JP7398227B2 - Work support systems and programs - Google Patents

Description

The present invention uses technology such as VR (Virtual Reality), AR (Augmented Reality), or MR (Mixed Reality) (hereinafter simply referred to as "VR etc.") to remotely The present invention relates to a work support system and program that can efficiently give instructions to workers.

2. Description of the Related Art Conventionally, systems have been proposed in which a work supporter gives instructions to workers on site using technology such as VR.

For example, in Patent Document 1, an image observed by a worker is transmitted to an instructor (supporter) located at a remote location, and the instructor provides three-dimensional work support to the worker while observing this image. system is proposed. In this system, both the instructor and the worker carry a stylus equipped with a magnetic sensor and wear a head mounted display (HMD) equipped with a three-dimensional position and orientation sensor. Then, data indicating the position/posture relationship between the three-dimensional position/posture sensor of the HMD of the instructor and the stylus used by the instructor is transmitted from the instructor to the worker. On the worker's side, based on this data, a pointer indicating the instructor's stylus is displayed virtually according to the worker's viewpoint. The stylus is provided with a button, and by pressing this button, the virtual object can be moved.

JP2006-209664A

However, according to the technique disclosed in Patent Document 1, both the operator and the instructor hold the stylus in their hands to perform work or give instructions, which limits the content of the work. In addition, not only the worker but also the supporter must wear the HMD, which becomes complicated.

The present invention has been made in response to such conventional circumstances, and it is not necessary to use a stylus at the work site, and the work support person does not need to wear an HMD, and can provide work support for work using various tools. The purpose is to provide a work support system and program that allows for.

In order to achieve the above purpose, the work support system according to the present disclosure:
A work support system comprising a work support device for a work supporter to give instructions to a worker in a remote location, and a head-mounted display device worn by the worker,
The work support device includes a hand position photographing unit that photographs the hands and fingers of a supporter, a gesture determination unit that calculates position or movement information of the photographed hands and fingers of the supporter, and a gesture determination unit that calculates position or movement information of the hands and fingers of the supporter, and a gesture determination unit that calculates position or movement information of the photographed hands and fingers of the supporter. a hand position information transmission unit that transmits to the head-mounted display device; a video reception unit that receives video information sent from the head-mounted display device; and a video display unit that displays the received video information; Equipped with
The head-mounted display device includes a transparent display section that displays a virtual image superimposed on a transparently displayed image of the real space, and position or movement information of the supporter's fingers transmitted from the work support device. a virtual hand display section that displays a virtual image of the supporter's hand and fingers on the transparent display section based on the position or movement information of the supporter's hand; and the transparent display. The present invention is characterized by comprising a video transmitter that transmits an image of the real space and a virtual video displayed on the work support device as video information to the work support device.

According to the present disclosure, a virtual image of the hands and fingers of a work supporter is superimposed on a transparently displayed image (scene) of the real space on a head-mounted display device (for example, a transparent head-mounted display). Provide support.

Further, the head-mounted display device includes an own hand detection unit that detects the worker's fingers, a worker's hand detected by the own hand detection unit, and the virtual hand display unit that displays the worker's hand and fingers. The present invention includes a hand overlap determination unit that determines whether the supporter's fingers overlap with the virtual image, and an overlap determination display unit that outputs the determination result of the hand overlap determination unit.

In particular, the virtual hand display section sets a reference position of the virtual image of the hand and fingers based on the position of the worker's fingers when the hand overlap determination section determines that there is an overlap, and then the hand position information reception section The virtual image is moved on the transparent display section based on the reference position and the position or movement information of the hand and fingers received by the user.

This absorbs the difference between the spatial coordinate system of the work support device and the spatial coordinate system of the head-mounted display device at the site, and moves the virtual image of the work support person's hands and fingers in the coordinate system of the head-mounted display device. This makes it possible to support work on objects at the site. Further, the work support person does not need to wear a head-mounted display, and can give instructions to the workers at the site while visually checking the situation on the site via a general-purpose computer monitor.

The work support device also includes a video recording unit that records received video information. At this time, it is preferable that, when the video recording section detects the movement of a specific finger of the supporter, it records the video information for a predetermined period of time before and after the time of detection.

Further, in the work support system according to the present disclosure, when the gesture determination unit detects a specific finger movement, the hand position information transmission unit transmits a drawing start command or end command to the head-mounted display. send to the device,
When the head-mounted display device receives a drawing start command, it draws a virtual image on the transparent display section according to the subsequent hand and finger movements, and when it receives a drawing end command, it draws the virtual image. It is characterized by ending.
This enables more specific work support.

Furthermore, in the work support system according to the present disclosure,
The work support device includes a tool information storage unit that stores tool information for displaying a virtual image of the tool, and a tool information selection unit that selects the tool information, and the hand position information transmission unit transmitting the tool information selected by the information selection unit to the head-mounted display device;
Upon receiving the tool information, the head-mounted display device displays a virtual image of the tool based on the tool information, superimposed on a virtual image of the supporter's fingers displayed by the virtual hand display section. It is characterized by being equipped with a virtual tool display section.

In particular, the virtual tool display section includes a storage means for storing a virtual image of the supporter's hand and a virtual image of the tool in association with corresponding locations (for example, points of contact between the tool and the fingers), and the virtual tool display section stores the virtual image of the tool. Preferably, the image is displayed by rotating or deforming the virtual image of the tool so that the virtual image of the supporter's fingers and the corresponding location overlap. As a result, the virtual image of the tool moves with the movement of the supporter's fingers, making it possible to provide more realistic work support.

Further, in the head-mounted display device worn by a worker according to the present disclosure, a method for instructing a worker to perform a task based on information sent from a task support device used by a task supporter is as follows:
In the work support device, acquiring position or movement information of the supporter's fingers and transmitting the information to the head-mounted display device;
displaying a virtual image of the supporter's hand and fingers on a transparent display section based on the position or movement information of the supporter's finger;
detecting an overlap between an image of the worker's hand and fingers displayed on the transparent display section and the virtual image;
If an overlap is detected, a step of associating the coordinates of a predetermined part of the worker at the time when the overlap is first detected with the position or movement information of the supporter's fingers at that time;
identifying and displaying the virtual image when an overlap is first detected;
It is characterized by including.

Preferably, the head-mounted display device is capable of detecting a distance to the work object, and if the distance is greater than a predetermined threshold, the head-mounted display device transmits position or movement information of the supporter's fingers to the worker. It is used as the relative position or amount of movement based on the position of a predetermined part (including the head-mounted display device worn by the worker) of the It is preferable to use the position or movement information of the supporter's hand or finger as the relative position or movement amount with respect to the world coordinates of the virtual image at the time when it is detected that the distance becomes equal to or less than the threshold value.

In the above, the coordinates of a predetermined part of the worker may be, for example, the coordinates of the worker's fingers, or the coordinates of the position of, for example, an infrared distance sensor of a wearable display device worn by the worker. good. Further, the association may be associated with each data and held, or may be obtained by determining a conversion parameter from the coordinates on the work support device side to the coordinates on the wearable display device side.

In addition, the program related to this disclosure is
A head-mounted display that includes a transparent display section that receives position or movement information of the hands and fingers of a work supporter and displays a virtual image of the supporter's hands and fingers superimposed on a transparently displayed image of the real space. A computer-executable program running on a device, the program comprising:
Own hand detection processing that detects the worker's fingers;
hand overlap determination processing that determines whether or not there is an overlap between the worker's hand detected by the own hand detection processing and a virtual image of the supporter's hand and fingers;
The position of the hand and finger of the worker when the hand overlap determination process determines that there is an overlap is set as the reference position of the virtual image of the hand, and the position or movement information of the hand and finger that is subsequently received is used in the transparent display section. virtual hand display processing for moving the virtual image;
It is characterized by carrying out.

According to the present disclosure, a work support person does not need to wear an HMD, and can perform work support for work using various tools with simple equipment.

FIG. 1 is a functional block diagram of a work support system according to an embodiment of the present invention. It is an explanatory diagram of calibration processing in an embodiment of the present invention. FIG. 2 is an explanatory diagram of the operation of the hand overlap determining section in FIG. 1; FIG. 2 is an explanatory diagram of the operation of the virtual tool display section of FIG. 1; 3 is a flowchart showing a procedure of calibration processing according to an embodiment of the present invention. 12 is a flowchart illustrating an example of a procedure for moving a virtual image of a supporter's fingers after calibration according to an embodiment of the present invention.

A work support system according to an embodiment of the present invention will be described below with reference to the drawings. Note that a transmissive head-mounted display (hereinafter simply referred to as "HMD") is used as an example of the head-mounted display device.

In FIG. 1, a work support system 1 includes a work support device 50 for use by a work supporter to instruct on-site workers to perform work, and an HMD 10 worn by on-site workers. The work support device 50 and the HMD 10 are connected via a wired or wireless communication line 2.

The work support device 50 detects the position and movement of the work supporter's fingers, transmits the position and movement to the HMD 10 on the worker side, receives video information sent from the HMD 10, and displays the video. It is displayed in section 51. Further, the work support device 50 allows the worker to select a tool to be used. Identification information of the selected tool is sent to the HMD 10. The work support device 50 having such a function can be realized using a general-purpose computer device. Existing motion gesture tracking technology can be used to detect the position and movement of fingers. Examples include Japanese Patent Publication No. 2009-525538 and "LEAP MOTION" (registered trademark) by Leap Motion, Inc. of the United States.

The HMD 10 receives the movement information of the hand and fingers of the work supporter sent from the work support device 50, and displays a virtual image of the hand and fingers superimposed on the real image. Further, upon receiving the identification information of the tool, the HMD 10 virtually displays the hand and fingers of the worker holding the tool. The virtual image and the real image are combined and sent to the work support device 50.

(composition)
The configuration of each device 50, 10 will be described in detail below.
The work support device 50 includes a hand position photographing unit 52 for photographing the hand and fingers of the work supporter, a gesture determination unit 54 for calculating position or movement information of the photographed hand, and transmitting position information of the supporter's hand and fingers to the HMD 10. A hand position information transmitting unit 55 that stores information on tools used for work, a tool information storage unit 1 (61) that stores a list of tool information used for work, a tool selection unit 56 that selects a tool from the tool information storage unit 1 (61), and a hand position information transmission unit 55 that stores information on tools used for work. It includes a video receiving section 53 that receives incoming video information, a video display section 51 that displays the received video information, and a video recording section 60 that records the video information.

Note that a general-purpose display device can be used as the video display section 51. Further, the hand position photographing section 52 and the gesture determining section 54 can use the technique of the motion gesture tracking device described above.

The HMD 10 includes a transparent display unit 11 that superimposes and displays a virtual image on a transparently displayed real image of the real space, and a work support device that uses the image of the real space and the virtual image displayed on the transparent display unit 11 as video information. 50; a hand position information receiving unit 13 that receives the position or movement information of the work supporter's fingers transmitted from the work support device 50; a virtual hand display section 14 that displays a virtual image of the worker's hand and fingers on the transparent display section 11 based on the information, a self-hand detection section 16 that detects the worker's hand and fingers displayed on the transparent display section 11, and a self-hand detection section A hand overlap determination section 17 determines whether or not there is an overlap between the worker's hand and fingers detected by the virtual hand display section 16 and the virtual image of the worker's hand and fingers displayed on the virtual hand display section 14, and the result of this determination is displayed on a transparent display section. 11, the tool information storage section 2 (20) that stores tool information for displaying a virtual image of the tool, and the tool information sent from the work support device 50. A virtual image of the tool is displayed on the transparent display section 11 based on the tool information stored in the tool information storage section 2 (20), superimposed on the virtual image of the hand and fingers of the work supporter displayed by the virtual hand display section 14. A virtual tool display section 15 is provided to display the virtual tool.

In addition to the above functions, the HMD 10 includes a three-dimensional position and orientation sensor and a distance sensor (not shown). Conventional techniques can be used for these, and the position of an object in real space can be acquired in a three-dimensional coordinate system. For example, an infrared distance sensor can be used as the distance sensor. After activating the HMD 10, the operator can obtain distances and three-dimensional coordinates of surrounding objects by rotating around himself/herself.

In the above, each section 12 to 18 in the HMD 10 and each section 53 to 56 in the work support device 50 can be realized by a program as a computer function.

(motion)
First, a description will be given of a calibration process performed at an initial stage when the HMD 10 and the work support device 50 are connected for communication. This process is a process of replacing the hand and finger movements of the work supporter on the work support device 50 side with the coordinate system on the HMD 10 side. By performing this processing, the hand and fingers 70 of the work supporter can be virtually displayed at appropriate positions in the real image displayed on the transparent display section 11 of the HMD 10, and effective work support can be achieved.

1. Calibration processing (overview of calibration processing)
As shown in FIG. 2, when the work supporter holds his hand above the hand position imaging unit 52, the gesture determination unit 54, which operates using the above-described motion gesture tracking technology, captures the position information of the supporter's fingers. This position information is transmitted to the HMD 10 by the hand position information transmitter 55. When the hand position information receiving unit 13 of the HMD 10 receives this position information, the virtual hand display unit 14 displays it at a predetermined position on the transparent display unit 11. The virtual hand display section 14 also passes the positional coordinates of the supporter's fingers to be virtually displayed to the hand overlap determination section 17 .

As shown in FIG. 3, the worker places his or her hand over the virtually displayed hand of the supporter. At the time of calibration, it is preferable to determine in advance so that the positional relationship between the supporter's line of sight and the supporter's hand is approximately the same as the positional relationship between the worker's line of sight and the worker's hand.

When the own hand detection section 16 of the HMD 10 detects the own hand and fingers displayed on the transparent display section 11, it passes the coordinates to the hand overlap determination section 17. The hand overlap determination unit 17 determines whether or not the two overlap, based on the coordinates of the hand and fingers of the supporter's own hand and the coordinates of the virtually displayed hand and fingers of the supporter.

As a result of this determination, if it is determined that they overlap, the hand overlap determination unit 17 uses the positional coordinates of the supporter's fingers on the work support device 50 side received by the hand position information receiving unit 13 and the virtual display on the HMD 10. The positional coordinates of the supporter's fingers 30, that is, the positional coordinates of the worker's fingers 40 are stored in a storage unit (not shown) in association with each other. Further, the positional coordinates of the supporter's fingers 30 on the HMD 10 at this time (namely, the positional coordinates of the worker's fingers 40) can be used as reference coordinates when moving the supporter's fingers.

For example, when the work supporter moves his or her fingers after the calibration process, the virtual hand display unit 14 receives the position coordinates of the hand and fingers after the movement sent from the work support device 50, and displays the previously received position coordinates of the work supporter. The amount of change from the position coordinates of the hand and fingers is changed from the reference coordinates. The coordinates after this change become the current position of the supporter's fingers on the HMD 10 after the movement. When hand and finger movement information (change direction and change amount) is sent from the work support device 50, the coordinates moved by the change direction and change amount with respect to the reference coordinates are the supporter's hand and finger on the current HMD 10. position. Note that the reference coordinates are not limited to the above, and may be based on, for example, the relative position from the infrared distance sensor of the HMD 10.

(Details of calibration process)
On the side of the work support device 50, the hand position imaging unit 52 is preferably placed at a position where movement of the supporter's hand in any direction within the working range can be detected. As an example, the supporter places the video display section 51 in front of himself and places the hand position photographing section 52 below the center of his arm when his hand is extended toward the front (see FIG. 2). The supporter may sit on a chair and place the motion gesture on a desk or on the floor. The hand position photographing section 52 is arranged as described above, and the hand is held above the hand and the palm is spread out. This position of the palm becomes the position at the time of calibration (neutral position).

Hereinafter, the procedure of the calibration process according to this embodiment will be described with reference to FIG. 5.

When the gesture determination unit 54 detects a finger, it transmits the position coordinates to the HMD 10 (S101, S102). Note that the positional coordinates transmitted from the work support device 50 to the HMD 10 during calibration may be any spatial coordinates managed by the gesture determination unit 54. For example, it may be the origin (0, 0, 0) or other coordinates.

When the HMD 10 receives the position coordinates of the hand and fingers sent from the work support device 50, the HMD 10 determines the coordinates in real space so that the coordinates are displayed at a predetermined position determined in advance for display on the transparent display unit 11 (S103 ).

For example, suppose that the left and right, top and bottom, and depth of the worker side (HMD 10 side) and the supporter side (work support device 50 side) are represented by three-dimensional coordinates (X, Y, Z) and (x, y, z), respectively. , calculates the coordinates (X1, Y1) of the center of the worker's visual field when the coordinates of the hand and fingers are first received, and converts these coordinates (X1, Y1) into the coordinates (x1, Y1) sent from the work support device 50. y1). Regarding the depth (Z-axis) direction, a predetermined distance (Z1) in the depth direction as seen from the operator is made to correspond to the coordinates (z1) sent from the work support device 50. Note that the predetermined distance in the depth direction (Z1) is preferably set to match the distance in the depth direction from the supporter's viewpoint to the neutral position.

In the above example, if the coordinates (x1, y1) are the coordinates of the middle position of both hands of the supporter, the hands and fingers of the supporter can be virtually displayed at positions equidistant to the left and right in the visual field of the worker. The virtual display is not limited to this, and it is sufficient that the supporter's hands are displayed within the field of view of the worker while maintaining the distance between them. Note that the coordinate system on the HMD 10 side can be the world coordinate system, but the three-dimensional coordinates (X, Y, Z) on the HMD 10 side at the time of calibration are determined by the operator (for example, an infrared distance sensor mounted on the HMD 10). It is preferable to use a coordinate system with the reference (origin) as the reference point.

As described above, the coordinates (x1, y1, z1) of the supporter's fingers on the work support device 50 side are converted to the coordinates (X1, Y1, Z1) on the HMD 10 side, and the supporter's fingers are virtually placed at the position of the coordinates. Display. Note that the above explanation was given using one point among the coordinates of the hand and fingers as an example, but when the coordinates of the hand and fingers are expressed by multiple feature points, any one point is transformed as described above, and for the other feature points, The coordinate system on the work support device 50 side can be converted to the coordinate system on the HMD 10 side using relative coordinates from the one point.

The method of coordinate transformation during calibration has been described above, but the coordinate system is not limited to the above, and for example, the dimension may be increased by further adding rotation amounts with respect to each axis. Further, movement information (direction and amount of movement) may be sent instead of or together with the position coordinates.

(Alignment with worker's fingers)
As shown in FIG. 3, the worker places his or her own hand on the supporter's hand that is virtually displayed on the transparent display section 11.

When the worker's own hand detection unit 16 of the HMD 10 detects the worker's own hand displayed on the transparent display unit 11, it acquires the coordinates thereof (S104). If the coordinates of the worker's hand and the virtually displayed coordinates of the supporter's hand are within a certain range, the hand overlap determination unit 17 determines that the worker's hand and the supporter's fingers overlap (S105). .

Note that the feature points of the worker's hand and fingers detected by the own hand detection unit 16 of the HMD 10 and the feature points of the supporter's hand and fingers detected by the gesture determination unit 54 of the work support device 50 do not need to completely match. For example, it may be determined that they overlap when the average value of the coordinates of each feature point on the fingers of the worker and the support person falls within a certain range. When determining that the hands and fingers of the worker and the supporter overlap, the overlap determination display unit 18 identifies and displays a virtual image of the supporter's fingers so that it can be seen that an overlap has been detected (S106).

The hand overlap detection unit 16 detects that the positions of the worker's hand and the supporter's hand overlap, and determines the position coordinates of the worker's fingers on the HMD 10 at this time (position coordinates of the worker's fingers). , are registered in the storage unit as the reference coordinates of the supporter's fingers (S107). This allows the supporter to move his/her own fingers within the space of the HMD 10 in subsequent instruction operations.

Note that there are three ways to move the supporter's fingers in the space of the HMD 10 as follows.
1) The position of the supporter's fingers is expressed in relative coordinates with respect to the worker's position, and the supporter's fingers are moved as the worker moves. Furthermore, the supporter moves his or her fingers in the space on the HMD 10 side in the amount of movement and in the direction of movement.
2) Represent the supporter's fingers in absolute coordinates (world coordinate system) on the HMD side. In this case, the supporter's fingers do not move as the worker moves.
3) Only the depth (for example, the Z-axis direction) as seen from the worker is expressed by the amount of movement on the HMD side from the Z coordinate of the worker as a reference. The left, right, top, and bottom (X, Y directions) are based on reference (absolute) coordinates at the time of calibration.

Which method to use may be initialized in advance, or may be changed by the supporter's selection.

An example of how to move the virtual image of the supporter's fingers after calibration will be described below with reference to FIG. 6. A feature of this process is that the coordinate system 1) or 3) and the coordinate system 2) are switched depending on the distance between the worker and the work object. Although this coordinate system switching function can be provided independently as a function of the HMD 10, it may also be realized as a function of the virtual hand display section 14. The functions of the virtual hand display section 14 will be explained below.

When the virtual hand display unit 14 receives the position or movement information of the fingers from the hand position information receiving unit 13, it executes the following process.

First, the distance between the worker and the work object measured by an infrared distance sensor mounted on the HMD 10 is acquired (S201). Whether the object is a work target or not can be determined by a gesture such as a pointing finger by the worker or the support person. Note that the distance between the worker and the work object may be, for example, the distance between the worker's fingers and the work object, or the distance between the supporter's virtual image position and the work object.

Next, it is determined whether the distance acquired in step S201 is less than or equal to a threshold value (S202), and if no, that is, if the worker is a certain distance from the work object, the position of the worker (HMD 10) is determined. The reference coordinates (hereinafter referred to as reference coordinates A) are set (S203). Then, a virtual image of the supporter's hand and fingers is identified and displayed so that it can be seen that the reference coordinate A is set (S204), and the position or movement information of the hand and finger received by the hand position information receiving unit 13 is displayed in the space of the HMD 10. The virtual image is displayed as a movement from the reference coordinate A (S205).

On the other hand, in step S202, if the distance is less than the threshold value, that is, if the worker approaches the work object, the world coordinates of the supporter's fingers at the time when the distance becomes less than the threshold value are set as reference coordinates (hereinafter referred to as reference coordinates). B) (S206). Then, a virtual image of the supporter's hand and fingers is identified and displayed so that it can be seen that the reference coordinate B is set (S207), and the position or movement information of the hand and finger received by the hand position information receiving unit 13 is then displayed in the space of the HMD 10. The virtual image is displayed as a movement from the reference coordinate B (S208).

Through the above processing, by moving the virtual image of the supporter's fingers, the virtual image will follow the worker until the worker approaches the work object, and when the worker approaches the work object, the virtual image will also move. The virtual image is separated from the movement of the worker, allowing stable movement of the work object.

The transparent display section 11 of the HMD 10 is a three-dimensional (3D) display, and the video display section 51 of the work support device 50 is a two-dimensional (2D) display. Although the problem is whether to make the work support person recognize it, the above method allows the position and movement of the hand and fingers of the support person on the work support device 50 side to be appropriately represented in the space on the HMD 10 side.

2. Tool Selection and Display Processing The supporter selects a tool in the work support device 50. To select a tool, a palette may be displayed and the tool selected from a list. When the tool selection section 56 detects a movement or shape of a finger, the tool information storage section stores the identification information of the tool corresponding to the movement. 1 (61) and transmit it to the HMD 10.

When the HMD 10 receives the identification information of the tool, the virtual tool display unit 15 extracts virtual image information corresponding to the identification information of the tool from the tool information storage unit 2 (20), and displays the image of the hand and fingers holding the tool. A virtual image is displayed on the transparent display section 11.

The virtually displayed video and the real video are combined, transmitted to the work support device 50, and displayed on the video display section 51.

As shown in FIG. 4, the tool has at least two finger feature points associated with a predetermined position of the tool. Then, based on the positional relationship of the feature points of the fingers, a predetermined position of the tool is aligned with the coordinates of the feature points of the fingers, and the entire tool is displayed.

To explain in detail using FIG. 4, feature points A and B on the tool and the fingers are associated with each other, and are the positions where the tool comes into contact with the fingers when the tool is gripped. Then, in the state before moving the tool in Fig. 4(a), the coordinates of point A are (X1, Y1, Z1), the coordinates of point B are (X2, Y2, Z2), and the tool in Fig. 4(b) is Suppose that the coordinates of point A in the state after moving are (X1', Y1', Z1') and the coordinates of point B are (X2', Y2', Z2').

At this time, the virtual hand display unit 14 calculates the coordinates in real space (the space on the HMD 10 side) using the position coordinates of the worker's fingers received by the hand position information receiving unit 13, and displays the worker's fingers in a virtual manner. indicate. The virtual tool display unit 15 acquires the coordinates of the feature points A and B from among the coordinate information used for virtual display in the virtual hand display unit 14. In the state shown in Fig. 4(a), the tool is extended a predetermined length in the direction of point A on a straight line connecting the coordinates of point A (X1, Y1, Z1) and the coordinates of point B (X2, Y2, Z2). virtual display. When transitioning to the state shown in FIG. 4(b), point A is located on the straight line connecting the coordinates of point A (X1', Y1', Z1') and the coordinates of point B (X2', Y2', Z2') The tool is virtually displayed by extending it a predetermined length in the direction. Note that if movement in the rotational direction about a straight line is important, it is preferable to add more feature points or increase the dimension.

3. Work Support When the worker approaches the work object, the supporter's hands and fingers also approach. Then, when the supporter extends his hand, the virtually displayed hand and fingers move in the direction of the work target. Note that an alarm may be output when the worker's hand and the virtually displayed supporter's hand are separated by a certain distance or more.

Note that when the distance between the virtually displayed hand and the tool and the real work object becomes less than a certain value, or when contact is detected, the virtually displayed hand and the tool may be identified and displayed. This allows the support person to know the distance to the work target.

The worker grasps the same tool as the virtually displayed tool and places his or her own finger over the virtually displayed finger. This allows the worker to learn how to hold and use the tool from the virtually displayed image.

As described above, according to the present embodiment, the reference position for the movement of the hand and fingers of the work supporter is determined in the real space of the worker side immediately after starting both the devices 10 and 50 before the work, so that the subsequent work can be performed. Able to provide support and instructions easily and efficiently. Further, the work support person does not need to wear a 3D display device such as an HMD, and can support work using various tools with simple equipment.

The present invention is not limited to the embodiments described above, and can be implemented with various modifications without departing from the gist thereof.

For example, in the work support device 50, when the gesture determination unit 54 detects a specific finger movement, the hand position information transmission unit 55 transmits a drawing start command or end command to the head-mounted display device. When the HMD 10 receives the drawing start command, it draws the virtual image on the transparent display unit 11 according to the subsequent hand and finger movements, and when it receives the drawing end command, it finishes drawing the virtual image. It's okay.

Furthermore, when the gesture determination unit 54 detects a specific finger movement, the video information received from the HMD 10 may be recorded for a certain period of time before and after.

As a simpler process for the virtual tool display unit 15, virtual image information of the hand holding the tool is stored in the tool information storage unit 2 (20) in advance, and the position information received by the hand position information receiving unit 13 is Based on this, this virtual image may be rotated, transformed, etc., and displayed.

1 Work support system 2 Communication line 10 Head-mounted display device (HMD)
11 Transparent display section 12 Video transmission section 13 Hand position information reception section 14 Virtual hand display section 15 Virtual tool display section 16 Own hand detection section 17 Hand overlap determination section 20 Tool information storage section 2
30 Virtually displayed supporter's fingers 31 Virtually displayed tools 40 Worker's fingers in real space 50 Work support device 51 Video display section 52 Hand position photographing section 53 Video receiving section 54 Gesture determining section 55 Hand position information transmitting section 56 Tool selection section 60 Video recording section 70 Hands and fingers of the work supporter on the work support device side

Claims (7)

A work support system comprising a work support device for a work supporter to give instructions to a worker, and a head-mounted display device worn by the worker, the work support system comprising:
The work support device includes:
a hand position photographing unit that photographs the hands and fingers of the supporter;
a gesture determination unit that calculates position or movement information of the photographed fingers of the supporter;
a hand position information transmitting unit that transmits position or movement information of the hand and fingers to the head-mounted display device;
a video receiving unit that receives video information sent from the head-mounted display device;
a video display unit that displays the received video information;
a tool information storage unit that stores tool information for displaying a virtual image of the tool;
a tool information selection unit that selects the tool information;
Equipped with
The head-mounted display device includes:
a transparent display section that superimposes and displays a virtual image on the transparently displayed image of the real space;
a hand position information receiving unit that receives position or movement information of the supporter's fingers transmitted from the work support device;
a virtual hand display unit that displays a virtual image of the supporter's fingers on the transparent display unit based on position or movement information of the supporter's fingers;
a video transmitting unit that transmits the real space image and virtual video displayed on the transparent display unit as video information to the work support device;
Equipped with
The hand position information transmitting unit transmits the tool information selected by the tool information selecting unit to the head-mounted display device,
Upon receiving the tool information, the head-mounted display device displays a virtual image of the tool based on the tool information, superimposed on a virtual image of the supporter's fingers displayed by the virtual hand display section. A work support system characterized by being equipped with a virtual tool display section . The virtual image of the supporter's fingers and the virtual image of the tool are associated with corresponding locations, and the virtual tool display section displays the virtual image of the tool with the virtual image of the supporter's fingers. 2. The work support system according to claim 1, wherein the virtual image of the tool is displayed while being rotated or transformed so that the corresponding locations overlap. The head-mounted display device includes:
an own hand detection unit that detects the worker's fingers;
a hand overlap determination unit that determines whether there is an overlap between the worker's hand detected by the own hand detection unit and a virtual image of the supporter's hand and fingers displayed by the virtual hand display unit;
an overlap determination display unit that outputs the determination result of the hand overlap determination unit;
The work support system according to claim 1 or 2, further comprising: The virtual hand display section sets a reference position of the virtual image of the hand and fingers based on the position of the worker's hand and fingers when the hand overlap determination section determines that there is an overlap, and the virtual hand image is then received by the hand position information reception section. 4. The work support system according to claim 3 , wherein the virtual image is moved on the transparent display section based on the position or movement information of the hand and fingers and the reference position. The work support system according to any one of claims 1 to 4 , wherein the work support device includes a video recording unit that records the received video information. In the work support device, when the gesture determination unit detects a specific finger movement, the hand position information transmission unit transmits a drawing start command or end command to the head-mounted display device,
When the head-mounted display device receives a drawing start command, it draws a virtual image on the transparent display section according to the subsequent hand and finger movements, and when it receives a drawing end command, it draws the virtual image. The work support system according to any one of claims 1 to 5 , characterized in that: A head-mounted display that includes a transparent display section that receives position or movement information of the hands and fingers of a work supporter and displays a virtual image of the supporter's hands and fingers superimposed on a transparently displayed image of the real space. A computer-executable program running on a device, the program comprising:
Own hand detection processing that detects the worker's fingers;
hand overlap determination processing that determines whether or not there is an overlap between the worker's hand detected by the own hand detection processing and a virtual image of the supporter's hand and fingers;
The position of the hand and finger of the worker when the hand overlap determination process determines that there is an overlap is set as the reference position of the virtual image of the hand, and the position or movement information of the hand and finger that is subsequently received is used in the transparent display section. virtual hand display processing for moving the virtual image;
A program characterized by executing.

Images