JP2020086630A - Work assisting system and program - Google Patents

Abstract

To provide a work assisting system which can improve efficiency of first work which a first user and a second user perform in coordination with each other.SOLUTION: A work assisting device is the work assisting system which causes a first display device wearable on the head of a first user and a second display device installed in a preliminarily determined room to display images and assists the first user and a second user in performing first work in coordination with each other. When the first display device is located in a first actual position out of positions within a first actual space, a first virtual object is disposed in a position corresponding to the first actual position out of positions within a virtual space and the second display device is caused to display the first virtual object, and a second virtual object is disposed in a second virtual position out of the positions within the virtual space and the first display device is caused to display the second virtual object. The first virtual object is a virtual object indicative of the first user, and the second virtual object is a virtual object indicative of the second user.SELECTED DRAWING: Figure 5

Description

The present invention relates to a work assistance system and a program.

Research and development are being conducted on devices that assist human work.

In this regard, there is known a device that assists a work performed by a worker using mixed reality (see Patent Document 1).

JP, 2017-044496, A

Here, the worker may be assisted by an assistant in a space different from the space in which the worker works. However, the conventional device as described in Patent Document 1 displays a virtual object on a transmissive display device worn by the worker on the head, and provides the worker with a space in which the virtual object is superimposed on the real space. It just makes it visible. For this reason, it may be difficult for the apparatus to improve the efficiency of the work performed by the cooperation between the worker and the assistant. Here, in this specification, a real space is a real space. Further, in this specification, the virtual space is a virtual three-dimensional space. Further, in the present specification, the virtual object is a virtual object arranged in the virtual space.

The present invention has been made in consideration of such circumstances, and provides a work assistance system and a program capable of improving the efficiency of the first work performed by the first user and the second user in cooperation with each other. This is an issue.

According to one aspect of the present invention, an image is displayed on a first display device that can be worn on the head of a first user and a second display device installed in a predetermined room, and the first user and the second user can display the image. A work assistance system for assisting a first work performed in cooperation with a user, wherein the first display device includes a transmissive first display unit and an imaging unit for imaging a real space, and the imaging A virtual space corresponding to the first real space is generated based on a captured image in which a predetermined first real space is captured by the unit, and each position of the generated virtual space is calculated based on the captured image. A virtual space generation unit that associates each position in the first real space with a virtual object located at a first virtual position among the positions in the virtual space; In the virtual space, according to a first display control unit that causes the first display device to display the first user so as to be visually recognized by the first user at a position corresponding to the first virtual position, and an operation received from the second user. A second display control unit that causes the second display device to display the virtual space as viewed from the line-of-sight direction of a second virtual position corresponding to the operation. A second virtual object is arranged at the second virtual position of the positions in the virtual space and the second virtual object is displayed on the first display device, and the second display control unit is configured to display the first real object. When the first display device is located at the first real position of the positions in the space, the first virtual object is arranged at a position corresponding to the first real position of the positions in the virtual space. Causing the second display device to display the first virtual object, the first virtual object being a virtual object representing the first user, and the second virtual object being a virtual object representing the second user. , A work assistance system.

Further, according to an aspect of the present invention, in the work assistance system, an image processing unit that specifies a position of the first display device within the first actual position based on the captured image is provided, and the first display control unit is provided. May arrange the first virtual object in the virtual space and display the first virtual object on the second display device based on the position specified by the image processing unit. Good.

Further, according to an aspect of the present invention, in the work assistance system, the image processing unit specifies a line-of-sight direction of the first display device in the first real space based on the captured image, and performs the first display. A control unit arranges the first virtual object in the virtual space and causes the second display device to display the first virtual object based on the position and the line-of-sight direction specified by the image processing unit. May be used.

Further, according to an aspect of the present invention, in the work assistance system, when the captured image includes the hand of the first user, the image processing unit determines the hand of the first user based on the captured image. Of the third virtual object indicating the hand of the first user together with the first virtual object in the virtual space by further specifying the position of the third virtual object in the first real space. A configuration for displaying one virtual object and the third virtual object may be used.

Further, according to one aspect of the present invention, in the work assistance system, when a first display operation for displaying a fourth virtual object on the first display device is received from the first user, the first display control unit and the second display operation are performed. A configuration may be used in which the display control unit arranges the fourth virtual object at a position corresponding to the first display operation among the positions in the virtual space.

Further, according to one aspect of the present invention, in the work assistance system, a server including the virtual space generation unit is provided, the first display device includes the first display control unit, and the second display device includes the A configuration may be used in which a second display control unit is provided, and the server is communicably connected to each of the first display device and the second display device.

Further, according to an aspect of the present invention, in the work assistance system, a configuration including a server including the virtual space generation unit, the first display control unit, and the second display control unit may be used.

Further, according to an aspect of the present invention, in the work assistance system, the first display control unit creates a virtual object that virtually surrounds the first display device based on a virtual operation received from the first user. When the virtual object that is arranged in the virtual space and that virtually surrounds the first display device is arranged in the virtual space, the second display control unit virtualizes the virtual object that surrounds the second virtual position. The function of the virtual object arranged in the space and virtually surrounding the first display device is the same as the function of the virtual object surrounding the second virtual position, and virtually surrounding the first display device. The virtual object and the virtual object that virtually surrounds the second display device have the same shape, and the display mode of the virtual object that virtually surrounds the first display device is displayed on the first display device. And a display mode of the virtual object virtually surrounding the second display device on the second display device may be the same display mode.

Further, according to an aspect of the present invention, in the work assistance system, respective shapes of a virtual object virtually surrounding the first display device and a virtual object virtually surrounding the second display device are the same cylinder. A configuration that is a shape may be used.

Further, according to one aspect of the present invention, an image is displayed on a first display device that can be worn on the head of the first user and a second display device installed in a predetermined room, and the first user and A program for controlling a work assistance system for assisting a first work performed in cooperation with a second user, wherein the first display device includes a transmissive first display unit, and an imaging unit for imaging a real space. A virtual space corresponding to the first real space is generated based on a captured image in which the predetermined first real space is captured by the image capturing unit, each position of the generated virtual space, and the captured image. A virtual space generating step of associating each position of the first real space calculated based on an image, and a virtual object arranged at a first virtual position of the positions in the virtual space with the first real space. A first display control step of displaying on the first display device so as to be visually recognized by the first user at a position corresponding to the first virtual position among the positions inside; and an operation received from the second user. A second display control step of causing the second display device to display the inside of the virtual space viewed from the line-of-sight direction of the second virtual position corresponding to the operation among the positions in the virtual space. A program, wherein the first display control step arranges a second virtual object at the second virtual position of the positions in the virtual space, and causes the first display device to display the second virtual object, In the second display control step, when the first display device is located at a first real position among the positions within the first real space, the second display control step is performed at the first real position among the positions within the virtual space. A first virtual object is arranged at a corresponding position to cause the second display device to display the first virtual object, the first virtual object being a virtual object indicating the first user, and the second virtual object. Is a program that is a virtual object representing the second user.

According to the present invention, it is possible to improve the efficiency of the first work performed by the first user and the second user in cooperation with each other.

It is a figure which shows an example of a structure of the work assistance system 1 which concerns on embodiment. It is a figure which shows an example of the hardware constitutions of the 1st display apparatus 10. 3 is a diagram showing an example of a hardware configuration of a second display device 20. FIG. It is a figure which shows an example of the hardware constitutions of the server 30. It is a figure showing an example of each functional composition of the 1st display 10, the 2nd display 20, and a server 30. It is a figure which shows an example of the relative positional relationship of virtual position X1 and virtual position X2 in virtual space VS. FIG. 7 is a diagram showing an example of a display surface of a first display section 11 when the positional relationship between the virtual position X1 and the virtual position X2 is the positional relationship shown in FIG. FIG. 7 is a diagram showing an example of a display surface of a second display unit 21 when the positional relationship between the virtual position X1 and the virtual position X2 is the positional relationship shown in FIG. It is a figure which shows an example of the display surface DP1 in case the virtual object VM1 is displayed on the display surface DP1. It is a figure which shows an example of the display surface DP2 in case the virtual object VM1 is displayed on the display surface DP2. It is a figure which shows an example of the display surface DP2 in case the virtual object VM2 is displayed on the display surface DP2. It is a figure which shows an example of the display surface DP1 in case the virtual object VM2 is displayed on the display surface DP1. It is a figure which shows an example of the flow of the process which the 1st display apparatus 10 superimposes virtual space VS on 1st real space RS. It is a figure which shows an example of the flow of the process which the 2nd display 20 displays the inside of virtual space VS. It is a figure which shows an example of the flow of the process which the 1st display apparatus 10 outputs 1st information to the 2nd display apparatus 20. It is a figure which shows an example of the flow of the process which the 2nd display apparatus 20 displays the 1st virtual object A1. It is a figure which shows an example of virtual object CM. It is a figure which shows an example of the display surface DP1 of the 1st display part 11 when the virtual object CM is arrange|positioned in the 1st virtual space VS1. It is a figure which shows an example of the display surface DP2 of the 2nd display part 21 when the virtual object CM is arrange|positioned in the 2nd virtual space VS2.

<Embodiment>
Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<Outline of work assistance system>
First, an outline of the work assistance system according to the embodiment will be described.

The work assistance system displays an image on a first display device that can be worn on the head of the first user and a second display device installed in a predetermined room so that the first user and the second user can Assist the first work done in cooperation. The first display device also includes a transmissive first display unit and an image capturing unit that captures an image of the real space. Here, the first work may be any work as long as the first user and the second user work together.

In addition, the work assistance system includes a virtual space generation unit, a first display control unit, and a second display control unit.

The virtual space generation unit generates a virtual space corresponding to the first real space based on the captured image in which the predetermined first real space is captured by the imaging unit. The virtual space generation unit associates each position in the generated virtual space with each position in the first real space calculated based on the captured image in which the first real space is captured by the imaging unit. That is, each position in the virtual space generated based on the captured image in which the first real space is captured by the imaging unit is the first position calculated based on the captured image in which the first real space is captured by the imaging unit. It is associated with each position in the real space.

The first display control unit visually recognizes the virtual object arranged at the first virtual position of the positions in the virtual space by the first user at the position corresponding to the first virtual position of the positions in the first real space. To be displayed on the first display device as described above.

In response to an operation received from the second user, the second display control unit causes the second display device to display the virtual space as viewed from the line-of-sight direction of the second virtual position corresponding to the operation in the virtual space. Display it.

Here, the first display control unit further arranges the second virtual object at the second virtual position of the positions in the virtual space and causes the first display device to display the second virtual object. Here, the second virtual object is a virtual object indicating the second user. Further, the second virtual position may be any position as long as it is a position corresponding to an operation received by the second display control unit from the second user among the positions in the virtual space.

In addition, the second display control unit further corresponds to the first real position of the positions in the virtual space when the first display device is located at the first real position of the positions in the first real space. The first virtual object is arranged at the position and the second virtual image is displayed on the second display device. Here, the first virtual object is a virtual object indicating the first user. Further, the first real position may be any position as long as it is a position in the first real space.

As a result, the work assistance system can improve the efficiency of the first work performed in cooperation with the first user and the second user. The configuration of such a work assistance system and the processing performed by the work assistance system will be described in detail below.

<Configuration of work assistance system>
Hereinafter, the configuration of the work assistance system 1 according to the embodiment will be described with reference to FIG. 1. FIG. 1 is a diagram showing an example of the configuration of a work assistance system 1 according to the embodiment.

Hereinafter, as an example, a case where the first user of the work assistance system 1 is the worker O shown in FIG. 1 will be described. Moreover, below, as an example, a case where the second user of the work assistance system 1 is the assistant H shown in FIG. 1 will be described.

The worker O is a person who performs the predetermined first work W1 together with the assistant H. The first work W1 is an example of the above-mentioned first work. The first work W1 may be any work as long as the work is performed by the worker O together with the assistant H. In the following, as an example, a case where the first work W1 is a working work on the object E shown in FIG. 1 will be described. The object E is an example of an object on which the worker O performs the first work W1. The assistant H is a person who assists the work of the worker O. The worker O may be configured so that three or more workers perform the first work W1.

In the example shown in FIG. 1, the worker O performs the first work W1 in the room R1. Further, in the example, the assistant H assists the first work W1 performed by the worker O in the room R2 different from the room R1. For example, the room R1 and the room R2 are different rooms installed in a building. That is, the assistant H is a person who assists the worker O from the room R2 located at a position distant from the room R1. Note that the room R1 and the room R2 may be rooms in different buildings, or may be different rooms in a single building.

The work assistance system 1 is an example of the work assistance system described above. The work assistance system 1 includes a first display device 10 that can be mounted on the head of the worker O, a second display device 20 installed in a predetermined room R2, and a server 30. The first display device 10 also includes a transmissive first display unit 11 and an imaging unit 12 that images the real space.

In the work assistance system 1, the first display device 10 and the server 30 may be integrally configured, or the second display device 20 and the server 30 may be integrally configured. The work assistance system 1 may be configured to include other devices in addition to the first display device 10, the second display device 20, and the server 30. The first display device 10 may have a configuration including other functional units in addition to the first display unit 11 and the imaging unit 12.

In the work assistance system 1, the first display device 10, the second display device 20, and the server 30 are communicably connected to each other by wireless communication. The wireless communication is wireless communication according to a standard such as a wireless LAN (Local Area Network) and a mobile communication network. The wireless communication may be wireless communication of another standard. Further, some or all of the first display device 10, the second display device 20, and the server 30 may be configured to be communicably connected to each other by wired communication.

In the work assistance system 1, the server 30 generates the virtual space VS corresponding to the first real space RS based on the captured image in which the predetermined first real space RS is captured by the image capturing unit 12 of the first display device 10. It Then, in the work assistance system 1, the virtual space VS generated by the server 30 is stored in each of the first display device 10 and the second display device 20. The first real space RS may be any real space as long as it can be imaged by the imaging unit 12. In the example shown in FIG. 1, the first real space RS is a space inside the room R1 in which the worker O performs the first work W1. The virtual space VS is a three-dimensional virtual space whose shape is almost the same as the shape of the first real space RS. In the following description, for convenience of description, a position in the first real space RS is referred to as a real position, and a position in the virtual space VS is referred to as a virtual position. Therefore, hereinafter, for example, when the real position of an object is referred to, the position of the object in the first real space RS is shown. Further, hereinafter, for example, when the virtual position of the object is referred to, the position of the object in the virtual space VS is shown.

Further, in the work assistance system 1, the first display device 10 superimposes the virtual space VS stored in the first display device 10 on the first real space RS by the mixed reality, and the first display unit 11 Display virtual objects on. In addition, in the work assistance system 1, the second display device 20 displays the virtual space VS stored in the second display device 20 by virtual reality. As a result, the work assistance system 1 can improve the efficiency of the first work W1 performed in cooperation with the worker O and the assistant H. Hereinafter, for convenience of explanation, the virtual space VS stored in the first display device 10 is referred to as a first virtual space VS1, and the virtual space VS stored in the second display device 20 is referred to as a second virtual space VS2. To do. That is, in this example, the first virtual space VS1 and the second virtual space VS2 are the same virtual space.

Here, in the embodiment, in the mixed reality, the virtual space VS corresponding to the first real space RS is superimposed on the first real space RS by using the first display unit 11, and the virtual space is added to the first real space RS. This is a technique for allowing a person to visually recognize a space where VS is superimposed.

In the mixed reality, superimposing the virtual space VS on the first real space RS means superimposing a virtual object arranged in the virtual space VS on the first real space RS. In mixed reality, overlapping positions are associated with each other when the virtual space VS is superimposed on the first real space RS. Therefore, in mixed reality, for example, when a certain virtual object VO is arranged at a certain virtual position VP, the virtual object VO positioned at the real position RP associated with the virtual position VP mounts the first display device 10. It is visually recognized by the operator O.

In mixed reality, the first display device 10 is mounted as a position where a virtual object VO is displayed and a position where the virtual object VO is located among the positions on the display surface of the first display unit 11 described above. The actual position visually recognized by the operator O is associated in advance by calibration or the like. Accordingly, in the mixed reality, when the real position RP is included in the area where the worker O can be seen through the first display unit 11, the virtual object VO is displayed at the position XP on the display surface associated with the real position RP. Then, the virtual object VO located at the actual position RP can be visually recognized by the worker O wearing the first display device 10.

As described above, in mixed reality, placing the virtual object VO at the virtual position VP means that the operator O visually recognizes the virtual object VO located at the real position RP associated with the virtual position VP. This means that the VO is displayed at the position XP on the display surface of the first display unit 11.

Moreover, in mixed reality, when the virtual space VS is superimposed on the first real space RS, a certain object RO arranged in the first real space RS is closer to the first display device 10 side than the virtual object VO. When the object RO and the virtual object VO partially overlap each other, the first display unit 11 does not display the part of the virtual object VO that overlaps the object RO. On the other hand, in the mixed reality, in that case, when the virtual object VO is located closer to the first display device 10 side than the object RO and the object RO and the virtual object VO partially overlap, the first display unit 11 displays. Displays the entire virtual object VO. That is, in mixed reality, in the space in which the virtual space VS is superimposed on the first real space RS, the object RO arranged in the first real space RS and the virtual object VO arranged in the virtual space VS are relative to each other. The operator O wearing the first display device 10 visually recognizes the front-rear relationship between the object RO and the virtual object VO while maintaining the positional relationship.

In mixed reality, the size of the virtual object VO displayed on the first display unit 11 of the first display device 10 is the distance between the virtual object VO and the first display device 10 in the first real space RS. Changes accordingly. The longer the distance, the smaller the size of the virtual object VO displayed on the first display unit 11 of the first display device 10. Further, the smaller the distance is, the larger the size of the virtual object VO displayed on the first display unit 11 of the first display device 10 is. That is, the mixed reality can make the worker O wearing the first display device 10 feel a perspective according to the distance between the virtual object VO and the first display device 10 in the first real space RS. it can.

In mixed reality, an area in which the worker O can be seen through the first display unit 11 is specified based on the captured image captured by the imaging unit 12 in the first real space RS. In mixed reality, the first virtual current position is specified based on the captured image captured by the image capturing unit 12. The first virtual current position is a virtual position where the first display device 10 is virtually positioned. In mixed reality, when the first virtual current position is specified, the first real current position is specified. The first actual current position is the actual position corresponding to the first virtual current position, and is the actual position where the first display device 10 is located. In mixed reality, the first virtual line-of-sight direction is specified based on the captured image. The first virtual line-of-sight direction is the optical axis direction of the imaging unit 12 of the first display device 10 that is virtually located at the first virtual current position in the virtual space VS.

In mixed reality, the position of the hand of the worker O in the first real space RS is recorded (tracked) based on the captured image captured by the image capturing unit 12, and the position is recorded at each time in the virtual space VS. A virtual position where the hand is virtually located is specified. Thus, in mixed reality, the worker O can perform a virtual operation with the hand on the virtual object VO visually recognized by the worker O in the first real space RS.

The method for realizing such mixed reality may be a known method or a method to be developed.

Returning to the description of the work assistance system 1.

In the work assistance system 1, the first display device 10 acquires the captured image captured by the image capturing unit 12 from the first display device 10. The first display device 10 superimposes the first virtual space VS1 on the first real space RS by mixed reality based on the acquired captured image. Thus, the first display device 10 causes the first display unit 11 to display the virtual object arranged in the first virtual space VS1. That is, the first display device 10 displays the virtual object on the first display unit so that the virtual object placed at the virtual position VP is visually recognized by the worker O at the actual position RP corresponding to the virtual position VP. 11 is displayed.

Further, in the work assistance system 1, the second display device 20 displays the inside of the second virtual space VS2. More specifically, the second display device 20 receives a predetermined designation operation from the assistant H. The specifying operation includes an operation of specifying the second virtual current position. The second virtual current position is a position at which the second virtual space VS2 is virtually viewed in the second virtual space VS2. That is, the second virtual current position is, so to speak, the virtual position of the assistant H who is virtually positioned in the second virtual space VS2. Further, the designation operation includes the operation of designating the second virtual line-of-sight direction. The second virtual line-of-sight direction is the direction in which the second virtual space VS2 is viewed from the second virtual current position within the second virtual space VS2 (the line-of-sight direction of the second virtual current position). That is, the line-of-sight direction of the second virtual current position is, so to speak, the line-of-sight direction of the assistant H who is virtually located in the second virtual space VS2. In the second virtual current position corresponding to the designation operation received from the assistant H (the second virtual current position designated by the designation operation), the second display device 20 receives the second virtual line-of-sight direction corresponding to the designation operation ( The inside of the second virtual space VS2 viewed from the second virtual line-of-sight direction designated by the designation operation is displayed. In this way, the second display device 20 causes the second display device 10 to move within the second virtual space VS2 that is the same virtual space as the first virtual space VS1 superimposed on the first real space RS in the first display device 10. 20 to display.

Further, in the work assistance system 1, when the virtual object in the first virtual space VS1 is arranged, the first display device 10 displays information indicating the virtual position of the virtual object in the first virtual space VS1 as the virtual object. The information indicating the object is output to the second display device 20 via the server 30. Then, in the work assistance system 1, the second display device 20 acquires information indicating the virtual position of the virtual object and information indicating the virtual object from the first display device 10, and sets the virtual object as the second virtual object. It is arranged at the virtual position in the space VS2.

Further, in the work assistance system 1, when the second display device 20 arranges a virtual object in the second virtual space VS2, the second display device 20 displays information indicating the virtual position of the virtual object in the second virtual space VS2 as the virtual position. The information indicating the object is output to the first display device 10 via the server 30. Then, in the work assistance system 1, the first display device 10 obtains information indicating the virtual position of the virtual object and information indicating the virtual object from the second display device 20, and the virtual object is the first virtual object. It is arranged at the virtual position in the space VS1.

Therefore, in the work assistance system 1, when a certain virtual object is arranged in the first virtual space VS1 or the second virtual space VS2, the worker O visually recognizes the virtual object by the mixed reality by the first display device 10. At the same time, the assistant H can visually recognize the virtual object by the virtual reality of the second display device 20.

Further, in the work assistance system 1, when the first display device 10 is located at a certain actual position, the second display device 20 is a virtual position corresponding to the actual position among the virtual positions in the second virtual space VS2. The first virtual object A1 is placed on the second display device 20 to display the first virtual object A1. At this time, the first display device 10 may arrange the first virtual object A1 in the first virtual space VS1 or may not arrange the first virtual object A1 in the first virtual space VS1. Hereinafter, as an example, a case where the first display device 10 does not arrange the first virtual object A1 in the first virtual space VS1 will be described. Here, the first virtual object A1 is a virtual object indicating the worker O. The first virtual object A1 is, for example, the avatar of the worker O, but is not limited to this.

More specifically, in the work assistance system 1, the first display device 10 identifies the above-described first virtual current position and first virtual line-of-sight direction based on the captured image captured by the imaging unit 12. The first display device 10 outputs, to the second display device 20 via the server 30, first information including information indicating each of the identified first or current sending position and the first virtual line-of-sight direction. The second display device 20 acquires the first information from the first display device 10, and among the virtual positions in the second virtual space VS2, the first virtual current position indicated by the information included in the acquired first information (that is, The first virtual object A1 is arranged at the virtual position where the worker O is virtually located in the virtual space VS, and the second virtual image A1 is displayed on the second display device 20. At this time, the second display device 20 matches the direction in which the first virtual object A1 is facing with the first virtual line-of-sight direction indicated by the information included in the acquired first information. Thereby, in the second display device 20, the assistant H has the virtual position where the first display device 10 is virtually positioned in the second virtual space VS2 and the first display device in the second virtual space VS2. The direction in which 10 is virtually facing can be easily visually recognized by the first virtual object A1 displayed by virtual reality.

Further, when the work assisting system 1 receives the above-described designation operation from the assistant H, the second information including information indicating each of the second virtual current position and the second virtual line-of-sight direction designated by the received designation operation. Is output to the first display device 10 via the server 30. The first display device 10 acquires the second information from the second display device 20, arranges the second virtual object A2 at the second virtual current position indicated by the information included in the acquired second information, and then displays the first display device. The second virtual object A2 is displayed on 10. At this time, the first display device 10 matches the direction in which the second virtual object A2 is facing with the second virtual line-of-sight direction indicated by the information included in the acquired second information. Here, the second virtual object A2 is a virtual object showing the assistant H. As a result, in the first display device 10, the worker O has the virtual position where the assistant H is virtually located in the first virtual space VS1 and the assistant H in the first virtual space VS1. The direction in which the virtual direction is facing can be visually easily known by the second virtual object A2 displayed by mixed reality. The second virtual object A2 is, for example, the avatar of the assistant H, but is not limited to this.

As described above, in the work assistance system 1, the worker O and the assistant H can visually know the positions of each other in the virtual space VS. Thereby, for example, in the work assistance system 1, the worker O can be guided by the second virtual object A2 to the position where the worker O performs the first work W1 in the room R1. As a result, the work assistance system 1 can improve the efficiency of the first work W1 performed by the worker O and the assistant H in cooperation with each other.

Here, the first display device 10 is an example of the above-described first display device. As described above, the first display device 10 can be mounted on the head of the worker O. For example, the first display device 10 is a head mounted display. In the example shown in FIG. 1, the first display device 10 is mounted on the head of the worker O.

The second display device 20 is an example of the aforementioned second display device. The second display device 20 is an information processing device capable of displaying images, such as a workstation, a desktop PC (Personal Computer), a notebook PC, or a tablet PC.

The server 30 is an information processing apparatus that can be operated as a server such as a workstation or a desktop PC (Personal Computer).

As described above, the server 30 generates the virtual space VS corresponding to the first real space RS based on the captured image in which the first real space RS is captured by the image capturing unit 12. The server 30 associates each position of the generated virtual space VS with each position of the first real space RS calculated based on the captured image.

<Hardware configuration of the first display device>
Hereinafter, the hardware configuration of the first display device 10 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of the hardware configuration of the first display device 10. The first display device 10 includes, for example, a first display unit 11, an imaging unit 12, a processor 13, a memory 14, a communication unit 15, a voice input unit 16, and a voice output unit 17. These components are communicatively connected to each other via a bus. The first display device 10 also communicates with the server 30 via the communication unit 15. The communication may be relayed by another device such as a server different from the server 30.

The first display unit 11 is an example of the above-described first display unit. For example, the first display unit 11 includes a transmissive liquid crystal display. Note that the first display unit 11 may include another transmissive display instead of the transmissive liquid crystal display.

The imaging unit 12 is, for example, a stereo camera including a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like as an imaging element that converts the condensed light into an electric signal. The image capturing unit 12 captures a moving image. The image capturing unit 12 may be configured to capture a still image.

The imaging unit 12 is provided at a predetermined installation position among the positions of the first display device 10. The installation position includes an area that can be imaged by the imaging unit 12 and an area that the worker O can see through the first display unit 11 when the first display device 10 is mounted on the head of the worker O. It is almost the same position.

The processor 13 is, for example, a CPU (Central Processing Unit). The processor 13 may be another processor such as FPGA (Field Programmable Gate Array). The processor 13 executes various commands stored in the memory 14 included in the first display device 10.

The memory 14 includes, for example, an SSD (Solid State Drive), an EEPROM (Electrically Erasable Programmable Read-Only Memory), a ROM (Read-Only Memory), a RAM (Random Access Memory), and the like. Note that the memory 14 may be an external storage device connected by a digital input/output port such as a USB (Universal Serial Bus) instead of being built in the first display device 10. The memory 14 stores various information processed by the processor 13, various commands executable by a computer, various images, the above-mentioned first virtual space VS1, and the like. Here, the various instructions are, for example, programs, codes, and the like for displaying various images of mixed reality on the first display unit 11.

The communication unit 15 is configured to include, for example, a digital input/output port such as USB and an Ethernet (registered trademark) port.

The voice input unit 16 is, for example, a microphone. Note that the voice input unit 16 may be another voice input device that can detect voice and output information indicating the detected voice instead of the microphone.

The audio output unit 17 is, for example, a speaker. The audio output unit 17 may be another audio output device capable of outputting audio instead of the speaker.

<Hardware configuration of the second display device>
Hereinafter, the hardware configuration of the second display device 20 will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of the hardware configuration of the second display device 20. The second display device 20 includes, for example, a second display unit 21, a processor 23, a memory 24, a communication unit 25, a voice input unit 26, a voice output unit 27, and an input reception unit 28. These components are communicatively connected to each other via a bus. The second display device 20 also communicates with the server 30 via the communication unit 25. The communication may be relayed by another device such as a server different from the server 30.

The second display unit 21 is an example of the second display unit described above. For example, the second display unit 21 is a display device including a liquid crystal display panel, an organic EL (ElectroLuminescence) display panel, and the like. The second display unit 21 may be separate from the second display device 20.

The processor 23 is, for example, a CPU. The processor 23 may be another processor such as FPGA. The processor 23 executes various commands stored in the memory 24 included in the second display device 20.

The memory 24 includes, for example, a HDD (Hard Disk Drive), SSD, EEPROM, ROM, RAM and the like. The memory 24 may be an external storage device connected to a digital input/output port such as a USB instead of being built in the second display device 20. The memory 24 stores various information processed by the processor 23, various commands executable by a computer, various images, the above-mentioned second virtual space VS2, and the like. Here, the various instructions are, for example, programs, codes, and the like for displaying various images of virtual reality on the second display unit 21.

The communication unit 25 is configured to include, for example, a digital input/output port such as USB and an Ethernet (registered trademark) port.

The voice input unit 26 is, for example, a microphone. Instead of the microphone, the voice input unit 26 may be another voice input device capable of detecting voice and outputting information indicating the detected voice.

The audio output unit 27 is, for example, a speaker. The audio output unit 27 may be another audio output device capable of outputting audio instead of the speaker.

The input receiving unit 28 is, for example, an input device such as a keyboard, a mouse, or a touch pad. The input receiving unit 28 may form a touch panel together with the second display unit 21. The input receiving unit 28 may be separate from the second display device 20.

<Server hardware configuration>
Hereinafter, the hardware configuration of the server 30 will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of the hardware configuration of the server 30. The server 30 includes, for example, a processor 33, a memory 34, and a communication unit 35. These components are communicatively connected to each other via a bus. Further, the server 30 communicates with each of the first display device 10 and the second display device 20 via the communication unit 35. The communication may be relayed by another device such as a server different from the server 30.

The processor 33 is, for example, a CPU. The processor 33 may be another processor such as FPGA. The processor 33 executes various commands stored in the memory 34 included in the server 30.

The memory 34 includes, for example, HDD, SSD, EEPROM, ROM, RAM and the like. It should be noted that the memory 24 may be an external storage device connected by a digital input/output port such as a USB instead of being built in the server 30. The memory 34 stores various information processed by the processor 33, various instructions executable by the computer, various images, the virtual space VS described above, and the like. Here, the various commands are, for example, programs, codes, and the like that generate a virtual space.

The communication unit 35 includes, for example, a digital input/output port such as USB, an Ethernet (registered trademark) port, and the like.

<Functional configuration of each of the first display device, the second display device, and the server>
Hereinafter, the functional configurations of the first display device 10, the second display device 20, and the server 30 will be described with reference to FIG. FIG. 5 is a diagram showing an example of a functional configuration of each of the first display device 10, the second display device 20, and the server 30. In addition, a part or all of the functional units included in the server 30 may be included in either one or both of the first display device 10 and the second display device 20. Further, some or all of the functional units included in the first display device 10 may be included in the server 30. Further, some or all of the functional units included in the second display device 20 may be included in the server 30.

The first display device 10 includes a first display unit 11, an imaging unit 12, a memory 14, a communication unit 15, a voice input unit 16, a voice output unit 17, and a control unit 19.

The control unit 19 controls the entire first display device 10. The control unit 19 includes a first display control unit 191, an imaging control unit 192, an image acquisition unit 193, and an image processing unit 194. These functional units included in the control unit 19 are realized by the processor 13 executing various commands stored in the memory 14, for example. Further, some or all of these functional units may be hardware functional units such as LSI (Large Scale Integration) and ASIC (Application Specific Integrated Circuit).

The first display control unit 191 controls the display on the first display unit 11. Further, the first display control unit 191 generates the above-mentioned first information and outputs the generated first information to the second display device 20 via the server 30.

The image capturing control unit 192 causes the image capturing unit 12 to capture an image of a region that the image capturing unit 12 can capture.

The image acquisition unit 193 acquires from the imaging unit 12 the captured image captured by the imaging unit 12. Here, in the embodiment, the image capturing unit 12 captures a moving image. In this case, the image acquisition unit 193 acquires each frame of the captured image captured by the imaging unit 12 as a moving image in chronological order. Hereinafter, for convenience of description, when referred to as a captured image captured by the image capturing unit 12, it means each frame thus acquired.

The image processing unit 194 performs image processing based on the captured image acquired by the image acquisition unit 193.

The second display device 20 includes a second display unit 21, a memory 24, a communication unit 25, a voice input unit 26, a voice output unit 27, an input receiving unit 28, and a control unit 29.

The control unit 29 controls the entire second display device 20. The control unit 29 includes a second display control unit 291. The second display control unit 291 included in the control unit 29 is realized, for example, by the processor 23 executing various commands stored in the memory 24. The second display control unit 291 may be a hardware function unit such as an LSI or ASIC.

The second display control unit 291 controls the display on the second display unit 21. In addition, the second display control unit 291 generates the above-mentioned second information, and outputs the generated second information to the first display device 10 via the server 30.

The server 30 includes a memory 34, a communication unit 35, and a control unit 39.

The control unit 39 controls the entire server 30. The control unit 39 includes a virtual space generation unit 391 and a voice relay unit 392. These functional units included in the control unit 39 are realized, for example, by the processor 33 executing various commands stored in the memory 34. Further, some or all of these functional units may be hardware functional units such as LSI and ASIC.

The virtual space generation unit 391 generates a virtual space VS corresponding to the first real space RS in the storage area of the memory 34 based on the captured image acquired from the first display device 10.

The voice relay unit 392 relays voice transmission between the voice input unit 16 of the first display device 10 and the voice output unit 27 of the second display device 20. The voice relay unit 395 relays voice transmission between the voice input unit 26 of the second display device 20 and the voice output unit 17 of the first display device 10. Accordingly, in the work assistance system 1, voice communication can be performed between the worker O who wears the first display device 10 and the helper H who operates the second display device 20.

<Second Virtual Object Displayed on First Display Device and First Virtual Object Displayed on Second Display Device>
Hereinafter, each of the second virtual object A2 displayed on the first display device 10 and the first virtual object A1 displayed on the second display device 20 will be described. In the following, as an example, the above-mentioned first virtual current position matches the virtual position X1 in the virtual space VS, and the above-mentioned second virtual current position matches the virtual position X2 in the virtual space VS. Will be described. Further, hereinafter, as an example, a case where the relative positional relationship between the virtual position X1 and the virtual position X2 in the virtual space VS is the positional relationship shown in FIG. 6 will be described. FIG. 6 is a diagram showing an example of a relative positional relationship between the virtual position X1 and the virtual position X2 in the virtual space VS. It should be noted that each of the object VO1 and the object VO2 shown in FIG. 6 is an example of some kind of rectangular parallelepiped virtual object arranged in the virtual space VS. The objects VO1 and VO2 may or may not be arranged in the virtual space VS. Therefore, description of the objects VO1 and VO2 is omitted. Further, FIG. 6 is a diagram showing an example of a region including the virtual position X1 and the virtual position X2 among the regions in the virtual space VS as viewed from above. Note that, in the virtual space VS, the upper side is the direction opposite to the direction corresponding to the gravity direction of the first real space RS among the directions in the virtual space VS. Further, in the virtual space VS, the lower side is a direction corresponding to the gravity direction of the first real space RS among the directions in the virtual space VS.

When the positional relationship between the virtual position X1 and the virtual position X2 is the positional relationship shown in FIG. 6, the virtual display as shown in FIG. 7 is displayed on the first display section 11 of the first display device 10 in the work assistance system 1. The object is displayed. FIG. 7 is a diagram showing an example of the display surface of the first display unit 11 when the positional relationship between the virtual position X1 and the virtual position X2 is the positional relationship shown in FIG. The display surface DP1 shown in FIG. 7 is the display surface of the first display unit 11.

From the display surface DP1 shown in FIG. 7, a part of the area in the first real space RS is visible through the display surface DP1. Further, from the display surface DP1 shown in FIG. 7, the operator's hand UH is visible through the display surface DP1. Further, on the display surface DP1 shown in FIG. 7, the object VO1, the object VO2, and the second virtual object A2 located at the virtual position X2 in the first virtual space VS1 are displayed.

In the example shown in FIG. 7, the second virtual object A2 has the head A2H of the second virtual object A2 corresponding to the head of the assistant H. In the embodiment, the virtual position of the second virtual object A2 is represented by the virtual position of the center of gravity of the head A2H. The virtual position of the second virtual object A2 may be represented by virtual positions of other parts corresponding to the second virtual object A2 instead of the virtual position of the center of gravity.

On the other hand, when the positional relationship between the virtual position X1 and the virtual position X2 is the positional relationship shown in FIG. 6, the second display section 21 of the second display device 20 in the work assistance system 1 is as shown in FIG. Virtual objects are displayed. FIG. 8 is a diagram showing an example of the display surface of the second display unit 21 when the positional relationship between the virtual position X1 and the virtual position X2 is the positional relationship shown in FIG. The display surface DP2 shown in FIG. 8 is the display surface of the second display unit 21.

On the display surface DP2 shown in FIG. 8, among the areas in the virtual space VS, the object VO1, the object VO2, and the first virtual object A1 located at the virtual position X2 in the second virtual space VS2 are displayed. The placed area is displayed. In the example shown in FIG. 8, the first virtual object A1 has the head A1H of the first virtual object A1 corresponding to the head of the worker O. In the embodiment, the virtual position of the first virtual object A1 is represented by the virtual position of the center of gravity of the head A1H. The virtual position of the first virtual object A1 may be represented by virtual positions of other parts corresponding to the first virtual object A1, instead of the virtual position of the center of gravity.

Further, on the display surface DP2 shown in FIG. 8, a virtual hand AF corresponding to the hand of the worker O is displayed. In the second display device 20, the hand AF is arranged in the second virtual space VS2 when the captured image captured by the imaging unit 12 includes the hand UH of the worker O. The virtual position of the hand AF in the second virtual space VS2 is a position corresponding to the position of the hand UH in the first real space RS among the virtual positions in the second virtual space VS2. The virtual position of the hand AF in the second virtual space VS2 is represented by the virtual position of the center of gravity of the hand AF. The virtual position of the hand AF in the second virtual space VS2 may be represented by virtual positions of other parts corresponding to the hand AF instead of the virtual position of the center of gravity. Further, the display surface DP2 may be configured so that the hand AF is not displayed.

Here, when the first display device 10 receives a virtual operation for displaying a certain virtual object VM1 on the first display unit 11 from the worker O, the display surface DP1 is displayed as shown in FIG. The virtual object VM1 is displayed at a virtual position corresponding to the operation among the virtual positions in the one virtual space VS1. FIG. 9 is a diagram showing an example of the display surface DP1 when the virtual object VM1 is displayed on the display surface DP1. The virtual object VM1 is, for example, a virtual procedure manual in which the work procedure of the first work W1 is described. The virtual object VM1 may be another virtual object such as a virtual menu where some virtual operation is performed instead of the virtual procedure manual.

On the other hand, when the first display device 10 receives a virtual operation for displaying the virtual object VM1 on the first display unit 11 from the worker O, the second virtual image is displayed on the display surface DP2 as shown in FIG. The virtual object VM1 is displayed at a virtual position corresponding to the operation among the virtual positions in the space VS2. FIG. 10 is a diagram showing an example of the display surface DP2 when the virtual object VM1 is displayed on the display surface DP2.

As described above, in the work assistance system 1, the virtual object displayed on the display surface DP1 by the virtual operation by the worker O is also displayed on the display surface DP2. As a result, in the work assistance system 1, the assistant H, for example, by looking at the state in the virtual space VS displayed on the second display unit 21, indicates what work the worker O is currently performing. Can be confirmed. As a result, the work assistance system 1 can improve the efficiency of the first work W1 performed by the worker O and the assistant H in cooperation with each other.

Here, when the second display device 20 receives an operation for displaying a certain virtual object VM2 on the second display unit 21 from the assistant H, the second virtual space is displayed on the display surface DP2 as shown in FIG. The virtual object VM2 is displayed at a virtual position corresponding to the operation among the virtual positions in the VS2. FIG. 11 is a diagram showing an example of the display surface DP2 when the virtual object VM2 is displayed on the display surface DP2. The virtual object VM2 is, for example, a virtual layout diagram in which the layout of facilities and the like in the first real space RS is described. The virtual object VM2 may be another virtual object such as a virtual menu where some virtual operation is performed instead of the virtual layout drawing.

On the other hand, when the second display device 20 receives a virtual operation for displaying the virtual object VM2 on the second display unit 21 from the worker O, the second virtual image is displayed on the display surface DP1 as shown in FIG. The virtual object VM2 is displayed at a virtual position corresponding to the operation among the virtual positions in the space VS2. FIG. 12 is a diagram showing an example of the display surface DP1 when the virtual object VM2 is displayed on the display surface DP1.

As described above, in the work assistance system 1, the virtual object displayed on the display surface DP2 by the operation of the assistant H is also displayed on the display surface DP1. Thus, in the work assistance system 1, the worker O, for example, by looking at the virtual object displayed on the first display unit 11, based on what the assistant H is currently instructing the worker O. Can be confirmed. As a result, the work assistance system 1 can improve the efficiency of the first work W1 performed by the worker O and the assistant H in cooperation with each other.

<Process in which the first display device superimposes the virtual space on the first real space>
Hereinafter, with reference to FIG. 13, a process in which the first display device 10 superimposes the virtual space VS on the first real space RS will be described. FIG. 13 is a diagram showing an example of the flow of processing in which the first display device 10 superimposes the virtual space VS on the first real space RS.

The first display control unit 191 acquires the virtual space VS previously stored in the server 30 from the server 30 (step S110).

More specifically, in step S110, the first display control unit 191 outputs a virtual space VS acquisition request to the server 30. The server 30 outputs the virtual space VS to the first display device 10 in response to the acquisition request acquired from the first display control unit 191 of the first display device 10. Then, the first display control unit 191 acquires the virtual space VS from the server 30. Then, the first display control unit 191 stores the acquired virtual space VS in the memory 14 as the first virtual space VS1.

Here, the virtual space VS that the first display device 10 acquires from the server 30 is information generated by the virtual space generation unit 394 by space recognition based on a captured image of the room R1 captured by the first display device 10. It may be present or may be information generated by another device. When the virtual space generation unit 394 generates the virtual space VS, the virtual space generation unit 394 uses the first real space based on the captured image in which the first real space RS is captured by the image capturing unit 12 of the first display device 10. A virtual space VS corresponding to RS is generated. The virtual space generation unit 394 associates each position of the generated virtual space VS with each position of the first real space RS calculated based on the captured image in which the first real space RS is imaged by the imaging unit 12.

Next, the image capturing control unit 192 causes the image capturing unit 12 to capture an image of a region that the image capturing unit 12 can capture. Then, the image acquisition unit 193 acquires the captured image captured by the imaging unit 12 from the imaging unit 12 (step S120).

Next, the image processing unit 194 performs matching between the imaging area and the first virtual space VS1 stored in the memory 14. The imaging region is a region in the first real space RS that is captured in the captured image acquired by the image acquisition unit 193 in step S120. That is, the image processing unit 194 identifies the area corresponding to the imaging area in the area in the first virtual space VS1. The first display control unit 191 causes the first display unit 11 to display various virtual objects arranged in the area specified by the image processing unit 194, and superimposes the first virtual space VS1 on the first real space RS. (Step S130). As a result, the first display control unit 191 causes the worker O to visually recognize a space in which the first virtual space VS1 (that is, the virtual space VS) is superimposed on the first real space RS. Further, in step S130, the image processing unit 194 specifies the first virtual current position and the first virtual line-of-sight direction based on the specified area. Further, when the area of the hand includes the hand of the worker O, the image processing unit 194 specifies the virtual position of the hand in the first virtual space VS1 as the eleventh virtual current position.

Next, the first display control unit 191 determines whether or not a superimposition end operation has been received (step S140). The superposition ending operation is an operation of ending the superposition of the first virtual space VS1 on the first real space RS. When the first display control unit 191 determines that the operation to end the superimposition has not been received (step S140-NO), the imaging control unit 192 transitions to step S120 and images the region that the imaging unit 12 can image. The part 12 is made to image again. On the other hand, when the first display control unit 191 determines that the operation for ending the superimposition has been accepted (step S140-YES), the process is ended.

<Process in which the second display device displays the virtual space>
Hereinafter, with reference to FIG. 14, a process in which the second display device 20 displays the inside of the virtual space VS will be described. FIG. 14 is a diagram showing an example of the flow of processing in which the second display device 20 displays the inside of the virtual space VS.

The second display control unit 291 acquires the virtual space VS previously stored in the server 30 from the server 30 (step S210).

More specifically, in step S210, the second display control unit 291 outputs a virtual space VS acquisition request to the server 30. The server 30 outputs the virtual space VS to the second display device 20 in response to the acquisition request acquired from the second display control unit 291 of the second display device 20. Then, the second display control unit 291 acquires the virtual space VS from the server 30. The second display control unit 291 stores the acquired virtual space VS in the memory 24 as the second virtual space VS2.

Then, the second display control unit 291 acquires the initial information stored in advance in the server 30 from the server 30 (step S220). Here, the initial information is information that specifies each of the predetermined second virtual current position and the second virtual line-of-sight direction as the initial value of each of the second virtual current position and the second virtual line-of-sight direction. The initial information may be stored in advance in the server 30 or may be stored in the server 30 from the assistant H via the second display device 20.

Next, the second display control unit 291 sets the second virtual current position specified by the information included in the initial information acquired in step S220 among the virtual positions in the second virtual space VS2 stored in the memory 24, The second virtual space VS2 viewed from the second virtual line-of-sight direction specified by the information is displayed on the second display unit 21 (step S230).

Next, the second display control unit 291 waits until the above-mentioned designation operation is accepted (step S240).

When receiving the designation operation (step S240-YES), the second display control unit 291 updates the display of the second virtual space VS2 (step S250). More specifically, the second display control unit 291 performs the designation operation at the second virtual current position designated by the designation operation received from the virtual positions in the second virtual space VS2 stored in the memory 24. The second virtual space VS2 viewed from the designated second virtual line-of-sight direction is displayed on the second display unit 21.

Next, the second display control unit 291 determines whether or not a display end operation has been received (step S260). The display ending operation is an operation of ending the display of the second virtual space VS2. When the second display control unit 291 determines that the operation for ending the display has not been received (step S260-NO), the second display control unit 291 transitions to step S240 and waits again until the designated operation is received. On the other hand, when the second display control unit 291 determines that the operation for ending the display has been received (step S260-YES), the process is ended.

<Process in which the first display device outputs the first information to the second display device>
Hereinafter, a process in which the first display device 10 outputs the first information to the second display device 20 will be described with reference to FIG. 15. FIG. 15 is a diagram showing an example of the flow of processing in which the first display device 10 outputs the first information to the second display device 20. The process of the flowchart shown in FIG. 15 is executed by the first display control unit 191 every time the first virtual current position and the first virtual line-of-sight direction are specified in step S130 shown in FIG. The process of the flowchart may be performed by another functional unit included in the first display device 10, instead of the first display control unit 191.

The first display control unit 191 generates first information including information indicating each of the first virtual current position and the first virtual line-of-sight direction specified in step S130 executed immediately before (step S310). When the image processing unit 194 specifies the eleventh virtual current position in step S130 executed immediately before, the first display control unit 191 displays the information indicating the first virtual current position and the first virtual line-of-sight direction. And the information indicating the eleventh virtual current position are generated.

Next, the 1st display control part 191 outputs the 1st information produced|generated in step S310 to the 2nd display device 20 via the server 30 (step S320), and complete|finishes a process.

The second display control unit 291 generates the above-mentioned second information by the same processing as the flowchart shown in FIG. 15, and outputs the generated second information to the first display device 10. Specifically, the second display control unit 291 generates the second information every time the second virtual current position and the second virtual line-of-sight direction are specified by the above-described initial information or the specifying operation, and the generated second information is generated. The information is output to the first display device 10 via the server 30. Under these circumstances, the description of the flow of processing in which the second display control unit 291 generates the second information and outputs the generated second information to the first display device 10 will be omitted.

In addition, when the first display control unit 191 receives an operation of arranging the virtual object desired by the worker O in the first virtual space VS1, the first display control unit 191 is similar to the flowchart illustrated in FIG. By the processing, the third information is generated, and the generated third information is output to the second display device 20. Here, the third information is information including the information indicating the virtual position designated by the operation and the information indicating the virtual object. Under these circumstances, the description with reference to the drawing regarding the flow of processing in which the first display control unit 191 generates the third information and outputs the generated third information to the second display device 20 in this case is as follows. Omit it.

<Process in which the second display device displays the first virtual object>
Hereinafter, with reference to FIG. 16, a process in which the second display device 20 displays the first virtual object A1 will be described. FIG. 16 is a diagram showing an example of the flow of processing in which the second display device 20 displays the first virtual object A1. In the process of the flowchart shown in FIG. 16, after the inside of the second virtual space VS2 is displayed on the second display unit 21 in step S230 shown in FIG. 14, a display end operation is accepted in step S260 shown in FIG. Will be executed until.

The second display control unit 291 waits until the first information is acquired from the first display device 10 (step S410).

When the second display control unit 291 acquires the first information from the first display device 10 (step S410-YES), the second display control unit 291 performs the first arrangement process (step S420). Here, the first placement processing will be described.

The first placement process is a process including a process of placing the first virtual object A1 in the second virtual space VS2. When acquiring the first information from the first display device 10, the second display control unit 291 sets the first virtual current position indicated by the information included in the first information among the virtual positions in the second virtual space VS2 to the first virtual current position. One virtual object A1 is arranged to display the first virtual object A1 on the second display device 20. At this time, the second display control unit 291 matches the direction in which the first virtual object A1 is facing with the first virtual line-of-sight direction indicated by the information included in the acquired first information. In addition, when the first information includes information indicating the eleventh virtual current position, the second display control unit 291 selects the eleventh virtual current position indicated by the information among the positions in the second virtual space VS2. The hand AF is arranged and the hand AF is displayed on the second display device 20. In this way, the second display control unit 291 performs the process of displaying the first virtual object A1 or the process of displaying the first virtual object A1 and the hand AF as the first placement process.

After the first arrangement process is performed in step S420, the second display control unit 291 transitions to step S410 and waits again until the first information is acquired from the first display device 10.

Note that when the first display control unit 191 acquires the second information, the first display control unit 191 performs the second arrangement process instead of the first arrangement process by the same process as the flowchart illustrated in FIG. 16. .. The second placement process is a process including a process of placing the second virtual object A2 in the first virtual space VS1. When acquiring the second information from the second display device 20, the first display control unit 191 sets the second virtual current position indicated by the information included in the second information among the virtual positions in the first virtual space VS1 to the second virtual current position. Two virtual objects A2 are arranged and the second virtual object A2 is displayed on the first display device 10. At this time, the first display control unit 191 matches the direction in which the second virtual object A2 is facing with the second virtual line-of-sight direction indicated by the information included in the acquired second information. Under such circumstances, the description of the flow of the process in which the first display control unit 191 performs the second placement process in that case will be omitted.

Further, when the second display control unit 291 acquires the third information, the second display control unit 291 performs the third arrangement process instead of the first arrangement process by the same process as the flowchart illustrated in FIG. 16. .. The third placement process is a process including a process of placing the virtual object indicated by the information included in the third information in the second virtual space VS2. That is, when the third information is acquired from the first display device 10, the second display control unit 291 sets the virtual information in the virtual position in the second virtual space VS2 to the virtual position indicated by the information included in the third information. The virtual object indicated by is arranged and the second display device 20 is caused to display the virtual object. Under such circumstances, the description of the flow of processing in which the second display control unit 291 performs the third placement processing will be omitted in this case.

In the embodiment described above, only one worker, O, performs the first work W1 in the room R1. However, in the work assistance system 1, two or more people may perform the first work W1 in the room R1. In this case, the work assistance system 1 includes a plurality of first display devices 10. The operation of each first display device 10 is the same as the operation of the first display device 10 described above.

Further, the line-of-sight direction described above may be replaced with the posture, may be indicated by the posture, and may be indicated by another amount capable of indicating the line-of-sight direction.

As described above, the work assistance system according to the embodiment (the work assistance system 1 in the above-described example) can be mounted on the head of the first user (the worker O in the above-described example). One display device (the first display device 10 in the example described above) and a second display device (in the example described above in the above-described example, room R2) installed in a predetermined room (room R2 in the example described above). , A second display device 20) and an image is displayed on the second display device 20) to assist the first work (the first work W1 in the example described above) performed in cooperation with the first user and the second user. Therefore, the first display device includes a transmissive first display unit (the first display unit 11 in the example described above) and an image capturing unit that captures an image of a real space (the image capturing unit 12 in the example described above). ) And a predetermined first real space (in the example described above, the first real space RS), the virtual space corresponding to the first real space (above). In the example described in 1 above, the virtual space generation unit (described above) that generates the virtual space VS and associates each position of the generated virtual space with each position of the first real space calculated based on the captured image. In the example, the virtual space generation unit 391) and the virtual object located at the first virtual position (in the example described above, the virtual position VP) of the positions in the virtual space are set as the positions in the first real space. A first display control unit that causes the first display device to display the first user so as to be visually recognized by the first user at a position corresponding to the first virtual position (the actual position RP in the example described above). Then, according to the first display control unit 191) and the operation (specified operation in the example described above) received from the second user, the second virtual position (position corresponding to the operation in the virtual space) ( In the example described above, the second display control unit (second display control unit that causes the second display device to display the virtual space viewed from the line-of-sight direction of the second virtual current position) (the second virtual line-of-sight direction in the example described above). In the example described above, the second display control unit 291) is provided, and the first display control unit is arranged such that the second virtual object (in the example described above in the example described above) is located at the second virtual position of the positions in the virtual space. , The second virtual object A2) is arranged to display the second virtual object on the first display device, and the second display control unit controls the first real position (described above) of the positions in the first real space. In the example, when the first display device is located at the first actual current position), A first virtual object (first virtual object A1 in the example described above) is arranged at a position (first virtual current position in the example described above) corresponding to the first actual position among positions in the virtual space. Then, the first virtual object is displayed on the second display device, and the first virtual object is the virtual object indicating the first user (in the example described above, the avatar indicating the worker O), and the second virtual object. Is a virtual object indicating the second user (an avatar indicating the assistant H in the example described above). Accordingly, the work assistance system can improve the efficiency of the first work performed by the first user and the second user in cooperation with each other.

In addition, the work assistance system includes an image processing unit (in the example described above, the image processing unit 194) that specifies the position of the first display device in the first actual position based on the captured image, and the first display The control unit may arrange the first virtual object in the virtual space and display the first virtual object on the second display device based on the position specified by the image processing unit.

In the work assistance system, the image processing unit specifies the line-of-sight direction of the first display device in the first real space based on the captured image, and the first display control unit is specified by the image processing unit. A configuration may be used in which the first virtual object is arranged in the virtual space and the second display device displays the first virtual object based on the position and the line-of-sight direction.

In the work assistance system, when the captured image includes the first user's hand, the image processing unit further specifies the position of the first user's hand in the first real space based on the captured image. , A third virtual object (the hand AF in the example described above) indicating the first user's hand is arranged in the virtual space together with the first virtual object, and the first virtual object and the third virtual object are displayed on the second display device. A configuration may be used that displays an object.

Further, in the work assistance system, when the first display operation for displaying the fourth virtual object (the virtual object VM1 in the example described above) on the first display device is received from the first user, the first display control unit and The 2nd display control part may use the composition which arranges the 4th virtual object in the position according to the 1st display operation among the positions in virtual space.

<Modification of Embodiment>
Hereinafter, modified examples of the embodiment will be described. In the modification of the embodiment, the same components as those in the embodiment are designated by the same reference numerals and the description thereof will be omitted.

In the modification of the embodiment, the first display control unit 191 sets the virtual object CM that virtually surrounds the first display device 10 in the first virtual space VS1 based on the virtual operation received from the worker O. Deploy. At this time, the first display control unit 191 displays the virtual object CM so that the first virtual current position specified by the image processing unit 194 immediately before the timing of receiving the operation is included inside the virtual object CM. It is arranged in one virtual space VS1.

The virtual object CM is, for example, a cylindrical virtual object as shown in FIG. FIG. 17 is a diagram showing an example of the virtual object CM. The shape of the virtual object CM may be any shape as long as it surrounds the first display device 10, instead of the cylindrical shape. For example, when arranging the virtual object CM having a cylindrical shape in the first virtual space VS1, the first display control unit 191 makes the virtual object CM so that the first virtual current position passes through the central axis of the virtual object CM. Are arranged in the first virtual space VS1.

The inner surface of the virtual object CM is various virtual menus operated by the worker O, various UIs (User Interfaces) such as virtual buttons for the worker O to reproduce a moving image, and the worker O. It is divided into a plurality of areas including each of various virtual materials to be browsed. When the virtual object CM is displayed, the worker O can perform virtual operations on a plurality of UIs, browse a plurality of materials, and the like without moving the room R1.

Further, in the area on the inner surface of the virtual object CM that does not include the UI, the material, etc., the UI, the material, etc. desired by the worker O can be displayed. According to the virtual operation received from the worker O, the first display control unit 191 fixes the UI corresponding to the operation, the material corresponding to the operation, and the like in the area.

Here, FIG. 18 is a diagram illustrating an example of the display surface DP1 of the first display unit 11 when the virtual object CM is arranged in the first virtual space VS1. As shown in FIG. 18, since the virtual object CM is arranged in the first virtual space VS1 so as to surround the first virtual current position on the display surface DP1, a part of the inner surface of the virtual object CM is displayed. Is displayed.

Further, when the virtual object CM that virtually surrounds the first display device 10 is arranged in the first virtual space VS1, the first display control unit 191 determines that the virtual object CM is arranged in the first virtual space VS1. The information shown is output to the second display device 20 via the server 30. When the information is acquired, the second display control unit 291 arranges the virtual object CM in the second virtual space VS2 so as to surround the second virtual current position at the timing when the information is acquired. As a result, a part of the inner surface of the virtual object CM is displayed on the display surface DP2 of the second display unit 21 as shown in FIG. FIG. 19 is a diagram showing an example of the display surface DP2 of the second display unit 21 when the virtual object CM is arranged in the second virtual space VS2.

Here, the display mode of the virtual object CM on the display surface DP2 shown in FIG. 19 is almost the same as the display mode of the virtual object CM on the display surface DP1 shown in FIG. Therefore, in the work assistance system 1, the assistant H can operate the virtual object CM with the same operation feeling as that of the operator O performing a virtual operation on the virtual object CM. In other words, in the work assistance system 1, the worker O can operate the virtual object CM with the same operation feeling as that of the assistant H operating the virtual object CM. That is, in the work assistance system 1, the operation feeling when the worker O wearing the first display device 10 makes a virtual operation on the virtual object CM and the assistant H through the second display device 20 causes the virtual object to appear. The operation feeling when operating the CM can be made substantially the same.

As described above, in the work assistance system according to the modified example of the embodiment, the first display control unit creates a virtual object that virtually surrounds the first display device based on the virtual operation received from the first user. When the virtual object that is placed in the virtual space and virtually surrounds the first display device (in the example described above, the virtual object CM) is placed in the virtual space, the second display controller displays the second virtual position. The virtual object surrounding the first display device has the same function as that of the virtual object surrounding the second virtual position. The virtual object that virtually surrounds the second display device and the virtual object that virtually surrounds the second display device have the same shape, and the virtual object that virtually surrounds the first display device is displayed on the first display device. A configuration may be used in which the display modes of the virtual object that virtually surrounds the second display device on the second display device are the same as each other.

Further, in the work assisting system, the configurations of the virtual object virtually surrounding the first display device and the virtual object virtually surrounding the second display device are the same cylindrical shape. Good.

Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and modifications, replacements, deletions, etc. are possible without departing from the gist of the present invention. May be done.

Further, the program for realizing the function of any component of the above-described device may be recorded in a computer-readable recording medium, and the program may be read by a computer system and executed. Here, the device is, for example, the first display device 10, the second display device 20, the server 30, or the like. It should be noted that the “computer system” mentioned here includes an OS (Operating System) and hardware such as peripheral devices. The "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD (Compact Disk)-ROM, or a storage device such as a hard disk built in a computer system. .. Furthermore, "computer-readable recording medium" refers to a fixed amount of volatile memory inside the computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. Those that hold time programs are also included.

Further, the above program may be transmitted from a computer system that stores the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium having a function of transmitting information, such as a network such as the Internet or a communication line such as a telephone line.
Further, the above program may be a program for realizing a part of the functions described above. Furthermore, the above program may be a program that can realize the above-mentioned functions in combination with a program already recorded in a computer system, that is, a so-called difference file or difference program.

DESCRIPTION OF SYMBOLS 1... Work assistance system, 10... 1st display device, 11... 1st display part, 12... Imaging part, 13, 23, 33... Processor, 14, 24, 34... Memory, 15, 25, 35... Communication part, 16, 26... Voice input unit, 17, 27... Voice output unit, 19, 29, 39... Control unit, 20... Second display device, 21... Second display unit, 28... Input receiving unit, 30... Server, 191 ... 1st display control part, 192... Imaging control part, 193... Image acquisition part, 194... Image processing part, 291... 2nd display control part, 391... Virtual space generation part, 392... Audio relay part, 394... Virtual space Generation unit, 395... Voice relay unit, A1... First virtual object, A2... Second virtual object, CM... Virtual object, DP1... Display surface, DP2... Display surface, H... Assistant, O... Worker, RS... First real space, VS... Virtual space, VS1... First virtual space, VS2... Second virtual space

Claims (10)

An image is displayed on a first display device that can be worn on the head of the first user and a second display device installed in a predetermined room, and the first user and the second user cooperate with each other. A work assistance system for assisting the first work,
The first display device includes a transmissive first display unit and an image capturing unit that captures an image of a real space,
A virtual space corresponding to the first real space is generated based on a captured image in which a predetermined first real space is captured by the image capturing unit, and based on each position of the generated virtual space and the captured image. A virtual space generation unit that associates the calculated positions of the first real space with each other,
A virtual object arranged at a first virtual position of the positions in the virtual space is visually recognized by the first user at a position corresponding to the first virtual position of the positions in the first real space. A first display controller for displaying on the first display device,
A second display device that displays, in response to an operation received from the second user, the virtual space viewed from the line-of-sight direction of a second virtual position corresponding to the operation, of the positions in the virtual space; A display control unit,
Equipped with
The first display control unit arranges a second virtual object at the second virtual position of the positions in the virtual space and causes the first display device to display the second virtual object,
When the first display device is located at a first real position of the positions within the first real space, the second display control unit selects the first real position of the positions within the virtual space. Arranging a first virtual object at a corresponding position to display the first virtual object on the second display device,
The first virtual object is a virtual object indicating the first user,
The second virtual object is a virtual object indicating the second user,
Work assistance system. An image processing unit that specifies the position of the first display device within the first actual position based on the captured image,
The first display control unit arranges the first virtual object in the virtual space based on the position specified by the image processing unit and causes the second display device to display the first virtual object,
The work assistance system according to claim 1. The image processing unit specifies a line-of-sight direction of the first display device in the first real space based on the captured image,
The first display control unit arranges the first virtual object in the virtual space based on the position and the line-of-sight direction identified by the image processing unit and displays the first virtual object on the second display device. To display,
The work assistance system according to claim 2. When the captured image includes the hand of the first user, the image processing unit further specifies the position of the hand of the first user in the first real space, based on the captured image,
In the virtual space, a third virtual object indicating the hand of the first user is arranged together with the first virtual object, and the second display device displays the first virtual object and the third virtual object.
The work assistance system according to claim 2 or 3. When a first display operation for displaying a fourth virtual object on the first display device is received from the first user,
The first display control unit and the second display control unit arrange the fourth virtual object at a position corresponding to the first display operation among the positions in the virtual space,
The work assistance system according to any one of claims 1 to 4. A server including the virtual space generation unit,
The first display device includes the first display control unit,
The second display device includes the second display control unit,
The server is communicably connected to each of the first display device and the second display device,
The work assistance system according to any one of claims 1 to 5. A server including the virtual space generation unit, the first display control unit, and the second display control unit;
The work assistance system according to any one of claims 1 to 5. The first display control unit arranges a virtual object that virtually surrounds the first display device in the virtual space based on a virtual operation received from the first user,
When a virtual object that virtually surrounds the first display device is arranged in the virtual space, the second display control unit arranges a virtual object that surrounds the second virtual position in the virtual space,
The function of the virtual object virtually surrounding the first display device is the same as the function of the virtual object surrounding the second virtual position,
The respective shapes of the virtual object virtually surrounding the first display device and the virtual object virtually surrounding the second display device are the same as each other,
The display mode of the virtual object virtually surrounding the first display device on the first display device and the display mode of the virtual object virtually surrounding the second display device on the second display device are the same display. Is an aspect,
The work assistance system according to any one of claims 1 to 6. The respective shapes of the virtual object virtually surrounding the first display device and the virtual object virtually surrounding the second display device are the same cylindrical shape.
The work assistance system according to claim 8. An image is displayed on a first display device that can be worn on the head of the first user and a second display device installed in a predetermined room, and the first user and the second user cooperate with each other. A program for controlling a work assistance system for assisting the first work,
The first display device includes a transmissive first display unit and an image capturing unit that captures an image of a real space,
A virtual space corresponding to the first real space is generated based on a captured image in which a predetermined first real space is captured by the image capturing unit, and based on each position of the generated virtual space and the captured image. A virtual space generating step of associating each calculated position of the first real space with each other;
A virtual object arranged at a first virtual position of the positions in the virtual space is visually recognized by the first user at a position corresponding to the first virtual position of the positions in the first real space. A first display control step of displaying on the first display device,
A second display device that displays, in response to an operation received from the second user, the virtual space viewed from the line-of-sight direction of a second virtual position corresponding to the operation, of the positions in the virtual space; A display control step,
Is a program for executing
The first display control step arranges a second virtual object at the second virtual position of the positions in the virtual space and causes the first display device to display the second virtual object,
In the second display control step, when the first display device is located at a first real position among the positions within the first real space, the second display control step is performed at the first real position among the positions within the virtual space. Arranging a first virtual object at a corresponding position to display the first virtual object on the second display device,
The first virtual object is a virtual object indicating the first user,
The second virtual object is a virtual object indicating the second user,
program.

Images