WO2025027877A1 - Information processing device, information processing method, and program - Google Patents

Abstract

Provided is an information processing device of an information processing system in which a voice input from a first user present in a first space and image data obtained by imaging the first space are presented to a second user present in a second space away from the first space, and a voice input from the second user is presented to the first user, said information processing device comprising a control circuit which controls the display of display information corresponding to information pertaining to the viewpoint of the second user and the image data and information pertaining to an object present in the first space by a display.

Description

Information processing device, information processing method, and program

This disclosure relates to an information processing device, an information processing method, and a program.

In recent years, technology has become known that allows a user who is in a location separate from the managed space to manage the status of the managed space. For example, Patent Document 1 discloses technology in which the managed space is a work site, the user is an administrator who manages the status of the work site, and the administrator manages the status of the work site.

JP 2022-42303 A

However, it is desirable for users who are located far away from the managed space to be able to manage the status of the managed space more efficiently.

In order to solve the above problem, according to one aspect of the present disclosure, there is provided an information processing device in an information processing system in which voice input from a first user present in a first space and image data obtained by capturing an image of the first space are presented to a second user present in a second space separated from the first space, and voice input from the second user is presented to the first user, the information processing device including a control circuit for controlling the display, on a display, of information relating to the viewpoint of the second user and display information corresponding to the image data, and information relating to an object present in the first space.

According to another aspect of the present disclosure, there is provided an information processing method in an information processing system in which voice input from a first user present in a first space and image data obtained by capturing an image of the first space are presented to a second user present in a second space separate from the first space, and voice input from the second user is presented to the first user, the information processing method including a processor controlling the display, on a display, of information relating to the viewpoint of the second user and display information corresponding to the image data, and information relating to an object present in the first space.

According to another aspect of the present disclosure, there is provided an information processing device in an information processing system in which voice input from a first user present in a first space and image data obtained by capturing an image of the first space are presented to a second user present in a second space separated from the first space, and voice input from the second user is presented to the first user, and a program is provided that causes a computer to function as a control circuit for controlling the display, on a display, of information relating to the viewpoint of the second user and display information corresponding to the image data, and information relating to an object present in the first space.

FIG. 1 is a diagram illustrating a configuration example of an information processing system according to an embodiment of the present disclosure. 2 is a diagram showing an example of data transmitted and received between devices constituting the information processing system 1 according to an embodiment of the present disclosure. FIG. FIG. 2 is a diagram illustrating an example of an overall process performed by an information processing system 1 according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a functional configuration of a field worker device 10 according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a functional configuration of a remote participant device 20 according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a functional configuration of a server 30 according to an embodiment of the present disclosure. 13 is a diagram for explaining details of a multimodal process performed when a remote participant makes an indication. FIG. 13 is a diagram for explaining details of a multi-modal process performed when a site worker makes an indication. FIG. 11 is a diagram for explaining an example of voice recognition performed by a control unit 340. FIG. 11 is a diagram for explaining an example of recognition of indication information performed by a control unit 340. FIG. FIG. 13 is a diagram for explaining an example of recognition of an object that is being noticed by a remote participant. FIG. 13 is a diagram for explaining an example of recognition of an object that is being noticed by a site worker. 11 is a diagram for explaining an example of response information recognition performed by a control unit 340. FIG. 13 is a diagram showing an example of meta-information in a tag recorded in the storage unit 350 by the control unit 340. FIG. FIG. 13 is a diagram for explaining a modified example of the recognition model. FIG. 11 is a diagram for explaining an example of an association start operation. 13 is a diagram for explaining a modified example of recognition of indicated information. FIG. FIG. 13 is a diagram for explaining a modified example in which a speaker identification result is used for tagging. FIG. 13 is a diagram for explaining an example of collaboration with AI. 11 is a diagram for explaining an example in which meta information in a tag is reflected in real-time video. FIG. FIG. 13 is a diagram for explaining a case where meta information in a tag is added to a timeline. 11 is a diagram for explaining a case where meta information in a tag is added to a map of a work site and a displayed image. FIG. FIG. 11 illustrates an example of tag management. FIG. 13 illustrates an example of a report. FIG. 1 is a diagram showing an example of a work site with a two-dimensional code. 2 is a diagram for explaining hardware processing executed by the field worker device 10. FIG. 11 is a diagram for explaining an example of recognition of a two-dimensional code based on a 360-degree image and a camera image. FIG. FIG. 2 is a diagram for explaining software processing executed by the server 30. FIG. 13 shows a first example of a map displayed by a remote participant device 20-1. FIG. 13 is a diagram showing an example of a details screen displayed by the remote participant device 20-1 when the details screen display button b2 is selected. FIG. 13 shows a second example of a map displayed by a remote participant device 20-1. FIG. 13 is a diagram showing an example of a details screen displayed by the remote participant device 20-1 when the details screen display button b8 is selected. FIG. 13 is a diagram showing an example of a details screen after filtering using a search key. FIG. 9 is a block diagram showing an example of a hardware configuration of an information processing device 900.

Below, a preferred embodiment of the present disclosure will be described in detail with reference to the attached drawings. Note that in this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals to avoid redundant description.

The explanation will be given in the following order.
0. Overview 1. Details of the embodiment 1.1. Example of the configuration of the information processing system 1.2. Example of the functional configuration of the on-site worker device 1.3. Example of the functional configuration of the remote participant device 1.4. Example of the functional configuration of the server 1.5. Details of multimodal processing 1.6. Use of two-dimensional codes 2. Example of the hardware configuration 3. Summary

<0. Overview>
First, an overview of the embodiment of the present disclosure will be described.

Generally, the situation at a work site (such as a construction site) where work is being performed by on-site workers is managed. The site manager (hereinafter simply referred to as the "foreman"), who is responsible for managing the situation at the work site, visits the work site to patrol for safety and manage the progress of construction on a daily basis, and sometimes compiles and manages the progress of work in reports. However, the costs involved in the foreman traveling to the work site, the costs involved in managing matters that need to be corrected, and the costs involved in creating reports can be enormous.

In order to reduce the cost required for the manager to travel to the work site, a technology that allows the manager to manage the situation at the work site from a remote location (hereinafter also referred to as "remote presence") is known. Also, in order for the manager to manage the overall situation at the work site, a system that records the situation at the work site using image data (e.g., still image data or video image data) captured at the work site may be introduced. Note that as an example of image data, video image data captured in all directions from the imaging point (hereinafter also referred to as "360-degree video") may be used.

However, each of these tasks - remote safety patrols, recording the work site situation using 360-degree video, managing items that need to be corrected, and creating reports - is usually done separately. As a result, it is very costly to complete all of these tasks.

Therefore, there is a demand for a system that uses remote presence to record the situation at the work site using 360-degree video, manages items that need to be corrected, and creates reports while linking them all together. It is expected that such a system will enable a manager in a remote location to manage the situation at the work site more efficiently.

The work site is an example of a managed space and an example of a first space. The remote location is an example of a place separate from the managed space and an example of a second space separate from the first space. The on-site worker is an example of a first user who exists in the managed space. The director is merely an example of a remote participant. Therefore, the remote participant may be someone other than the director (for example, a site supervisor, a site technical staff member, a head office staff member, or a client). The remote participant is an example of a second user who manages the status of the managed space.

The above provides an overview of the embodiments of the present disclosure.

1. Details of the embodiment
Next, the embodiments of the present disclosure will be described in detail.

(1.1. Configuration example of information processing system)
First, a configuration example of an information processing system according to an embodiment of the present disclosure will be described with reference to FIG.

FIG. 1 is a diagram showing an example configuration of an information processing system according to an embodiment of the present disclosure. As shown in FIG. 1, an information processing system 1 according to an embodiment of the present disclosure includes an on-site worker device 10, remote participant devices 20-1 to 20-3, a server 30, and a network 40.

The on-site worker device 10, the remote participant devices 20-1 to 20-3, and the server 30 are each connected to a network 40. The on-site worker device 10, the remote participant devices 20-1 to 20-3, and the server 30 are configured to be able to communicate with each other via the network 40. Note that in the following description, the remote participant devices 20-1 to 20-3 may be referred to as "remote participant devices 20" without distinguishing between them.

(Field Worker Device 10)
The on-site worker device 10 is an information processing device realized by a computer and used by a on-site worker. For example, the on-site worker device 10 may be realized by a wearable device attached to the head of the on-site worker. The on-site worker device 10 has a camera 110 ( FIG. 4 ) capable of capturing 360-degree images of the work site. The on-site worker device 10 also has a microphone 120 ( FIG. 4 ) that detects sounds at the work site (hereinafter also referred to as "on-site sounds"). The on-site sounds may include on-site environmental sounds and the sounds of the on-site workers.

The on-site worker device 10 also has a speaker 170 (FIG. 4) that can output the voices of the remote participants received from each of the remote participant devices 20-1 to 20-3 via the server 30. Note that in the example shown in FIG. 1, the on-site worker device 10 does not have a display, but the on-site worker device 10 may have a display, and such a display may be capable of displaying information similar to the information that the remote participant device 20 can display.

(Remote Participant Device 20)
The remote participant device 20 is an information processing device realized by a computer and used by a remote participant. In the example shown in Fig. 1, the remote participant device 20-1 is a PC (Personal Computer), the remote participant device 20-2 is a HMD (Head Mounted Display), and the remote participant device 20-3 is a smartphone. Thus, the specific form of the remote participant device 20 is not particularly limited.

The HMD may be realized by a VR (Virtual Reality) device, an AR (Augmented Reality) device, or an MR (Mixed Reality) device. In the following description, it is assumed that the remote participant using the remote participant device 20-1 is the director. Furthermore, in the example shown in FIG. 1, the number of remote participant devices 20 provided in the information processing system 1 is three. However, the number of remote participant devices 20 provided in the information processing system 1 may be one or two, or may be four or more.

The remote participant device 20 also has a display 280 (FIG. 5) capable of displaying display information corresponding to the 360-degree video of the work site and information related to the viewpoint of the remote participant received from the on-site worker device 10. The remote participant device 20 also has a microphone 220 (FIG. 5) that detects sound in a remote location (hereinafter also referred to as "remote audio"). The remote audio may include the voice of the remote participant. The on-site worker device 10 also has a speaker 270 (FIG. 5) that can output the voice of the on-site worker received from the on-site worker device 10 via the server 30, and the voice of the remote participant received from another remote participant device 20 via the server 30.

(Server 30)
The server 30 is realized by a computer and functions as an example of an information processing device. In the following description, the functions of the on-site worker device 10, the remote participant device 20, and the server 30 will be described in more detail, but some or all of the functions of the server 30 may be possessed by the on-site worker device 10 or the remote participant device 20 instead of the server 30.

(Network 40)
The network 40 relays communication between devices connected to the network 40. The type of the network 40 is not limited. As a specific example, the network 40 may be configured as a so-called wireless network, such as a network based on the Wi-Fi (registered trademark) standard. As another example, the network 40 may be configured as the Internet, a dedicated line, a local area network (LAN), a wide area network (WAN), or the like. The network 40 may include multiple networks, and at least a part of the network may be configured as a wired network.

(Data transmitted between devices)
FIG. 2 is a diagram showing an example of data transmitted and received between devices constituting the information processing system 1 according to an embodiment of the present disclosure.

As shown in FIG. 2, the on-site worker device 10 controls so that display information (hereinafter also referred to as "display image") corresponding to the 360-degree image of the work site obtained by the camera 110 (FIG. 4) and information related to the viewpoints of the remote participants is transmitted in real time to each of the remote participant devices 20-1 to 20-3 via the server 30. Each of the remote participant devices 20-1 to 20-3 receives the display image and displays the received display image on the display 280 (FIG. 5).

Remote participants can grasp the situation at the work site by visually checking the displayed image.

Furthermore, interactions may take place between any combination of on-site workers using on-site worker device 10 and remote participants using remote participant devices 20-1 to 20-3. However, due to space limitations, interactions between a remote participant using remote participant device 20-1 and a remote participant using remote participant device 20-3 are omitted from FIG. 2.

Such dialogue can facilitate smooth communication about the situation at the work site. More specifically, the dialogue can be realized by having one of the on-site worker using the on-site worker device 10 and the remote participants using the remote participant devices 20-1 to 20-3 act as the speaker and the other person as the listener, with the speaker's voice being input to the microphone of the device used by the speaker and presented to the listener via the speaker of the device used by the listener.

(Overall processing example)
Fig. 3 is a diagram showing an example of the overall processing performed by the information processing system 1 according to an embodiment of the present disclosure. As shown in Fig. 3, the overall processing is divided into "real-time processing" and "post-processing". "Real-time processing" is processing performed at the same timing as the 360-degree video of the work site is distributed in real time. "Post-processing" is processing performed after the 360-degree video of the work site is saved in the server 30.

The "real-time processing" includes the transmission of 360-degree video from the on-site worker device 10 to the server 30 (S31), the transmission of on-site audio from the on-site worker device 10 to the server 30 (S32), and the transmission of the on-site worker's position information from the on-site worker device 10 to the server 30 (S33).

The "real-time processing" also includes transmitting remote audio from the remote participant device 20 to the server 30 (S34), transmitting the remote participant's viewpoint direction, FoV (Field of View), and cursor position from the remote participant device 20 to the server 30 (S35), and the like. The viewpoint direction and FoV may be included in the information related to the viewpoint described above.

In this specification, the viewpoint is the position at which the remote participant is looking at the 360-degree video. Therefore, the viewpoint direction can be expressed by the angles (roll angle, pitch angle, and yaw angle) around the three axes (X-axis, Y-axis, and Z-axis) of the direction in which the remote participant is looking at the 360-degree video.

The FoV is the magnification ratio of the displayed image relative to the 360-degree image. Therefore, the displayed image is the image of the 360-degree image within the range defined by the FoV, based on the viewpoint direction of the remote participant. The FoV becomes larger when the remote participant operates to increase the magnification ratio, and becomes smaller when the remote participant operates to decrease the magnification ratio.

The cursor position is the position of the cursor superimposed on the displayed image. The cursor position can be moved on the displayed image by a cursor movement operation by the remote participant. Then, a process corresponding to the cursor position can be executed by a confirm operation by the remote participant.

The "real-time processing" also includes multimodal processing executed by the server 30 (S36). For example, the multimodal processing includes a process in which the server 30 generates a display image based on the 360-degree video, the viewing direction, and the FoV, a process in which the server 30 transmits the display image to the remote participant device 20, and a process in which the remote participant device 20 displays the display image (S37). Note that, below, the display image that is displayed in real time before the 360-degree video is saved by the server 30 is also referred to as the "real-time video."

The multimodal processing also includes a process in which the server 30 extracts the voice of the on-site worker from the on-site voice and transmits the voice of the on-site worker to the remote participant device 20, and a process in which the remote participant device 20 outputs the voice of the on-site worker (S38).

Furthermore, the multimodal processing includes a process in which the server 30 extracts the voice of the remote participant from the remote voice and transmits the voice of the remote participant to the on-site worker device 10, and a process in which the on-site worker device 10 outputs the voice of the remote participant (S39). Note that, although not shown in FIG. 3, the multimodal processing also includes a process in which the server 30 transmits the voice of the remote participant to a remote participant device 20 used by another remote participant, and a process in which the remote participant device 20 used by the other remote participant outputs the voice of the remote participant.

Multimodal processing also includes processing to acquire meta-information such as time information indicating the current time, position information of on-site workers, viewpoint direction of remote participants, FoV, and cursor position. The meta-information is incorporated into tags associated with the video, on-site audio, and remote audio. As will be explained later, other meta-information may also be incorporated into the tag later. In the following explanation, incorporating meta-information into a tag is also referred to as "tagging."

The "real-time processing" also includes a process (S40) executed by the server 30 to save various data (360-degree video, dialogue history, meta information, etc.). The dialogue history includes the voice of the on-site worker and the voice of the remote participant.

"Post-processing" includes a process in which the server 30 transmits to the remote participant device 20 a display image obtained by performing image processing on the stored 360-degree image, and a process in which the server 30 transmits to the remote participant device 20 an audio obtained by performing audio processing on the stored audio of the on-site worker and the stored audio of the remote participant (S41). In addition, "post-processing" includes a process in which the remote participant device 20 displays the display image as a past display image, and a process in which the remote participant device 20 outputs the audio as a past audio (S42).

Note that below, the display image that is displayed based on the 360-degree image that has been temporarily stored by the server 30 and then retrieved is also referred to as "archived image."

The "post-processing" includes a process (S43) in which the server 30 automatically creates a report (reporting) based on the saved tags. The "post-processing" also includes a process (S44) in which the server 30 transmits the report to the remote participant device 20, and a process (S44) in which the remote participant device 20 displays the report.

The above describes an example configuration of the information processing system 1 according to an embodiment of the present disclosure.

(1.2. Example of Functional Configuration of Field Worker Device)
Next, an example of the functional configuration of the field worker device 10 according to the embodiment of the present disclosure will be described with reference to FIG.

FIG. 4 is a diagram showing an example of the functional configuration of the field worker device 10 according to an embodiment of the present disclosure. As shown in FIG. 4, the field worker device 10 includes a camera 110, a microphone 120, a detection unit 130, a control unit 140, a storage unit 150, a communication unit 160, and a speaker 170.

(Camera 110)
The camera 110 captures a 360-degree image by capturing an image of the work site where the worker is present. Here, it is mainly assumed that the camera 110 has two imaging devices, and the images captured by the two imaging devices are combined to obtain the 360-degree image. However, the number of imaging devices that the camera 110 has may be three or more, or may be one.

For example, the lens included in the imaging device may be a fisheye lens. When the imaging device captures an image through a fisheye lens, a wide range of image is obtained, although distortion occurs in the image obtained by the capture. As an example, the angle of view of the fisheye lens may be a value greater than 180 degrees (e.g., 185 degrees). However, the lens included in the imaging device may be a lens other than a fisheye lens.

For example, the lens of the imaging device may be an ultra-wide-angle lens. When the imaging device captures images through the ultra-wide-angle lens, a wide range of image can be obtained (although there is a possibility that a narrower range of image can be obtained compared to when the imaging device captures images through a fisheye lens). Furthermore, when the imaging device captures images through the ultra-wide-angle lens, it is possible to reduce distortion that occurs in the image obtained by imaging.

(Microphone 120)
The microphone 120 detects on-site sounds. As described above, the on-site sounds may include on-site environmental sounds and the voices of on-site workers.

(Detection Unit 130)
The detection unit 130 detects the position information of the on-site worker. For example, the detection unit 130 has a position detection sensor, and detects the position information of the on-site worker by the position detection sensor. For example, the position detection sensor may be a GPS (Global Positioning System) sensor. Alternatively, the detection unit 130 may detect the position information of the on-site worker by a self-position estimation technique based on an image captured by the camera 110. Alternatively, the image captured by the camera 110 may be transmitted to the server 30 by the communication unit 160, and the server 30 may detect the position information of the on-site worker based on the image. In addition, the process of detecting the position information of the on-site worker may be performed in real time processing or in post processing.

The format of the location information of the on-site worker is not limited. As an example, the location information of the on-site worker may be expressed by coordinates on a map of the work site.

(Control unit 140)
The control unit 140 is configured with one or more processors. The processor may be a CPU (Central Processing Unit) or the like. When the control unit 140 is configured with a processor, the processor may be configured with an electronic circuit. The control unit 140 can be realized by the processor executing a program.

(Memory unit 150)
The storage unit 150 is a recording medium including a memory, which stores a program executed by the control unit 140 and stores data necessary for executing the program. The storage unit 150 also temporarily stores data for calculations by the control unit 140. The storage unit 150 is configured by a magnetic storage device, a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.

(Communication unit 160)
The communication unit 160 includes a communication interface and communicates with each of the remote participant devices 20-1 to 20-3 and the server 30 via the network 40. For example, the communication unit 160 transmits on-site audio detected by the microphone 120 to the server 30. The communication unit 160 also receives audio of the remote participants transmitted from the server 30.

(Speaker 170)
The speaker 170 outputs sound under the control of the control unit 140. For example, the speaker 170 outputs the voice of a remote participant received by the communication unit 160.

The above describes an example of the functional configuration of the field worker device 10 according to an embodiment of the present disclosure.

(1.3. Example of Functional Configuration of Remote Participant Device)
Next, an example of the functional configuration of the remote participant device 20 according to an embodiment of the present disclosure will be described with reference to FIG.

FIG. 5 is a diagram showing an example of the functional configuration of a remote participant device 20 according to an embodiment of the present disclosure. As shown in FIG. 5, the remote participant device 20 includes a viewpoint detection unit 210, a microphone 220, an operation detection unit 230, a control unit 240, a storage unit 250, a communication unit 260, a speaker 270, and a display 280.

(Viewpoint detection unit 210)
The viewpoint detection unit 210 detects the viewpoint direction of the remote participant. As described above, the viewpoint direction of the remote participant can be expressed by angles (roll angle, pitch angle, and yaw angle) around three axes in the direction in which the remote participant is looking at the 360-degree video. For example, the viewpoint detection unit 210 may have an acceleration sensor and an orientation sensor. In this case, the viewpoint detection unit 210 may detect the roll angle and pitch angle by the acceleration sensor, and detect the yaw angle by the orientation sensor.

(Microphone 220)
The microphone 220 detects distant audio, which, as noted above, may include the voices of remote participants.

(Operation detection unit 230)
The operation detection unit 230 detects various operations input by remote participants. The specific form of the operation detection unit 230 is not limited. For example, the operation detection unit 230 provided in the remote participant device 20-1 has a mouse and a keyboard and detects operations by the mouse and the keyboard. The operation detection unit 230 provided in the remote participant device 20-2 has a button and detects operations by the button. The operation detection unit 230 provided in the remote participant device 20-3 has a touch panel and detects operations by the touch panel.

(Control unit 240)
The control unit 240 is configured with one or more processors. The processor may be a CPU (Central Processing Unit) or the like. When the control unit 240 is configured with a processor, the processor may be configured with an electronic circuit. The control unit 240 can be realized by the processor executing a program.

(Memory unit 250)
The storage unit 250 is a recording medium including a memory, which stores a program executed by the control unit 240 and stores data necessary for executing the program. The storage unit 250 also temporarily stores data for calculations by the control unit 240. The storage unit 250 is configured by a magnetic storage device, a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.

(Communication unit 260)
The communication unit 260 includes a communication interface and communicates with the on-site worker device 10, the other remote participant devices 20, and the server 30 via the network 40. For example, the communication unit 260 transmits remote voice detected by the microphone 220 to the server 30. The communication unit 260 also receives the voice and display image of the on-site worker transmitted from the server 30.

(Speaker 270)
The speaker 270 outputs sound in accordance with the control of the control unit 240. For example, the speaker 270 outputs the voice of a site worker received by the communication unit 260.

(Display 280)
The display 280 displays various images under the control of the control unit 240. For example, the display 280 displays a display image received by the communication unit 260. As described above, the display image may be an image generated by the server 30 based on the 360-degree image, the viewing direction, and the FoV.

The above describes an example of the functional configuration of the remote participant device 20 according to an embodiment of the present disclosure.

(1.4. Example of Server Functional Configuration)
Next, an example of a functional configuration of the server 30 according to an embodiment of the present disclosure will be described with reference to FIG.

FIG. 6 is a diagram showing an example of the functional configuration of the server 30 according to an embodiment of the present disclosure. As shown in FIG. 6, the server 30 includes a control unit 340, a storage unit 350, and a communication unit 360.

(Control unit 340)
The control unit 340 is configured with one or more processors. The processor may be a CPU (Central Processing Unit) or the like. When the control unit 340 is configured with a processor, the processor may be configured with an electronic circuit. The electronic circuit may also be referred to as a control circuit. The control unit 340 may be realized by the processor executing a program.

(Memory unit 350)
The storage unit 350 is a recording medium including a memory, which stores a program executed by the control unit 340 and stores data necessary for executing the program. The storage unit 350 also temporarily stores data for calculations by the control unit 340. The storage unit 350 is configured by a magnetic storage device, a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.

(Communication unit 360)
The communication unit 360 includes a communication interface, and communicates with the on-site worker device 10 and each of the remote participant devices 20-1 to 20-3 via the network 40.

For example, the communication unit 360 receives 360-degree video and on-site audio from the on-site worker device 10. The communication unit 360 also transmits the display video and the audio of the on-site worker to each of the remote participant devices 20-1 to 20-3. The communication unit 360 also receives remote audio from the remote participant device 20 used by the speaker. The communication unit 360 also transmits the audio of the remote participant to each of the on-site worker device 10 and the remote participant device 20 used by the listener.

The above describes an example of the functional configuration of the server 30 according to an embodiment of the present disclosure.

(1.5. Details of multimodal processing)
Next, details of the multimodal processing according to the embodiment of the present disclosure will be described. As described above, the on-site worker is present at the work site and can directly view the work site. Meanwhile, the remote participant can view the 360-degree video and a display video according to the viewpoint direction and FoV of the remote participant. The on-site worker and the remote participant can have a conversation while checking their own field of view.

For example, there may be cases where an on-site worker wants to make a comment (e.g., a question) about an object that exists at the work site. At this time, there may be cases where a remote participant wants to reply to the comment. Alternatively, there may be cases where a remote participant wants to make a comment (e.g., a request for correction) about an object that exists at the work site. At this time, there may be cases where an on-site worker wants to reply to the comment (e.g., a response about the current status of the response).

Below, we will explain the details of the multimodal processing performed when a remote participant makes an indication, with reference to Figure 7, and explain the details of the multimodal processing performed when an on-site worker makes an indication, with reference to Figure 8.

(When a remote participant makes a comment)
FIG. 7 is a diagram for explaining details of the multimodal processing performed when a remote participant makes an indication.

As shown in FIG. 7, in the server 30, the control unit 340 performs voice recognition (S11) based on the voice of the remote participant (D11) to obtain a voice recognition result. The voice recognition result is obtained as text data. The control unit 340 recognizes indication information indicating an indication based on the voice recognition result obtained from the voice of the remote participant (S12).

Meanwhile, the control unit 340 performs voice recognition (S13) based on the voice of the on-site worker (D12) to obtain a voice recognition result. The voice recognition result is obtained as text data. The control unit 340 recognizes response information indicating a response to the indication based on the voice recognition result obtained from the voice of the on-site worker (S14).

The control unit 340 obtains information about the objects by recognizing the objects present at the work site based on the 360-degree video (D13) (S15). The information about the object may include the type of object and the position information of the object in the 360-degree video. Note that instead of the 360-degree video, the object may be recognized based on the voice recognition results obtained from the voice of the remote participant. The control unit 340 may also take into account the site environmental sounds (D15) when recognizing the object.

The object recognized by the control unit 340 in this manner may be considered to be the object that is being noticed by the remote participant. However, here, in order to increase the accuracy of recognizing the object that is being noticed by the remote participant, the control unit 340 sets the recognized object as a candidate object that is being noticed by the remote participant. Here, it is mainly assumed that there is one object candidate. However, there may be multiple object candidates.

The control unit 340 recognizes the object (point of attention) that the remote participant is paying attention to from among the object candidates that the remote participant is paying attention to, based on the viewpoint direction, FoV, and cursor position (D14) of the remote participant (S16). Note that the cursor position is taken into consideration in order to further improve the accuracy of recognizing the object that the remote participant is paying attention to. Therefore, in cases where there is no cursor position, the cursor position does not need to be taken into consideration.

When recognizing the pointed-out information in S12, it is conceivable that the type of object may be recognized based on the voice recognition results obtained from the voice of the remote participant. In such a case, the control unit 340 may recognize the object pointed out by the remote participant only from the object type recognized together with the pointed-out information.

When the condition that the object pointed out by the remote participant has been recognized is met, the control unit 340 generates tag T11 by associating the meta information already incorporated in tag T11, such as time information (D17) and on-site worker position information (D16), with the meta information to be newly incorporated in tag T11, such as information indicating that the pointer is a remote participant, the type of object pointed out by the remote participant, the information pointed out by the remote participant, and the response information by the on-site worker, and records the generated tag T11 in the storage unit 350. Although not shown, meta information such as the remote participant's viewpoint direction, FoV, and cursor position has also already been incorporated in tag T11.

Note that the condition of recognizing an object that is being called out by a remote participant corresponds to an example of an association start condition for a remote participant. The association start condition for a remote participant is a condition for the control unit 340 to associate new meta information with meta information already incorporated in a tag, and the control unit 340 associates the new meta information with the meta information already incorporated in a tag based on the association start condition for a remote participant being satisfied. However, as will be explained later, the association start condition for a remote participant may be a condition other than the condition of recognizing an object that is being called out by a remote participant.

In the example shown in FIG. 7, tag T11 incorporates "xx:xx" as time information, "position □" as location information, "object A" as the type of object, "△△ (attention such as a request for correction)" as pointed out information, and "response status ●●" as response information. Although not shown, tag T11 may also incorporate additional location information of the object being pointed out by the remote participant.

(When a worker on-site makes a comment)
FIG. 8 is a diagram for explaining details of the multimodal processing performed when a site worker issues an indication.

As shown in FIG. 8, in the server 30, the control unit 340 performs voice recognition (S22) based on the voice of the on-site worker (D23) to obtain a voice recognition result. The voice recognition result is obtained as text data. The control unit 340 recognizes indication information indicating an indication based on the voice recognition result obtained from the voice of the on-site worker (S24).

Meanwhile, the control unit 340 performs voice recognition (S25) based on the voice of the remote participant (D24) to obtain a voice recognition result. The voice recognition result is obtained as text data. The control unit 340 recognizes response information indicating a response to the indication based on the voice recognition result obtained from the voice of the remote participant (S26).

Note that the view direction, FoV, and cursor position (D25) of the remote participant may be taken into account when recognizing the response information. Note that the cursor position is taken into account in order to further increase the accuracy of recognition of the response information by the remote participant. Therefore, in cases where the cursor position does not exist, the cursor position does not need to be taken into account.

The control unit 340 obtains information about the objects by recognizing the objects present at the work site based on the 360-degree video (D21) (S21). The information about the object may include the type of object and the position information of the object in the 360-degree video. Note that instead of the 360-degree video, the object may be recognized based on the voice recognition results obtained from the voice of the on-site worker. The control unit 340 may also take into account the site environmental sounds (D22) when recognizing the object.

The object recognized by the control unit 340 in this manner may be considered to be the object that has been drawn attention to by the on-site worker. However, here, in order to increase the accuracy of recognizing the object that has been drawn attention to by the on-site worker, the control unit 340 sets the recognized object as a candidate object that has been drawn attention to by the on-site worker. Here, it is mainly assumed that there is one object candidate. However, there may be multiple object candidates.

The control unit 340 recognizes the object (point of attention) that the on-site worker is paying attention to from among the candidate objects that the on-site worker is paying attention to, based on the position indicated by the on-site worker (S23). Note that examples of the position indicated by the on-site worker will be described in detail later.

When recognizing the indicated information in S24, it is conceivable that the type of object may be recognized based on the voice recognition results obtained from the voice of the on-site worker. In such a case, the control unit 340 may recognize the object that has been pointed out by the on-site worker only from the object type recognized together with the indicated information.

When the condition that an object called out by a field worker has been recognized is met, the control unit 340 generates tag T21 by associating the meta information already incorporated in tag T21, namely, time information (D27) and location information of the field worker (D26), with the meta information to be newly incorporated in tag T21, namely, information indicating that the person who pointed out the object is a field worker, the type of object called out by the field worker, the information pointed out by the field worker, and response information by the remote participant, and records the generated tag T21 in the storage unit 350. Although not shown, meta information such as the viewpoint direction, FoV, and cursor position of the remote participant has also already been incorporated in tag T21.

The condition of recognizing an object that is being called out by a field worker corresponds to an example of an association start condition related to a field worker. The association start condition related to a field worker is a condition for the control unit 340 to associate new meta information with meta information already incorporated in a tag, and the control unit 340 associates the new meta information with the meta information already incorporated in a tag based on the fact that the association start condition related to a field worker is satisfied. However, as will be explained later, the association start condition related to a field worker may be a condition other than the condition of recognizing an object that is being called out by a field worker.

In the example shown in FIG. 8, tag T21 incorporates "xx:xx" as time information, "position □" as location information, "object A" as object type, "●●" as pointed out information, and "△△" as response information. Although not shown, tag T21 may also incorporate additional location information of an object that has been pointed out by a site worker.

(Voice Recognition)
Next, an example of voice recognition performed by the control unit 340 will be described with reference to FIG.

FIG. 9 is a diagram for explaining an example of voice recognition performed by the control unit 340. Referring to FIG. 9, a voice recognition model M11 is shown. The voice recognition model M11 is a trained model that has been generated in advance by machine learning. For example, the control unit 340 may input voice input from an on-site worker or a remote participant into the voice recognition model M11, and obtain characters output from the voice recognition model M11 as the voice recognition result.

In the example shown in FIG. 9, the "voice" input to the voice recognition model M11 is the voice input from a remote participant. Furthermore, the characters output from the voice recognition model M11 are the characters "let me see it up close." Note that the characters may correspond to text data.

(Recognition of pointed out information)
Next, an example of recognition of indication information performed by the control unit 340 will be described with reference to FIG.

FIG. 10 is a diagram for explaining an example of the recognition of pointed out information performed by the control unit 340. Referring to FIG. 10, a pointed out information recognition model M21 is shown. The pointed out information recognition model M21 is a trained model that has been generated in advance by machine learning.

For example, the control unit 340 may obtain the indication information output from the indication information recognition model M21 based on inputting a sentence composed of a set of characters to the indication information recognition model M21. In this case, if information about an object is additionally output from the indication information recognition model M21 based on inputting a sentence composed of a set of characters to the indication information recognition model M21, the control unit 340 may obtain information about the object in addition to the indication information.

More specifically, the pointed-out information recognition model M21 performs a process of extracting a sentence of interest from multiple subsequent sentences (hereinafter also referred to as the "sentence extraction process"), and then performs a process of classifying the extracted sentence into what it is saying and in what context (hereinafter also referred to as the "sentence classification process").

For example, the sentence extraction process may be a process of setting a window of a predetermined time width, quantifying the attention level within the window, and extracting sentences whose quantified attention level is equal to or exceeds a threshold value. The attention level may be quantified using a recursive function such as a recurrent neural network (RNN).

Alternatively, the sentence extraction process may be a process of extracting sentences using an attention mechanism. Furthermore, the sentence extraction process may be a process in which sentences specialized in the field to which the work belongs are prepared in advance, and sentences are extracted on a rule-based basis using the prepared sentences.

The text classification process may be a process that uses a method such as dependency analysis to extract subjects and predicates, and then performs classification using the extracted subjects and predicates. Alternatively, the text classification process may be a process that uses machine learning, such as an attention mechanism, to perform classification. Alternatively, the text classification process may determine the pointed-out information by setting a label for each piece of pointed-out information and judging which label the text is classified into.

In the example shown in FIG. 10, the "sentence" input to the pointed-out information recognition model M21 is the sentence "Please go to the right. This bag is dangerous, so put it back later" that is recognized based on the voice input from the remote participant. The classification result output from the pointed-out information recognition model M21 is "Object type: bag, pointed-out information: dangerous, put it back."

(Recognition of objects attended to by remote participants)
Next, an example of recognition of an object that is being attended to by a remote participant will be described with reference to FIG.

FIG. 11 is a diagram for explaining an example of recognition of an object that is being noted by a remote participant. Referring to FIG. 11, an object recognition model M31 is shown. The object recognition model M31 is a trained model that has been generated in advance by machine learning.

The 360-degree video G11 is an image obtained by capturing an image of the work site, and is transmitted from the on-site worker device 10 and received by the server 30. Objects B11 and B12 are captured in the 360-degree video G11. Each of objects B11 and B12 is a bag. Objects B13 and B14 are also captured in the 360-degree video G11. Each of objects B13 and B14 is a on-site worker.

Displayed image H11 is an image of the 360-degree image G11 in a range that corresponds to the viewpoint direction and FoV of the remote participant who made the point. Objects B11 and B12 are shown in displayed image H11, but objects B13 and B14 are not. For example, control unit 340 may input 360-degree image G11, the viewpoint direction of the remote participant who made the point, FoV, and cursor position to object recognition model M31, and obtain information about the object that is output from object recognition model M31.

In the example shown in Figure 11, the information obtained about the object is "Object type: bag, Object position information: □".

As described above, instead of the 360-degree video, objects may be recognized based on voice recognition results obtained from the voices of remote participants. The control unit 340 may also take into account on-site environmental sounds when recognizing objects. Alternatively, instead of on-site environmental sounds, characteristics of the on-site environmental sounds (such as information indicating the direction from which the on-site environmental sounds are coming to the on-site worker device 10) may be taken into account.

Furthermore, as described above, it is conceivable that the type of object may be recognized based on the voice recognition results obtained from the voice of the remote participant. In such a case, the control unit 340 may recognize the object that is being pointed out by the remote participant only from the object type that is recognized together with the pointed out information.

(Recognition of objects attended to by field workers)
Next, an example of recognition of an object that is being noticed by a site worker will be described with reference to FIG.

FIG. 12 is a diagram for explaining an example of recognizing an object that is being noted by a site worker. Referring to FIG. 12, a 360-degree video G21 is shown. Objects B21 and B22 are shown in the 360-degree video G21. Objects B21 and B22 are each a bag. Furthermore, object B23 is shown in the 360-degree video G21. Object B23 is a box.

Each of the objects B21 to B23 is assumed to be a candidate object that has been drawn attention of by the on-site worker. Here, it is assumed that the object that the on-site worker wishes to point out is object B23. In this case, the on-site worker indicates the position of object B23 by the position of his/her hand F11.

The control unit 340 may then recognize the object (point of attention) that the on-site worker is drawing attention to from among the candidate objects that the on-site worker is drawing attention to, based on the position indicated by the on-site worker. In the example shown in FIG. 12, object B23 is recognized as the object (point of attention) that the on-site worker is drawing attention to.

(Recognition of response information)
Next, an example of response information recognition performed by the control unit 340 will be described with reference to FIG.

FIG. 13 is a diagram for explaining an example of response information recognition performed by the control unit 340. Referring to FIG. 13, a response information recognition model M41 is shown. The response information recognition model M41 is a trained model that has been generated in advance by machine learning.

For example, the control unit 340 may obtain response information output from the response information recognition model M41 based on inputting a sentence composed of a set of characters into the response information recognition model M41.

More specifically, the response information recognition model M41 performs sentence extraction processing and sentence classification processing, similar to the pointed-out information recognition model M21 (Figure 10).

In the example shown in FIG. 13, the instruction information from the remote participant is the sentence "Please go to the right. The bag here is dangerous, so please put it back later" that is recognized based on the voice input from the remote participant.

Also, if the sentence input to the response information recognition model M41 is "Understood. I will deal with it later," the classification result output from the response information recognition model M41 is "Needs correction."Also, if the sentence input to the response information recognition model M41 is "I have just returned it," the classification result output from the response information recognition model M41 is "Affected."

(Meta information in tags)
Next, an example of meta-information in a tag recorded in the storage unit 350 by the control unit 340 will be described with reference to FIG.

FIG. 14 is a diagram showing an example of meta information in a tag recorded in storage unit 350 by control unit 340. Referring to FIG. 14, tag T31 is shown as an example of a tag recorded in storage unit 350 by control unit 340.

In tag T31, "xx:xx" is incorporated as time information, "position □" is incorporated as the position information of the on-site worker, "bag" is incorporated as the type of object warned about by the remote participant, "it's dangerous so it should be put back" is incorporated as the indication information, and "response status: not yet responded" is incorporated as the response information. Other meta information is also incorporated in tag T31.

Other meta information includes information indicating the person who pointed out the object, position information of the object in the 360-degree video, the viewpoint direction of the remote participant, FoV, and cursor position. If tag T31 shown in FIG. 14 is recorded in storage unit 350, it will be possible to later refer to tag T31 to determine the location of the pointed out object and to play back remote audio and on-site audio at the time the object was pointed out based on time information.

(Modifications of the recognition model)
In the above, an example has been described in which the voice recognition model M11, the pointed-out information recognition model M21, the object recognition model M31, and the response information recognition model M41 are provided separately, but a single recognition model having the same functions as those of these models may be adopted.

FIG. 15 is a diagram for explaining a modified example of the recognition model. Referring to FIG. 15, recognition model M51 is shown. For example, the control unit 340 may input the voice of a remote participant or an on-site worker and a 360-degree video to the recognition model M51, and obtain the type of object, object position information, indication information, and response information that are output from the recognition model M51.

(Modification of Association Start Condition)
In the above, it is mainly assumed that the association start condition for the remote participant is a condition that an object that is being noticed by the remote participant is recognized. However, the association start condition for the remote participant may be a condition other than the condition that an object that is being noticed by the remote participant is recognized. For example, the association start condition for the remote participant may be a condition that a predetermined association start operation is input from the remote participant.

FIG. 16 is a diagram for explaining an example of an association start operation. Referring to FIG. 16, a remote participant device 20-1 is shown. The display 280 of the remote participant device 20-1 displays a display image W11 in real time. The display 280 also displays an association start button W12. For example, the association start operation for a remote participant may be an operation in which the remote participant presses the association start button W12.

(Modification of the recognition of indicated information)
In the above description, it is mainly assumed that the information pointed out by the remote participant is recognized based on the voice recognition result obtained from the voice of the remote participant. However, the information pointed out by the remote participant may be input to the operation detection unit 230.

FIG. 17 is a diagram for explaining a modified example of the recognition of pointed out information. Referring to FIG. 17, a pointed out information input screen W21 displayed on the remote participant device 20 is shown. The pointed out information input screen W21 includes a pointed out information input field W211 and a response status input field W212.

For example, the control unit 340 may recognize the indication information input into the indication information input field W211 based on an operation performed on the operation detection unit 230 to input indication information into the indication information input field W211. Furthermore, the control unit 340 may recognize the response status input into the response status input field W212 based on an operation performed on the operation detection unit 230 to input a response status into the response status input field W212.

Note that the recognition of pointed out information based on the voice recognition results obtained from the voice of the remote participant and the recognition of pointed out information input by the remote participant may be used together. For example, pointed out information recognized based on the voice recognition results obtained from the voice of the remote participant may be corrected by pointed out information input by the remote participant.

(Speaker Identification)
In the above, it is mainly assumed that there is a field worker using the field worker device 10 at the work site. However, in reality, there may be a field worker who is not using the field worker device 10 near the field worker who uses the field worker device 10. Furthermore, as described above, it may be assumed that there are a plurality of remote participants using the remote participant device 20.

In this case, as an example, there is a possibility that the indication information and the information related to the object may not be properly associated. Therefore, it is desirable to identify the speaker who is making the sound, and use the result of identifying the speaker for tagging. With reference to FIG. 18, a modified example in which the result of identifying the speaker is used for tagging will be described.

FIG. 18 is a diagram for explaining a modified example in which the speaker identification result is used for tagging. On-site worker U12 is a field worker who uses the field worker device 10. On-site worker U11 is a field worker who is near the field worker U12 and is not using the field worker device 10. There is a possibility that the field worker U11 and the field worker U12 are visually recognizing different objects. However, there is a possibility that the pointed-out information by the field worker U11 will be associated with information about the object based on the viewpoint direction of the field worker U12.

The control unit 340 should therefore determine whether or not the on-site worker U12 is the speaker. Here, it is assumed that the on-site worker U12 has been identified as the speaker. In such a case, the control unit 340 should recognize the indication information based on the voice recognition results obtained from the voice of the on-site worker U12, based on the fact that the on-site worker U12 has been identified as the speaker.

This can reduce the possibility of matching the information pointed out by on-site worker U11 with information about an object based on the viewpoint direction of on-site worker U12. In other words, even if there are multiple on-site workers at the work site, the possibility of correctly matching the information pointed out by on-site worker U12 with information about an object based on the viewpoint direction of on-site worker U12 increases.

Whether or not a field worker is a speaker may be determined based on the sound detection results by the microphone 120 used by the field worker. As an example, if the volume detected by the microphone 120 used by the field worker is equal to or greater than a threshold, the field worker may be determined to be a speaker. Alternatively, whether or not a field worker is a speaker may be determined based on the sound detection results by the microphone 120 used by the field worker and a machine learning model. Alternatively, if the direction from which the sound detected by the microphone 120 comes can be estimated, whether or not the field worker is a speaker may be determined based on whether or not the direction from which the sound comes is a specific direction (for example, the direction from the position of the field worker's mouth toward the microphone 120).

Furthermore, the viewpoint direction of remote participant A and the viewpoint direction of remote participant B are shown. When referring to the viewpoint direction of remote participant A and the viewpoint direction of remote participant B, it is possible that remote participant A and remote participant B are viewing different objects. However, there is a possibility that the information pointed out by remote participant A will be associated with information about the object based on the viewpoint direction of remote participant B.

The control unit 340 should therefore determine whether or not each of remote participant A and remote participant B is a speaker. Here, it is assumed that remote participant A has been identified as the speaker. In such a case, the control unit 340 should recognize pointed-out information based on the voice recognition results obtained from the voice of remote participant A, based on the fact that remote participant A has been identified as the speaker.

This can reduce the possibility of matching the information pointed out by remote participant A with information about an object based on the viewpoint direction of remote participant B. In other words, even when there are multiple remote participants in remote locations, the possibility of correctly matching the information pointed out by remote participant A with information about an object based on the viewpoint direction of remote participant A increases, and the possibility of correctly matching the information pointed out by remote participant B with information about an object based on the viewpoint direction of remote participant B increases.

Whether or not a remote participant is a speaker may be determined based on the results of sound detection by the microphone 220 used by the remote participant. As an example, if the volume detected by the microphone 220 used by the remote participant is equal to or greater than a threshold, the remote participant may be determined to be a speaker.

(Example of collaboration with AI)
In the above, it is mainly assumed that the remote participant thinks about what to point out. However, by cooperating with AI (Artificial Intelligence), the remote participant can think about what to point out based on information provided by the AI. An example of cooperation with AI will be described with reference to FIG. 19.

FIG. 19 is a diagram for explaining an example of collaboration with AI. Referring to FIG. 19, prediction model M61 is a trained model that has been generated in advance by machine learning. Prediction model M61 is a model that estimates, for each object, the possibility that a remote participant will point out something from the 360-degree video (hereinafter, also referred to as "pointing possibility").

The control unit 340 inputs the 360-degree video G11 into the prediction model M61, and estimates the possibility of each object being pointed out by a remote participant based on the possibility of each object being pointed out output from the prediction model M61.

In the example shown in FIG. 19, the probability that a remote participant will point out object B11 is "90%", the probability that a remote participant will point out object B12 is "60%", the probability that a remote participant will point out object B13 is "75%", and the probability that a remote participant will point out object B14 is "70%".

The control unit 340 generates a display image according to the pointing possibility for each object. The control unit 340 then transmits the generated display image via the communication unit 360 to the remote participant device 20 used by the remote participant. In the remote participant device 20, the control unit 240 then obtains the display image via the communication unit 260 and controls the display of the display image on the display 280.

As an example, when an object exists whose pointing probability is higher than a threshold, the control unit 340 may superimpose an additional image corresponding to the position of the object onto the displayed image. Here, the threshold may be any specific value, but it is assumed that the threshold is 50%. In this case, the control unit 340 determines that the pointing probability of each of objects B11 to B14 is higher than the threshold.

Displayed image H11 is an image of the 360-degree image G11 in a range that corresponds to the viewing direction and FoV of the remote participant who made the pointing. Objects B11 and B12 are shown in displayed image H11. The control unit 340 superimposes additional image W31 and additional image W32 at positions in the 360-degree image G11 that correspond to the relative positions of objects B11 to B14, which have a higher possibility of being pointed out than a threshold, based on the position of displayed image H11.

Here, in the 360-degree video G11, objects B13-B14, whose pointing possibility is higher than the threshold, are located to the right of the position of the display video H11. Therefore, the control unit 340 superimposes the additional image W32 to the right of the objects B11-B12 appearing in the display video H11. Also, objects B11-B12, whose pointing possibility is higher than the threshold, are located inside the display video H11 in the 360-degree video G11. Therefore, the control unit 340 superimposes an additional image W31, which is different from the additional image W32, inside the display video H11.

For example, the remote participant may be able to input an operation to select the additional image W32. At this time, the control unit 340 may change the display image H11 to a range in which the object appears, based on the operation to select the additional image W32 being input by the remote participant to the operation detection unit 230.

The remote participant can consider whether or not to point out each of the objects B11 to B14 whose pointing probability is higher than the threshold value, while taking into account the pointing probability for each object. In addition, the remote participant can instruct the on-site worker by voice to approach the object, so that the on-site worker can quickly check the object.

(Using tags for real-time video)
As described above, the tag is generated by the control unit 340, and the meta information in the tag generated by the control unit 340 may be reflected in the real-time video. An example in which the meta information in the tag is reflected in the real-time video will be described with reference to Fig. 20 .

Figure 20 is a diagram to explain an example in which meta information in a tag is reflected in real-time video.

As shown in FIG. 20, the control unit 340 generates a display image H11 from the 360-degree image G11, the range of which corresponds to the viewpoint direction and FoV of the remote participant who made the point. The control unit 340 transmits the generated display image H11 via the communication unit 360 to the remote participant device 20 used by the remote participant. Then, in the remote participant device 20, the control unit 240 acquires the display image H11 via the communication unit 260, and controls the display of the display image H11 on the display 280.

Furthermore, the control unit 340 may add a pin P11 to a position in the display image H11 that corresponds to the position of the object B11. Furthermore, when an operation to select the pin P11 is input by a remote participant, the control unit 340 may add a tag T41 to a position in the display image H11 that corresponds to the position of the object B11. The tag T41 is similar to the first half of the tag T31 (FIG. 14).

Here, the "bag" included in tag T41 corresponds to the type of object B11 that is being noted by the remote participant. Also, the type of object B11 is an example of information related to object B11 that is being noted by the remote participant.

As the viewpoint of the remote participant moves, the position of the displayed image in the 360-degree image changes, so object B11 may move in the displayed image. Therefore, the control unit 340 moves pin P11 and tag 41 so that they follow object B11. Note that because the position information of the object relative to the 360-degree image is incorporated in the tag, even if the remote participant moves his or her viewpoint, pin P11 and tag 41 can follow the movement of the remote participant's viewpoint.

Referring to FIG. 20, as the on-site worker's viewpoint moves, the 360-degree image G11 changes to a 360-degree image G31. As the remote participant's viewpoint moves, the displayed image H11 changes to a displayed image H21. However, because the pin P11 and tag 41 follow the movement of the remote participant's viewpoint, the tag T41 is added at a position corresponding to the position of the object B11 in the displayed image H21, and the pin P11 is added at a position corresponding to the position of the object B11 in the displayed image H21.

(Using tags for archived footage)
The meta information in the tag generated by the control unit 340 may be reflected in the archived video. An example in which the meta information in the tag is reflected in the archived video will be described with reference to Fig. 21 and Fig. 22. A case in which the meta information in the tag is attached to a timeline will be described with reference to Fig. 21, and a case in which the meta information in the tag is attached to a map of a work site and a displayed video will be described with reference to Fig. 22.

FIG. 21 is a diagram for explaining a case where meta information in a tag is added to a timeline. Referring to FIG. 21, a display image H11 is shown as an example of archived video.

Based on the satisfaction of a predetermined output start condition, the control unit 240 controls the display 280 to display the 360-degree image retrieved from the storage unit 350 and the display image H11 corresponding to the viewpoint direction and FoV of the remote participant. Because the viewpoint direction and FoV of the remote participant remain in the tag, the display image H11 is displayed taking into account the viewpoint direction and FoV of the remote participant, making it possible to efficiently access the pointed-out information from the display image H11.

Here, the specified output start condition may include a condition that an output start operation for the display image H11 is input from a remote participant. The output start operation is not particularly limited. For example, if an output start button is displayed on the display 280, the output start operation may be an operation of selecting the output start button.

A timeline H41 is added to the display image H11. The control unit 340 may add pins P11 and P21 to positions on the timeline H41 that correspond to the playback position in the display image H11 where object B11, which was noted by the remote participant, was recognized. When the remote participant inputs an operation to select pin P21, the control unit 340 may add a tag to a position in the display image H11 that corresponds to the position of the object corresponding to pin P21.

For example, when a remote participant adjusts the playback position to a position corresponding to pin P11, a display image, the voice of the on-site worker, and the voice of the remote participant corresponding to the playback position in the display image H11 where the object B11 corresponding to pin P11 is recognized can be output. Similarly, when a remote participant adjusts the playback position to a position corresponding to pin P21, a display image, the voice of the on-site worker, and the voice of the remote participant corresponding to the playback position in the display image H11 where the object corresponding to pin P21 is recognized can be output.

FIG. 22 is a diagram for explaining a case where meta information in a tag is attached to a map of the work site and a displayed image. Referring to FIG. 22, a map W13 of the work site is shown. The control unit 340 may add a pin P21 to a position on the map W13 of the work site that corresponds to the position information of the object. Furthermore, referring to FIG. 22, a displayed image W11 is shown. The control unit 340 may add a pin P21 to a position on the displayed image W11 that corresponds to the position information of the object.

In addition, because the various data obtained through multimodal processing (360-degree video, dialogue history, meta information, etc.) is stored, remote participants can utilize post-processing to more efficiently access the desired information.

(Automatic highlights from archive footage)
If the entire 360-degree video were to be stored in the storage unit 350 of the server 30, the management costs of the server 30 would become enormous, and the communication load when accessing the archived video would also become large.

Therefore, the control unit 340 highlights the period from a few seconds before the object B11 is recognized to a few seconds after the object B11 is recognized, and reduces the frame rate of the 360-degree video during other periods, thereby significantly reducing the amount of data for the 360-degree video.

(Increasing the resolution of the image of the pointed out area)
There is a demand for the object pointed out by the on-site worker or the remote participant to be made clearer, so the control unit 340 may use super-resolution to increase the resolution of only the periphery of the object pointed out by the on-site worker or the remote participant in the displayed image.

The control unit 340 may also correct blurring caused by shaking of the displayed image, overexposure, and other color tones. The control unit 340 may also utilize stabilization based on image features and super-resolution processing using machine learning.

(Recognizing and displaying overlooked areas)
In the above, a technique for estimating the possibility of pointing out for each object in real time and displaying information about objects with a higher possibility of pointing out than a threshold in real time has been described. However, when displaying the archived video, the remote participant device 20 may display in real time points that may be pointed out by the remote participant that were overlooked by the remote participant.

It is also expected that high-load calculation methods and feedback using information from the future will be used. This will enable various recognitions to be performed with higher accuracy than when real-time processing is performed.

(Tag Management)
The tag obtained by the control unit 340 can be used to manage the situation of the work site. For example, the remote participants can manage the situation of the work site using the SCRUM method. With reference to FIG. 23 , an example of managing the situation of the work site using a general SCRUM method will be described.

FIG. 23 is a diagram showing an example of tag management. Referring to FIG. 23, the tags change in the order of tag T51, tag T52, and tag T53. Remote participants can manage tags as tickets and modify the tag information, such as changes in the situation at the work site, feedback and review to the person who pointed out the problem, and completion.

(Report output)
Furthermore, the tag can be output as a report showing the progress of the work. For example, the site manager needs to report the progress of the work every day to a senior manager or a client. According to the embodiment of the present disclosure, the site manager can automatically output a report showing the progress of the work.

FIG. 24 is a diagram showing an example of a report. Referring to FIG. 24, as an example of a report showing the progress of work, a map W13 of the work site with a pin P21 added, a display image W11 with a pin P21 added, and output information R11 including a tag T41 are shown. The control unit 340 in the server 30 automatically creates and outputs the output information R11. For example, the control unit 340 may automatically create and output the output information R11 based on a report output instruction from the site manager. The output information R12 and output information R13 are each similar information to the output information R11.

For example, suppose that an operation specifying a period (e.g., a day or a month) is input to the operation detection unit 230 by a remote participant. For example, it is assumed that the tag T41 is a tag including time information belonging to that period. It is assumed that the display image W11 is a display image corresponding to the viewpoint direction and FoV associated with the time information belonging to that period. It is assumed that the map W13 of the work site with the pin P21 added is a map of the work site corresponding to the position information of an object associated with the time information belonging to that period.

In such a case, the time information included in output information R11 belongs to the period specified by the remote participants. Similarly, it is assumed that the time information included in output information R12 and output information R13 also belongs to the period specified by the remote participants.

Based on the input of an operation to specify a period, the control unit 340 controls the display 280 to display the output information R11, output information R12, and output information R13, which include time information belonging to that period.

The above describes the details of multimodal processing according to an embodiment of the present disclosure.

(1.6. Use of two-dimensional codes)
Next, the use of two-dimensional codes according to an embodiment of the present disclosure will be described.

(Overview of 2D code usage)
As described above, the remote participants can easily access the desired display video by using the meta information embedded in the tag through tagging by the server 30. In particular, by incorporating the location information of the on-site worker into the tag as meta information, the remote participants can efficiently access the desired display video by using the location information of the on-site worker embedded in the tag.

As described above, the location information of the on-site worker may be detected by a GPS sensor in the on-site worker device 10, or may be detected by self-location estimation technology. Alternatively, the location information of the on-site worker may be detected by the server 30 based on video of the work site. Furthermore, the representation format of the location information of the on-site worker is not limited. As described above, the location information of the on-site worker may be represented by coordinates on a map of the work site. For example, a map may be prepared for each floor of the work site, and one map may be divided into multiple areas.

Here, depending on the location of the on-site worker, the detection accuracy of the GPS sensor may be low. Therefore, in order to improve detection accuracy regardless of the location of the on-site worker, it is possible to use self-location estimation technology. However, detection using self-location estimation technology tends to require high processing costs for detection, and it is difficult to achieve high immediacy of detection. Furthermore, work sites often have a hierarchical structure consisting of multiple floors, and it takes a lot of time to understand the hierarchical structure.

In the following explanation, we propose attaching a two-dimensional code to the work site. First, with reference to Figure 25, we will explain an example of a work site with a two-dimensional code.

FIG. 25 is a diagram showing an example of a work site to which a two-dimensional code has been attached. Referring to FIG. 25, objects B21 and B22 exist at work site J. Furthermore, at work site J, there exists a field worker using a field worker device 10. Furthermore, a two-dimensional code C1 has been attached to a specific location at work site J. The position information of the location to which the two-dimensional code C1 has been attached and the two-dimensional code C1 are associated in advance.

When the two-dimensional code C1 comes within the imaging range of the camera 110 (FIG. 4) provided on the field worker device 10, the two-dimensional code C1 is read by the camera 110, and the time when the two-dimensional code C1 is read is associated with the position information as time information. This makes it clear that the field worker has arrived near the location where the two-dimensional code C1 is attached at the time indicated by the time information.

Furthermore, if the location information of the location where the two-dimensional code C1 is attached incorporates information indicating the floor and information indicating the area on the floor, it can be known when the field worker arrived at which area on which floor. As an example, if it is confirmed in real time that time information is associated with the location information of the location where the two-dimensional code C1 is attached, it can be instantly known that the field worker has arrived near the location where the two-dimensional code C1 is attached.

Furthermore, 360-degree video and on-site audio may be associated with the location information of the location to which the two-dimensional code C1 is attached. By checking the 360-degree video and on-site audio, remote participants can periodically check the situation near the location to which the two-dimensional code C1 is attached.

In the example shown in FIG. 25, the two-dimensional code C1 is a QR (Quick Response) code (registered trademark). However, as will be explained later, the two-dimensional code C1 does not have to be limited to a QR code (registered trademark).

(Hardware processing)
Fig. 26 is a diagram for explaining hardware processing executed by the field worker device 10. As shown in Fig. 26, the control unit 140 included in the field worker device 10 includes a GPU (Graphics Processing Unit) 141 and a CPU 145. The GPU 141 includes a video acquisition unit 142, an image processing unit 143, and an encoder 144.

Multiple camera images obtained by the cameras 110 (FIG. 4) provided on the field worker device 10 are input to the GPU 141 and acquired by the image acquisition unit 142. Meanwhile, IMU information obtained by an IMU (Inertial Measurement Unit) (not shown) provided on the field worker device 10 is input to the CPU 145. The CPU 145 acquires the IMU information and performs filtering on the IMU information to obtain a rotation angle (S147). The CPU 145 outputs the rotation angle to the GPU 141 (S146).

The image processing unit 143 stitches the multiple camera images to generate a composite image (S141), and generates a 360-degree video by rotating the composite image based on the rotation angle (S142). The image processing unit 143 outputs the 360-degree video to the encoder 144 and also to the CPU 145. The encoder 144 encodes the 360-degree video. The encoded 360-degree video is distributed to the remote participant device 20 in real time via the server 30, or is stored in the server 30.

The CPU 145 performs code recognition to recognize the two-dimensional code based on the 360-degree video (S148). In this way, the encoding by the GPU 141 and the code recognition by the CPU 145 are performed in parallel, so that the code recognition can be performed without reducing the processing speed of the GPU 141. The two-dimensional code recognized by the CPU 145 is transmitted to the server 30.

If the field worker device 10 is equipped with a lamp, the CPU 145 may notify the field worker that a two-dimensional code has been recognized by illuminating the lamp. Alternatively, the CPU 145 may notify the field worker that a two-dimensional code has been recognized by outputting a specified sound from the microphone 120. At this time, the CPU 145 may change the color of the illuminated lamp or the sound output for each recognized two-dimensional code.

The CPU 145 also executes hardware control based on the recognition of the two-dimensional code (S149). Here, the hardware control may be hardware control related mainly to the 360-degree video. That is, the CPU 145 may perform control related to the 360-degree video based on the recognition of the two-dimensional code. For example, the control related to the 360-degree video may change depending on which two-dimensional code is recognized.

For example, there may be dangerous locations within a work site. In such a case, a two-dimensional code may be attached near the dangerous location (for example, at the entrance to the dangerous location). In this case, the GPU 141 may control the 360-degree video by automatically starting transmission of the 360-degree video to the server 30 based on the CPU 145 recognizing the two-dimensional code attached near the dangerous location.

In the server 30, the control unit 340 may automatically start to distribute the 360-degree video and the display video according to the viewpoint direction and FoV of the remote participant to the remote participant device 20. Then, in the remote participant device 20, the control unit 240 may start to display the display video on the display 280. This allows the remote participant to easily grasp the situation of the field worker who is in a dangerous location. In addition, in the server 30, the control unit 340 may automatically start to save the 360-degree video.

Alternatively, there may be a private place (such as a toilet) within the work site. In such a case, a two-dimensional code may be attached near the private place (such as the entrance to the private place). In this case, when the two-dimensional code attached near the private place begins to be recognized by the CPU 145 (i.e., when the site worker enters the toilet), the GPU 141 may control the 360-degree video by automatically stopping the transmission of the 360-degree video to the server 30.

In the server 30, the control unit 340 may automatically stop the delivery of the 360-degree video and the display video according to the viewpoint direction and FoV of the remote participant to the remote participant device 20. Then, in the remote participant device 20, the control unit 240 may stop the display of the display video on the display 280. This prevents the remote participants from being aware of the situation of the on-site worker who is in a private location. In addition, in the server 30, the control unit 340 may automatically stop saving the 360-degree video.

On the other hand, when the recognition by the CPU 145 of a two-dimensional code placed near a private location is interrupted and then resumed (i.e., when the field worker leaves the toilet, etc.), the GPU 141 may control the 360-degree video by automatically starting transmission of the 360-degree video to the server 30, in the same way as when a two-dimensional code placed near a dangerous location is recognized.

Alternatively, there may be a place in the work site where the imaging environment is poor (e.g., a dark room). In such a case, a two-dimensional code may be attached near the place where the imaging environment is poor (e.g., the entrance to the place where the imaging environment is poor). In this case, the GPU 141 may control the 360-degree video by changing a predetermined parameter related to the 360-degree video based on the CPU 145 recognizing the two-dimensional code attached near the place where the imaging environment is poor.

More specifically, GPU 141 may control the 360-degree video by changing a specific parameter of camera 110 (FIG. 4) that captures multiple camera images to generate the 360-degree video. For example, the parameter change may be to increase the EV (Exposure Value) to a specific value. This is expected to increase the brightness of the 360-degree video.

It is a concern that the same two-dimensional code may be recognized multiple times by the field worker device 10 within a short period of time. Therefore, the CPU 145 may determine one of the multiple two-dimensional codes recognized within a specified time period as a two-dimensional code that represents the multiple two-dimensional codes (hereinafter also referred to as a "representative two-dimensional code"), and transmit the determined representative two-dimensional code to the server 30. For example, the length of the specified time period may be one minute. Furthermore, the length of the specified time period may be changeable by the field worker, etc.

Also, any one of the multiple two-dimensional codes may be determined as the representative two-dimensional code. For example, the two-dimensional code that is recognized in the middle of the multiple two-dimensional codes may be determined as the representative two-dimensional code.

Alternatively, it may be the case that recognition of a two-dimensional code placed near a private location is initiated (for example, when a field worker enters a restroom). In such a case, the first two-dimensional code to be recognized among multiple two-dimensional codes may be determined to be the representative two-dimensional code.

Alternatively, it may be possible that recognition of a two-dimensional code placed near a private location is interrupted and then resumed (for example, when a field worker leaves a bathroom). In such a case, the last two-dimensional code recognized among multiple two-dimensional codes may be determined as the representative two-dimensional code.

Here, the two-dimensional code may be recognized directly from the 360-degree video, but the two-dimensional code may appear distorted in the 360-degree video. Therefore, when the two-dimensional code is recognized directly from the 360-degree video, the recognition accuracy of the two-dimensional code may not be improved. Therefore, the two-dimensional code may be recognized directly from camera images with less distortion while referring to the 360-degree video. An example of recognition of a two-dimensional code based on such a 360-degree video and camera images will be described with reference to FIG. 27.

FIG. 27 is a diagram for explaining an example of recognition of a two-dimensional code based on a 360-degree image and a camera image. Referring to FIG. 27, a 360-degree image G51, a camera image H51, and a camera image H52 are shown.

Camera image H51 is a camera image from the viewpoint of the on-site worker, and corresponds to area G511 of the 360-degree image G51. Camera image H52 is a camera image from a position moved horizontally from camera image H51, and corresponds to area G512 of the 360-degree image G51. The 360-degree image G51 shows a two-dimensional code C1 with a lot of distortion. On the other hand, camera image H51 shows a two-dimensional code C1 with little distortion.

The CPU 145 then attempts to recognize the two-dimensional code from the camera image H51 corresponding to area G511, and attempts to recognize the two-dimensional code from the camera image H52 corresponding to area G512. In this way, the CPU 145 attempts to recognize the two-dimensional code in the camera images corresponding to areas in all directions while moving the area in the 360-degree image G51 horizontally. This is expected to improve the accuracy of recognizing the two-dimensional code.

(Software processing)
Fig. 28 is a diagram for explaining software processing executed by server 30. As shown in Fig. 28, storage unit 350 included in server 30 stores a database in which "ID", "map", "coordinates" on the map, "time information", and "arrival name" are associated with each other.

In the example shown in FIG. 28, "Map M1" is registered in "Map," and the coordinates (x1, y1) on Map M1 are registered in "Coordinates." The combination of "Map" and "Coordinates" corresponds to an example of location information for a field worker. Furthermore, data may not be registered in "Time Information" and "Arrival Name," or data may already be registered.

When the two-dimensional code is recognized by the field worker device 10, the "two-dimensional code" is transmitted from the field worker device 10 to the server 30. At this time, not only is the two-dimensional code transmitted to the server 30, but the name of the field worker pre-set in the field worker device 10 may also be transmitted from the field worker device 10 to the server 30 as the "arrival name." In the example shown in FIG. 28, the "two-dimensional code" is "1" and the "arrival name" is "Technical staff A."

In the server 30, upon receipt of the two-dimensional code by the communication unit 360, the control unit 340 registers time information indicating the current time in the time information associated with the ID corresponding to the two-dimensional code. In the example shown in FIG. 28, the "time information" is "9/24 14:15." The current time can also be expressed as the time when the two-dimensional code was recognized by the field worker device 10. The control unit 340 also registers "Technical Staff A" in the "Arrival Name" associated with the ID corresponding to the two-dimensional code.

The control unit 340 can transmit the map M1, the coordinates (x, y) on the map M1, the time information "14:15", and "Arrival name: Technical staff A" to the remote participant device 20 via the communication unit 360.

In this way, the time information at which the two-dimensional code was recognized by the field worker device 10 is associated with the map and the coordinates on the map, which are examples of the field worker's location information. This associates the 360-degree image associated with the field worker's location information with the time information, making it easy to identify the 360-degree image that corresponds to the time information at which the two-dimensional code was recognized.

(Regarding location detection of field workers)
As described above, the position of the on-site worker may be detected by a GPS sensor, may be detected by self-location estimation technology, or may be detected by reading a two-dimensional code. In particular, a GPS sensor is suitable for detecting a position outdoors. Therefore, when a work site is outdoors, it is desirable to detect the position of the on-site worker by a GPS sensor.

On the other hand, indoor work sites may have a hierarchical structure consisting of multiple floors. In such cases where the work site is an indoor location consisting of multiple floors, it is desirable to detect the location of on-site workers by reading a two-dimensional code.

Furthermore, multiple position detection methods may be used in combination. For example, if the work site is outdoors, position detection using a GPS sensor and position detection using reading a two-dimensional code may be used in combination. Furthermore, if the work site is indoors, position detection using self-location estimation technology and position detection using reading a two-dimensional code may be used in combination.

Alternatively, the arrival of a field worker at a key location may be detected by reading a two-dimensional code, and the location of the field worker between one key location and another key location may be detected by a GPS sensor or self-location estimation technology.

More specifically, the floor or area where the field worker is present may be detected by reading a two-dimensional code. The detailed location of the field worker within the floor may then be detected by a GPS sensor. Also, the detailed location of the field worker in an outdoor area may be detected by a GPS sensor, and the detailed location of the field worker in an indoor area may be detected by self-location estimation technology.

(Viewed by remote participant device)
As described above, by tagging, a tag is associated with a 360-degree image of a work site, and the 360-degree image and the tag are stored in the storage unit 350 of the server 30. At this time, on-site audio and remote audio related to the 360-degree image may also be stored in the storage unit 350. The tag includes meta information. Examples of meta information include time information, location information of the on-site worker, type of object, name of the person who pointed out the problem, pointed out information, name of the person who responded, and response information. For example, location information of the on-site worker may be expressed by a combination of a map and coordinates on the map.

As described above, there may be cases where an on-site worker is the one who points out a problem and a remote participant is the one who responds, and there may be cases where a remote participant is the one who points out a problem and an on-site worker is the one who responds. In other words, one of the on-site worker and the remote worker may be the one who points out a problem and the other may be the one who responds. However, there may be cases where the pointer and the person who responds are the same or different on-site workers, and there may be cases where the pointer and the person who responds are the same or different remote workers.

In addition, based on the fact that the two-dimensional code attached to the work site is read by the field worker device 10, the location information associated with the two-dimensional code is associated with the time information when the two-dimensional code was read and the name of the field worker using the field worker device 10 (i.e., the name of the person who arrived). The time information and the name of the person who arrived associated with the location information are stored in the memory unit 350 in the server 30.

Furthermore, based on the two-dimensional code being read by the on-site worker device 10, the 360-degree video of the work site may be distributed to the remote participant device 20 via the server 30, or may be stored in the memory unit 350 of the server 30. Hereinafter, the distribution of the 360-degree video to the remote participant device 20 is also simply referred to as "distribution", and the storage of the 360-degree video in the memory unit 350 of the server 30 is also simply referred to as "recording".

In the above, it has been mainly assumed that the distribution or recording of 360-degree video is started based on the field worker device 10 reading a two-dimensional code. However, the distribution of 360-degree video may also be started based on the field worker inputting a predetermined operation to start distribution to the field worker device 10. Also, the recording of 360-degree video may also be started based on the field worker inputting a predetermined operation to start recording to the field worker device 10.

Similarly, the distribution of the 360-degree video may be stopped based on a predetermined operation to stop distribution being input from the field worker to the field worker device 10. Alternatively, the recording of the 360-degree video may be stopped based on a predetermined operation to stop recording being input from the field worker to the field worker device 10.

For example, when the remote participant device 20-1 inputs an operation to select a map to the remote participant device 20, the selected map is displayed by the remote participant device 20-1.

In this case, it is assumed that a tag is associated with the map. In such a case, meta information included in the tag may be displayed. For example, the meta information may be pointed out information, the name of the person who pointed it out, etc. It is also assumed that coordinates on the map, time information when the two-dimensional code was read, and the name of the person who arrived may be associated with the map. In such a case, the coordinates on the map, time information, and the name of the person who arrived may be displayed.

(First display example by remote participant device)
Fig. 29 is a diagram showing a first example of a map displayed by the remote participant device 20-1. As shown in Fig. 29, a map is displayed by the remote participant device 20-1.

Here, it is assumed that a tag is associated with the map, and that the tag includes the coordinate p1 on the map, the name of the person who pointed out the problem "Director", and the pointed out information "Pointed out about XX". In such a case, the remote participant device 20-1 may display the coordinate p1 on the map using a balloon display F1. The balloon display F1 includes the name of the person who pointed out the problem "Director" and the pointed out information "Pointed out about XX".

The balloon display F1 also includes a details screen display button b1. The operation when the details screen display button b1 is selected is almost the same as the operation when the details screen display button b8 in FIG. 31 is selected, so a detailed explanation of the details screen display button b1 will be omitted.

It is also assumed that other combinations of the name of the person who pointed out the problem and the pointed out information are associated with coordinate p1 on the map, but only the combination of the name of the person who pointed out the problem "Director" and the pointed out information "Pointed out about XX" associated with the most recent time information is displayed.

Furthermore, it is assumed that the map coordinate p2, the time information "14:15", and the name of the person who arrived "Technical Staff A" are associated with each other on the map. In such a case, the remote participant device 20-1 may display the map coordinate p2 using a balloon display F2. The balloon display F2 includes the time information "14:15" and the name of the person who arrived "Technical Staff A". The balloon display F2 also includes a details screen display button b2. The details screen display button b2 will be explained in detail later.

Note that, although not shown in FIG. 29, the area to which coordinate p2 belongs is referred to as "Area Rc." It is also assumed that other combinations of time information and arriving person names are associated with coordinate p2 on the map, but only the most recent combination of time information "14:15" and arriving person name "Technical Staff A" is displayed.

In the following explanation, it is assumed that the on-site worker using the on-site worker device 10 is "Technical Staff A" and the remote participant using the remote participant device 20-1 is the "Manager." Furthermore, there is also a on-site worker "Technical Staff B" at the work site who is different from the on-site worker device 10 used by the on-site worker device 10 used by the on-site worker "Technical Staff A." The on-site worker "Technical Staff B" uses a on-site worker device 10 that is different from the on-site worker device 10 used by the on-site worker "Technical Staff A."

The current location of "Technical Staff A" is coordinate p3 on the map, and coordinate p3 on the map is displayed by balloon display F3. Balloon display F3 includes "Technical Staff A's" current status "Recording". Balloon display F3 also includes a distribution request button b3.

When a remote participant using the remote participant device 20-1 inputs an operation to select the distribution request button b3, the remote participant can make a distribution request to technical staff A. For example, if the on-site worker device 10 is equipped with a lamp, the distribution request can be realized by turning on the lamp. Alternatively, the distribution request can be realized by outputting a specified sound from the microphone 120. If the area to which the coordinate p3 belongs is "area Ra," the area "area Ra" may be displayed at the current position p as the current location of "technical staff A." Alternatively, the corresponding floor (e.g., the first floor) on the map may be displayed together with area Ra as the current location of "technical staff A."

The current location of "Technical Staff B" is coordinate p4 on the map, and coordinate p4 on the map is displayed by balloon display F4. Balloon display F4 includes "Technical Staff B's" current status, "Streaming." Balloon display F4 also includes a dialogue participation button b4.

In addition to "recording" and "streaming," other types of status include "no recording stream" and "unavailable." "No recording stream" means that the 360-degree video is neither streamed nor saved. "Unavailable" means that a streaming request cannot be fulfilled, even if one is received from a remote participant. The status is associated with the name of the on-site worker.

The status of the on-site worker may automatically switch from "recording" to "unavailable". For example, the timing of the switch may be when a specified two-dimensional code is recognized by the on-site worker device 10. Alternatively, the timing of the switch may be when the volume detected by the microphone 120 provided in the on-site worker device 10 is greater than a threshold value, or when the brightness of the image detected by the camera 110 is lower than a threshold value. Alternatively, the timing of the switch may be when the communication quality by the communication unit 160 falls below a specified quality.

When a remote participant using the remote participant device 20-1 inputs an operation to select the dialogue participation button b4, the remote participant can join the dialogue being conducted by technical staff B. For example, participation in a dialogue can be achieved by the remote participant's voice being presented to technical staff B, and the voice spoken by technical staff B being presented to the remote participant. If the area to which coordinate p4 belongs is "area Rb," then area "area Rb" may be displayed at current position p as the current location of "technical staff B." Alternatively, the corresponding floor (e.g., the first floor) on the map may be displayed together with area Rb as the current location of "technical staff B."

The video list L1 is a list of 360-degree videos associated with the map displayed by the remote participant device 20-1. The pointed-out information list L2 is a list of pointed-out information included in tags associated with the map.

It is also conceivable that the amount of information displayed by the remote participant device 20-1 (for example, speech bubble display F1 and speech bubble display F2) will increase. For this reason, filtering of the displayed information may be possible. The search key input field E1, emergency button b5, pointed out button b6, and execute button b7 are used for filtering. However, filtering using these is similar to filtering on the details screen 51 (FIG. 30) and the details screen 52 (FIG. 32), so a detailed description of filtering will be omitted here.

Let us assume that a remote participant inputs an operation to select the details screen display button b2. In such a case, the remote participant device 20-1 transmits the coordinate p2 displayed by the speech bubble display F2 to the server 30, and the server 30 transmits a details screen based on the time information and the name of the person who arrived associated with the coordinate p2 to the remote participant device 20-1. In the remote participant device 20-1, the control unit 240 controls the display of the details screen on the display 280.

FIG. 30 shows an example of a details screen displayed by the remote participant device 20-1 when the details screen display button b2 is selected. Referring to FIG. 30, details screen 51 is shown as an example of such a details screen.

Detail screen 51 includes a combination of time information "9/22 16:35" and the name of the person who arrived "Director," a combination of time information "9/23 11:15" and the name of the person who arrived "Technical Staff B," and a combination of time information "9/24 14:15" and the name of the person who arrived "Technical Staff A," all of which are associated with coordinate p2. In addition, selectable times t11 to t13 are displayed at the positions corresponding to each piece of time information.

When a remote participant inputs an operation to select one of the selection target times t11 to t13, time information indicating the selection target time selected by the remote participant, as well as the viewpoint direction and FoV of the remote participant, are transmitted from the remote participant device 20-1 to the server 30.

The server 30 transmits to the remote participant device 20-1 a specific frame in the 360-degree video associated with the time information received from the remote participant device 20-1, and a still image corresponding to the received viewpoint direction and FoV of the remote participant. The remote participant device 20-1 displays the still image transmitted from the server 30. By viewing the still images displayed in this way, the remote participant can check the situation of the same area of the work site in chronological order.

Note that the specified frame may be any of the multiple frames that make up the 360-degree video. For example, the specified frame may be the last frame or the first frame of the multiple frames that make up the 360-degree video.

In the example shown in FIG. 30, based on the selection of the selection target time t13 by the remote participant, a 360-degree image associated with the time information "9/24 14:15" indicating the selection target time t13 and a still image H31 according to the viewpoint direction and FoV of the remote participant are displayed.

In addition, when the remote participant inputs an operation to select the video display button b11, a 360-degree image associated with the time information "9/24 14:15" indicating the selection target time t13 and a video corresponding to the viewpoint direction and FoV of the remote participant may be displayed. By viewing the video displayed in this way, the remote participant can check the detailed changes in the situation at the work site over time.

Also, a remote participant may input an operation to select any two of the selection target times t11 to t13 and select the side-by-side comparison button b12, so that still images related to the two selected selection target times are displayed side-by-side. By looking at the two still images displayed in this way, the remote participant can compare and confirm the situation at the work site at the two different times.

The details screen 51 includes a search key input field E1, a period input field E2, and an execute button b7, which are used for filtering.

For example, filtering by search key may be possible. For example, assume that a remote participant inputs "Director" into the search key input field E1 and inputs an operation to select the execute button b7 to the operation detection unit 230. In such a case, in the remote participant device 20-1, the control unit 240 controls the communication unit 260 to transmit "Director" input into the search key input field E1 to the server 30 as the specified search key.

In the server 30, the communication unit 360 receives the search key "Director", and the control unit 340 identifies the arrival person who includes the search key "Director" from among the arrival people associated with the coordinate p2. The control unit 340 then controls the communication unit 360 so that only the combination of the arrival person "Director" who includes the search key "Director" and the time information "9/22 16:35" is sent to the remote participant device 20-1.

As a result, on the remote participant device 20-1, only the combination of the time information "9/22 16:35" and the arrival person's name "Director", and the selection target time t11 corresponding to this combination, are displayed. This type of filtering makes it easy for remote participants to access the still images and videos they wish to view, for example, when they wish to view only the still images and videos associated with the arrival person's name "Director".

Alternatively, assume that a remote participant inputs "9/24~" into the period input field E2 and inputs an operation to select the execute button b7 to the operation detection unit 230. In such a case, in the remote participant device 20-1, the control unit 240 controls the communication unit 260 so that "9/24~" input into the period input field E2 is sent to the server 30 as the specified period.

In the server 30, the communication unit 360 receives the period "9/24~", and the control unit 340 identifies time information belonging to the period "9/24~" from the time information associated with the coordinate p2. The control unit 340 then controls the communication unit 360 so that only the combination of time information "9/24 14:15" belonging to the period "9/24~" and the arrival person "Technical Staff A" is transmitted to the remote participant device 20-1.

As a result, only the selection target time t13 corresponding to the combination of the time information "9/24 14:15" and the arrival person's name "Technical Staff A" is displayed on the remote participant device 20-1. This type of filtering makes it easy for remote participants to access the still images and videos they wish to view, for example, when they wish to view only the still images and videos associated with the time information "9/24 14:15" that belongs to the period "9/24~".

(Second Example Display by Remote Participant Device)
Fig. 31 is a diagram showing a second example of a map displayed by the remote participant device 20-1. As shown in Fig. 31, a map is displayed by the remote participant device 20-1. As in the example shown in Fig. 29, a coordinate p1 on the map is displayed by a balloon display F1.

Furthermore, assume that a tag including the coordinate p5 on the map, the respondent's name "Technical Staff A", and the response information "XX is unavailable" is associated with the map. In such a case, the remote participant device 20-1 may display the coordinate p5 on the map using a balloon display F8. The balloon display F8 includes the respondent's name "Technical Staff A" and the response information "XX is unavailable". The balloon display F8 also includes a details screen display button b8. Although not shown in FIG. 31, the area to which the coordinate p5 belongs is referred to as "area Rd".

It is assumed that other combinations of the name of the person who pointed out the problem and the pointed out information, and the name of the person who responded and the response information may be associated with coordinate p5 on the map, but only the combination of the responder name "Technical Staff A" and the response information "Cannot handle XX" associated with the most recent time information is displayed.

Let us assume that a remote participant inputs an operation to select the details screen display button b8. In this case, the remote participant device 20-1 transmits the coordinate p5 displayed by the speech bubble display F8 to the server 30, and the server 30 transmits to the remote participant device 20-1 a details screen based on the time information, the name of the person who pointed out the problem or the name of the person who responded, and the pointed out information or the response information associated with the coordinate p5 in the area Rd. In the remote participant device 20-1, the control unit 240 controls the display of the details screen on the display 280.

FIG. 32 is a diagram showing an example of a details screen displayed by the remote participant device 20-1 when the details screen display button b8 is selected. Referring to FIG. 32, details screen 52 is shown as an example of such a details screen.

Details screen 52 includes a combination of the problem-taker's name "Director" and problem information "Point raised about XX," a combination of the problem-taker's name "Director" and problem information "Point raised about △△," a combination of the responder's name "Technical staff member A" and reply information "Status check on △△," and a combination of the responder's name "Technical staff member B" and reply information "△△ dealt with," all of which are associated with coordinates in area Rd to which coordinate p5 belongs.

Due to space limitations, the time information associated with coordinate p5 has been omitted from the details screen 52. Selection target times t24 to t28 are displayed at positions corresponding to the respective time information. The method of using selection target times t24 to t28 is the same as the method of using selection target times t11 to t13 shown in FIG. 30.

In the example shown in FIG. 32, a still image H61 according to the viewpoint direction and FoV of the remote participant and the 360-degree video associated with the time information indicating the selection target time t28 is displayed based on the selection target time t28 being selected by the remote participant. The method of using the video display button b11 and the side-by-side comparison button b12 has already been described with reference to FIG. 30.

The details screen 52 includes a search key input field E1, an emergency button b5, a pointed out button b6, a period input field E2, and an execute button b7, which are used for filtering. The method of using the period input field E2 has already been explained with reference to FIG. 30. The method of using the search key input field E1 shown in FIG. 32 is almost the same as the method of using the search key input field E1 explained with reference to FIG. 30, but instead of the name of the recipient, the name of the person who pointed out the problem, the pointed out information, the name of the person who responded, and the response information may be searched for. Here, an example is explained in which the pointed out information and the response information are searched for.

For example, assume that a remote participant inputs the object type "XX" into the search key input field E1 and inputs an operation to select the execute button b7 to the operation detection unit 230. In such a case, in the remote participant device 20-1, the control unit 240 controls the communication unit 260 so that the "XX" input into the search key input field E1 is sent to the server 30 as the specified search key.

In the server 30, the communication unit 360 receives the search key "XX", and the control unit 340 identifies the indication information and response information that contain the search key "XX" from among the indication information associated with coordinates in area Rd to which coordinate p5 belongs. The control unit 340 then controls the communication unit 360 so that only the indication information that contains the search key "XX", the name of the person who indicated the indication and the time information associated with the indication information, the response information that contains the search key "XX", and the name of the person who responded and the time information associated with the response information are transmitted to the remote participant device 20-1.

FIG. 33 is a diagram showing an example of a details screen after filtering using a search key. Referring to FIG. 33, details screen 53 is shown as an example of such a details screen. As shown in FIG. 33, on the remote participant device 20-1, only the selection target time t24 corresponding to the time information associated with the pointed out information "Pointed out about XX" containing the search key "XX" and the selection target time t28 corresponding to the time information associated with the response information "Cannot handle XX" containing the search key "XX" are displayed. In addition, the selection target time t24 is selected, and a tag T41 and a still image H62 corresponding to the time information indicating the selection target time t24 are displayed.

Furthermore, filtering by the category to which the pointed out information belongs may be possible. For example, the category may be information indicating whether or not an action needs to be taken on the object (whether or not correction is required). As one example, in the server 30, the control unit 340 may determine whether or not an action needs to be taken on the object based on whether or not the pointed out information contains a predetermined term indicating a need for action (for example, "please" or "please")

Alternatively, the category may be information indicating the degree of urgency of responding to the object. As an example, in the server 30, the control unit 340 may determine whether or not responding to the object is urgent based on whether or not the indication information contains a predetermined term indicating urgency (e.g., "urgent"). The category may be associated with the indication information by the control unit 340.

For example, assume that a remote participant selects the emergency button b5 and then inputs an operation to select the execute button b7 to the operation detection unit 230. In such a case, in the remote participant device 20-1, the control unit 240 controls the communication unit 260 so that the category "emergency" is sent to the server 30 as the specified category.

In the server 30, the communication unit 360 receives the category "emergency", and the control unit 340 identifies the indication information belonging to the category "emergency" from the indication information associated with coordinates in the area Rd to which the coordinate p5 belongs. The control unit 340 then controls the communication unit 360 so that only the combination of the indication information belonging to the category "emergency", the name of the person who indicated the indication, and time information is transmitted to the remote participant device 20-1.

As a result, on the remote participant device 20-1, only the combination of the criticism information, the name of the person who criticized, and the time information sent from the server 30, and the selection target time corresponding to that combination are displayed. Note that a category to which the response information belongs may be used instead of the criticism information. For example, the category may be information indicating whether or not the information is a praise from the remote participant to the on-site worker.

Furthermore, filtering may be possible based on the response status of the object. For example, the response status may be information indicating whether or not the object has been pointed out (pointed out or not pointed out). As an example, in the server 30, the control unit 340 may determine whether or not the object has been pointed out based on whether or not the pointed out information has been recognized by the pointed out information recognition model M21.

Alternatively, the response status may be information indicating whether or not the object has been confirmed (confirmed or unconfirmed). As an example, in the server 30, the control unit 340 may determine whether or not the object has been confirmed based on whether or not the response information contains a predetermined term indicating confirmation (e.g., "confirmed").

Alternatively, the response status may be information indicating whether or not an object has been responded to (responded to or not responded to). As an example, in the server 30, the control unit 340 may determine whether or not an object has been responded to based on whether or not the response information contains a predetermined term indicating that a response has been made (e.g., "responded to"). The response status may be associated with information about the object by the control unit 340.

For example, assume that a remote participant selects the "Indicated" button b6 and then inputs an operation to select the "Execute" button b7 to the operation detection unit 230. In such a case, in the remote participant device 20-1, the control unit 240 controls the communication unit 260 to send the response status "Indicated" to the server 30 as the specified response status.

In the server 30, the communication unit 360 receives the response status "pointed out", and the control unit 340 identifies information about the object associated with the response status "pointed out" from information about objects associated with coordinates in the area Rd to which the coordinate p5 belongs. The control unit 340 then controls the communication unit 360 so that only the combination of the pointing out information, the name of the person who pointed out the problem, and time information associated with the information about the object is sent to the remote participant device 20-1.

As a result, on the remote participant device 20-1, only the combination of the pointed out information, the name of the person who pointed out the problem, and the time information sent from the server 30, and the selection target time corresponding to that combination are displayed. The pointed out information may include the type of object as an example of information about the object. For example, the "XX" included in the pointed out information "Pointed out about XX" may correspond to the type of object.

Similarly, filtering by the name of the person who pointed out the problem may be possible. That is, the control unit 340 may identify the name of the person who pointed out the problem that matches the name of the person who pointed out the problem specified by the remote participant from among the names of the people who pointed out the problem that are associated with coordinates in the area Rd to which the coordinate p5 belongs. Then, in the remote participant device 20-1, the control unit 240 may control the display on the display 280 of the pointed out information and time information associated with the identified name of the person who pointed out the problem.

Similarly, filtering by responder name may be possible. That is, the control unit 340 may identify a responder name that matches the responder name specified by the remote participant from among the responder names associated with coordinates in area Rd to which coordinate p5 belongs. Then, in the remote participant device 20-1, the control unit 240 may control the display on the display 280 of the response information and time information associated with the identified responder name.

Filtering by status may be possible. That is, the control unit 340 may identify a status that matches the status specified by the remote participant from among the statuses associated with the names of on-site workers. Then, in the remote participant device 20-1, the control unit 240 may control the display 280 to display the names of on-site workers associated with the identified status.

The above describes the details of the embodiment of the present disclosure.

2. Hardware configuration example
Next, a hardware configuration example of an information processing device 900 as an example of the server 30 according to an embodiment of the present disclosure will be described with reference to Fig. 34. Fig. 34 is a block diagram showing a hardware configuration example of the information processing device 900. Note that the server 30 does not necessarily have to have all of the hardware configuration shown in Fig. 34, and some of the hardware configuration shown in Fig. 34 may not be present in the server 30.

As shown in FIG. 34, the information processing device 900 includes a CPU (Central Processing unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925. The information processing device 900 may have a processing circuit such as a DSP (Digital Signal Processor) or an ASIC (Application Specific Integrated Circuit) instead of or in addition to the CPU 901.

The CPU 901 functions as an arithmetic processing device and control device, and controls all or part of the operations within the information processing device 900 in accordance with various programs recorded in the ROM 902, RAM 903, storage device 919, or removable recording medium 927. The ROM 902 stores programs and arithmetic parameters used by the CPU 901. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change appropriately during the execution. The CPU 901, ROM 902, and RAM 903 are interconnected by a host bus 907 that is composed of an internal bus such as a CPU bus. Furthermore, the host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect/Interface) bus via a bridge 909.

The input device 915 is a device operated by a user, such as a button. The input device 915 may include a mouse, a keyboard, a touch panel, a switch, a lever, and the like. The input device 915 may also include a microphone that detects the user's voice. The input device 915 may be, for example, a remote control device that uses infrared rays or other radio waves, or an externally connected device 929 such as a mobile phone that supports operation of the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs it to the CPU 901. The user operates the input device 915 to input various data to the information processing device 900 and instruct processing operations.

The output device 917 is configured with a device capable of visually or audibly notifying the user of acquired information. The output device 917 may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or a sound output device such as a speaker or a headphone. The output device 917 may also include a PDP (Plasma Display Panel), a projector, a hologram, a printer device, or the like. The output device 917 outputs the results obtained by the processing of the information processing device 900 as a video such as text or an image, or as a sound such as voice or audio. The output device 917 may also include a light to brighten the surroundings.

The storage device 919 is a device for storing data, configured as an example of a storage unit of the information processing device 900. The storage device 919 is configured, for example, with a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. This storage device 919 stores the programs and various data executed by the CPU 901, as well as various data acquired from the outside.

The drive 921 is a reader/writer for a removable recording medium 927 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and is built into the information processing device 900 or is externally attached. The drive 921 reads out information recorded on the attached removable recording medium 927 and outputs it to the RAM 903. The drive 921 also writes information to the attached removable recording medium 927.

The connection port 923 is a port for directly connecting a device to the information processing device 900. The connection port 923 may be, for example, a Universal Serial Bus (USB) port, an IEEE 1394 port, or a Small Computer System Interface (SCSI) port. The connection port 923 may also be an RS-232C port, an optical audio terminal, or a High-Definition Multimedia Interface (HDMI) (registered trademark) port. By connecting an external device 929 to the connection port 923, various types of data may be exchanged between the information processing device 900 and the external device 929.

The communication device 925 is, for example, a communication interface configured with a communication device for connecting to the network 931. The communication device 925 may be, for example, a wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or a communication card for WUSB (Wireless USB). The communication device 925 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communications. The communication device 925 transmits and receives signals, for example, between the Internet and other communication devices using a predetermined protocol such as TCP/IP. The network 931 connected to the communication device 925 is a network connected by wire or wirelessly, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

<3. Summary>
According to an embodiment of the present disclosure, an information processing device is provided in which audio input from a first user present in a first space and image data obtained by capturing an image of the first space are presented to a second user present in a second space separated from the first space, and audio input from the second user is presented to the first user, and the information relating to the second user's viewpoint and display information corresponding to the image data are displayed on a display.The information processing device is provided with a control unit that controls the display of information on a display device that is related to the second user's viewpoint and the image data.

This configuration allows a user in a second space separate from the first space to more efficiently manage the situation in the first space.

The above describes in detail preferred embodiments of the present disclosure with reference to the attached drawings, but the technical scope of the present disclosure is not limited to such examples. It is clear that a person with ordinary knowledge in the technical field of the present disclosure can conceive of various modified or revised examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

For example, in the above, it is mainly assumed that the work performed by the on-site worker is construction work. However, as mentioned above, the work performed by the on-site worker is not limited to construction work. For example, the work performed by the on-site worker may be inspection work to check electronic devices, etc. In this case, the electronic device to be inspected may be the object, but if the electronic device is a stationary type, the position of the object may be registered in advance. Then, if it is detected that the worker is approaching the object, some kind of notification may be given to the worker.

The work performed by the field worker may also be work necessary for agriculture (i.e., farm work). In this case, a farm may correspond to the work site, and the spatial unit in which the crops are grown (e.g., crop area, greenhouse, ridge, etc.) may correspond to the area at the work site. Filtering similar to that described above may be possible for each spatial unit in which the crops are grown. Alternatively, the work performed by the field worker may be surveillance work to monitor a specified surveillance target (e.g., dangers in the surveillance area, suspicious people, etc.).

In the above, a two-dimensional code is associated with location information in advance, and based on the two-dimensional code being read by the camera 110 equipped in the field worker device 10, the server 30 associates the time information at which the two-dimensional code is read with the location information. However, a two-dimensional code other than a QR code (registered trademark) may be used instead of the two-dimensional code. Alternatively, a code having a dimension other than two dimensions (such as a one-dimensional code) may be used instead of the two-dimensional code. Alternatively, an AR (Augmented Reality) marker may be used instead of the two-dimensional code.

Furthermore, reading a two-dimensional code is merely one example of receiving a specific signal. That is, based on the reception of a specific signal by the field worker device 10, the server 30 may associate time information indicating the time the specific signal was received with location information previously associated with the two-dimensional code. For example, the specific signal may be a signal emitted by a beacon. Alternatively, the specific signal may be a signal emitted by a Wi-Fi (registered trademark) base station.

In the above, the case where the on-site worker device 10 is mounted on the head of a field worker has been mainly described. However, the form of the on-site worker device 10 does not have to be limited to a form mounted on the head of a field worker.

For example, the field worker device 10 may be smart glasses. Smart glasses may include a scouter type in which the display is viewed with one eye. Alternatively, the field worker device 10 may be a body-worn camera that is attached to the body of the field worker. Alternatively, the field worker device 10 may be a shoulder-mounted device or a neck-mounted device.

Furthermore, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology disclosed herein may achieve other effects that are apparent to a person skilled in the art from the description in this specification, in addition to or in place of the above effects.

Note that the following configurations also fall within the technical scope of the present disclosure.
(1)
An information processing device in an information processing system, in which a voice input from a first user present in a first space and image data obtained by capturing an image of the first space are presented to a second user present in a second space separated from the first space, and the voice input from the second user is presented to the first user,
a control circuit for controlling display, by a display, of display information corresponding to information related to a viewpoint of the second user and the image data, and information related to an object existing in the first space;
Information processing device.
(2)
A pointability of the object is estimated based on the image data;
The control circuit controls the display of the display information on the display in accordance with the pointing possibility.
The information processing device according to (1).
(3)
the control circuit controls, when an object whose pointing possibility is higher than a threshold value is present, a display on the display such that an additional image according to a position of the object is superimposed on the display information.
The information processing device according to (2).
(4)
the control circuit controls the display so that the display information is changed to a range in which the object is shown, based on the selection of the additional image.
The information processing device according to (3).
(5)
The information about the object is recognized based on a voice input from the first user or the second user.
The information processing device according to any one of (1) to (4).
(6)
The information about the object is recognized based on the image data.
The information processing device according to any one of (1) to (4).
(7)
Based on an association initiation condition for the second user, information about the object is associated with information about the viewpoint;
The association start condition for the second user is a condition that the object is recognized based on a voice input from the second user, a condition that the object is recognized based on the image data, or a condition that a predetermined association start operation is input from the second user.
The information processing device according to any one of (1) to (6).
(8)
The information about the object includes a type of the object and position information of the object in the image data.
The information processing device according to any one of (1) to (7).
(9)
The information about the object is associated with a correspondence status for the object;
the control circuit controls the display of the information about the object on the display based on the fact that the information about the object is identified as being associated with the designated correspondence situation.
The information processing device according to any one of (1) to (7).
(10)
The response status is information indicating whether or not a problem has been pointed out regarding the object, information indicating whether or not the object has been confirmed, or information indicating whether or not a response has been made to the object.
The information processing device according to (9) above.
(11)
The control circuit controls the display of time information associated with the information about the viewpoint on the display.
The information processing device according to any one of (1) to (10).
(12)
The time information is information indicating a time when a predetermined signal is received by the device of the first user.
The information processing device according to (11) above.
(13)
Based on the reception of the predetermined signal, the time information is associated with position information previously associated with the predetermined signal.
The information processing device according to (12).
(14)
the control circuit controls, based on an operation for designating a period, the display of the display information corresponding to information related to the viewpoint associated with time information belonging to the period, on the display.
The information processing device according to any one of (11) to (13).
(15)
The control circuit controls the display of the first user's location information on the display.
The information processing device according to any one of (1) to (14).
(16)
The control circuit controls the display of information indicated by one of the first user and the second user, the information being recognized based on a voice input from the one user.
The information processing device according to any one of (1) to (15).
(17)
The control circuit controls the display of the indication information on the display based on the fact that the one user is identified as matching the designated indicater.
The information processing device according to (16).
(18)
The control circuit controls display of the indication information on the display based on the fact that the indication information is identified as including a specified search key.
The information processing device according to (16).
(19)
The indicated information is associated with a category to which the indicated information belongs,
The control circuit controls display of the indication information on the display based on the fact that the indication information is identified as being associated with a designated category.
The information processing device according to (16).
(20)
The category is information indicating whether or not a response to the object is required, or information indicating a degree of urgency of a response to the object.
The information processing device according to (19).
(21)
The recognition of the indication information based on the voice input from the one user is determined based on the fact that the one user is identified as a speaker.
The information processing device according to any one of (16) to (20).
(22)
the control circuit controls the display, by the display, of the display information corresponding to the information relating to the viewpoint and a map corresponding to position information of an object existing in the first space;
The information processing device according to any one of (1) to (21).
(23)
The control circuit controls the display of indication information by one of the first user and the second user, which is recognized based on a voice input from the one user, and response information by the other user, which is recognized based on a voice input from the other user.
23. The information processing device according to any one of (1) to (22).
(24)
the control circuitry controls the display of response information by the other user on the display based on the other user being identified as matching a designated respondent.
The information processing device according to (23).
(25)
The control circuit controls the display of the display information on the display based on a predetermined output start condition being satisfied,
the predetermined output start condition includes a condition that an operation to start output of the display information is input.
The information processing device according to any one of (1) to (24).
(26)
Control is performed on the image data based on a predetermined signal being received by the device of the first user.
The information processing device according to any one of (1) to (25).
(27)
the control circuit performs control related to the image data by starting or stopping the display of the display information;
The information processing device described in (26).
(28)
The control related to the image data is a change of a predetermined parameter related to the image data.
The information processing device described in (26).
(29)
The information regarding the viewpoint includes a viewpoint direction of the second user and a magnification ratio of the display information with respect to the image data.
The information processing device according to any one of (1) to (28).
(30)
The control circuit includes:
storing information about the object;
automatically generating a report of the work progress in the first space based on the stored information;
30. The information processing device according to any one of (1) to (29).
(31)
the report includes information about the object, the display information, and a map of the vicinity of the location of the object;
The information processing device according to (30).
(32)
An information processing method in an information processing system, in which a voice input from a first user present in a first space and image data obtained by capturing an image of the first space are presented to a second user present in a second space separated from the first space, and the voice input from the second user is presented to the first user,
and a processor controls a display of display information corresponding to the information on the viewpoint of the second user and the image data, and information on an object present in the first space, on a display.
Information processing methods.
(33)
Computer,
An information processing device in an information processing system, in which a voice input from a first user present in a first space and image data obtained by capturing an image of the first space are presented to a second user present in a second space separated from the first space, and the voice input from the second user is presented to the first user,
A program that functions as a control circuit that controls the display, on a display, of information regarding the second user's viewpoint and display information corresponding to the image data, and information regarding objects existing in the first space.

REFERENCE SIGNS LIST 1 Information processing system 10 Site worker device 110 Camera 120 Microphone 130 Detection unit 140 Control unit 150 Storage unit 160 Communication unit 170 Speaker 20 Remote participant device 210 Viewpoint detection unit 220 Microphone 230 Operation detection unit 240 Control unit 250 Storage unit 260 Communication unit 270 Speaker 280 Display 30 Server 340 Control unit 350 Storage unit 360 Communication unit

Claims (33)

An information processing device in an information processing system, in which a voice input from a first user present in a first space and image data obtained by capturing an image of the first space are presented to a second user present in a second space separated from the first space, and the voice input from the second user is presented to the first user,
Display information corresponding to information related to the second user's viewpoint and the image data;
A control circuit for controlling a display of information relating to an object present in the first space by a display.
Information processing device. A pointability of the object is estimated based on the image data;
The control circuit controls the display of the display information on the display in accordance with the pointing possibility.
The information processing device according to claim 1 . the control circuit controls, when an object whose pointing possibility is higher than a threshold value is present, a display on the display such that an additional image according to a position of the object is superimposed on the display information.
The information processing device according to claim 2 . the control circuit controls the display so that the display information is changed to a range in which the object is shown, based on the selection of the additional image.
The information processing device according to claim 3 . The information about the object is recognized based on a voice input from the first user or the second user.
The information processing device according to claim 1 . The information about the object is recognized based on the image data.
The information processing device according to claim 1 . Based on an association initiation condition for the second user, information about the object is associated with information about the viewpoint;
The association start condition for the second user is a condition that the object is recognized based on a voice input from the second user, a condition that the object is recognized based on the image data, or a condition that a predetermined association start operation is input from the second user.
The information processing device according to claim 1 . The information about the object includes a type of the object and position information of the object in the image data.
The information processing device according to claim 1 . The information about the object is associated with a correspondence status for the object;
The control circuit controls the display of the information about the object on the display based on the fact that the information about the object is identified as being associated with the designated correspondence situation.
The information processing device according to claim 1 . The response status is information indicating whether or not a notice has been issued for the object, information indicating whether or not the object has been confirmed, or information indicating whether or not a response has been issued for the object.
The information processing device according to claim 9. The control circuit controls the display of time information associated with the information about the viewpoint on the display.
The information processing device according to claim 1 . The time information is information indicating a time when a predetermined signal is received by the device of the first user.
The information processing device according to claim 11. Based on the reception of the predetermined signal, the time information is associated with position information previously associated with the predetermined signal.
The information processing device according to claim 12. the control circuit controls, based on an operation for designating a period, the display of the display information on the display in accordance with information related to the viewpoint associated with time information belonging to the period.
The information processing device according to claim 11. The control circuit controls the display of the first user's location information on the display.
The information processing device according to claim 1 . The control circuit controls the display of information indicated by one of the first user and the second user, the information being recognized based on a voice input from the one user.
The information processing device according to claim 1 . The control circuit controls the display of the indication information on the display based on the fact that the one user is identified as matching the designated indicater.
The information processing device according to claim 16. The control circuit controls display of the indication information on the display based on the fact that the indication information is identified as including a specified search key.
The information processing device according to claim 16. The indicated information is associated with a category to which the indicated information belongs,
The control circuit controls the display of the indication information on the display based on the fact that the indication information is identified as being associated with a designated category.
The information processing device according to claim 16. The category is information indicating whether or not a response to the object is required, or information indicating a degree of urgency of a response to the object.
The information processing device according to claim 19. The recognition of the indication information based on the voice input from the one user is determined based on the fact that the one user is identified as a speaker.
The information processing device according to claim 16. the control circuit controls the display, by the display, of the display information corresponding to the information relating to the viewpoint and a map corresponding to position information of an object present in the first space;
The information processing device according to claim 1 . The control circuit controls the display of indication information by one of the first user and the second user, which is recognized based on a voice input from the one user, and response information by the other user, which is recognized based on a voice input from the other user.
The information processing device according to claim 1 . the control circuitry controls the display of response information by the other user on the display based on the other user being identified as matching a designated respondent.
The information processing device according to claim 23. The control circuit controls the display of the display information on the display based on a predetermined output start condition being satisfied,
the predetermined output start condition includes a condition that an operation to start output of the display information is input.
The information processing device according to claim 1 . Control is performed on the image data based on a predetermined signal being received by the device of the first user.
The information processing device according to claim 1 . the control circuit performs control related to the image data by starting or stopping the display of the display information;
27. The information processing device according to claim 26. The control related to the image data is a change of a predetermined parameter related to the image data.
27. The information processing device according to claim 26. The information regarding the viewpoint includes a viewpoint direction of the second user and a magnification ratio of the display information with respect to the image data.
The information processing device according to claim 1 . The control circuit includes:
storing information about the object;
automatically generating a report of the work progress in the first space based on the stored information;
The information processing device according to claim 1 . the report includes information about the object, the display information, and a map of the vicinity of the location of the object;
The information processing device according to claim 30. An information processing method in an information processing system, in which a voice input from a first user present in a first space and image data obtained by capturing an image of the first space are presented to a second user present in a second space separated from the first space, and the voice input from the second user is presented to the first user,
and a processor controls a display of display information corresponding to the information on the viewpoint of the second user and the image data, and information on an object present in the first space, on a display.
Information processing methods. Computer,
An information processing device in an information processing system, in which a voice input from a first user present in a first space and image data obtained by capturing an image of the first space are presented to a second user present in a second space separated from the first space, and the voice input from the second user is presented to the first user,
A program that functions as a control circuit that controls the display, on a display, of information regarding the second user's viewpoint and display information corresponding to the image data, and information regarding objects existing in the first space.

Images