JP6528566B2 - Display control method, information processing apparatus and display control program - Google Patents

Description

  The present invention relates to a display control method, an information processing apparatus, and a display control program.

  Using AR (Augmented Reality) technology, a predetermined marker is detected from the image of the real space acquired by the camera of the mobile terminal, and the virtual object associated with the marker is superimposed on the image of the real space Techniques for displaying on a display are known (see, for example, Patent Document 1).

  As one of the applications of such technology, a predetermined procedure etc. is determined based on an AR marker (reference material) attached in advance to a predetermined location of equipment in inspection work of equipment such as a plant or a building. An object of AR content indicating that the image is displayed is superimposed on a camera image and displayed to support work. When the operator wants to know the details such as the procedure indicated by the object, the user can select and confirm the object to display the notes at the time of the operation and the detailed information of the equipment.

JP 2012-103789 A

  The selection of the object described above can be performed without any problem in touch panel operation of a smartphone, a tablet or the like. FIG. 1 is a view showing an example of selecting AR content by tap operation, but tapping on the object of “Procedure 1” with a finger or the like allows coordinate positions on the screen tapped and coordinates of four corners of the object. The coordinate area of the object defined by the position is compared, and if the coordinate area of the object includes a tap position, the object is selected.

  However, in the field work, an HMD (Head Mounted Display) or the like is often used from the viewpoint of work efficiency, and in many cases, it is not possible to perform an operation of tapping the touch panel with a finger or the like. In an environment where such a hand can not be used, an operation is performed on the terminal by a voice command. In that case, the terminal shifts the focus to the next object in accordance with, for example, a voice command "next" based on a list of all the objects prepared in advance as AR content for display. Therefore, there is no problem if the transition destination is close to the object with the current focus, but if the transition destination is far, many voice commands must be issued before the focus transitions to the desired object, and operability is improved. It was bad.

  FIG. 2 is a diagram showing an example of selecting AR content by a voice command. In FIG. 2A, when there is a focus in “Procedure 1” and you want to shift the focus to “Procedure 3”, the focus is transferred from “Procedure 1” to “Procedure 2” with a voice command, and further “Procedure 2” It suffices to shift the focus by the voice command in "Procedure 3". FIG. 2 (b) shows the case where the focus is on "Procedure 1" and it is desired to shift the focus to "Procedure 100", but a total of 99 voices are generated to transit the focus from "Procedure 1" to "Procedure 100". It has to issue a command, which is very complicated. In addition, the focus may shift to AR content not displayed on the screen, and there is also a problem that the usability is lowered.

  Therefore, an object of the present invention is to provide a control technique capable of improving the selectivity of an object being displayed.

In one aspect, each AR (Augmented Reality) content for each reference object recognized from the image captured by the imaging device is displayed superimposed on the image, and display is performed among the AR content groups to which order information is assigned. A plurality of AR contents selected as targets to be displayed on the unit, and the second order of the plurality of AR contents specified based on the first order information respectively assigned to the specified AR contents The computer executes processing for giving information and controlling the selection order of the plurality of AR contents specified according to the given second order information.

  The selectivity of the object being displayed can be improved.

It is a figure which shows the example which selects AR content by tap operation. It is a figure which shows the example which selects AR content by an audio | voice command. It is a figure showing an example of composition of a system concerning one embodiment. It is a figure which shows the software structural example of a worker terminal. It is a figure which shows the software structural example of a server. It is a figure which shows the hardware structural example of a worker terminal. It is a figure showing an example of composition of a server. It is a figure which shows the example of data which a worker terminal hold | maintains. It is a figure which shows the example of data which a server hold | maintains. It is a flowchart which shows the process example of a worker terminal. It is a figure which shows the example of recognition of AR marker. It is a figure which shows the operation example by a voice command.

  Hereinafter, preferred embodiments of the present invention will be described.

<Configuration>
FIG. 3 is a diagram showing an example of the configuration of a system according to an embodiment. In FIG. 3, at the work site, photographing (imaging) is performed by the camera (imaging device) of the worker terminal 1 used by the worker. When an AR marker (reference object) M attached to each place such as a facility is included in the field of view to be photographed, AR content is displayed superimposed on the worker by the AR technology to support the work. The display on the worker terminal 1 is performed by an HMD (Head Mounted Display) or the like, and it is assumed that the operator can perform an operation by a voice command. However, even when the HMD is not used, the case where the operation is performed by the voice command corresponding to the situation where it is difficult for the operator to operate the operator terminal 1 by hand is included. The server 2 provides the worker terminal 1 with basic data for AR display when online.

  FIG. 4 is a view showing an example of the software configuration of the worker terminal 1. In FIG. 4, the worker terminal 1 has, as functions realized by the AR application 11, an AR content generation unit 12, an imaging unit 13, an AR marker recognition unit 14, an AR content display unit 15, a focus order control unit 16 and an audio command. And an application unit 17.

  The AR content generation unit 12 has a function of acquiring basic data for AR display from the server 2 in the online environment and generating AR content to be used for display in the offline environment in advance. The photographing unit 13 has a function of photographing by a camera function. The AR marker recognition unit 14 has a function of recognizing an AR marker (identifying the AR marker and recognizing the arrangement, and the like) from the captured image. The AR content display unit 15 has a function of superimposing and displaying AR content corresponding to the recognized AR marker on the image of the camera. The focus order control unit 16 has a function of controlling the order of focus transition with respect to AR content superimposed and displayed on the image. The voice command application unit 17 has a function of receiving a voice command and applying an operation corresponding to the voice command.

  FIG. 5 is a view showing an example of the software configuration of the server 2. In FIG. 5, the server 2 includes an AR content information provision unit 21. The AR content information providing unit 21 has a function of providing basic data (AR content information) for AR display in response to a request from the worker terminal 1.

  FIG. 6 is a diagram showing an example of the hardware configuration of the worker terminal 1. In FIG. 6, the worker terminal 1 is connected to the bus 100, and the microphone 101, the speaker 102, the camera 103, the display unit 104, the operation unit 105, the sensor unit 106, the power unit 107, the wireless unit 108, and the short distance communication unit 109, an auxiliary storage unit 110, a main storage unit 111, a CPU 112, a drive unit 113, and a recording medium 114.

  The microphone 101 inputs voices uttered by the user and other sounds. The speaker 102 outputs the voice of the other party at the time of using the call function, and outputs the ring tone, the sound effect of the application, and the like. The camera 103 captures an image (moving image, still image) of the real space within the angle of view preset in the terminal. The display unit 104 displays a screen (a screen provided by the terminal OS as a standard, an image captured by a camera, AR object data projected on the screen, and the like) set by the OS and various applications to the user. The screen of the display unit 104 may be a touch panel display or the like, and also has a function as an input unit that acquires input information when the user performs tap / flick / scroll on the screen.

  The operation unit 105 is an operation button displayed on the screen of the display unit 104, a button provided outside the terminal, or the like. The operation button may be a power button, a home button, a volume control button, a back button, or the like. The sensor unit 106 detects a certain point in time or continuous position, orientation, operation and the like of the terminal. Examples include GPS, acceleration sensors, azimuth sensors, geomagnetic sensors, gyro sensors, and the like. The power unit 107 supplies power to each unit of the terminal. The wireless unit 108 is, for example, a transmission / reception unit of communication data which receives a wireless signal / communication data from a base station (mobile network) using an antenna or the like and transmits a wireless signal to the base station. The short distance communication unit 109 can perform short distance communication with a computer such as another terminal by using, for example, a short distance communication method such as infrared communication or WiFi or Bluetooth (registered trademark).

  The auxiliary storage device 110 is a storage such as a hard disk drive (HDD) or a solid state drive (SSD). Based on a control signal from the CPU 112, an application program and a control program provided in the computer are recorded, and input / output is performed as necessary. The main storage device 111 stores an execution program and the like read from the auxiliary storage device 110 according to an instruction from the CPU 112, and stores various information and the like obtained during program execution. The CPU 112 controls each processing of the entire computer, such as various operations and data input / output with each hardware component based on a control program such as an OS or an execution program stored in the main storage device. Implement the process. For example, the drive device 113 can detachably set a storage medium, etc., can read various information recorded in the set recording medium, and can write predetermined information on the recording medium. The recording medium 114 is a computer-readable recording medium that stores an execution program and the like. The functions of the respective units of the worker terminal 1 shown in FIG. 4 are realized by execution of a program by the CPU 112. The program may be provided by a recording medium or may be provided via a network.

  FIG. 7 is a view showing a configuration example of the server 2. In FIG. 7, the server 2 includes a central processing unit (CPU) 202 connected to a system bus 201, a read only memory (ROM) 203, a random access memory (RAM) 204, and a non-volatile random access memory (NVRAM) 205. Have. The server 2 also includes an I / F (Interface) 206 and an I / O (Input / Output Device) 207 connected to the I / F 206, an HDD (Hard Disk Drive) / flash memory 208, and an NIC (Network Interface Card). ), A monitor 210 connected to the I / O 207, a keyboard 211, a mouse 212 and the like. A CD / DVD (Compact Disk / Digital Versatile Disk) drive or the like can also be connected to the I / O 207. The functions of the respective units of the server 2 illustrated in FIG. 5 are realized by execution of a program by the CPU 202. The program may be provided by a recording medium or may be provided via a network.

  FIG. 8 is a view showing an example of data held by the worker terminal 1. In FIG. 8, the worker terminal 1 generates an AR content management table based on the information acquired from the server 2, an in-screen display AR content management table and a focus target AR content management table generated in the worker terminal 1, and voice in advance. It holds a voice command management table set for handling.

  The AR content management table is a table for managing information of AR content to be displayed for each AR marker, and “marker ID” “AR content ID” “coordinate value” “rotation angle” “scale ratio” “texture path” Items such as "display information at the time of selection confirmation" are included. “Marker ID” is information for identifying an AR marker. "AR content ID" is information for identifying AR content. The “coordinate value” is a position at which AR content is displayed (a relative value to the position of the recognized AR marker). The “rotation angle” is an angle at which an image is rotated when displaying AR content. The “scale ratio” is a ratio at which an image is scaled up or down when displaying AR content. "Texture path" is a path in which an image of AR content is stored. The “display information at the time of selection confirmation” is a path in which detailed information and the like displayed when the AR content is selected and confirmed is stored.

  The in-screen display AR content management table is a table for managing the transition sequence of the AR content displayed on the screen and the focus among the AR content corresponding to the recognized AR marker, “AR content ID”, “focus transition” Items such as "order" are included. “AR content ID” is information for identifying AR content displayed in the screen. The “focus transition order” indicates the order in which the focus transitions in the AR content in the table, and as the default, the AR content IDs are in ascending order. However, new order information may be added according to the display position of the AR content ID in the screen. For example, the objects displayed on the left side of the screen can be given an order prior to the objects displayed on the right side.

  The focus target AR content management table is a table corresponding to toggle type focus transition, and is a table for managing AR content with focus at present and next and previous AR content, “AR content ID” “next focus transition target AR Items such as content ID, "previous focus transition target AR content ID" and the like are included. The “AR content ID” is information that identifies AR content that is currently in focus. The “next focus transition target AR content ID” is information identifying the AR content of the transition destination when receiving a forward transition operation (“next” of a voice command or the like). The “previous focus transition target AR content ID” is information for identifying AR content of the transition destination when a transition operation in the reverse direction (“preceding” of a voice command or the like) is received.

  The voice command management table is a table for managing voice commands, and includes items such as “voice command ID”, “voice command”, and “process”. "Voice command ID" is information for identifying a voice command. "Voice command" is the content of the voice command. "Process" is the processing content executed by the voice command.

  FIG. 9 is a view showing an example of data held by the server 2 and holding an AR content management table. The contents are similar to those shown in FIG.

<Operation>
FIG. 10 is a flowchart showing an example of processing of the worker terminal 1. In FIG. 10, when the process is started, the worker terminal 1 activates the AR application 11 (step S101). The AR application 11 also activates a camera function at the time of activation, and the imaging unit 13 starts imaging.

  The AR content generation unit 12 of the activated AR application 11 acquires the latest AR content information (AR content management table) and the like from the server 2 when it is in the online environment (step S102), and generates AR content (step S103). ). When not in the online environment, the AR content generation unit 12 generates AR content based on the previously acquired AR content information, if any.

  Next, the AR marker recognition unit 14 stands by to recognize an AR marker (reference object) in the captured image (step S104). FIG. 11 is a diagram showing an example of recognition of an AR marker. That is, after the AR marker recognition unit 14 captures the AR marker M from the inside of the captured image by the outline, it identifies the AR marker ID by the pattern recorded inside the outline. Then, based on the distortion of the image of the AR marker M, the three-dimensional arrangement of the AR marker M (the inclination, the rotation angle, etc. of the AR marker M) is recognized.

  Returning to FIG. 10, when the AR marker is recognized (Yes in step S104), the AR content display unit 15 superimposes the generated AR content corresponding to the AR marker on the captured image and displays it (step S105).

  Next, the AR content display unit 15 focuses on the AR content with the smallest AR content ID among the AR content initially displayed, as the focus target, and emphasizes the display of the AR content (step S106).

  Next, the focus order control unit 16 calculates AR content within the captured camera angle of view (step S107), and calculates the focus order of AR content within the camera angle of view (step S108). Whether or not there is AR content within the camera angle of view compares the coordinate values of the four corners of the camera screen with the display coordinate values of the AR content (the coordinate values of the four corners in the case of a rectangle), and the coordinate value range of the camera screen It is judged by whether or not the display coordinate value of the AR content is included in the inside. The focus order is, in principle, given in order from the smaller AR content ID, but new order information may be given according to the display position in the screen. For example, the objects displayed on the left side of the screen can be given an order prior to the objects displayed on the right side. The calculated focus order is recorded in the in-screen display AR content management table and the focus target AR content management table.

  Next, when detecting the input of the voice command (Yes in step S109), the voice command application unit 17 determines whether the input is a voice command for changing the focus state of the AR content (step S110). The voice command IDs “1” to “6” in the voice command management table of FIG. 8 are voice commands for changing the focus state of AR content.

  Returning to FIG. 10, if it is an input of a voice command to make the focus state transition (Yes in step S110), the focus order control unit 16 changes the AR content to be the focus target (step S111). The change of the AR content to be focused is performed on the focus target AR content management table.

  After changing the AR content to be focused (step S111), or when the AR marker is not recognized (No in step S104), when the voice command is not input (No in step S109), the focus state is changed When it is not an input of a voice command (No in step S110), the process proceeds to determination of termination of the AR application 11 (step S112).

  If the AR application 11 does not end (No in step S112), the process returns to the AR marker recognition determination (step S104). If the AR application 11 ends (Yes in step S112), the AR application 11 ends (step S113).

  FIG. 12 is a diagram showing an operation example by a voice command. FIG. 12A shows a state in which the AR content of “Procedure 1”, “Procedure 2” and “Procedure 100” is displayed, and the AR content of “Procedure 1” is focused.

  In this state, when the operator issues a voice command of "next", "done" or "miki", as shown in FIG. 12B, the focus shifts to AR content of "procedure 2".

  Furthermore, when the worker issues a voice command of "next", "done" or "miki", as shown in FIG. 12C, the focus shifts to the AR content of "procedure 100".

  In the case of shifting the focus in the reverse direction, an operation can be performed by issuing a voice command of "previous" or "above" or "hidori".

  In addition to or in place of voice commands in Japanese, voice commands in other languages such as English may be used. For example, "next" is an example corresponding to "next".

<Summary>
As described above, according to the present embodiment, the selectivity of the object being displayed can be improved.

  Hereinabove, the preferred embodiments have been described. Although specific embodiments have been shown and described herein, it is apparent that various modifications and changes can be made to these embodiments without departing from the broad spirit and scope as defined in the claims. . That is, it should not be construed as being limited by the details of the specific example and the attached drawings.

  The AR content managed by being assigned an AR content ID in the AR content management table (FIG. 8) is an example of the “object data group”. The AR content managed by the AR content ID being registered in the in-screen display AR content management table (FIG. 8) is an example of “a plurality of object data”. The focus transition order of the in-screen display AR content management table (FIG. 8) is an example of “order information”. The display unit 104 (FIG. 6) is an example of the “display unit”.

The following sections will be further disclosed regarding the above description.
(Supplementary Note 1) Among the object data group to which order information is assigned, a plurality of object data selected as targets to be displayed on the display unit are specified,
Order information is newly added to the identified plurality of object data;
The order of selection of the object data is controlled according to the given order information.
A display control method characterized in that a computer executes a process.
(Supplementary Note 2) The new order information is added according to the display position of the identified plurality of object data.
The display control method according to appendix 1, characterized in that
(Supplementary Note 3) With respect to the objects displayed on the left side of the display unit, the object data displayed on the left side of the display unit is given a previous order than the objects displayed on the right side of the display unit.
The display control method according to appendix 1, characterized in that
(Supplementary Note 4) If the input voice information is analyzed and it is detected that the voice information includes a voice indicating a predetermined instruction, the selection order is the first one from the selection of the first object data. Switch to the selection of the second corresponding object data next to the object,
The display control method according to appendix 1, characterized in that
(Supplementary Note 5) The voice is voice corresponding to the right, next, or next,
The display control method according to appendix 4, characterized in that:
(Supplementary Note 6) A means for specifying a plurality of object data selected as objects to be displayed on the display unit among object data groups to which order information is assigned, and
A means for newly providing order information for the identified plurality of object data;
A unit for controlling the selection order of the object data according to the given order information;
An information processing apparatus comprising:
(Supplementary Note 7) The new order information is added according to the display position of the identified plurality of object data.
The information processing apparatus according to claim 6, characterized in that
(Supplementary Note 8) With respect to the objects displayed on the left side of the display unit, the object data displayed on the left side of the display unit is given a previous order than the objects displayed on the right side of the display unit.
The information processing apparatus according to claim 6, characterized in that
(Supplementary Note 9) When the input voice information is analyzed to detect that the voice information includes a voice indicating a predetermined instruction, the selection order is the first one from the selection of the first object data. Switch to the selection of the second corresponding object data next to the object,
The information processing apparatus according to claim 6, characterized in that
(Supplementary Note 10) The voice is a voice corresponding to the right, next, or next,
The information processing apparatus according to appendix 9, characterized in that
(Supplementary note 11) Among the object data group to which order information is assigned, a plurality of object data selected as targets to be displayed on the display unit are specified,
Order information is newly added to the identified plurality of object data;
The order of selection of the object data is controlled according to the given order information.
Display control program that causes a computer to execute processing.
(Supplementary Note 12) The new order information is added according to the display position of the plurality of identified object data.
The display control program according to appendix 11.
(Supplementary Note 13) With respect to the objects displayed on the left side of the display unit, the object data displayed on the left side of the display unit is given a previous order than the objects displayed on the right side of the display unit.
The display control program according to appendix 11.
(Supplementary Note 14) When the input voice information is analyzed to detect that the voice information includes a voice indicating a predetermined instruction, the selection order is the first one from the selection of the first object data. Switch to the selection of the second corresponding object data next to the object,
The display control program according to appendix 11.
(Supplementary Note 15) The voice is a voice corresponding to the right, next, or next,
The display control program according to appendix 14.

DESCRIPTION OF SYMBOLS 1 Worker terminal 11 AR application 12 AR content generation part 13 Shooting part 14 AR marker recognition part 15 AR content display part 16 Focus order control part 17 Voice command application part 2 Server 21 AR content information provision part

Claims (9)

Each AR (Augmented Reality) content for each reference object recognized from the image captured by the imaging device is displayed superimposed on the image,
Among the AR content groups to which order information is assigned, a plurality of AR contents selected as targets to be displayed on the display unit are specified,
Second order information is added to each of the identified plurality of AR contents based on first order information assigned to each of the identified plurality of AR contents ,
The selection order of the plurality of AR contents specified is controlled according to the given second order information,
A display control method characterized in that a computer executes a process.   When it is detected that the reference object is included in the image photographed by the photographing device, the object data group associated with the reference object is specified,
  Extract a plurality of object data to be displayed on the display device from the specified object data group;
  Order information is given to each of the plurality of extracted object data;
  The selection order of the plurality of extracted object data is controlled according to the given order information.
A display control method characterized in that a computer executes a process.   The plurality of object data to be displayed on the display device are extracted based on the display positions of the plurality of object data included in the object data group and the display range of the display device.
  The display control method according to claim 2, characterized in that: By analyzing the input speech information, detects that it contains sound indicating a predetermined instruction to the audio information, the selection of the first AR content, the selection order of the first AR content follows Switch to the selection of the second AR content that corresponds to
The display control method according to any one of claims 1 to 3 , characterized in that: The voice, following, or a sound corresponding to the next,
The display control method according to claim 4 , characterized in that: Means for superimposing each AR (Augmented Reality) content for each reference object recognized from the image captured by the imaging device on the image, and
A unit for specifying a plurality of AR content items selected as targets to be displayed on the display unit among the AR content groups to which order information is assigned;
A unit for applying second order information to each of the plurality of AR contents specified based on the first order information respectively assigned to each of the plurality of AR contents specified ;
A unit for controlling the selection order of the plurality of AR contents specified according to the given second order information;
An information processing apparatus comprising:   A means for specifying an object data group associated with the reference object when it is detected that the reference object is included in the image captured by the imaging device;
  A unit for extracting a plurality of object data to be displayed on a display device from the specified object data group;
  A unit for providing order information to each of the plurality of extracted object data;
  A control unit configured to control a selection order of the plurality of extracted object data in accordance with the assigned order information;
An information processing apparatus comprising: Each AR (Augmented Reality) content for each reference object recognized from the image captured by the imaging device is displayed superimposed on the image,
Among the AR content groups to which order information is assigned, a plurality of AR contents selected as targets to be displayed on the display unit are specified,
Second order information is added to each of the identified plurality of AR contents based on first order information assigned to each of the identified plurality of AR contents ,
The selection order of the plurality of AR contents specified is controlled according to the given second order information,
Display control program that causes a computer to execute processing.   When it is detected that the reference object is included in the image photographed by the photographing device, the object data group associated with the reference object is specified,
  Extract a plurality of object data to be displayed on the display device from the specified object data group;
  Order information is given to each of the plurality of extracted object data;
  The selection order of the plurality of extracted object data is controlled according to the given order information.
Display control program that causes a computer to execute processing.