JP6125076B2 - Display device, control method, and control program - Google Patents

Description

  The present application relates to a display device, a control method, and a control program.

  Some display devices including a display unit can display an image or the like in a three-dimensional manner (for example, see Patent Document 1). Stereoscopic display is realized using parallax of both eyes.

JP 2011-95547 A

  Although the stereoscopic display is a display format familiar to the user, in the conventional display device, the stereoscopic display is used only for the purpose of viewing and has not been used for improving the convenience of operation. An object of the present invention is to provide a display device, a control method, and a control program capable of providing a user-friendly operation method.

  A display device according to one aspect displays a three-dimensional object for performing a process related to a spreadsheet by displaying an image corresponding to each of both eyes of the user by being mounted. A storage unit that stores a rule in which an operation on the object is associated with a process item included in the process, a detection unit that detects the operation, and a detection unit that detects the operation based on the rule. And a control unit that determines the processing item to be executed according to the operation.

  A control method according to one aspect is executed by a display device that displays an image corresponding to each of both eyes of a user by being worn, and displays an object for executing processing related to a spreadsheet in a three-dimensional manner. A three-dimensional display of the three-dimensional object on a display device, a step of detecting an operation on the object, the operation, and a processing item included in the process are associated with each other Determining a processing item to be executed according to the detected operation with reference to a storage unit storing a rule.

  A control program according to one aspect displays a three-dimensional display of an object for performing a process related to a spreadsheet by displaying an image corresponding to each of the user's eyes when worn. Refer to a storage unit that stores a rule in which the object is three-dimensionally displayed on a display device, a step of detecting an operation on the object, a rule in which the operation is associated with a processing item included in the processing Then, the step of determining the processing item to be executed according to the detected operation is executed.

FIG. 1 is a perspective view of a display device according to the first embodiment. FIG. 2 is a front view of the display device worn by the user. FIG. 3 is a diagram illustrating a modification of the display device. FIG. 4 is a diagram illustrating another modification of the display device. FIG. 5 is a diagram illustrating another modification of the display device. FIG. 6 is a block diagram of the display device according to the first embodiment. FIG. 7 is a diagram illustrating an example of control based on the function provided by the control program. FIG. 8 is a diagram illustrating an example of information stored in object data. FIG. 9 is a diagram illustrating an example of information stored in the action data. FIG. 10 is a diagram illustrating an example of information stored in the action data. FIG. 11 is a diagram illustrating an example of information stored in the action data. FIG. 12 is a diagram illustrating an example of information stored in the action data. FIG. 13 is a diagram illustrating an example of information stored in the action data. FIG. 14 is a diagram illustrating an example of information stored in the action data. FIG. 15 is a diagram for describing a first example of detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. FIG. 16 is a diagram for describing a first example of detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. FIG. 17 is a flowchart illustrating a processing procedure of contact detection processing in the first example. FIG. 18 is a flowchart showing the procedure of the operation detection process in the first example. FIG. 19 is a diagram for describing a second example of detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. FIG. 20 is a flowchart showing the procedure of the operation detection process in the second example. FIG. 21 is a diagram for describing a third example of detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. FIG. 22 is a diagram for describing a third example of detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. FIG. 23 is a flowchart illustrating the procedure of the operation detection process in the third example. FIG. 24 is a diagram for describing a first example of detection of an operation performed by picking up a three-dimensional object. FIG. 25 is a flowchart illustrating a processing procedure of selection detection processing in the first example. FIG. 26 is a flowchart illustrating the procedure of the operation detection process in the first example. FIG. 27 is a diagram for describing a modified example of the first example of detecting an operation performed by picking up a three-dimensional object. FIG. 28 is a diagram for describing a second example of detecting an operation performed by picking up a three-dimensional object. FIG. 29 is a flowchart illustrating a processing procedure of selection detection processing in the second example. FIG. 30 is a diagram for explaining a modified example of the second example of the operation detection performed by picking up the three-dimensional object. FIG. 31 is a diagram for describing a third example of detection of an operation performed by picking up a three-dimensional object. FIG. 32 is a diagram for describing a third example of detection of an operation performed by picking up a three-dimensional object. FIG. 33 is a flowchart of the process procedure of the selection detection process in the third example. FIG. 34 is a flowchart illustrating the procedure of the operation detection process in the third example. FIG. 35 is a diagram for describing a modified example of the third example of detecting an operation performed by picking up a three-dimensional object. FIG. 36 is a block diagram of a display device according to the second embodiment. FIG. 37 is a diagram illustrating a state of a three-dimensional object displayed in the display space when viewed from the user. FIG. 38 is a development view of a three-dimensional object. FIG. 39 is a diagram illustrating an example of a process item determination rule in which an operation detected for a cube or a worksheet is associated with a process item included in a process related to a spreadsheet. FIG. 40 is a diagram illustrating an example of an operation performed on a cube. FIG. 41 is a diagram illustrating an example of a cube display mode. FIG. 42 is a diagram illustrating an example of a cube display mode. FIG. 43 is a diagram illustrating an example of a place where a process related to a spreadsheet is executed in a cube. FIG. 44 is a diagram illustrating a configuration example of a worksheet. FIG. 45 is a diagram illustrating an example of a three-dimensional cell. FIG. 46 is a diagram illustrating an example of displaying a graph in a cube. FIG. 47 shows the concept of cube column or row rotation. FIG. 48 is a diagram illustrating the concept of cube division. FIG. 49 is a diagram showing the concept of selecting cell cubes on a worksheet constituting a cube. FIG. 50 is a diagram illustrating the concept of crushing a cube. FIG. 51 is a diagram showing the concept of erasing one column of a cube. FIG. 52 is a diagram showing the concept of separation of a cell cube from a cube. FIG. 53 is a flowchart illustrating a processing procedure according to the second embodiment.

  Hereinafter, the present invention will be described in detail with reference to the drawings. The present invention is not limited to the following description. In addition, constituent elements in the following description include those that can be easily assumed by those skilled in the art, those that are substantially the same, and those in a so-called equivalent range.

  First, the overall configuration of the display device 1 according to the first embodiment will be described with reference to FIGS. 1 and 2. FIG. 1 is a perspective view of the display device 1. FIG. 2 is a front view of the display device 1 attached by the user. As shown in FIG. 1 and FIG. 2, the display device 1 is a head mount type device that is worn on the user's head.

  The display device 1 includes a front surface portion 1a, a side surface portion 1b, and a side surface portion 1c. Front part 1a is arranged in front of the user so as to cover both eyes of the user when worn. The side surface portion 1b is connected to one end portion of the front surface portion 1a, and the side surface portion 1c is connected to the other end portion of the front surface portion 1a. The side surface portion 1b and the side surface portion 1c are supported by a user's ear like a vine of glasses when worn, and stabilize the display device 1. The side part 1b and the side part 1c may be configured to be connected to the back of the user's head when worn.

  The front part 1a is provided with a display part 32a and a display part 32b on the surface facing the user's eyes when worn. The display unit 32a is disposed at a position facing the user's right eye when worn, and the display unit 32b is disposed at a position facing the user's left eye when worn. The display unit 32a displays a right-eye image, and the display unit 32b displays a left-eye image. Thus, by providing the display unit 32a and the display unit 32b that display an image corresponding to each eye of the user when worn, the display device 1 realizes three-dimensional display using binocular parallax. Can do.

  The display unit 32a and the display unit 32b may be configured by a single display device as long as different images can be independently provided to the right and left eyes of the user. For example, one display device may be configured to independently provide different images for the right eye and the left eye by switching at high speed a shutter that blocks the displayed image so that only one eye can see it. . The front part 1a may be configured to cover the eyes of the user so that external light does not enter the eyes of the user when worn.

  The front surface portion 1a includes a photographing unit 40 and a photographing unit 42 on the surface opposite to the surface on which the display unit 32a and the display unit 32b are provided. The photographing unit 40 is disposed in the vicinity of one end portion (right eye side when worn) of the front surface portion 1a, and the photographing portion 42 is located in the vicinity of the other end portion (left eye side when worn) of the front surface portion 1a. Arranged. The imaging unit 40 acquires an image in a range corresponding to the field of view of the user's right eye. The imaging unit 42 acquires an image in a range corresponding to the field of view of the user's left eye. The field of view here is a field of view when the user is looking at the front, for example.

  The display device 1 displays an image photographed by the photographing unit 40 on the display unit 32a as an image for the right eye, and displays an image photographed by the photographing unit 42 on the display unit 32b as an image for the left eye. For this reason, the display device 1 can provide the user with the same scene as when the display device 1 is not worn, even if the field of view is blocked by the front surface portion 1a.

  In addition to the function of providing an actual scene to the user, the display device 1 displays virtual information three-dimensionally and allows the user to operate the virtual information. Have According to the display device 1, the virtual information is displayed so as to be superimposed on the actual scene as if it actually exists. Then, the user can operate the virtual information as if he / she actually touched the virtual information by using his / her hand, for example, to change the virtual information. As described above, the display device 1 provides an intuitive and highly convenient operation method for virtual information. In the following description, virtual information displayed three-dimensionally by the display device 1 may be referred to as a “three-dimensional object”.

  The display device 1 provides the user with the same wide field of view as when the display device 1 is not mounted. And the display apparatus 1 can arrange | position a three-dimensional object by arbitrary magnitude | sizes in the arbitrary positions in this wide visual field. As described above, the display device 1 can display three-dimensional objects having various sizes at various positions in a wide space without being restricted by the size of the display device.

  1 and 2 show an example in which the display device 1 has a shape like glasses (goggles), the shape of the display device 1 is not limited to this. For example, the display device 1 may have a helmet-type shape that covers substantially the upper half of the user's head, like the display device 2 shown in FIG. 3. Alternatively, the display device 1 may have a mask type shape that covers almost the entire face of the user, like the display device 3 shown in FIG. The display device 1 may be configured to be connected to an external device 4d such as an information processing device or a battery device by wire or wirelessly like the display device 4 shown in FIG.

  Next, a functional configuration of the display device 1 will be described with reference to FIG. FIG. 6 is a block diagram of the display device 1 according to the first embodiment. As shown in FIG. 6, the display device 1 includes an operation unit 13, a control unit 22, a storage unit 24, display units 32a and 32b, photographing units 40 and 42, a detection unit 44, and a distance measurement unit 46. And have. The operation unit 13 receives basic operations such as starting and stopping the display device 1 and changing the operation mode.

  The display units 32 a and 32 b include display devices such as a liquid crystal display (LCD) and an organic EL (Organic Electro-Luminescence) panel, and display various types of information according to control signals input from the control unit 22. The display units 32a and 32b may be projection devices that project an image onto the user's retina using a light source such as a laser beam.

  The imaging units 40 and 42 electronically capture an image using an image sensor such as a charge coupled device image sensor (CCD) or a complementary metal oxide semiconductor (CMOS). The photographing units 40 and 42 convert the photographed images into signals and output the signals to the control unit 22.

  The detection unit 44 detects an actual object that exists in the imaging range of the imaging units 40 and 42. For example, the detection unit 44 detects an object that matches a pre-registered shape (for example, the shape of a human hand) among real objects existing in the imaging range. The detection unit 44 detects the range (shape and size) of the actual object in the image based on the brightness, saturation, hue edge, and the like of the object even if the shape is not registered in advance. It may be configured.

  The distance measuring unit 46 measures the distance to an actual object existing in the imaging range of the imaging units 40 and 42. The distance to the actual object is measured for each eye based on the position of each eye of the user wearing the display device 1. For this reason, when the reference position where the distance measuring unit 46 measures the distance is shifted from the position of each eye, the measured value of the distance measuring unit 46 indicates the distance to the eye position according to the difference. It is corrected to represent.

  In the present embodiment, the imaging units 40 and 42 also serve as the detection unit 44 and the distance measurement unit 46. That is, in this embodiment, an object within the shooting range is detected by analyzing the images shot by the shooting units 40 and 42. Furthermore, the distance from the object is measured (calculated) by comparing the object included in the image captured by the capturing unit 40 with the object included in the image captured by the capturing unit 42.

  The display device 1 may include a detection unit 44 separately from the imaging units 40 and 42. The detection unit 44 may be, for example, a sensor that detects an actual object existing in the imaging range using at least one of visible light, infrared light, ultraviolet light, radio waves, sound waves, magnetism, and capacitance. The display device 1 may include a distance measuring unit 46 separately from the imaging units 40 and 42. The distance measuring unit 46 is a sensor that detects the distance to an actual object existing in the imaging range using at least one of visible light, infrared light, ultraviolet light, radio waves, sound waves, magnetism, and capacitance, for example. Also good. The display device 1 may include a sensor that can also serve as the detection unit 44 and the distance measurement unit 46, such as a sensor using a TOF (Time-of-Flight) method.

  The control unit 22 includes a CPU (Central Processing Unit) that is a calculation unit and a memory that is a storage unit, and implements various functions by executing programs using these hardware resources. Specifically, the control unit 22 reads a program or data stored in the storage unit 24 and expands it in a memory, and causes the CPU to execute instructions included in the program expanded in the memory. The control unit 22 reads / writes data from / to the memory and the storage unit 24 and controls the operation of the display unit 32a and the like according to the execution result of the instruction by the CPU. When the CPU executes an instruction, the data developed in the memory and the operation detected via the detection unit 44 are used as a part of parameters and determination conditions.

  The storage unit 24 includes a nonvolatile storage device such as a flash memory, and stores various programs and data. The program stored in the storage unit 24 includes a control program 24a. The data stored in the storage unit 24 includes object data 24b, action data 24c, and virtual space data 24d. The storage unit 24 may be configured by a combination of a portable storage medium such as a memory card and a read / write device that reads from and writes to the storage medium. In this case, the control program 24a, object data 24b, action data 24c, and virtual space data 24d may be stored in a storage medium. Further, the control program 24a, the object data 24b, the action data 24c, and the virtual space data 24d may be acquired from other devices such as a server device by wireless communication or wired communication.

  The control program 24 a provides functions related to various controls for operating the display device 1. The functions provided by the control program 24a include a function for superimposing a three-dimensional object on the images acquired by the photographing units 40 and 42 and displaying the same on the display units 32a and 32b, a function for detecting an operation on the three-dimensional object, and a detected operation. A function for changing the three-dimensional object in response to this is included.

  The control program 24 a includes a detection processing unit 25, a display object control unit 26, and an image composition unit 27. The detection processing unit 25 provides a function for detecting an actual object existing in the imaging range of the imaging units 40 and 42. The function provided by the detection processing unit 25 includes a function of measuring the distance to each detected object.

  The display object control unit 26 provides a function for managing what three-dimensional objects are arranged in the virtual space and what state each three-dimensional object is in. The functions provided by the display object control unit 26 include detecting an operation on a three-dimensional object based on the movement of an actual object detected by the function of the detection processing unit 25, and changing the three-dimensional object based on the detected operation. The function to be included is included.

  The image composition unit 27 provides a function for generating an image to be displayed on the display unit 32a and an image to be displayed on the display unit 32b by synthesizing the image in the real space and the image in the virtual space. The function provided by the image compositing unit 27 includes the real object and 3 based on the distance to the real object measured by the function of the detection processing unit 25 and the distance from the viewpoint to the three-dimensional object in the virtual space. A function to determine the context of the dimension object and adjust the overlap is included.

  The object data 24b includes information regarding the shape and properties of the three-dimensional object. The object data 24b is used for displaying a three-dimensional object. The action data 24c includes information regarding how an operation on the displayed three-dimensional object acts on the three-dimensional object. The action data 24c is used to determine how to change the three-dimensional object when an operation on the displayed three-dimensional object is detected. The change here includes movement, rotation, deformation, disappearance, replacement, and the like. The virtual space data 24d holds information regarding the state of the three-dimensional object arranged in the virtual space. The state of the three-dimensional object includes, for example, a position, a posture, a deformation state, and the like. Note that replacement means replacing one object with another object.

  Next, an example of control based on the function provided by the control program 24a will be described with reference to FIG. The image P1a is an image obtained by the photographing unit 40, that is, an image corresponding to a sight of a real space viewed with the right eye. The table P1 and the user's hand H1 are shown in the image P1a. The display device 1 also acquires an image obtained by photographing the same scene by the photographing unit 42, that is, an image corresponding to a sight when the real space is viewed with the left eye.

  The image P2a is a right-eye image generated based on the virtual space data 24d and the object data 24b. In this example, the virtual space data 24d holds information related to the state of the block-shaped three-dimensional object BL1 existing in the virtual space, and the object data 24b holds information related to the shape and properties of the three-dimensional object BL1. The display device 1 reproduces the virtual space based on these pieces of information, and generates an image P2a obtained by viewing the reproduced virtual space from the viewpoint of the right eye. The position of the right eye (viewpoint) in the virtual space is determined based on a predetermined rule. Similarly, the display device 1 also generates an image obtained by viewing the reproduced virtual space from the viewpoint of the left eye. That is, the display device 1 also generates an image in which the three-dimensional object BL1 is displayed three-dimensionally by combining with the image P2a.

  In step S1 shown in FIG. 7, the display device 1 combines the image P1a and the image P2a to generate an image P3a. The image P3a is an image displayed on the display unit 32a as a right-eye image. At this time, the display device 1 determines the front-rear relationship between the real object in the shooting range of the shooting unit 40 and the three-dimensional object existing in the virtual space, with the position of the user's right eye as a reference. When the real object and the three-dimensional object overlap, the overlap is adjusted so that the object closer to the user's right eye can be seen in the front.

  Such overlapping adjustment is performed for each range of a predetermined size (for example, for each pixel) in the region on the image where the real object and the three-dimensional object overlap. For this reason, the distance from the viewpoint in the real space to the real object is measured for each range of a predetermined size on the image. Further, the distance from the viewpoint in the virtual space to the three-dimensional object is calculated for each range of a predetermined size on the image in consideration of the position, shape, posture, etc. of the three-dimensional object.

  In the scene of step S1 shown in FIG. 7, the three-dimensional object BL1 is arranged in a position corresponding to the position immediately above the position where the table T1 exists in the real space in the virtual space. Furthermore, in the scene of step S1 shown in FIG. 7, the user's hand H1 and the three-dimensional object BL1 exist in substantially the same direction and at substantially the same distance with respect to the position of the user's right eye. For this reason, by adjusting the overlap for each range of a predetermined size, in the combined image P3a, in the portion corresponding to the thumb of the hand H1 in the region where the hand H1 and the three-dimensional object BL1 overlap. The hand H1 appears on the front surface, and the three-dimensional object BL1 appears on the front surface in other portions. Furthermore, in the area where the table T1 and the three-dimensional object BL1 overlap, the three-dimensional object BL1 appears on the front surface.

  By adjusting the overlap, in step S1 shown in FIG. 7, an image P3a is obtained as if the three-dimensional object BL1 is placed on the table T1 and the user grasps the three-dimensional object BL1 with the hand H1. It has been. The display device 1 generates an image displayed on the display unit 32b as an image for the left eye by synthesizing the image captured by the imaging unit 42 and the image obtained by viewing the virtual space from the viewpoint of the left eye by the same processing. When generating an image for the left eye, the overlap between the real object and the three-dimensional object is adjusted based on the position of the user's left eye.

  The display device 1 displays the composite image thus generated on the display units 32a and 32b. As a result, the user can see a scene as if the three-dimensional object BL1 is placed on the table T1 and the user grasps the three-dimensional object BL1 with his / her hand H1.

  In the scene of step S2 shown in FIG. 7, the user moves the hand H1 in the direction of the arrow A1. In this case, the image obtained by the imaging unit 40 changes to an image P1b in which the position of the hand H1 has moved to the right. Further, the display device 1 determines that the movement of the hand H1 is an operation of moving the hand H1 to the right while holding the three-dimensional object, and moves the position of the three-dimensional object in the virtual space to the right according to the operation. The movement of the three-dimensional object in the virtual space is reflected in the virtual space data 24d. As a result, the image for the right eye generated based on the virtual space data 24d and the object data 24b changes to an image P2b in which the position of the three-dimensional object BL1 has moved to the right. Details of operation detection by the display device 1 will be described later.

  The display device 1 combines the image P1b and the image P2b to generate an image P3b for the right eye. The image P3b is an image as if the user is holding the three-dimensional object BL1 with the hand H1 at a position on the right side of the table T1 as compared with the image P3a. The display device 1 similarly generates a composite image for the left eye. Then, the display device 1 displays the composite image thus generated on the display units 32a and 32b. As a result, the user can see a scene as if he / she picked the three-dimensional object BL1 with his / her hand H1 and moved it to the right.

  Such a composite image for display is updated at a frequency (for example, 30 times per second) equal to the frame rate of a general moving image. As a result, the change of the three-dimensional object according to the user's operation is reflected in the image displayed on the display device 1 in almost real time, and the user can feel the three-dimensional without any sense of incongruity as if it actually exists. You can manipulate objects. Furthermore, in the configuration according to the present embodiment, the user's hand that operates the three-dimensional object is not positioned between the user's eyes and the display units 32a and 32b. The operation can be performed without worrying about the display of the object being interrupted.

  Next, the object data 24b and the action data 24c shown in FIG. 6 will be described in more detail with reference to FIGS. FIG. 8 is a diagram illustrating an example of information stored in the object data 24b. 9 to 14 are diagrams illustrating examples of information stored in the action data 24c.

  As shown in FIG. 8, the object data 24b stores information including type, shape information, color, transparency, etc. for each three-dimensional object. The type indicates the physical property of the three-dimensional object. The type takes values such as “rigid body” and “elastic body”, for example. The shape information is information indicating the shape of the three-dimensional object. The shape information is, for example, a set of vertex coordinates of the surfaces constituting the three-dimensional object. The color is the color of the surface of the three-dimensional object. Transparency is the degree to which a three-dimensional object transmits light. The object data 24b can hold information regarding a plurality of three-dimensional objects.

  In the example shown in FIGS. 9 to 14, information regarding a change when a pressing operation is detected is stored in the action data 24 c for each type of three-dimensional object. As shown in FIG. 9, when the type of the three-dimensional object is “rigid”, the change when the pressing operation is detected according to the presence / absence of the fulcrum, the presence / absence of the obstacle in the pressed direction, and the pressing speed changes. Different. The obstacle here may be another three-dimensional object or an actual object. Further, whether the pressing speed is fast or slow is determined based on a threshold value.

  If the three-dimensional object has no fulcrum and there is no obstacle in the pushed direction, the three-dimensional object is displayed so as to move according to the amount pushed in the pushed direction. The three-dimensional object displayed in this way is, for example, a building block, a pen, or a book. As to how to move, whether to slide or rotate may be determined based on the shape of the three-dimensional object. Further, whether to move with the pushed object or move away from the object so as to be repelled by the pushed object may be determined based on the pushing speed, and calculation of frictional resistance between the three-dimensional object and the bottom surface. You may determine based on a value or a setting value.

  If the 3D object has no fulcrum and there is an obstacle fixed in the pushed direction, the 3D object moves according to the amount pushed in the pushed direction and moves when it touches the obstacle. Appears to stop. The three-dimensional object displayed in this way is, for example, a building block, a pen, or a book. When the pushed speed is high, the three-dimensional object may destroy the obstacle and continue to move. Also, when the 3D object comes in contact with an obstacle while being moved away from the object so as to be repelled by the pressed object, the 3D object may be moved in the opposite direction so as to bounce off.

  If the 3D object has no fulcrum and there are other rigid bodies that are not fixed in the pushed direction, and the pushed speed is slow, the 3D object moves according to the amount pushed in the pushed direction, After contacting another rigid body, the other rigid bodies are displayed to move together. If the 3D object has no fulcrum and there is another rigid body that is not fixed in the pressed direction, and the pressed speed is fast, the 3D object moves according to the amount pressed in the pressed direction. Is displayed. After the three-dimensional object comes into contact with another rigid body, the other rigid body is displayed so as to be repelled and moved. After coming into contact with another rigid body, the three-dimensional object may stop on the spot or continue to move at a reduced speed. The combination of the three-dimensional object and the other rigid body displayed in this way is, for example, a combination of a bowling ball and a pin, or a combination of marbles.

  If there is another rigid body that does not have a fulcrum in the 3D object and is not fixed in the pressed direction, but the other rigid body can pass through, the 3D object will depend on the amount pressed in the pressed direction. After moving and contacting another rigid body, it is displayed to continue moving as it passes through the other rigid body. In reality, a rigid body does not slip through a rigid body, but by enabling such a slip-through, a novel experience can be provided to the user. The combination of the three-dimensional object and the other rigid body displayed in this way is, for example, a combination of a bowling ball and a pin, or a combination of marbles. A threshold may be provided for the pushed speed, and when the pushed speed is equal to or less than the threshold, the three-dimensional object may not pass through another rigid body.

  When the three-dimensional object has a fulcrum, the three-dimensional object is displayed so as to rotate around the fulcrum according to the pressed direction and amount. The rotation referred to here may be a rotation that rotates around 360 degrees, or a rotation that reciprocates within a predetermined rotation range. The three-dimensional object displayed in this way is, for example, a pendulum, a boxing sandback, or a windmill.

  Also, as shown in FIG. 10, when the type of the three-dimensional object is “elastic body”, the change when the pressing operation is detected differs depending on the material, the presence / absence of the change amount limitation, and the pressing speed. The material here is an assumed material of the three-dimensional object and is defined in the object data 24b.

  If the material of the 3D object is rubber, there is no limit on the amount of change, and the pressing speed is slow, the 3D object is deformed according to the amount pressed in the pressed direction and released from the pressed state. When it is done, it is displayed so as to return to the original shape. In addition, when the material of the three-dimensional object is rubber, there is no limit on the amount of change and the pressing speed is fast, the three-dimensional object is deformed according to the amount pressed in the pressing direction, and then the flip It is displayed to move in the pressed direction while returning to the original shape. The three-dimensional object displayed in this way is, for example, a rubber ball or an eraser.

  If the material of the 3D object is rubber, and the amount of change is limited, the 3D object will be deformed to the deformable range according to the amount pressed in the pressed direction, and the pressing operation will be detected thereafter. Then, it is displayed to move in the pushed direction while returning to the original shape. The three-dimensional object displayed in this way is, for example, a rubber ball or an eraser.

  If the material of the 3D object is metal, the 3D object will be deformed to the deformable range according to the amount pressed in the pressed direction, and will return to its original shape when released from the pressed state. To repeat (vibrate). When pressed in a direction other than the deformable direction, the three-dimensional object moves in the same manner as a rigid body. The three-dimensional object displayed in this way is, for example, a leaf spring or a string spring.

  As shown in FIG. 11, when the type of the three-dimensional object is “plastic”, the three-dimensional object is displayed such that the pressed portion is recessed and the overall shape changes. The three-dimensional object displayed in this way is, for example, clay.

  Also, as shown in FIG. 12, when the type of the three-dimensional object is “liquid”, the change when the pressing operation is detected differs depending on the pressing speed. When the pushing speed is slow, the object to be pushed is displayed so as to be in contact with the three-dimensional object, ie, the liquid. When the pressing speed is medium, the object to be pressed is displayed on the liquid so that the ripple spreads on the liquid. When the pushed speed is high, the object to be pushed is touched by the liquid, and it is displayed that the liquid splashes. The three-dimensional object displayed in this way is, for example, water in a cup.

  Also, as shown in FIG. 13, when the type of the three-dimensional object is “gas”, the change when the pressing operation is detected differs depending on the pressing speed. When the pushed speed is slow, a three-dimensional object, i.e., a gas is interrupted by the pushed object and is displayed to drift around it. When the pressing speed is medium, it is displayed that the gas is scattered by the pressing object. When the pushing speed is high, the vortex is generated in the gas behind the pushing object in the moving direction. The three-dimensional object displayed in this way is, for example, smoke.

  As shown in FIG. 14, when the type of the three-dimensional object is “aggregate”, the change when the pressing operation is detected differs depending on the coupling state of the elements of the aggregate. When there is no connection of the elements of the aggregate, the three-dimensional object is displayed such that the pressed portion is recessed and the overall shape of the aggregate changes. The three-dimensional object displayed in this way is, for example, sand or sugar.

  When there is a combination of the elements of the aggregate, the three-dimensional object is displayed such that the pressed portion is recessed and the overall shape of the aggregate changes. Further, an element other than the pressed part is displayed so as to be moved by being pulled by the element at the pressed part. The three-dimensional object displayed in this way is, for example, a chain.

  When there is no combination of the elements of the aggregate, but an attractive force or a repulsive force is applied to the pushing object, the three-dimensional object is displayed so as to move even if it does not contact the pushing object. When an attractive force acts between the pressed object and the pressed object, the three-dimensional object is attracted to the pressed object when it is within a predetermined distance from the pressed object without contacting the pressed object. In addition, when a repulsive force acts between the pressed object and the pressed object, the three-dimensional object moves away from the pressed object when it is within a predetermined distance from the pressed object without contacting the pressed object. The combination of the displayed three-dimensional object and the pressed object is, for example, a combination of iron powder and a magnet.

  As described above, by changing the three-dimensional object based on the information stored in the object data 24b and the information stored in the action data 24c, the three-dimensional object can be variously changed according to the pressing operation. it can. The information stored in the object data 24b and the action data 24c is not limited to the above example, and may be changed as appropriate according to the application. For example, the method of changing the three-dimensional object may be set according to the type and size of the pressed object and the size of the contact area between the pressed object and the three-dimensional object.

  Next, detection of an operation of pushing a three-dimensional object and change of the three-dimensional object according to the detected operation will be described with reference to FIGS. 15 and 16. In the following description, a space viewed by a user wearing the display device 1 may be referred to as a display space. The display device 1 can display an actual object and a three-dimensional object three-dimensionally (three-dimensionally) in the display space by providing images corresponding to the eyes of the user's right eye and left eye. . The display device 1 associates the virtual space reproduced based on the virtual space data 24d with the actual space photographed by the photographing units 40 and 42 based on a predetermined rule, and displays a space in which these spaces overlap. Display as space.

  15 and 16 are diagrams for explaining detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. In step S11 illustrated in FIG. 15, the display device 1 displays the three-dimensional object OB1 in a three-dimensional manner in the display space 50. The three-dimensional object OB1 is an object that imitates a ball, for example. In step S11, a bottom surface B1 that supports the three-dimensional object OB1 is displayed.

  In step S12, the user places the finger F1 in a position in contact with the three-dimensional object OB1 and keeps it still. The display device 1 determines that the three-dimensional object OB1 is selected as the operation target when a real object is detected in the display space and the state where the object is in contact with the three-dimensional object OB1 continues for a predetermined time or longer. . Then, the display device 1 notifies the user that the three-dimensional object OB1 has been selected as an operation target, for example, by changing the display mode of the three-dimensional object OB1.

  Whether the object is in contact with the three-dimensional object OB1 is determined based on the position of the object in the real space, the shape and orientation of the three-dimensional object OB1, the position in the virtual space, and the like. The comparison between the position in the real space and the position in the virtual space may be performed by converting the position in one space into the position in the other space based on the above-mentioned predetermined rule. You may perform by converting a position into the position of the space for a comparison. When a finger is detected as an actual object, the position of the finger toe may be processed as the position of the object. Since a person often uses a finger toe when operating something, by processing the position of the finger toe as the position of an object, a more natural operation feeling can be provided to the user.

  The notification of selection as the operation target may be, for example, changing the overall color of the three-dimensional object OB1, or changing the color near the position in contact with the object on the surface of the three-dimensional object OB1. Realized by changing. The display device 1 may perform notification by sound or vibration instead of such visual notification or in addition to such visual notification.

  As described above, the display device 1 determines that the three-dimensional object OB1 is selected as the operation target when a state where a real object such as a finger is in contact with the three-dimensional object OB1 is continuously detected for a predetermined time or longer. . By adding the condition that the contact state is continuously detected for a predetermined time or longer, for example, in the process of moving a finger to operate another three-dimensional object, an unintended three-dimensional object is selected as an operation target. The possibility of being made can be lowered.

  After the three-dimensional object OB1 is selected as the operation target, it is assumed that the user has intruded the finger F1 inside the three-dimensional object OB1 so as to press the three-dimensional object OB1, as shown in step S13. The display device 1 changes the three-dimensional object according to the operation when an operation for intruding the object into the three-dimensional object selected as the operation target is detected. How to change the three-dimensional object is determined by the type of the three-dimensional object defined in the object data 24b and the change rule defined in association with the type in the action data 24c.

  For example, in the object data 24b, the three-dimensional object OB1 is defined as an elastic body, and in the action data 24c, it is defined that the elastic body is deformed in the pressed direction according to the amount of pressing when pressed. Shall be. In this case, as shown in step S14, the display device 1 changes the three-dimensional object OB1 so that the portion into which the finger F1 has entered is pressed and recessed.

  Further, the object data 24b defines the three-dimensional object OB1 as a rigid body, and the action data 24c defines that the rigid body moves in the pressed direction according to the amount of pressing when pressed. And In this case, as shown in step S15 in FIG. 16, the display device 1 moves the three-dimensional object OB1 in the traveling direction of the finger F1 as if pressed by the finger F1. In step S15 of FIG. 16, since the three-dimensional object OB1 is supported by the bottom surface B1, it moves according to the horizontal component of the bottom surface B1 of the force applied by the rigid body.

  As described above, when an operation of pushing the three-dimensional object is detected, the three-dimensional object is changed variously according to the operation by changing the three-dimensional object OB1 based on the object data 24b and the action data 24c. Can do. The operation of pressing is an operation that is used in various scenes in the real world. By detecting the operation of pressing the three-dimensional object OB1 and executing the corresponding process, the operation is intuitive and highly convenient. Can be realized.

  An object used for operating a three-dimensional object is not limited to a finger, and may be a hand, a foot, a stick, an instrument, or the like. The mode of changing the three-dimensional object in accordance with the pressing operation may be in accordance with actual physical laws or may not be in reality.

  The display device 1 may limit a space for detecting an operation on the three-dimensional object to the operable range 51. The operable range 51 is, for example, a range that can be reached by a user wearing the display device 1. In this way, by limiting the space for detecting the operation on the three-dimensional object, it is possible to reduce the load of the arithmetic processing executed by the display device 1 to detect the operation.

  Next, a first example of a processing procedure executed by the display device 1 regarding an operation of pressing a three-dimensional object will be described with reference to FIGS. 17 and 18. FIG. 17 is a flowchart illustrating a processing procedure of contact detection processing of a three-dimensional object. The processing procedure illustrated in FIG. 17 is realized by the control unit 22 executing the control program 24a.

  As shown in FIG. 17, first, the control unit 22 synthesizes and displays a virtual space image including a three-dimensional object and a real space image in step SA01.

  Subsequently, in step SA02, the control unit 22 determines whether a predetermined object is detected by the detection unit 44, that is, the imaging units 40 and 42. The predetermined object is, for example, a user's finger. When the predetermined object is not detected (step SA02, No), the control unit 22 determines whether the operation end is detected as step SA08.

  The operation end is detected, for example, when a predetermined operation is performed on the operation unit 13. When the operation end is detected (step SA08, Yes), the control unit 22 ends the contact detection process. When the operation end is not detected (No at Step SA08), the control unit 22 re-executes Step SA02 and the subsequent steps.

  When a predetermined object is detected (step SA02, Yes), the control unit 22 determines the type of the predetermined object as step SA03. The type of the predetermined object is determined based on, for example, the size, shape, color, and the like of the object in the images photographed by the photographing units 40 and 42. Subsequently, in step SA04, the control unit 22 searches for a three-dimensional object that is in contact with a predetermined object. If there is no three-dimensional object in contact with the predetermined object (step SA05, No), the control unit 22 proceeds to step SA08.

  When a three-dimensional object in contact with the predetermined object is found (step SA05, Yes), the control unit 22 determines, as step SA06, the three-dimensional object in contact with the predetermined object based on the object data 24b. Determine the type. And the control part 22 performs the operation detection process mentioned later as step SA07. Thereafter, the control unit 22 proceeds to Step SA08.

  FIG. 18 is a flowchart illustrating a procedure of the operation detection process. The processing procedure shown in FIG. 18 is realized by the control unit 22 executing the control program 24a.

  As shown in FIG. 18, the control unit 22 first acquires the contact time between the predetermined object and the three-dimensional object as step SB01. Then, the control unit 22 determines whether the predetermined object has moved into the three-dimensional object as Step SB02. When the predetermined object has not moved inside the three-dimensional object (step SB02, No), the control unit 22 re-executes step SB01 and the subsequent steps.

  When the predetermined object has moved to the inside of the three-dimensional object (step SB02, Yes), the control unit 22 determines whether the contact time is a predetermined time or more as step SB03. When the contact time is shorter than the predetermined time (step SB03, No), it is determined that the three-dimensional object is not an operation target, and thus the control unit 22 ends the operation detection process.

  When the contact time is equal to or longer than the predetermined time (step SB03, Yes), the control unit 22 calculates the speed of the predetermined object as step SB04. In step SB05, the control unit 22 changes the three-dimensional object based on the type, position, and speed of the predetermined object, the type of the three-dimensional object, and the like. A specific method of change is determined according to the action data 24c.

  Subsequently, in Step SB06, the control unit 22 determines whether the predetermined object has moved outside the three-dimensional object. When the predetermined object has not moved to the outside of the three-dimensional object, that is, when the pressing operation is continued (No in Step SB06), the control unit 22 re-executes Step SB04 and the subsequent steps.

  When the predetermined object moves to the outside of the three-dimensional object, that is, when the three-dimensional object is released (step SB06, Yes), the control unit 22 determines whether the change of the three-dimensional object continues as step SB07. judge. For example, when the action data 24c defines that the vibration continues for a predetermined time after release, it is determined that the change of the three-dimensional object continues.

  When the change of the three-dimensional object continues (step SB07, Yes), the control unit 22 changes the three-dimensional object as step SB08, and then re-executes step SB07 and the subsequent steps. When the change of the three-dimensional object is not continued (step SB07, No), the control unit 22 ends the operation detection process.

  As described above, in the first example, the three-dimensional object is variously changed according to the pressing operation, thereby providing a convenient operation method for the user.

  A second example of a processing procedure related to an operation of pressing a three-dimensional object will be described. The contact detection process in the second example is the same as the contact detection process in the first example. Therefore, in the second example, the description overlapping the first example is omitted, and the operation detection process will be mainly described.

  First, detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation will be described with reference to FIG. FIG. 19 is a diagram for describing detection of an operation of pressing a three-dimensional object and changes in the three-dimensional object according to the detected operation. In step S21 shown in FIG. 19, the user contacts the finger F1 with the three-dimensional object OB1, and in step S22, the user causes the finger F1 to enter the inside of the three-dimensional object OB1.

  When a real object is detected in the display space and the state in which the object moves to the inside of the three-dimensional object OB1 after the object contacts the three-dimensional object OB1 continues for a predetermined time or longer, the display device 1 It is determined that OB1 has been selected as the operation target. Then, the display device 1 notifies the user that the three-dimensional object OB1 has been selected as an operation target, for example, by changing the display mode of the three-dimensional object OB1. Further, as shown in step S23, the display device 1 responds to the operation with the finger F1 after the contact detection as if the three-dimensional object OB1 was already selected as an operation target to be pressed in the step S21. Change.

  As described above, after detecting the contact between the object and the three-dimensional object, the user can quickly start the operation of pressing the three-dimensional object by making it possible to detect the pressing operation even if the object does not stay on the spot. can do. In addition, by adding the condition that the state in which the object moves to the inside of the three-dimensional object OB1 after contact continues for a predetermined time or longer, for example, in the process of moving a finger to operate another three-dimensional object, The possibility that an unintended three-dimensional object is selected as an operation target can be reduced.

  Next, the procedure of the operation detection process in the second example will be described with reference to FIG. FIG. 20 is a flowchart illustrating the processing procedure of the operation detection process. The processing procedure shown in FIG. 20 is realized by the control unit 22 executing the control program 24a. The processing procedure of the contact detection processing is the same as the procedure shown in FIG.

  As shown in FIG. 20, the control unit 22 first determines whether a predetermined object has moved inside the three-dimensional object as step SC01. If the predetermined object has not moved to the inside of the three-dimensional object (No at Step SC01), since the three-dimensional object is determined not to be an operation target, the control unit 22 ends the operation detection process.

  When the predetermined object has moved to the inside of the three-dimensional object (step SC01, Yes), the control unit 22 determines whether the elapsed time from the contact detection is a predetermined time or more as step SC02. When the elapsed time is shorter than the predetermined time (No at Step SC02), the control unit 22 re-executes Step SC01 and the subsequent steps.

  When the contact time is equal to or longer than the predetermined time (step SC02, Yes), the control unit 22 calculates the speed of the predetermined object as step SC03. In step SC04, the control unit 22 changes the three-dimensional object based on the type, position, and speed of the predetermined object and the type of the three-dimensional object. A specific method of change is determined according to the action data 24c.

  Subsequently, in step SC05, the control unit 22 determines whether the predetermined object has moved outside the three-dimensional object. When the predetermined object has not moved outside the three-dimensional object, that is, when the pressing operation is continued (No at Step SC05), the control unit 22 re-executes Step SC03 and the subsequent steps.

  When the predetermined object moves outside the three-dimensional object, that is, when the three-dimensional object is released (step SC05, Yes), the control unit 22 determines whether the change of the three-dimensional object continues as step SC06. judge. For example, when the action data 24c defines that the vibration continues for a predetermined time after release, it is determined that the change of the three-dimensional object continues.

  When the change of the three-dimensional object continues (step SC06, Yes), the control unit 22 changes the three-dimensional object as step SC07, and then re-executes step SC06 and subsequent steps. When the change of the three-dimensional object does not continue (No at Step SC06), the control unit 22 ends the operation detection process.

  As described above, in the second example, since the pressing operation is recognized even when the object such as the finger is in contact with the three-dimensional object does not continue for a predetermined time or longer, the user can recognize the three-dimensional object. The operation of pushing the object can be started quickly.

  A third example of a processing procedure related to an operation of pressing a three-dimensional object will be described. The contact detection process in the third example is the same as the contact detection process in the first example. Therefore, in the third example, the description overlapping with the first example is omitted, and the operation detection process is mainly described.

  First, detection of an operation of pushing a three-dimensional object and change of the three-dimensional object according to the detected operation will be described with reference to FIGS. 21 and 22. 21 and 22 are diagrams for explaining detection of an operation of pushing a three-dimensional object and a change of the three-dimensional object according to the detected operation. In step S31 shown in FIG. 21, the three-dimensional object OB1 is displayed three-dimensionally in the display space. In addition, the user is bringing the finger F1 into contact with the three-dimensional object OB1.

  Here, it is assumed that the user has caused the finger F1 to enter the inside of the three-dimensional object OB1. When the display device 1 detects that the object in contact with the three-dimensional object OB1 has moved to the inside of the three-dimensional object OB1, as shown in step S32, the display device 1 responds to the operation with the finger F1 from that point in time. Change. In the example shown in FIG. 21, in step S32, the three-dimensional object OB1 starts moving in accordance with the movement of the finger F1.

  Then, as shown in step S33, the display device 1 determines the three-dimensional object OB1 as an operation target when the finger F1 continues to move inward of the three-dimensional object OB1 for a predetermined time or more. Then, the display device 1 notifies the user that the three-dimensional object OB1 has been determined as an operation target, for example, by changing the display mode of the three-dimensional object OB1. Thereafter, while the movement of the finger F1 to the inside of the three-dimensional object OB1 is detected, the display device 1 continues to change the three-dimensional object OB1.

  As shown in step S34 of FIG. 22, when the movement of the finger F1 to the inside of the three-dimensional object OB1 is not detected before the predetermined time has elapsed, the display device 1 is opposite to the change made so far. A change is applied to the three-dimensional object OB1. As a result, the three-dimensional object OB1 is displayed in the same state at the same position as in the step S31. The speed at which the reverse change is applied to the three-dimensional object OB1 may be faster than the speed at which the change has been applied to the three-dimensional object OB1 so far. That is, the three-dimensional object OB1 may be reversely changed as if it is reversely played back at high speed.

  In this way, by starting to apply changes to the 3D object from the stage where the intrusion of the object inside the 3D object is detected, the user can confirm that the 3D object is being selected before the selection is confirmed. Can be recognized from. As a result, the user can know at an early stage whether or not the intended three-dimensional object has been selected. When an unintended three-dimensional object is selected, the user can return the original three-dimensional object to an original state by canceling the operation before a predetermined time elapses.

  Until the movement of the finger F1 to the inside of the three-dimensional object OB1 continues for a predetermined time or longer, the mode is different from the state in which the selected three-dimensional object is changed as the operation target at both the normal time (for example, half-time). (Transparent) may be displayed. By changing the display mode in this way, the user can easily determine the state of the three-dimensional object.

  Next, the procedure of the operation detection process in the third example will be described with reference to FIG. FIG. 23 is a flowchart illustrating the processing procedure of the operation detection process. The processing procedure shown in FIG. 23 is realized by the control unit 22 executing the control program 24a. The processing procedure of the contact detection processing is the same as the procedure shown in FIG.

  As shown in FIG. 23, the control unit 22 first determines whether a predetermined object has moved inside the three-dimensional object as step SD01. If the predetermined object has not moved to the inside of the three-dimensional object (step SD01, No), since the three-dimensional object is determined not to be an operation target, the control unit 22 ends the operation detection process.

  When the predetermined object is moving inside the three-dimensional object (step SD01, Yes), the control unit 22 calculates the speed of the predetermined object as step SD02. In step SD03, the control unit 22 changes the three-dimensional object based on the type, position, and speed of the predetermined object, the type of the three-dimensional object, and the like. A specific method of change is determined according to the action data 24c.

  Subsequently, as step SD04, the control unit 22 determines whether or not the elapsed time from the contact detection is a predetermined time or more. When the elapsed time is shorter than the predetermined time, that is, when the three-dimensional object is not fixed as the target of the pressing operation (No at Step SD04), the control unit 22 determines the three-dimensional object of the predetermined object as Step SD05. It is determined whether the movement in the internal direction continues.

  When the movement of the three-dimensional object in the internal direction continues (step SD05, Yes), the control unit 22 re-executes step SD02 and subsequent steps. When the movement of the three-dimensional object in the internal direction is not continued (No at Step SD05), the control unit 22 reversely changes the three-dimensional object and returns it to the original state as Step SD06. Then, the control unit 22 ends the operation detection process.

  When the elapsed time from the contact detection is equal to or longer than the predetermined time (step SD04, Yes), the control unit 22 determines whether the predetermined object has moved outside the three-dimensional object as step SD07. When the predetermined object has not moved outside the three-dimensional object, that is, when the pressing operation is continued (No at Step SD07), the control unit 22 re-executes Step SD02 and the subsequent steps.

  When the predetermined object moves outside the three-dimensional object, that is, when the three-dimensional object is released (step SD07, Yes), the control unit 22 determines whether the change of the three-dimensional object continues as step SD08. judge. For example, when the action data 24c defines that the vibration continues for a predetermined time after release, it is determined that the change of the three-dimensional object continues.

  When the change of the three-dimensional object continues (step SD08, Yes), the control unit 22 changes the three-dimensional object as step SD09, and then re-executes step SD08 and subsequent steps. When the change of the three-dimensional object does not continue (No at step SD08), the control unit 22 ends the operation detection process.

  As described above, in the third example, since the three-dimensional object is changed in accordance with the operation from the time when the pressing operation is detected, the three-dimensional object that is the target of the pressing operation by the user. Easy to recognize objects.

  Although the operation of pushing the 3D object has been described as the operation related to the 3D object, the operation detected by the display device 1 concerning the 3D object is not limited to the pushing operation. The display device 1 can also detect an operation performed by a user picking a three-dimensional object. Hereinafter, an operation performed by picking a three-dimensional object will be described.

  With reference to FIG. 24, detection of an operation performed by picking a three-dimensional object will be described. FIG. 24 is a diagram for describing detection of an operation performed by picking up a three-dimensional object. In step S41 shown in FIG. 24, the three-dimensional object OB1 is displayed three-dimensionally in the display space 50.

  Here, it is assumed that the user wants to perform some operation by picking up the three-dimensional object OB1. In order to pick up the three-dimensional object OB1 and perform some operation, it is first necessary to select the three-dimensional object OB1 as an operation target. In order to select the three-dimensional object OB1, the user moves the finger F1 and the finger F2 so that the three-dimensional object OB1 is positioned between the finger F1 and the finger F2, as shown in step S42, and changes its state. Maintain for a predetermined time or longer.

  The display device 1 selects the three-dimensional object OB1 when two real objects are detected in the display space and the state where the three-dimensional object OB1 is located between the two objects continues for a predetermined time or longer. It is determined that the three-dimensional object OB1 is selected. Then, the display device 1 notifies the user that the three-dimensional object OB1 has been selected, for example, by changing the display mode of the three-dimensional object OB1.

  Whether the three-dimensional object OB1 is located between the two objects is determined based on the positions of the two objects in the real space, the shape and orientation of the three-dimensional object OB1, the position in the virtual space, and the like. The comparison between the position in the real space and the position in the virtual space may be performed by converting the position in one space into the position in the other space based on the above-mentioned predetermined rule. You may perform by converting a position into the position of the space for a comparison. When a finger is detected as an actual object, the position of the finger toe may be processed as the position of the object.

  As described above, the display device 1 determines that the three-dimensional object OB1 is selected when the state in which the three-dimensional object OB1 is located between actual objects such as fingers is continuously detected for a predetermined time or more. The operation of placing a finger so that the three-dimensional object OB1 is sandwiched between the fingers is similar to the operation of a person picking something to select something in the real space. Therefore, such an operation is intuitive and easy to understand as an operation for selecting a three-dimensional object. Further, by adding a condition that the state is continuously detected for a predetermined time or longer, for example, an unintended three-dimensional object is selected in the process of moving a finger to select another three-dimensional object. The possibility can be reduced.

  After determining that the three-dimensional object OB1 is in the selected state, the display device 1 adds changes such as movement, deformation, disappearance, and replacement to the three-dimensional object OB1 according to the movement of the finger F1 and the finger F2.

  Next, a first example of a processing procedure executed by the display device 1 regarding an operation performed by picking a three-dimensional object will be described with reference to FIGS. FIG. 25 is a flowchart illustrating a processing procedure of a three-dimensional object selection detection process. The processing procedure shown in FIG. 25 is realized by the control unit 22 executing the control program 24a.

  As shown in FIG. 25, the control unit 22 first synthesizes and displays a virtual space image including a three-dimensional object and a real space image in step SE01.

  Subsequently, in step SE02, the control unit 22 determines whether the detection unit 44, that is, the imaging units 40 and 42, has detected the first object and the second object. The first object and the second object are real objects, for example, user's fingers. When the first object and the second object are not detected (step SE02, No), the control unit 22 determines whether the operation end is detected as step SE10.

  The operation end is detected, for example, when a predetermined operation is performed on the operation unit 13. When the operation end is detected (step SE10, Yes), the control unit 22 ends the selection detection process. When the operation end is not detected (No at Step SE10), the control unit 22 re-executes Step SE02 and the subsequent steps.

  When the first object and the second object are detected (step SE02, Yes), the control unit 22 determines, as step SE03, the first object and the second object from the displayed three-dimensional objects. Search for the three-dimensional object displayed between. When there is no corresponding three-dimensional object (step SE04, No), the control unit 22 proceeds to step SE10.

  When a three-dimensional object displayed between the first object and the second object is found (step SE04, Yes), the control unit 22 determines that the first object, the second object, and the second object are step SE05. The time during which the three-dimensional object is located is acquired. When the acquired time is less than the predetermined time (step SE06, No), the control unit 22 proceeds to step SE10.

  When the acquired time is equal to or longer than the predetermined time (step SE06, Yes), the control unit 22 calculates the distance between the first object and the second object as step SE07. Moreover, the control part 22 makes the three-dimensional object currently displayed between the 1st object and the 2nd object a selection state as step SE08. And control part 22 performs operation detection processing mentioned below as Step SE09, and changes the three-dimensional object in a selection state according to detected operation in it. After the operation detection process ends, the control unit 22 proceeds to step SE10.

  FIG. 26 is a flowchart showing the procedure of the operation detection process. The processing procedure shown in FIG. 26 is realized by the control unit 22 executing the control program 24a.

  As shown in FIG. 26, the control unit 22 first calculates the distance between the first object and the second object as step SF01. And control part 22 judges whether the distance of the 1st object after the start time of operation detection processing and the 2nd object is almost constant as Step SF02. The distance is substantially constant, for example, when the amount of change in the distance between the first object and the second object at the current time is compared with the distance at the start of the operation detection process in a predetermined range (first object and This means that the second object is within ± 10% of the maximum change in distance when moving at a normal speed. Also, when the distance between the first object and the second object continues to shrink after the start of the operation detection process (when the first object and the second object are moving in the direction of crushing the three-dimensional object) In addition, it may be determined that the distance is substantially constant. Further, when the distance between the two changes only within a range such as camera shake, the distance may be determined to be substantially constant.

  When the distance between the first object and the second object is substantially constant (step SF02, Yes), the control unit 22 calculates the moving speeds of the first object and the second object as step SF03. Subsequently, in step SF04, the control unit 22 determines whether the calculated moving speed is equal to or less than a threshold value. The threshold value used here is, for example, the moving speed of the fingertip when a person throws a thing. The moving speed compared with the threshold value may be the average of the moving speed of the first object and the moving speed of the second object, whichever is faster, or whichever is slower. There may be.

  When the moving speed is less than or equal to the threshold (step SF04, Yes), the control unit 22 changes the three-dimensional object according to the detected movements of the first object and the second object as step SF05. For example, when the rightward movement of the first object and the second object is detected, the control unit 22 moves the three-dimensional object to the right according to the movement of the first object and the second object. Let When the counterclockwise rotation of the first object and the second object is detected, the control unit 22 rotates the three-dimensional object counterclockwise in accordance with the rotation of the first object and the second object. When movement and rotation are detected simultaneously, movement and rotation are performed simultaneously. When there is an obstacle to the movement and rotation of the three-dimensional object, the movement and rotation of the three-dimensional object may be stopped when the three-dimensional object contacts the obstacle. The obstacle may be a real object or another three-dimensional object. Thereafter, the control unit 22 re-executes Step SF01 and subsequent steps.

  When the moving speed is faster than the threshold (No at Step SF04), the control unit 22 deletes the three-dimensional object as Step SF06. When deleting the three-dimensional object, an animation may be displayed so that the three-dimensional object flies in the moving direction of the first object and the second object. Then, the control unit 22 ends the operation detection process. As described above, when the first object and the second object move at high speed so as to throw the 3D object, the 3D object is erased by an intuitive operation by erasing the 3D object. be able to. Instead of the operation of moving the first object and the second object at a high speed, for example, erasure of the three-dimensional object may be assigned to an operation of crushing the three-dimensional object. Further, instead of deleting the three-dimensional object, the three-dimensional object may be returned to the original placement location. The display device 1 may not perform the processes of steps SF03, SF04, and SF06. That is, the display device 1 may execute step SF05 when the distance between the first object and the second object is determined to be substantially constant in step SF02, regardless of the moving speed of the two objects.

  When the distance between the first object and the second object is not substantially constant (step SF02, No), the control unit 22 determines whether the distance is a three-dimensional object, that is, the start point of the operation detection process, as step SF07. It is judged whether it is expanding. When the distance is increased (step SF07, Yes), the control unit 22 cancels the selected state of the three-dimensional object as step SF08. The operation of increasing the distance between the first object and the second object is similar to the operation of releasing the actual object being picked. Therefore, this operation is intuitive and easy to understand as an operation for canceling the selection of the three-dimensional object.

  Subsequently, in step SF09, the control unit 22 moves the selected three-dimensional object according to gravity or the like. Then, the control unit 22 ends the operation detection process. The movement here is displayed, for example, so that the three-dimensional object falls according to gravity and stops on the floor or table. Before stopping the movement of the three-dimensional object, the three-dimensional object may be bounced according to the elasticity of the three-dimensional object and the hardness of the floor or table. The magnitude of impact when the three-dimensional object collides with the floor or table may be calculated, and if the impact is greater than a predetermined value, the three-dimensional object may be displayed as damaged. Further, the three-dimensional object may be moved more slowly than when actual gravity works.

  When the distance between the first object and the second object is shorter than when the three-dimensional object is selected (step SF07, No), the control unit 22 deforms the three-dimensional object according to the distance as step SF10. . Then, the control unit 22 re-executes Step SF01 and subsequent steps. For example, the degree of deformation of the three-dimensional object may be changed according to the elasticity set as an attribute of the three-dimensional object. The control unit 22 increases the degree of deformation of an object having a low hardness as an attribute, such as a three-dimensional object simulating a rubber ball, as the distance between the first object and the second object decreases. It's okay. In addition, the control unit 22 reduces the degree of deformation of an object having a high hardness as an attribute, such as a three-dimensional object imitating a building block, even if the distance between the first object and the second object is reduced. You can keep it.

  When the distance between the first object and the second object is shorter than when the three-dimensional object is selected, the display device 1 may reduce the three-dimensional object instead of deforming it. When the distance between the first object and the second object becomes a predetermined value or less, the display device 1 may display the three-dimensional object as if it was damaged.

  As described above, in the first example, the three-dimensional object is selected when the state where the three-dimensional object is positioned between the objects such as the fingers continues for a predetermined time or longer. Selection can be realized by an intuitive and easy-to-understand operation.

  As illustrated in FIG. 27, the display device 1 selects a three-dimensional object that a state in which at least one of the first object and the second object is in contact with the three-dimensional object continues for a predetermined time or longer. It is good also as conditions. By making contact with a three-dimensional object as a selection condition, it becomes easier for a user to select a desired three-dimensional object when a plurality of three-dimensional objects are displayed close to each other.

  A second example of a processing procedure related to an operation performed by picking up a three-dimensional object will be described. The operation detection process in the second example is the same as the operation detection process in the first example. Therefore, in the second example, description overlapping with the first example is omitted, and mainly the selection detection process will be described.

  First, detection of an operation performed by picking a three-dimensional object will be described with reference to FIG. FIG. 28 is a diagram for explaining the detection of an operation performed by picking up a three-dimensional object. In step S51 shown in FIG. 28, the three-dimensional object OB1 is three-dimensionally displayed in the display space. In order to select the three-dimensional object OB1, the user moves the finger F1 and the finger F2 so that the three-dimensional object OB1 is positioned between the finger F1 and the finger F2.

  When two real objects are detected in the display space and the three-dimensional object OB1 is located between the two objects, the display device 1 monitors a change in the distance between the two objects. If the distance is substantially constant for a predetermined time or more, it is determined that the three-dimensional object OB1 has been selected, and the three-dimensional object OB1 is selected. Then, the display device 1 notifies the user that the three-dimensional object OB1 has been selected, for example, by changing the display mode of the three-dimensional object OB1.

  While the display device 1 monitors a change in the distance between the two objects, the two objects do not need to stay at a position between which the three-dimensional object OB1 is sandwiched. That is, the user moves the finger F1 and the finger F2 so that the three-dimensional object OB1 is positioned between the finger F1 and the finger F2 as shown in step S51, and then maintains the state of the finger F1. The finger F2 may be moved to another position.

  Assume that the user has moved from the state of step S51 while keeping the distance D1 between the finger F1 and the finger F2 substantially constant as shown in step S52. In this case, as shown in step S53, the display device 1 puts the three-dimensional object OB1 into a selected state when the distance D1 between the finger F1 and the finger F2 is kept almost constant for a predetermined time or longer. Then, the display device 1 moves the three-dimensional object OB1 between the finger F1 and the finger F2 as if they were already selected in the step S51. The display device 1 may store the movements of the finger F1 and the finger F2 from step S51 to step S53, and rotate the three-dimensional object OB1 according to the stored movement. Thereafter, the display device 1 adds changes such as movement, deformation, disappearance, and replacement to the three-dimensional object OB1 according to the movement of the finger F1 and the finger F2.

  In this way, once the two objects have moved to a position sandwiching the 3D object, the user can select the 3D object by making the 3D object selectable even if the object does not stay on the spot. After that, the operation can be started quickly.

  Next, the procedure of the selection detection process in the second example will be described with reference to FIG. FIG. 29 is a flowchart illustrating a processing procedure of a three-dimensional object selection detection process. The processing procedure shown in FIG. 29 is realized by the control unit 22 executing the control program 24a.

  As shown in FIG. 29, first, the control unit 22 synthesizes and displays a virtual space image including a three-dimensional object and a real space image in step SG01. Subsequently, in step SG02, the control unit 22 determines whether the first object and the second object are detected by the detection unit 44, that is, the imaging units 40 and 42. When the first object and the second object are not detected (step SG02, No), the control unit 22 cancels the temporarily selected state of the three-dimensional object if there is a temporarily selected three-dimensional object as step SG14. To do. The provisional selection state is a state in which a state in which a three-dimensional object is displayed between two objects is detected and whether the distance between the two objects is maintained substantially constant is monitored.

  And control part 22 judges whether the end of operation was detected as Step SG15. When the operation end is detected (step SG15, Yes), the control unit 22 ends the selection detection process. When the operation end is not detected (No at Step SG15), the control unit 22 re-executes Step SG02 and the subsequent steps.

  When the first object and the second object are detected (step SG02, Yes), the control unit 22 determines whether there is a temporarily selected three-dimensional object as step SG03. When there is no temporarily selected three-dimensional object (step SG03, No), the control unit 22 displays the first three-dimensional object between the first and second objects as step SG04. Find the 3D object that is being used. When there is no corresponding three-dimensional object (step SG05, No), the control unit 22 proceeds to step SG15.

  When a three-dimensional object displayed between the first object and the second object is found (step SG05, Yes), the control unit 22 determines whether the first object, the second object, and the second object are step SG06. The three-dimensional object displayed during is temporarily set to the selected state. Moreover, the control part 22 calculates the distance of a 1st object and a 2nd object as step SG07. And the control part 22 progresses to step SG15.

  When the first object and the second object are detected and there is a provisionally selected three-dimensional object (step SG03, Yes), the control unit 22 sets the first object and the second object as step SG08. The distance is calculated. And the control part 22 determines whether distance is substantially constant as step SG09. When the distance is not substantially constant (No in Step SG09), the control unit 22 cancels the temporary selection state of the three-dimensional object in the temporary selection state as Step SG14. And the control part 22 progresses to step SG15.

  When the distance between the first object and the second object is substantially constant (step SG09, Yes), the control unit 22 determines whether the period during which the distance is kept substantially constant is a predetermined time or more as step SG10. judge. When the period during which the distance is kept substantially constant is less than the predetermined time (step SG10, No), the control unit 22 proceeds to step SG15.

  When the period during which the distance is maintained substantially constant is equal to or longer than the predetermined time (step SG10, Yes), the control unit 22 is displayed between the first object and the second object as step SG11. Put the dimension object in the selected state. Moreover, the control part 22 moves a three-dimensional object between a 1st object and a 2nd object as step SG12. And control part 22 performs operation detection processing shown in Drawing 26 as Step SG13, and changes the three-dimensional object in a selection state according to detected operation in it. After the operation detection process is completed, the control unit 22 proceeds to step SG15.

  As described above, in the second example, after a three-dimensional object is positioned between objects such as fingers, the three-dimensional object is selected when the distance of the object is kept substantially constant for a predetermined time or more. Therefore, the user can quickly start the operation after selecting the three-dimensional object.

  As in Step S61 to Step S63 illustrated in FIG. 30, the display device 1 determines the distance between the first object and the second object after at least one of the first object and the second object contacts the three-dimensional object. May be kept substantially constant for a predetermined time or more as a condition for selecting a three-dimensional object. By making contact with a three-dimensional object as a selection condition, it becomes easier for a user to select a desired three-dimensional object when a plurality of three-dimensional objects are displayed close to each other.

  A third example of a processing procedure related to an operation performed by picking up a three-dimensional object will be described. Regarding the third example, the description overlapping with the first example is omitted, and mainly the selection detection process and the operation detection process will be described.

  First, detection of an operation performed by picking up a three-dimensional object will be described with reference to FIGS. 31 and 32. In step S71 shown in FIG. 31, the three-dimensional object OB1 is displayed three-dimensionally in the display space. In order to select the three-dimensional object OB1, the user moves the finger F1 and the finger F2 so that the three-dimensional object OB1 is positioned between the finger F1 and the finger F2.

  When two real objects are detected in the display space and the three-dimensional object OB1 is located between the two objects, the display device 1 monitors a change in the distance between the two objects. If the distance is substantially constant for a predetermined time or more, it is determined that the three-dimensional object OB1 has been selected, and the three-dimensional object OB1 is selected. Then, the display device 1 notifies the user that the three-dimensional object OB1 has been selected, for example, by changing the display mode of the three-dimensional object OB1.

  While the display device 1 monitors a change in the distance between the two objects, the two objects do not need to stay at a position between which the three-dimensional object OB1 is sandwiched. That is, the user moves the finger F1 and the finger F2 so that the three-dimensional object OB1 is positioned between the finger F1 and the finger F2 as shown in step S71, and then maintains the state of the finger F1. The finger F2 may be moved to another position.

  Assume that the user has moved from the state of step S71 while keeping the distance D1 between the finger F1 and the finger F2 substantially constant as shown in step S72. In this case, the display device 1 responds to the movement of the finger F1 and the finger F2 from the stage where it is detected that the three-dimensional object OB1 is displayed between the fingers F1 and F2, that is, from the stage of step S71. Changes such as movement, deformation, disappearance, and replacement are added to the three-dimensional object OB1. Then, as shown in step S73, the display device 1 places the three-dimensional object OB1 in a selected state when the distance D1 between the finger F1 and the finger F2 is kept substantially constant for a predetermined time or longer.

  As shown in step S74 of FIG. 32, when the distance D1 between the finger F1 and the finger F2 is expanded before the predetermined time has passed, that is, when the selection is not performed, the display device 1 has been added so far. A change opposite to the change is added to the three-dimensional object OB1. As a result, the three-dimensional object OB1 is displayed in the same state at the same position as in the step S71. The speed at which the reverse change is applied to the three-dimensional object OB1 may be faster than the speed at which the change has been applied to the three-dimensional object OB1 so far. That is, the three-dimensional object OB1 may be reversely changed as if it is reversely played back at high speed.

  Thus, by starting to apply a change to the 3D object from the stage where it is detected that the 3D object is displayed between the two objects, the user can confirm that the 3D object is being selected. It can be recognized before the selection is confirmed. As a result, the user can know at an early stage whether or not the intended three-dimensional object has been selected. The display device 1 has a mode in which the changed three-dimensional object is different from the normal state and the selected state (for example, translucent) until the distance between the two objects is kept substantially constant for a predetermined time or longer. ) May make it easier for the user to determine the state of the three-dimensional object.

  Next, a processing procedure executed by the display device 1 regarding an operation performed by picking up a three-dimensional object will be described with reference to FIGS. 33 and 34. FIG. 33 is a flowchart illustrating a processing procedure of a three-dimensional object selection detection process. The processing procedure shown in FIG. 33 is realized by the control unit 22 executing the control program 24a.

  As shown in FIG. 33, the control unit 22 first combines and displays a virtual space image including a three-dimensional object and a real space image in step SH01. Subsequently, in step SH02, the control unit 22 determines whether the first object and the second object are detected by the detection unit 44, that is, the imaging units 40 and 42. When the first object and the second object are not detected (step SH02, No), the control unit 22 cancels the temporarily selected state of the three-dimensional object if there is a temporarily selected three-dimensional object as step SH10. To do.

  And the control part 22 determines whether operation completion was detected as step SH11. When the operation end is detected (step SH11, Yes), the control unit 22 ends the selection detection process. When the operation end is not detected (No at Step SH11), the control unit 22 re-executes Step SH02 and the subsequent steps.

  When the first object and the second object are detected (step SH02, Yes), the control unit 22 determines whether or not there is a temporarily selected three-dimensional object as step SH03. When there is no provisionally selected three-dimensional object (step SH03, No), the control unit 22 displays as a step SH04 between the first object and the second object among the displayed three-dimensional objects. Find the 3D object that is being used. When there is no corresponding three-dimensional object (step SH05, No), the control unit 22 proceeds to step SH11.

  When a three-dimensional object displayed between the first object and the second object is found (step SH05, Yes), the control unit 22 determines that the first object, the second object, and the second object are step SH06. The three-dimensional object displayed during is temporarily set to the selected state. Moreover, the control part 22 calculates the distance of a 1st object and a 2nd object as step SH07. Then, the control unit 22 proceeds to step SH11.

  When the first object and the second object are detected and there is a temporarily selected three-dimensional object (step SH03, Yes), the control unit 22 sets the first object and the second object as step SH08. It is determined whether at least one of them is moving. When neither the first object nor the second object has moved (step SH08, No), the control unit 22 proceeds to step SH11.

  When at least one of the first object and the second object is moving (step SH08, Yes), the control unit 22 executes the operation detection process shown in FIG. 34 as step SH09, and selects among them. The three-dimensional object in the state is changed according to the detected operation. After the operation detection process is completed, the control unit 22 proceeds to step SH11.

  FIG. 34 is a flowchart illustrating the procedure of the operation detection process. The processing procedure shown in FIG. 34 is realized by the control unit 22 executing the control program 24a. As shown in FIG. 34, the controller 22 first calculates the distance between the first object and the second object as step SI01. And control part 22 judges whether the distance of the 1st object after the start time of operation detection processing and the 2nd object is almost constant as Step SI02.

  When the distance between the first object and the second object is substantially constant (step SI02, Yes), the control unit 22 determines whether a predetermined time has elapsed since the operation detection process was started as step SI03. If the predetermined time has elapsed (step SI03, Yes), the control unit 22 sets the three-dimensional object in the selected state if there is a temporarily selected three-dimensional object as step SI04. If the predetermined time has not elapsed (step SI03, No), step SI04 is not executed.

  Subsequently, as step SI05, the control unit 22 calculates the moving speeds of the first object and the second object. And the control part 22 determines whether the calculated moving speed is below a threshold value as step SI06. When the moving speed is equal to or lower than the threshold (step SI06, Yes), the control unit 22 moves or rotates the three-dimensional object according to the detected movements of the first object and the second object as step SI07. To do. Then, the control unit 22 re-executes step SI01 and subsequent steps.

  When the moving speed is faster than the threshold (step SI06, No), the control unit 22 deletes the three-dimensional object as step SI08. When deleting the three-dimensional object, an animation may be displayed so that the three-dimensional object flies in the moving direction of the first object and the second object. Then, the control unit 22 ends the operation detection process. Instead of the operation of moving the first object and the second object at a high speed, for example, erasure of the three-dimensional object may be assigned to an operation of crushing the three-dimensional object. Instead of deleting the three-dimensional object, the three-dimensional object may be returned to the original placement location. The display device 1 does not have to perform the processes of steps SI05, SI06, and step SI08. That is, the display device 1 may execute step SI07 regardless of the moving speed of the two objects after determining NO in step SI03 or after executing step SI04.

  When the distance between the first object and the second object is not substantially constant (No in step SI02), the control unit 22 determines whether the distance is a three-dimensional object, that is, the start point of the operation detection process, as step SI09. It is judged whether it is expanding. When the distance is increased (step SI09, Yes), the control unit 22 determines whether the three-dimensional object displayed between the first object and the second object is in a temporarily selected state as step SI10. judge.

  When the three-dimensional object is in the temporary selection state (step SI10, Yes), the control unit 22 cancels the temporary selection state of the three-dimensional object as step SI11. Moreover, the control part 22 reversely changes a three-dimensional object and returns it to the original state as step SI12. Then, the control unit 22 ends the operation detection process.

  When the three-dimensional object is not in the temporary selection state, that is, in the selection state (step SI10, No), the control unit 22 cancels the selection state of the three-dimensional object as step SI13. Moreover, the control part 22 moves the three-dimensional object which canceled the selection state according to gravity etc. as step SI14. Then, the control unit 22 ends the operation detection process. The movement here is displayed, for example, so that the three-dimensional object falls according to gravity and stops on the floor or table. Before stopping the movement of the three-dimensional object, the three-dimensional object may be bounced according to the elasticity of the three-dimensional object and the hardness of the floor or table. The magnitude of impact when the three-dimensional object collides with the floor or table may be calculated, and if the impact is greater than a predetermined value, the three-dimensional object may be displayed as damaged. Further, the three-dimensional object may be moved more slowly than when actual gravity works.

  When the distance between the first object and the second object is shorter than when the three-dimensional object is selected (step SI09, No), the control unit 22 deforms the three-dimensional object according to the distance as step SI15. . Then, the control unit 22 re-executes step SI01 and subsequent steps. For example, the degree of deformation of the three-dimensional object may be changed according to the hardness set as an attribute of the three-dimensional object.

  As described above, in the third example, since it is detected that the three-dimensional object is positioned between objects such as fingers, the three-dimensional object is changed according to the operation. It is easy for the user to recognize the selection of the three-dimensional object.

  As in Step S81 to Step S83 shown in FIG. 35, after at least one of the first object and the second object comes into contact with the three-dimensional object, the distance between the first object and the second object is approximately equal to or longer than a predetermined time. It may be maintained as a condition for selecting a three-dimensional object. By making contact with a three-dimensional object as a selection condition, it becomes easier for a user to select a desired three-dimensional object when a plurality of three-dimensional objects are displayed close to each other.

  The display device 1 described in the above embodiment can be applied to various uses. The three-dimensional object (display object) to be operated may be an object that imitates an actual object such as a book, a building block, a spoon, a chopstick, a playing card, clay, or a musical instrument, or a virtual object. It may be an object that does not exist, such as an avatar, a game character, or a virtual reality AR tag. Further, the change applied to the three-dimensional object according to the detected operation is not limited to the above movement, deformation, disappearance, replacement, and the like. Further, the change applied to the three-dimensional object according to the pressing operation is not limited to the above-described embodiment, and may be changed according to the type of the three-dimensional object.

  For example, when a three-dimensional object imitating clay (hereinafter simply referred to as “clay”) is an operation target, the clay is deformed according to the pressing operation so that the user can shape the clay into an arbitrary shape. Also good. Moreover, the clay may be hardened as if the clay dries over time. Further, when an operation of pushing clay with a finger or hand immersed in a three-dimensional object of water is detected, the viscosity of the clay may be improved.

  In addition, when a three-dimensional object that imitates a record board (hereinafter simply referred to as “record board”) is an operation target, the record board rotates around a fulcrum and plays a sound according to the pressing operation. Also good. A technique such as scratching by a disc jockey may be virtually realized by linking rotation and sound reproduction.

  In the first embodiment, the method in which the display device detects an operation on the three-dimensional object and changes the three-dimensional object according to the detected operation has been described. In Example 2 below, an example will be described in which the method described in Example 1 is applied to, for example, a case where a spreadsheet application is executed in a three-dimensional virtual space.

  The functional configuration of the display device 1 will be described with reference to FIG. FIG. 36 is a block diagram of the display device 1 according to the second embodiment. The functional configuration of the display device 1 according to the second embodiment is basically the same as that of the first embodiment (FIG. 6), except for the points described below.

  Similar to the first embodiment, the control program 24a stored in the storage unit 24 includes a detection processing unit 25, a display object control unit 26, and an image composition unit 27, and further includes a spreadsheet process. A part 28 is included. The data stored in the storage unit 24 includes object data 24b, action data 24c, and virtual space data 24d, and further includes spreadsheet data 24e.

  The spreadsheet processing unit 28 provides a function for realizing processing related to spreadsheets such as aggregation, calculation, and analysis of numerical data. The spreadsheet processing unit 28 displays a three-dimensional object whose surfaces are configured by a worksheet for performing processing related to the spreadsheet. The spreadsheet processing unit 28 implements processing related to spreadsheets according to operations detected on the worksheet. In the worksheet, for example, a plurality of grid cells are arranged vertically and horizontally. Operations detected on the worksheet include operations on the cells.

  A rule for the spreadsheet processing unit 28 to determine the processing content will be described with reference to FIGS. FIG. 37 is a diagram illustrating a state of a three-dimensional object displayed in the display space when viewed from the user. FIG. 38 is a development view of a three-dimensional object. E1 shown in FIG. 37 indicates the eyes of the user.

  As shown in FIG. 37, the user recognizes the display space 50 as a space having a depth in the Z-axis direction parallel to the user's line-of-sight direction α1. Then, the user determines whether or not a three-dimensional object (hereinafter referred to as a cube CB1) whose surfaces are composed of a worksheet for performing processing related to spreadsheets is floating in the display space 50. To recognize. In the following description, the notation of the Z-axis direction is not limited to either the direction from the user toward the back side of the display space 50 or the direction from the back side of the display space 50 toward the user. The notation of the direction (+) means a direction away from the user toward the back side of the display space 50, and means a direction opposite to the direction toward the origin of the three-dimensional coordinate system. Reference to the X-axis direction means at least the direction parallel to the X-axis. The expression “Y-axis direction” means at least a direction parallel to the Y-axis. When expressed as the Y-axis direction (+), it means the direction opposite to the direction toward the origin of the three-dimensional coordinate system. When expressed as the Y-axis direction (−), it means a direction toward the origin of the three-dimensional coordinate system.

  As shown in FIG. 38, each surface of the cube CB1 includes a worksheet WS1, a worksheet WS2, a worksheet WS3, a worksheet WS4, a worksheet WS5, and a worksheet WS6. Each of the worksheets WS1 to WS6 independently realizes a process related to a spreadsheet according to an operation from the user.

  FIG. 39 is a diagram illustrating an example of a process item determination rule in which an operation detected for a cube or a worksheet is associated with a process item included in a process related to a spreadsheet. The spreadsheet processing unit 28 determines a processing item to be executed according to the operation detected on the worksheet based on the processing item determination rule shown in FIG. 39, and executes the processing of the determined processing item.

  No. shown in FIG. The first rule includes a detection operation of “moving one surface of the cube in the Z-axis direction (+)” and “a portion where the surfaces constituting the cube CB1 are orthogonal to each other (a portion where cells of different worksheets are adjacent). In FIG. 5, when an operation is defined, a processing item “update the operation result” is associated.

  No. shown in FIG. The rule of 2 includes a detection operation “rotate a column or row of a cube” and a processing item “update the operation result and graph when there is an operation and a graph that refer to the rotating column or row”. Are associated.

  No. shown in FIG. The rule No. 3 is associated with the detection operation “cut the cube by hand” and the processing item “update the configuration information of the worksheet according to the division of the cube”. When the cube is divided, the number of worksheets, the number of cells included in the worksheet, the number of rows, the number of columns, and the like are updated.

  No. shown in FIG. The rule No. 4 is associated with a detection operation “move two cells in the X-axis direction to collide” and a processing item “multiply numeric data of cells”.

  No. shown in FIG. The rule No. 5 includes a detection operation of “moving two cells in the Y-axis direction to make them collide” and “numerical data in cells in the Y-axis direction (+) are changed to numerical data in cells in the Y-axis direction (−)”. Is associated with the process item “divide by”.

  No. shown in FIG. The rule “6” corresponds to the detection operation “crush the cube in the Y-axis direction (−)” and the processing item “erase worksheet data according to the amount of movement in the Y-axis direction (−)”. It is attached.

  No. shown in FIG. The rule No. 7 is associated with a detection operation “press a specific cell in the cube” and a processing item “delete a cell in a column or row including the cell”.

  No. shown in FIG. The rule No. 8 includes a detection operation of “colliding the separated cell with another worksheet” and a process of “updating the data of the worksheet from which the cell is separated and the data of the worksheet in which the cell collides”. Items are associated with each other.

  The process item determination rule shown in FIG. 39 is merely an example, and in addition to the example shown in FIG. 39, the type of operation to be detected and the content of the process associated with the operation may be added, changed, or deleted. Good.

  The change in the display mode of the cube CB1 (the worksheet constituting the six faces of the cube) according to the operation detected for the cube CB1 (the worksheet constituting the six faces of the cube) is detected by the detection processing unit 25, the display object. This is realized based on the functions provided by the control unit 26 and the image composition unit 27, and the object data 24b, the action data 24c, and the virtual space data 24d.

  Similar to the first embodiment, the detection processing unit 25 provides a function for detecting an actual object existing in the imaging range of the imaging units 40 and 42. The function provided by the detection processing unit 25 includes a function of measuring the distance to each detected object. In the second embodiment, the detection processing unit 25 mainly detects the user's hand or finger and measures the distance to the detected user's hand or finger.

  The display object control unit 26 provides a function for managing what three-dimensional objects are arranged in the virtual space and in what state each three-dimensional object is in the same manner as in the first embodiment. To do. The functions provided by the display object control unit 26 include detecting an operation on a three-dimensional object based on the movement of an actual object detected by the function of the detection processing unit 25, and changing the three-dimensional object based on the detected operation. The function to be included is included. In the second embodiment, the display object control unit 26 mainly applies to the cube CB1 (worksheet constituting the six surfaces of the cube) based on the movement of the user's hand detected by the function of the detection processing unit 25. The operation is detected, and the display mode of the cube CB1 (the worksheet constituting the six surfaces of the cube) is changed based on the detected operation.

  As in the first embodiment, the image composition unit 27 generates an image to be displayed on the display unit 32a and an image to be displayed on the display unit 32b by synthesizing the image in the real space and the image in the virtual space. Provide the function to do. The function provided by the image compositing unit 27 includes the real object and 3 based on the distance to the real object measured by the function of the detection processing unit 25 and the distance from the viewpoint to the three-dimensional object in the virtual space. A function to determine the context of the dimension object and adjust the overlap is included. In the second embodiment, the image composition unit 27 synthesizes the image of the cube CB1 and the image of the actual space. Further, the image compositing unit 27 uses the distance between the user's hand and the cube CB1 based on the distance to the user's hand detected by the function of the detection processing unit 25 and the distance from the viewpoint in the virtual space to the cube CB1. Determine the context and adjust the overlap.

  In the second embodiment, the object data 24b is used, for example, for displaying the cube CB1 as a three-dimensional object. In the second embodiment, the object data 24b stores information including the type, shape information, color, transparency, and the like of the cube CB1 (worksheet constituting the six surfaces of the cube). The type indicates the physical property of the cube CB1 (the worksheet that forms the six surfaces of the cube). The type takes values such as “rigid body” and “elastic body”, for example. The shape information is information indicating the shape of the cube CB1 (the worksheet that forms the six surfaces of the cube). The shape information is, for example, a set of vertex coordinates of the surfaces constituting the cube CB1 (the worksheet that constitutes the six surfaces of the cube). The color is the color of the surface of the cube CB1 (the worksheet constituting the six surfaces of the cube) or the color configuration. Transparency is the degree to which the cube CB1 (the worksheet that forms the six surfaces of the cube) transmits light.

  In the second embodiment, the action data 24c is generated when the operation on the cube CB1 (worksheet constituting the six faces of the cube) displayed as a three-dimensional object is detected. This is used to determine how to change the display mode of the worksheet. These changes include the disappearance, movement, rotation, deformation, replacement of the cube CB1, the disappearance of the cells of the worksheet constituting the six sides of the cube CB1, duplication, enlargement, reduction, deformation, replacement, vertical and horizontal of the worksheet. This includes disappearance, duplication, enlargement, reduction, deformation, replacement, etc. of a plurality of cells arranged in a lattice pattern. The replacement means that one cube, cell, or sheet is replaced with another cube, cell, or sheet.

  In the second embodiment, the virtual space data 24d holds information related to the state of the cube CB1 (worksheet constituting the six surfaces of the cube) displayed as a three-dimensional object. The state of the cube CB1 (worksheet constituting the six surfaces of the cube) includes, for example, the position, posture, deformation state, and the like.

  The spreadsheet data 24e includes the configuration information of the cube CB1 (the worksheet that constitutes the six faces of the cube), the data held in the cube CB1 (the worksheet that constitutes the six faces of the cube), and the cube calculated by the spreadsheet processor 28. Stores data related to various processes executed on CB1 (worksheets constituting six surfaces of a cube).

  Hereinafter, various controls based on the functions provided by the control program 24a according to the second embodiment will be described. The various controls based on the functions provided by the control program 24a according to the second embodiment are determined by the rules according to the operations detected for the cube CB1 (the worksheet constituting the six surfaces of the cube). And control of processing for changing the display mode of the worksheet according to the operation detected for the cube CB1 (worksheet constituting the six surfaces of the cube).

  FIG. 40 is a diagram illustrating an example of operations performed on the cube CB1. FIG. 41 is a diagram illustrating an example of a display mode of the cube CB1.

  For example, as illustrated in FIG. 41, when the display device 1 is operated to rotate the cube CB1 around the Y axis, for example, as illustrated in FIG. 41, the display surface (worksheet) of the cube CB1 is displayed. change. For example, the display device 1 stores in advance the movement of the user's finger for rotating the cube CB1 about the Y axis, and rotates the cube CB1 according to the movement of the user's finger to The display surface (worksheet) of CB1 is changed.

  For example, as shown in step S91 in FIG. 41, the display device 1 is centered on the Y axis in a state where the worksheet WS1 constituting one surface of the cube CB1 is displayed facing the front of the user. When an operation of rotating counterclockwise by 90 degrees is performed, as shown in step S92 of FIG. 41, the worksheet WS2 constituting the other surface of the cube CB1 is displayed so as to face the front of the user. For example, as shown in step S91 in FIG. 41, the display device 1 is centered on the Y axis in a state where the worksheet WS1 constituting one surface of the cube CB1 is displayed facing the front of the user. When an operation of rotating counterclockwise by 45 degrees is performed, as shown in step S93 of FIG. 41, a worksheet WS1 that forms one surface of the cube CB1 and a worksheet WS2 that forms the other surface of the cube CB1. Are displayed so that they face the front of the user.

  In the case shown in step S93 in FIG. 41, the display device 1 is not particularly described in the rule shown in FIG. 39, but switches the processing to the valid state for the worksheet WS1 and the worksheet WS2, and displays the spreadsheet data 24e. You may record that.

  FIG. 42 is a diagram illustrating an example of operations performed on the cube CB1. FIG. 43 is a diagram illustrating an example of a place where processing related to the spreadsheet is executed in the cube CB1. As a premise for the description of FIG. 42 and FIG. 43, in the portion where the worksheets constituting each surface of the cube CB1 are orthogonal (portion where cells of different worksheets are adjacent), input to the cells arranged in the respective worksheets It is assumed that the mathematical formula is defined so that the calculation is performed with reference to the data that is stored.

  As shown in FIG. 42, the display device 1 is an operation of moving the surface formed by the worksheet WS1 among the surfaces forming the cube CB1 in the Z-axis direction (+) with the palm of the user's hand H1. As shown in FIG. 43, the operated surface is displayed as if it were parallel. The display device 1 moves the surface constituted by the worksheet WS1 according to the distance that the user's hand H1 moves in the Z-axis direction (+).

  At the same time, the display device 1 displays the rule No. 1 shown in FIG. Based on 1, the processing related to the spreadsheet is executed. That is, the display device 1 determines whether the worksheet WS1 and the worksheet WS2 are in accordance with operations detected on the worksheet WS1 of the cube CB1 by the detection unit (for example, the detection processing unit 25 and the display object control unit 26). Refer to the data input to the cells arranged in the worksheet WS1 and the data input to the cells arranged in the worksheet WS2, which are adjacent to the position forming the angle (90 degrees). The arithmetic processing to be executed is executed. Similarly, the display device 1 has the data and the work input to the cell arranged in the work sheet WS1 adjacent to the position where the work sheet WS1 and the work sheet WS5 form an angle (90 degrees). An arithmetic process for referring to data input to a cell arranged on the sheet WS5 is executed. Similarly, the display device 1 is configured such that the data input to the cell arranged in the worksheet WS1 and the workpiece adjacent to the position where the worksheet WS1 and the worksheet WS4 form an angle (90 degrees). An arithmetic process for referring to data input to a cell arranged on the sheet WS4 is executed. Similarly, the display device 1 has the data and the work input to the cell arranged in the work sheet WS1 adjacent to the position where the work sheet WS1 and the work sheet WS6 form an angle (90 degrees). An arithmetic process for referring to data input to a cell arranged on the sheet WS6 is executed.

  42, for example, as shown in FIG. 43, a surface OP1 in which a surface constituted by the worksheet WS1 and a surface constituted by the worksheet WS2 intersect, among the surfaces constituting the cube CB1, and the worksheet It is composed of a surface OP2 where a surface composed of WS1 and a surface composed of a worksheet WS5 intersect, a surface OP3 where a surface composed of a worksheet WS1 and a surface composed of a worksheet WS4 intersect, and a worksheet WS1. 42 is changed from the case shown in FIG. 42, the display device 1 executes the calculation again, and stores the calculation stored in the spreadsheet data 24e. Update the result.

  For example, even before the operation shown in FIG. 42 is performed, that is, before the surface where the worksheet WS1 and the worksheets WS2, WS5, WS4, and WS6 intersect is changed, the display device 1 similarly, Arithmetic processing that refers to adjacent cells between worksheets can be executed.

  FIG. 44 is a diagram illustrating a configuration example of a worksheet. FIG. 45 is a diagram illustrating an example of a three-dimensional cell.

  As shown in FIG. 44, each cell of the worksheet constituting each surface of the cube CB1 may be constituted by a cubic object (hereinafter referred to as a cell cube). In the case shown in FIG. 44, the work sheet WS1 is in a state in which, for example, the surface SF1 facing the user side and the surface BF1 parallel to this surface among the surfaces constituting the cell cube CC1 are effective in processing relating to the spreadsheet. It is good. The display device 1 may display the surface BF1 by performing a transparent process on the surface SF1, or may display the surface BF1 when a hand is inserted into the cube CB1. Alternatively, when the display device 1 picks up the cell cube CC1 and detects an operation of rotating the cell cube CC1 around the Y axis, the display device 1 may display the surface SF1 and the surface BF1 as if they were switched.

  FIG. 46 is a diagram illustrating an example of displaying a graph in the cube CB1. As illustrated in FIG. 46, the display device 1 may display a graph GR1 corresponding to the calculation result executed on the worksheet in the cube CB1. The display device 1 creates a graph to be displayed in the cube CB1 based on the calculation result data stored in the spreadsheet data 24e.

  FIG. 47 is a diagram showing the concept of rotation of the column or row of the cube CB1. In FIG. 47, in order to explain the concept of rotation of the columns or rows of the cube CB1, the number of cell cubes, the number of columns and the rows formed by the cell cubes are arranged on the worksheets constituting each surface of the cube CB1. For simplicity, a cube composed of 3 columns × 3 rows is shown. For example, when the display device 1 detects an operation of rotating nine cell cubes including the cell cube CCx as a lump (rotating in the direction of the arrow 38-1) about the rotation axis Py, The cell cube is displayed as if it were rotated as a group of lines. Similarly, when the display device 1 detects an operation of rotating about the rotation axis Px (an operation of rotating in the direction of the arrow 38-2), is the nine cell cubes rotated as a whole in one row? Is displayed.

  At this time, when the graph shown in FIG. 46 refers to the calculation result based on the cell data of the worksheet WS1 constituting one surface of the cube CB1, the display device 1 determines the rule No. 1 shown in FIG. 2, the display device 1 updates the calculation result and the graph stored in the spreadsheet data 24e. The display device 1 reflects the update in the graph shown in FIG.

  FIG. 48 is a diagram showing the concept of division of the cube CB1. For example, when the display device 1 detects an operation of dividing the cube CB1 by hand, as shown in FIG. 48, the display device 1 displays the cube CB1 as if it was divided into a cube CB2 and a cube CB3. Good. For example, the display device 1 detects a location where the palm of the user's hand H1 moving in the Z-axis direction (+) crosses the inside of the cube CB1 in the Z-axis direction (+), and the cube CB1 is extracted from the corresponding location. To divide. The operation for displaying the cube CB1 as if it was divided is not limited to the operation of dividing the cube CB1 by hand, and may be any operation.

  At the same time, the display device 1 displays the rule No. 1 shown in FIG. 3, the data stored in the spreadsheet data 24e is updated. That is, the display device 1 updates the configuration information of the worksheet (the number of worksheets, the number of cells included in the worksheet, the number of rows, the number of columns, etc.) according to the division of the cube.

  FIG. 49 is a diagram showing the concept of selecting cell cubes on the worksheet that constitutes the cube CB1. The display device 1 is displayed so that, for example, a worksheet WS1 that forms one surface of the cube CB1 and a worksheet WS2 that forms the other surface of the cube CB1 face each other in half. In the case (for example, see step S93 in FIG. 41), the cell cube CC1 of the worksheet WS1 and the cell cube CC2 of the worksheet WS2 can be selected simultaneously.

  For example, when the display device 1 detects an operation of gripping the cell cube CC1 and the cell cube CC2 and moving the cell cube CC1 and the cell cube CC2 so as to collide with each other in the X-axis direction, Make the cell cube disappear. Then, in accordance with the timing of erasing the cell, the display device 1 displays the calculation result on a predetermined cell cube or the like of the worksheet in which the calculation is defined in advance. It is assumed that the cell calculation is executed in parallel with the control of the display mode of the cell cube.

  At the same time, the display device 1 displays the rule No. 1 shown in FIG. 4, the data stored in the spreadsheet data 24e is updated. Note that the display device 1 has the rule No. 1 shown in FIG. The “cell” in the detection operation 4 is automatically read as “cell cube” and the process is executed. That is, the display device 1 multiplies the numerical data of the cells and updates the result of the multiplication.

  FIG. 50 is a diagram showing a concept of crushing the cube CB1. When the display device 1 detects an operation of crushing in the Y-axis direction (−) from the side of the worksheet WS5 constituting one surface of the cube CB1, as shown in step S101 of FIG. 50, the display device 1 shows the process of step S102 of FIG. As described above, the nine cell cubes that are opposite to the side on which the crushing operation is performed are lost as if the cube CB1 was crushed. The display device 1 may adjust the amount of cell cubes to be extinguished according to the amount of movement of the palm of the user's hand H1 in the Y-axis direction (−).

  At the same time, the display device 1 displays the rule No. 1 shown in FIG. 6 to update the data stored in the spreadsheet data 24e. That is, the display device 1 erases the worksheet data corresponding to the cell cube to be erased. In the example shown in FIG. 50, a part of the cell data of the worksheets WS1 to WS4 and the data of the worksheet WS6 are completely erased.

  FIG. 51 is a diagram showing a concept of erasing one column of the cube CB1. As shown in FIG. 51, when the display device 1 detects an operation of pushing the cell cube CC1 of the worksheet WS1 constituting one surface of the cube CB1 in the Z-axis direction (+), the display device 1 of the worksheet WS1 including the cell cube CC1 is detected. Disappear column L1. For example, the display device 1 determines that the cell cube CC1 has been selected when a certain period of time has elapsed with the user's hand H1 and the cell cube CC1 in contact with each other, and the column L1 including the cell cube CC1. To be determined as the disappearance target. At this time, the display device 1 may discriminate whether to erase the column itself or to erase only the data written in the cells constituting the column according to the detected operation. Alternatively, the display device 1 may execute the disappearance of the column only on the worksheet WS1 so as not to affect other worksheets. In this case, the display device 1 automatically generates a new column in place of the deleted column.

  At the same time, the display device 1 displays the rule No. 1 shown in FIG. 7, the data stored in the spreadsheet data 24e is updated. That is, the display device 1 deletes the cell in the column or row including the cell pressed by the user's hand H1.

  FIG. 52 is a diagram showing the concept of separation of the cell cube from the cube CB1. As shown in FIG. 52, when the display device 1 detects an operation of picking and pulling the cell cube CCx included in the worksheet constituting one surface of the cube CB1, it is as if the cell cube CCx is separated from the cube CB1. indicate. Subsequently, for example, when the display device 1 detects an operation of causing the cell cube CCx displayed as if it has been separated to collide with another worksheet, the data of the cell cube CCx is copied as if the data of the cell cube CCx was copied. The data is displayed on the worksheet on which the cell cube CCx is collided.

  At the same time, the display device 1 displays the rule No. 1 shown in FIG. 8, the data stored in the spreadsheet data 24e is updated. That is, the display apparatus 1 updates the data of the worksheet from which the cells are separated and the data of the worksheet in which the cells collide.

  In addition, when the display device 1 cannot display the data written in the cell by the area of the cell on the display, the display device 1 performs, for example, the cell according to the operation of sliding the surface of the cell with the user's finger. The data entered in may be scrolled.

  Alternatively, the display device 1 can display data (characters, characters) written on a worksheet that is visible in front of the user even when the cube CB1 is rotated around an arbitrary axis in response to the user's operation. The vertical direction of the numbers may be automatically displayed so as not to give the user a sense of incongruity.

  FIG. 53 is a flowchart illustrating a processing procedure according to the second embodiment. The processing procedure shown in FIG. 53 is realized by the control unit 22 executing the control program 24a.

  As shown in FIG. 53, the control unit 22 synthesizes and displays a cube CB1 having six worksheets as step SJ01.

  Subsequently, in step SJ02, the control unit 22 determines whether an operation on the worksheet is detected. As a result of the determination, when an operation on the worksheet is detected (step SJ02, Yes), the control unit 22 executes a process according to the detected operation as step SJ03. For example, the processing executed in step SJ03 includes control of processing related to the table calculation determined by the rule according to the operation detected for the cube CB1 (worksheet constituting the six surfaces of the cube), and Control of the process which changes the display mode of a worksheet according to operation detected with respect to cube CB1 (the worksheet which comprises 6 surfaces of a cube) is included (refer FIGS. 40-52 etc.).

  Subsequently, the control unit 22 determines whether the operation end is detected as Step SJ04. The operation end is detected, for example, when a predetermined operation is performed on the operation unit 13. When the operation end is detected (step SJ04, Yes), the control unit 22 ends the processing procedure shown in FIG. When the operation end is not detected (No at Step SJ04), the control unit 22 re-executes Step SJ02 and the subsequent steps.

  In step SJ02, if the result of determination is that an operation on the worksheet has not been detected (No in step SJ02), the control unit 22 proceeds to the processing procedure in step SJ04.

  The aspects of the present invention shown in the above embodiments can be arbitrarily changed without departing from the gist of the present invention. Moreover, you may combine each said Example suitably. For example, the control program shown in the above embodiment may be divided into a plurality of modules or may be integrated with other programs.

  In the above embodiment, the example in which the user himself / herself operates the three-dimensional object has been described. However, the display device may detect the movement of another person, an animal, a machine, or the like in the photographing range of the photographing unit as the three-dimensional object. . In addition, the display device may share the virtual space with other devices. That is, the display device may be configured such that a person other than the user of the display device can view and operate the three-dimensional object in the virtual space through another device.

  In the above embodiment, the display device alone detects an operation on the three-dimensional object. However, the display device may detect an operation on the three-dimensional object in cooperation with the server device. In this case, the display device sequentially transmits the image captured by the imaging unit or the information detected by the detection unit to the server device, and the server device detects the operation and notifies the display device of the detection result. With this configuration, the load on the display device can be reduced.

  Various controls executed by the display device described in the above embodiments can also be executed by a portable device such as a smartphone or a tablet.

DESCRIPTION OF SYMBOLS 1 Display apparatus 1a Front part 1b Side part 1c Side part 4d External device 13 Operation part 22 Control part 24 Storage part 24a Control program 24b Object data 24c Action data 24d Virtual space data 24e Spreadsheet data 25 Detection process part 26 Display object control part 27 Image composition unit 28 Spreadsheet processing unit 32a, 32b Display unit 40, 42 Image capturing unit 44 Detection unit 46 Distance measuring unit

Claims (6)

A display unit that is arranged in front of the user in a state of being worn on the head and displays a three-dimensional object for executing processing related to spreadsheet;
A storage unit that stores a rule in which an operation on the object is associated with a process item included in the process;
A detection unit for detecting the operation;
Based on the rule, it has a control unit for determining the processing items to be executed in response to the operation detected by the detecting unit,
The display unit displays three-dimensional graphs independent of each other in three dimensions . The display unit displays the three-dimensional object in which each surface is configured with a planar object for executing processing related to spreadsheet,
The display device according to claim 1, wherein the control unit determines, based on the rule, the processing items to be simultaneously executed on the planar object in accordance with the operation detected by the detection unit. . In the planar object, a plurality of cells, which are processing areas into which data is input, are arranged in a grid pattern,
The control unit, in response to the operation detected by the detection unit, out of the planar objects, data input to the cell arranged in the first planar object, and second The display device according to claim 2, wherein a process of simultaneously displaying data input to the cells arranged in the planar object is performed. In the planar object, a plurality of cells, which are processing areas into which data is input, are arranged in a grid pattern,
The control unit is adjacent to a position where a first planar object and a second planar object form an angle among the planar objects according to the operation detected by the detection unit. The data input to the cell arranged in the first planar object and the data input to the cell arranged in the second planar object. The display device according to claim 2, which executes arithmetic processing. A control method executed by a display device arranged in front of the user's eyes in a state of being worn on the head and displaying a three-dimensional object for executing processing related to spreadsheets,
Displaying the three-dimensional object on a display device in three dimensions;
Detecting an operation on the object;
Determining a processing item to be executed according to the detected operation with reference to a storage unit storing a rule in which the operation and a processing item included in the processing are associated ;
And a step of three-dimensionally displaying a three-dimensional graph independent of each other by the display device . A display device that is placed in front of the user in a state of being worn on the head and displays a three-dimensional object for executing processing related to spreadsheet,
Displaying the three-dimensional object on a display device in three dimensions;
Detecting an operation on the object;
Determining a processing item to be executed according to the detected operation with reference to a storage unit storing a rule in which the operation and a processing item included in the processing are associated ;
And a step of causing the display device to perform three-dimensional display of three-dimensional graphs independent of each other .

Images