KR102692701B1 - Middleware apparatus and system for group physics interaction - Google Patents

Abstract

A middleware device and system for group physical interaction in a multi-mobile HMD environment, which can provide a more predictable user interface than before, are proposed. The proposed middleware device includes a mobile HMD communication unit which performs communication with each mobile HMD existing in a mobile HMD operating space, and an operating environment analysis unit which analyzes the environment in which each mobile HMD is operating in the mobile HMD operating space, selects a mobile HMD to provide a physical effect file based on the analysis result, and generates a physical effect file to be provided to the selected mobile HMD.

Description

{Middleware apparatus and system for group physics interaction}

The present invention relates to a middleware device and system for group physical interaction, and more particularly, to a middleware device and system for group physical interaction in an environment capable of operating a plurality of mobile HMDs.

Conventional virtual reality-based haptic technology provides vibration-based haptics to hands, feet, chest, and abdomen through form factors such as gloves, shoes, and suits.

PC-based (i.e. PC-connected) virtual reality HMDs had their own controllers with vibration haptic capabilities.

In contrast, most mobile-based (i.e., smartphone-insertable) HMDs do not have their own haptic controllers, and thus cannot apply haptic effects to various interactions in virtual reality. Accordingly, user actions such as clicking, selecting, and moving in virtual reality cannot be expressed as haptic interactions.

In addition, in cases where multiple users use mobile-based HMDs, safety accidents due to collisions between users are frequent (because their eyes are covered by the HMD), and it is impossible to provide physical effects according to clustering and location.

Prior art 1: Republic of Korea Publication Patent No. 10-2016-0036915 (Mobile device, HMD and system) Prior art 2: Republic of Korea Publication Patent No. 10-2014-0064647 (System and method for simulated physical interaction with haptic effects) Prior art 3: Republic of Korea Publication Patent No. 10-2016-0075333 (Systems and methods for enabling interaction with objects)

The present invention has been proposed to solve the above-mentioned conventional problems, and its purpose is to provide a middleware device and system for group physical interaction in a multi-mobile HMD environment that can provide a more predictable user interface than the conventional one.

Another object of the present invention is to provide a middleware device and system for group physical interaction in a multi-mobile HMD environment that can maximize usability in a multi-user mobile-based HMD usage situation.

Another object of the present invention is to provide a middleware device and system for group physical interaction in a multi-mobile HMD environment, which can improve the driving performance of the overall system.

In order to achieve the above-mentioned purpose, a middleware device for group physical interaction according to a preferred embodiment of the present invention includes: a mobile HMD communication unit for performing communication with each mobile HMD existing in a mobile HMD operating space; and an operating environment analysis unit for analyzing an environment in which each of the mobile HMDs is operating in the mobile HMD operating space, selecting a mobile HMD to provide a physical effect file based on the analysis result, and generating a physical effect file to be provided to the selected mobile HMD.

The above-described operating environment analysis unit may include a user positioning receiver unit that receives positioning information from a user positioning unit within the mobile HMD operating space; and a user cluster/location/characteristics analysis unit that analyzes clustering and location and movement characteristics of a plurality of users existing in the mobile HMD operating space based on the positioning information to select a mobile HMD to provide the physical effect file, and generates a physical effect file to be provided to the selected mobile HMD.

The above-mentioned operating environment analysis unit may additionally include an HMD terminal status analysis unit that analyzes the status of each mobile HMD based on system information of each mobile HMD and generates a physical effect file to be provided to the corresponding mobile HMD.

The above mobile HMD communication unit may include an HMD terminal status receiving unit that receives system information of each mobile HMD and sends it to the HMD terminal status analysis unit; and a file transmitting unit that transmits a physical effect file generated by the HMD terminal status analysis unit based on the system information of each mobile HMD to the corresponding mobile HMD.

When users within the above mobile HMD operating space are divided into a population group and a departure group, the operating environment analysis unit analyzes positioning information from a user positioning unit within the mobile HMD operating space to recognize the population group and the departure group, and transmits a physical effect file for inducing the departure group to return to the population group to the mobile HMDs of users of the departure group through the mobile HMD communication unit.

A middleware device for group physical interaction according to a preferred embodiment of the present invention may further include a non-HMD effect output unit that receives a physical effect file from the operating environment analysis unit and outputs it to a non-HMD effect output terminal.

The above non-HMD effect output terminal may be a separate effect output terminal other than the mobile HMD existing in the mobile HMD operating space.

When users within the above mobile HMD operating space are divided into a population group and a departure group, the operating environment analysis unit analyzes positioning information from a user positioning unit within the above mobile HMD operating space to recognize the population group and the departure group, and transmits a physical effect file for inducing the departure group to return to the population group to a non-HMD effect output terminal located at the shortest distance from the departure group through the non-HMD effect output unit.

The above physical effect file may be a file containing a haptic effect pattern for driving an actuator equipped in the mobile HMD.

And, a system for group physical interaction according to a preferred embodiment of the present invention includes a middleware device including a user positioning unit for positioning each user within a mobile HMD operating space; a mobile HMD worn by each of the users; and a mobile HMD communication unit for performing communication with each of the mobile HMDs; and an operating environment analysis unit for analyzing an environment in which each of the mobile HMDs is operated within the mobile HMD operating space, selecting a mobile HMD to provide a physical effect file based on the analysis result, and generating a physical effect file to be provided to the selected mobile HMD.

The above mobile HMD may include a game engine that drives virtual reality content; a haptic operation unit that outputs physical effects that a user can feel; and a middleware communication processing unit that collects system information of the mobile HMD and transmits it to the middleware device, transmits a physical effect file from the middleware device to the haptic operation unit, and transmits interaction data in the game engine according to user input to the haptic operation unit so that physical effects according to user input are output.

The above haptic operating unit may include an actuator driving unit that drives an actuator provided in the mobile HMD.

According to the present invention of this configuration, user convenience can be maximized by providing a more predictable and tactile user interface by utilizing a vibration actuator of a mobile device inserted into an HMD.

In addition, it is highly practical because it is interface compatible with not only advanced input technologies such as motion tracking but also conventional input devices such as joysticks.

In addition, it can be said to be economical in that the user experience of virtual reality can be improved only through software modifications without the purchase of additional hardware.

In addition, through linkage with middleware, not only haptic effects based on the content embedded in the mobile-based HMD itself, but also external haptic effects delivered through the middleware can be linked to the mobile-based HMD. Accordingly, it can be said that its expandability and cost-effectiveness are superior to those of conventional technologies.

In addition, in cases where multiple users use mobile-based HMDs, it prevents safety accidents caused by collisions between users, and enables the provision of physical effects according to clustering and location. This maximizes the creative freedom of content creators.

In addition, the introduction of middleware decentralizes computing resources for positioning, physical effect generation, and output, which were previously concentrated on mobile-based HMDs. Accordingly, not only can the operating performance of the overall system be improved, but the processing graphics level of the mobile-based HMD can also be improved.

FIG. 1 is a drawing showing an example in which a system for group physical interaction according to an embodiment of the present invention is employed in a mobile HMD operating space.
Figure 2 is an internal configuration diagram of the mobile HMD and middleware device illustrated in Figure 1.
FIG. 3 is a drawing for explaining an example of group physical interaction in the present invention, and is a drawing for explaining an example of transmitting physical effects to users of a disengaged group through a mobile HMD.
FIG. 4 is a drawing for explaining another example of group physical interaction in the present invention, and is a drawing for explaining an example of transmitting a physical effect to users of a disengaged group through a non-HMD effect output terminal.

The present invention may have various modifications and embodiments, and specific embodiments are illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, but should be understood to include all modifications, equivalents, or substitutes included in the spirit and technical scope of the present invention.

The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprises" or "has" and the like are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms defined in commonly used dictionaries, such as those defined in common usage, should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and shall not be interpreted in an idealized or overly formal sense unless expressly defined in this application.

Hereinafter, with reference to the attached drawings, a preferred embodiment of the present invention will be described in more detail. In order to facilitate an overall understanding in describing the present invention, the same reference numerals are used for the same components in the drawings, and redundant descriptions of the same components are omitted.

FIG. 1 is a diagram showing an example in which a system for group physical interaction according to an embodiment of the present invention is employed in a mobile HMD operating space, and FIG. 2 is a diagram showing the internal configuration of the mobile HMD and middleware device shown in FIG. 1.

A system for group physical interaction according to an embodiment of the present invention may include a user positioning unit (10), a mobile HMD (40), and a middleware device (50).

A system for group physical interaction can be installed in the mobile HMD operating space (60).

The mobile HMD operating space (60) refers to a space where multiple users use the same indoor/outdoor space while each wearing a mobile HMD (40) (i.e., mobile-based HMD). In Fig. 1, only one user and only one mobile HMD (40) are illustrated for convenience. In reality, it is better to understand that multiple users and multiple mobile HMDs (40) exist.

A user refers to a person who uses a mobile HMD operating space (60) and can experience virtual reality content in a multi-mobile HMD operating environment including a mobile HMD (40) and a non-HMD user input terminal (30).

The mobile HMD operating space (60) may be equipped with a non-HMD effect output terminal (20), a non-HMD user input terminal (30), and a system for group physical interaction (i.e., including a user positioning unit (10), a mobile HMD (40), and a middleware device (50)). Of course, the non-HMD effect output terminal (20) and the non-HMD user input terminal (30) may be included in the system for group physical interaction.

The user positioning unit (10) is a positioning terminal equipped with a positioning terminal (10a) and a positioning information transmitter (10b), and positions each user within the mobile HMD operating space (60) (positioning in two-dimensional coordinates).

The positioning terminal (10a) can position the user using a method such as Zigbee or GPS. The positioning terminal (10a) transmits the user's positioning information to the positioning information transmitter (10b).

The positioning information transmitter (10b) transmits user positioning information to the user positioning information receiver (52a) in the operating environment analysis unit (52) of the middleware device (50) using wireless communication technology.

A non-HMD effect output terminal (20) is a terminal that outputs some kind of physical effect. A non-HMD effect output terminal (20) is a device that does not perform physical effect output using components inside a mobile HMD (40), and is a device other than a mobile HMD (i.e., a mobile-based HMD) (40). In other words, a non-HMD effect output terminal (20) means a separate effect output terminal (effect output such as sense of movement, sense of smell, etc.) other than a mobile HMD (40) placed in a mobile HMD operating space (60).

The non-HMD effect output terminal (20) can provide the user with physical effects corresponding to the five senses of a human being. Here, the physical effect does not mean only a specific physical effect, but all physical effects including the five senses (e.g., effects of movement, sense of smell, etc.). Therefore, the non-HMD effect output terminal (20) can output various types of physical effects.

The non-HMD effect output terminal (20) includes an effect receiving unit (20a) and an effect driving unit (20b).

The effect receiving unit (20a) receives a physical effect file (i.e., non-HMD effect output) generated from the middleware device (50) and transmits it to the effect driving unit (20b).

The effect driver (20b) outputs a physical effect that can be felt by the user according to a certain physical effect file received through the effect receiver (20a).

A non-HMD user input terminal (30) can provide input for interaction with a virtual reality scene (42) within a game engine (41) when a user uses a mobile HMD (40) in a mobile HMD operating space (60).

Representative examples of non-HMD user input terminals (30) may include joysticks, portable terminals equipped with motion sensing functions, etc.

Accordingly, the user can manipulate or modify the cursor (42a) or object (42b) of the virtual reality scene (42) by operating buttons on the non-HMD user input terminal (30), using gestures, etc.

The mobile HMD (40) (i.e., mobile head mounted display) is a body-mounted display that each user in the mobile HMD operating space (60) wears on his or her head.

Mobile HMD (40) is used by mounting a mobile communication terminal such as a smartphone on the HMD.

The mobile HMD (40) can process communication with the middleware device (50).

The mobile HMD (40) includes a game engine (41) that drives virtual reality content, a middleware communication processing unit (46) that analyzes data, a user input recognition unit (45) that receives user input by sensors built into the mobile HMD (40) other than a non-HMD user input terminal (30), and a haptic operation unit (44) that operates a haptic effect received from the middleware communication processing unit (46). It should be noted that in the embodiment of the present invention, the mobile HMD (40) is presented as a concept that includes both attached hardware and software.

The game engine (41) is a physics engine used to drive virtual reality. In other words, the game engine (41) is a collection of software on an operating system (OS) for driving a virtual reality scene (42) and a virtual reality interaction recognition unit (43), and is responsible for all graphic (image) operations in a mobile HMD (40). Representative examples include Unity 3D and Unreal Engine.

A virtual reality scene (42) may include background graphics that compose the scene, a cursor (42a) having a two-dimensional and three-dimensional shape that can be manipulated through user input, and an object (42b) having a two-dimensional and three-dimensional shape that is created to enable interaction with the cursor (42a).

The cursor (42a) can take the shape of a circle, a wedge, a straight line, a specific shape, etc., and can have a volume (three dimensions) considering that it is a virtual reality scene (42) rather than just a two-dimensional area.

The object (42b) is a virtual object within the virtual reality scene (42) that interacts with the cursor (42a). The object (42b) may appear as an interface element such as a button or a general object shape such as a specific three-dimensional/plane figure. This is at the discretion of the author who creates the virtual reality graphics, and can be said to apply to most virtual reality three-dimensional works.

The virtual reality interaction recognition unit (43) is operated together with the virtual reality scene (42).

The virtual reality interaction recognition unit (43) may include an interaction monitoring unit (43a) that monitors interactions within a virtual reality scene (42), and an interaction information transmission unit (43b) that transmits interaction data extracted through monitoring by the interaction monitoring unit (43a) to the effect input/output unit (47) of the haptic operation unit (44) through a middleware communication processing unit (46).

The interaction monitoring unit (43a) monitors and recognizes the interaction between the cursor (42a) and the object (42b). Here, the interaction may include movement, transition, or change in physical characteristics of the object (42b) due to the cursor (42a).

The interaction information transmitter (43b) applies interaction data from the interaction monitoring unit (43a) to the effect input/output unit (47) of the haptic operation unit (44) through the middleware communication processing unit (46). Accordingly, the effect input/output unit (47) outputs a haptic effect according to the received interaction data.

The middleware communication processing unit (46) can collect basic system status data (e.g., battery status, operating time, temperature within the terminal, etc.) of the mobile HMD (40) and transmit it to the HMD terminal status analysis unit (52c) through the HMD terminal status receiving unit (56a) of the middleware device (50).

The middleware communication processing unit (46) can transmit a haptic effect file from the middleware device (50) to the effect input/output unit (47) of the haptic operation unit (44) of the corresponding mobile HMD (40). Here, the haptic effect file from the middleware device (50) can be of two types. That is, the first can be a haptic effect file according to the result of analyzing the status of the corresponding mobile HMD based on the system information (battery status (remaining amount), operating time, heat status, throttling characteristics, etc.) of each mobile HMD (40) in the mobile HMD operation space (60). The second can be a haptic effect file according to the result of analyzing the cluster map and location and movement characteristics of a plurality of users existing in the mobile HMD operation space (60) based on the user positioning information of each mobile HMD (40) in the mobile HMD operation space (60).

The middleware communication processing unit (46) receives interaction data output from the interaction information transmitting unit (43b) and transmits it to the effect input/output unit (47) of the haptic operation unit (44), thereby enabling a haptic effect to be driven according to user input.

The user input recognition unit (45) can recognize the user's input using an input sensor (45a) built into the mobile HMD (40) other than the non-HMD user input terminal (30). Here, the input sensor (45a) may include a camera, a microphone, a magnetic field sensor, an accelerometer, etc.

User input data generated in the user input recognition unit (45) is provided to the game engine (41). The cursor (42a) of the game engine (41) is manipulated according to the user input data, and as the cursor (42a) is manipulated, objects (42b) within the virtual reality scene (42) interact with each other.

The haptic operation unit (44) outputs a haptic effect corresponding to the interaction data from the virtual reality interaction recognition unit (43) received through the middleware communication processing unit (46), but outputs it in a form that can be felt by the user.

The haptic operating unit (44) includes an effect input/output unit (47) that includes an actuator driving unit (47a).

The effect input/output unit (47) can receive a physical effect file in which the driving strength and pattern of the actuator driving unit (47a) are specified. That is, the effect input/output unit (47) receives a physical effect file (e.g., a haptic effect file) transmitted from the middleware communication processing unit (46) and performs the task of converting the physical effect file into a form in which the actuator driving unit (47a) can smoothly drive the haptic effect.

The actuator driving unit (47a) drives an actuator equipped in the mobile HMD (40). Here, the actuator equipped in the mobile HMD (40) may be a linear actuator, a coin actuator, or the like, which are commonly applied to a mobile communication terminal mounted in the mobile HMD (40).

The actuator driving unit (47a) can output a haptic effect in a form that can be felt by the user. That is, the actuator driving unit (47a) enables the user to feel the haptic effect by driving the actuator.

The middleware device (50) mediates a non-HMD effect output terminal (20), a non-HMD user input terminal (30), and a plurality of mobile HMDs (40), thereby reducing the data processing load of each terminal and enabling integrated management.

The middleware device (50) is equipped and operated on a terminal (e.g., a PC, etc.) separate from the non-HMD effect output terminal (20), the non-HMD user input terminal (30), and a plurality of mobile HMDs (40).

The middleware device (50) includes a mobile HMD communication unit (51), an operating environment analysis unit (52), an operating environment library (53), an operating environment authoring unit (54), and a non-HMD effect output unit (55).

The mobile HMD communication unit (51) is responsible for communication with each mobile HMD (40) existing in the mobile HMD operating space (60). The mobile HMD communication unit (51) includes an input/output unit (56).

The input/output unit (56) includes an HMD terminal status receiving unit (56a) that receives an HMD terminal status, and a haptic effect file transmitting unit (56b) that transmits a physical effect file (e.g., a haptic effect file). Here, the haptic effect file transmitting unit (56b) may be an example of a file transmitting unit described in the patent claims of the present invention.

The HMD terminal status receiving unit (56a) receives system information (e.g., battery status, operating time, heat status, throttling characteristics, etc.) of each mobile HMD (40) from the middleware communication processing unit (46).

The HMD terminal status receiving unit (56a) sends the system information of each received mobile HMD (40) to the HMD terminal status analysis unit (52c) of the operating environment analysis unit (52).

The haptic effect file transmission unit (56b) transmits the haptic effect file (i.e., physical effect file) generated by the operating environment analysis unit (52) to the haptic operation unit (44) through the middleware communication processing unit (46) of the mobile HMD (40). Here, the haptic effect file generated by the operating environment analysis unit (52) can be generated by the user group/location/characteristic analysis unit (52b) and the HMD terminal status analysis unit (52c), respectively.

The operating environment analysis unit (52) analyzes the environment in which the mobile HMD (40) is being operated by the user in the mobile HMD operating space (60), and generates a physical effect file (e.g., a haptic effect file) to be operated in the mobile HMD (40) and/or non-HMD effect output terminal (20) based on the analysis results.

The operating environment analysis unit (52) includes a user positioning information receiving unit (52a), a user cluster/location/characteristics analysis unit (52b), and an HMD terminal status analysis unit (52c).

The user positioning government receiving unit (52a) receives the positioning information generated by the user positioning unit (10) and transmits it to the user cluster/location/characteristics analysis unit (52b).

The user cluster/location/characteristic analysis unit (52b) analyzes clusters, locations, movement characteristics (patterns), etc. of multiple users based on positioning information from the user positioning government receiving unit (52a), and generates a physical effect file (e.g., a haptic effect file) that meets preset conditions. Here, the preset conditions mean specific clusters, specific locations, and specific movement characteristics within the mobile HMD operating space (60).

The physical effect file generated in the user cluster/location/characteristic analysis unit (52b) is transmitted to the mobile HMD (40) and/or non-HMD effect output terminal (20), and is experienced by the user.

The HMD terminal status analysis unit (52c) analyzes the status of the corresponding mobile HMD (40) based on the system information (battery status, operating time, heat status, throttling characteristics, etc.) of the mobile HMD (40) received by the HMD terminal status receiving unit (56a). Then, the HMD terminal status analysis unit (52c) transmits a specific physical effect file to the corresponding mobile HMD (40) through the haptic effect file transmitting unit (56b) so that the user can recognize it. This is to prevent safety accidents or inconveniences in use while the mobile HMD (40) is being operated in a multi-user environment, and to provide the user with related information in a form that can be experienced in advance.

In other words, the HMD terminal status analysis unit (52c) can analyze the system information of each mobile HMD (40) and transmit a specific physical effect file suitable for each mobile HMD (40) accordingly. That is, the HMD terminal status analysis unit (52c) analyzes the status based on the system information of each mobile HMD (40) and transmits a specific physical effect file to the corresponding mobile HMD (40). Here, the physical effect file may mean a file including a haptic effect pattern for driving an actuator (including vibration intensity). The system information of the mobile HMD (40) may mean information such as remaining battery level and overheating, and different haptic effect files may be transmitted to the corresponding mobile HMD (40) through the mobile HMD communication unit (51) according to the system information of each mobile HMD (40).

Finally, the operating environment analysis unit (52) analyzes the environment in which each mobile HMD (40) in the mobile HMD operating space (60) is being operated in the mobile HMD operating space (60). Then, the operating environment analysis unit (52) selects a mobile HMD to provide a physical effect file (e.g., a haptic effect file) based on the analysis result, and can generate a physical effect file to be provided to the selected mobile HMD. Here, the environmental analysis means analyzing the status of each mobile HMD based on the system information (battery status (remaining amount), operating time, heat status, throttling characteristics, etc.) of each mobile HMD. In addition, the environmental analysis may also mean analyzing the clustering map and location and movement characteristics of multiple users existing in the mobile HMD operating space (60) based on the user positioning information of each mobile HMD (40). This is because the operating environment analysis unit (52) can perform both analysis based on system information of each mobile HMD (40) and analysis based on user positioning information.

The operating environment library (53) provides pre-written mapping information required when the user cluster/location/characteristic analysis unit (52b) analyzes user positioning information. The mapping information can preset the occurrence of physical effects at specific locations within the mobile HMD operating space (60), the occurrence of physical effects at specific clusters, etc. according to location.

The operating environment authoring unit (54) can create mapping information of the operating environment library (53) and author physical effects to be attached to the mapping information.

The operating environment authoring unit (54) includes a mapping unit (54a) and a haptic authoring unit (54b).

The mapping unit (54a) presets the occurrence of physical effects at specific locations within the mobile HMD operating space (60), the occurrence of physical effects at specific clusters, etc. according to location, and generates mapping information based on the preset information.

The haptic authoring unit (54b) authores the pattern, intensity, duration, etc. of the haptic effect, which is a physical effect to be attached to the mapping information.

The non-HMD effect output unit (55) receives a physical effect file generated according to the user's movement and group characteristics from the user group/location/characteristic analysis unit (52b), and outputs the received physical effect file as an effect file for driving the non-HMD effect output terminal (20).

The embodiment of the present invention configured as described above can provide a more predictable user interface by using a vibration actuator equipped in a smart phone terminal inserted into a mobile HMD (40). Here, a predictable interface means an interface that can predict how the system will operate when a user performs a certain operation through some kind of medium without a separate explanation of the interface itself. In general, an interface with high predictability provides a superior user experience.

In addition, the usability in a multi-user mobile HMD usage situation can be maximized through the user positioning unit (10) and the middleware device (50). That is, in the past, when multiple users used the mobile HMD (40), safety accidents due to collisions between each other were frequent (because their eyes were covered by the HMD), and it was impossible to provide physical effects according to clustering and location, but in the embodiment of the present invention, this can be solved by introducing the user positioning unit (10) and the middleware device (50).

In addition, through the introduction of a middleware device (50), computing resources for positioning, physical effect generation, and output, which were previously focused on the mobile HMD, are distributed, thereby improving the operating performance of the overall system.

FIG. 3 is a drawing for explaining an example of group physical interaction in the present invention, and is a drawing for explaining an example of transmitting physical effects to users of a disengaged group through a mobile HMD.

In Fig. 3, a number of users within the mobile HMD operating space (60) are divided into two groups (i.e., the population group (101) and the exit group (102)).

The situation in the virtual reality scene (42) is assumed to be a multi-user war game, and a scene is assumed in which friendly units break away from the population (101) and move.

It is assumed that each user within the mobile HMD operating space (60) is wearing a mobile HMD (40).

Finally, Fig. 3 assumes a situation in which a group (102) is created in a population (101), and a physical effect file (e.g., a haptic effect file) is transmitted to the mobile HMD (40) of users of the group (102). That is, user positioning, file transmission, and effect output are performed as follows.

First, the user positioning unit (10) within the mobile HMD operating space (60) positions the user group (i.e., the population (101) and the exit group (102)).

Accordingly, the positioning information generated in the user positioning unit (10) (i.e., the user positioning information of the population (101) and the user positioning information of the departure group (102)) is transmitted to the middleware device (50) through the communication environment (103). Here, the middleware device (50) is assumed to be stored in the middleware operating system hardware (100).

The middleware device (50) recognizes that the population (101) and the departing group (102) have occurred by analyzing the received positioning information (i.e., the user positioning information of the population (101) and the user positioning information of the departing group (102)). In other words, the user cluster/location/characteristic analysis unit (52b) of the operating environment analysis unit (52) of the middleware device (50) analyzes the clustering, location, movement characteristics, etc. of multiple users based on the positioning information from the user positioning information receiving unit (52a) (i.e., the user positioning information of the population (101) and the user positioning information of the departing group (102)). Through this analysis, the user cluster/location/characteristic analysis unit (52b) can identify the location of the population (10), the number of its members, the locations of its members, etc., and the location of the departing group (102), the number of its members, the locations of its members, etc. The user cluster/location/characteristic analysis unit (52b) can estimate the group with a larger number of members among the population (101) and the outgoing group (102) as the population.

Thereafter, the middleware device (50) transmits a message and a physical effect file (i.e., a haptic effect file) for inducing the return of the leaving group (102) to the mobile HMD (40) of the users of the leaving group (102). In other words, the user cluster/location/characteristic analysis unit (52b) of the middleware device (50) generates a message and a haptic pattern for inducing the return of the group (101), and transmits the generated message and haptic pattern to the mobile HMD (40) of the users of the leaving group (102) through the input/output unit (51).

In this way, the haptic pattern that induces the return to the population (101) received by the middleware communication processing unit (46) of the mobile HMD (40) of the users of the departure group (102) is output through the effect input/output unit (47) of the haptic operation unit (44). In addition, the message that induces the return to the population (101) received by the middleware communication processing unit (46) can be displayed in the form of a pop-up in the virtual reality scene (42).

By this, users of the leaving group (102) can return to the population (101).

FIG. 4 is a drawing for explaining another example of group physical interaction in the present invention, and is a drawing for explaining an example of transmitting a physical effect to users of a disengaged group through a non-HMD effect output terminal.

In Fig. 4, a number of users within the mobile HMD operating space (60) are divided into two groups (i.e., the population group (101) and the exit group (102)).

The situation in the virtual reality scene (42) is assumed to be a multi-user war game, and a scene is assumed in which friendly units break away from the population (101) and move.

It is assumed that each user within the mobile HMD operating space (60) is wearing a mobile HMD (40).

Finally, Fig. 4 assumes a situation in which a group (102) is created in a population (101), and a physical effect file of some kind is transmitted to a non-HMD effect output terminal (20) located at the shortest distance from the group (102). That is, user positioning, file transmission, and effect output are performed as follows.

First, the user positioning unit (10) within the mobile HMD operating space (60) positions the user group (i.e., the population (101) and the exit group (102)).

Accordingly, the positioning information generated in the user positioning unit (10) (i.e., the user positioning information of the population (101) and the user positioning information of the departure group (102)) is transmitted to the middleware device (50) through the communication environment (103). Here, the middleware device (50) is assumed to be stored in the middleware operating system hardware (100).

The middleware device (50) recognizes that the population (101) and the departing group (102) have occurred by analyzing the received positioning information (i.e., the user positioning information of the population (101) and the user positioning information of the departing group (102)). In other words, the user cluster/location/characteristic analysis unit (52b) of the operating environment analysis unit (52) of the middleware device (50) analyzes the clustering, location, movement characteristics, etc. of multiple users based on the positioning information from the user positioning information receiving unit (52a) (i.e., the user positioning information of the population (101) and the user positioning information of the departing group (102)). Through this analysis, the user cluster/location/characteristic analysis unit (52b) can identify the location of the population (10), the number of its members, the locations of its members, etc., and the location of the departing group (102), the number of its members, the locations of its members, etc. The user cluster/location/characteristic analysis unit (52b) can estimate the group with a larger number of members among the population (101) and the outgoing group (102) as the population.

Thereafter, the middleware device (50) transmits a message and a physical effect file (e.g., a haptic effect file) for inducing the return of the leaving group (102) to the population (101) to the non-HMD effect output terminal (20) located at the shortest distance from the leaving group (102). In other words, the user cluster/location/characteristic analysis unit (52b) of the middleware device (50) generates a message and a physical effect file for inducing the return of the population (101), and transmits the generated message and the physical effect file to the non-HMD effect output terminal (20) through the non-HMD effect output unit (55).

By this, the non-HMD effect output terminal (20) provides a message and a predetermined physical effect to users of the leaving group (102). Accordingly, users of the leaving group (102) can return to the population (101).

As described above, the best embodiment has been disclosed in the drawings and the specification. Although specific terms have been used herein, they have been used only for the purpose of describing the present invention and have not been used to limit the meaning or the scope of the present invention described in the claims. Therefore, those skilled in the art will understand that various modifications and equivalent other embodiments are possible from this. Accordingly, the true technical protection scope of the present invention should be determined by the technical idea of the appended claims.

10: User positioning unit 10a: Positioning terminal
10b: Positioning information transmitter 20: Non-HMD effect output terminal
20a: Effect receiver 20b: Effect driver
30: Non-HMD user input terminal
40: Mobile HMD 41: Game Engine
42: Virtual Reality Scene 43: Virtual Reality Interaction Cognitive Unit
44: Haptic operation unit 45: User input recognition unit
46: Middleware communication processing unit 47: Effect input/output unit
50: Middleware device 51: Mobile HMD communication unit
52: Operational Environment Analysis Department 53: Operational Environment Library
54: Operating environment authoring section 55: Non-HMD effect output section
56 : Input/output section

Claims (20)

A mobile HMD communication unit that performs communication with each mobile HMD existing in the mobile HMD operating space; and
An operating environment analysis unit that analyzes the environment in which each of the above mobile HMDs is operated in the mobile HMD operating space, selects a mobile HMD to provide a physical effect file based on the results of the analysis, and generates a physical effect file to be provided to the selected mobile HMD; including,
The above operating environment analysis department is,
A user positioning information receiving unit that receives positioning information from a user positioning unit within the above mobile HMD operating space;
A user cluster/location/characteristics analysis unit that analyzes the cluster map and location and movement characteristics of multiple users existing in the mobile HMD operating space based on the above positioning information, selects a mobile HMD to provide the physical effect file, and generates a physical effect file to be provided to the selected mobile HMD; and
An HMD terminal status analysis unit that analyzes the status of each mobile HMD based on the system information of each of the above mobile HMDs and generates a physical effect file to be provided to each of the above mobile HMDs; including:
The system information of the above mobile HMD is battery level, operating time, heat status, and throttling characteristics.
The above physical effect file is a file containing a haptic effect pattern for driving an actuator equipped in the mobile HMD.
The above user cluster/location/characteristics analysis section,
When users in the above mobile HMD operating space are divided into a population group and a departure group, the population group and the departure group are recognized by analyzing the positioning information from the user positioning unit in the above mobile HMD operating space, and a physical effect file that induces the departure group to return to the population is transmitted to the non-HMD effect output terminal located at the shortest distance from the departure group through the non-HMD effect output unit.
When recognizing the above population and the above outgroup,
Identify the location of the above population, the number of members, and the location of the members.
Identify the location of the above-mentioned breakaway group, the number of its members, and the location of its members;
A middleware device for group physical interaction, characterized in that it estimates a group with a large number of members among the above population and the above departure group as the population. delete delete In claim 1,
The above mobile HMD communication unit,
An HMD terminal status receiving unit that receives system information of each of the above mobile HMDs and sends it to the HMD terminal status analysis unit; and
A middleware device for group physical interaction, characterized by including a file transmission unit that transmits a physical effect file generated based on the system information of each mobile HMD in the HMD terminal status analysis unit to the corresponding mobile HMD. delete delete In claim 1,
The above non-HMD effect output terminal is,
A middleware device for group physical interaction, characterized in that it is a separate effect output terminal other than the mobile HMD existing in the above mobile HMD operating space. delete delete Each mobile HMD existing in the mobile HMD operating space; and
Including a middleware device that performs communication with each of the above mobile HMDs,
The above middleware device,
A mobile HMD communication unit that performs communication with each of the above mobile HMDs;
An operating environment analysis unit that analyzes the environment in which each of the above mobile HMDs is operated in the mobile HMD operating space, selects a mobile HMD to provide a physical effect file based on the results of the analysis, and generates a physical effect file to be provided to the selected mobile HMD; including,
The above operating environment analysis department is,
A user positioning information receiving unit that receives positioning information from a user positioning unit within the above mobile HMD operating space;
A user cluster/location/characteristics analysis unit that analyzes the cluster map and location and movement characteristics of multiple users existing in the mobile HMD operating space based on the above positioning information, selects a mobile HMD to provide the physical effect file, and generates a physical effect file to be provided to the selected mobile HMD; and
An HMD terminal status analysis unit that analyzes the status of each mobile HMD based on the system information of each of the above mobile HMDs and generates a physical effect file to be provided to each of the above mobile HMDs; including:
The system information of the above mobile HMD is battery level, operating time, heat status, and throttling characteristics.
The above physical effect file is a file containing a haptic effect pattern for driving an actuator equipped in the mobile HMD.
The above user cluster/location/characteristics analysis section,
When users in the above mobile HMD operating space are divided into a population group and a departure group, the population group and the departure group are recognized by analyzing the positioning information from the user positioning unit in the above mobile HMD operating space, and a physical effect file that induces the departure group to return to the population is transmitted to the non-HMD effect output terminal located at the shortest distance from the departure group through the non-HMD effect output unit.
When recognizing the above population and the above outgroup,
Identify the location of the above population, the number of members, and the location of the members.
Identify the location of the above-mentioned breakaway group, the number of its members, and the location of its members;
A system for group physical interaction, characterized in that a group with a large number of members among the above population and the above departure group is estimated as the population. In claim 10,
The above mobile HMD,
A game engine that drives virtual reality content;
A haptic operating unit that outputs physical effects that the user can feel; and
A system for group physical interaction, characterized by including a middleware communication processing unit that collects system information of the mobile HMD and transmits it to the middleware device, transmits a physical effect file from the middleware device to the haptic operation unit, and transmits interaction data from the game engine according to user input to the haptic operation unit so that a physical effect according to the user input is output. In claim 11,
The above haptic operating unit,
A system for group physical interaction, characterized by including an actuator driving unit that drives an actuator provided in the mobile HMD. delete delete In claim 10,
The above mobile HMD communication unit,
An HMD terminal status receiving unit that receives system information of each of the above mobile HMDs and sends it to the HMD terminal status analysis unit; and
A system for group physical interaction, characterized by including a file transmission unit that transmits a physical effect file generated based on the system information of each mobile HMD in the HMD terminal status analysis unit to the corresponding mobile HMD. delete delete In claim 10,
The above non-HMD effect output terminal is,
A system for group physical interaction, characterized in that it is a separate effect output terminal other than the mobile HMD existing in the above mobile HMD operating space. delete delete

Images