CN115840506A - Data interaction method, device, equipment and medium based on wearable equipment - Google Patents

Abstract

The present disclosure relates to a wearable device-based data interaction method, device, device, and medium. The method includes: identifying terminal devices within a set space range based on wearable devices, and generating virtual wireframes and network protocols of the terminal devices address, in response to detecting the user's palm within the set space range, identifying the user's palm to generate a virtual hand corresponding to the user's palm in the wearable device, and the distance between the virtual hand and the virtual wireframe is less than a distance threshold, And when the target gesture of the virtual hand matches the preset interaction gesture, determine the data interaction process between the wearable device and the terminal device, and perform data exchange between the wearable device and the terminal device according to the data interaction process and the network protocol address. interact. Therefore, the application of wearable devices through gesture interaction can realize data interaction between devices, so that the interactive experience is stronger and the interactive operation is more convenient.

Description

Wearable device-based data interaction method, device, device and medium

technical field

The present disclosure relates to the technical field of mixed reality, and in particular to a wearable device-based data interaction method, device, device and medium.

Background technique

In related technologies, the data interaction between mobile devices is mainly through the establishment of wired or wireless connections between devices for data transmission, but it is limited by the problem of connection lines between devices, which may easily lead to data transmission between devices. Blocked, and the data interaction process needs to establish a communication connection between devices, resulting in cumbersome interaction process and affecting interaction efficiency. For example, during an indoor conference, when the user needs to display files on the projection device, it is necessary to establish a communication connection between the mobile device and the projection device through a wired or wireless method, and then transfer the data file from the mobile device to the projection device. In the device, if the connection between the mobile device and the projection device is blocked, or the connection is interrupted during the transmission process, normal data interaction between the mobile device and the projection device will not be possible.

Contents of the invention

In order to overcome the problems existing in related technologies, the present disclosure provides a wearable device-based data interaction method, device, device and medium.

According to the first aspect of an embodiment of the present disclosure, a wearable device-based data interaction method is provided, including:

Identifying terminal devices within a set space range based on the wearable device, and generating a virtual wireframe and a network protocol address of the terminal device;

Responding to detecting the user's palm within the set space range, identifying the user's palm, so as to generate a virtual hand corresponding to the user's palm in the wearable device;

When the distance between the virtual hand and the virtual wireframe is less than the distance threshold, and the target gesture of the virtual hand matches the preset interaction gesture, determine the distance between the wearable device and the terminal device data interaction process;

Perform data interaction between the wearable device and the terminal device according to the data interaction process and the network protocol address.

Optionally, the determining the data interaction process between the wearable device and the terminal device includes:

monitoring the dynamic trajectory of the virtual hand to generate gestures of the virtual hand;

When the gesture action is a set extraction action, determine the current extraction target of the virtual hand in the wearable device;

If the current extraction target is not within the display range of the virtual wireframe, then determine that the data interaction process is a terminal device display process;

If the current extraction target is within the display range of the virtual wireframe, it is determined that the data interaction process is a wearable device display process.

Optionally, the performing data interaction between the wearable device and the terminal device according to the data interaction process and the network protocol address includes:

When the data interaction process is the display process of the wearable device, the network protocol address is sent to the network end, and the network protocol address is used to instruct the network end to send the network protocol address to the terminal according to the network protocol address The device requests the first target data corresponding to the current extraction target;

In response to receiving the first target data fed back by the network terminal, displaying the first target data at a set position of the wearable device;

When the data interaction process is the display process of the terminal device, the second target data corresponding to the current extraction target and the network protocol address are sent to the network end, and the network protocol address is used to indicate the The network end sends the second target data to the terminal device according to the network protocol address, so as to display the second target data in the terminal device.

Optionally, the identifying the user's palm to generate a virtual hand corresponding to the user's palm in the wearable device includes:

Carrying out key point recognition on the user's palm to generate a plurality of palm key points corresponding to the user's palm;

Based on the user's palm displayed in the wearable device, the plurality of palm key points are connected to generate the virtual hand.

Optionally, the method includes:

monitoring the movement trajectories of the plurality of palm key points in the virtual hand to generate the target gesture of the virtual hand;

If the similarity between the target gesture and the preset interaction gesture reaches a similarity threshold, it is determined that the target gesture matches the preset interaction gesture.

Optionally, the identifying the terminal device within the set space range based on the wearable device, and generating the virtual wireframe and network protocol address of the terminal device include:

Identifying the terminal device within a set space range based on the wearable device, so as to generate appearance features of the terminal device;

generating the virtual wireframe according to the appearance features;

sending the virtual wireframe to a network, where the virtual wireframe is used to instruct the network to determine the network protocol address of the terminal device according to the virtual wireframe;

The network protocol address fed back by the network terminal is received.

Optionally, the identifying the terminal device within the set space range based on the wearable device, and generating the network protocol address of the terminal device includes:

Identifying terminal devices within a set spatial range based on the wearable device to determine an actual spatial distance between the wearable device and the terminal device;

reporting the location information and the actual spatial distance of the wearable device to the network, where the location information and the actual spatial distance are used to instruct the network to determine the wearable device according to the location information and the actual spatial distance The network protocol address of the terminal device;

The network protocol address fed back by the network terminal is received.

According to a second aspect of an embodiment of the present disclosure, a wearable device-based data interaction device is provided, including:

The first generation module is configured to identify terminal devices within a set space range based on the wearable device, and generate a virtual wireframe and a network protocol address of the terminal device;

The second generation module is configured to, in response to detecting the user's palm within the set space range, identify the user's palm, so as to generate a virtual hand corresponding to the user's palm in the wearable device;

A determination module configured to determine that the wearable device is compatible with the wearable device when the distance between the virtual hand and the virtual wireframe is less than a distance threshold and the target gesture of the virtual hand matches a preset interaction gesture. A data interaction process between the terminal devices;

The execution module is configured to perform data interaction between the wearable device and the terminal device according to the data interaction process and the network protocol address.

According to a third aspect of an embodiment of the present disclosure, a wearable device is provided, including:

processor;

memory for storing processor-executable instructions;

Wherein, the processor is configured to implement the steps of any one of the methods in the first aspect of the present disclosure when executing the executable instructions.

According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, on which computer program instructions are stored, and when the program instructions are executed by a processor, the wearable device-based data provided by the first aspect of the present disclosure is realized. The steps of the interactive method.

The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

Through the above method, based on the wearable device, the terminal device within the set space range is identified, the virtual wireframe and network protocol address of the terminal device are generated, and the user's palm is recognized in response to the detection of the user's palm within the set space range , to generate a virtual hand corresponding to the user's palm in the wearable device, and determine that the wearable device The data interaction process with the terminal device, according to the data interaction process and the network protocol address, performs data interaction between the wearable device and the terminal device. Therefore, the application of wearable devices through gesture interaction can realize data interaction between devices, so that the interactive experience is stronger and the interactive operation is more convenient.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure.

Fig. 1 is a flow chart of a data interaction method based on a wearable device according to an exemplary embodiment.

Fig. 2 is an exemplary diagram showing a data interaction method for mixed reality glasses according to an exemplary embodiment.

Fig. 3 is a flowchart showing a method for generating a mixed reality scene according to an exemplary embodiment.

Fig. 4 is a flowchart showing a method for determining a gesture event according to an exemplary embodiment.

Fig. 5 is a flowchart showing a data interaction method based on a wearable device according to an exemplary embodiment.

Fig. 6 is a schematic diagram of a virtual hand according to an exemplary embodiment.

Fig. 7 is a flow chart showing a method for recognizing a target gesture according to an exemplary embodiment.

Fig. 8 is a schematic diagram showing a display process of a wearable device according to an exemplary embodiment.

Fig. 9 is a schematic diagram showing a display process of a terminal device according to an exemplary embodiment.

Fig. 10 is a flowchart showing a data interaction method based on mixed reality glasses according to an exemplary embodiment.

Fig. 11 is a block diagram of a data interaction device based on a wearable device according to an exemplary embodiment.

Fig. 12 is a block diagram of a wearable device according to an exemplary embodiment.

Detailed ways

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present disclosure as recited in the appended claims.

It should be noted that all actions to obtain signals, information or data in this application are carried out under the premise of complying with the corresponding data protection laws and policies of the country where the location is located, and with the authorization given by the corresponding device owner.

Before explaining the data interaction method in this application, it is necessary to introduce the wearable device in this application. In this application, the wearable device is used to realize the mixed reality technology, and the mixed reality (Mix Reality, MR) scene is an intermediary The form between the virtual scene and the real scene, including the augmented reality scene and the augmented virtual scene, refers to a new visual environment produced by merging reality and virtuality. In the new visual environment, physical objects and virtual objects coexist, and various interactions can occur between the two. In this application, the information flow interaction between multiple devices is based on wearable devices equipped with mixed reality technology, and indirect communication between devices is achieved through built-in services in multiple devices.

In related technologies, virtual objects have been endowed with more realistic attributes in various ways, such as: fixing virtual objects through spatial positioning technology, achieving a clear hierarchy between real objects and virtual objects through virtual and real occlusion technology, and making virtual objects Object synchronization realizes the lighting information of the object. The immersive interaction in mixed reality enables virtual objects to be integrated into every corner of life, so that the transmission medium can be transferred from the mobile phone screen to a thinner and more portable wearable device, and can be obtained from other electronic devices through image recognition and data transmission technology. Display specific information, and realize more data interaction in the mixed reality space of the wearable device according to the corresponding control gestures.

Fig. 1 is a flow chart of a data interaction method based on a wearable device according to an exemplary embodiment. As shown in Fig. 1 , the method is used in a wearable device and includes the following steps.

In step S101, terminal devices within a set space range are identified based on the wearable device, and a virtual wireframe and a network protocol address of the terminal device are generated.

It is worth mentioning that this embodiment is applied to wearable devices, which are equipped with mixed reality technology. Wearable devices can be mixed reality glasses, mixed reality helmets, etc. A mixed reality scene is displayed in the device. When the user wears the wearable device, the mixed reality scene can be presented to the user through the display device, wherein the spatial range displayed to the user in the mixed reality scene through the display device is the set spatial range that the wearable device can capture . In order to facilitate mixed reality interaction, when the user wears the wearable device, the wearable device will scan the set space range to determine the terminal devices within the set space range. For example, usually the terminal devices have relatively Fixed appearance features, for example, TVs, mobile phones, personal computers, etc., all have quadrangular outer frames and display devices, and terminal devices within a set space can be identified based on the appearance features, and the terminal devices can be identified according to the outer frame of the terminal devices. A virtual wireframe of the terminal device is generated at a corresponding position of the terminal device in the display device of the wearable device.

It should be noted that the principle of data interaction between the wearable device and the terminal device within the set space range in this embodiment is to share the data currently displayed on the display screen of the terminal device to the local area network through the local area network or the Internet or the Internet, and send the data to the wearable device for display through the LAN or the Internet, or share the data currently displayed in the wearable device to the LAN or the Internet, and send the data to the wearable device through the LAN or the Internet. The data is sent to the terminal device for display. Wherein, the data may be image data, video data, file data, etc., which is not limited in this embodiment. Therefore, the corresponding terminal device is a smart device that can access the network, where the smart device may include: a LAN device that needs to be connected to the Internet through a transit router, such as a TV, a laptop, a tablet computer, etc., and a device that can be connected directly through a network card Internet devices for the Internet, such as mobile phones, desktop computers connected to network cables, etc. For example, FIG. 2 is an example diagram showing a data interaction method for mixed reality glasses according to an exemplary embodiment. As shown in FIG. 2, the wearable device in this embodiment is mixed reality glasses. When the terminal device When it is a local area network device (TV, personal computer), the mixed reality glasses and the local area network device are connected to the Internet through the same router. At this time, the router acts as a data transfer interface to realize the indirect data interaction between the mixed reality glasses and the local area network device; When the terminal device is an Internet device (smart phone), the mixed reality glasses have a built-in networking function and can be connected to the Internet. At this time, the Internet serves as a data transfer interface between the mixed reality glasses and the Internet device to realize the connection between the mixed reality glasses and the Internet device. Indirect data interaction between Internet devices.

The network protocol address is the unique identity of the terminal device in the process of network data interaction. The network protocol addresses of different terminal devices are different. By identifying the network protocol address, the identity of the terminal device in the process of network data interaction can be determined, so that it can be determined The data file currently browsed by the terminal device. For example, in this embodiment, for LAN devices, in the router transfer station corresponding to the LAN, the mapping relationship between the corresponding appearance characteristics of each terminal device and the network protocol address of the terminal device is pre-configured. When the user wears the wearable device , identify the terminal devices within the set space range, thereby extracting the appearance features of the corresponding terminal devices, where the appearance features may include size features, color features, brand features, etc., and send the appearance features to the router transfer station , the router transfer station determines the network protocol address of the terminal device according to the appearance characteristics and the mapping relationship, and then feeds the network protocol address back to the wearable device, and the wearable device displays the network protocol address on the corresponding virtual line frame of the terminal device Location. For example, Fig. 3 is a flow chart showing a method for generating a mixed reality scene according to an exemplary embodiment. As shown in Fig. 3 , the method includes the following steps.

(1) Scan the scene within the set range through the wearable device to generate map information;

(2) Identify the terminal device in the scene according to the map information;

(3) If there is an identified terminal device in the scene, the virtual wireframe corresponding to the terminal device is retrieved from the database according to the device name of the terminal device, and the display position of the terminal device in the wearable device is used to prevent the Virtual wireframe; if the terminal device in the scene has not been recognized by the wearable device, then generate a corresponding virtual wireframe based on the position of the terminal device in the scene;

(4) Identify the terminal device. Enter the network IP address corresponding to the terminal device in the wearable device to produce a map file corresponding to the scene.

Optionally, the above step of generating the network protocol address of the terminal device may include:

Identify the terminal devices within the set space range based on the wearable device to determine the actual spatial distance between the wearable device and the terminal device;

Report the location information and actual spatial distance of the wearable device to the network, and the location information and actual spatial distance are used to instruct the network to determine the network protocol address of the terminal device based on the location information and actual spatial distance;

Network protocol address for receiving feedback from the network side.

It is worth mentioning that in this embodiment, the wearable device and the terminal device are connected to the same network. For example, if the terminal device is a LAN device, the wearable device and the terminal device are connected to the Internet through the same router transfer station; if The terminal device is an Internet device, and the wearable device and the terminal device are connected to the Internet through the same base station. As an example, the wearable device in this embodiment has a precise positioning function and a distance detection function, and the actual space between the wearable device and the terminal device can be determined through the display ratio and display distance of the terminal device in the display device of the wearable device distance, wherein the actual spatial distance may be a relative three-dimensional coordinate in a three-dimensional coordinate system. Send the actual space distance and the location information of the wearable device to the network terminal, and the network terminal determines the location of the wearable device according to the location information, and determines multiple other terminal devices within the preset range of the wearable device according to the precise positioning method , the network side determines the terminal device displayed on the display device of the wearable device from multiple other terminal devices according to the actual spatial distance between the wearable device and the terminal device, and according to the data interaction between the terminal device and the network side Record, determine the network protocol address corresponding to the terminal device, and feed back the network protocol address to the wearable device.

It should be noted that in this embodiment, the wearable device can scan and determine multiple terminal devices within the set space range, and the virtual wireframe and network protocol address corresponding to multiple terminal devices can be determined through the above-mentioned method, thereby increasing the available The practicability of wearable devices enables wearable devices to better realize data interaction with any terminal device within a set space range.

Optionally, in another implementation manner, the above step of generating the network protocol address of the terminal device may also include:

Identify terminal devices within a set space based on wearable devices to generate appearance features of terminal devices;

Generate a virtual wireframe based on appearance features;

Sending the virtual wireframe to the network side, the virtual wireframe is used to instruct the network side to determine the network protocol address of the terminal device according to the virtual wireframe;

The address of the network protocol that receives the feedback from the network.

It is worth mentioning that in this embodiment, there is a unique correspondence between the virtual wireframe of each terminal device within the set space and the terminal device, and the network protocol address corresponding to the virtual wireframe and the terminal device is set on the network side The mapping relationship among them, the network protocol address corresponding to the virtual wireframe can be determined by referring to the mapping relationship. As an example, in this embodiment, the wearable device identifies terminal devices within a set range, generates appearance features of the terminal devices, and generates a virtual wireframe according to the appearance features. The virtual wireframe is sent to the network side, and the network side determines the network protocol address corresponding to the virtual wireframe according to the above mapping relationship, and feeds back the network protocol address of the terminal device to the wearable device.

In step S102, in response to detecting the user's palm within the set space range, the user's palm is recognized, so as to generate a virtual hand corresponding to the user's palm in the wearable device.

It is worth mentioning that in this embodiment, the data interaction and data display in the wearable device are mainly controlled by the user's gestures, and the gestures in the wearable device are controlled by detecting the gestures corresponding to the user's palm in the display device of the wearable device. Video playback, image display, file display, and data interaction between wearable devices and terminal devices. For example, it is possible to control the display of corresponding image data on the display device of the wearable device through a fixed gesture, because the wearable device is loaded with mixed reality technology, so the image data is displayed on the real scene captured by the current wearable device , the image data is displayed on the real scene with a set spatial range, at this time, the user can also control the display position of the image data in the real scene according to corresponding gestures. Therefore, in order to identify the user intention of the corresponding user, the wearable device detects the user's palm within the set space range. The user is the center, and the palm image within the preset threshold range of the set spatial range is detected. When the palm image is detected within the preset threshold range, it is determined that the palm image is the user's palm, and the user's palm is recognized. To determine the gesture action corresponding to the user's palm.

For example, in this embodiment, important features such as finger size, palm size, and finger position of the user's palm can be detected, and a virtual hand that can be used to represent the user's palm can be generated in the wearable device according to the corresponding features. It is worth mentioning that the change of the user's palm in the wearable device includes position change and gesture change. When the virtual data displayed in the wearable device needs to be dragged, the user's palm is required to drag the virtual data in the wearable device , so that the position changes; when it is necessary to operate the virtual data displayed in the wearable device, it is necessary to issue an operation command to the wearable device through the fingers corresponding to the user's palm, so that the gesture changes. Therefore, the virtual hand includes the corresponding palm and fingers of the user to represent the position change feature and gesture change feature of the user's palm. Wherein, the display position of the virtual hand in the wearable device corresponds to the display position of the user's palm in the wearable device. When the user's palm changes in the wearable device, the virtual hand will follow the user's palm in the wearable device. Transform or move in the display device.

In step S103, when the distance between the virtual hand and the virtual wireframe is less than the distance threshold and the target gesture of the virtual hand matches the preset interaction gesture, determine the data interaction process between the wearable device and the terminal device.

For example, after the virtual hand is generated through the above steps, the distance between the virtual hand and the virtual wireframe is detected to determine whether relevant data interaction has been performed between the user's palm and the terminal device. When the user's palm touches the terminal device on the display device of the wearable device, and the corresponding virtual hand intersects with the virtual wireframe, it is determined that the user has the intention to interact with the terminal device, and then the target gesture corresponding to the user's virtual hand is determined. Detection is performed, and when the target gesture matches the preset interaction gesture, the data interaction process between the wearable device and the terminal device is determined through the target gesture.

As an example, the data interaction process between the wearable device and the terminal device includes (1) extracting the currently displayed data from the terminal device to the wearable device through the transfer station for data display; (2) transferring the wearable data through the transfer station The data currently displayed in the device is extracted to the terminal device for display. Therefore, the preset interaction gestures between the wearable device and the terminal device in this embodiment include: a data capture gesture and a data release gesture, and the interaction between the wearable device and the terminal device can be determined by detecting the position where the gesture corresponding to the virtual hand is generated. data interaction process. Fig. 4 is a flowchart of a method for determining a gesture event according to an exemplary embodiment. As shown in Fig. 4 , the method for determining includes the following steps.

(1) Identify the position and gesture type of the virtual hand;

(2) Shoot corresponding induction rays directly in front of the virtual hand;

(3) When the induction ray hits the virtual wireframe corresponding to the terminal device, determine the distance between the virtual hand and the virtual wireframe;

(4) When the distance is less than the threshold, determine the gesture corresponding to the current virtual hand;

(5) If the gesture matches the preset interaction gesture, it is determined that the gesture control process has passed, and the user is giving a control command to the wearable device based on the gesture event.

In step S104, data interaction is performed between the wearable device and the terminal device according to the data interaction process and the network protocol address.

For example, in this embodiment, after the data interaction process between the wearable device and the terminal device is determined through the above steps, according to the network protocol address of the terminal device, through the data transfer network end, the wearable device and the terminal device are exchanged. Data interaction. For example, when the data interaction process is that the wearable device extracts the data information currently displayed in the terminal device, the wearable device sends the network protocol address and data request instructions to the network side, and the network side determines the corresponding terminal according to the network protocol address After the device is installed, the data request command is sent to the terminal device, and the terminal device sends the data currently displayed on the display screen to the network terminal based on the data request command, and the network terminal sends the data to the wearable device for data display; When the data interaction process is for the wearable device to transmit data to the terminal device for data display, the currently displayed data and network protocol address are sent to the network through the wearable device, where the currently displayed data is the virtual hand selected in the wearable device According to the specified data that needs to be shared, the network end sends the data to the terminal device according to the network protocol address, so that the terminal device displays the data on the current screen after receiving the data.

Through the above method, based on the wearable device, the terminal device within the set space range is identified, the virtual wireframe and network protocol address of the terminal device are generated, and the user's palm is recognized in response to the detection of the user's palm within the set space range , to generate a virtual hand corresponding to the user's palm in the wearable device, and determine that the wearable device The data interaction process with the terminal device, according to the data interaction process and the network protocol address, performs data interaction between the wearable device and the terminal device. Therefore, the application of wearable devices through gesture interaction can realize data interaction between devices, so that the interactive experience is stronger and the interactive operation is more convenient.

Fig. 5 is a flow chart of a data interaction method based on a wearable device according to an exemplary embodiment. As shown in Fig. 5, the method is applied to a wearable device and includes the following steps.

In step S201, the wearable device identifies the terminal device within the set space range, and generates a virtual wireframe and a network protocol address of the terminal device.

As an example, the method of generating the virtual wireframe and the network protocol address in this embodiment is the same as that in the above step S101, and reference may be made to the above step S101, and details will not be repeated here.

In step S202, in response to detecting the user's palm within the set space range, the user's palm is recognized, so as to generate a virtual hand corresponding to the user's palm in the wearable device.

As an example, the method of generating the virtual hand in this embodiment is the same as that in the above step S102, and reference may be made to the above step S102, which will not be repeated here.

Optionally, in an implementation manner, the above step S202 includes:

Perform key point recognition on the user's palm to generate multiple palm key points corresponding to the user's palm.

Based on the user's palm displayed in the wearable device, multiple palm key points are connected to generate a virtual hand.

As an example, in this embodiment, key points are identified on the user's palm according to the activity points of the user's palm. The palm key points corresponding to the index finger are generated at the direct position of the different activities of the index finger. Recognize the user's palm based on the same recognition logic, and generate multiple palm key points corresponding to the user's palm. It should be noted that in this embodiment, the generated position of the palm key points in the wearable device fits the user's palm, and it can be observed from the user's perspective that the palm key points are attached to the user's palm. The display position and display outline in the wearable device connect multiple palm key points to generate a virtual hand corresponding to the user's palm. Fig. 6 is a schematic diagram of a virtual hand according to an exemplary embodiment. As shown in Fig. 6, by recognizing the virtual point of the user's palm, and according to the outline of the user's palm in the wearable device, the virtual point After the bits are connected, a virtual hand as shown in the figure is generated. It is worth mentioning that in this embodiment, the virtual points are numbered according to their arrangement order, so as to facilitate the subsequent recognition of the target gesture corresponding to the virtual hand, so as to analyze the user's control intention. For example, Fig. 7 is a flowchart of a target gesture recognition method according to an exemplary embodiment. As shown in Fig. 7, the method is applied to mixed reality glasses, and the method includes the following steps.

(1) In response to the start of the scene in the mixed reality glasses, turn on the camera of the mixed reality glasses;

(2) Analyze the image captured by the camera through the camera of the mixed reality glasses to determine whether there is a user's palm in the image;

(3) When there is a user's palm, generate key points corresponding to the user's palm in the space of the mixed reality glasses;

(4) Determine the gesture corresponding to the user's palm according to the position of the key point.

Optionally, in another implementation manner, the interaction method further includes:

The movement trajectory of multiple palm key points in the virtual hand is monitored to generate the target gesture of the virtual hand.

When the similarity between the target gesture and the preset interactive gesture reaches the similarity threshold, it is determined that the target gesture matches the preset interactive gesture.

It is worth mentioning that the position of the virtual hand in the wearable device fits the user's palm, and the virtual hand will follow the movement of the user's palm. Therefore, in this embodiment, the movement trajectory of multiple palm key points corresponding to the virtual hand is monitored. According to the relative relationship between the key points of each palm, determine the target gesture corresponding to the user's palm, and compare the target gesture with the preset interactive gesture to determine the similarity between the two. When the similarity reaches the similarity threshold, Then it is determined that the target gesture matches the preset interaction gesture.

In step S203, the dynamic trajectory of the virtual hand is monitored to generate gestures of the virtual hand.

In step S204, if the gesture action is a set extraction action, determine the current extraction target of the virtual hand in the wearable device.

It is worth mentioning that when performing data interaction between a wearable device and a terminal device, it is necessary to issue data interaction instructions to the wearable device through the user's palm. Different data interaction processes include data extraction actions, and can be based on data extraction. The position where the action is generated determines the data interaction process. As an example, the preset interactive gestures in this embodiment include a set extraction action and a set release action, and the wearable device monitors the dynamic trajectory of each finger in the virtual hand to generate gesture actions of the virtual hand. The gesture action is compared with the preset interaction gesture, and when the gesture action is a set extraction action, it is determined that the user needs to select the current extraction target in the display device of the wearable device.

In step S205, if the current extraction target is not within the display range of the virtual wireframe, it is determined that the data interaction process is a terminal device display process.

In step S206, if the current extraction target is within the display range of the virtual wireframe, it is determined that the data interaction process is a wearable device display process.

It is worth mentioning that the virtual wireframe generated in the above steps in this embodiment may be the boundary line of the terminal device, and the terminal device is equipped with a display screen. Usually, the boundary line of the terminal device is around the display screen Therefore, the display range of the virtual wireframe is the display range of the terminal device. Therefore, in this embodiment, the interaction process between the wearable device and the terminal device is determined by determining whether the extraction target corresponding to the set extraction action is within the display range of the virtual wireframe. For example, if the current extraction target is within the display range of the virtual wireframe corresponding to the terminal device, it is determined that the user intends to extract data from the terminal device and display it on the wearable device, and the corresponding data interaction process is the wearable device display process . If the current extraction target is not within the display range of the virtual wireframe corresponding to the terminal device, it is determined that the user intends to extract data from the wearable device to the terminal device for data display, and the corresponding data interaction process is the terminal device display process.

In step S207, data interaction is performed between the wearable device and the terminal device according to the data interaction process and the network protocol address.

As an example, the data interaction method in this embodiment is the same as that in the above step S104, and reference may be made to the above step S104, and details are not repeated here.

Optionally, in an implementation manner, the above step S207 includes:

When the data interaction process is a wearable device display process, the network protocol address is sent to the network end, and the network protocol address is used to instruct the network end to request the terminal device for the first target data corresponding to the current extraction target according to the network protocol address.

In response to receiving the first target data fed back by the network terminal, the first target data is displayed at the set position of the wearable device.

In the case that the data interaction process is a terminal device display process, send the second target data and network protocol address corresponding to the current extraction target to the network side, and the network protocol address is used to instruct the network side to send the second target to the terminal device according to the network protocol address data to display the second target data in the terminal device.

For example, FIG. 8 is a schematic diagram of a display process of a wearable device according to an exemplary embodiment. As shown in FIG. 8, when the data interaction process is determined to be a display process of a wearable device through the above steps in this embodiment, it can be The wearable device sends a network protocol address to the network, and the network sends a data request to the corresponding terminal device according to the network protocol address. After receiving the data request, the terminal device sends the first target data currently displayed on the display to the network. The first target data is fed back to the wearable device through the network transfer, and the first target data is displayed on the set position of the display device of the wearable device, so that the first target data can be shared between the wearable device and the terminal device. Synchronous display in . Fig. 9 is a schematic diagram of a display process of a terminal device according to an exemplary embodiment. As shown in Fig. 9, when the data interaction process is determined to be a display process of a terminal device, the wearable device will extract the second target corresponding to the current target The data and the network protocol address are sent to the network terminal. After the network terminal determines the corresponding terminal device according to the network protocol address, it sends the second target data to the terminal device. After receiving the second target data, the terminal device displays the Display the second target data.

Through the above method, after starting the wearable device, the user can set the interactive gestures in the wearable device to realize the data transmission between the wearable device and the terminal device, and complete the synchronous display between the wearable device and the terminal device, so that The interactive reality experience is stronger, and the interactive operation is more convenient.

Fig. 10 is a flow chart of a data interaction method based on mixed reality glasses according to an exemplary embodiment. As shown in Fig. 10, the method is applied to mixed reality glasses, and the method includes the following steps.

(1) Interactively determine the target gesture corresponding to the user's palm through the mixed reality glasses;

(2) When it is determined that there is a gesture interaction event, obtain the device IP address corresponding to the terminal device in the mixed reality glasses;

(3) Determine the interaction process between the terminal device and the mixed reality glasses according to the interaction track of the target gesture in the mixed reality glasses. For example, when the user's palm drags the file in the mixed reality glasses space to the virtual line represented by the terminal device Frame, determine the interaction process as the mixed reality glasses transmit data to the terminal device for display; when the user's palm drags the file displayed in the terminal device APP to any space of the mixed reality glasses, determine the interaction process as the mixed reality glasses request the terminal device data, and display the data in the corresponding position of the mixed reality glasses;

(4) The mixed reality glasses transmit information to the terminal device through the server according to the interaction process;

(5) When the interaction process is that the mixed reality glasses transmit data to the terminal device, the mixed reality glasses send information to the server, which includes the target data currently selected by the user's palm and the device IP address corresponding to the terminal device; The address sends the target data to the terminal device; after the background APP of the terminal device receives the target data, it displays the target data on the display screen;

(6) When the interaction process is that the mixed reality glasses request data from the terminal device, the mixed reality glasses send the data request and the device IP address to the server, and the server sends the data request to the corresponding terminal device according to the device IP address, and the terminal device According to the data request, the corresponding target data is sent to the server, the server transmits the target data to the mixed reality glasses, and the mixed reality glasses display the target data on a corresponding display position after receiving the target data.

Through the above method, after starting the wearable device, the user can set the interactive gestures in the wearable device to realize the data transmission between the wearable device and the terminal device, and complete the synchronous display between the wearable device and the terminal device, so that The interactive reality experience is stronger, and the interactive operation is more convenient.

Fig. 11 is a block diagram of a data interaction device based on a wearable device according to an exemplary embodiment. Referring to FIG. 11 , the device 100 includes: a first generation module 110 , a second generation module 120 , a determination module 130 and an execution module 140 .

The first generating module 110 is configured to identify terminal devices within a set space range based on the wearable device, and generate a virtual wireframe and a network protocol address of the terminal device;

The second generation module 120 is configured to recognize the user's palm in response to detecting the user's palm within the set space range, so as to generate a virtual hand corresponding to the user's palm in the wearable device;

The determination module 130 is configured to determine the data interaction between the wearable device and the terminal device when the distance between the virtual hand and the virtual wireframe is less than a distance threshold and the target gesture of the virtual hand matches the preset interaction gesture process;

The execution module 140 is configured to perform data interaction between the wearable device and the terminal device according to the data interaction process and the network protocol address.

Optionally, the determination module 130 is configured to:

Monitor the dynamic trajectory of the virtual hand to generate gestures of the virtual hand;

In the case that the gesture action is a set extraction action, determine the current extraction target of the virtual hand in the wearable device;

If the current extraction target is not within the display range of the virtual wireframe, determine the data interaction process as the terminal device display process;

If the current extraction target is within the display range of the virtual wireframe, it is determined that the data interaction process is a wearable device display process.

Optionally, the execution module 140 is configured to:

When the data interaction process is a wearable device display process, a network protocol address is sent to the network terminal, and the network protocol address is used to instruct the network terminal to request the terminal device for the first target data corresponding to the current extraction target according to the network protocol address;

displaying the first target data at a set position of the wearable device in response to receiving the first target data fed back by the network terminal;

In the case that the data interaction process is a terminal device display process, send the second target data and network protocol address corresponding to the current extraction target to the network side, and the network protocol address is used to instruct the network side to send the second target to the terminal device according to the network protocol address data to display the second target data in the terminal device.

Optionally, the second generating module 120 is configured to:

Carry out key point recognition on the user's palm to generate multiple palm key points corresponding to the user's palm;

Based on the user's palm displayed in the wearable device, multiple palm key points are connected to generate a virtual hand.

Optionally, the apparatus 100 includes a determination module, the determination module is configured to:

Monitor the movement trajectory of multiple palm key points in the virtual hand to generate the target gesture of the virtual hand;

When the similarity between the target gesture and the preset interactive gesture reaches the similarity threshold, it is determined that the target gesture matches the preset interactive gesture.

Optionally, the first generating module 110 is configured to:

Identify terminal devices within a set space based on wearable devices to generate appearance features of terminal devices;

Generate a virtual wireframe based on appearance features;

Sending the virtual wireframe to the network side, the virtual wireframe is used to instruct the network side to determine the network protocol address of the terminal device according to the virtual wireframe;

Network protocol address for receiving feedback from the network side.

Optionally, the first generating module 110 is configured to:

Identify the terminal devices within the set space range based on the wearable device to determine the actual spatial distance between the wearable device and the terminal device;

Report the location information and actual spatial distance of the wearable device to the network, and the location information and actual spatial distance are used to instruct the network to determine the network protocol address of the terminal device based on the location information and actual spatial distance;

Network protocol address for receiving feedback from the network side.

Regarding the apparatus in the foregoing embodiments, the specific manner in which each module executes operations has been described in detail in the embodiments related to the method, and will not be described in detail here.

The present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored. When the program instructions are executed by a processor, the steps of the wearable device-based data interaction method provided by the present disclosure are implemented.

Fig. 12 is a block diagram of a wearable device according to an exemplary embodiment. For example, the device 1200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and the like.

12, device 1200 may include one or more of the following components: processing component 1202, memory 1204, power supply component 1206, multimedia component 1208, audio component 1210, input/output interface 1212, sensor component 1214, and communication component 1216.

Processing component 1202 generally controls the overall operations of device 1200, such as those associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 1202 may include one or more processors 1220 to execute instructions to complete all or part of the steps of the above method for data interaction based on wearable devices. Additionally, processing component 1202 may include one or more modules that facilitate interaction between processing component 1202 and other components. For example, processing component 1202 may include a multimedia module to facilitate interaction between multimedia component 1208 and processing component 1202 .

The memory 1204 is configured to store various types of data to support operations at the device 1200 . Examples of such data include instructions for any application or method operating on device 1200, contact data, phonebook data, messages, pictures, videos, and the like. The memory 1204 can be realized by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

The power supply component 1206 provides power to various components of the device 1200 . Power components 1206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for device 1200 .

The multimedia component 1208 includes a screen that provides an output interface between the device 1200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may not only sense a boundary of a touch or swipe action, but also detect duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 1208 includes a front camera and/or a rear camera. When the device 1200 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capability.

The audio component 1210 is configured to output and/or input audio signals. For example, the audio component 1210 includes a microphone (MIC), which is configured to receive external audio signals when the device 1200 is in operation modes, such as call mode, recording mode and voice recognition mode. Received audio signals may be further stored in memory 1204 or sent via communication component 1216 . In some embodiments, the audio component 1210 also includes a speaker for outputting audio signals.

The input/output interface 1212 provides an interface between the processing component 1202 and a peripheral interface module. The peripheral interface module may be a keyboard, a click wheel, a button, and the like. These buttons may include, but are not limited to: a home button, volume buttons, start button, and lock button.

Sensor assembly 1214 includes one or more sensors for providing status assessments of various aspects of device 1200 . For example, the sensor component 1214 can detect the open/closed state of the device 1200, the relative positioning of components, such as the display and the keypad of the device 1200, and the sensor component 1214 can also detect a change in the position of the device 1200 or a component of the device 1200 , the presence or absence of user contact with the device 1200 , the device 1200 orientation or acceleration/deceleration and the temperature change of the device 1200 . Sensor assembly 1214 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 1214 may also include optical sensors, such as CMOS or CCD image sensors, for use in imaging applications. In some embodiments, the sensor component 1214 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

Communication component 1216 is configured to facilitate wired or wireless communications between device 1200 and other devices. The device 1200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In one exemplary embodiment, the communication component 1216 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 1216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, device 1200 may be programmed by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable Realized by a gate array (FPGA), controller, microcontroller, microprocessor or other electronic components, it is used to execute the above-mentioned data interaction method based on the wearable device.

In an exemplary embodiment, there is also provided a non-transitory computer-readable storage medium including instructions, such as a memory 1204 including instructions, which can be executed by the processor 1220 of the device 1200 to complete the above-mentioned wearable device-based data storage medium. interactive method. For example, the non-transitory computer readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

In addition to being an independent electronic device, the above-mentioned device can also be a part of an independent electronic device. For example, in one embodiment, the device can be an integrated circuit (Integrated Circuit, IC) or a chip, wherein the integrated circuit can be an IC , can also be a collection of multiple ICs; the chip can include but not limited to the following types: GPU (Graphics Processing Unit, graphics processing unit), CPU (Central Processing Unit, central processing unit), FPGA (Field Programmable Gate Array, programmable logic array), DSP (Digital Signal Processor, digital signal processor), ASIC (Application Specific Integrated Circuit, application specific integrated circuit), SOC (System on Chip, SoC, system on chip or system-on-a-chip), etc. The above-mentioned integrated circuits or chips can be used to execute executable instructions (or codes), so as to realize the above-mentioned wearable device-based data interaction method. The executable instructions may be stored in the integrated circuit or chip, or may be obtained from other devices or devices, for example, the integrated circuit or chip includes a processor, a memory, and an interface for communicating with other devices. The executable instruction can be stored in the memory, and when the executable instruction is executed by the processor, the above-mentioned data interaction method based on the wearable device can be realized; or, the integrated circuit or chip can receive the executable instruction and transmit it through the interface. Execute to the processor to realize the above-mentioned data interaction method based on the wearable device.

In another exemplary embodiment, there is also provided a computer program product comprising a computer program executable by a programmable device, the computer program having a function for performing the above-mentioned The code part of the wearable device-based data interaction method.

Other embodiments of the disclosure will be readily apparent to those skilled in the art from consideration of the specification and practice of the disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, and these modifications, uses or adaptations follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not disclosed in the present disclosure . The specification and examples are to be considered exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It should be understood that the present disclosure is not limited to the precise constructions which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (10)

1. A data interaction method based on a wearable device, characterized in that, comprising: Identifying terminal devices within a set space range based on the wearable device, and generating a virtual wireframe and a network protocol address of the terminal device; Responding to detecting the user's palm within the set space range, identifying the user's palm, so as to generate a virtual hand corresponding to the user's palm in the wearable device; When the distance between the virtual hand and the virtual wireframe is less than the distance threshold, and the target gesture of the virtual hand matches the preset interaction gesture, determine the distance between the wearable device and the terminal device data interaction process; Perform data interaction between the wearable device and the terminal device according to the data interaction process and the network protocol address. 2. The interaction method according to claim 1, wherein the determining the data interaction process between the wearable device and the terminal device comprises: monitoring the dynamic trajectory of the virtual hand to generate gestures of the virtual hand; When the gesture action is a set extraction action, determine the current extraction target of the virtual hand in the wearable device; If the current extraction target is not within the display range of the virtual wireframe, then determine that the data interaction process is a terminal device display process; If the current extraction target is within the display range of the virtual wireframe, it is determined that the data interaction process is a wearable device display process. 3. The interaction method according to claim 2, wherein the data interaction between the wearable device and the terminal device according to the data interaction process and the network protocol address comprises: When the data interaction process is the display process of the wearable device, the network protocol address is sent to the network end, and the network protocol address is used to instruct the network end to send the network protocol address to the terminal according to the network protocol address The device requests the first target data corresponding to the current extraction target; In response to receiving the first target data fed back by the network terminal, displaying the first target data at a set position of the wearable device; When the data interaction process is the display process of the terminal device, the second target data corresponding to the current extraction target and the network protocol address are sent to the network end, and the network protocol address is used to indicate the The network end sends the second target data to the terminal device according to the network protocol address, so as to display the second target data in the terminal device. 4. The interaction method according to claim 1, wherein the identifying the user's palm to generate a virtual hand corresponding to the user's palm in the wearable device comprises: Carrying out key point recognition on the user's palm to generate a plurality of palm key points corresponding to the user's palm; Based on the user's palm displayed in the wearable device, the plurality of palm key points are connected to generate the virtual hand. 5. The interactive method according to claim 4, characterized in that the method comprises: monitoring the movement trajectories of the plurality of palm key points in the virtual hand to generate the target gesture of the virtual hand; If the similarity between the target gesture and the preset interaction gesture reaches a similarity threshold, it is determined that the target gesture matches the preset interaction gesture. 6. The interaction method according to claim 1, wherein the terminal device within the set space range is identified based on the wearable device, and a virtual wireframe and a network protocol address of the terminal device are generated, include: Identifying the terminal device within a set space range based on the wearable device, so as to generate appearance features of the terminal device; generating the virtual wireframe according to the appearance features; sending the virtual wireframe to a network, where the virtual wireframe is used to instruct the network to determine the network protocol address of the terminal device according to the virtual wireframe; The network protocol address fed back by the network terminal is received. 7. The interaction method according to claim 1, wherein the identifying the terminal device within the set space range based on the wearable device, and generating the network protocol address of the terminal device includes: Identifying terminal devices within a set spatial range based on the wearable device to determine an actual spatial distance between the wearable device and the terminal device; reporting the location information and the actual spatial distance of the wearable device to the network, where the location information and the actual spatial distance are used to instruct the network to determine the wearable device according to the location information and the actual spatial distance The network protocol address of the terminal device; The network protocol address fed back by the network terminal is received. 8. A data interaction device based on a wearable device, characterized in that it comprises: The first generation module is configured to identify terminal devices within a set space range based on the wearable device, and generate a virtual wireframe and a network protocol address of the terminal device; The second generation module is configured to, in response to detecting the user's palm within the set space range, identify the user's palm, so as to generate a virtual hand corresponding to the user's palm in the wearable device; A determination module configured to determine that the wearable device is compatible with the wearable device when the distance between the virtual hand and the virtual wireframe is less than a distance threshold and the target gesture of the virtual hand matches a preset interaction gesture. A data interaction process between the terminal devices; The execution module is configured to perform data interaction between the wearable device and the terminal device according to the data interaction process and the network protocol address. 9. A wearable device, characterized in that, comprising: processor; memory for storing processor-executable instructions; Wherein, the processor is configured to implement the steps of the method according to any one of claims 1-7 when executing the executable instructions. 10. A computer-readable storage medium, on which computer program instructions are stored, wherein the steps of the method according to any one of claims 1-7 are implemented when the program instructions are executed by a processor.

Images