CN118118717A - Screen sharing method, device, equipment and medium - Google Patents

Abstract

The application provides a screen sharing method, a device, equipment and a medium, wherein the method comprises the following steps: responding to sensor data sent by a host side, and determining a screen sharing mode selected by the host side; if the screen sharing mode selected by the anchor terminal is the target sharing mode, performing debouncing processing on the sensor data to obtain optimized sensor data; rendering the display picture according to the optimized sensor data to obtain a target image picture; and sending the target image frames to the anchor side and the audience side so that the anchor side and the audience side display the target image frames. The application can reduce or even avoid discomfort such as dizziness and the like caused by image shake to human eyes.

Description

Screen sharing method, device, equipment and medium

Technical Field

The embodiments of the present application relate to the field of data processing technology, and in particular to a screen sharing method, apparatus, device and medium.

Background technique

With the development and popularization of Extended Reality (XR) devices, users can share the screen of XR devices with other users through screen sharing when using XR devices, thereby improving the experience of XR devices in multi-person interaction. Among them, screen sharing, as the name suggests, is to share the screen display content with others as it is.

At present, when sharing the screen of an XR device with other users, the user usually wears the XR device and uses the terminal device as the screen sharing client to upload data related to the screen content of the XR device to the cloud server. Then, the cloud server calls the graphics processing resources based on the received data for rendering, audio and video encoding and other operations. After that, the cloud server sends the processed data to the user and other users, so that the display devices of each user end perform decoding, rendering and other operations according to the data sent by the cloud server, and display the corresponding image screen and audio data on the screen. The specific process can be seen in Figure 1.

However, when the image on the screen of the user's XR device shakes in three-dimensional space, such as high-frequency shaking of pitch, roll (roll), and yaw, this shaking will cause the image to shake unnaturally. Moreover, this shaking picture will be rendered by the cloud server in real time and sent to the user and other users without any difference. Therefore, after the user and other users see the shaking picture, they will be affected by motion sickness caused by vestibular vision mismatch and will obviously feel dizzy and other discomfort.

Summary of the invention

The present application provides a screen sharing method, apparatus, device and medium, which can reduce or even avoid discomfort such as dizziness caused to the human eye due to image jitter.

In a first aspect, an embodiment of the present application provides a screen sharing method, including:

In response to sensor data sent by the host terminal, determining a screen sharing mode selected by the host terminal;

If the screen sharing mode selected by the anchor terminal is the target sharing mode, de-jittering the sensor data to obtain optimized sensor data;

Rendering the display screen according to the optimized sensor data to obtain a target image screen;

The target image screen is sent to the host end and the audience end so that the host end and the audience end display the target image screen.

In a second aspect, an embodiment of the present application provides a screen sharing device, including:

A mode determination module, configured to determine the screen sharing mode selected by the anchor terminal in response to sensor data sent by the anchor terminal;

A de-jitter processing module, configured to perform de-jitter processing on the sensor data to obtain optimized sensor data if the screen sharing mode selected by the anchor terminal is the target sharing mode;

An image rendering module, used to render a display image according to the optimized sensor data to obtain a target image;

The image distribution module is used to send the target image to the host terminal and the audience terminal so that the host terminal and the audience terminal display the target image.

In a third aspect, an embodiment of the present application provides an electronic device, including:

A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the screen sharing method described in the first aspect embodiment or its various implementations.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program enables a computer to execute the screen sharing method as described in the embodiment of the first aspect or its various implementations.

In a fifth aspect, an embodiment of the present application provides a computer program product comprising program instructions, which, when executed on an electronic device, enables the electronic device to execute the screen sharing method as described in the embodiment of the first aspect or its various implementations.

The technical solution disclosed in the embodiments of the present application has at least the following beneficial effects:

When receiving the sensor data sent by the host, the screen sharing mode selected by the host is determined. If the screen sharing mode is the target sharing mode, the sensor data is de-jittered to obtain optimized sensor data, and based on the optimized sensor data, the display screen is rendered to obtain the target image screen, and then the target image screen is sent to the host and the audience, so that the host and the audience display the target image screen. The present application determines the screen sharing mode selected by the host during screen sharing, and de-jitters the sensor data when the screen sharing mode is the target sharing mode, so as to filter out the subtle and high-frequency unnatural jitters in the unstable sensor data and obtain stable sensor data. Then, when the display screen is rendered based on the stable sensor data, a stable image screen can be rendered, so that the discomfort such as dizziness caused by image jitter to the human eye can be reduced or even avoided based on the stable image screen.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 is a schematic diagram of a screen sharing method provided in the related art;

FIG2 is a flow chart of a screen sharing method provided in an embodiment of the present application;

FIG3 is a schematic diagram of a sharing mode selection interface provided in an embodiment of the present application;

FIG4 is a schematic diagram of a user looking outward in a spherical model provided by an embodiment of the present application;

FIG5 is a flow chart of another screen sharing method provided in an embodiment of the present application;

FIG6 is a schematic diagram of de-jittering processing for 6DoF data provided by an embodiment of the present application;

FIG7 is a schematic block diagram of a screen sharing device provided in an embodiment of the present application;

FIG8 is a schematic block diagram of an electronic device provided in an embodiment of the present application.

Detailed ways

The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. According to the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

This application is applicable to the XR device screen sharing scenario. Considering that when the image of the XR device screen is currently shared, the image will shake in three-dimensional space due to the user's movement, such as high-frequency shaking of pitch, roll, and yaw. This shaking will cause the image to shake unnaturally, and this shaking image will be rendered by the cloud server without any difference and sent to the user. Therefore, when the user watches the shaking image, he will be affected by motion sickness caused by vestibular vision mismatch and will obviously feel dizziness and other discomfort. Therefore, this application designs a screen sharing solution, which can reduce or even avoid the discomfort such as dizziness caused by image shaking to the human eye.

In order to facilitate understanding of the embodiments of the present application, before describing each embodiment of the present application, some concepts involved in all embodiments of the present application are first appropriately explained, as follows:

1) Virtual Reality (VR), a technology for creating and experiencing a virtual world, generates a virtual environment by computation, and is a multi-source information (the virtual reality mentioned in this article includes at least visual perception, and can also include auditory perception, tactile perception, motion perception, and even taste perception, olfactory perception, etc.), realizing the fusion of virtual environment, interactive three-dimensional dynamic vision and simulation of entity behavior, allowing users to immerse themselves in a simulated virtual reality environment, and realizing applications in various virtual environments such as maps, games, videos, education, medical care, simulation, collaborative training, sales, assisted manufacturing, maintenance and repair.

2) Virtual reality devices (VR devices), terminals for realizing virtual reality effects, can usually be provided in the form of glasses, helmet-mounted displays (Head Mount Display, HMD), and contact lenses to realize visual perception and other forms of perception. Of course, the form of virtual reality devices is not limited to this, and can be further miniaturized or enlarged according to actual needs.

Optionally, the virtual reality devices described in the embodiments of the present application may include but are not limited to the following types:

2.1) PC-based virtual reality (PCVR) devices use the PC to perform related calculations and data output for virtual reality functions. External PC-based virtual reality devices use the data output by the PC to achieve virtual reality effects.

2.2) Mobile virtual reality devices support the configuration of mobile terminals (such as smart phones) in various ways (such as head-mounted displays with dedicated card slots). Through wired or wireless connection with the mobile terminal, the mobile terminal performs relevant calculations of virtual reality functions and outputs data to the mobile virtual reality device, such as watching virtual reality videos through the mobile terminal's APP.

2.3) All-in-one virtual reality devices have a processor for performing relevant calculations of virtual functions, and thus have independent virtual reality input and output functions. They do not need to be connected to a PC or mobile terminal and have a high degree of freedom in use.

3) Augmented Reality (AR): A technology that calculates the camera's posture parameters in the real world (or three-dimensional world, real world) in real time during the process of the camera capturing images, and adds virtual elements to the images captured by the camera based on the camera posture parameters. Virtual elements include but are not limited to: images, videos, and three-dimensional models. The goal of AR technology is to integrate the virtual world with the real world on the screen for interaction.

4) Mixed Reality (MR): A simulated setting that integrates computer-created sensory input (e.g., virtual objects) with sensory input from a physical setting or its representation. In some MR settings, the computer-created sensory input can adapt to changes in sensory input from the physical setting. In addition, some electronic systems used to present MR settings can monitor the orientation and/or position relative to the physical setting so that virtual objects can interact with real objects (i.e., physical elements from the physical setting or their representations). For example, the system can monitor movement so that virtual plants appear stationary relative to physical buildings.

5) Extended Reality (XR) refers to all real and virtual combined environments and human-computer interactions generated by computer technology and wearable devices, which include virtual reality (VR), augmented reality (AR) and mixed reality (MR) and other forms.

After introducing some concepts involved in the embodiments of the present application, a screen sharing method provided in the embodiments of the present application is specifically described below with reference to the accompanying drawings.

FIG2 is a flow chart of a screen sharing method provided in an embodiment of the present application. The embodiment of the present application is applicable to a scenario in which an XR device is used for screen sharing, and the screen sharing method can be performed by a screen sharing device. The screen sharing device can be composed of hardware and/or software and can be integrated into an electronic device. In an embodiment of the present application, the electronic device can be selected as a server, such as a cloud server, etc., which is not limited here.

As shown in FIG. 2 , the method may include the following steps:

S101, in response to sensor data sent by the anchor terminal, determining a screen sharing mode selected by the anchor terminal.

In the embodiment of the present application, the host end refers to the host end device, and the host end device is controlled by the host user. Among them, the host end device is an XR device, and the XR device can be a VR device, an AR device, or an MR device, etc., and this application does not make specific restrictions on it.

Considering that XR devices are usually used together with handheld devices, users can use handheld devices to interact with XR devices. Among them, the handheld device can be selected as a handle or a hand controller. Therefore, the anchor end device used by the anchor user in this application can be an XR device and a handheld device. Of course, if the XR device has other interactive functions, such as gesture interaction function, voice interaction function or eye tracking function, the anchor end device that the anchor user of this application can choose to use is only an XR device, and there is no specific restriction on it here.

It should be understood that the sensor data in the present application is specifically 6-DOF data.

The degree of freedom refers to the number of directions in which a user can move in a 3D space, and the number of directions is the degree of freedom. In this application, the number of directions representing the degree of freedom is 6 in total.

In this application, 6 degrees of freedom (DoF) means that the user has the ability to rotate on the X, Y and Z axes, as well as the ability to move on the X, Y and Z axes. That is, 6DoF includes: translational freedom and rotational freedom. Among them, translational freedom is divided into three types: forward/backward, up/down, left/right, and rotational freedom is divided into three types: pitch, roll, and yaw. In other words, 6 degrees of freedom can be composed of three types of translational freedom and three types of rotational freedom.

Considering that human body movements can be roughly divided into two categories: rotation and translation, and 6DoF is composed of rotational freedom and translational freedom, XR devices that support 6DoF can simulate almost all of the user's head movements.

That is to say, no matter how complex any possible movement of the object is, it can be represented by a combination of translation and rotation, that is, it can be expressed by 6DoF data.

As mentioned above, when the anchor user uses an XR device and a handheld device, the sensor data in this application is the 6DoF data of the XR device and the 6DoF data of the handheld device. When the anchor user uses an XR device, the sensor data in this application is the 6DoF data of the XR device.

Optionally, when the anchor user needs to share the anchor device screen with other users, he can first create a virtual room for screen sharing, and then invite other users as viewers into the virtual room to lay the foundation for subsequent screen sharing.

The audience users invited here are actually audience-end devices. In this embodiment, the audience-end device can be any hardware device with a display screen, such as an XR device, a personal computer, a smart phone, or a laptop computer, etc., and there is no restriction on it here.

In the embodiment of the present application, creating a virtual room includes the following situations:

In the first case, if a screen sharing client is installed on the host device, the host user can control the host device to use the screen sharing client to send an application instruction to the server to create a virtual room. Then, when the server receives the application instruction sent by the host device, it allocates a virtual room to the host device and returns the virtual room identifier to the host device. The host device then enters the virtual room based on the virtual room identifier returned by the server to complete the creation of the virtual room. Among them, the virtual room identifier refers to information that can uniquely identify the virtual room, such as the virtual room number or virtual room ID, etc., which is not limited here.

In the second case, if the screen sharing client is not installed on the host device, the host user can control the host device to use the terminal device as the screen sharing client to send an application instruction to the server to create a virtual room. Then, when the server receives the application instruction sent by the host device, it allocates a virtual room to the host device and returns the virtual room identifier to the host device. The host device then enters the virtual room based on the virtual room identifier returned by the server to complete the creation of the virtual room.

It should be understood that the above-mentioned terminal device can be selected as an intelligent terminal such as a personal computer, a smart phone or a laptop computer.

Furthermore, the host user can share the virtual room identifier with the audience devices controlled by other users based on the host device, so that the audience devices can enter the created virtual room based on the virtual room identifier; or, the host user can actively invite the audience devices controlled by other users to enter the created virtual room based on the host device.

When the host device and the audience device enter the same virtual room, the host user can control the host device to perform screen sharing operations in the virtual room.

Optionally, before starting screen sharing, the host-end device first displays a sharing mode selection interface on the host-end device screen based on the detected screen sharing instruction. The sharing mode selection interface includes: prompt information, target sharing mode controls, and normal sharing mode controls, as shown in Figure 3. Among them, the prompt information can be selected as "Please select the screen sharing mode you want." When it is detected that the host user triggers the target sharing mode control, it means that the host user has completed the setting operation of the screen sharing mode, and the host-end device closes the sharing mode selection interface. Alternatively, when the specified display time is reached and the trigger instruction of the host user is not detected, the target sharing mode is selected by default for the host user, and the sharing mode selection interface is closed.

It should be understood that the target sharing mode in the present application refers to the de-jitter sharing mode. The de-jitter sharing mode is mainly used to filter out unnatural jitters in 6DoF data to obtain smooth 6DoF data.

Furthermore, the anchor user can use the anchor-end device to upload shared content, such as speech scripts or live video data to the server according to the guidance information, so that the server can perform screen sharing operations to the audience-end devices in the same virtual room based on the shared content.

After the upload is complete, the host device can start sharing the screen. When sharing the screen, the host user adjusts the image displayed on the screen by controlling the host device. At the same time, the host device will obtain its own 6DoF data in real time and send the obtained 6DoF data to the server in real time, so that the server can perform image rendering and image sharing operations based on the 6DoF data to achieve screen sharing.

In an embodiment of the present application, when the anchor end device obtains its own 6DoF data, it can collect its own 6DoF data in real time by controlling the sensors in itself.

It should be understood that the sensor in the anchor-end device can be a sensor, such as a nine-axis sensor, or an inertial measurement unit (IMU), etc., and no specific limitation is made here.

In addition, when the host device sends the acquired 6DoF data to the server in real time, if the host device is an XR device and a handheld device, the handheld device first sends its own 6DoF data to the XR device, and then the XR device sends its own 6DoF data and the handheld device's 6DoF data to the server in a pre-specified transmission format. If the host device is an XR device, the XR device sends its own 6DoF data to the server in a pre-specified transmission format.

Among them, the pre-defined transmission format may be as follows:

Among them, TransQuaternion rotation represents a quaternion data, and TransVector3 position represents a three-dimensional coordinate data, as shown below:

Considering that in actual use, it is usually necessary to convert the quaternion into Euler angle matrix data, the present application can convert the above quaternion into Euler angle matrix data by the following formula:

Among them, q ₀ , q ₁ , q ₂ and q ₃ are quaternions, corresponding to x, y, z, w in TransQuaternion rotation respectively.

Considering that in actual use, the host user's body will move unconsciously when controlling the host-side device, and the body movement will cause the image on the host-side device screen to shake in three-dimensional space, and this shaking will cause the image to shake unnaturally. Therefore, in order to reduce or even avoid simulated motion sickness caused by image shaking, which causes dizziness and other discomfort to the user's eyes when watching the image displayed on the device, the host user usually selects the de-shaking sharing mode when sharing the screen, so that the server performs de-shaking processing based on the de-shaking sharing mode, and then renders the display screen based on the de-shaking data to obtain a stable image, and share the image with different user terminals.

Based on the above reasons, after the server receives the 6DoF data sent by the anchor device, it can determine whether the screen sharing mode selected by the anchor device is the de-jitter sharing mode or the normal sharing mode, and then perform corresponding image rendering and image sharing operations according to the determined screen sharing mode.

When the server determines the screen sharing mode selected by the host device, it can determine the screen sharing mode identifier corresponding to the host device by querying the data list. Then, according to the screen sharing mode identifier, the screen sharing mode selected by the host device is determined. The screen sharing mode identifier is used to represent the screen sharing mode selected by the host user.

For example, assuming that the screen sharing mode identifier "true" is used to identify the de-jitter sharing mode, and the screen sharing mode identifier "false" is used to identify the normal sharing mode, then when it is determined that the screen sharing mode identifier corresponding to the host-end device is "true", it can be determined that the host-end device has selected the de-jitter sharing mode. Conversely, when it is determined that the screen sharing mode identifier corresponding to the host-end device is "false", it can be determined that the host-end device has selected the normal sharing mode.

It should be noted that the simulated motion sickness mentioned above is caused by the inconsistency between the state observed by the user visually and the actual state of the body. For example, when the user is sitting or standing and controlling the movement of the character through the handle, the visual information obtained by the user is "I am moving", but the vestibular organ in the middle ear responsible for sensing the body's state sends a signal to the brain that "I am not moving". This contradictory signal will make the brain think that "itself" is in an abnormal and dangerous state. At this time, the brain will produce a strong sense of dizziness to warn the user to get out of the current state as soon as possible.

S102: If the screen sharing mode selected by the anchor terminal is the target sharing mode, de-jitter processing is performed on the sensor data to obtain optimized sensor data.

When it is determined that the screen sharing mode selected by the anchor-end device is the target sharing mode (de-jitter sharing mode), the server of this application can de-jitter the 6DoF data sent by the anchor-end device to filter out unnatural jitters in the 6DoF data and obtain stable 6DoF data, that is, optimized sensor data, to provide conditions for obtaining stable image pictures later.

Optionally, the server in the present application may use a preset de-jitter processing rule to de-jitter the 6DoF data to obtain optimized 6DoF data. Exemplarily, the 6DoF data may be smoothed using a data smoothing algorithm to filter out subtle and high-frequency unnatural jitters in the 6DoF data to obtain optimized 6DoF data.

The data smoothing algorithm may be a sliding average method, an exponential sliding average method, or a convolution smoothing algorithm (Savitzky-Golay filtering, SG filtering for short), etc., and no specific limitation is imposed on it here.

S103, rendering the display picture according to the optimized sensor data to obtain a target image picture.

It should be understood that in the present application, the target image picture may be selected as a texture image.

Optionally, the server may import the optimized 6DoF data into a rendering resource to perform a rendering operation of a specified viewport image. Alternatively, the server may also import the optimized 6DoF data into a cloud rendering server to perform a rendering operation of a specified viewport image, etc., which is not limited here.

Usually, the image displayed in the virtual space is a spherical model. The user is equivalent to standing at the center of the sphere and looking outward. Due to the limited viewing angle of the human eye, the user can only see a small part of the 360-degree sphere at the same time. When the user rotates the viewing angle, the other parts of the sphere can be seen, as shown in Figure 4. Among them, the part of the 360-degree sphere that the user sees at the same time is the viewport, and the image displayed on the viewport is the viewport image.

It should be understood that since 6DoF data can determine the position and angle of the anchor user's line of sight, the server can determine the corresponding viewport, i.e., the designated viewport, based on the line of sight position and angle determined by the 6DoF data. Thus, the server can perform image rendering processing on the display screen corresponding to the designated viewport to obtain the designated viewport image screen.

In some possible implementations, the server of the present application renders the optimized 6DoF data, and when obtaining the target image, the optimized 6DoF data can be imported according to the import rules of the interface by calling the application interface opened by the Steamvr platform, such as the openvr interface. Then, the Steamvr platform renders the display screen of the corresponding viewport according to the imported optimized 6DoF data to obtain the texture image of the corresponding viewport.

The DriverPose_t data structure provided by the above openvr interface includes the following information:

Among them, the imported optimized 6DoF data is mainly imported into the following fields in the above data structure:

double vecPosition[3];

double vecVelocity[3];

double vecAcceleration[3];

vr::HmdQuaternion_t qRotation;

double vecAngularVelocity[3];

double vecAngularAcceleration[3];

In addition, after importing the optimized 6DoF data into the openvr interface, it is necessary to convert the optimized 6DoF data (Rotation and Position) into a rotation matrix in the SubmitLayer interface of openvr.

In this application, the rotation matrix can be obtained by calling the general function glm::quat_cast(rotationMatrix).

S104, sending the target image to the host terminal and the audience terminal, so that the host terminal and the audience terminal display the target image.

Optionally, the server of the present application may encode and package the rendered target image, and then synchronously forward the packaged data to the host-end device and the audience-end device in the same virtual room through the SFU (Selective Forwarding Unit) data forwarding method, so that after the host-end device and the audience-end device receive the packaged data sent by the server, they decode and render the packaged data to display a stable image on the host-end device and the audience-end device, thereby reducing or even avoiding discomfort such as dizziness caused by image jitter to the human eye based on the stable image.

In the embodiment of the present application, SFU is a server program that routes and forwards audio and video data streams of Web Real-Time Communication (WebRTC) clients. The core function of the SFU server is to establish a link with each WebRTC Peer client, receive audio and video data from them respectively, and implement one-to-many capabilities (i.e. forwarding a client's stream to other WebRTC Peer clients).

In another implementation scenario of the present application, if after receiving the 6DoF data sent by the host end device, it is determined that the screen sharing mode selected by the host end device is the normal sharing mode, then the server renders and processes the received 6DoF data according to the existing screen sharing method to obtain the target image, and sends the target image to the host end device and the viewer end device.

It is understandable that this application adds a de-jitter sharing mode on the basis of the existing screen sharing solution, and uses the de-jitter sharing mode to smooth the 6DoF data uploaded by the anchor end device with unnatural jitter, so as to filter out the subtle and high-frequency jitter in the 6DoF data and obtain stable 6DoF data. Furthermore, after rendering the display screen based on the stable 6DoF data, a stable image screen can be obtained, so that when the anchor user and the audience user watch the stable image screen displayed by their respective devices, it can reduce or even avoid simulated motion sickness caused by image jitter, which causes dizziness and other discomfort when the user watches the image screen displayed by the device.

The screen sharing method provided in the embodiment of the present application determines the screen sharing mode selected by the host end when receiving the sensor data sent by the host end. If it is determined that the screen sharing mode selected by the host end is the target sharing mode, the sensor data is de-jittered to obtain the optimized sensor data, and based on the optimized sensor data, the display screen is rendered to obtain the target image screen, and then the target image screen is sent to the host end and the audience end, so that the host end and the audience end display the target image screen. The present application determines the screen sharing mode selected by the host end during screen sharing, and de-jitters the sensor data when the screen sharing mode is the target sharing mode, so as to filter out the subtle and high-frequency unnatural jitter in the unstable sensor data, and obtain stable sensor data. Then, when the display screen is rendered based on the stable sensor data, a stable image screen can be rendered, so that the discomfort such as dizziness caused by image jitter to the human eye can be reduced or even avoided based on the stable image screen. In addition, when the audience end and/or the host end is an XR device that can provide a virtual space, the present application can not only reduce or eliminate the user's dizziness, but also extend the time the user is immersed in the virtual space and increase the user's reuse rate.

From the above introduction, it can be seen that the present application performs de-jitter processing on the sensor data according to the de-jitter sharing mode selected by the anchor end, and then renders a stable target image based on the optimized sensor data, thereby reducing the jitter phenomenon in the rendered image, thereby reducing or even avoiding the user's dizziness caused by image jitter.

Based on the above embodiment, the de-jittering process of 6DoF data in the present application is further explained in conjunction with FIG. 5 .

As shown in FIG5 , the method may include the following steps:

S201, in response to sensor data sent by the host end, determining a screen sharing mode selected by the host end.

S202: If the screen sharing mode selected by the anchor end is the target sharing mode, a target smoothing algorithm is obtained.

S203, determining a target threshold range corresponding to the target smoothing algorithm.

The target smoothing algorithm in the present application can be selected as any of the conventional data smoothing algorithms, such as the sliding average method, the exponential sliding average method or the convolution smoothing algorithm (Savitzky-Golay filtering, referred to as SG filtering), etc.

That is, the server can randomly select an algorithm from all configured regular data smoothing algorithms as the target smoothing algorithm.

Considering that different smoothing algorithms have different threshold ranges based on which they perform smoothing processing, after the server selects a target smoothing algorithm from conventional data smoothing algorithms, it is necessary to determine a target threshold range corresponding to the target smoothing algorithm.

Optionally, the present application can determine the target threshold range corresponding to the target smoothing algorithm by querying the correspondence between the smoothing algorithm and the threshold range. Among them, in the correspondence between the smoothing algorithm and the threshold range, the threshold range can be a default threshold range of the smoothing algorithm, or it can also be a threshold range customized by the user, such as the manufacturer, based on the de-jitter accuracy, and there is no specific restriction on it here.

S204, performing de-jitter processing on the sensor data according to the target smoothing algorithm and the target threshold range to obtain optimized sensor data.

Optionally, the server of this application first compares the received 6DoF data with the target threshold range to determine whether the 6DoF data is within the target threshold range. If it is determined that the 6DoF data is within the target threshold range, it means that the 6DoF data has unnatural jitter, and the 6DoF data needs to be optimized. If it is determined that the 6DoF data is not within the target threshold range, it means that the 6DoF data is stable data, and there is no need to optimize the 6DoF data, so as to keep the viewport position unchanged.

When the present application performs de-jitter processing on 6DoF data with unnatural jitter, an average value is first calculated based on the target threshold range, and then the average value is used as a smoothing value, so that the server can use a target smoothing algorithm based on the smoothing value to de-jitter the 6DoF data with unnatural jitter and obtain stable 6DoF data.

When de-jittering 6DoF data with unnatural jitter is processed based on the smoothing value using the target smoothing algorithm, the magnitude relationship between the 6DoF data and the smoothing value may be first determined, and if the 6DoF data is greater than the smoothing value, the 6DoF data is adjusted to a first value. If the 6DoF data is less than or equal to the smoothing value, the 6DoF data is adjusted to a second value.

The first value can be selected as 1, and the second value can be selected as 0.

For example, assuming that the target threshold range is [a, b] and the 6DoF data is k, when k is not located at [a, b], the server calculates the average value based on [a, b] as: Furthermore, compare k with/> When k is greater than/> When k is less than / > , the 6DoF data is adjusted to 0, as shown in FIG6 .

That is to say, when the present application performs de-jittering on the 6DoF data based on the smoothing value calculated based on the target threshold range, the 6DoF data greater than the smoothing value is adjusted to a first value, and the 6DoF data less than or equal to the smoothing value is adjusted to a second value, so that the optimized 6DoF data can maintain the same direction, rather than the data effect of adjusting back and forth in the positive and negative directions, so that the viewport image can be kept within a stable range as much as possible, thus reducing or even avoiding the dizziness of the user's eyes caused by image jitter due to small-range motion jitter of the user.

S205, rendering the display picture according to the optimized sensor data to obtain a target image picture.

S206, sending the target image to the host terminal and the audience terminal, so that the host terminal and the audience terminal display the target image.

The screen sharing method provided in the embodiment of the present application determines the screen sharing mode selected by the host end when receiving the sensor data sent by the host end. If it is determined that the screen sharing mode selected by the host end is the target sharing mode, the sensor data is de-jittered to obtain the optimized sensor data, and based on the optimized sensor data, the display screen is rendered to obtain the target image screen, and then the target image screen is sent to the host end and the audience end, so that the host end and the audience end display the target image screen. The present application determines the screen sharing mode selected by the host end during screen sharing, and de-jitters the sensor data when the screen sharing mode is the target sharing mode, so as to filter out the subtle and high-frequency unnatural jitter in the unstable sensor data, and obtain stable sensor data. Then, when the display screen is rendered based on the stable sensor data, a stable image screen can be rendered, so that the discomfort such as dizziness caused by image jitter to the human eye can be reduced or even avoided based on the stable image screen. In addition, when the audience end and/or the host end is an XR device that can provide a virtual space, the present application can not only reduce or eliminate the user's dizziness, but also extend the time the user is immersed in the virtual space and increase the user's reuse rate.

A screen sharing device provided in an embodiment of the present application is described in detail below with reference to FIG7 . FIG7 is a schematic block diagram of a screen sharing device provided in an embodiment of the present application.

As shown in FIG. 7 , the screen sharing device 300 includes: a mode determination module 310 , a de-jitter processing module 320 , an image rendering module 330 and an image distribution module 340 .

The mode determination module 310 is used to determine the screen sharing mode selected by the host terminal in response to the sensor data sent by the host terminal;

A de-jitter processing module 320 is used to de-jitter the sensor data to obtain optimized sensor data if the screen sharing mode selected by the anchor terminal is the target sharing mode;

An image rendering module 330 is used to render a display image according to the optimized sensor data to obtain a target image;

The image distribution module 340 is used to send the target image to the host terminal and the audience terminal so that the host terminal and the audience terminal display the target image.

In an optional implementation of the embodiment of the present application, the de-jitter processing module 320 includes:

An algorithm acquisition unit, which acquires a target smoothing algorithm;

A range determination unit, used to determine a target threshold range corresponding to the target smoothing algorithm;

A de-jitter unit is used to perform de-jitter processing on the sensor data according to the target smoothing algorithm and the target threshold range to obtain optimized sensor data.

In an optional implementation of the embodiment of the present application, the de-jitter unit is specifically used to:

determining whether the sensor data is within the target threshold range;

If it is within the target threshold range, determining a smoothing value according to the target threshold range;

The target smoothing algorithm is controlled to utilize the smoothing value to perform a de-jittering process on the sensor data to obtain optimized sensor data.

In an optional implementation of the embodiment of the present application, the de-jitter unit is further configured to:

An average value is determined according to the target threshold range, and a smoothing value is determined by using the average value.

In an optional implementation of the embodiment of the present application, the de-jitter unit is further configured to:

If the sensor data is greater than the smoothed value, adjusting the sensor data to a first value;

If the sensor data is less than or equal to the smoothed value, the sensor data is adjusted to a second value.

In an optional implementation of the embodiment of the present application, the mode determination module 310 is specifically configured to:

Determining the screen sharing mode selected by the host terminal according to the recorded screen sharing mode identifier;

The screen sharing mode identifier is used to represent the screen sharing mode selected by the anchor terminal.

In an optional implementation of the embodiment of the present application, the sensor data is 6-DOF data.

The screen sharing device provided in the embodiment of the present application determines the screen sharing mode selected by the host end when receiving the sensor data sent by the host end. If it is determined that the screen sharing mode selected by the host end is the target sharing mode, the sensor data is de-jittered to obtain the optimized sensor data, and based on the optimized sensor data, the display screen is rendered to obtain the target image screen, and then the target image screen is sent to the host end and the audience end, so that the host end and the audience end display the target image screen. The present application determines the screen sharing mode selected by the host end during screen sharing, and de-jitters the sensor data when the screen sharing mode is the target sharing mode, so as to filter out the subtle and high-frequency unnatural jitter in the unstable sensor data, and obtain stable sensor data. Then, when the display screen is rendered based on the stable sensor data, a stable image screen can be rendered, so that the discomfort such as dizziness caused to the human eye by image jitter can be reduced or even avoided based on the stable image screen. In addition, when the audience end and/or the host end is an XR device that can provide a virtual space, the present application can not only reduce or eliminate the user's dizziness, but also prolong the time the user is immersed in the virtual space and improve the user's reuse rate.

It should be understood that the device embodiment and the aforementioned method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the device 300 shown in FIG7 can execute the method embodiment corresponding to FIG2, and the aforementioned and other operations and/or functions of each module in the device 300 are respectively for implementing the corresponding processes in each method in FIG2, and for the sake of brevity, no further description is given here.

In the above, the device 300 of the embodiment of the present application is described from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the first aspect method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and/or software form instructions in the processor, and the steps of the first aspect method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above-mentioned first aspect method embodiment in conjunction with its hardware.

FIG8 is a schematic block diagram of an electronic device provided in an embodiment of the present application. As shown in FIG8 , the electronic device 800 may include:

The memory 410 and the processor 420, the memory 410 is used to store the computer program and transmit the program code to the processor 420. In other words, the processor 420 can call and run the computer program from the memory 410 to implement the screen sharing method in the embodiment of the present application.

For example, the processor 420 may be configured to execute the above screen sharing method embodiment according to instructions in the computer program.

In some embodiments of the present application, the processor 420 may include but is not limited to:

General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

In some embodiments of the present application, the memory 410 includes but is not limited to:

Volatile memory and/or non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 410 and executed by the processor 420 to complete the screen sharing method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

As shown in FIG8 , the electronic device 400 may further include:

The transceiver 430 may be connected to the processor 420 or the memory 410 .

The processor 420 may control the transceiver 430 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 430 may include a transmitter and a receiver. The transceiver 430 may further include an antenna, and the number of antennas may be one or more.

It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

The present application also provides a computer storage medium on which a computer program is stored. When the computer program is executed by a computer, the computer is enabled to execute the method of the above method embodiment.

An embodiment of the present application also provides a computer program product including program instructions. When the program instructions are executed on an electronic device, the electronic device executes the method of the above method embodiment.

When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (11)

1. A screen sharing method, comprising:

responding to sensor data sent by a host, and determining a screen sharing mode selected by the host;

If the screen sharing mode selected by the anchor terminal is a target sharing mode, performing debouncing processing on the sensor data to obtain optimized sensor data;

Rendering the display picture according to the optimized sensor data to obtain a target image picture;

And sending the target image frames to the anchor side and the audience side so that the anchor side and the audience side display the target image frames.

2. The method of claim 1, wherein the debouncing the sensor data to obtain optimized sensor data comprises:

acquiring a target smoothing algorithm;

determining a target threshold range corresponding to the target smoothing algorithm;

And performing debouncing processing on the sensor data according to the target smoothing algorithm and the target threshold range to obtain optimized sensor data.

3. The method according to claim 2, wherein the performing a de-dithering process on the sensor data according to the target smoothing algorithm and the target threshold range to obtain optimized sensor data includes:

Determining whether the sensor data is within the target threshold range;

if the value is within the target threshold range, determining a smooth value according to the target threshold range;

And controlling the target smoothing algorithm to perform debouncing processing on the sensor data by using the smoothing value to obtain optimized sensor data.

4. A method according to claim 3, wherein determining a smoothed value from the target threshold range comprises:

and determining an average value according to the target threshold range, and determining a smooth value from the average value.

5. A method according to claim 3, wherein controlling the target smoothing algorithm to perform a de-dithering process on the sensor data using the smoothed value to obtain optimized sensor data comprises:

If the sensor data is greater than the smoothed value, adjusting the sensor data to a first value;

and if the sensor data is less than or equal to the smoothed value, adjusting the sensor data to a second value.

6. The method of claim 1, wherein the determining the screen sharing mode selected by the anchor comprises:

determining a screen sharing mode selected by the anchor terminal according to the recorded screen sharing mode identification;

Wherein the screen sharing mode identification is used for representing the screen sharing mode selected by the anchor.

7. The method of any one of claims 1-6, wherein the sensor data is 6 degrees of freedom data.

8. A screen sharing apparatus, comprising:

The mode determining module is used for responding to the sensor data sent by the anchor terminal and determining a screen sharing mode selected by the anchor terminal;

the debouncing processing module is used for debouncing the sensor data to obtain optimized sensor data if the screen sharing mode selected by the anchor terminal is a target sharing mode;

the image rendering module is used for rendering the display picture according to the optimized sensor data to obtain a target image picture;

And the image distribution module is used for sending the target image frames to the anchor side and the audience side so as to enable the anchor side and the audience side to display the target image frames.

9. An electronic device, comprising:

A processor and a memory for storing a computer program, the processor for invoking and running the computer program stored in the memory to perform the screen sharing method of any of claims 1 to 7.

10. A computer-readable storage medium storing a computer program that causes a computer to execute the screen sharing method according to any one of claims 1 to 7.

11. A computer program product comprising program instructions which, when run on an electronic device, cause the electronic device to perform the screen sharing method of any of claims 1 to 7.