CN118433367A - Image processing method, device, medium, head mounted display device and program product - Google Patents

Abstract

The application provides an image processing method, an image processing device, a medium, a head-mounted display device and a program product, wherein the method is applied to the head-mounted display device and comprises the following steps: collecting camera data and inertial data; transmitting the camera data and the inertial data to a computing device; acquiring rendering image coding data, wherein the rendering image coding data is obtained by a computing device determining a rendering image of the head-mounted display device according to the camera data and the inertia data and coding the rendering image; acquiring preset display frame rate, current inertial data and display time information corresponding to a screen display image corresponding to a previously determined historical rendering image; and displaying a target screen display image corresponding to the rendered image according to the rendered image coding data, the preset display frame rate, the current inertia data and the display time information, so that the calculated amount of the head-mounted display device is reduced, the sizes and the weights of the radiator and the battery are further reduced, and the miniaturization and the light weight of the head-mounted display device are facilitated.

Description

Image processing method, device, medium, head mounted display device and program product

Technical Field

The present invention relates to the field of computer technology, and more specifically, to an image processing method, apparatus, medium, device and program product.

Background technique

Head-mounted display (HMD), also known as headset, sends optical signals to the eyes through various headsets to achieve different effects such as virtual reality (VR), augmented reality (AR), and mixed reality (MR).

At present, the miniaturization and lightweighting of head-mounted displays are the key research directions. However, since head-mounted displays need to perform a lot of calculations, they have high requirements for the computing power of the processor, the heat dissipation capacity of the radiator, and the battery life. As a result, the structural space occupied by the radiator and battery is large, which is not conducive to the miniaturization and lightweighting of head-mounted displays.

Summary of the invention

The embodiments of the present application provide an image processing method, apparatus, medium, head-mounted display device, and program product, which can reduce the amount of calculation of the head-mounted display device, thereby reducing the power consumption of the head-mounted display device, and further reducing the size and weight of the radiator and battery, which is conducive to the miniaturization and lightweight of the head-mounted display device.

In a first aspect, an embodiment of the present application provides an image processing method, which is applied to a head mounted display device, and the method includes:

Collect camera data and inertial data;

sending the camera data and the inertial data to a computing device;

Acquire rendered image encoding data, where the rendered image encoding data is obtained by the computing device determining a rendered image of the head mounted display device according to the camera data and the inertial data, and encoding the rendered image;

Obtaining a preset display frame rate, current inertial data, and display time information corresponding to a screen display image corresponding to a previously determined historical rendering image;

A target screen display image corresponding to the rendered image is displayed according to the rendered image encoding data, the preset display frame rate, the current inertia data, and the display time information.

In a second aspect, an embodiment of the present application provides an image processing method, which is applied to a computing device, and the method includes:

Get camera data and inertial data of the head-mounted display device;

Determining a rendered image of the head mounted display device based on the camera data and the inertial data;

The rendered image is encoded to obtain rendered image encoded data, and the rendered image encoded data is sent to the head-mounted display device, so that the head-mounted display device obtains a preset display frame rate, current inertia data, and display time information corresponding to a screen display image corresponding to a historical rendered image determined last time, and displays a target screen display image corresponding to the rendered image according to the rendered image encoded data, the preset display frame rate, the current inertia data, and the display time information.

In a third aspect, an embodiment of the present application provides an image processing device, which is applied to a head mounted display device, comprising:

Acquisition module, used to collect camera data and inertial data;

A sending module, used for sending the camera data and the inertial data to a computing device;

A first acquisition module is used to acquire rendered image encoding data, where the rendered image encoding data is obtained by the computing device determining a rendered image of the head mounted display device according to the camera data and the inertial data, and encoding the rendered image;

A second acquisition module is used to acquire a preset display frame rate, current inertial data, and display time information corresponding to a screen display image corresponding to a previously determined historical rendering image;

The display module is used to display a target screen display image corresponding to the rendered image according to the rendered image encoding data, the preset display frame rate, the current inertia data, and the display time information.

In a fourth aspect, an embodiment of the present application provides an image processing device, applied to a computing device, comprising:

An acquisition module is used to acquire camera data and inertial data of a head mounted display device;

A determination module, configured to determine a rendered image of the head mounted display device according to the camera data and the inertial data;

The encoding module is used to encode the rendered image to obtain rendered image encoded data, and send the rendered image encoded data to the head-mounted display device, so that the head-mounted display device obtains a preset display frame rate, current inertia data, and display time information corresponding to a screen display image corresponding to a historical rendered image determined last time, and displays a target screen display image corresponding to the rendered image according to the rendered image encoded data, the preset display frame rate, the current inertia data, and the display time information.

In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is suitable for a processor to load to execute the image processing method as described in the first aspect above, or to execute the image processing method as described in the second aspect above.

In a sixth aspect, an embodiment of the present application provides a head-mounted display device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor executes the image processing method described in the first aspect above by calling the computer program stored in the memory.

In the seventh aspect, an embodiment of the present application provides a computing device, which includes a processor and a memory, wherein the memory stores a computer program, and the processor executes the image processing method described in the second aspect above by calling the computer program stored in the memory.

In an eighth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the image processing method described in the first aspect above, or implements the image processing method described in the second aspect above.

The embodiments of the present application provide an image processing method, apparatus, medium, head-mounted display device and program product, wherein the method is applied to a head-mounted display device, and comprises: collecting camera data and inertial data; sending the camera data and the inertial data to a computing device; obtaining rendering image coding data, wherein the rendering image coding data is obtained by the computing device determining a rendering image of the head-mounted display device according to the camera data and the inertial data, and encoding the rendering image; obtaining a preset display frame rate, current inertial data, and display time information corresponding to a screen display image corresponding to a historical rendering image determined previously; and displaying a target screen display image corresponding to the rendering image according to the rendering image coding data, the preset display frame rate, the current inertial data, and the display time information. In the present application, after the head-mounted display device collects camera data and inertial data, the camera data and inertial data are sent to a computing device, and the computing device completes tracking calculations and image rendering. The head-mounted display device only needs to obtain the rendered image and then predict the target screen display image and display the target screen display image without performing tracking calculations and image rendering, thereby reducing the amount of calculation of the head-mounted display device and reducing the power consumption of the head-mounted display device, thereby reducing the size and weight of the radiator and battery, which is conducive to the miniaturization and lightweight of the head-mounted display device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of an image processing method provided in an embodiment of the present application.

FIG. 2 is a schematic diagram of a first application scenario of the image processing method provided in an embodiment of the present application.

FIG3 is a schematic diagram of a second application scenario of the image processing method provided in an embodiment of the present application.

FIG. 4 is a schematic diagram of a third application scenario of the image processing method provided in an embodiment of the present application.

FIG. 5 is a schematic diagram of a fourth application scenario of the image processing method provided in an embodiment of the present application.

FIG. 6 is another schematic flow chart of the image processing method provided in an embodiment of the present application.

FIG. 7 is a schematic diagram of a fifth application scenario of the image processing method provided in an embodiment of the present application.

FIG. 8 is a schematic block diagram of an image processing device provided in an embodiment of the present application.

FIG. 9 is a schematic block diagram of an image processing device provided in an embodiment of the present application.

FIG. 10 is a schematic block diagram of a head mounted display device provided in an embodiment of the present application.

Detailed ways

The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. For the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

Unless otherwise defined, the technical terms or scientific terms used in this application should be the common sense understood by people of ordinary skill in the field to which this disclosure belongs. "First", "second" and similar words used in this disclosure are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiment of the embodiment of the present application described here can be implemented in a sequence other than those illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

First, some terms in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

Virtual Reality (VR), as the name implies, is the combination of virtuality and reality. Theoretically, VR technology is a computer simulation system that can create and experience a virtual world. It uses computers to generate a simulated environment and immerse users in it. Virtual reality technology uses real-life data, electronic signals generated by computer technology, and combines them with various output devices to transform them into phenomena that people can feel. These phenomena can be real objects in reality or substances that are invisible to our naked eyes, expressed through three-dimensional models. Because these phenomena are not what we can see directly, but are simulated in the real world through computer technology, they are called virtual reality.

Augmented Reality (AR) technology is a technology that cleverly integrates virtual information with the real world. It widely uses a variety of technical means such as multimedia, three-dimensional modeling, real-time tracking and registration, intelligent interaction, and sensing. It simulates computer-generated virtual information such as text, images, three-dimensional models, music, and videos, and applies them to the real world. The two types of information complement each other, thereby achieving "enhancement" of the real world.

Mixed Reality (MR) is a further development of virtual reality technology. This technology presents virtual scene information in real scenes, builds an interactive feedback information loop between the real world, the virtual world and the user, so as to enhance the realism of the user experience.

SLAM (simultaneous localization and mapping) is a commonly used technology in the field of autonomous driving and augmented reality. It mainly studies the perception and positioning of devices in unknown environments through various sensors.

IMU (Inertial Measurement Unit) is a sensor mainly used to detect and measure acceleration and rotational motion. IMU sensors can include a combination of one or more of accelerometers, gyroscopes, and magnetometers. Imagine a Cartesian coordinate system with an x-axis, y-axis, and z-axis. IMU can measure linear motion in each axis direction, as well as rotational motion around each axis.

6DoF tracking: Six degrees of freedom tracking. Objects can move in three-dimensional space with six degrees of freedom (6DOF). The six angles are (1) forward/backward, (2) up/down, (3) left/right, (4) yaw, (5) pitch, and (6) roll. Using a VR system that allows 6DOF, you can move freely in a limited space, which allows users to take full advantage of all 6 degrees of freedom: yaw, pitch, roll, forward/backward, up/down, and left/right. This makes the field of view more realistic and real.

Frame rate is the frequency (rate) at which bitmap images in units of frames appear continuously on the display.

With the rapid development of virtual reality technology, virtual reality head-mounted display technology has also developed rapidly. For example, there have been great technical improvements in tracking technologies such as head-mounted display device tracking, eye tracking, and facial tracking. The optical aspect is also developing rapidly. The miniaturization and lightweight of head-mounted display devices are the key directions for the next development. The miniaturization and lightweight of head-mounted display devices mean that the head-mounted display devices are required to have smaller size and lighter weight. However, since the current head-mounted display devices need to perform tracking calculations such as head-mounted display device tracking, eye tracking, expression tracking, gesture tracking, and image rendering, among which the tracking algorithm and image rendering both require strong processor computing power, resulting in high power consumption during the use of head-mounted display devices, which makes the current head-mounted display device battery and radiator larger in size, resulting in large size and weight of head-mounted display devices, which is not conducive to the miniaturization and lightweight development of head-mounted display devices. To solve this problem, the embodiments of the present application provide an image processing method, apparatus, medium, head-mounted display device and program product. By having a head-mounted display device responsible for data acquisition and a computing device responsible for tracking and rendering, the computing workload of the head-mounted display device is reduced to reduce the power consumption of the head-mounted display device, thereby reducing the size and weight of the battery and the radiator, and further reducing the size and weight of the head-mounted display device, which is conducive to the miniaturization and lightweight development of the head-mounted display device.

It should be noted that the order of description of the following embodiments is not intended to limit the priority order of the embodiments.

Please refer to FIG. 1 , which shows a schematic flow chart of an image processing method described in an embodiment of the present application. The image processing method can be applied to a head mounted display device. The method mainly includes steps S101 to S105, which are described as follows:

Step S101, collecting camera data and inertial data,

Step S102, sending the camera data and inertial data to a computing device.

The head-mounted display device may be a head-mounted display of virtual reality, augmented reality or mixed reality, and the computing device may be a server, such as a local server or a cloud server, or may be a computer, a smart phone or other device. The head-mounted display device and the computing device may be connected via a network. Referring to FIG2 , the head-mounted display device may include a SLAM tracking camera, a mouth expression tracking camera, an MR camera and an eye tracking camera, wherein the SLAM tracking camera may be used to collect images of the external environment, the mouth expression tracking camera may be used to collect images of the user's mouth expression, the MR camera may be used to collect images of the external environment, and the eye tracking camera may be used to collect images of the user's eye expression. The head-mounted display device may also include an IMU sensor, which is used to collect the user's head-mounted display device motion data.

The head mounted display device motion data may include the position data of the head mounted display device, and the position data may include the position information along the three rectangular coordinate axes of X, Y, and Z, and the angle information Pitch, Yaw, and Roll rotating around the three rectangular coordinate axes of X, Y, and Z, wherein Pitch is the pitch angle rotating around the X axis, Yaw is the yaw angle rotating around the Y axis, and Roll is the roll angle rotating around the Z axis. Usually, the position information along the three rectangular coordinate axes of X, Y, and Z and the angle information Pitch, Yaw, and Roll rotating around the three rectangular coordinate axes of X, Y, and Z are collectively referred to as six degrees of freedom information.

In some embodiments, the head-mounted display device includes multiple cameras, and the method may also include: adjusting the exposure time of the multiple cameras to align the exposure center points of the multiple cameras; and determining the timestamp data of the camera data based on the exposure center points of the multiple cameras.

Please refer to Figure 3. It is easy to understand that the ambient light brightness corresponding to each camera is different, so the exposure time needs to be adjusted when shooting. Different cameras can have different exposure time periods. In order to improve the accuracy of data alignment, the exposure center point of each camera can be selected as the basis for adding the system timestamp. For SLAM, the exposure time is not concerned, but a unified moment is required to process the data. That is, the moment when the image is generated needs to be aligned.

Specifically, a timestamp is usually a character sequence that uniquely identifies a certain moment in time. Since the head mounted display device collects camera data and inertial data in a continuous real-time process, each data can be uniquely identified by a timestamp.

In this embodiment, the head mounted display device includes an inertial sensor, and the method may further include: determining a timestamp of the inertial data according to an interruption time of the inertial sensor.

When IMU collects data, an interrupt will be generated for each data collected. The interrupt time point of IMU can be selected as the basis for adding system timestamp to improve tracking accuracy.

Specifically, as shown in Figure 4, the head-mounted display device and the computing device may both include wireless modules (a first wireless module, a second wireless module), and the two are connected through a wireless module network. The wireless module may be a 60GHz Wi-Fi (wireless network communication technology) module, a UWB (UltraWideBand) module, a 5G module, or other high-bandwidth, low-latency transmission module.

Among them, 5G is the fifth generation of mobile communication technology. In order to meet the increased demand for wireless data services since the deployment of the 4G communication system, efforts have been made to develop an improved 5G communication system. 5G has faster transmission speeds, greater transmission capacity and extremely low latency, which reduces the time for data transmission and thus reduces latency. In the field of virtual reality, in order for users to have a better user experience, the latency needs to be as small as possible, otherwise there will be symptoms such as dizziness.

Specifically, as shown in FIG2 , the head-mounted display device may further include a processor, which can synthesize and encode images after acquiring the camera data collected by the camera. For example, the environmental images captured by four SLAM tracking cameras can be synthesized into one environmental image, the images captured by two eye tracking cameras and a mouth expression tracking camera can be synthesized into one image, and the images captured by multiple MR cameras can be synthesized into one image. Then, the synthesized composite image is image encoded to obtain camera image encoding data, and the camera encoding data and IMU sensor data are sent to the computing device. Among them, the composite image can be encoded into an H.264 code stream, or Moving Picture Experts Group-2 (MPEG-2), source coding standard (Audio Video Coding Standard, AVS), etc.

For example, referring to FIG4 , the processor may include a SLAM image acquisition module, an eye expression image acquisition module, an MR image acquisition module, an inertial measurement module, a first image encoding module, and a first image decoding module, wherein the SLAM image acquisition module may be used to process data collected by a SLAM tracking camera, the eye expression image acquisition module may be used to process data collected by an eye tracking camera, the MR image acquisition module may be used to process data collected by an MR camera, the inertial measurement module may be used to process data acquired by an inertial sensor, the first image encoding module may be used to encode images, and the first image decoding module may be used to decode images. The processor may also include an optical module, which may be used to magnify different images to different distances. The difference between object distance and image distance is achieved through the optical module, and the optical module is equivalent to an optical lens.

In some embodiments, step S101 may mainly include: collecting camera data according to a first preset collection frame rate of a camera of a head-mounted display device; collecting inertial data according to a second preset collection frame rate, wherein the second preset collection frame rate is greater than the first preset collection frame rate of the camera of the head-mounted display device.

Among them, the second preset acquisition frame rate is a frame rate much higher than the preset display frame rate of the head-mounted display device, and the preset display frame rate of the head-mounted display device is greater than the first preset acquisition frame rate of the camera of the head-mounted display device. For example, the second preset acquisition frame rate may be 1KHz, and the preset display frame rate of the head-mounted display device may be 72Hz. Thus, at the subsequent display moment of the head-mounted display device, the current position of the head-mounted display device can be predicted based on the previous multiple frames of inertial data. Specifically, the higher the first preset acquisition frame rate of the inertial data, the higher the accuracy of the position tracking and prediction of the head-mounted display device.

It is easy to understand that the current head-mounted display device can simultaneously support three preset display frame rates of 72Hz, 90Hz and 120Hz. However, the shooting frame rate of the camera may be only 30Hz, and the highest is only 60Hz. In order to meet the display effect of the head-mounted display device, it is not possible to render only 30 frames or 60 frames. Therefore, the inertial data can be collected according to the second preset acquisition frame rate and sent to the computing device. The computing device can perform interpolation rendering of the time axis according to the collected inertial data and camera data. For example, if the preset display frame rate of the head-mounted display device is 120Hz, the first preset acquisition frame rate of the camera is 60Hz, and the first preset acquisition frame rate of the inertial data is 1KHz, then 1000 frames of inertial data and 60 frames of camera data will be collected. At this time, the computing device can perform interpolation rendering in the order of the time axis, that is, render 120 frames of images.

It is easy to understand that interpolation rendering can also be performed by the processor of the head-mounted display device. The head-mounted display device can cache the collected inertial data. If the number of frames of the acquired rendered image does not match the frame rate of the head-mounted display device, interpolation rendering is performed according to the cached inertial data in the timeline order.

Step S103, obtaining the rendered image encoding data, where the rendered image encoding data is obtained by the computing device determining the rendered image of the head mounted display device according to the camera data and the inertial data, and encoding the rendered image.

Specifically, the head-mounted display device sends the camera data and inertial data to the computing device, which can implement 6DoF tracking, eye and expression tracking, gesture tracking, etc. of the head-mounted display device based on the camera data and inertial data, and render the display images of the left and right eyes in real time based on the calculation results, and then encode the rendered images and send them to the head-mounted display device.

In some embodiments, the method may further include: if the rendered image encoding data is not obtained, displaying a target screen display image corresponding to the rendered image according to the previous frame image encoding data, inertial data corresponding to the previous frame image encoding data, and current inertial data.

The previous frame image encoding data may be the rendered image encoding data acquired last time.

It is easy to understand that since the head-mounted display device requires the computing device to send the rendered image encoding data, wireless interference, computing device abnormalities, insufficient rendering performance, long rendering time, etc. may occur, resulting in image frame loss. At this time, the head-mounted display device will display the same image as the previous frame. However, if the user's head turns, the light of the same image frame will fall on different positions on the user's retina, causing the seen picture to shake. Therefore, the head-mounted display device can predict the target screen display image of the frame based on the previous frame rendered image and inertial data, that is, predict the head-mounted display device movement change based on the inertial data, and then spatially shift and distort the previous frame rendered image to obtain the target screen display image of the frame, so as to cooperate with the head-mounted display device movement change, reduce picture jitter, avoid user dizziness caused by picture jitter, and improve user experience.

Step S104, obtaining a preset display frame rate, current inertia data, and display time information corresponding to a screen display image corresponding to a previously determined historical rendering image.

Step S105 , displaying a target screen display image corresponding to the rendered image according to the rendered image encoding data, the preset display frame rate, the current inertia data, and the display time information.

In step S105, displaying a target screen display image corresponding to the rendered image according to the rendered image encoding data, the preset display frame rate, the current inertia data, and the display time information includes:

Determine a rendered image and rendering time information corresponding to the rendered image according to the rendered image encoding data;

Determine target display time information according to the preset display frame rate and the display time information;

The target screen display image corresponding to the rendered image is displayed according to the rendering time information corresponding to the rendered image, the target display time information and the current inertial data.

In some optional embodiments, determining the target display time information according to the preset display frame rate and the display time information includes:

Determining the target time interval information based on the preset display frame rate, wherein the target time interval information is the reciprocal of the preset display frame rate;

The sum of the display time information and the target time interval information is determined as the target display time information.

Specifically, after the head-mounted display device obtains the rendered image coded data, first, the rendered image coded data can be decoded to obtain the rendered image and the rendering time information corresponding to the rendered image, and the rendering time information is used to indicate the rendering time of the rendered image, and the rendering time information can be a rendering timestamp. It is easy to understand that since the rendered image coded data is sent from the computing device to the head-mounted display device, there is a certain delay, and the head-mounted display device will not display the rendered image code immediately after obtaining it, and there is also a certain delay. In order to eliminate the delay, in the subsequent steps, the head-mounted display device can determine the rendered image corresponding to the target display time information based on the rendering time information, the target display time information and the measured current inertial data, and then display it through the display driver module to eliminate the delay.

It is easy to understand that, as shown in Figure 5, the head-mounted display device sends the camera data and inertial data to the computing device at time T. The computing device can predict the rendered image at time T based on the point cloud database, camera data and inertial data. After the computing device encodes the rendered image and sends it to the head-mounted display device, a certain delay is actually generated, and the time T+Delay has been reached. At this time, the position of the head-mounted display device may have shifted. At this time, the inertial data obtained is the inertial data at time t=T+Delay. The head-mounted display device predicts the image at time T+Delay based on the target display time information and the inertial measurement unit data of the head-mounted display device.

For example, as shown in FIG4 , the processor may include an image prediction module. After the head-mounted display device receives the rendered image encoding data, the first image decoding module obtains the rendered image and the rendering time information of the rendered image. Then, the image prediction module may determine the target screen display image based on the rendering time information, the current time information, the target display time information and the current inertial data. Specifically, the head-mounted display device collects data and sends it to the computing device for rendering. The computing device then sends the rendered image to the head-mounted server. There is a time difference in this process, including the processor, the computing device, the wireless module, and other processing processes. There will be delays. In order to reduce the delay, the head-mounted display device can determine the target screen display image based on the collected current inertial data. For example, the delay may be 10 milliseconds or 20 milliseconds, and an action frame may be missing in the middle, so it is necessary to predict the target screen display image of the action frame.

Specifically, the head mounted display device motion data corresponding to the target display time information can be determined based on the rendering time information, the current time information, the current inertia data, and the target display time information. For example, the head mounted display device may rotate a certain angle compared to the rendering time. The rendered image is adjusted based on the predicted head mounted display device motion data corresponding to the target display time information to obtain the target screen display image, and the target screen display image is displayed on the display screen. All of the above technical solutions can be combined in any way to form optional embodiments of the present application, and will not be described one by one here.

An embodiment of the present application provides an image processing method, which is applied to a head-mounted display device, including: collecting camera data and inertial data; sending the camera data and the inertial data to a computing device; obtaining rendering image encoding data, wherein the rendering image encoding data is obtained by the computing device determining a rendering image of the head-mounted display device according to the camera data and the inertial data, and encoding the rendering image; obtaining a preset display frame rate, current inertial data, and display time information corresponding to a screen display image corresponding to a historical rendering image determined last time; and displaying a target screen display image corresponding to the rendering image according to the rendering image encoding data, the preset display frame rate, the current inertial data, and the display time information. In the present application, after the head-mounted display device collects camera data and inertial data, the camera data and inertial data are sent to a computing device, and the computing device completes tracking calculations and image rendering. The head-mounted display device only needs to obtain the rendered image and then predict the target screen display image and display the target screen display image without performing tracking calculations and image rendering, thereby reducing the amount of calculation of the head-mounted display device and reducing the power consumption of the head-mounted display device, thereby reducing the size and weight of the radiator and battery, which is conducive to the miniaturization and lightweight of the head-mounted display device.

Please refer to FIG. 6 , which shows a schematic flow chart of an image processing method described in an embodiment of the present application. The image processing method can be applied to a computing device. The method mainly includes steps S201 to S203, which are described as follows:

Step S201, obtaining camera data and inertial data of a head mounted display device.

The computing device may be a local server or a cloud server, or a computer or a smart phone, the head mounted display device may be a head mounted display device of virtual reality, augmented reality or mixed reality, and the head mounted display device and the computing device may be connected via a network. Referring to FIG2 , the head mounted display device may include a SLAM tracking camera, a mouth expression tracking camera, an MR camera and an eye tracking camera, wherein the camera data may be the encoded data of the external environment image collected by the SLAM tracking camera, the encoded data of the user's mouth expression image collected by the mouth expression tracking camera, the encoded data of the external environment image collected by the MR camera, and the encoded data of the user's eye expression image collected by the eye tracking camera. The head mounted display device may also include an IMU sensor, and the inertial data may be the head mounted display device motion data of the user collected by the IMU sensor.

Step S202: determining a rendered image of a head mounted display device according to the camera data and the inertial data.

The computing device may decode the camera data to obtain an external environment image captured by the SLAM tracking camera, a user's mouth expression image captured by the mouth expression tracking camera, an external environment image captured by the MR camera, and an eye expression image of the user captured by the eye tracking camera. The computing device may synthesize multiple external environment images captured by the SLAM tracking camera into a panoramic environment image, synthesize multiple external environment images captured by the MR camera into a panoramic environment image, and synthesize the mouth expression image and two eye expression images into one expression image.

Specifically, the computing device can implement 6DoF tracking of the head-mounted display device, eye and expression tracking, gesture tracking, etc. according to the camera data, inertial data, and the point cloud database, and then render the above-mentioned panoramic environment image according to the tracking results to generate a rendered image. Among them, rendering refers to the process of adding properties and methods to corresponding components, and then adding components to corresponding containers. For example, in PS (Adobe Photoshop), rendering refers to the process of adding attributes such as color and size to the current canvas, in which the canvas can be regarded as a container with many small components, and each component has its own corresponding attributes. Among them, the point cloud database (pointclouddata) is equivalent to a collection of all 3D coordinate points in the spatial environment. Point cloud data refers to a collection of a set of vectors in a three-dimensional coordinate system.

The computing device may render the rendered images corresponding to the left eye perspective and the right eye perspective respectively according to the tracking result. The rendering order may be to render the left eye image first and then the right eye image, and the rendering may be performed in the same manner.

Step S203, encode the rendered image to obtain rendered image encoded data, and send the rendered image encoded data to a head-mounted display device, so that the head-mounted display device obtains a preset display frame rate, current inertia data, and display time information corresponding to a screen display image corresponding to a previously determined historical rendered image, and displays a target screen display image corresponding to the rendered image according to the rendered image encoded data, the preset display frame rate, the current inertia data, and the display time information.

Among them, the computing device can encode the left-eye rendered image and the right-eye rendered image, and send them to the head-mounted display device. The head-mounted display device determines the target screen display image according to the rendering time information of the rendered image and the current inertial data and target display time information measured by the head-mounted display device, and then the head-mounted display device displays the target screen display image through the display driver.

In this embodiment, step S202 may specifically include: obtaining a preset display frame rate of the head-mounted display device, and determining a target rendering frame number according to the preset display frame rate of the head-mounted display device; and determining a rendered image of the head-mounted display device according to the camera data, the inertial data, and the target rendering frame number.

It is easy to understand that the current head-mounted display device can simultaneously support three preset display frame rates of 72Hz, 90Hz and 120Hz. However, the camera's shooting frame rate may be only 30Hz, and the highest is only 60Hz. In order to meet the display effect of the head-mounted display device, it is not possible to render only 30 frames or 60 frames. Therefore, the IMU will collect inertial data and send it to the computing device according to the second preset acquisition frame rate that is much larger than the preset display frame rate of the head-mounted display device. The computing device can perform interpolation rendering of the time axis based on the collected inertial data and camera data. For example, if the preset display frame rate of the head-mounted display device is 120Hz, the camera's shooting frame rate is 60Hz, and the second preset acquisition frame rate is 1KHz, then the computing device can receive inertial data corresponding to 1000 frames of image frames and camera data corresponding to 60 frames of image frames. At this time, according to the time axis, it can be determined that there is no camera data at a certain moment, and the rendered image at that moment can be predicted based on the camera data of the previous frame at that moment and the multiple frames of inertial data before that moment. Specifically, the computing device may perform interpolation rendering according to the above method in the order of the timeline, that is, render 120 frames or more of images.

Please refer to FIG7 . The computing device can also connect multiple sets of head-mounted display devices through a wireless module, and can realize the tracking and use of multiple head-mounted display devices in the same environmental coordinate system. For example, when multiple users play games together in the same environment, multiple head-mounted display devices can be connected to the computing device at the same time, and collect camera data and inertial data and send them to the computing device. The computing device can track multiple head-mounted display devices according to the camera data, inertial data, and point cloud database corresponding to the multiple head-mounted display devices, and realize the synchronization of multiple head-mounted devices.

All of the above technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

The embodiment of the present application provides an image processing method, which is applied to a computing device, including: obtaining camera data and inertial data of a head-mounted display device; determining a rendering image of the head-mounted display device according to the camera data and inertial data; encoding the rendering image to obtain rendering image encoding data, and sending the rendering image encoding data to the head-mounted display device, so that the head-mounted display device obtains a preset display frame rate, current inertial data, and display time information corresponding to a screen display image corresponding to a historical rendering image determined previously, and displays a target screen display image corresponding to the rendering image according to the rendering image encoding data, the preset display frame rate, the current inertial data, and the display time information. In the present application, after the head-mounted display device collects the camera data and inertial data, the camera data and inertial data are sent to the computing device, and the computing device completes the tracking calculation and image rendering. The head-mounted display device only needs to predict the target screen display image and display the target screen display image after obtaining the rendering image, without the need for tracking calculation and image rendering, thereby reducing the amount of calculation of the head-mounted display device, and reducing the power consumption of the head-mounted display device, thereby reducing the size and weight of the radiator and the battery, which is conducive to the miniaturization and lightweight of the head-mounted display device.

The method embodiment of the present application is described in detail above. The device embodiment of the present application is described in detail below in conjunction with Figure 8. It should be understood that the device embodiment and the method embodiment correspond to each other, and similar descriptions can refer to the method embodiment.

FIG8 is a schematic structural diagram of an image processing device 10 according to an embodiment of the present application. As shown in FIG8 , the image processing device 10 may include:

The acquisition module 11 is used to acquire camera data and inertial data;

A sending module 12, used for sending the camera data and the inertial data to the computing device;

A first acquisition module 13 is used to acquire rendered image coding data, where the rendered image coding data is obtained by a computing device determining a rendered image of a head mounted display device according to camera data and inertial data, and encoding the rendered image;

The second acquisition module 14 is used to acquire a preset display frame rate, current inertial data, and display time information corresponding to a screen display image corresponding to a previously determined historical rendering image;

The display module 15 is used to display a target screen display image corresponding to the rendered image according to the rendered image encoding data, the preset display frame rate, the current inertia data, and the display time information.

Optionally, the display module 15 may also be used to: if the rendered image coding data is not obtained, then display the target screen display image corresponding to the rendered image according to the previous frame image coding data, the inertial data corresponding to the previous frame image coding data and the current inertial data.

Optionally, the head-mounted display device includes multiple cameras, and the image processing device 10 may include a third determination module, used to: adjust the exposure time of the multiple cameras to align the exposure center points of the multiple cameras; determine the timestamp data of the camera data based on the exposure center points of the multiple cameras.

Optionally, the head mounted display device includes an inertial sensor, and the image processing device 10 may also be used to determine a timestamp of the inertial data according to an interruption time of the inertial sensor.

Optionally, the acquisition module 11 can be used to: acquire camera data according to a first preset acquisition frame rate of a camera of the head mounted display device; and acquire inertial data according to a second preset acquisition frame rate.

It should be noted that the functions of each module in the image processing device 10 in the embodiment of the present application can correspond to the specific implementation methods in the above-mentioned method embodiments, which will not be repeated here.

Each module in the above-mentioned image processing device 10 can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of the processor in the head-mounted display device in the form of hardware, or can be stored in the memory in the head-mounted display device in the form of software, so that the processor can call and execute the corresponding operations of each module.

The image processing device 10 provided in the embodiment of the present application collects camera data and inertial data through the collection module 11, and the sending module 12 sends the camera data and inertial data to the computing device. Then, the first acquisition module 13 obtains the rendering image coding data, which is the rendering image of the head-mounted display device determined by the computing device according to the camera data and inertial data, and is obtained by encoding the rendering image. Then, the second acquisition module 14 obtains the preset display frame rate, the current inertial data, and the display time information corresponding to the screen display image corresponding to the historical rendering image determined last time. The display module 15 displays the target screen display image corresponding to the rendering image according to the rendering image coding data, the preset display frame rate, the current inertial data, and the display time information, thereby reducing the calculation amount of the head-mounted display device, reducing the power consumption of the head-mounted display device, and further reducing the size and weight of the radiator and the battery, which is conducive to the miniaturization and lightweight of the head-mounted display device.

FIG. 9 is a schematic structural diagram of an image processing device 20 according to an embodiment of the present application. As shown in FIG. 9 , the image processing device 20 is applied to a computing device and may include:

An acquisition module 21 is used to acquire camera data and inertial data of a head mounted display device;

A determination module 22, configured to determine a rendered image of a head mounted display device according to the camera data and the inertial data;

The encoding module 23 is used to encode the rendered image to obtain the rendered image encoding data, and send the rendered image encoding data to the head-mounted display device, so that the head-mounted display device obtains the preset display frame rate, the current inertia data, and the display time information corresponding to the screen display image corresponding to the historical rendered image determined last time, and displays the target screen display image corresponding to the rendered image according to the rendered image encoding data, the preset display frame rate, the current inertia data, and the display time information.

Optionally, the determination module 22 can be used to: obtain a preset display frame rate of the head-mounted display device, and determine a target rendering frame number based on the preset display frame rate of the head-mounted display device; and determine a rendered image of the head-mounted display device based on camera data, inertial data and the target rendering frame number.

It should be noted that the functions of each module in the image processing device 20 in the embodiment of the present application can correspond to the specific implementation methods in the above-mentioned method embodiments, which will not be repeated here.

Each module in the above-mentioned image processing device 20 can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of the processor in the head-mounted display device in the form of hardware, or can be stored in the memory in the head-mounted display device in the form of software, so that the processor can call and execute the corresponding operations of each module.

The image processing device 20 provided in the embodiment of the present application obtains camera data and inertial data of a head-mounted display device through an acquisition module 21, and then the determination module 22 determines a rendered image of the head-mounted display device according to the camera data and the inertial data. Then the encoding module 23 encodes the rendered image to obtain rendered image encoded data, and sends the rendered image encoded data to the head-mounted display device, so that the head-mounted display device obtains a preset display frame rate, current inertial data, and display time information corresponding to a screen display image corresponding to a historical rendered image determined last time, and displays a target screen display image corresponding to the rendered image according to the rendered image encoded data, the preset display frame rate, the current inertial data, and the display time information, thereby reducing the amount of calculation of the head-mounted display device, reducing the power consumption of the head-mounted display device, and further reducing the size and weight of the radiator and the battery, which is conducive to the miniaturization and lightweight of the head-mounted display device.

Please refer to Figure 10, which shows a schematic block diagram of a head mounted display device provided in an embodiment of the present application. The head mounted display device 30 shown in Figure 10 is only an example and should not bring any limitation to the functions and scope of use of the embodiment of the present application.

The head mounted display device 30 may include: a processor 31, a memory 32, an input device 33, an output device 34, a communication device 35, a communication bus 36, and an input/output (I/O) interface 37. The processor 31, the memory 32, and the input/output (I/O) interface 37 communicate with each other through the communication bus 36. Generally, the following devices may be connected to the I/O interface 37: an input device 33 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 34 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; and a communication device 35. The communication device 35 may allow the head mounted display device 30 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 10 shows a head mounted display device 30 having various devices, it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead.

Specifically, the memory 32 may be used to store software programs and modules, and the processor 31 runs the software programs and modules stored in the memory 32, such as the software programs of the corresponding operations in the aforementioned method embodiments.

In some embodiments, the processor 31 can call software programs and modules stored in the memory 32 to perform the following operations: collect camera data and inertial data; send the camera data and the inertial data to a computing device; obtain rendering image coding data, wherein the rendering image coding data is obtained by the computing device determining the rendering image of the head-mounted display device based on the camera data and the inertial data, and encoding the rendering image; obtain a preset display frame rate, current inertial data, and display time information corresponding to a screen display image corresponding to a historical rendering image determined last time; and display a target screen display image corresponding to the rendering image according to the rendering image coding data, the preset display frame rate, the current inertial data, and the display time information.

Specifically, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device 35, or installed from a memory 32. When the computer program is executed by the processor 31, the above-mentioned functions defined in the aforementioned method embodiment of the present application are executed.

It should be noted that it should be understood that the division of the various devices and terminals in FIG. 10 is only a division of logical functions, and in actual implementation, they can be fully or partially integrated into one physical entity, or physically separated. Moreover, these modules can be all implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware.

The embodiment of the present disclosure also provides a computing device, which includes a processor and a memory, wherein a computer program is stored in the memory, and the processor can perform the following operations by calling the computer program stored in the memory: obtaining camera data and inertial data of a head-mounted display device; determining a rendering image of the head-mounted display device according to the camera data and the inertial data; encoding the rendering image to obtain rendering image encoding data, and sending the rendering image encoding data to the head-mounted display device, so that the head-mounted display device obtains a preset display frame rate, current inertial data, and display time information corresponding to a screen display image corresponding to a historical rendering image determined last time, and displays a target screen display image corresponding to the rendering image according to the rendering image encoding data, the preset display frame rate, the current inertial data, and the display time information.

The present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the image processing method described in the above method embodiment are executed. The storage medium may be a volatile or non-volatile computer-readable storage medium.

The embodiments of the present disclosure also provide a computer program product, which carries a program code. The instructions included in the program code can be used to execute the steps of the image processing method described in the above method embodiment. For details, please refer to the above method embodiment, which will not be repeated here.

The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is implemented as a computer storage medium. In another optional embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

It should be understood that the processor of the embodiment of the present application may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method embodiment can be completed by an integrated logic circuit of hardware in the processor or instructions in the form of software. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit, or a processor that can process signals.

(Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

It can be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DRRAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

It should be understood that the above-mentioned memory is exemplary but not restrictive. For example, the memory in the embodiment of the present application may also be a static random access memory (static RAM, SRAM), a dynamic random access memory (dynamic RAM, DRAM), a synchronous dynamic random access memory, or a synchronous dynamic random access memory.

(synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous linked dynamic random access memory

(synchlink DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DRRAM), etc. That is, the memory in the embodiments of the present application is intended to include but is not limited to these and any other suitable types of memory.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the device or unit can be electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (13)

1. An image processing method, applied to a head-mounted display device, comprising:

collecting camera data and inertial data;

Transmitting the camera data and the inertial data to a computing device;

Acquiring rendering image coding data, wherein the rendering image coding data is obtained by determining a rendering image of the head-mounted display device according to the camera data and the inertia data by the computing device and coding the rendering image;

Acquiring preset display frame rate, current inertial data and display time information corresponding to a screen display image corresponding to a previously determined historical rendering image;

And displaying a target screen display image corresponding to the rendering image according to the rendering image coding data, the preset display frame rate, the current inertia data and the display time information.

2. The image processing method according to claim 1, wherein after transmitting the camera data and the inertial data to a computing device, the method further comprises:

And if the rendering image coding data is not acquired, displaying a target screen display image corresponding to the rendering image according to the previous frame image coding data, the inertia data corresponding to the previous frame image coding data and the current inertia data.

3. The image processing method of claim 1, wherein the head mounted display device comprises a plurality of cameras, the method further comprising:

adjusting exposure time of the cameras so as to align exposure center points of the cameras;

and determining the timestamp data of the camera data according to the exposure center points of the cameras.

4. The image processing method of claim 3, wherein the head mounted display device includes an inertial sensor, the method further comprising:

and determining the time stamp of the inertial data according to the interruption time of the inertial sensor.

5. The image processing method according to claim 1, wherein the acquiring camera data and inertial data includes:

acquiring the camera data according to a first preset acquisition frame rate of a camera of the head-mounted display device;

and acquiring the inertial data according to a second preset acquisition frame rate, wherein the second preset acquisition frame rate is larger than the first preset acquisition frame rate.

6. An image processing method, applied to a computing device, the method comprising:

acquiring camera data and inertial data of the head-mounted display device;

determining a rendered image of the head-mounted display device according to the camera data and the inertial data;

And encoding the rendering image to obtain rendering image encoding data, and sending the rendering image encoding data to the head-mounted display device, so that the head-mounted display device obtains preset display frame rate, current inertia data and display time information corresponding to a screen display image corresponding to a previously determined historical rendering image, and displays a target screen display image corresponding to the rendering image according to the rendering image encoding data, the preset display frame rate, the current inertia data and the display time information.

7. The image processing method of claim 6, wherein the determining a rendered image of the head mounted display device from the camera data and inertial data comprises:

acquiring a preset display frame rate of the head-mounted display device, and determining a target rendering frame number according to the preset display frame rate of the head-mounted display device;

and determining a rendering image of the head-mounted display device according to the camera data, the inertia data and the target rendering frame number.

8. An image processing apparatus, characterized by being applied to a head-mounted display device, comprising:

the acquisition module is used for acquiring camera data and inertial data;

A transmitting module for transmitting the camera data and the inertial data to a computing device;

The first acquisition module is used for acquiring rendering image coding data, wherein the rendering image coding data is obtained by the computing device through coding a rendering image of the head-mounted display device determined according to the camera data and the inertia data;

the second acquisition module is used for acquiring preset display frame rate, current inertial data and display time information corresponding to a screen display image corresponding to a previously determined historical rendering image;

and the display module is used for displaying a target screen display image corresponding to the rendering image according to the rendering image coding data, the preset display frame rate, the current inertia data and the display time information.

9. An image processing apparatus, characterized by being applied to a computing apparatus, comprising:

The acquisition module is used for acquiring camera data and inertial data of the head-mounted display device;

The determining module is used for determining a rendering image of the head-mounted display device according to the camera data and the inertia data;

The encoding module is used for encoding the rendering image to obtain rendering image encoding data, and sending the rendering image encoding data to the head-mounted display device, so that the head-mounted display device obtains preset display frame rate, current inertia data and display time information corresponding to a screen display image corresponding to a previously determined historical rendering image, and displays a target screen display image corresponding to the rendering image according to the rendering image encoding data, the preset display frame rate, the current inertia data and the display time information.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which is adapted to be loaded by a processor for performing the image processing method according to any one of claims 1-5 or for performing the image processing method according to any one of claims 6-7.

11. A head mounted display device comprising a processor and a memory, the memory having stored therein a computer program, the processor being operable to perform the image processing method of any of claims 1-5 by invoking the computer program stored in the memory.

12. A computing device comprising a processor and a memory, the memory having stored therein a computer program for executing the image processing method according to any one of claims 6 to 7 by calling the computer program stored in the memory.

13. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the image processing method of any one of claims 1-5 or the image processing method of any one of claims 6-7.