CN112017300B - Mixed reality image processing method, device and equipment - Google Patents

Abstract

The present invention discloses a method, device and equipment for processing mixed reality images. The method comprises: acquiring an environmental scene image; identifying a target object contained in the environmental scene image to obtain spatial position information of the target object; and superimposing description information corresponding to the target object onto the environmental scene image to obtain a mixed reality image.

Description

Mixed reality image processing method, device and equipment

Technical Field

The embodiments of the present disclosure relate to the field of image processing technology. More specifically, the embodiments of the present disclosure relate to a method for processing mixed reality images, a device for processing mixed reality images, and a head-mounted display device.

Background Art

Mixed Reality (MR) technology is a further development of virtual reality technology. This technology introduces real-world scene information into the virtual environment, builds an interactive feedback information loop between the virtual world, the real world and the user, and enhances the realism of the user experience.

At present, in the field of early childhood education, augmented reality technology can be used to process teaching videos. When using head-mounted display devices for early childhood education, more realistic scenes can be displayed to users. However, users cannot interact with objects in the scene, and the user experience is poor.

Therefore, it is necessary to provide a new solution for processing mixed reality images.

Summary of the invention

The purpose of the embodiments of the present disclosure is to provide a new technical solution for processing mixed reality images.

According to a first aspect of an embodiment of the present disclosure, a method for processing a mixed reality image is provided, the method comprising:

Acquire environmental scene images;

Identify the target object contained in the environmental scene image, and obtain the spatial position information of the target object;

Based on the spatial position information, the description information corresponding to the target object is superimposed on the environmental scene image to obtain a mixed reality image.

Optionally, after the step of identifying the target object contained in the environmental scene image and obtaining the spatial position information of the target object, the method further includes:

Obtaining an initial data set of virtual objects to be displayed;

Rendering the virtual object to be displayed in the environment scene image according to the plane parameters of the target plane in the environment scene image and the initial data set of the virtual object to be displayed;

Based on the spatial position information and the plane parameters, the description information corresponding to the target object and the description information corresponding to the virtual object to be displayed are superimposed on the environmental scene image to obtain a mixed reality image.

Optionally, the environmental scene image includes a first image and a second image captured by a binocular camera of the head mounted display device;

The step of identifying the target object contained in the environmental scene image and obtaining the spatial position information of the target object comprises:

Identify the target object contained in the first image based on a predetermined recognition model, and obtain first position information of the area where the target object is located;

Identify the target object contained in the second image based on a predetermined recognition model, and obtain second position information of the area where the target object is located;

The spatial position information of the target object is determined according to the first position information and the second position information.

Optionally, the method further comprises: a step of training a predetermined recognition model:

Acquire historical environmental scene images;

Determining a region where a target object is located in the historical environment scene image, and marking position information of the region where the target object is located and category information of the target object;

Generate a data set based on the annotated historical environment scene images;

The recognition model is trained according to the data set.

Optionally, the method further comprises:

Identify the target object contained in the environmental scene image and obtain category information of the target object;

According to the category information of the target object, description information corresponding to the target object is selected from a pre-established database.

Optionally, the head mounted display device includes a first camera and a second camera;

The step of superimposing the description information corresponding to the target object onto the environment scene image based on the spatial position information to obtain a mixed reality image comprises:

Determining three-dimensional coordinate information of a display position of the description information corresponding to the target object according to the spatial position information and a predetermined first offset;

Converting the three-dimensional coordinate information to obtain a first pixel coordinate of the display position in the image coordinate system of the first camera and a second pixel coordinate of the display position in the image coordinate system of the second camera;

According to the first pixel coordinates and the second pixel coordinates, the description information corresponding to the target object is superimposed on the environmental scene image to obtain a mixed reality image.

Optionally, the method further comprises: a step of determining plane parameters of a target plane in the environment scene image:

Extracting feature points from the environment scene image;

Construct a feature point cloud based on the extracted feature points;

Performing plane detection on the environment scene image based on the feature point cloud to determine a target plane;

The plane parameters of the target plane are obtained, wherein the plane parameters include center point coordinates and a normal vector.

Optionally, the step of rendering the virtual object to be displayed in the environmental scene image according to the plane parameters of the target plane in the environmental scene image and the initial data set of the virtual object to be displayed comprises:

Acquire an initial data set of a virtual object to be displayed, the initial data set including three-dimensional coordinate information of a plurality of feature points for constructing the virtual object;

Determining a placement position of the virtual object to be displayed according to the center point coordinates of the target plane, and determining a placement direction of the virtual object to be displayed according to the normal vector of the target plane;

Determining a target data set of the virtual object to be displayed according to the initial data set of the virtual object to be displayed, and the placement position and placement direction of the virtual object to be displayed;

According to the target data set of the virtual object to be displayed, the virtual object to be displayed is rendered in the environment scene image.

Optionally, the step of superimposing the description information corresponding to the target object and the description information corresponding to the virtual object to be displayed on the environment scene image based on the spatial position information and the plane parameters to obtain the mixed reality image includes:

Determine first position information of a display position of the description information corresponding to the target object according to the spatial position information and a predetermined first offset;

Determining second position information of a display position of the description information corresponding to the virtual object to be displayed according to the center point coordinates of the target plane and a predetermined second offset;

According to the first position information and the second position information, the description information corresponding to the target object and the description information corresponding to the virtual object to be displayed are superimposed on the environmental scene image to obtain a mixed reality image.

According to a second aspect of an embodiment of the present disclosure, a device for processing a mixed reality image is provided, the device comprising:

An acquisition module is used to acquire an environmental scene image;

A recognition model is used to recognize a target object contained in the environmental scene image and obtain spatial position information of the target object;

An image generation module, configured to superimpose the description information corresponding to the target object onto the environment scene image based on the spatial position information to obtain a mixed reality image;

or,

The device includes a processor and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, the method described in any one of the first aspects of the embodiments of the present disclosure is executed.

According to a third aspect of an embodiment of the present disclosure, a head-mounted display device is provided, comprising a binocular camera and the apparatus for processing mixed reality images according to the second aspect of an embodiment of the present disclosure.

According to the embodiments of the present disclosure, by processing the mixed reality image, the description information related to the target object can be displayed to the user while displaying the target object. According to the embodiments of the present disclosure, by processing each frame of the video, the description information of the target object can be integrated into the real-life video, so that the user can obtain the description information corresponding to the target object while watching the real-life video through the head-mounted display device, and the user experience is better.

According to the embodiment of the present disclosure, it can be applied to teaching scenarios. While displaying the target object to the user, the descriptive information related to the target object is displayed to the user, which makes it easier for the user to understand the target object, increases the fun of teaching, and provides a better user experience.

Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative work.

FIG1 is a schematic diagram of a hardware configuration of a head mounted display device that can be used to implement an embodiment of the present disclosure;

FIG2 is a schematic diagram of a process of a method for processing a mixed reality image according to an embodiment of the present disclosure;

FIG3 is a schematic diagram of a process of a mixed reality image processing method according to an embodiment of the present disclosure;

FIG4 is a schematic diagram of a scene of a method for processing a mixed reality image according to an embodiment of the present disclosure;

FIG5 is a structural block diagram of a device for processing mixed reality images according to an embodiment of the present disclosure;

FIG6 is a structural block diagram of a device for processing mixed reality images according to an embodiment of the present disclosure;

FIG. 7 is a schematic diagram of the structure of a head mounted display device according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

<Hardware Configuration>

FIG1 is a schematic diagram of the hardware configuration of a head mounted display device 100 that can be used to implement the method for processing mixed reality images according to an embodiment of the present disclosure.

In one embodiment, the head mounted display device 100 may be an intelligent device such as a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (Mixed Reality) device.

In one embodiment, the head mounted display device 100 includes a first camera and a second camera, and the first camera and the second camera are used to simulate human eyes.

In one embodiment, as shown in FIG1 , the head mounted display device 100 may include a processor 110, a memory 120, an interface device 130, a communication device 140, a display device 150, an input device 160, a speaker 170, a microphone 180, a camera 190, etc. Among them, the processor 110 may include, for example, but not limited to a central processing unit CPU, a microprocessor MCU, etc. The processor 110 may also include, for example, a graphics processor GPU (Graphics Processing Unit), etc. The memory 120 may include, for example, but not limited to ROM (Read Only Memory), RAM (Random Access Memory), a non-volatile memory such as a hard disk, etc. The interface device 130 may include, for example, but not limited to a USB interface, a serial interface, a parallel interface, an infrared interface, etc. The communication device 140 may, for example, be capable of wired or wireless communication, and may specifically include WiFi communication, Bluetooth communication, 2G/3G/4G/5G communication, etc. The display device 150 may, for example, be a liquid crystal display, an LED display, a touch display, etc. The input device 160 may include, for example, but not limited to a touch screen, a keyboard, somatosensory input, etc. The speaker 170 and the microphone 180 may be used to output/input voice information. The camera 180 may be used to obtain image information, for example, and the camera 190 may be a binocular camera, for example. Although multiple devices are shown for the head mounted display device 100 in FIG1 , the present invention may only involve some of the devices.

As used in the embodiments of the present disclosure, the memory 120 of the head mounted display device 100 is used to store instructions, and the instructions are used to control the processor 110 to operate to support the implementation of the mixed reality image processing method of any embodiment provided in the first aspect of the present disclosure. The technicians can design instructions according to the scheme disclosed in the embodiments of the present disclosure. How the instructions control the processor to operate is well known in the art, so it will not be described in detail here.

<Method Example 1>

Referring to FIG. 2 , a method for processing a mixed reality image provided by an embodiment of the present disclosure is described. The method relates to a head mounted display device, which may be the head mounted display device 100 as shown in FIG. 1 . The method for processing a mixed reality image comprises the following steps:

Step 210: Acquire an environmental scene image.

In this embodiment, an environmental scene image is acquired through a head-mounted display device. The head-mounted display device includes a binocular camera. The environmental scene image includes a first image captured by a first camera in the binocular camera and a second image captured by a second camera in the binocular camera. The first image and the second image are captured at the same time. Optionally, the same clock trigger source can be used to trigger the first camera and the second camera to ensure hardware synchronization of the first camera and the second camera. In this embodiment, the image size of the first image and the second image is consistent, wherein the image size can be set in a variety of ways.

In one embodiment, in a teaching scenario, different objects can be displayed to the user through an environmental scene image, and corresponding teaching content can be displayed for the displayed objects. The environmental scene image includes a target object. The target object can be an object used to display to the user in the environmental scene image. The environmental scene image can be, for example, an indoor scene image or an outdoor scene image. The target objects contained in the indoor scene image can be household items, food, etc. Household items can be, for example, sofas, dining tables, chairs, etc. Food can be vegetables, fruits, snacks, etc. For example, apples, bananas, pitaya, tomatoes, green vegetables, etc. The target objects contained in the outdoor scene image can be, for example, shops, buses, traffic lights, etc.

After acquiring the environment scene image, proceed to step 220.

Step 220: Identify the target object contained in the environmental scene image and obtain the spatial position information of the target object.

In one embodiment, the step of identifying the target object contained in the environmental scene image and obtaining the spatial position information of the target object may further include: identifying the target object contained in the environmental scene image based on a predetermined recognition model and obtaining the spatial position information of the target object. According to an embodiment of the present disclosure, locating the target object in the environmental scene image based on a predetermined recognition model can improve the accuracy of recognition.

In this embodiment, the method for processing mixed reality images further includes the step of training a predetermined recognition model. The step of training a predetermined recognition model may further include: steps 410-440.

Step 410: Acquire a historical environment scene image.

In this example, the historical environment scene image may be an image containing a target object. The environment scene may be an indoor environment or an outdoor environment. The target object may be an object for display to a user.

Step 420: determine the area where the target object in the historical environment scene image is located, and mark the position information of the area where the target object is located and the category information of the target object.

In this example, the target object in the historical environment scene image can be selected through the selection window. That is to say, the area where the target object is located can be the area selected by the selection window. The position information of the area where the target object is located may include the coordinates of the corner points of the selection window and the size information of the selection window. The size information of the selection window may include the length information and width information of the selection window. Optionally, the size information of the selection window may be a predetermined fixed value. Optionally, corresponding selection windows are set for target objects of different categories, and the size information of the selection windows corresponding to target objects of different categories may be the same or different. The shape of the selection window can also be set according to actual needs. For example, the selection window is rectangular. The embodiments of the present disclosure are not limited to this.

Step 430: Generate a data set based on the annotated historical environment scene images.

Step 440: Train the recognition model according to the data set.

In this embodiment, the annotated historical environment scene images are used as training samples, and a data set is generated based on a plurality of annotated historical environment scene images. The more training samples there are in the data set, the higher the accuracy of the training result. Moreover, when the number of training samples in the data set reaches a certain value, as the number of training samples increases, the improvement in the accuracy of the training result gradually decreases until it becomes stable. Here, the number of required training samples, i.e., the number of historical environment scene images, can be determined by taking into account both the accuracy of the training result and the data processing cost.

In a more specific example, the step of generating a data set based on the annotated historical environment scene images may further include: steps 431-434.

Step 431: Acquire a first predetermined number of annotated historical environment scene images as a first data set.

In this step, a small number of historical environment scene images may be selected, and the selected historical environment scene images may be manually annotated to annotate the location information of the area where the target object is located in the historical environment scene images and the category information of the target object.

Step 432: Acquire a second predetermined number of unlabeled historical environment scene images as a second data set.

In this step, the second predetermined number is much larger than the first predetermined number.

Step 433: cluster the historical environment scene images in the second data set according to the first data set to obtain the location information of the area where the target object is located and the category information of the target object in each historical environment scene image in the second data set.

Step 434: Use the first data set and the clustered second data set as a data set.

According to the disclosed embodiment, a large number of unlabeled second data sets can be clustered based on a small number of labeled first data sets to determine the location information of the area where the target object is located in each historical environment scene image in the second data set and the category information of the target object, and the first data set and the clustered second data set are used as data sets to train the recognition model. In this way, the efficiency of obtaining data sets can be improved and labor costs can be reduced. Furthermore, the accuracy of recognition model recognition can also be improved.

In a more specific example, the step of identifying the target object contained in the environmental scene image and obtaining the spatial position information of the target object may further include: steps 510-530.

Step 510: Identify a target object contained in the first image based on a predetermined recognition model, and obtain first position information of an area where the target object is located.

Step 520: Identify the target object contained in the second image based on a predetermined recognition model, and obtain second position information of the area where the target object is located.

Step 530: Determine the spatial position information of the target object according to the first position information and the second position information.

In this example, the head mounted display device includes binocular cameras, namely, a first camera and a second camera. The first image and the second image are respectively acquired by the first camera and the second camera, and the first image and the second image are acquired at the same time. That is, the first image and the second image contain the same target object. The first position information may be the position information of the area where the target object is located in the image coordinate system of the first camera. The second position information may be the position information of the area where the target object is located in the image coordinate system of the second camera. For example, the position information of the area where the target object is located may be the coordinates of a corner point of the selection window and the size information of the selection window. According to the coordinates of a corner point of the selection window and the size information of the selection window, the coordinates of the four corner points of the selection window may be calculated. Further, using the principle of stereo triangulation positioning, the spatial position information of the area where the target object is located in the camera coordinate system of the first camera or the camera coordinate system of the second camera may be calculated according to the first position information and the second position information. Among them, the spatial position information of the area where the target object is located may be the three-dimensional coordinates of the four corner points of the selection window in the camera coordinate system. According to the embodiment of the present disclosure, the positioning of the target object is realized by positioning the area where the target object is located, and there is no need to position the multiple feature points constituting the target object, thereby improving the efficiency of positioning.

After identifying the target object contained in the environmental scene image and obtaining the spatial position information of the target object, the process proceeds to step 230 .

Step 230: Based on the spatial position information, superimpose the description information corresponding to the target object onto the environmental scene image to obtain a mixed reality image.

In this embodiment, the description information at least includes the teaching information of the target object. The description information can be a subtitle or an illustration. The description information of the target object is superimposed on the environmental scene image, and the description information of the target object can also be displayed to the user while the target object is displayed to the user. In this embodiment, according to the description information of the target object, the relevant content of the target object can be explained to the user, thereby realizing the display and teaching of the target object. The description information can be different for target objects of different categories. For example, the description information of household products can include the name of the household products and the usage scenario. The description information corresponding to the fruit can include the name of the fruit, the origin of the fruit and the growing environment of the fruit. The description information of the traffic light can include the type, purpose and usage scenario of the traffic light. For example, as shown in Figure 4, description information can be added to the fruit in the scene.

In one embodiment, the mixed reality image processing method may further include: identifying a target object contained in the environment scene image, obtaining category information of the target object, and selecting description information corresponding to the target object from a pre-established database according to the category information of the target object.

In one embodiment, based on the spatial position information, the description information corresponding to the target object is superimposed on the environmental scene image to obtain a mixed reality image, which may further include: steps 610-630.

Step 610: Determine the three-dimensional coordinate information of the display position of the description information corresponding to the target object according to the spatial position information and a predetermined first offset.

In one example, the three-dimensional coordinates of the corner point of the upper left corner of the selection window of the target object are obtained; based on the three-dimensional coordinates of the corner point and a predetermined first offset, the three-dimensional coordinate information of the display position of the description information corresponding to the target object can be determined. Determining the three-dimensional coordinate information of the display position of the description information based on the predetermined first offset can ensure that the description information is displayed near the target object, and can also prevent the description information from blocking the target object.

In one example, determining the display position of the description information of the target object may be determining the position of the information display window for displaying the description information. Specifically, the three-dimensional coordinates of the corner point of the upper left corner of the selection window of the target object are obtained; based on the three-dimensional coordinates of the corner point and a predetermined first offset, the three-dimensional coordinate information of the corner point of the upper left corner of the information display window may be determined; based on the three-dimensional coordinate information of the corner point of the upper left corner of the information display window and the size of the information display window, description information is added to the target object. The description information may be a subtitle or an illustration.

In one example, based on the SLAM (Simultaneous Localization And Mapping) algorithm, the posture information of the head-mounted display device relative to the external environment can be obtained. The posture information includes the rotation matrix R _HMD and the translation vector T _HMD of the head-mounted display device in the world coordinate system. According to the posture information of the head-mounted display device, the three-dimensional coordinate information of the display position of the description information is transformed into a coordinate system, and the three-dimensional coordinate information of the display position of the description information in the world coordinate system is determined. Specifically, the three-dimensional coordinate information of the display position of the description information is transformed into a coordinate system based on the following formula (1).

_Pw ＝ _RHMD * _Pc + _THMD (1)

Wherein, _Pw is the three-dimensional coordinate information of the display position of the description information in the world coordinate system, _Pc is the three-dimensional coordinate information of the display position of the description information in the camera coordinate system, R _HMD is the rotation matrix of the head-mounted display device in the world coordinate system, and T _HMD is the translation vector of the head-mounted display device in the world coordinate system.

Step 620: convert the three-dimensional coordinate information to obtain a first pixel coordinate of the display position in the image coordinate system of the first camera and a second pixel coordinate of the display position in the image coordinate system of the second camera.

In one example, the first pixel coordinate and the second pixel coordinate are calculated based on the following formulas (2)-(4).

P _uv1 = k ₁ * E * P _w (2)

P _uv2 = k ₂ * E * P _w (3)

Among them, P _uv1 is the first pixel coordinate of the display position in the image coordinate system of the first camera, P _uv2 is the second pixel coordinate of the display position in the image coordinate system of the second camera, k ₁ is the intrinsic parameter matrix of the first camera, k ₂ is the intrinsic parameter matrix of the second camera, P _w is the three-dimensional coordinate information of the display position of the description information in the world coordinate system, E is the transformation matrix of the head-mounted display device, R _HMD is the rotation matrix of the head-mounted display device in the world coordinate system, and T _HMD is the translation vector of the head-mounted display device in the world coordinate system.

Step 630: Superimpose the description information corresponding to the target object onto the environment scene image according to the first pixel coordinates and the second pixel coordinates to obtain a mixed reality image.

In this embodiment, the binocular camera of the head mounted display device can overlay the description information on the environment scene image according to the obtained first pixel coordinates and second pixel coordinates. Further, according to each environment scene image of the real scene video, the description information of the target object can be overlayed and displayed in the real scene video.

According to the embodiments of the present disclosure, by processing the mixed reality image, the description information related to the target object can be displayed to the user while displaying the target object. According to the embodiments of the present disclosure, by processing each frame of the video, the description information of the target object can be integrated into the real-life video, so that the user can obtain the description information corresponding to the target object while watching the real-life video through the head-mounted display device, and the user experience is better.

According to the embodiment of the present disclosure, it can be applied to teaching scenarios. While displaying the target object to the user, the descriptive information related to the target object is displayed to the user, which makes it easier for the user to understand the target object, increases the fun of teaching, and provides a better user experience.

<Method Example 2>

Referring to Fig. 3, a method for processing a mixed reality image provided by an embodiment of the present disclosure is described. The method relates to a head mounted display device, which may be the head mounted display device 100 as shown in Fig. 1. The method for processing a mixed reality image includes the following steps.

Step 310: Acquire an environmental scene image.

Step 320: Identify the target object contained in the environmental scene image and obtain the spatial position information of the target object.

In this embodiment, the specific process of acquiring the environmental scene image in step 310 and identifying the spatial position information of the target object in step 320 is as described in the above embodiment and will not be repeated here.

Step 330: Obtain an initial data set of virtual objects to be displayed.

In one embodiment, the initial data set of the virtual object to be displayed includes three-dimensional coordinate information of multiple feature points for constructing the virtual object. Based on the initial data set, the virtual object can be constructed. According to the actual usage scenario of the user, multiple initial data sets of virtual objects to be displayed can be pre-generated. Optionally, the initial data of the virtual object to be displayed can be stored in the head-mounted display device or in the server. After the initial data set of the virtual object to be displayed, enter step 340.

Step 340: Render the virtual object to be displayed in the environment scene image according to the plane parameters of the target plane in the environment scene image and the initial data set of the virtual object to be displayed.

In this embodiment, the target plane in the environmental scene image can be the ground, a desktop, a platform, etc. A virtual object can be generated based on the target plane. According to the embodiment of the present disclosure, in the process of teaching through a head-mounted display device, a virtual object can be added to a real-scene video, so that a user can interact with the virtual object while watching the real-scene video through the head-mounted display device, which is convenient for the user to observe the virtual object from different angles, thereby improving the effect of teaching based on the head-mounted display device and providing a better user experience.

In one embodiment, the step of rendering the virtual object to be displayed in the environmental scene image according to the plane parameters of the target plane in the environmental scene image and the initial data set of the virtual object to be displayed may further include: steps 710-740.

Step 710: Acquire an initial data set of a virtual object to be displayed, wherein the initial data set includes three-dimensional coordinate information of a plurality of feature points for constructing the virtual object.

Step 720: Determine the placement position of the virtual object to be displayed according to the center point coordinates of the target plane, and determine the placement direction of the virtual object to be displayed according to the normal vector of the target plane.

Optionally, when generating a virtual object, the center of gravity of the virtual object can be made to coincide with the center point of the target plane, and the virtual object can be stacked along the normal vector direction of the target plane. The placement position and placement direction of the virtual object to be displayed can be determined according to the plane parameters of the target plane, so that the virtual object to be displayed can be displayed in the middle of the mixed reality image, and the displayed virtual object can be prevented from being tilted.

Step 730: Determine a target data set of the virtual object to be displayed according to the initial data set of the virtual object to be displayed, and the placement position and placement direction of the virtual object to be displayed.

Step 740: Render the virtual object to be displayed in the environment scene image according to the target data set of the virtual object to be displayed.

In one example, the target data set of the virtual object includes three-dimensional coordinate information of multiple feature points for constructing the virtual object, and the three-dimensional coordinate information of the feature points is coordinate information converted according to the plane parameters of the target plane. Rendering the virtual object to be displayed in the environmental scene image specifically includes converting the three-dimensional coordinate information of each feature point in the target data set; obtaining the third pixel coordinate of the feature point in the image coordinate system of the first camera and the fourth pixel coordinate of the feature point in the image coordinate system of the second camera; the binocular camera of the head mounted display device renders the virtual object to be displayed in the environmental scene image according to the third pixel coordinate and the fourth pixel coordinate of each feature point in the target data set.

In one embodiment, the method for processing mixed reality images may further include the step of determining the plane parameters of the target plane in the environment scene image. Specifically, it may further include steps 710-740.

Step 710: extract feature points from the environment scene image.

In this step, the environmental scene image includes a first image captured by a first camera in a binocular camera and a second image captured by a second camera in a binocular camera. Feature points are extracted from the first image to obtain a plurality of first feature points. Feature points of the image can be extracted by using a FAST (Feature from Accelerated Segment Test) corner detection algorithm, a SIFT (Scale Invariant Feature Transform) algorithm, or an ORB (Oriented FAST and Rotated BRIEF) algorithm, etc., without any limitation here.

Step 720: construct a feature point cloud based on the extracted feature points.

In one example, the feature point cloud may include three-dimensional coordinate information of multiple first feature points in a world coordinate system. Specifically, the epipolar matching method is used to match feature points in the second image, and multiple second feature points in the second image that match the multiple first feature points are obtained. Through triangulation, the three-dimensional coordinate information of the multiple first feature points in the first camera coordinate system is calculated based on the first feature points and the second feature points. The coordinate system is converted according to the posture of the head-mounted display device when the environmental scene image of the current frame is collected, and the three-dimensional coordinate information of the multiple first feature points in the world coordinate system is obtained, that is, the feature point cloud is obtained.

Step 730: Perform plane detection on the environment scene image based on the feature point cloud to determine the target plane.

In one example, the current plane is determined based on three feature points randomly extracted from the feature point cloud; the normal vector of the current plane and the number of first inner points are obtained, and when the normal vector of the current plane and the number of first inner points meet a predetermined condition, the current plane is determined as the current valid plane. The first inner point is a feature point whose distance from the current plane is less than a first predetermined distance. After determining multiple current valid planes, the plane with the largest number of first inner points is selected as the target plane.

Step 740: Obtain plane parameters of the target plane, wherein the plane parameters include center point coordinates and a normal vector.

The following is a specific example to illustrate the determination of rendering a virtual object to be displayed in an environmental scene image, wherein the virtual object to be displayed is a dinosaur model.

Step 901: Acquire an environmental scene image.

Step 902: Determine the plane parameters of the target plane in the environment scene image. The plane parameters include the three-dimensional coordinates P _cplane of the center point of the target plane and the normal vector V _cplane .

Step 903: Based on the following formulas (5) and (6), coordinate system transformation is performed on the three-dimensional coordinates P _cplane of the center point of the target plane and the normal vector V _cplane according to the posture information of the head mounted display device.

P _wplane ＝R _HMD *P _cplane +T _HMD (5)

V _wplane ＝R _HMD *V _cplane +T _HMD (6)

Among them, P _wplane is the three-dimensional coordinate information of the center point of the target plane in the world coordinate system, P _cplane is the three-dimensional coordinate information of the center point of the target plane in the camera coordinate system, W _wplane is the normal vector of the target plane in the world coordinate system, V _cplane is the normal vector of the target plane in the camera coordinate system, R _HMD is the rotation matrix of the head-mounted display device in the world coordinate system, and T _HMD is the translation vector of the head-mounted display device in the world coordinate system.

Step 904: obtaining an initial data set of the dinosaur model to be displayed;

Step 905 : Determine the target data set of the virtual object to be displayed according to the initial data set of the dinosaur model to be displayed, the three-dimensional coordinates P _wplane of the center point of the target plane in the world coordinate system, and the normal vector V _wplane .

Step 906: Based on the following formulas (7)-(9), the three-dimensional coordinate information of each feature point in the target data set of the dinosaur model is converted to obtain the third pixel coordinate of the feature point in the image coordinate system of the first camera and the fourth pixel coordinate of the feature point in the image coordinate system of the second camera.

P _uv3 ＝k ₁ *E*P _wdinosant (7)

P _uv4 ＝k ₂ *E*P _wdinosant (8)

Among them, P _wdinosant is the three-dimensional coordinate information of any feature point in the target data set in the world coordinate system, P _uv3 is the third pixel coordinate of any feature point in the target data set in the image coordinate system of the first camera, P _uv4 is the third pixel coordinate of any feature point in the target data set in the image coordinate system of the first camera, k ₁ is the intrinsic parameter matrix of the first camera, k ₂ is the intrinsic parameter matrix of the second camera, E is the transformation matrix of the head-mounted display device, R _HMD is the rotation matrix of the head-mounted display device in the world coordinate system, and T _HMD is the translation vector of the head-mounted display device in the world coordinate system.

Step 907: Render the dinosaur model to be displayed in the environment scene image according to the third pixel coordinates and the fourth pixel coordinates of each feature point in the target data set of the dinosaur model.

The virtual object to be displayed is rendered in the environment scene image, and then the process proceeds to step 350 .

Step 350: Based on the spatial position information and the plane parameters, the description information corresponding to the target object and the description information corresponding to the virtual object to be displayed are superimposed on the environmental scene image to obtain a mixed reality image.

In one embodiment, the step of superimposing the description information corresponding to the target object and the description information corresponding to the virtual object to be displayed on the environmental scene image based on the spatial position information and the plane parameters to obtain a mixed reality image may further include: steps 810-830.

Step 810: Determine first position information of a display position of description information corresponding to the target object according to the spatial position information and a predetermined first offset.

The description information corresponding to the target object at least includes the teaching information of the target object. The description information of the target object is superimposed on the environmental scene image, and while the target object is displayed to the user, the relevant teaching information of the target object can also be displayed to the user. In one embodiment, the target object contained in the environmental scene image is identified to obtain the category information of the target object. According to the category information of the target object, the description information corresponding to the target object is selected from a pre-established database. The process of determining the first position information is as described in the above embodiment, and will not be repeated here.

Step 820: Determine second position information of a display position of the description information corresponding to the virtual object to be displayed according to the center point coordinates of the target plane and a predetermined second offset.

The description information corresponding to the virtual object to be displayed includes at least the teaching information of the virtual object. The description information may be a subtitle or an illustration. After the virtual object to be displayed is rendered in the environmental scene image, the description information of the virtual object may be superimposed on the environmental scene image, so that while the virtual object is displayed to the user, the relevant teaching information of the virtual object may also be displayed to the user. In this embodiment, according to the description information of the virtual object, the relevant content of the target object may be explained to the user, thereby realizing the display and teaching of the virtual object.

Step 830: Based on the first position information and the second position information, superimpose the description information corresponding to the target object and the description information corresponding to the virtual object to be displayed onto the environmental scene image to obtain a mixed reality image.

By determining the first position information of the display position of the target object description information according to a predetermined first offset, and determining the second position information of the display position of the virtual object description information according to a predetermined second offset, it can be ensured that while the corresponding description information is displayed near the target object and the virtual object, the description information can be prevented from obstructing the target object or the virtual object.

According to the disclosed embodiment, in the process of teaching through a head-mounted display device, a virtual object can be added to a real-scene video, so that the user can watch the real-scene video through the head-mounted display device while interacting with the virtual object, which is convenient for the user to observe the virtual object from different angles, thereby improving the effect of teaching based on the head-mounted display device and providing a better user experience. In addition, in the process of teaching through a head-mounted display device, the description information corresponding to the target object and the virtual object can be integrated into the real-scene video, so that the user can watch the real-scene video through the head-mounted display device while obtaining the teaching information corresponding to the target object and the virtual object, which provides a better user experience.

<Device Example 1>

5 , an embodiment of the present disclosure provides a mixed reality image processing device 50 , wherein the mixed reality image processing device 50 includes an acquisition module 51 , a recognition module 52 , and an image generation module 53 .

The acquisition module 51 can be used to acquire an environment scene image.

The recognition module 52 can be used to recognize the target object contained in the environmental scene image and obtain the spatial position information of the target object.

In a specific example, the recognition module 52 is specifically used to identify the target object contained in the first image based on a predetermined recognition model, and obtain first position information of the area where the target object is located; identify the target object contained in the second image based on a predetermined recognition model, and obtain second position information of the area where the target object is located; and determine the spatial position information of the target object based on the first position information and the second position information.

In a specific example, the recognition module 52 is specifically used to recognize the target object contained in the environmental scene image and obtain the category information of the target object.

The image generation module 53 can be used to superimpose the description information corresponding to the target object onto the environmental scene image based on the spatial position information to obtain a mixed reality image.

In one embodiment, the description information at least includes teaching information of the target object.

In a specific example, the image generation module 53 is specifically used to determine the three-dimensional coordinate information of the display position of the description information corresponding to the target object according to the spatial position information and the predetermined first offset;

Converting the three-dimensional coordinate information to obtain a first pixel coordinate of the display position in the image coordinate system of the first camera and a second pixel coordinate of the display position in the image coordinate system of the second camera;

According to the first pixel coordinates and the second pixel coordinates, the description information corresponding to the target object is superimposed on the environmental scene image to obtain a mixed reality image.

In one embodiment, the mixed reality image processing device 50 may further include a description information acquisition module, which may be used to select description information corresponding to the target object from a pre-established database according to the category information of the target object.

In one embodiment, the mixed reality image processing device 50 may further include an initial data set acquisition module and a virtual object generation module.

In this embodiment, the initial data set acquisition module can be used to acquire an initial data set of a virtual object to be displayed.

The virtual object generation module can be used to render the virtual object to be displayed in the environmental scene image according to the plane parameters of the target plane in the environmental scene image and the initial data set of the virtual object to be displayed.

In a specific example, the virtual object generation module is specifically used to obtain an initial data set of a virtual object to be displayed, the initial data set including three-dimensional coordinate information of multiple feature points for constructing the virtual object; determine the placement position of the virtual object to be displayed according to the center point coordinates of the target plane, and determine the placement direction of the virtual object to be displayed according to the normal vector of the target plane; determine the target data set of the virtual object to be displayed according to the initial data set of the virtual object to be displayed, the placement position and placement direction of the virtual object to be displayed; and render the virtual object to be displayed in the environmental scene image according to the target data set of the virtual object to be displayed.

The image generation module 53 can also be used to superimpose the description information corresponding to the target object and the description information corresponding to the virtual object to be displayed on the environmental scene image based on the spatial position information and the plane parameters to obtain a mixed reality image.

In a specific example, the image generation module 53 is specifically used to determine the first position information of the display position of the description information corresponding to the target object based on the spatial position information and a predetermined first offset; determine the second position information of the display position of the description information corresponding to the virtual object to be displayed based on the center point coordinates of the target plane and a predetermined second offset; and superimpose the description information corresponding to the target object and the description information corresponding to the virtual object to be displayed on the environmental scene image based on the first position information and the second position information to obtain a mixed reality image.

6 , the embodiment of the present disclosure provides a mixed reality image processing device 60, which includes a processor 61 and a memory 62. The memory 62 is used to store a computer program, and when the computer program is executed by the processor 61, the mixed reality image processing method disclosed in any of the above embodiments is implemented.

<Device Example 2>

7 , an embodiment of the present disclosure provides a head mounted display device 70 , which may be the head mounted display device 100 as shown in FIG. 1 . The head mounted display device 70 includes a binocular camera 71 and an image processing device 72 .

In one embodiment, the image processing device 72 may be, for example, the mixed reality image processing device 50 as shown in FIG. 5 , or may be the mixed reality image processing device 60 as shown in FIG. 6 .

In one embodiment, the head mounted display device 100 may be an intelligent device such as a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (Mixed Reality) device.

According to the disclosed embodiment, in the process of teaching through a head-mounted display device, a virtual object can be added to a real-scene video, so that the user can watch the real-scene video through the head-mounted display device while interacting with the virtual object, which is convenient for the user to observe the virtual object from different angles, thereby improving the effect of teaching based on the head-mounted display device and providing a better user experience. In addition, in the process of teaching through a head-mounted display device, the description information corresponding to the target object and the virtual object can be integrated into the real-scene video, so that the user can watch the real-scene video through the head-mounted display device while obtaining the teaching information corresponding to the target object and the virtual object, which provides a better user experience.

Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments, but it should be clear to those skilled in the art that the above embodiments can be used alone or in combination with each other as needed. In addition, for the device embodiment, since it corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description of the corresponding part of the method embodiment. The system embodiment described above is merely illustrative, and the modules described as separate components may or may not be physically separated.

The present invention may be a system, a method and/or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing/processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers. The network adapter card or network interface in each computing/processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing/processing device.

The computer program instructions for performing the operation of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "Like" languages or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, the electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be executed by using the state information of the computer-readable program instructions to personalize and customize the electronic circuit, thereby implementing various aspects of the present invention.

Various aspects of the present invention are described herein with reference to the flow charts and/or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and/or block diagram and the combination of each box in the flow chart and/or block diagram can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions/actions specified in one or more boxes in the flowchart and/or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and/or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions/actions specified in one or more boxes in the flowchart and/or block diagram.

Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions/actions specified in one or more boxes in the flowchart and/or block diagram.

The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and/or flow chart, and the combination of the boxes in the block diagram and/or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims (8)

1. A processing method of a mixed reality image, which is applied to a head-mounted display device, and is characterized in that the head-mounted display device comprises a first camera and a second camera; the method comprises the following steps:

acquiring an environmental scene image;

Identifying a target object contained in the environment scene image, and obtaining space position information and category information of the target object;

selecting description information corresponding to the target object from a pre-established database according to the category information of the target object;

Based on the spatial position information, overlapping the description information corresponding to the target object on the environment scene image to obtain a mixed reality image, wherein the method comprises the following steps:

According to the space position information and a preset first offset, determining three-dimensional coordinate information of a display position of the descriptive information corresponding to the target object;

Converting the three-dimensional coordinate information based on the posture information of the head-mounted display device to obtain a first pixel coordinate of the display position under the image coordinate system of the first camera and a second pixel coordinate of the display position under the image coordinate system of the second camera; wherein the gesture information comprises a rotation matrix and a translation vector of the head-mounted display device in a world coordinate system;

According to the first pixel coordinates and the second pixel coordinates, the description information corresponding to the target object is overlapped on the environment scene image, and a mixed reality image is obtained;

Wherein, after the step of identifying the target object contained in the environmental scene image and obtaining the spatial position information and the category information of the target object, the method further comprises:

Acquiring an initial data set of a virtual object to be displayed;

rendering the virtual object to be displayed in the environment scene image according to the plane parameter of the target plane in the environment scene image and the initial data set of the virtual object to be displayed;

And based on the spatial position information and the plane parameter, overlapping the description information corresponding to the target object and the description information corresponding to the virtual object to be displayed on the environment scene image to obtain a mixed reality image.

2. The method of claim 1, wherein the environmental scene image comprises a first image and a second image acquired by a binocular camera of the head mounted display device;

The step of identifying a target object contained in the environmental scene image and obtaining spatial position information of the target object includes:

Identifying a target object contained in the first image based on a preset identification model, and obtaining first position information of an area where the target object is located;

Identifying a target object contained in the second image based on a preset identification model, and obtaining second position information of an area where the target object is located;

And determining the spatial position information of the target object according to the first position information and the second position information.

3. The method of claim 2, wherein the method further comprises: training a predetermined recognition model:

acquiring a historical environmental scene image;

determining the region of a target object in the historical environment scene image, and marking the position information of the region of the target object and the category information of the target object;

generating a data set according to the marked historical environmental scene image;

training the recognition model according to the data set.

4. The method of claim 2, wherein the method further comprises: determining plane parameters of a target plane in the environmental scene image:

extracting feature points from the environmental scene image;

constructing a characteristic point cloud according to the extracted characteristic points;

performing plane detection on the environmental scene image based on the characteristic point cloud, and determining a target plane;

and obtaining plane parameters of the target plane, wherein the plane parameters comprise coordinates of a central point and normal vectors.

5. The method of claim 4, wherein the rendering the virtual object to be presented in the environmental scene image based on the plane parameters of the target plane in the environmental scene image and the initial data set of the virtual object to be presented comprises:

acquiring an initial data set of a virtual object to be displayed, wherein the initial data set comprises three-dimensional coordinate information of a plurality of feature points for constructing the virtual object;

determining the placement position of the virtual object to be displayed according to the center point coordinates of the target plane, and determining the placement direction of the virtual object to be displayed according to the normal vector of the target plane;

determining a target data set of the virtual object to be displayed according to the initial data set of the virtual object to be displayed, the placement position and the placement direction of the virtual object to be displayed;

and rendering the virtual object to be displayed in the environment scene image according to the target data set of the virtual object to be displayed.

6. The method of claim 5, wherein the step of superimposing the description information corresponding to the target object and the description information corresponding to the virtual object to be displayed on the environmental scene image based on the spatial location information and the plane parameter, and obtaining a mixed reality image includes:

determining first position information of a display position of the descriptive information corresponding to the target object according to the spatial position information and a preset first offset;

Determining second position information of the display position of the description information corresponding to the virtual object to be displayed according to the center point coordinate of the target plane and a preset second offset;

And according to the first position information and the second position information, the description information corresponding to the target object and the description information corresponding to the virtual object to be displayed are overlapped on the environment scene image, so that a mixed reality image is obtained.

7. A processing device of a mixed reality image, which is applied to a head-mounted display device, and is characterized in that the head-mounted display device comprises a first camera and a second camera; the device comprises:

The acquisition module is used for acquiring the environment scene image;

The identification model is used for identifying a target object contained in the environment scene image and obtaining space position information and category information of the target object;

the description information acquisition module is used for selecting the description information corresponding to the target object from a pre-established database according to the category information of the target object;

The image generating module is configured to superimpose description information corresponding to the target object on the environmental scene image based on the spatial position information, and obtain a mixed reality image, and includes:

According to the space position information and a preset first offset, determining three-dimensional coordinate information of a display position of the descriptive information corresponding to the target object;

Converting the three-dimensional coordinate information based on the posture information of the head-mounted display device to obtain a first pixel coordinate of the display position under the image coordinate system of the first camera and a second pixel coordinate of the display position under the image coordinate system of the second camera; wherein the gesture information comprises a rotation matrix and a translation vector of the head-mounted display device in a world coordinate system;

According to the first pixel coordinates and the second pixel coordinates, the description information corresponding to the target object is overlapped on the environment scene image, and a mixed reality image is obtained;

The initial data set acquisition module is used for acquiring an initial data set of the virtual object to be displayed;

The virtual object generation module is used for rendering the virtual object to be displayed in the environment scene image according to the plane parameter of the target plane in the environment scene image and the initial data set of the virtual object to be displayed;

The image generation module is further configured to superimpose, on the environmental scene image, description information corresponding to the target object and description information corresponding to the virtual object to be displayed based on the spatial position information and the plane parameter, to obtain a mixed reality image;

or the apparatus comprises a processor and a memory storing computer instructions which, when executed by the processor, perform the method of any of claims 1-6.

8. A head-mounted display device comprising a binocular camera and the mixed reality image processing apparatus of claim 7.