CN116095353A - Live broadcast method and device based on volume video, electronic equipment and storage medium - Google Patents

Abstract

The embodiment of the application discloses a live broadcast method, a live broadcast device, electronic equipment and a storage medium based on volume video; in the embodiment of the application, live video data of the three-dimensional character model and shooting video of the fusion object can be obtained; performing image recognition processing on the shot video of the fusion object to obtain the virtual image of the fusion object; analyzing the live video data of the three-dimensional character model to obtain background information of the live video data; carrying out fusion processing on the virtual image of the fusion object and the background information of the live video data to obtain fused live video data; acquiring viewing angle information, and determining the display content of the three-dimensional character model and the virtual image of the fusion object in the fused live video data based on the viewing angle information; based on the display content, the fused live video data is displayed, and interaction between the watched object and the three-dimensional character model in the live scene can be supported, so that interactivity and interestingness of the volume video are improved.

Description

Live streaming method, device, electronic equipment and storage medium based on volumetric video

technical field

The present application relates to the technical field of image processing, and in particular to a volume video-based live broadcast method, device, electronic equipment, and storage medium.

Background technique

Volumetric Video (also known as volumetric video, spatial video, volumetric 3D video or 6-DOF video, etc.) is a technology that captures information in 3D space (such as depth information and color information, etc.) and generates a 3D model sequence. Compared with traditional videos, volumetric videos add the concept of space to videos, and use 3D models to better restore the real 3D world, instead of using 2D flat videos and moving mirrors to simulate the sense of space in the real 3D world. Since the volumetric video is essentially a sequence of 3D models, users can adjust it to any viewing angle to watch according to their preferences, which has a higher degree of restoration and immersion than 2D flat video. Volumetric video can be applied in many different scenarios, for example, volumetric video can be applied to live broadcasting scenarios. However, when the volumetric video is applied to the live broadcast scene, it cannot support the interaction between the viewing object and the 3D character model in the live broadcast scene.

Contents of the invention

Embodiments of the present application provide a live broadcast method, device, device, storage medium, and program product based on volumetric video, which can support interaction between viewing objects and 3D character models in a live broadcast scene, thereby improving the interactivity and interest of volumetric video.

An embodiment of the present application provides a live broadcast method based on volumetric video, including:

Obtain the live video data of the 3D character model and the shooting video of the fused object;

performing image recognition processing on the shot video of the fusion object to obtain the virtual image of the fusion object;

Analyzing the live video data of the three-dimensional character model to obtain the background information of the live video data;

performing fusion processing on the avatar of the fusion object and the background information of the live video data to obtain the fused live video data;

Obtain viewing angle information, and determine the display content of the 3D character model in the fused live video data and the avatar of the fused object based on the viewing angle information;

The fused live video data is displayed based on the display content.

Correspondingly, an embodiment of the present application provides a live broadcast device based on volumetric video, including:

An acquisition unit, configured to acquire the live video data of the 3D character model and the shooting video of the fused object;

An image recognition unit, configured to perform image recognition processing on the captured video of the fusion object to obtain the virtual image of the fusion object;

An analysis unit, configured to analyze the live video data of the 3D character model to obtain the background information of the live video data;

A fusion unit, configured to fuse the avatar of the fusion object and the background information of the live video data to obtain fused live video data;

A determining unit, configured to obtain viewing angle information, and determine the display content of the 3D character model in the fused live video data and the avatar of the fused object based on the viewing angle information;

A display unit, configured to display the fused live video data based on the display content.

In addition, an embodiment of the present application also provides an electronic device, including a processor and a memory, the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the volumetric video-based live broadcast method provided in the embodiment of the present application .

In addition, the embodiment of the present application also provides a computer-readable storage medium, the above-mentioned computer-readable storage medium stores a computer program, and the above-mentioned computer program is suitable for being loaded by a processor, so as to execute any one of the methods provided in the embodiment of the present application based on Live broadcast method of volumetric video.

In addition, an embodiment of the present application further provides a computer program product, including a computer program. When the above computer program is executed by a processor, any method for live broadcast based on volumetric video provided in the embodiment of the present application is implemented.

In the embodiment of the present application, the live video data of the 3D character model and the shot video of the fused object can be obtained; the image recognition processing is performed on the shot video of the fused object to obtain the virtual image of the fused object; the live video data of the 3D character model is processed Analyze to obtain the background information of the live video data; fuse the avatar of the fusion object and the background information of the live video data to obtain the fused live video data; obtain the viewing angle information, and determine the information in the fused live video data based on the viewing angle information. The display content of the 3D character model and the avatar of the fused object; displaying the fused live video data based on the display content can support the interaction between the viewing object and the 3D character model in the live scene, thereby improving the interactivity and interest of the volumetric video .

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

FIG. 1 is a schematic flow diagram of a live broadcast method based on volumetric video provided in an embodiment of the present application;

Fig. 2 is a schematic diagram of different viewing angles provided by the embodiment of the present application;

FIG. 3 is a schematic structural diagram of a live broadcast device based on volumetric video provided by an embodiment of the present application;

FIG. 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts belong to the scope of protection of this application.

Embodiments of the present application provide a live broadcast method, device, electronic device, and storage medium based on volumetric video. The apparatus for live broadcasting based on volumetric video can be integrated in electronic equipment, and the electronic equipment can be a server or a terminal or other equipment.

Among them, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication , middleware services, domain name services, security services, network acceleration services (Content Delivery Network, CDN), and cloud servers for basic cloud computing services such as big data and artificial intelligence platforms.

The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be connected directly or indirectly through wired or wireless communication, which is not limited in this application.

In addition, "multiple" in the embodiments of the present application refers to two or more. "First" and "second" in the embodiments of the present application are used to distinguish descriptions, and should not be understood as implying relative importance.

Each will be described in detail below. It should be noted that the description sequence of the following embodiments is not intended to limit the preferred sequence of the embodiments.

In this embodiment, a description will be made from the perspective of a volumetric video-based live broadcast device. In order to facilitate the description of the volumetric video-based live broadcast method of the present application, the following will describe in detail the integration of a volumetric video-based live broadcast device in a terminal. That is, the terminal is used as the execution subject for detailed description.

Please refer to FIG. 1 . FIG. 1 is a schematic flowchart of a live broadcast method based on volumetric video provided by an embodiment of the present application. The live broadcast method based on volumetric video may include:

101. Acquire the live video data of the three-dimensional character model and the shooting video of the fused object.

Wherein, the live video in this embodiment of the present application may be a video generated based on a volumetric video. For example, the live video may be a live concert, and so on.

Wherein, the three-dimensional character model may be a character constructed based on volume video. For example, the three-dimensional character model may be a model constructed according to a singer.

Among them, volumetric video (also known as volumetric video, spatial video, volumetric 3D video or 6-DOF video, etc.) is a method that captures information in 3D space (such as depth information and color information, etc.) and generates a 3D model sequence. technology. Compared with traditional videos, volumetric videos add the concept of space to videos, and use 3D models to better restore the real 3D world, instead of using 2D flat videos and moving mirrors to simulate the sense of space in the real 3D world. Since the volumetric video is essentially a sequence of 3D models, users can adjust it to any viewing angle to watch according to their preferences, which has a higher degree of restoration and immersion than 2D flat video.

In an embodiment, in this application, the 3D model used to form the volumetric video can be reconstructed as follows:

First obtain the color images and depth images of different viewing angles of the subject, as well as the camera parameters corresponding to the color images; then, according to the acquired color images and their corresponding depth images and camera parameters, train the neural network that implicitly expresses the 3D model of the subject model, and extract the isosurface based on the trained neural network model, realize the 3D reconstruction of the shooting object, and obtain the 3D model of the shooting object.

It should be noted that there is no specific limitation on the architecture of the neural network model used in the embodiment of the present application, which can be selected by those skilled in the art according to actual needs. For example, a multilayer perceptron (MLP) without a normalization layer can be selected as the basic model for model training.

The 3D model reconstruction method provided by this application will be described in detail below.

First of all, multiple color cameras and depth cameras can be used simultaneously to take multi-angle shots of the target object (the target object is the subject) that needs to be reconstructed in three dimensions, and obtain color images of the target object at multiple different angles of view and the corresponding depth. Image, that is, at the same shooting time (the difference between the actual shooting time is less than or equal to the time threshold, the shooting time is considered to be the same), the color cameras of each viewing angle will capture the color image of the target object at the corresponding viewing angle, and correspondingly, the depth of each viewing angle The camera will capture the depth image of the target object at the corresponding viewing angle. It should be noted that the target object may be any object, including but not limited to living objects such as people, animals, and plants, or non-living objects such as machinery, furniture, and dolls.

In this way, the color images of the target object at different viewing angles have corresponding depth images, that is, when shooting, the color camera and the depth camera can adopt the configuration of the camera group, and the color camera at the same viewing angle cooperates with the depth camera to simultaneously shoot the same target object . For example, a studio can be built. The central area of the studio is the shooting area, and around the shooting area, multiple groups of color cameras and depth cameras are paired at certain angles in the horizontal and vertical directions. When the target object is in the shooting area surrounded by these color cameras and depth cameras, color images of the target object at different viewing angles and corresponding depth images can be obtained by shooting these color cameras and depth cameras.

In addition, the camera parameters of the color camera corresponding to each color image are further acquired. Among them, the camera parameters include the internal and external parameters of the color camera, which can be determined through calibration. The internal parameters of the camera are parameters related to the characteristics of the color camera itself, including but not limited to the focal length, pixels and other data of the color camera. The external parameters of the camera are the coordinates of the color camera in the world The parameters in the system, including but not limited to the position (coordinates) of the color camera and the rotation direction of the camera and other data.

As above, after acquiring multiple color images of different viewing angles and their corresponding depth images at the same shooting moment of the target object, three-dimensional reconstruction of the target object can be performed based on these color images and their corresponding depth images. Different from the method of converting depth information into point cloud for 3D reconstruction in related technologies, this application trains a neural network model to realize the implicit expression of the 3D model of the target object, so as to realize the 3D reconstruction of the target object based on the neural network model. Three-dimensional reconstruction.

Optionally, this application selects an MLP that does not include a normalization layer as the basic model, and performs training as follows:

Convert the pixels in each color image into rays based on the corresponding camera parameters;

Sampling multiple sampling points on the ray, and determining the first coordinate information of each sampling point and the directed distance (Signed Distance Field, SDF) value of each sampling point from the pixel point;

Input the first coordinate information of the sampling point into the basic model, and obtain the predicted SDF value and the predicted RGB color value of each sampling point output by the basic model;

Based on the first difference between the predicted SDF value and the SDF value, and the second difference between the predicted RGB color value and the RGB color value of the pixel, the parameters of the basic model are adjusted until a preset stop condition is met;

The basic model that satisfies the preset stop condition is used as a neural network model that implicitly expresses the 3D model of the target object.

First, based on the camera parameters corresponding to the color image, a pixel in the color image is converted into a ray, which can be a ray passing through the pixel and perpendicular to the color image surface; then, sampling multiple sampling points on the ray, The sampling process of the sampling points can be performed in two steps. You can first uniformly sample some sampling points, and then further sample multiple sampling points at key points based on the depth value of the pixel points, so as to ensure that as much as possible can be sampled near the surface of the model. Sampling point; then, calculate the first coordinate information of each sampling point in the world coordinate system obtained by sampling and the SDF value of each sampling point according to the camera parameters and the depth value of the pixel point, where the SDF value can be a pixel point The difference between the depth value of and the distance between the sampling point and the camera imaging surface. The difference is a signed value. When the difference is positive, it means that the sampling point is outside the 3D model. When the difference is negative When , it means that the sampling point is inside the 3D model, and when the difference is zero, it means that the sampling point is on the surface of the 3D model; then, after completing the sampling of the sampling point and calculating the SDF value corresponding to each sampling point, further divide The first coordinate information of the sampling point in the world coordinate system is input into the basic model (the basic model is configured to map the input coordinate information into SDF values and RGB color values and then output), and record the SDF value output by the basic model as the predicted SDF value , record the RGB color value output by the basic model as the predicted RGB color value; then, based on the first difference between the predicted SDF value and the SDF value corresponding to the sampling point, and the RGB color value of the pixel corresponding to the predicted RGB color value and the sampling point The second difference between the color values, making adjustments to the parameters of the base model.

In addition, for other pixels in the color image, the sampling points are also sampled in the same manner as above, and then the coordinate information of the sampling points in the world coordinate system is input to the basic model to obtain the corresponding predicted SDF value and predicted RGB color value for Adjust the parameters of the basic model until the preset stopping condition is met. For example, the preset stopping condition can be configured as the number of iterations of the basic model reaches the preset number, or the preset stopping condition can be configured as the basic model converges. When the iteration of the basic model satisfies the preset stop condition, a neural network model capable of accurately and implicitly expressing the three-dimensional model of the shooting object is obtained. Finally, the isosurface extraction algorithm can be used to extract the surface of the three-dimensional model of the neural network model, so as to obtain the three-dimensional model of the shooting object.

Optionally, in some embodiments, the imaging plane of the color image is determined according to camera parameters; and a ray passing through a pixel in the color image and perpendicular to the imaging plane is determined as a ray corresponding to the pixel.

Wherein, the coordinate information of the color image in the world coordinate system may be determined according to the camera parameters of the color camera corresponding to the color image, that is, the imaging plane may be determined. Then, it can be determined that the ray passing through the pixel in the color image and perpendicular to the imaging plane is the ray corresponding to the pixel.

Optionally, in some embodiments, the second coordinate information and the rotation angle of the color camera in the world coordinate system are determined according to the camera parameters; and the imaging plane of the color image is determined according to the second coordinate information and the rotation angle.

Optionally, in some embodiments, a first number of first sampling points are sampled at equal intervals on the ray; multiple key sampling points are determined according to the depth value of the pixel point, and a second number of second sampling points are sampled according to the key sampling points. Sampling points: determining the first number of first sampling points and the second number of second sampling points as a plurality of sampling points obtained by sampling on the ray.

Among them, first uniformly sample n (i.e. the first number) first sampling points on the ray, n is a positive integer greater than 2; then, according to the depth value of the aforementioned pixel points, determine the The preset number of key sampling points closest to the aforementioned pixel points, or determine the key sampling points from the n first sampling points that are less than the distance threshold from the aforementioned pixel points; then, re-sampling m according to the determined key sampling points For the second sampling point, m is a positive integer greater than 1; finally, the n+m sampling points obtained by sampling are determined as a plurality of sampling points obtained by sampling on the ray. Among them, sampling m more sampling points at key sampling points can make the training effect of the model more accurate on the surface of the 3D model, thereby improving the reconstruction accuracy of the 3D model.

Optionally, in some embodiments, the depth value corresponding to the pixel is determined according to the depth image corresponding to the color image; the SDF value of each sampling point from the pixel is calculated based on the depth value; the SDF value of each sampling point is calculated according to the camera parameters and the depth value Point coordinate information.

Among them, after sampling multiple sampling points on the ray corresponding to each pixel point, for each sampling point, determine the distance between the shooting position of the color camera and the corresponding point on the target object according to the camera parameters and the depth value of the pixel point , and then calculate the SDF value of each sampling point one by one based on the distance and calculate the coordinate information of each sampling point.

It should be noted that after the training of the basic model is completed, for the coordinate information of any given point, the corresponding SDF value can be predicted by the basic model that has completed the training, and the predicted SDF value represents the relationship between the point and The positional relationship (internal, external or surface) of the 3D model of the target object realizes the implicit expression of the 3D model of the target object, and obtains a neural network model for implicitly expressing the 3D model of the target object.

Finally, extract the isosurface of the above neural network model. For example, you can use the isosurface extraction algorithm (Marching cubes, MC) to draw the surface of the 3D model to obtain the surface of the 3D model, and then obtain the 3D surface of the target object based on the surface of the 3D model. Model.

The 3D reconstruction solution provided by this application uses the neural network to implicitly model the 3D model of the target object, and adds depth information to improve the speed and accuracy of model training. Using the 3D reconstruction scheme provided by this application, the 3D reconstruction of the photographed object is carried out continuously in time sequence, and the 3D model of the photographed object at different moments can be obtained. Volumetric video captured by the subject. In this way, "volume video shooting" can be performed on any subject to obtain a volume video presented with specific content. For example, it is possible to shoot a volumetric video of a dancing subject to obtain a volumetric video that can be watched from any angle. It is possible to shoot a volumetric video of a teaching subject to obtain a volumetric video that can be viewed from any angle. etc.

It should be noted that the volumetric videos involved in the following embodiments of the present application can be captured by the above volumetric video shooting methods.

Wherein, the fused objects may include users watching live video. In an embodiment, the embodiments of the present application may support the integration of users who watch the live video and the live video, so as to improve the interactivity of the live video. For example, a user can shoot his own video, and then the video of the watching user can be fused with the live video.

In an embodiment, before the live video is played or when the live video is played, the user may request to merge the video of the watching user with the live video.

For example, before the live video is played, the video playback interface may include a fusion request control, and the user may integrate the process of watching the live video with the live video by triggering the fusion request control. After the user triggers the integration request control, the user's terminal device can collect the user's shooting video data, and then, the terminal device can transmit the user's shooting video data to the live broadcast terminal. Then, when playing the live video, the live terminal can acquire the live video data of the 3D character model and the shot video of the fused object.

In an embodiment, since there are many possibilities to watch the live video, the number of users that can be integrated into the background of the live video is limited. For example, the number of users that can be integrated in the background of the live video is 100. Therefore, when the preset fusion positions in the live video are all reserved, the fusion request control on the video playback interface will be disabled, thereby avoiding too many fusion objects and affecting the quality of the live broadcast.

102. Perform image recognition processing on the captured video of the fused object to obtain a virtual image of the fused object.

In an embodiment, the user can also choose whether to directly fuse the captured video with the live video, or to fuse with the live video in the form of an avatar.

For example, after the user triggers the fusion request control, the avatar generation control and the fusion selection control may be displayed on the video playback interface.

When the user selects the fusion selection control, the live broadcast device can directly fuse the user's shot video with the live video. When the user selects the avatar generation control, the avatar of the fused object can be generated according to the captured video of the fused object.

Among them, there are many ways to generate the virtual image of the fused object.

For example, the live broadcast device can generate multiple avatar templates in advance, and the user can select the avatar template he needs to generate his own avatar.

For another example, the live broadcast device can intelligently generate the avatar of the fusion object according to the video shot by the user. For example, the outline image of the fused object can be recognized from the captured video, and then the style conversion can be performed on the outline object of the fused object to obtain the virtual image of the fused object. Specifically, the step of "performing image recognition processing on the captured video of the fused object to obtain the virtual image of the fused object" may include:

Perform frame division processing on the captured video to obtain at least one video frame of the captured video;

Perform contour recognition on the video frame to obtain the contour image of the fused object;

Perform style conversion on the outline image of the fused object to obtain the virtual image of the fused object.

In an embodiment, frame division processing may be performed on the shot video to obtain at least one video frame of the shot video. For example, an open source computer vision library (Open Source Computer Vision Library, openCV) can be used to perform frame division processing on the captured video to obtain at least one video frame of the captured video. Among them, openCV is a cross-platform computer vision and machine learning software library, which can run on multiple operating systems, and provides interfaces of multiple programming languages, and realizes many general algorithms in the direction of image processing and computer vision.

In an embodiment, after obtaining at least one video frame of the captured video, contour recognition may be performed on the video frame to obtain a contour image of the fused object. Among them, there are many methods to perform contour recognition on video frames to obtain contour images of fused objects.

For example, artificial intelligence algorithms can be used to perform contour recognition on video frames to obtain contour images of fused objects. For example, artificial intelligence algorithms such as Convolutional Neural Networks (CNN), De-Convolutional Networks (DN) or Deep Neural Networks (DNN) can be used to perform contour recognition on video frames , to obtain the contour image of the fused object.

For another example, the contour recognition of the video frame may be performed through pixels to obtain the contour image of the fused object. For example, pixel information of a video frame can be extracted. Then, contour detection can be performed on the fused object in the video frame according to the pixel point information to obtain the position information of the contour pixel point. Then, the contour image of the fused object can be cropped according to the position information of the contour pixels.

In an embodiment, style conversion may be performed on the outline image of the fused object to obtain the avatar of the fused object. Wherein, performing style conversion on the outline image of the fused object may refer to converting the image of the fused object in the outline image into an avatar. For example, you can convert the image of the fused object in the outline image into an anime style. For another example, the image of the fused object in the outline image can be converted into an oil painting style, and so on. In an embodiment, an artificial intelligence algorithm may be used to perform style conversion on the outline image of the fused object to obtain the avatar of the fused object. For example, artificial intelligence algorithms such as CNN or DNN can be used to perform style conversion on the outline image of the fused object to obtain the virtual image of the fused object.

103. Analyzing the live video data of the 3D character model to obtain background information of the live video data.

Wherein, the background information of the live video data may include information equivalent to a three-dimensional character model in the live video data. For example, in live video data, a 3D character model may belong to foreground information, while information other than the 3D character model may be background information.

In an embodiment, the live video data may be analyzed to obtain the background information of the live video data. For example, foreground separation may be performed on video frames of live video to obtain background information of live video data. Wherein, performing foreground separation on the video frames of the live video may refer to separating the three-dimensional character model and the background information in the live video. For example, artificial intelligence technology can be used to separate the 3D character model and background information in the live video.

104. Perform fusion processing on the virtual image of the fusion object and the background information of the live video data to obtain the fused live video data.

In one embodiment, in order to improve the interactivity and interest of the volumetric video, the avatar of the fused object and the background information of the live video data may be fused to obtain fused live video data.

In an embodiment, in order to ensure the quality of the live video, objects that can be fused in the background of the live video are limited, therefore, the background of the live video may include at least one preset fusion position. Then, the virtual image of the fused object and the preset fused position in the background video can be fused to obtain fused live video data.

Among them, there are many ways to fuse the virtual image of the fusion object and the background information of the live video data to obtain the fused live video data.

For example, the avatar of the fused object and the background information of the live video data can be randomly fused to obtain fused live video data. For example, a fusion position may be randomly assigned to the virtual image of the fusion object, and then the virtual information of the fusion object may be fused into the background information according to the fusion position.

In an embodiment, the fusion object and the background information may also be fused according to the level information of the fusion object. Specifically, the step of "fusion processing the virtual image of the fused object and the background information of the live video data to obtain the fused live video data" may include:

Obtain the level information of the fusion object;

Determining the fusion position of the virtual image of the fusion object in the background information of the live video data based on the level information of the fusion object;

According to the fused position of the avatar, the avatar and the background information of the live video data are fused to obtain the fused live video data.

Wherein, the level information of the fusion object may include the account level of the fusion object, the amount paid by the predetermined fusion position, and so on. For example, the higher the amount paid out when the fused object is scheduled to be fused, the higher its level will be. For another example, the higher the account level of the fusion object is, the higher its level is, and so on.

Then, the fused position of the avatar of the fused object in the background information of the live video data can be determined according to the level information of the fused object. Specifically, the step of "determining the fusion position of the virtual image of the fusion object in the background information of the live video data based on the level information of the fusion object" may include:

Matching the level information of the fusion object with the level information of at least one preset fusion position in the background information to obtain a matching result;

The fusion position of the avatar in the background information of the live video data is determined in at least one preset fusion position based on the matching result.

For example, the preset fusion positions in the background information all have corresponding level information. For example, the closer the preset fusion position is to the three-dimensional character model, the higher the level is. Conversely, the farther away from the 3D character model the lower the level of the preset fusion position.

In an embodiment, the level information of the fused object may be matched with the level information of at least one preset fused position in the background information to obtain a matching result. Then, the fusion position of the avatar in the background information of the live video data may be determined in at least one preset fusion position based on the matching result. For example, the level information of the fused object may be compared with the level information of the preset fused position. If the level information of the fused object is the same as the level information of the preset fused position, it means that the fused object matches the preset fused position, and then the preset fused position can be determined as the fused position of the fused object.

Then, the avatar and the background information of the live video data may be fused according to the fused position of the avatar to obtain the fused live video data.

105. Obtain viewing angle information, and determine display content of the 3D character model and the avatar of the fused object in the fused live video data based on the viewing angle information.

In an embodiment, after the fused live video data is generated, the live video based on the 3D character model can be displayed in 360 degrees. Therefore, the viewer can change the displayed content of the live video by adjusting his viewing angle. For example, as shown in FIG. 2 , 001 may be the live content viewed by the audience through the first viewing angle, and 002 may be the live content viewed by the audience through the second viewing angle. Wherein, the first viewing angle and the second viewing angle are different viewing angles, so the content displayed in the live video is also different. Since the user can adjust his own viewing angle, the user's viewing angle information can be obtained, and based on the viewing angle information, the display content of the 3D character model and the avatar of the fused object in the fused live video data can be determined.

Wherein, the viewing angle information may refer to the change between the user's original viewing angle and the current viewing angle, that is, the angle change between the first viewing angle and the second viewing angle. For example, as shown in FIG. 2 , the first viewing angle and the second viewing angle in the image change by 180 degrees.

In one embodiment, the step of "determining the display content of the 3D character model and the avatar of the fused object in the fused live video data based on the viewing angle information" may include:

Perform angle calculation processing on the viewing angle information to obtain the viewing angle;

Map the viewing angle to the fused live video data to obtain the display range corresponding to the live video data;

The display content of the 3D character model and the avatar of the fused object in the fused live video data is determined according to the display range.

In an embodiment, angle calculation processing may be performed on the viewing angle information to obtain the viewing angle. For example, the viewing angle information may be used to describe the angle change between the first angle of view and the second angle of view. Then, an arithmetic operation may be performed on the angle information corresponding to the first viewing angle and the angle change to obtain the viewing angle. For example, the viewing angle may be obtained by adding or subtracting the angle information corresponding to the first viewing angle and the viewing angle change.

In an embodiment, the viewing angle may be mapped to the fused live video data to obtain a display range corresponding to the live video data. For example, there is a pre-set three-dimensional coordinate in the live video. The viewing angle may be mapped to the three-dimensional coordinates, and then the display range corresponding to the target video data is determined through the three-dimensional coordinates.

Then, the display content of the 3D character model and the avatar of the fused object in the fused live video data may be determined according to the display range.

106. Display the fused live video data based on the display content.

In an embodiment, after the display content of the 3D character model and the avatar of the fused object in the fused live video data is determined, the fused live video data may be displayed based on the display content.

In an embodiment, in order to further improve the interest of the live video, the avatar in the fused live video data can also be changed following the movement of the 3D character model. For example, when the hand of the 3D character model moves up, the avatar in the fused live video data will follow the hand of the 3D character model to rise. For another example, when the 3D character model turns around, the avatar in the fused live video data will follow the 3D character model to rotate.

Specifically, this embodiment of the application may also include:

Detect the movement of the 3D character model;

When it is detected that the action of the three-dimensional character model matches the preset trigger action, extracting the avatar control logic corresponding to the preset trigger action;

According to the control logic of the virtual image, the position information of the live video data after fusion of the virtual image is adjusted and processed.

In an embodiment, the motion of the 3D character model in the fused live video data may be detected. By detecting the actions of the three-dimensional character model, it is possible to know what actions the three-dimensional character model has performed.

In one embodiment, it may be preset which action of the three-dimensional character model will cause the position of the avatar to change. Therefore, the action of the three-dimensional character model can be matched with the preset trigger action.

Wherein, when it is detected that the action of the three-dimensional character model matches the preset trigger action, the avatar control logic corresponding to the preset trigger action may be extracted. Wherein, the avatar control logic may be used to describe how the position of the avatar should be changed. For example, the avatar control logic may indicate that when the hand of the three-dimensional character model rises, the position of the avatar will follow the hand of the three-dimensional character model. For another example, the avatar control logic may indicate that when the hand of the 3D character model falls, the position of the avatar will follow the hand of the 3D character model.

Then, the location information of the fused live video data of the avatar is adjusted according to the avatar control logic. Specifically, the step of "adjusting and processing the location information of the fused live video data of the avatar according to the avatar control logic" may include:

According to the action of the 3D character model, calculate the dynamic acceleration;

According to the virtual image control logic, the dynamic acceleration is transformed into noise information acting on the virtual image;

The noise information is added to the avatar so as to adjust the position information of the fused live video data of the avatar.

In an embodiment, when detecting the motion of the three-dimensional character model, the speed of change of the motion of the three-dimensional character model may also be detected. Then, the dynamic change acceleration can be calculated according to the movement change speed of the three-dimensional character model. By dynamically changing the acceleration, the position of the avatar can be changed.

In one embodiment, the dynamically changing acceleration can be converted into noise information acting on the avatar according to the avatar control logic. Wherein, the noise information can change the position of the avatar. For example, the noise information may be Perlin noise (Per l i n noise). Wherein, according to the virtual image control logic, the dynamic acceleration can be converted into noise information acting on the virtual image. For example, when the control logic of the avatar is that the hand of the 3D character model rises, the position of the avatar will follow the rise of the hand of the 3D character model, and noise information in an upward direction can be generated. By adding the noise information to the avatar, the position information of the fused live video data of the avatar can be adjusted and processed.

In one embodiment, in order to further improve the interactivity between watching live video and live video, this embodiment of the present application can also support users watching live video to operate their avatar freely in the scene based on cloud rendering technology. Mobile, enabling thousands of people to watch the concert.

For example, it can be judged whether the user of the live video is qualified to control his avatar to move freely in the scene according to the level of the user watching the live video. For example, the grades of users who watch the live video may include grades 1 to 5. Among them, users at level 5 can control their avatars to move in the scene by themselves. However, users of levels 1 to 4 cannot control their avatars to move in the scene by themselves.

For another example, according to the position of the avatar of the user watching the live video in the live video, it can be judged whether the user of the live video can control his avatar to move freely in the scene. For example, it is possible to limit that the avatars at certain positions in the live video background can move, and users at these positions can operate their avatars to move.

As can be seen from the above, in the embodiment of the present application, the live video data of the 3D character model and the shooting video of the fusion object can be obtained; image recognition processing is performed on the shooting video of the fusion object to obtain the virtual image of the fusion object; The live video data is analyzed to obtain the background information of the live video data; the avatar of the fusion object and the background information of the live video data are fused to obtain the fused live video data; the viewing angle information is obtained, and the fused image is determined based on the viewing angle information. The display content of the 3D character model and the avatar of the fused object in the live video data; displaying the fused live video data based on the display content can support the interaction between the viewing object and the 3D character model in the live scene, thereby improving the interaction of the volumetric video sex and fun.

In order to facilitate better implementation of the volumetric video-based live broadcast method provided by the embodiment of the present application, the embodiment of the present application further provides a device based on the volumetric video-based live broadcast method. The meanings of the nouns are the same as those in the volumetric video-based live broadcast method above, and for specific implementation details, please refer to the descriptions in the method embodiments.

For example, as shown in Figure 3, the live broadcast device based on volumetric video may include:

An acquisition unit 301, configured to acquire the live video data of the 3D character model and the captured video of the fused object;

An image recognition unit 302, configured to perform image recognition processing on the shot video of the fusion object to obtain the virtual image of the fusion object;

An analysis unit 303, configured to analyze the live video data of the 3D character model to obtain the background information of the live video data;

A fusion unit 304, configured to perform fusion processing on the avatar of the fusion object and the background information of the live video data to obtain fused live video data;

The determination unit 305 is configured to obtain viewing angle information, and determine the display content of the 3D character model in the fused live video data and the avatar of the fused object based on the viewing angle information;

The display unit 306 is configured to display the fused live video data based on the display content.

In one embodiment, the image recognition unit 302 may include:

A frame division processing subunit, configured to perform frame division processing on the captured video to obtain at least one video frame of the captured video;

A contour recognition subunit, configured to perform contour recognition on the video frame to obtain a contour image of the fused object;

The style conversion subunit is configured to perform style conversion on the outline image of the fusion object to obtain the virtual image of the fusion object.

In an embodiment, the fusion unit 304 may include:

an information acquisition subunit, configured to acquire level information of the fusion object;

A position determining subunit, configured to determine the fusion position of the avatar of the fusion object in the background information of the live video data based on the level information of the fusion object;

The fusion subunit is configured to perform fusion processing on the background information of the virtual image and the live video data according to the fusion position of the virtual image to obtain fused live video data.

In an embodiment, the position determination subunit may include:

A matching module, configured to match the level information of the fusion object with the level information of at least one preset fusion position in the background information to obtain a matching result;

A position determining module, configured to determine a fusion position of the avatar in the background information of the live video data in the at least one preset fusion position based on the matching result.

In an embodiment, the determining unit 305 may include:

An angle calculation subunit, configured to perform angle calculation processing on the viewing angle information to obtain a viewing angle;

A mapping subunit, configured to map the viewing angle to the fused live video data to obtain a display range corresponding to the live video data;

The content determination subunit is configured to determine the display content of the 3D character model and the avatar of the fused object in the fused live video data according to the display range.

In an embodiment, the live broadcast device may further include:

a detection unit, configured to detect the actions of the three-dimensional character model;

An extracting unit, configured to extract the avatar control logic corresponding to the preset trigger action when it is detected that the action of the three-dimensional character model matches the preset trigger action;

The adjusting unit is configured to adjust the location information of the avatar in the fused live video data according to the avatar control logic.

In an embodiment, the adjustment unit may include:

A calculation subunit, configured to calculate the dynamic acceleration according to the action of the three-dimensional character model;

A conversion subunit, configured to convert the dynamic change acceleration into noise information acting on the avatar according to the avatar control logic;

The adding subunit is configured to add the noise information to the avatar, so as to adjust the position information of the avatar in the fused live video data.

During specific implementation, each of the above modules can be implemented as an independent entity, or can be combined arbitrarily as the same or several entities. For the specific implementation of each of the above modules and the corresponding beneficial effects, please refer to the previous method embodiments. I won't repeat them here.

The embodiment of the present application also provides an electronic device, which may be a server or a terminal, etc., as shown in FIG. 4 , which shows a schematic structural diagram of the electronic device involved in the embodiment of the present application. Specifically:

The electronic device may include a processor 601 of one or more processing cores, a memory 602 of one or more computer-readable storage media, a power supply 603, an input unit 604 and other components. Those skilled in the art can understand that the structure of the electronic device shown in FIG. 4 does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine some components, or arrange different components. in:

The processor 601 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device, by running or executing computer programs and/or modules stored in the memory 602, and calling the Data, perform various functions of electronic devices and process data. Optionally, the processor 601 may include one or more processing cores; preferably, the processor 601 may integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces, and application programs, etc. , the modem processor mainly handles wireless communications. It can be understood that the foregoing modem processor may not be integrated into the processor 601 .

The memory 602 can be used to store computer programs and modules, and the processor 601 executes various functional applications and data processing by running the computer programs and modules stored in the memory 602 . The memory 602 can mainly include a program storage area and a data storage area, wherein the program storage area can store operating systems, computer programs required by at least one function (such as sound playback function, image playback function, etc.); Data created by the use of electronic devices, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 602 may further include a memory controller to provide the processor 601 with access to the memory 602 .

The electronic device also includes a power supply 603 for supplying power to various components. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, so that functions such as charging, discharging, and power consumption management can be implemented through the power management system. The power supply 603 may also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

The electronic device can also include an input unit 604, which can be used to receive input numbers or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

Although not shown, the electronic device may also include a display unit, etc., which will not be repeated here. Specifically, in this embodiment, the processor 601 in the electronic device will load the executable file corresponding to the process of one or more computer programs into the memory 602 according to the following instructions, and the processor 601 will run the executable file stored in the The computer program in memory 602, thereby realizes various functions, such as:

Obtain the live video data of the 3D character model and the shooting video of the fused object;

performing image recognition processing on the shot video of the fusion object to obtain the virtual image of the fusion object;

Analyzing the live video data of the three-dimensional character model to obtain the background information of the live video data;

performing fusion processing on the avatar of the fusion object and the background information of the live video data to obtain the fused live video data;

Obtain viewing angle information, and determine the display content of the 3D character model in the fused live video data and the avatar of the fused object based on the viewing angle information;

The fused live video data is displayed based on the display content.

For the specific implementation manners of the above operations and the corresponding beneficial effects, refer to the detailed description of the live broadcast method based on volumetric video above, and details are not repeated here.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above-mentioned embodiments can be completed by a computer program, or by controlling related hardware through a computer program, and the computer program can be stored in a computer-readable storage media and is loaded and executed by the processor.

To this end, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored, and the computer program can be loaded by a processor to execute any volumetric video-based live broadcast method provided in the embodiment of the present application. in the steps. For example, the computer program can perform the following steps:

Obtain the live video data of the 3D character model and the shooting video of the fused object;

performing image recognition processing on the shot video of the fusion object to obtain the virtual image of the fusion object;

Analyzing the live video data of the three-dimensional character model to obtain the background information of the live video data;

performing fusion processing on the avatar of the fusion object and the background information of the live video data to obtain the fused live video data;

Obtain viewing angle information, and determine the display content of the 3D character model in the fused live video data and the avatar of the fused object based on the viewing angle information;

The fused live video data is displayed based on the display content.

For the specific implementation manners of the above operations and the corresponding beneficial effects, reference may be made to the foregoing embodiments, and details are not repeated here.

Wherein, the computer-readable storage medium may include: a read-only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and the like.

Because the computer program stored in the computer-readable storage medium can execute the steps in any volume video-based live broadcast method provided in the embodiments of the present application, therefore, any one of the methods provided in the embodiments of the present application can be realized. For the beneficial effects that can be achieved by the live broadcast method based on volumetric video, refer to the previous embodiments for details, and will not be repeated here.

Wherein, according to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above volumetric video-based live broadcast method.

The volumetric video-based live broadcast method, device, electronic equipment, and storage medium provided by the embodiments of the present application are described above in detail. In this paper, specific examples are used to illustrate the principles and implementation methods of the present application. The above embodiments The description is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation and application scope, in summary , the contents of this specification should not be construed as limiting the application.

Claims (10)

1. A live broadcast method based on volume video, characterized in that, comprising: Obtain the live video data of the 3D character model and the shooting video of the fused object; performing image recognition processing on the shot video of the fusion object to obtain the virtual image of the fusion object; Analyzing the live video data of the three-dimensional character model to obtain the background information of the live video data; performing fusion processing on the avatar of the fusion object and the background information of the live video data to obtain the fused live video data; Obtain viewing angle information, and determine the display content of the 3D character model in the fused live video data and the avatar of the fused object based on the viewing angle information; The fused live video data is displayed based on the display content. 2. The method according to claim 1, wherein said performing image recognition processing on the shot video of said fusion object to obtain the virtual image of said fusion object comprises: performing frame division processing on the captured video to obtain at least one video frame of the captured video; Perform contour recognition on the video frame to obtain a contour image of the fusion object; Performing style conversion on the outline image of the fused object to obtain a virtual image of the fused object. 3. The method according to claim 1, wherein said performing fusion processing on the avatar of the fusion object and the background information of the live video data to obtain the fused live video data comprises: Obtaining level information of the fusion object; determining the fusion position of the avatar of the fusion object in the background information of the live video data based on the level information of the fusion object; The avatar and the background information of the live video data are fused according to the fused position of the avatar to obtain fused live video data. 4. The method according to claim 3, wherein the determining the fusion position of the avatar of the fusion object in the background information of the live video data based on the level information of the fusion object comprises: matching the level information of the fusion object with the level information of at least one preset fusion position in the background information to obtain a matching result; A fusion position of the avatar in the background information of the live video data is determined in the at least one preset fusion position based on the matching result. 5. The method according to claim 1, wherein the determination of the display content of the three-dimensional character model in the live video data after the fusion and the avatar of the fusion object based on the viewing angle information includes: performing angle calculation processing on the viewing angle information to obtain a viewing angle; Mapping the viewing angle to the fused live video data to obtain a display range corresponding to the live video data; The display content of the 3D character model in the fused live video data and the avatar of the fused object is determined according to the display range. 6. The method according to claim 1 or 5, characterized in that the method further comprises: Detecting the actions of the three-dimensional character model; When it is detected that the action of the three-dimensional character model matches the preset trigger action, extracting the avatar control logic corresponding to the preset trigger action; Adjusting the location information of the avatar in the fused live video data according to the avatar control logic. 7. The method according to claim 6, characterized in that, according to the avatar control logic, the adjustment processing of the position information of the avatar in the live video data after the fusion includes: Calculating the dynamic change acceleration according to the action of the three-dimensional character model; According to the avatar control logic, the dynamic change acceleration is converted into noise information acting on the avatar; The noise information is added to the avatar, so as to adjust the position information of the avatar in the fused live video data. 8. A live broadcast device based on volumetric video, comprising: An acquisition unit, configured to acquire the live video data of the 3D character model and the shooting video of the fused object; An image recognition unit, configured to perform image recognition processing on the captured video of the fusion object to obtain the virtual image of the fusion object; An analysis unit, configured to analyze the live video data of the 3D character model to obtain the background information of the live video data; A fusion unit, configured to fuse the avatar of the fusion object and the background information of the live video data to obtain fused live video data; A determining unit, configured to obtain viewing angle information, and determine the display content of the 3D character model in the fused live video data and the avatar of the fused object based on the viewing angle information; A display unit, configured to display the fused live video data based on the display content. 9. An electronic device, characterized in that it comprises a processor and a memory, the memory stores a computer program, and the processor is used to run the computer program in the memory to execute any one of claims 1 to 7. The volumetric video-based live broadcast method described above. 10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the method described in any one of claims 1 to 7. Volumetric Video-Based Live Streaming Method.

Images