CN116129526A - Method and device for controlling photographing, electronic equipment and storage medium - Google Patents

Abstract

The present application provides a co-production control method, device, electronic equipment, and computer-readable storage medium. The co-shooting control method includes: presenting a shooting picture containing a user character; performing gesture recognition on the user character in the shooting picture to obtain the current gesture of the user character; if the current gesture is consistent with the preset position control gesture match, obtain the pointing position of the current gesture; place a virtual object in the shooting frame according to the pointing point, so as to control the relative position of the virtual object and the user character in the shooting frame. In this application, the user character can control and know the position of the virtual object, so that the user character can make actions and expressions that are more natural and coordinated with the virtual object when co-shooting with the virtual object, reducing the abruptness of the co-shooting picture, and making the co-shooting picture more effective nature.

Description

Co-production control method, device, electronic device and storage medium

technical field

The present application relates to the technical field of shooting processing, and in particular to a co-shooting control method, device, electronic equipment, and computer-readable storage medium.

Background technique

With the development of image capture technology, users can use electronic devices to capture various videos or images, and users have more and more diverse requirements for video or image capture. For example, users expect to be able to interact with virtual objects (such as volume video ) to capture a video or image.

However, the inventors of the embodiments of the present application found in the actual research and development process that during the co-shooting process, since the user character does not know the relative position of the virtual object in the screen, the actions and expressions of the user character and the virtual object are not coordinated. , resulting in a more abrupt picture effect in co-production.

Contents of the invention

The present application provides a co-pacing control method, device, electronic equipment, and computer-readable storage medium, which can enable the user character to control and know the position of the virtual object, and then make the user character coordinate with the virtual object more naturally when co-pacing with the virtual object The movements and expressions reduce the abruptness of the co-shooting pictures, making the co-shooting pictures more natural.

In a first aspect, the present application provides a co-pacing control method, the method comprising:

Present a shooting screen containing the user's character;

performing gesture recognition on the user character in the photographed picture to obtain the current gesture of the user character;

If the current gesture matches the preset position control gesture, then acquire the pointed position point of the current gesture;

The virtual object is placed in the shooting frame according to the pointing position point, so as to control the relative position of the virtual object and the user character in the shooting frame.

In a second aspect, the present application provides a co-pacing control device, the co-pacing control device includes:

a display unit, configured to present a shooting screen including a user character;

A recognition unit, configured to perform gesture recognition on the user character in the photographed picture to obtain the current gesture of the user character;

An acquisition unit, configured to acquire the pointing position point of the current gesture if the current gesture matches a preset position control gesture;

A control unit, configured to place a virtual object in the shooting frame according to the pointed position point, so as to control the relative positions of the virtual object and the user character in the shooting frame.

In some embodiments, the acquiring unit is specifically used for:

If the current gesture matches the preset position control gesture, then acquire the finger corresponding to the current gesture to point to a ray formed in the three-dimensional space where the shooting picture is located;

Obtain an intersection point between the ray and the supporting surface of the virtual object as the pointing position of the current gesture.

In some embodiments, before obtaining the intersection point between the ray and the supporting surface of the virtual object as the pointed position point of the current gesture, the obtaining bit unit is specifically used for:

adding the standing surface of the user character in the shooting picture to the set of candidate planes in the shooting picture;

Adding a plane whose angle between the shooting picture and the standing surface is smaller than a preset angle threshold to the set of candidate planes;

From each of the planes in the set of candidate planes, a plane that has an intersection point with the ray and is closest to a starting point of the ray is obtained as a supporting surface of the virtual object.

In some embodiments, the acquiring unit is specifically used for:

detecting the gesture type of the current gesture of the user character;

If the gesture type is a position control gesture and the current gesture matches a preset position control gesture, the pointing position point of the current gesture is acquired.

In some embodiments, the control unit is specifically used for:

If the gesture type is an orientation control gesture, then acquire the associated orientation of the current gesture;

According to the associated orientation, the relative orientation of the virtual object and the user character in the shooting screen is controlled.

In some embodiments, the control unit is specifically used for:

In response to the change of the orientation of the user character, the orientation of the virtual object is updated, so that the relative orientation of the user character and the virtual object remains as the associated orientation.

In some embodiments, the control unit is specifically used for:

If the gesture type is a distance control gesture, then obtain the associated distance of the current gesture;

According to the association distance, the relative distance between the virtual object and the user character in the photographing frame is controlled.

In some embodiments, the control unit is specifically used for:

In response to a touch operation on the shooting control, the shooting screen is shot to obtain a co-shooting video of the virtual object and the target of the user character.

In some embodiments, the control unit is specifically used for:

Responding to a touch operation on the shooting control, shooting the shooting screen to obtain a preliminary co-shooting video of the virtual object and the user character;

Perform control gesture recognition on the video frame in the preliminary co-shooting video to obtain a target video frame containing the control gesture;

The target video frame is filtered out from the preliminary co-shooting video to obtain the target co-shooting video.

In some embodiments, the control unit is specifically used for:

When the current gesture matches the preset control gesture, detect whether the camera of the shooting picture is in a shooting state;

If the camera is in the shooting state, then switch the camera from the shooting state to the pause state;

Switching the camera from a pause state to a shooting state until the virtual object is placed at the pointing position in the shooting frame.

In some embodiments, the virtual object is a three-dimensional model in a volumetric video, and the control unit is specifically used for:

The three-dimensional model in the volumetric video is placed in the shooting frame according to the pointed position point.

In a third aspect, the present application also provides an electronic device, the electronic device includes a processor and a memory, and a computer program is stored in the memory, and when the processor invokes the computer program in the memory, the computer program provided by the present application is executed. Any method of co-tempo control.

In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is loaded by a processor to execute the co-time control method.

This application obtains the current gesture of the user character by performing gesture recognition on the user character in the shooting screen; if the current gesture matches the preset position control gesture, then obtains the pointing position point of the current gesture; according to the pointing position point in the shooting screen Placing a virtual object allows the user character to control the position of the virtual object through gestures, so that the approximate position of the virtual object that the user character is in sync with, and then makes the user character take a more natural and coordinated action with the virtual object when in sync with the virtual object. Expressions reduce the abruptness of the co-shooting pictures and make the co-shooting pictures more natural.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can also be obtained based on these drawings without any creative effort.

FIG. 1 is a schematic diagram of a scene of a co-production control system provided by an embodiment of the present application;

FIG. 2 is a schematic flowchart of a co-pacing control method provided in an embodiment of the present application;

FIG. 3 is a schematic diagram of a comparison between the shooting picture and the shooting scene provided in the embodiment of the present application;

FIG. 4 is a schematic diagram of a scene of a shooting screen provided in an embodiment of the present application;

FIG. 5 is a schematic diagram of another scene of a shooting screen provided in an embodiment of the present application;

FIG. 6 is a schematic diagram of another scene of the shooting screen provided in the embodiment of the present application;

Fig. 7 is a schematic structural diagram of an embodiment of the co-pacing control device provided in the embodiment of the present application;

Fig. 8 is a schematic structural diagram of an embodiment of the electronic device provided in the embodiment of the present application.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the application with reference to the drawings in the embodiments of the application. Apparently, the described embodiments are only some of the embodiments of the application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts belong to the scope of protection of this application.

In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are only used for descriptive purposes, and cannot be interpreted as indicating or implying relative importance or implicitly indicating the indicated technical features. quantity. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of said features. In the description of the embodiments of the present application, "plurality" means two or more, unless otherwise specifically defined.

The following description is given to enable any person skilled in the art to make and use the application. In the following description, details are set forth for purposes of explanation. It should be understood that one of ordinary skill in the art would recognize that the present application may be practiced without these specific details. In other instances, well known procedures have not been described in detail to avoid obscuring the description of the embodiments of the present application with unnecessary detail. Therefore, the present application is not intended to be limited to the illustrated embodiments, but to be consistent with the broadest scope consistent with the principles and features disclosed in the embodiments of the present application.

Volumetric Video (also known as volumetric video, spatial video, volumetric 3D video or 6-DOF video, etc.) is a technology that captures information in 3D space (such as depth information and color information, etc.) and generates a 3D model sequence. Compared with traditional videos, volumetric videos add the concept of space to videos, and use 3D models to better restore the real 3D world, instead of using 2D flat videos and moving mirrors to simulate the sense of space in the real 3D world. Since the volumetric video is essentially a sequence of 3D models, users can adjust it to any viewing angle to watch according to their preferences, which has a higher degree of restoration and immersion than 2D flat video.

Optionally, in this application, the 3D model used to form the volumetric video can be reconstructed as follows:

First obtain the color images and depth images of different viewing angles of the subject, as well as the camera parameters corresponding to the color images; then, according to the acquired color images and their corresponding depth images and camera parameters, train the neural network that implicitly expresses the 3D model of the subject model, and extract the isosurface based on the trained neural network model, realize the 3D reconstruction of the shooting object, and obtain the 3D model of the shooting object.

It should be noted that there is no specific limitation on the architecture of the neural network model used in the embodiment of the present application, which can be selected by those skilled in the art according to actual needs. For example, a multilayer perceptron (MLP) without a normalization layer can be selected as the basic model for model training.

The 3D model reconstruction method provided by this application will be described in detail below.

First of all, multiple color cameras and depth cameras can be used simultaneously to take multi-angle shots of the target object (the target object is the subject) that needs to be reconstructed in three dimensions, and obtain color images of the target object at multiple different angles of view and the corresponding depth. Image, that is, at the same shooting time (the difference between the actual shooting time is less than or equal to the time threshold, the shooting time is considered to be the same), the color cameras of each viewing angle will capture the color image of the target object at the corresponding viewing angle, and correspondingly, the depth of each viewing angle The camera will capture the depth image of the target object at the corresponding viewing angle. It should be noted that the target object may be any object, including but not limited to living objects such as people, animals, and plants, or non-living objects such as machinery, furniture, and dolls.

In this way, the color images of the target object at different viewing angles have corresponding depth images, that is, when shooting, the color camera and the depth camera can adopt the configuration of the camera group, and the color camera at the same viewing angle cooperates with the depth camera to simultaneously shoot the same target object . For example, a studio can be built. The central area of the studio is the shooting area, and around the shooting area, multiple groups of color cameras and depth cameras are paired at certain angles in the horizontal and vertical directions. When the target object is in the shooting area surrounded by these color cameras and depth cameras, color images of the target object at different viewing angles and corresponding depth images can be obtained by shooting these color cameras and depth cameras.

In addition, the camera parameters of the color camera corresponding to each color image are further acquired. Among them, the camera parameters include the internal and external parameters of the color camera, which can be determined through calibration. The internal parameters of the camera are parameters related to the characteristics of the color camera itself, including but not limited to the focal length, pixels and other data of the color camera. The external parameters of the camera are the coordinates of the color camera in the world The parameters in the system, including but not limited to the position (coordinates) of the color camera and the rotation direction of the camera and other data.

As above, after acquiring multiple color images of different viewing angles and their corresponding depth images at the same shooting moment of the target object, three-dimensional reconstruction of the target object can be performed based on these color images and their corresponding depth images. Different from the method of converting depth information into point cloud for 3D reconstruction in related technologies, this application trains a neural network model to realize the implicit expression of the 3D model of the target object, so as to realize the 3D reconstruction of the target object based on the neural network model. Three-dimensional reconstruction.

Optionally, the present application selects a multilayer perceptron (Multilayer Perceptron, MLP) that does not include a normalization layer as the basic model, and trains as follows:

Convert the pixels in each color image into rays based on the corresponding camera parameters;

Sampling a plurality of sampling points on the ray, and determining the first coordinate information of each sampling point and the SDF value of each sampling point from the pixel point;

Input the first coordinate information of the sampling point into the basic model, and obtain the predicted SDF value and the predicted RGB color value of each sampling point output by the basic model;

Based on the first difference between the predicted SDF value and the SDF value, and the second difference between the predicted RGB color value and the RGB color value of the pixel, the parameters of the basic model are adjusted until a preset stop condition is met;

The basic model that satisfies the preset stop condition is used as a neural network model that implicitly expresses the 3D model of the target object.

First, based on the camera parameters corresponding to the color image, a pixel in the color image is converted into a ray, which can be a ray passing through the pixel and perpendicular to the color image surface; then, sampling multiple sampling points on the ray, The sampling process of the sampling points can be performed in two steps. You can first uniformly sample some sampling points, and then further sample multiple sampling points at key points based on the depth value of the pixel points, so as to ensure that as much as possible can be sampled near the surface of the model. Sampling point; then, calculate the first coordinate information of each sampling point in the world coordinate system obtained by sampling and the directional distance (Signed Distance Field, SDF) value of each sampling point according to the camera parameters and the depth value of the pixel point , where the SDF value can be the difference between the depth value of the pixel point and the distance between the sampling point and the imaging surface of the camera. The difference is a signed value. When the difference is positive, it means that the sampling point is in the 3D model When the difference is negative, it means that the sampling point is inside the 3D model, and when the difference is zero, it means that the sampling point is on the surface of the 3D model; then, after completing the sampling of the sampling point and calculating each sample After the SDF value corresponding to the point, the first coordinate information of the sampling point in the world coordinate system is further input into the basic model (the basic model is configured to map the input coordinate information into SDF value and RGB color value and then output), and the basic model The output SDF value is recorded as the predicted SDF value, and the RGB color value output by the basic model is recorded as the predicted RGB color value; then, based on the first difference between the predicted SDF value and the SDF value corresponding to the sampling point, and the predicted RGB color value The second difference between the RGB color value of the pixel corresponding to the sampling point and the parameter of the basic model is adjusted.

In addition, for other pixels in the color image, the sampling points are also sampled in the same manner as above, and then the coordinate information of the sampling points in the world coordinate system is input to the basic model to obtain the corresponding predicted SDF value and predicted RGB color value for Adjust the parameters of the basic model until the preset stopping condition is met. For example, the preset stopping condition can be configured as the number of iterations of the basic model reaches the preset number, or the preset stopping condition can be configured as the basic model converges. When the iteration of the basic model satisfies the preset stop condition, a neural network model capable of accurately and implicitly expressing the three-dimensional model of the shooting object is obtained. Finally, the isosurface extraction algorithm can be used to extract the surface of the three-dimensional model of the neural network model, so as to obtain the three-dimensional model of the shooting object.

Optionally, in some embodiments, the imaging plane of the color image is determined according to camera parameters; and a ray passing through a pixel in the color image and perpendicular to the imaging plane is determined as a ray corresponding to the pixel.

Wherein, the coordinate information of the color image in the world coordinate system may be determined according to the camera parameters of the color camera corresponding to the color image, that is, the imaging plane may be determined. Then, it can be determined that the ray passing through the pixel in the color image and perpendicular to the imaging plane is the ray corresponding to the pixel.

Optionally, in some embodiments, the second coordinate information and the rotation angle of the color camera in the world coordinate system are determined according to the camera parameters; and the imaging plane of the color image is determined according to the second coordinate information and the rotation angle.

Optionally, in some embodiments, a first number of first sampling points are sampled at equal intervals on the ray; multiple key sampling points are determined according to the depth value of the pixel point, and a second number of second sampling points are sampled according to the key sampling points. Sampling points: determining the first number of first sampling points and the second number of second sampling points as a plurality of sampling points obtained by sampling on the ray.

Among them, first uniformly sample n (i.e. the first number) first sampling points on the ray, n is a positive integer greater than 2; then, according to the depth value of the aforementioned pixel points, determine the The preset number of key sampling points closest to the aforementioned pixel points, or determine the key sampling points from the n first sampling points that are less than the distance threshold from the aforementioned pixel points; then, re-sampling m according to the determined key sampling points For the second sampling point, m is a positive integer greater than 1; finally, the n+m sampling points obtained by sampling are determined as a plurality of sampling points obtained by sampling on the ray. Among them, sampling m more sampling points at key sampling points can make the training effect of the model more accurate on the surface of the 3D model, thereby improving the reconstruction accuracy of the 3D model.

Optionally, in some embodiments, the depth value corresponding to the pixel is determined according to the depth image corresponding to the color image; the SDF value of each sampling point from the pixel is calculated based on the depth value; the SDF value of each sampling point is calculated according to the camera parameters and the depth value Point coordinate information.

Among them, after sampling multiple sampling points on the ray corresponding to each pixel point, for each sampling point, determine the distance between the shooting position of the color camera and the corresponding point on the target object according to the camera parameters and the depth value of the pixel point , and then calculate the SDF value of each sampling point one by one based on the distance and calculate the coordinate information of each sampling point.

It should be noted that after the training of the basic model is completed, for the coordinate information of any given point, the corresponding SDF value can be predicted by the basic model that has completed the training, and the predicted SDF value represents the relationship between the point and The positional relationship (internal, external or surface) of the 3D model of the target object realizes the implicit expression of the 3D model of the target object, and obtains a neural network model for implicitly expressing the 3D model of the target object.

Finally, extract the isosurface of the above neural network model. For example, you can use the isosurface extraction algorithm (Marching cubes, MC) to draw the surface of the 3D model to obtain the surface of the 3D model, and then obtain the 3D surface of the target object based on the surface of the 3D model. Model.

The 3D reconstruction solution provided by this application uses the neural network to implicitly model the 3D model of the target object, and adds depth information to improve the speed and accuracy of model training. Using the 3D reconstruction scheme provided by this application, the 3D reconstruction of the photographed object is carried out continuously in time sequence, and the 3D model of the photographed object at different moments can be obtained. Volumetric video captured by the subject. In this way, "volume video shooting" can be performed on any subject to obtain a volume video presented with specific content. For example, it is possible to shoot a volumetric video of a dancing subject to obtain a volumetric video that can be watched from any angle. It is possible to shoot a volumetric video of a teaching subject to obtain a volumetric video that can be viewed from any angle. etc.

It should be noted that the volume video involved in the following embodiments of the present application can be captured by the above volume video shooting method.

Embodiments of the present application provide a method, device, electronic device, and computer-readable storage medium for co-pacing control. Wherein, the co-shooting control device may be integrated in an electronic device, and the electronic device may be a server, or a terminal or other equipment.

The co-shooting control method of the embodiment of the present application can be applied to the production process and use of the volumetric video, for example, the subject (such as a performer) in the volumetric video is used as a virtual object, and the co-shooting is performed according to the co-shooting control method of the embodiment of the present application. Exemplarily, the production process and use of the volumetric video are roughly as follows:

Step 1: Shoot and collect

Performers enter the camera array system deployed in a matrix, through which professional-level acquisition equipment such as infrared IR cameras and 4K ultra-high-definition industrial cameras will capture and extract performers' color information, material information, depth information and other data.

The second step: material generation

After collecting the data, upload the materials to the cloud, and then the algorithm can be deployed on the cloud to automatically generate volumetric videos (3D dynamic character model sequences).

Step Three: Using Volumetric Video

Put the volumetric video into UE4/UE5/Unity 3D through plug-ins, perfectly integrate with virtual scenes or CG special effects, support real-time rendering, or be used for AR co-production, etc.

The executor of the co-pacing control method in the embodiment of the present application may be the co-pacing control device provided in the embodiment of the present application, or a server device, a physical host, or a user equipment (User Equipment, UE) and other electronic devices of different types integrated with the co-pacing control device. , wherein, the co-production control device can be realized by hardware or software, and the UE can specifically be a terminal device such as a smart phone, a tablet computer, a notebook computer, a palmtop computer, a desktop computer or a personal digital assistant (Personal Digital Assistant, PDA). The electronic device can work independently or in a cluster, and the electronic device can integrate a camera or establish a network connection with the camera to realize the shooting of the shooting scene and form a shooting picture; the electronic device can also Integrate the display screen or establish a network connection with the display screen to display the shooting picture during the shooting process.

For example, the co-pacing control method provided in the embodiment of the present application may be applied to the co-pacing control system shown in FIG. 1 . Wherein, the co-production control system includes a terminal 101 and a server 102. The terminal 101 may be a device including both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing bidirectional communication on a bidirectional communication link. The terminal 101 can specifically be a terminal equipped with a camera, such as a mobile phone, a tablet computer, a notebook computer, etc., and is used to capture a shooting scene to obtain a shooting picture; the terminal 101 can also specifically be a camera installed on the shooting site to complete co-shooting picture shooting. The terminal 101 and the server 102 can carry out two-way communication through the network. The server 102 can be an independent server, or a server network or server cluster composed of servers, which includes but is not limited to computers, network hosts, single network servers, and multiple network servers. A cloud server composed of a set or multiple servers. Wherein, the cloud server is composed of a large number of computers or network servers based on cloud computing (Cloud Computing). Wherein, the server 102 may also include a display screen for displaying a shooting picture. The terminal 101 and the server 102 can jointly implement the co-shooting control method. For example, the terminal 101 can send a shooting picture to the server 102, and the server 102 can thus present a shooting picture containing a user character; Gesture recognition, obtaining the current gesture of the user character; if the current gesture matches the preset position control gesture, obtaining the pointing position point of the current gesture; placing the pointing position point in the shooting picture according to the pointing position point The virtual object is used to control the relative position of the virtual object and the user character in the shooting frame.

Those skilled in the art can understand that the application environment shown in Figure 1 is only an application scenario related to the solution of this application, and does not constitute a limitation on the application scenario of the solution of this application. More or fewer computer devices are shown. For example, only one server 102 is shown in FIG. 1 . It can be understood that the co-production control system may also include one or more other servers, which are not specifically limited here.

It should also be noted that the scene schematic diagram of the co-production control system shown in FIG. The limitations of the technical solutions provided by the embodiments of the present invention, those skilled in the art know, with the evolution of the co-production control system and the emergence of new business scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

Next, let’s start to introduce the co-shooting control method provided by the embodiment of the present application. In the embodiment of the present application, an electronic device is used as the execution subject, and the electronic device integrates a camera and a display screen for illustration. The execution body will be omitted.

Referring to FIG. 2 , FIG. 2 is a schematic flowchart of a co-pacing control method provided in an embodiment of the present application. It should be noted that although a logical order is shown in the flowchart shown in FIG. 2 or other figures, in some cases, the steps shown or described may be performed in a different order than here. The co-production control method includes steps 201-204, wherein:

201. Present a shooting screen including a user character.

Wherein, the user role refers to a user who performs co-production with the virtual object. For example, photographing people in a scene.

Wherein, the virtual object refers to an object that does not exist in the shooting scene but appears in the shooting screen and is in harmony with the user character. For example, as shown in FIG. 3 , the virtual object is a puppy, and the puppy does not exist in the shooting scene (shown in the dotted frame in FIG. 3 ), but the puppy appears in the shooting screen.

Wherein, the virtual object may specifically be a three-dimensional model in the volumetric video, or may also be a two-dimensional model.

Wherein, the shooting scene refers to a real scene where the user character is shooting.

Wherein, the shooting picture refers to a picture formed by capturing a picture of a shooting scene. The shooting picture can specifically be the picture captured when the camera is turned on but not officially entering the shooting state, or it can be the picture captured when the camera is turned on and the shooting state is officially in progress (such as after the user presses the "start shooting" button).

For example, when the camera of the electronic device is turned on, the camera will capture the current shooting scene to form a shooting picture, and the display screen of the electronic device will present the shooting picture including the user character. In some embodiments, when the camera of the electronic device is turned on, the virtual object can also be presented at the same time, that is, in step 201, the captured picture including the user character and the virtual object is presented simultaneously. In some other embodiments, the virtual object may also be presented only when the camera is officially shooting, that is, in step 201 , a shooting picture containing the user character but not containing the virtual object may be presented.

202. Perform gesture recognition on the user character in the captured image to obtain a current gesture of the user character.

Wherein, the current gesture refers to a gesture of the user character obtained by performing gesture recognition on the user character. For example, the current gesture of the user character may be "stretch the index finger, bend the other four fingers", "stretch the five fingers together", and so on.

Exemplarily, first, a screenshot of the shooting screen presented in step 201 can be taken to obtain a shooting screen image; then, a gesture recognition algorithm is performed based on the shooting screen image to obtain the current gesture of the user character.

For example, first, based on the training data set (including multiple sample images, and labeling the user's hand area in each sample image, and the gesture category corresponding to the hand area in each sample image), the preset gesture recognition algorithm is Training, so that the trained gesture recognition algorithm learns the characteristics of various gestures, so as to obtain the trained gesture recognition algorithm (suitable for detecting the hand area in the image and determining the gesture category corresponding to the hand area in the image). Among them, the preset gesture recognition algorithm can be an open source network model that can be used for classification tasks, such as EfficientNet model, YOLOv3 network, MobileNet network and so on. Specifically, an open source network whose model parameters are default values (usable for classification tasks) can be used as a preset gesture recognition algorithm.

Among them, you can set the gesture categories that the gesture recognition algorithm needs to learn according to the gestures that need to be recognized in the actual business scenario. For example, if you need to recognize whether the gesture of the user character is "stretch the index finger and bend the other four fingers", you can set two types Gesture categories (one is "stretch the index finger, bend the other four fingers", and the other is "other gestures") to train the preset gesture recognition algorithm; Such as "stretching the index finger, bending the remaining four fingers" gesture), category 2 (such as "five fingers combined and stretching" gesture), category 3 (forms other than category 1 and category 2), you can set 3 gesture categories (one for "Extend the index finger, bend the other four fingers", one is "five fingers merged and stretched", and the other is "other gestures") to train the preset gesture recognition algorithm.

Then, take a screenshot of the shooting picture presented in step 201 to obtain the shooting picture image and input it into the trained gesture recognition algorithm, so as to call the trained gesture recognition algorithm to classify the shooting picture image: first detect the user in the shooting picture image The hand area of the character, and then classify the hand area in the shot image to obtain the gesture category of the user character in the shot image, which is used as the current gesture of the user character.

203. If the current gesture matches a preset position control gesture, acquire a pointed position point of the current gesture.

Wherein, the preset position control gesture refers to a preset gesture for controlling the placement position of the virtual object. For example, the preset position control gesture may be "stretch the index finger and bend the other four fingers", or "stretch the five fingers together".

The preset position control gestures here are only examples. In fact, the specific presentation forms of the preset position control gestures can be set according to the actual business scene requirements. In this embodiment, the specific presentation forms of the preset position control gestures are different. Do limit.

Wherein, the pointed position point refers to the position pointed by the current gesture. Specifically, it may be the point pointed by the finger corresponding to the current gesture (as shown in the following case (1)), or it may be the pre-associated position point of the current gesture (as shown in the following case (2)).

After identifying the current gesture of the user character in step 202, it will detect whether the current gesture of the user character matches the preset position control gesture, and if the current gesture matches the preset position control gesture, it will enter step 203; otherwise, If the current gesture does not match the preset position control gesture, no further processing may be performed or step 202 may be re-executed to perform gesture recognition on the user character in the shooting screen to obtain the current gesture of the user character until the current gesture of the user character is detected. When the preset position control gesture matches, go to step 203 .

In step 203, there are many ways to determine the pointing point, for example, including:

Case (1): The pointing position is the point pointed by the finger corresponding to the current gesture. At this time, step 203 may specifically include the following steps 2031A-2032A:

2031A. If the current gesture matches the preset position control gesture, acquire the ray formed in the three-dimensional space where the shooting picture is located by the finger pointing corresponding to the current gesture.

2032A. Obtain an intersection point between the ray and the supporting surface of the virtual object as the pointed position point of the current gesture.

Wherein, the three-dimensional space where the shooting picture is located refers to the three-dimensional space of the shooting scene captured corresponding to the shooting picture.

Wherein, the ray refers to the ray formed in the three-dimensional space where the captured image is located by the finger pointing corresponding to the current gesture, which is referred to as the ray in this document. As shown in FIG. 3 , the ray can specifically be understood as: a ray starting from the tip of the finger corresponding to the current gesture, and pointing to the ray in a direction in which the ray extends.

In some embodiments, in step 2032A, the standing surface of the user character (such as the ground) can be directly used as the supporting surface of the virtual object. At this time, the intersection point between the ray and the standing surface of the user character can be directly used as the direction of the current gesture location point. For example, as shown in FIG. 4 , assuming that the preset position control gesture is "stretch the index finger and bend the other four fingers", through step 202, it can be recognized that the current gesture of the user character is "stretch the index finger and bend the other four fingers", then the current The gesture is matched with the preset position control gesture, which can identify the ray formed by the finger pointing corresponding to the current gesture of the user character (that is, the pointing of the index finger) in the three-dimensional space where the shooting picture is located, and then connect the ray to the standing surface of the user character The intersection point between (as shown at point A in FIG. 4 , where the extension line pointed by the finger intersects with the standing surface of the user character at point A) is used as the pointed position point of the current gesture.

In some embodiments, there are multiple planes (such as the floor plane, each step plane of the stairs) in the shooting scene at the same time. In step 2032A, any plane in the shooting scene can be designated as the supporting surface of the virtual object. At this time, step 2032A Before that, the supporting surface of the virtual object can be determined first, and then enter step 2032A to obtain the intersection point between the ray and the supporting surface of the virtual object as the pointed position point of the current gesture. The process of determining the supporting surface of the virtual object may specifically include: adding the standing surface of the user character in the shooting picture to the set of candidate planes in the shooting picture; The plane whose included angle is less than the preset included angle threshold is added to the set of candidate planes; from each plane in the set of candidate planes, the closest plane that intersects with the ray is obtained as the virtual The supporting surface of the object.

Wherein, the starting point of the ray refers to the starting point pointed by a finger, for example, a fingertip.

Wherein, the plane intersection point refers to the intersection point between the ray and the plane.

Wherein, the plane closest to the starting point of the ray refers to the plane having an intersection with the ray among the planes in the set of candidate planes, and the plane with the smallest distance between the plane intersection and the starting point of the ray.

For example, as shown in FIG. 6 , the photographed picture includes a desktop, a wall, and the ground, and the standing surface of the user character is the ground, wherein the angle between the desktop and the ground is less than a preset angle threshold (such as 5°), The intersection point of the ray and the ground is point A, and the intersection point of the ray and the desktop is point B, then the ground and the desktop will be added to the set of alternative planes; ) exists an intersection point, and there is an intersection point with the ray (as shown in Figure 6, the intersection point between the ray and the ground is point A, and the intersection point between the ray and the desktop is point B), and the plane closest to the starting point of the ray (as shown in Figure 6 is desktop) as a supporting surface for virtual objects.

It can be seen that in order to ensure that the virtual object can be placed normally, the supporting surface of the virtual object is the ground or a plane parallel to the ground (such as the desktop, the plane of each step of the stairs, etc.). The planes whose included angles are smaller than the preset included angle threshold are added to the set of candidate planes of the shooting screen, and then the plane with the closest intersection with the ray is selected from the set of candidate planes as the supporting surface of the virtual object; the first aspect , since the standing surface and the plane whose angle with the standing surface is less than the preset angle threshold are added to the set of candidate planes, so as to ensure that the plane that may be the support surface is retained for the determination of the support surface, and can be filtered Drop some non-supporting surfaces, thereby reducing the amount of calculations needed to determine the pointing position of the current gesture. In the second aspect, on the basis that the user can specify the support surface of the virtual object, it is possible to avoid using a plane that intersects with the ray but cannot normally place the virtual object (such as a wall in the shooting scene) as the support surface of the virtual object, thereby avoiding using Dummy placed on wrong plane. In the third aspect, since the plane with the nearest intersection with the ray is used as the supporting surface of the virtual object, it can avoid when the ray intersects with multiple points at the same time (for example, as shown in Figure 6, the ray intersects with the desktop and the ground at the same time) , at this time, the plane closest to the starting point of the ray, that is, the desktop, will be used as the support surface of the virtual object) The problem of misjudgment of the support surface. In the fourth aspect, the user can specify the placement plane of the virtual object without fixing a certain plane (such as the ground) as the placement plane of the virtual object, which improves the control diversity of the virtual object.

Case (2): The pointing position is the pre-associated point of the current gesture. For example, the preset position control gestures include: A gesture, B gesture, and C gesture. Assume that the pre-associated position points of A gesture, B gesture, and C gesture are respectively: 1 meter to the left of the user character, and 1 meter to the right of the user character 1 meter in front of the user character, if the current gesture is recognized as a C gesture, it proves that the current gesture matches the preset position control gesture, and the point 1 meter in front of the user character is used as the pointing position of the current gesture.

204. Place a virtual object in the shooting frame according to the pointed position point, so as to control a relative position between the virtual object and the user character in the shooting frame.

For example, as shown in Figure 4 and Figure 5, if the pointing position point is point A, a virtual object (such as a puppy) will be placed at the pointing position point A, so that the virtual object in the shooting screen is placed where the user wants position, so that the user character can control the relative position of the virtual object and the user character in the shooting screen, so that even if the user cannot see the shooting screen, the user can still roughly control and understand the placement position of the virtual object during the shooting process, so that it can be compared with the virtual object The more natural and coordinated movements and expressions of the subject reduce the abruptness of co-shooting videos or images, making the effect of co-shooting videos or images more natural.

Furthermore, in order to allow the co-shooter (that is, the user character in the shooting screen) to better control and understand the orientation information of the virtual object, in addition to controlling the placement point of the virtual object through gestures, the orientation of the virtual object can also be controlled through gestures ( For example, the relative orientation to the user character, the relative orientation to the camera, etc.), the relative distance between the virtual object and the user character. At this time, in step 203, the gesture type of the current gesture of the user character is detected; if the gesture type is a position control gesture and the current gesture matches the preset position control gesture, then the direction of the current gesture is obtained location point. For example, the preset position control gesture is "five fingers merged and stretched". If the current gesture is "index finger stretched, and the remaining four fingers are bent", the pointing position of the current gesture will not be obtained. If the current gesture is "five fingers merged and stretched", it proves that The current gesture is a preset position control gesture, and the pointing position of the current gesture will be obtained further. To ensure that the user is controlling the position of the virtual object, then obtain the execution position point, so as to avoid that the user's current gesture is not the position of controlling the virtual object, or the current gesture is controlling the orientation information such as the relative orientation and relative distance of the virtual object, so as to avoid execution The invalid detection of the position point ensures that the user can control different orientations of the virtual object (such as placement point, placement distance, placement orientation, etc.) through various types of gestures.

If the gesture type is an orientation control gesture, then acquire the associated orientation of the current gesture; and control the relative orientation of the virtual object and the user character in the shooting screen according to the associated orientation. For example, the preset orientation control gestures are "index finger parallel to the ground pointing straight ahead" and "index finger parallel to the ground pointing straight back", wherein the associated orientations of "index finger parallel to the ground pointing straight forward" and "index finger parallel to the ground pointing straight back" are respectively : The virtual object is facing the user character, and the virtual object is facing away from the user character. If the current gesture is "index finger parallel to the ground pointing straight ahead", it proves that the current gesture is a preset orientation control gesture, and the current gesture "index finger parallel to the ground pointing straight ahead" will be followed Move the virtual object in the shooting screen with the associated orientation "virtual object faces away from the user character", so that the relative orientation of the virtual object and the user character in the control shooting screen is "virtual object faces away from the user character". In this way, the user can control the different orientations of the virtual object through gestures, so that the user can roughly control and clearly understand the orientation of the virtual object even when the shooting screen cannot be seen, so as to make actions and expressions that are more natural and coordinated with the virtual object , reduce the abruptness of the co-production video, making the co-production video effect more natural.

Furthermore, after the user makes an orientation control gesture, the orientation of the user character may occur, such as changing from facing the camera directly to the side facing the camera, and the user essentially needs to control the virtual object to face the user character. For this reason, when the user is shooting After adjusting the orientation, the orientation of the virtual object can be controlled to change accordingly, so as to avoid the problem that the user needs to continuously control the orientation of the virtual object through gestures after fine-tuning the orientation during the shooting process, and realize the convenience of interaction. At this time, after controlling the relative orientation of the virtual object and the user character in the photographed screen according to the associated orientation, when a change in the orientation of the user character is detected, it may also respond to the orientation of the user character , updating the orientation of the virtual object, so that the relative orientation between the user character and the virtual object remains the associated orientation.

If the gesture type is a distance control gesture, then acquire the associated distance of the current gesture; according to the associated distance, control the relative distance between the virtual object and the user character in the shooting picture. For example, the preset distance control gestures are "raise 1 finger" and "raise 2 fingers", wherein the associated distances of "raise 1 finger" and "raise 2 fingers" are: virtual object distance The user character is 1 meter away, and the virtual object is 2 meters away from the user character. If the current gesture is "raise 1 finger", it proves that the current gesture is a preset distance control gesture, and the associated distance of the current gesture "raise 1 finger" will be used "The virtual object is 1 meter away from the user character" moves the virtual object in the shooting screen so that the relative distance between the virtual object in the shooting screen and the user character is "1 meter". In this way, the user can control the different distances of the virtual object through gestures, so that the user can roughly control and clearly understand the distance of the virtual object even when the user cannot see the shooting screen, so as to make actions and expressions that are more natural and coordinated with the virtual object , reduce the abruptness of the co-production video, making the co-production video effect more natural.

The shooter (it can be the user character himself or someone else) can choose to press the shooting control at any time to control the camera to enter the shooting state to shoot the shooting screen, and get the co-shooting video of the virtual object and the user character's target; electronic equipment The shooting screen will be shot in response to the touch operation on the shooting control, and a co-shooting video of the virtual object and the target of the user character will be obtained. Further, in order to avoid that after entering the shooting state, the control gesture made by the user character will be captured, resulting in a large amount of target co-shooting video data (for example, especially when using the 3D model in the volumetric video as a virtual object, It will cause the co-shooting video to increase significantly), or cause the user to cut out the control gesture frame in the video in the later stage, and can respond to the touch operation on the shooting control to shoot the shooting picture, and obtain the virtual object and Preliminary co-shooting video of the user role; control gesture recognition is performed on the video frame in the preliminary co-shooting video to obtain a target video frame containing a control gesture (such as identifying a position control gesture, an orientation control gesture, or a distance control gesture The video frame is used as the target video frame); the target video frame is filtered out from the preliminary co-shooting video to obtain the target co-shooting video. In this way, video frames containing control gestures can be filtered out before saving, thereby reducing the memory occupied by the target co-shooting video to a certain extent, and reducing the subsequent cropping processing required.

Further, when the photographer presses the shooting control, that is, the camera is in the shooting state, it can also automatically identify whether the current gesture is a preset control gesture (such as whether it is a position control gesture, an orientation control gesture, or a distance control gesture) ), if it is, the shooting will be paused automatically, and the shooting will continue after the control is completed. That is, the co-shooting control method further includes: when the current gesture matches the preset control gesture, detecting whether the camera of the shooting picture is in the shooting state; if the camera is in the shooting state, switching the camera from the shooting state Switching to a pause state; until the virtual object in the shooting screen is placed at the pointing position, switch the camera from the pause state to the shooting state. For example, in order to prevent the control gestures of the co-photographer from being recorded, it is possible to automatically skip or filter the pictures where the gestures appear. For example, even if the shooting device has pressed the "shoot" button, if the co-shooter uses gestures to control the position of the virtual object, the shooting will be automatically paused; after the virtual object is placed at the pointed position, the shooting will be continued automatically to reduce the risk of co-shooting with the virtual object. amount of video data.

Therefore, this embodiment can control the position, orientation, distance, etc. of the virtual object through the gesture of the co-shooter (that is, the user), so that the user can roughly control and understand the position of the virtual object (such as the 3D model in the volumetric video), so as to make a comparison with The more natural and coordinated movements and expressions of virtual objects reduce the abruptness of the co-production video, making the co-production video more natural. It can also be avoided to a certain extent that the photographer needs to manually adjust the position of the virtual object in the shooting screen and inform the subject that the subject cannot know the position of the virtual object accurately and quickly.

In order to better implement the co-pacing control method in the embodiment of the present application, on the basis of the co-pacing control method, the embodiment of the present application also provides a co-pacing control device, as shown in Figure 7, which is the co-pacing control device in the embodiment of the present application A structural schematic diagram of an embodiment, the co-beat control device 700 includes:

a display unit 701, configured to present a shooting picture including a user character;

A recognition unit 702, configured to perform gesture recognition on the user character in the captured image to obtain the current gesture of the user character;

An acquiring unit 703, configured to acquire a pointing position point of the current gesture if the current gesture matches a preset position control gesture;

The control unit 704 is configured to place a virtual object in the shooting frame according to the pointed position point, so as to control the relative positions of the virtual object and the user character in the shooting frame.

In some embodiments, the acquiring unit 703 is specifically configured to:

If the current gesture matches the preset position control gesture, then acquire the finger corresponding to the current gesture to point to a ray formed in the three-dimensional space where the shooting picture is located;

Obtain an intersection point between the ray and the supporting surface of the virtual object as the pointing position of the current gesture.

In some embodiments, before acquiring the intersection point between the ray and the supporting surface of the virtual object as the pointed position point of the current gesture, the acquiring unit 703 is specifically configured to:

adding the standing surface of the user character in the shooting picture to the set of candidate planes in the shooting picture;

Adding a plane whose angle between the shooting picture and the standing surface is smaller than a preset angle threshold to the set of candidate planes;

From each of the planes in the set of candidate planes, a plane that has an intersection point with the ray and is closest to a starting point of the ray is obtained as a supporting surface of the virtual object.

In some embodiments, the acquiring unit 703 is specifically configured to:

detecting the gesture type of the current gesture of the user character;

If the gesture type is a position control gesture and the current gesture matches a preset position control gesture, the pointing position point of the current gesture is acquired.

In some embodiments, the control unit 704 is specifically configured to:

If the gesture type is an orientation control gesture, then acquire the associated orientation of the current gesture;

According to the associated orientation, the relative orientation of the virtual object and the user character in the shooting screen is controlled.

In some embodiments, the control unit 704 is specifically configured to:

In response to the change of the orientation of the user character, the orientation of the virtual object is updated, so that the relative orientation of the user character and the virtual object remains as the associated orientation.

In some embodiments, the control unit 704 is specifically configured to:

If the gesture type is a distance control gesture, then obtain the associated distance of the current gesture;

According to the association distance, the relative distance between the virtual object and the user character in the photographing frame is controlled.

In some embodiments, the control unit 704 is specifically configured to:

In response to a touch operation on the shooting control, the shooting screen is shot to obtain a co-shooting video of the virtual object and the target of the user character.

In some embodiments, the control unit 704 is specifically configured to:

Responding to a touch operation on the shooting control, shooting the shooting screen to obtain a preliminary co-shooting video of the virtual object and the user character;

Perform control gesture recognition on the video frame in the preliminary co-shooting video to obtain a target video frame containing the control gesture;

The target video frame is filtered out from the preliminary co-shooting video to obtain the target co-shooting video.

In some embodiments, the control unit 704 is specifically configured to:

When the current gesture matches the preset control gesture, detect whether the camera of the shooting picture is in a shooting state;

If the camera is in the shooting state, then switch the camera from the shooting state to the pause state;

Switching the camera from a pause state to a shooting state until the virtual object is placed at the pointing position in the shooting frame.

In some embodiments, the virtual object is a three-dimensional model in a volumetric video, and the control unit 704 is specifically configured to:

The three-dimensional model in the volumetric video is placed in the shooting frame according to the pointed position point.

Therefore, the co-shooting control device 700 provided by the embodiment of the present application can bring the following technical effects: the user character can control the position of the virtual object through gestures, so that the approximate position of the virtual object that the user character is co-patting, and then the user character can control the position of the virtual object in the When co-shooting with a virtual object, make movements and expressions that are more natural and coordinated with the virtual object, reduce the abruptness of the co-shooting picture, and make the co-shooting picture effect more natural.

During specific implementation, each of the above units may be implemented as an independent entity, or may be combined arbitrarily as the same or several entities. The specific implementation of each of the above units may refer to the previous method embodiments, and will not be repeated here.

Correspondingly, the embodiment of the present application also provides an electronic device, the electronic device can be a terminal, and the terminal can be a smart phone, a tablet computer, a notebook computer, a touch screen, a personal computer (PC, Personal Computer), a personal digital assistant (Personal Digital Assistant, PDA) and other terminal equipment. As shown in FIG. 8 , FIG. 8 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 800 includes a processor 801 with one or more processing cores, a memory 802 with one or more computer-readable storage media, and computer programs stored in the memory 802 and operable on the processor. Wherein, the processor 801 is electrically connected with the memory 802 . Those skilled in the art can understand that the structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and may include more or less components than those shown in the figure, or combine some components, or arrange different components.

The processor 801 is the control center of the electronic device 800, and uses various interfaces and lines to connect various parts of the entire electronic device 800, by running or loading software programs and/or modules stored in the memory 802, and calling the software programs stored in the memory 802 Execute various functions of the electronic device 800 and process data, so as to monitor the electronic device 800 as a whole.

In the embodiment of the present application, the processor 801 in the electronic device 800 will load the instructions corresponding to the process of one or more application programs into the memory 802 according to the steps of any co-pacing control method provided in the present application, and The application program stored in the memory 802 is run by the processor 801, so as to realize the specific process of the above-mentioned co-time control method.

Optionally, as shown in FIG. 8 , the electronic device 800 further includes: a touch display screen 803 , a radio frequency circuit 804 , an audio circuit 805 , an input unit 806 and a power supply 807 . Wherein, the processor 801 is electrically connected with the touch screen 803 , the radio frequency circuit 804 , the audio circuit 805 , the input unit 806 and the power supply 807 respectively. Those skilled in the art can understand that the structure of the electronic device shown in FIG. 8 does not constitute a limitation on the electronic device, and may include more or less components than shown in the figure, or combine some components, or arrange different components.

The touch display screen 803 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 803 may include a display panel and a touch panel. Among them, the display panel can be used to display information input by or provided to the user and various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, videos and any combination thereof.

The radio frequency circuit 804 can be used to send and receive radio frequency signals to establish wireless communication with network equipment or other electronic equipment through wireless communication, and to send and receive signals with network equipment or other electronic equipment.

The audio circuit 805 may be used to provide an audio interface between the user and the electronic device through a speaker or a microphone. The audio circuit 805 can transmit the electrical signal converted from the received audio data to the speaker, and the speaker converts it into an audio signal for output; on the other hand, the microphone converts the collected audio signal into an electrical signal, which is converted by the audio circuit 805 It is audio data, and then the audio data is processed by the output processor 801, and then sent to another electronic device through the radio frequency circuit 804, or the audio data is output to the memory 802 for further processing. Audio circuitry 805 may also include an earphone jack to provide communication of peripheral headphones with the electronic device.

The input unit 806 can be used to receive input numbers, character information or user characteristic information (such as fingerprints, iris, face information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control .

The power supply 807 is used to supply power to various components of the electronic device 800 . Optionally, the power supply 807 may be logically connected to the processor 801 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption through the power management system. The power supply 807 may also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

Although not shown in FIG. 8 , the electronic device 800 may also include a camera, a sensor, a Wi-Fi module, a Bluetooth module, etc., which will not be repeated here.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by instructions controlling related hardware, and the instructions can be stored in a computer-readable storage medium, and loaded and executed by the processor.

To this end, the embodiment of the present application provides a computer-readable storage medium, which stores a plurality of computer programs, and the computer programs can be loaded by a processor to execute any of the co-pacing control methods provided in the embodiments of the present application. .

Wherein, the computer-readable storage medium may include: a read-only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and the like.

In the above embodiments of the co-production control device, computer-readable storage medium, and electronic equipment, the descriptions of each embodiment have their own emphases, and for parts not described in detail in a certain embodiment, refer to the relevant descriptions of other embodiments. Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process and beneficial effects of the above-described co-production control device, computer-readable storage medium, electronic equipment and their corresponding units can be Refer to the description of the co-beating control method in the above embodiment, and details are not repeated here.

The above is a detailed introduction to the co-pacing control method, device, electronic equipment and computer-readable storage medium provided by the embodiments of the present application. In this paper, specific examples are used to illustrate the principles and implementation methods of the present application. The above embodiments The description is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, there will be changes in the specific implementation and application scope, in summary , the contents of this specification should not be construed as limiting the application.

Claims (14)

1. A co-pacing control method, characterized in that the method comprises: Present a shooting screen containing the user's character; performing gesture recognition on the user character in the photographed picture to obtain the current gesture of the user character; If the current gesture matches the preset position control gesture, then acquire the pointed position point of the current gesture; The virtual object is placed in the shooting frame according to the pointing position point, so as to control the relative position of the virtual object and the user character in the shooting frame. 2. The co-shooting control method according to claim 1, wherein if the current gesture matches a preset position control gesture, obtaining the pointed position point of the current gesture comprises: If the current gesture matches the preset position control gesture, then acquire the finger corresponding to the current gesture to point to a ray formed in the three-dimensional space where the shooting picture is located; Obtain an intersection point between the ray and the supporting surface of the virtual object as the pointing position of the current gesture. 3. The co-shoot control method according to claim 2, characterized in that before acquiring the intersection point between the ray and the supporting surface of the virtual object as the pointing position point of the current gesture, further comprising: : adding the standing surface of the user character in the shooting picture to the set of candidate planes in the shooting picture; Adding a plane whose angle between the shooting picture and the standing surface is smaller than a preset angle threshold to the set of candidate planes; From each of the planes in the set of candidate planes, a plane that has an intersection point with the ray and is closest to a starting point of the ray is obtained as a supporting surface of the virtual object. 4. The co-shooting control method according to claim 1, wherein if the current gesture matches a preset position control gesture, obtaining the pointed position point of the current gesture comprises: detecting the gesture type of the current gesture of the user character; If the gesture type is a position control gesture and the current gesture matches a preset position control gesture, the pointing position point of the current gesture is acquired. 5. The co-production control method according to claim 4, characterized in that the method further comprises: If the gesture type is an orientation control gesture, then acquire the associated orientation of the current gesture; According to the associated orientation, the relative orientation of the virtual object and the user character in the shooting screen is controlled. 6. The co-shooting control method according to claim 5, characterized in that, after controlling the relative orientation of the virtual object and the user character in the shooting screen according to the associated orientation, further comprising: In response to the change of the orientation of the user character, the orientation of the virtual object is updated, so that the relative orientation of the user character and the virtual object remains as the associated orientation. 7. The co-production control method according to claim 4, characterized in that the method further comprises: If the gesture type is a distance control gesture, then obtain the associated distance of the current gesture; According to the association distance, the relative distance between the virtual object and the user character in the photographing frame is controlled. 8. The co-production control method according to claim 1, further comprising: In response to a touch operation on the shooting control, the shooting screen is shot to obtain a co-shooting video of the virtual object and the target of the user character. 9. The co-production control method according to claim 8, characterized in that the method further comprises: Responding to a touch operation on the shooting control, shooting the shooting screen to obtain a preliminary co-shooting video of the virtual object and the user character; Perform control gesture recognition on the video frame in the preliminary co-shooting video to obtain a target video frame containing the control gesture; The target video frame is filtered out from the preliminary co-shooting video to obtain the target co-shooting video. 10. The co-production control method according to claim 1, further comprising: When the current gesture matches the preset control gesture, detect whether the camera of the shooting picture is in a shooting state; If the camera is in the shooting state, then switch the camera from the shooting state to the pause state; Switching the camera from a pause state to a shooting state until the virtual object is placed at the pointing position in the shooting screen. 11. The co-shooting control method according to any one of claims 1-10, wherein the virtual object is a three-dimensional model in the volumetric video. 12. A co-beat control device, characterized in that the co-beat control device comprises: a display unit, configured to present a shooting screen including a user character; A recognition unit, configured to perform gesture recognition on the user character in the photographed picture to obtain the current gesture of the user character; An acquisition unit, configured to acquire the pointing position of the current gesture if the current gesture matches a preset position control gesture; A control unit, configured to place a virtual object in the shooting frame according to the pointed position point, so as to control the relative positions of the virtual object and the user character in the shooting frame. 13. An electronic device, characterized in that it comprises a processor and a memory, and a computer program is stored in the memory, and when the processor invokes the computer program in the memory, it executes the process described in any one of claims 1 to 11. The co-pacing control method described above. 14. A computer-readable storage medium, characterized in that a computer program is stored thereon, and the computer program is loaded by a processor to execute the co-production control method according to any one of claims 1 to 11.

Images