CN107644423B - Real-time video data processing method, device and computing device based on scene segmentation - Google Patents

Abstract

The invention discloses a real-time processing method, device, computing device and computer storage medium for video data based on scene segmentation, wherein the method comprises: acquiring in real time the video shot and/or recorded by an image acquisition device including a specific object. The current frame image; or, obtain the current frame image containing the specific object in the currently playing video in real time; input the current frame image into the scene segmentation network to obtain the scene segmentation result corresponding to the current frame image; determine the scene segmentation result according to the scene segmentation result Contour information of a specific object; add personalized special effects according to the contour information of a specific object to obtain a frame processed image; overlay the frame processed image over the current frame image to obtain processed video data; display the processed video data. The technical solution can obtain the scene segmentation result corresponding to the frame image in real time and accurately, and can add personalized special effects to the frame image more accurately based on the scene segmentation result.

Description

Real-time video data processing method, device and computing device based on scene segmentation

technical field

The present invention relates to the technical field of image processing, in particular to a real-time processing method, device, computing device and computer storage medium for video data based on scene segmentation.

Background technique

In the prior art, image scene segmentation processing methods are mainly based on fully convolutional neural networks in deep learning. These processing methods use the idea of transfer learning to transfer the network pre-trained on large-scale classification datasets to images. Train on the segmentation dataset to obtain a segmentation network for scene segmentation, and then use the segmentation network to segment the image.

The network architecture used in the segmentation network obtained in the prior art directly utilizes the image classification network, and the size of the convolution block in the convolution layer is fixed, so the size of the receptive field is fixed. The field refers to the area of the input image corresponding to the response of a node in the output feature map, and the receptive field with a fixed size is only suitable for capturing objects of fixed size and scale. However, for image scene segmentation, the scene often contains objects of different sizes. Using a segmentation network with a fixed size receptive field often has problems when dealing with too large and too small objects. For example, for smaller objects, the receptive field It will capture too much background around the target, thus confusing the target with the background, causing the target to be missed and wrongly judged as the background; for larger targets, the receptive field can only capture a part of the target, making the target category judgment biased, resulting in Discontinuous segmentation results. Therefore, the image scene segmentation processing method in the prior art has the problem of low accuracy of image scene segmentation, and the obtained segmentation result cannot be used to add personalized special effects to the frame images in the video well and accurately. The display effect of the obtained processed video data is poor.

SUMMARY OF THE INVENTION

In view of the above problems, the present invention is proposed to provide a real-time processing method, apparatus, computing device and computer storage medium for video data based on scene segmentation that overcome the above problems or at least partially solve the above problems.

According to one aspect of the present invention, a real-time processing method for video data based on scene segmentation is provided, the method is performed based on a trained scene segmentation network, and the method includes:

Acquire in real time the current frame image of the specific object contained in the video shot and/or recorded by the image acquisition device; or, acquire in real time the current frame image of the specific object contained in the currently played video;

The current frame image is input into the scene segmentation network, wherein, in the scene segmentation network at least one convolutional layer, the first convolutional block of the convolutional layer is scaled by the scale coefficient output by the scale regression layer, and the first convolutional block is obtained. Second convolution block, and then use the second convolution block to perform the convolution operation of the convolution layer to obtain the output result of the convolution layer; the scale regression layer is the middle convolution layer of the scene segmentation network;

Obtain the scene segmentation result corresponding to the current frame image;

Determine the contour information of the specific object according to the scene segmentation result corresponding to the current frame image;

According to the contour information of a specific object, add personalized special effects to obtain frame processing images;

Covering the frame processing image with the current frame image to obtain the processed video data;

Displays the processed video data.

Further, using the second convolution block to perform the convolution operation of the convolution layer, and obtaining the output result of the convolution layer further includes:

Using the linear interpolation method, the feature vector is sampled from the second convolution block to form the third convolution block;

According to the third convolution block and the convolution kernel of the convolution layer, the convolution operation is performed to obtain the output result of the convolution layer.

Further, the samples used in the training of the scene segmentation network include: a plurality of sample images stored in the sample library and the labeled scene segmentation results corresponding to the sample images.

Further, the training process of the scene segmentation network is completed through multiple iterations; in an iterative process, the sample images and the labeled scene segmentation results corresponding to the sample images are extracted from the sample library, and the scene segmentation is realized by using the sample images and the labeled scene segmentation results. training of the network.

Further, the training process of the scene segmentation network is completed through multiple iterations; one of the iterative processes includes:

Input the sample image to the scene segmentation network to obtain the sample scene segmentation result corresponding to the sample image;

According to the segmentation loss between the sample scene segmentation result and the labeled scene segmentation result, the scene segmentation network loss function is obtained, and the scene segmentation network loss function is used to realize the training of the scene segmentation network.

Further, the training steps of the scene segmentation network include:

Extract the sample images and the segmentation results of the labeled scene corresponding to the sample images from the sample library;

The sample image is input into the scene segmentation network for training, wherein, in the scene segmentation network at least one convolutional layer, the scale coefficient or the initial scale coefficient output by the scale regression layer in the last iteration process is used for the first step of the convolutional layer. The convolution block is scaled to obtain a second convolution block, and then the second convolution block is used to perform the convolution operation of the convolution layer to obtain the output result of the convolution layer;

Obtain the sample scene segmentation result corresponding to the sample image;

According to the segmentation loss between the sample scene segmentation result and the labeled scene segmentation result, the scene segmentation network loss function is obtained, and the weight parameters of the scene segmentation network are updated according to the scene segmentation network loss function;

The training steps of the scene segmentation network are performed iteratively until a predetermined convergence condition is satisfied.

Further, the predetermined convergence condition includes: the number of iterations reaches a preset number of iterations; and/or the output value of the loss function of the scene segmentation network is less than a preset threshold.

Further, the scale coefficient is a feature vector in the scale coefficient feature map output by the scale regression layer.

Further, the method further includes: at the beginning of the scene segmentation network training, initializing the weight parameters of the scale regression layer.

Further, displaying the processed video data further includes: displaying the processed video data in real time;

The method also includes: uploading the processed video data to a cloud server.

Further, uploading the processed video data to the cloud server further includes:

Upload the processed video data to the cloud video platform server for the cloud video platform server to display the video data on the cloud video platform.

Further, uploading the processed video data to the cloud server further includes:

Upload the processed video data to the cloud live broadcast server, so that the cloud live broadcast server can push the video data to the viewing user client in real time.

Further, uploading the processed video data to the cloud server further includes:

Upload the processed video data to the cloud official account server, so that the cloud official account server can push the video data to the official account attention client.

According to another aspect of the present invention, there is provided an apparatus for real-time processing of video data based on scene segmentation, the apparatus operates based on a trained scene segmentation network, and the apparatus includes:

an acquisition module, adapted to acquire the current frame image of the specific object in the video shot and/or recorded by the image acquisition device in real time; or, acquire the current frame image of the specific object in the currently played video in real time;

The segmentation module is suitable for inputting the current frame image into the scene segmentation network, wherein, in the scene segmentation network at least one convolutional layer, the first convolutional block of the convolutional layer is processed by the scale coefficient output by the scale regression layer. scaling to obtain a second convolution block, and then use the second convolution block to perform the convolution operation of the convolution layer to obtain the output result of the convolution layer; the scale regression layer is the middle convolution layer of the scene segmentation network;

a generation module, adapted to obtain a scene segmentation result corresponding to the current frame image;

a determination module, adapted to determine the contour information of a specific object according to the scene segmentation result corresponding to the current frame image;

The processing module is suitable for adding personalized special effects according to the contour information of the specific object to obtain the frame processing image;

an overlay module, adapted to overlay the frame processing image over the current frame image to obtain the processed video data;

The display module is adapted to display the processed video data.

Further, the segmentation module is further adapted to:

Using the linear interpolation method, the feature vector is sampled from the second convolution block to form the third convolution block;

According to the third convolution block and the convolution kernel of the convolution layer, the convolution operation is performed to obtain the output result of the convolution layer.

Further, the samples used in the training of the scene segmentation network include: a plurality of sample images stored in the sample library and the labeled scene segmentation results corresponding to the sample images.

Further, the device further includes: a scene segmentation network training module; the training process of the scene segmentation network is completed through multiple iterations;

The scene segmentation network training module is suitable for: in an iterative process, extract the sample images and the labeled scene segmentation results corresponding to the sample images from the sample library, and use the sample images and the labeled scene segmentation results to realize the training of the scene segmentation network.

Further, the device further includes: a scene segmentation network training module; the training process of the scene segmentation network is completed through multiple iterations;

The scene segmentation network training module is suitable for: in an iterative process, input the sample image to the scene segmentation network to obtain the sample scene segmentation result corresponding to the sample image;

According to the segmentation loss between the sample scene segmentation result and the labeled scene segmentation result, the scene segmentation network loss function is obtained, and the scene segmentation network loss function is used to realize the training of the scene segmentation network.

Further, the device also includes: a scene segmentation network training module;

The scene segmentation network training module includes:

an extraction unit, adapted to extract a sample image and a segmentation result of annotated scene corresponding to the sample image from the sample library;

The training unit is suitable for inputting the sample images into the scene segmentation network for training, wherein, in the scene segmentation network at least one convolutional layer, the scale coefficient or the initial scale coefficient output by the scale regression layer in the last iteration process is used for this volume. The first convolution block of the accumulation layer is scaled to obtain the second convolution block, and then the second convolution block is used to perform the convolution operation of the convolution layer to obtain the output result of the convolution layer;

an obtaining unit, adapted to obtain a sample scene segmentation result corresponding to the sample image;

The updating unit is adapted to obtain the scene segmentation network loss function according to the segmentation loss between the sample scene segmentation result and the labeled scene segmentation result, and update the weight parameter of the scene segmentation network according to the scene segmentation network loss function;

The scene segmentation network training module runs iteratively until a predetermined convergence condition is met.

Further, the predetermined convergence condition includes: the number of iterations reaches a preset number of iterations; and/or the output value of the loss function of the scene segmentation network is less than a preset threshold.

Further, the scale coefficient is a feature vector in the scale coefficient feature map output by the scale regression layer.

Further, the scene segmentation network training module is further adapted to: at the beginning of the scene segmentation network training, initialize the weight parameters of the scale regression layer.

Further, the display module is further adapted to: display the processed video data in real time;

The device also includes:

The uploading module is suitable for uploading the processed video data to the cloud server.

Further, the uploading module is further adapted to:

Upload the processed video data to the cloud video platform server for the cloud video platform server to display the video data on the cloud video platform.

Further, the uploading module is further adapted to:

Upload the processed video data to the cloud live broadcast server, so that the cloud live broadcast server can push the video data to the viewing user client in real time.

Further, the uploading module is further adapted to:

Upload the processed video data to the cloud official account server, so that the cloud official account server can push the video data to the official account attention client.

According to another aspect of the present invention, a computing device is provided, including: a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface communicate with each other through the communication bus;

The memory is used for storing at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the above-mentioned real-time processing method for video data based on scene segmentation.

According to yet another aspect of the present invention, a computer storage medium is provided, the storage medium stores at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the above-mentioned real-time processing method for video data based on scene segmentation.

According to the technical solution provided by the present invention, the current frame image containing the specific object in the video shot and/or recorded by the image acquisition device is acquired in real time; or, the current frame image containing the specific object in the currently played video is acquired in real time, and the The current frame image is input into the scene segmentation network, wherein, in the scene segmentation network, at least one convolutional layer is used to scale the first convolutional block of the convolutional layer by using the scale coefficient output by the scale regression layer to obtain the second convolutional block. Convolution block, and then use the second convolution block to perform the convolution operation of the convolution layer to obtain the output result of the convolution layer, and then obtain the scene segmentation result corresponding to the current frame image, according to the scene corresponding to the current frame image. From the segmentation result, determine the contour information of the specific object, add personalized special effects according to the contour information of the specific object, obtain the frame processed image, cover the current frame image with the frame processed image to obtain the processed video data, and display the processed video data. The technical scheme provided by the invention scales the convolution block according to the scale coefficient, realizes the adaptive scaling of the receptive field, and uses the trained scene segmentation network to obtain the scene segmentation result corresponding to the frame image in the video in real time and accurately, effectively The accuracy and processing efficiency of image scene segmentation are greatly improved, and personalized special effects can be added to frame images more accurately based on the obtained scene segmentation results, which beautifies the display effect of video data.

The above description is only an overview of the technical solutions of the present invention, in order to be able to understand the technical means of the present invention more clearly, it can be implemented according to the content of the description, and in order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand , the following specific embodiments of the present invention are given.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are for the purpose of illustrating preferred embodiments only and are not to be considered limiting of the invention. Also, the same components are denoted by the same reference numerals throughout the drawings. In the attached image:

1 shows a schematic flowchart of a real-time processing method for video data based on scene segmentation according to an embodiment of the present invention;

2 shows a schematic flowchart of a method for training a scene segmentation network according to an embodiment of the present invention;

3 shows a schematic flowchart of a real-time processing method for video data based on scene segmentation according to another embodiment of the present invention;

4 shows a structural block diagram of a real-time processing apparatus for video data based on scene segmentation according to an embodiment of the present invention;

5 shows a structural block diagram of a real-time processing apparatus for video data based on scene segmentation according to another embodiment of the present invention;

FIG. 6 shows a schematic structural diagram of a computing device according to an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thoroughly understood, and will fully convey the scope of the present disclosure to those skilled in the art.

1 shows a schematic flowchart of a real-time processing method for video data based on scene segmentation according to an embodiment of the present invention. The method is executed based on a trained scene segmentation network. As shown in FIG. 1 , the method includes the following steps:

Step S100, acquiring in real time a current frame image containing a specific object in a video shot and/or recorded by the image acquisition device; or acquiring in real time a current frame image containing a specific object in a currently playing video.

In this embodiment, the image acquisition device is described by taking a mobile terminal as an example. The current frame image of the camera of the mobile terminal when the video is recorded or the current frame image of the video is obtained in real time. Since the present invention processes a specific object, only the current frame image containing the specific object is acquired when the current frame image is acquired. In addition to acquiring the video shot and/or recorded by the image acquisition device in real time, it is also possible to acquire the current frame image of the specific object in the currently playing video in real time.

Step S101, the current frame image is input into the scene segmentation network.

The current frame image contains a specific object, such as a human body. In order to accurately add personalized special effects to the current frame image, the scene segmentation network needs to be used to segment the current frame image. Among them, the scene segmentation network is trained, and the trained scene segmentation network can use the scale coefficient output by the scale regression layer in the network to scale the convolution block of the convolution layer, so as to more accurately The current frame image is scene segmented. Specifically, the samples used in the training of the scene segmentation network include: a plurality of sample images stored in the sample library and the labeled scene segmentation results corresponding to the sample images. The labeling scene segmentation result is the segmentation result obtained by manual segmentation and labeling of each scene in the sample image.

Among them, the training process of the scene segmentation network is completed through multiple iterations. Optionally, in an iterative process, the sample images and the labeled scene segmentation results corresponding to the sample images are extracted from the sample library, and the scene segmentation network is trained by using the sample images and the labeled scene segmentation results.

Optionally, an iterative process includes: inputting the sample image into the scene segmentation network to obtain a sample scene segmentation result corresponding to the sample image; and obtaining the scene segmentation network loss according to the segmentation loss between the sample scene segmentation result and the labeled scene segmentation result. function, and use the scene segmentation network loss function to realize the training of the scene segmentation network.

Step S102, at least one convolution layer in the scene segmentation network, using the scale coefficient output by the scale regression layer to perform scaling processing on the first convolution block of the convolution layer to obtain a second convolution block.

Those skilled in the art can select which layer or which layers of convolutional blocks of the convolutional layer are to be scaled according to actual needs, which is not limited here. For convenience of distinction, in the present invention, the convolution block to be scaled is referred to as the first convolution block, and the convolution block after scaling is referred to as the second convolution block. Assuming that the first convolutional block of a certain convolutional layer in the scene segmentation network is scaled, then in this convolutional layer, the first convolutional block of the convolutional layer is scaled using the scale coefficient output by the scale regression layer. The scaling process is performed to obtain the second convolution block.

Among them, the scale regression layer is the middle convolution layer of the scene segmentation network, and the middle convolution layer refers to one or more convolution layers in the scene segmentation network. Those skilled in the art can choose the appropriate one in the scene segmentation network according to actual needs. One or more convolutional layers of , as scale regression layers, are not limited here. In the present invention, the feature map output by the scale regression layer is called the scale coefficient feature map, and the scale coefficient is the feature vector in the scale coefficient feature map output by the scale regression layer. The invention scales the convolution block according to the scale coefficient, thereby realizing the adaptive scaling of the receptive field, and can more accurately perform scene segmentation on the input current frame image, effectively improving the accuracy of image scene segmentation.

Step S103, using the second convolution block to perform the convolution operation of the convolution layer to obtain the output result of the convolution layer.

After the second convolution block is obtained, the second convolution block can be used to perform the convolution operation of the convolution layer to obtain the output result of the convolution layer.

In step S104, a scene segmentation result corresponding to the current frame image is obtained.

After obtaining the output result of the convolution layer in step S103, if there are other convolution layers after the convolution layer in the scene segmentation network, then the output result of the convolution layer is used as the input of the next convolution layer Perform subsequent convolution operations. After the convolution operation of all convolutional layers in the scene segmentation network, the scene segmentation result corresponding to the current frame image is obtained.

Step S105: Determine the contour information of the specific object according to the scene segmentation result corresponding to the current frame image.

After the scene segmentation result corresponding to the current frame image is obtained, the contour information of the specific object can be determined according to the scene segmentation result corresponding to the current frame image. Assuming that the specific object is a human body, then the outline information of the human body can be determined according to the scene segmentation result, so as to distinguish which areas in the current frame image are human bodies and which areas are not human bodies.

Step S106, adding personalized special effects according to the contour information of the specific object to obtain a frame-processed image.

After the contour information of the specific object is determined, personalized special effects can be added according to the contour information of the specific object to obtain a frame-processed image. Persons skilled in the art can set personalized special effects according to actual needs, which are not limited here. For example, according to the contour information of the specific object, an effect map can be added at the edge of the specific object. The effect map can be a static effect map or a dynamic effect map. Specifically, when the specific object is a human body, the effect map can be It is an effect map such as flames, beating notes, waves, etc.; when the specific object is a human head, the effect map can be an effect map such as a hair crown, shaking ears, etc. The specific settings are set according to the implementation, which is not limited here.

Step S107, overlaying the frame processed image over the current frame image to obtain processed video data.

Using the frame processing image to directly overwrite the original current frame image, the processed video data can be obtained directly. At the same time, the recorded user can also directly see the frame processing image.

When the frame-processed image is obtained, the frame-processed image will directly overwrite the original current frame image. The speed of coverage is faster, generally completed within 1/24 seconds. For the user, due to the relatively short time of the overlay processing, the human eye does not notice it obviously, that is, the human eye does not notice the process that the original current frame image in the video data is overwritten. In this way, when the processed video data is subsequently displayed, the processed video data is displayed in real time while shooting and/or recording and/or playing the video data, and the user will not feel that the frame images in the video data are overwritten. display effect.

Step S108, displaying the processed video data.

After the processed video data is obtained, it can be displayed in real time, and the user can directly see the display effect of the processed video data.

According to the real-time processing method for video data based on scene segmentation provided by this embodiment, the current frame image of the specific object contained in the video shot and/or recorded by the image acquisition device is acquired in real time; The current frame image of the specific object is input into the scene segmentation network, wherein, in the scene segmentation network at least one convolution layer, the scale coefficient output by the scale regression layer is used for the first convolution layer of the convolution layer. The block is scaled to obtain the second convolution block, and then the second convolution block is used to perform the convolution operation of the convolution layer to obtain the output result of the convolution layer, and then the scene segmentation result corresponding to the current frame image is obtained, According to the scene segmentation result corresponding to the current frame image, determine the contour information of the specific object, add personalized special effects according to the contour information of the specific object, obtain the frame processing image, and cover the frame processing image with the current frame image to obtain the processed video data, Displays the processed video data. The technical scheme provided by the invention scales the convolution block according to the scale coefficient, realizes the adaptive scaling of the receptive field, and uses the trained scene segmentation network to obtain the scene segmentation result corresponding to the frame image in the video in real time and accurately, effectively The accuracy and processing efficiency of image scene segmentation are greatly improved, and personalized special effects can be added to frame images more accurately based on the obtained scene segmentation results, which beautifies the display effect of video data.

FIG. 2 shows a schematic flowchart of a method for training a scene segmentation network according to an embodiment of the present invention. As shown in FIG. 2 , the training steps of the scene segmentation network include the following steps:

Step S200, extract the sample images and the segmentation result of the labeled scene corresponding to the sample images from the sample library.

The sample library not only stores the sample images, but also stores the segmentation results of the labeled scene corresponding to the sample images. Those skilled in the art can set the number of sample images stored in the sample library according to actual needs, which is not limited here. In step S200, a sample image is extracted from the sample library, and a segmentation result of the labeled scene corresponding to the sample image is extracted.

Step S201, input the sample image into the scene segmentation network for training.

After the sample images are extracted, the sample images are input into the scene segmentation network for training.

Step S202, at least one convolution layer in the scene segmentation network, using the scale coefficient or the initial scale coefficient output by the scale regression layer in the last iteration process to scale the first convolution block of the convolution layer to obtain the second volume Building blocks.

Those skilled in the art can select which layer or which layers of convolutional blocks of the convolutional layer are to be scaled according to actual needs, which is not limited here. Assuming that the first convolution block of a certain convolutional layer in the scene segmentation network is scaled, then in this convolutional layer, the scale coefficient or initial scale coefficient output by the scale regression layer in the previous iteration process is used for the convolutional layer. The first convolutional block of the layer is scaled to obtain the second convolutional block.

Specifically, in order to effectively train the scene segmentation network, at the beginning of the scene segmentation network training, the weight parameters of the scale regression layer can be initialized. Those skilled in the art can set specific initialized weight parameters according to actual needs, which are not limited here. The initial scale coefficient is the feature vector in the scale coefficient feature map output by the scale regression layer after initialization.

Step S203, using the second convolution block to perform the convolution operation of the convolution layer to obtain the output result of the convolution layer.

After the second convolution block is obtained, the second convolution block can be used to perform the convolution operation of the convolution layer to obtain the output result of the convolution layer. Since the second convolution block is obtained by scaling the first convolution block, the coordinates corresponding to the feature vectors in the second convolution block may not be integers. Eigenvectors corresponding to integer coordinates. Those skilled in the art can set the preset calculation method according to actual needs, which is not limited here. For example, the preset calculation method may be a linear interpolation method. Specifically, the linear interpolation method is used to sample the feature vector from the second convolution block to form a third convolution block, and then convolve the third convolution block with the convolution block. The convolution kernel of the layer performs the convolution operation to obtain the output result of the convolution layer.

After the output of the convolutional layer is obtained, if there are other convolutional layers after the convolutional layer in the scene segmentation network, the output of the convolutional layer is used as the input of the subsequent convolutional layer for subsequent convolution operation. After the convolution operation of all convolutional layers in the scene segmentation network, the scene segmentation result corresponding to the sample image is obtained.

Step S204, obtaining a sample scene segmentation result corresponding to the sample image.

Obtain the sample scene segmentation result corresponding to the sample image obtained by the scene segmentation network.

Step S205: Obtain a scene segmentation network loss function according to the segmentation loss between the sample scene segmentation result and the labeled scene segmentation result, and update the weight parameter of the scene segmentation network according to the scene segmentation network loss function.

Wherein, those skilled in the art can set the specific content of the scene segmentation network loss function according to actual needs, which is not limited here. According to the loss function of the scene segmentation network, a back propagation operation is performed, and the weight parameters of the scene segmentation network are updated through the operation result.

Step S206, iteratively execute the training steps of the scene segmentation network until a predetermined convergence condition is satisfied.

Wherein, those skilled in the art can set predetermined convergence conditions according to actual needs, which are not limited here. For example, the predetermined convergence condition may include: the number of iterations reaches a preset number of iterations; and/or, the output value of the scene segmentation network loss function is less than a preset threshold. Specifically, it can be judged whether the predetermined convergence condition is satisfied by judging whether the number of iterations reaches the preset number of iterations, or whether the predetermined convergence condition is satisfied according to whether the output value of the scene segmentation network loss function is smaller than a predetermined threshold. In step S206, the training steps of the scene segmentation network are iteratively executed until a predetermined convergence condition is satisfied, thereby obtaining a trained scene segmentation network.

其中，H_A为该输入特征图的高度参数，W_A为该输入特征图的宽度参数，C_A为该输入特征图的通道数；卷积层J的输出特征图为

其中，H_B为该输出特征图的高度参数，W_B为该输出特征图的宽度参数，C_B为该输出特征图的通道数；尺度回归层输出的尺度系数特征图为

其中，H_S为该尺度系数特征图的高度参数，W_S为该尺度系数特征图的宽度参数，该尺度系数特征图的通道数为1，具体地，H_S＝H_B，且W_S＝W_B。In a specific training process, for example, the first convolution block of a certain convolution layer in the scene segmentation network needs to be scaled. It is assumed that the convolution layer is called the convolution layer J. The input feature map is

Among them, HA is the height parameter of the input feature map, W _A is the width parameter of the input feature map, and _{C A is} the number of channels of the input feature map; the output feature map of the convolutional layer J is

Among them, _HB is the height parameter of the output feature map, _WB is the width parameter of the output feature map, and _CB is the number of channels of the output feature map; the scale coefficient feature map output by the scale regression layer is

Wherein, H _S is the height parameter of the scale coefficient feature map, W _S is the width parameter of the scale coefficient feature map, and the number of channels of the scale coefficient feature map is 1, specifically, H _S = _HB , and W _S = _WB .

In the scene segmentation network, an ordinary 3×3 convolutional layer can be selected as the scale regression layer, and the output feature map corresponding to the scale regression layer with the channel number of 1 is the scale coefficient feature map. In order to effectively train the scene segmentation network and prevent the scene segmentation network from collapsing during the training process, it is necessary to initialize the weight parameters of the scale regression layer at the beginning of the scene segmentation network training. Among them, the initialized weight parameters of the scale regression layer are

Among them, w ₀ is the initialized convolution kernel of the scale regression layer, a is any position in the convolution kernel, and b ₀ is the initialized bias term. In the initialization process of the weight parameters of the scale regression layer, the convolution kernel is set to satisfy the random coefficient σ of the Gaussian distribution, and its value is small, close to 0, and the bias term is set to 1. Therefore, after initializing The processed scale regression layer will all output values close to 1, that is, the initial scale coefficient is close to 1, then after applying the initial scale coefficient to the convolution layer J, the output results obtained are not much different from the standard convolution results. Thus, a relatively stable training process is provided, which effectively prevents the scene segmentation network from collapsing during the training process.

偏置为

卷积层J的输入特征图为

卷积层J的输出特征图为

卷积层J的第一卷积块为X^t，对第一卷积块X^t进行缩放处理后所得到的第二卷积块为Y^t，其中，一般情况下，k＝1。在输出特征图B中的任意位置t，对应的特征向量为

特征向量B^t为由该特征向量对应于输入特征图A中的第二卷积块Y^t与卷积核K内积得到的，其中，位置

For convolutional layer J, suppose the convolution kernel of convolutional layer J is

Biased to

The input feature map of the convolutional layer J is

The output feature map of convolutional layer J is

The first convolution block of the convolution layer J is X ^t , and the second convolution block obtained after scaling the first convolution block X ^t is Y ^t , where, in general, k=1. At any position t in the output feature map B, the corresponding feature vector is

The feature vector B ^t is obtained by the inner product of the feature vector corresponding to the second convolution block Y ^t in the input feature map A and the convolution kernel K, wherein, the position

是卷积的膨胀系数，

和

是输入特征图A中的坐标。第一卷积块X^t中将均匀选取(2k+1)×(2k+1)个特征向量与卷积核K进行相乘，具体地，这些特征向量的坐标为The first convolution block X ^t is a square area centered at (p ^t , q ^t ) in the input feature map A, and its side length is fixed to 2kd+1, where,

is the expansion coefficient of the convolution,

and

are the coordinates in the input feature map A. In the first convolution block X ^t , (2k+1)×(2k+1) feature vectors are uniformly selected and multiplied by the convolution kernel K. Specifically, the coordinates of these feature vectors are

in,

Assuming that s ^t is the scale coefficient of the feature vector B ^t corresponding to the position t in the output feature map B in the scale coefficient feature map, the position of s ^t in the scale coefficient feature map is also t, and the feature vector B ^t in the output feature map B in the same position.

第二卷积块Y^t中将均匀选取 (2k+1)×(2k+1)个特征向量与卷积核K进行相乘，具体地，这些特征向量的坐标为The first convolution block X ^t of the convolution layer J is scaled by the scale coefficient s ^t to obtain the second convolution block Y ^t . The second convolution block Y ^t is an input feature map A with (p ^t , q ^t ) as the center of the square area, its side length will change according to the scale coefficient s ^t as

In the second convolution block Y ^t , (2k+1)×(2k+1) feature vectors are uniformly selected and multiplied by the convolution kernel K. Specifically, the coordinates of these feature vectors are

的具体的计算公式为：Among them, the scale coefficient s ^t is a real value, then the coordinates x' _ij and y' _ij of the feature vector may not be integers. In the present invention, the eigenvectors corresponding to these non-integer coordinates are obtained by using a linear interpolation method. Using the linear interpolation method, the feature vector is sampled from the second convolution block Y ^t to form the third convolution block Z ^t , then for each feature vector in the third convolution block Z ^t

The specific calculation formula is:

若(x'_ij,y'_ij)超出了输入特征图A的范围，则对应的特征向量将被置为0作为填补。假设

是卷积核K与对应的特征向量相乘且输出通道为c的卷积向量，其中，

那么卷积运算中对应所有通道的按元素相乘过程可以用与

进行矩阵相乘表示，则前向传播(forward propagation)过程为

in,

If (x' _ij , y' _ij ) exceeds the range of the input feature map A, the corresponding feature vector will be set to 0 as padding. Assumption

is the convolution vector in which the convolution kernel K is multiplied by the corresponding feature vector and the output channel is c, where,

Then the element-wise multiplication process corresponding to all channels in the convolution operation can be used with

Perform matrix multiplication, then the forward propagation process is

During backpropagation, assuming the gradient g(B ^t ) transmitted from B ^t , the gradient is

g(b)=g(B ^t )

where g(·) represents the gradient function, and (·) ^T represents the matrix transpose. It is worth noting that in the process of calculating the gradient, the final gradient of the convolution kernel K and the bias b is the sum of the gradients obtained from all positions in the output feature map B. For the linear interpolation process, the partial derivative of the corresponding eigenvector is

The partial derivatives of the corresponding coordinates are

的偏导与上述

的公式类似，此处不再赘述。corresponding

The partial derivative of the above

The formula is similar and will not be repeated here.

Since the coordinates are calculated by the scale coefficient s ^t , the partial derivative of the scale coefficient corresponding to the coordinates is

Based on the above partial derivatives, the gradient of the scale coefficient feature map S and the input feature map A can be obtained by the following formula:

Therefore, the weight parameters of each convolution layer and the weight parameters of the scale regression layer in the scene segmentation network can be trained end-to-end. In addition, the gradient of the scale coefficient can be calculated by the gradient from the next layer, so the scale coefficient is obtained automatically and implicitly. In a specific implementation process, both the forward propagation process and the back propagation process can be performed in parallel on a graphics processing unit (GPU), which has high computational efficiency.

According to the scene segmentation network training method provided in this embodiment, a scene segmentation network that scales convolution blocks according to scale coefficients can be trained to achieve adaptive scaling of the receptive field, and the scene segmentation network can quickly obtain the corresponding The scene segmentation result effectively improves the accuracy and processing efficiency of image scene segmentation.

3 shows a schematic flowchart of a real-time processing method for video data based on scene segmentation according to another embodiment of the present invention. The method is executed based on a trained scene segmentation network. As shown in FIG. 3 , the method includes the following steps:

Step S300, acquiring in real time a current frame image containing a specific object in a video shot and/or recorded by the image acquisition device; or acquiring in real time a current frame image containing a specific object in a currently playing video.

Step S301, input the current frame image into the scene segmentation network.

Among them, the scene segmentation network is trained, and the trained scene segmentation network can use the scale coefficient output by the scale regression layer in the network to scale the convolution block of the convolution layer, and more accurately input the current frame. Image for scene segmentation.

Step S302, at least one convolution layer in the scene segmentation network, using the scale coefficient output by the scale regression layer to perform scaling processing on the first convolution block of the convolution layer to obtain a second convolution block.

Those skilled in the art can select which layer or which layers of convolutional blocks of the convolutional layer are to be scaled according to actual needs, which is not limited here. The scale coefficient is the feature vector in the scale coefficient feature map output by the scale regression layer. In step S302, the first convolution block of the convolution layer is scaled by the scale coefficient to obtain the second convolution block.

Step S303 , using a linear interpolation method to sample a feature vector from the second convolution block to form a third convolution block.

的具体的计算公式为：Since the second convolution block is obtained by scaling the first convolution block, the coordinates corresponding to the feature vectors in the second convolution block may not be integers. Therefore, the linear interpolation method can be used to obtain these non-integer values. The eigenvectors corresponding to the coordinates. Using the linear interpolation method, the feature vector is sampled from the second convolution block, and then the third convolution block is formed according to the sampled feature vector. Assuming that the second convolution block is Y ^t and the third convolution block is Z ^t , then for each feature vector in the third convolution block Z ^t

The specific calculation formula is:

d是卷积的膨胀系数，s^t是尺度系数，一般情况下，k＝1。in,

d is the expansion coefficient of the convolution, s ^t is the scale coefficient, in general, k=1.

Step S304, according to the third convolution block and the convolution kernel of the convolution layer, perform a convolution operation to obtain an output result of the convolution layer.

After the third convolution block is obtained, a convolution operation is performed on the third convolution block and the convolution kernel of the convolution layer to obtain the output result of the convolution layer.

In step S305, a scene segmentation result corresponding to the current frame image is obtained.

After obtaining the output result of the convolution layer in step S304, if there are other convolution layers after the convolution layer in the scene segmentation network, then the output result of the convolution layer is used as the input of the next convolution layer Perform subsequent convolution operations. After the convolution operation of all convolutional layers in the scene segmentation network, the scene segmentation result corresponding to the current frame image is obtained.

Step S306: Determine the contour information of the specific object according to the scene segmentation result corresponding to the current frame image.

After the scene segmentation result corresponding to the current frame image is obtained in step S305, the contour information of the specific object can be determined according to the scene segmentation result corresponding to the current frame image. Assuming that the specific object is a human body, then the outline information of the human body can be determined according to the scene segmentation result, so as to distinguish which areas in the current frame image are human bodies and which areas are not human bodies.

Step S307, adding personalized special effects according to the contour information of the specific object to obtain a frame-processed image.

After the contour information of the specific object is determined, personalized special effects can be added according to the contour information of the specific object to obtain a frame-processed image. Persons skilled in the art can set personalized special effects according to actual needs, which are not limited here.

For example, according to the contour information of the specific object, an effect map can be added at the edge of the specific object. The effect map can be a static effect map or a dynamic effect map. Specifically, when the specific object is a human body, the effect map can be It is an effect map such as flames, beating notes, waves, etc.; when the specific object is a human head, the effect map can be an effect map such as a hair crown, shaking ears, etc. The specific settings are set according to the implementation, which is not limited here. For another example, the specific object area and the non-specific object area can be determined according to the contour information of the specific object. The two-dimensional scene background image can also be a three-dimensional scene background image. Specifically, when the specific object is a human body, the image of the background area can be replaced with a three-dimensional scene background image such as a three-dimensional underwater scene background image and a three-dimensional volcano scene background image.

Step S308, overlaying the frame processed image over the current frame image to obtain processed video data.

Using the frame processing image to directly overwrite the original current frame image, the processed video data can be obtained directly. At the same time, the recorded user can also directly see the frame processing image.

Step S309, displaying the processed video data.

After the processed video data is obtained, it can be displayed in real time, and the user can directly see the display effect of the processed video data.

Step S310, upload the processed video data to the cloud server.

The processed video data can be directly uploaded to the cloud server. Specifically, the processed video data can be uploaded to one or more cloud video platform servers, such as iQiyi, Youku, Kuai Video and other cloud video platform servers. For the cloud video platform server to display video data on the cloud video platform. Alternatively, the processed video data can also be uploaded to the cloud live broadcast server. When a user of the live broadcast viewing terminal enters the cloud live broadcast server to watch, the cloud live broadcast server can push the video data to the viewing user client in real time. Alternatively, the processed video data can also be uploaded to the cloud official account server. When a user follows the official account, the cloud official account server will push the video data to the official account attention client; further, the cloud official account server can also Follow the viewing habits of users who follow the official account, and push video data that conforms to the user's habits to the official account's attention client.

According to the real-time processing method of video data based on scene segmentation provided in this embodiment, not only the convolution block is scaled according to the scale coefficient, so as to realize the adaptive scaling of the receptive field, but also the convolution block after scale processing is scaled by the linear interpolation method. Further processing is performed to solve the problem of selecting feature vectors whose coordinates are non-integers in the convolution block after scaling processing; and the scene segmentation results corresponding to the frame images in the video can be obtained in real time and accurately by using the trained scene segmentation network, effectively It improves the accuracy and processing efficiency of image scene segmentation, and can add personalized special effects to frame images more accurately based on the obtained scene segmentation results, beautify the video data display effect, and optimize the video data processing method.

FIG. 4 shows a structural block diagram of an apparatus for real-time processing of video data based on scene segmentation according to an embodiment of the present invention. The apparatus operates based on a trained scene segmentation network. As shown in FIG. 4 , the apparatus includes: an acquisition module 410 , a segmentation module 420 , a generation module 430 , a determination module 440 , a processing module 450 , an overlay module 460 and a display module 470 .

The acquiring module 410 is adapted to: acquire in real time a current frame image containing a specific object in a video shot and/or recorded by the image acquisition device; or acquire in real time a current frame image containing a specific object in a currently played video.

The segmentation module 420 is adapted to: input the current frame image into the scene segmentation network, wherein, in at least one convolutional layer in the scene segmentation network, the first convolutional block of the convolutional layer is used using the scale coefficient output by the scale regression layer Perform scaling processing to obtain a second convolution block, and then use the second convolution block to perform a convolution operation of the convolution layer to obtain an output result of the convolution layer.

The scene segmentation network is trained. Specifically, the samples used in the training of the scene segmentation network include: a plurality of sample images stored in the sample library and the labeled scene segmentation results corresponding to the sample images. The scale regression layer is the intermediate convolutional layer of the scene segmentation network. Those skilled in the art can select an appropriate one or more convolutional layers in the scene segmentation network as the scale regression layer according to actual needs, which is not limited here. The scale coefficient is the feature vector in the scale coefficient feature map output by the scale regression layer.

The generating module 430 is adapted to: obtain a scene segmentation result corresponding to the current frame image.

The determining module 440 is adapted to: determine the contour information of the specific object according to the scene segmentation result corresponding to the current frame image.

The processing module 450 is adapted to: add personalized special effects according to the contour information of the specific object to obtain a frame-processed image.

The overlay module 460 is adapted to: overlay the frame processed image over the current frame image to obtain the processed video data.

The overlay module 460 directly overlays the original current frame image with the frame processed image, and can directly obtain the processed video data. At the same time, the recorded user can also directly see the frame processing image.

The display module 470 is adapted to: display the processed video data.

After the display module 470 obtains the processed video data, it can display it in real time, and the user can directly see the display effect of the processed video data.

According to the real-time video data processing device based on scene segmentation provided in this embodiment, the convolution block can be scaled according to the scale coefficient, and the adaptive scaling of the receptive field is realized. The scene segmentation results corresponding to the frame images in the video effectively improve the accuracy and processing efficiency of image scene segmentation. Based on the obtained scene segmentation results, personalized special effects can be added to the frame images more accurately, and the video data display effect can be beautified. .

Fig. 5 shows a structural block diagram of an apparatus for real-time processing of video data based on scene segmentation according to another embodiment of the present invention. The apparatus operates based on a trained scene segmentation network. As shown in Fig. 5, the apparatus includes: an acquisition module 510 , a scene segmentation network training module 520 , a segmentation module 530 , a generation module 540 , a determination module 550 , a processing module 560 , an overlay module 570 , a display module 580 and an upload module 590 .

The acquiring module 510 is adapted to: acquire in real time the current frame image of the video shot and/or recorded by the image acquisition device that contains the specific object; or, acquire the current frame image of the currently played video containing the specific object in real time.

Among them, the training process of the scene segmentation network is completed through multiple iterations. The scene segmentation network training module 520 is adapted to: in an iterative process, extract sample images and labeled scene segmentation results corresponding to the sample images from the sample library, and use the sample images and labeled scene segmentation results to implement scene segmentation network training.

Optionally, the scene segmentation network training module 520 is adapted to: in an iterative process, input the sample image to the scene segmentation network to obtain the sample scene segmentation result corresponding to the sample image; according to the difference between the sample scene segmentation result and the labeled scene segmentation result; The segmentation loss between the two is obtained, and the scene segmentation network loss function is obtained, and the scene segmentation network training is realized by using the scene segmentation network loss function.

In a specific embodiment, the scene segmentation network training module 520 may include: an extraction unit 521 , a training unit 522 , an acquisition unit 523 and an update unit 524 .

Specifically, the extraction unit 521 is adapted to: extract the sample images and the segmentation results of the labeled scene corresponding to the sample images from the sample library.

The training unit 522 is adapted to: input the sample image into the scene segmentation network for training, wherein, in the scene segmentation network at least one convolution layer, the scale coefficient or the initial scale coefficient output by the scale regression layer in the previous iteration process is used. The first convolution block of the convolution layer is scaled to obtain the second convolution block, and then the second convolution block is used to perform the convolution operation of the convolution layer to obtain the output result of the convolution layer.

Among them, the scale regression layer is the middle convolution layer of the scene segmentation network, and the scale coefficient is the feature vector in the scale coefficient feature map output by the scale regression layer.

Optionally, the training unit 522 is further adapted to: use a linear interpolation method to sample a feature vector from the second convolution block to form a third convolution block; according to the third convolution block and the convolution kernel of the convolution layer Perform the convolution operation to obtain the output result of the convolution layer.

The obtaining unit 523 is adapted to: obtain a sample scene segmentation result corresponding to the sample image.

The updating unit 524 is adapted to: obtain a scene segmentation network loss function according to the segmentation loss between the sample scene segmentation result and the labeled scene segmentation result, and update the weight parameter of the scene segmentation network according to the scene segmentation network loss function.

The scene segmentation network training module 520 runs iteratively until a predetermined convergence condition is satisfied.

Wherein, those skilled in the art can set predetermined convergence conditions according to actual needs, which are not limited here. For example, the predetermined convergence condition may include: the number of iterations reaches a preset number of iterations; and/or, the output value of the scene segmentation network loss function is less than a preset threshold. Specifically, it can be judged whether the predetermined convergence condition is satisfied by judging whether the number of iterations reaches the preset number of iterations, or whether the predetermined convergence condition is satisfied according to whether the output value of the scene segmentation network loss function is smaller than a predetermined threshold.

Optionally, the scene segmentation network training module 520 is further adapted to: at the beginning of the scene segmentation network training, initialize the weight parameters of the scale regression layer.

The segmentation module 530 is adapted to: input the current frame image into the scene segmentation network, wherein at least one convolutional layer in the scene segmentation network uses the scale coefficient output by the scale regression layer to the first convolutional block of the convolutional layer. Perform scaling processing to obtain the second convolution block, and then use the linear interpolation method to sample the feature vector from the second convolution block to form a third convolution block; according to the convolution of the third convolution block and the convolution layer The kernel performs the convolution operation to obtain the output result of the convolution layer.

The generating module 540 is adapted to: obtain a scene segmentation result corresponding to the current frame image.

The determining module 550 is adapted to: determine the contour information of the specific object according to the scene segmentation result corresponding to the current frame image.

The processing module 560 is adapted to: add personalized special effects according to the contour information of the specific object to obtain a frame-processed image.

The overlay module 570 is adapted to: overlay the frame processed image over the current frame image to obtain the processed video data.

The display module 580 is adapted to: display the processed video data.

After the display module 580 obtains the processed video data, it can display it in real time, and the user can directly see the display effect of the processed video data.

The uploading module 590 is adapted to upload the processed video data to the cloud server.

The uploading module 590 can directly upload the processed video data to the cloud server. Specifically, the uploading module 590 can upload the processed video data to one or more cloud video platform servers, such as iQiyi, Youku, Kuai Video Wait for the cloud video platform server for the cloud video platform server to display video data on the cloud video platform. Alternatively, the uploading module 590 can also upload the processed video data to the cloud live broadcast server. When a user who has a live broadcast viewing terminal enters the cloud live broadcast server to watch, the cloud live broadcast server can push the video data to the viewing user client in real time. Or the uploading module 590 can also upload the processed video data to the cloud official account server, and when a user pays attention to the official account, the cloud official account server pushes the video data to the official account attention client; further, the cloud official account server According to the viewing habits of users who follow the official account, video data that conforms to the user's habits can also be pushed to the client of the official account.

According to the apparatus for real-time processing of video data based on scene segmentation provided in this embodiment, not only the convolution block is scaled according to the scale coefficient, so as to realize the adaptive scaling of the receptive field, but also the convolution block after scaling is processed by linear interpolation method. Further processing is performed to solve the problem of selecting feature vectors whose coordinates are non-integers in the convolution block after scaling processing; and the scene segmentation results corresponding to the frame images in the video can be obtained in real time and accurately by using the trained scene segmentation network, effectively It improves the accuracy and processing efficiency of image scene segmentation, and can add personalized special effects to frame images more accurately based on the obtained scene segmentation results, beautify the video data display effect, and optimize the video data processing method.

The present invention also provides a non-volatile computer storage medium, where the computer storage medium stores at least one executable instruction, and the executable instruction can execute the real-time processing method of video data based on scene segmentation in any of the above method embodiments.

FIG. 6 shows a schematic structural diagram of a computing device according to an embodiment of the present invention. The specific embodiment of the present invention does not limit the specific implementation of the computing device.

As shown in FIG. 6 , the computing device may include: a processor (processor) 602 , a communications interface (Communications Interface) 604 , a memory (memory) 606 , and a communication bus 608 .

in:

The processor 602 , the communication interface 604 , and the memory 606 communicate with each other through the communication bus 608 .

The communication interface 604 is used to communicate with network elements of other devices such as clients or other servers.

The processor 602 is configured to execute the program 610, and specifically may execute the relevant steps in the foregoing embodiments of the method for real-time processing of video data based on scene segmentation.

Specifically, the program 610 may include program code including computer operation instructions.

The processor 602 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The one or more processors included in the computing device may be the same type of processors, such as one or more CPUs; or may be different types of processors, such as one or more CPUs and one or more ASICs.

The memory 606 is used to store the program 610 . Memory 606 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

The program 610 can specifically be used to cause the processor 602 to execute the real-time processing method of video data based on scene segmentation in any of the foregoing method embodiments. For the specific implementation of the steps in the program 610, reference may be made to the corresponding descriptions in the corresponding steps and units in the above-mentioned embodiment of the real-time processing of video data based on scene segmentation, which will not be repeated here. Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the above-described devices and modules, reference may be made to the corresponding process descriptions in the foregoing method embodiments, which will not be repeated here.

The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used with teaching based on this. The structure required to construct such a system is apparent from the above description. Furthermore, the present invention is not directed to any particular programming language. It should be understood that various programming languages may be used to implement the inventions described herein, and that the descriptions of specific languages above are intended to disclose the best mode for carrying out the invention.

In the description provided herein, numerous specific details are set forth. It will be understood, however, that embodiments of the invention may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description.

Similarly, it is to be understood that in the above description of exemplary embodiments of the invention, various features of the invention are sometimes grouped together into a single embodiment, figure, or its description. This disclosure, however, should not be construed as reflecting an intention that the invention as claimed requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the claims following the Detailed Description are hereby expressly incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of this invention.

Those skilled in the art will understand that the modules in the device in the embodiment can be adaptively changed and arranged in one or more devices different from the embodiment. The modules or units or components in the embodiments may be combined into one module or unit or component, and further they may be divided into multiple sub-modules or sub-units or sub-assemblies. All features disclosed in this specification (including accompanying claims, abstract and drawings) and any method so disclosed may be employed in any combination, unless at least some of such features and/or procedures or elements are mutually exclusive. All processes or units of equipment are combined. Each feature disclosed in this specification (including accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise.

Furthermore, those skilled in the art will appreciate that although some of the embodiments described herein include certain features, but not others, included in other embodiments, that combinations of features of different embodiments are intended to be within the scope of the invention within and form different embodiments. For example, in the following claims, any of the claimed embodiments may be used in any combination.

Various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some or all of the components according to the embodiments of the present invention. The present invention can also be implemented as apparatus or apparatus programs (eg, computer programs and computer program products) for performing part or all of the methods described herein. Such a program implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such signals may be downloaded from Internet sites, or provided on carrier signals, or in any other form.

It should be noted that the above-described embodiments illustrate rather than limit the invention, and that alternative embodiments may be devised by those skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. do not denote any order. These words can be interpreted as names.

Claims (22)

1. A real-time processing method for video data based on scene segmentation, the method is performed based on a trained scene segmentation network, the method comprising: Acquire in real time the current frame image of the specific object contained in the video shot and/or recorded by the image acquisition device; or, acquire in real time the current frame image of the specific object contained in the currently played video; Inputting the current frame image into the scene segmentation network, wherein at least one convolutional layer in the scene segmentation network uses the scale coefficient output by the scale regression layer to scale the first convolutional block of the convolutional layer processing to obtain a second convolution block, and then using a linear interpolation method to sample a feature vector from the second convolution block to form a third convolution block; According to the third convolution block and the convolution kernel of the convolution layer, the convolution operation is performed to obtain the output result of the convolution layer; the scale regression layer is the middle convolution layer of the scene segmentation network; the The scale coefficient is the feature vector in the scale coefficient feature map output by the scale regression layer; Obtain the scene segmentation result corresponding to the current frame image; Determine the contour information of the specific object according to the scene segmentation result corresponding to the current frame image; According to the contour information of the specific object, adding personalized special effects to obtain a frame processing image; Covering the frame processing image with the current frame image to obtain the processed video data; Display the processed video data; Wherein, the training process of the scene segmentation network is completed through multiple iterations; in an iterative process, sample images are input into the scene segmentation network for training, wherein, in the scene segmentation network at least one convolutional layer, The first convolution block of the convolution layer is scaled by the scale coefficient or the initial scale coefficient output by the scale regression layer in the last iteration process to obtain the second convolution block. The feature vector is obtained by sampling in the accumulation block to form the third convolution block; According to the third convolution block and the convolution kernel of the convolution layer, the convolution operation is performed to obtain the output result of the convolution layer; Wherein, the displaying the processed video data further includes: displaying the processed video data in real time; The method further includes: uploading the processed video data to a cloud server. 2 . The method according to claim 1 , wherein the samples used in the scene segmentation network training comprise: a plurality of sample images stored in a sample library and the labeled scene segmentation results corresponding to the sample images. 3 . 3. The method according to claim 2, wherein, in an iterative process, a sample image and an annotation scene segmentation result corresponding to the sample image are extracted from the sample library, and the sample image and the annotated scene segmentation result are used for segmentation The results enable the training of the scene segmentation network. 4. The method of claim 3, wherein an iterative process comprises: Input the sample image to the scene segmentation network to obtain the sample scene segmentation result corresponding to the sample image; According to the segmentation loss between the sample scene segmentation result and the labeled scene segmentation result, a scene segmentation network loss function is obtained, and the scene segmentation network is trained by using the scene segmentation network loss function. 5. The method according to claim 4, wherein the training step of the scene segmentation network comprises: Extracting sample images and the segmentation result of the labeled scene corresponding to the sample images from the sample library; inputting the sample image into the scene segmentation network for training; Obtain the sample scene segmentation result corresponding to the sample image; According to the segmentation loss between the sample scene segmentation result and the labeled scene segmentation result, a scene segmentation network loss function is obtained, and the weight parameter of the scene segmentation network is updated according to the scene segmentation network loss function; The training steps of the scene segmentation network are iteratively performed until a predetermined convergence condition is satisfied. 6. The method according to claim 5, wherein the predetermined convergence condition comprises: the number of iterations reaches a preset number of iterations; and/or the output value of the scene segmentation network loss function is less than a preset threshold. 7. The method according to claim 6, wherein the method further comprises: at the beginning of the scene segmentation network training, initializing the weight parameters of the scale regression layer. 8. The method according to claim 7, wherein the uploading of the processed video data to the cloud server further comprises: Upload the processed video data to the cloud video platform server for the cloud video platform server to display the video data on the cloud video platform. 9. The method according to claim 8, wherein the uploading of the processed video data to the cloud server further comprises: Upload the processed video data to the cloud live broadcast server, so that the cloud live broadcast server can push the video data to the viewing user client in real time. 10. The method according to claim 9, wherein the uploading of the processed video data to the cloud server further comprises: Upload the processed video data to the cloud official account server, so that the cloud official account server can push the video data to the official account attention client. 11. A device for real-time processing of video data based on scene segmentation, the device operates based on a trained scene segmentation network, the device comprising: an acquisition module, adapted to acquire the current frame image of the specific object in the video shot and/or recorded by the image acquisition device in real time; or, acquire the current frame image of the specific object in the currently played video in real time; The segmentation module is adapted to input the current frame image into the scene segmentation network, wherein at least one convolutional layer in the scene segmentation network uses the scale coefficient output by the scale regression layer for the first convolutional layer. The convolution block is scaled to obtain a second convolution block, and then a linear interpolation method is used to sample a feature vector from the second convolution block to form a third convolution block; according to the third convolution block and The convolution kernel of the convolution layer performs convolution operation to obtain the output result of the convolution layer; the scale regression layer is the middle convolution layer of the scene segmentation network; the scale coefficient is the scale output by the scale regression layer eigenvectors in the coefficient feature map; a generation module, adapted to obtain a scene segmentation result corresponding to the current frame image; a determination module, adapted to determine the contour information of the specific object according to the scene segmentation result corresponding to the current frame image; a processing module, adapted to add personalized special effects according to the outline information of the specific object to obtain a frame processing image; an overlay module, adapted to overlay the frame processing image over the current frame image to obtain the processed video data; a display module, adapted to display the processed video data; Wherein, the training process of the scene segmentation network is completed through multiple iterations; in an iterative process, sample images are input into the scene segmentation network for training, wherein, in the scene segmentation network at least one convolutional layer, The first convolution block of the convolution layer is scaled by the scale coefficient or the initial scale coefficient output by the scale regression layer in the last iteration process to obtain the second convolution block. The feature vector is obtained by sampling in the accumulation block to form a third convolution block; according to the third convolution block and the convolution kernel of the convolution layer, the convolution operation is performed to obtain the output result of the convolution layer; Wherein, the display module is further adapted to: display the processed video data in real time; The device also includes: The uploading module is suitable for uploading the processed video data to the cloud server. 12 . The apparatus according to claim 11 , wherein the samples used for the training of the scene segmentation network comprise: a plurality of sample images stored in a sample library and the labeled scene segmentation results corresponding to the sample images. 13 . 13. The apparatus according to claim 12, wherein the apparatus further comprises: a scene segmentation network training module; The scene segmentation network training module is adapted to: in an iterative process, extract a sample image and a labeled scene segmentation result corresponding to the sample image from the sample library, and use the sample image and the labeled scene segmentation result to realize a scene Training of the segmentation network. 14. The apparatus according to claim 13, wherein the apparatus further comprises: a scene segmentation network training module; The scene segmentation network training module is adapted to: in an iterative process, input the sample image to the scene segmentation network to obtain the sample scene segmentation result corresponding to the sample image; According to the segmentation loss between the sample scene segmentation result and the labeled scene segmentation result, a scene segmentation network loss function is obtained, and the scene segmentation network is trained by using the scene segmentation network loss function. 15. The apparatus according to claim 14, wherein the apparatus further comprises: a scene segmentation network training module; The scene segmentation network training module includes: an extraction unit, adapted to extract a sample image and a segmentation result of annotated scene corresponding to the sample image from the sample library; a training unit, adapted to input the sample images into the scene segmentation network for training; an obtaining unit, adapted to obtain a sample scene segmentation result corresponding to the sample image; an update unit, adapted to obtain a scene segmentation network loss function according to the segmentation loss between the sample scene segmentation result and the labeled scene segmentation result, and update the weight parameter of the scene segmentation network according to the scene segmentation network loss function; The scene segmentation network training module runs iteratively until a predetermined convergence condition is satisfied. 16. The apparatus according to claim 15, wherein the predetermined convergence condition comprises: the number of iterations reaches a preset number of iterations; and/or the output value of the scene segmentation network loss function is less than a preset threshold. 17. The apparatus according to claim 16, wherein the scene segmentation network training module is further adapted to: initialize the weight parameters of the scale regression layer when the scene segmentation network training starts. 18. The apparatus of claim 17, wherein the upload module is further adapted to: Upload the processed video data to the cloud video platform server for the cloud video platform server to display the video data on the cloud video platform. 19. The apparatus of claim 18, wherein the uploading module is further adapted to: Upload the processed video data to the cloud live broadcast server, so that the cloud live broadcast server can push the video data to the viewing user client in real time. 20. The apparatus of claim 19, wherein the uploading module is further adapted to: Upload the processed video data to the cloud official account server, so that the cloud official account server can push the video data to the official account attention client. 21. A computing device, comprising: a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform an operation corresponding to the real-time processing method for video data based on scene segmentation according to any one of claims 1-10. 22. A computer storage medium, wherein at least one executable instruction is stored in the storage medium, and the executable instruction causes a processor to execute the scene segmentation-based video data real-time according to any one of claims 1-10. The operation corresponding to the processing method.

Images