CN114511808A - Video feature determination method, related device and readable storage medium - Google Patents

Abstract

The present application discloses a video feature determination method, a related device and a readable storage medium. For computer vision tasks, the image area of the sound source orientation in the video is often the key information with a higher degree of importance. In this solution, after acquiring the video to be processed, the image frame is determined for the video frame of the video to be processed. The corresponding sound source orientation attention feature map is based on each image frame and the sound source orientation attention feature map corresponding to each image frame to determine the video features of the video to be processed. The video features extracted by this scheme can enhance the ability to express the image area of the sound source azimuth in the video.

Description

Video feature determination method, related device and readable storage medium

technical field

The present application relates to the technical field of video processing, and more particularly, to a video feature determination method, a related device and a readable storage medium.

Background technique

In recent years, with the development of Internet technology and artificial intelligence technology, video-based computer vision tasks (such as target detection, video classification, target tracking, real-time human pose estimation, etc.) have played a key role in the fields of smart security and smart elderly care. Video-based computer vision tasks refer to extracting features from videos collected by video capture devices (such as cameras, etc.), determining video features, and finally decoding the video features based on task requirements to output the results corresponding to the current task. Videos contain a lot of information, among which, for video-based computer vision tasks, more important key information needs to be paid more attention.

Therefore, how to determine the video features so that the determined video features can enhance the ability to express key information in the video has become a technical problem to be solved urgently by those skilled in the art.

SUMMARY OF THE INVENTION

In view of the above problems, the present application proposes a video feature determination method, a related device, and a readable storage medium. The specific plans are as follows:

A method for determining a video feature, the method comprising:

Get pending video;

For the video frame of the video to be processed, determine the sound source orientation attention feature map corresponding to the image frame;

Based on each image frame and the sound source orientation attention feature map corresponding to each image frame, video features of the video to be processed are determined.

Optionally, the determining the sound source orientation attention feature map corresponding to the image frame includes:

acquiring the sound signal corresponding to the image frame;

Determine the sound source orientation based on the sound signal corresponding to the image frame;

According to the image frame and the sound source position, an attention feature map of the sound source position corresponding to the image frame is determined.

Optionally, determining the sound source orientation based on the sound signal corresponding to the image frame, including:

acquiring the coordinates of at least one microphone array element that collects the sound signal;

Based on the time difference between the sound signal reaching the at least one microphone array element, the sound source coordinates are calculated, and the sound source coordinates are used to represent the sound source orientation.

Optionally, determining the sound source orientation attention feature map corresponding to the image frame according to the image frame and the sound source orientation, comprising:

Based on the sound source coordinates, determine the image plane coordinates of the sound source in the image frame;

generating a two-dimensional random Gaussian distribution with the image plane coordinates as the mean value and the random value as the standard deviation;

The two-dimensional random Gaussian distribution is normalized to obtain a sound source orientation attention feature map corresponding to the image frame.

Optionally, determining the video features of the video to be processed based on each image frame and the sound source orientation attention feature map corresponding to each image frame, including:

The respective image frames and the sound source orientation attention feature map corresponding to each image frame are input into a feature generation network, and the feature generation network is based on the respective image frames and the sound source orientation attention feature corresponding to each image frame. Figure outputs the video features of the video to be processed;

The feature generation network takes the training image frame and the sound source orientation attention feature map corresponding to the training image frame as the training sample, and takes the preset computer vision processing task result marked with the training image frame as the sample label, and the training result is obtained. of.

Optionally, the feature generation network includes an encoding module and a feature fusion module;

Then, the feature generation network outputs the video features of the video to be processed based on the respective image frames and the sound source orientation attention feature maps corresponding to the respective image frames, including:

For the video frame, the encoding module encodes the image frame and the sound source orientation attention feature map corresponding to the image frame to obtain an enhanced feature map of the image frame;

The feature fusion module performs a feature fusion operation on the enhanced feature maps of each image frame to obtain the video features of the video to be processed.

Optionally, the encoding module includes an image frame encoding module and a sound source orientation attention feature map encoding module, the image frame encoding module includes a cascade of 1 encoding units, each encoding unit includes a convolution layer, a Downsampling layer and a feature fusion layer;

The sound source orientation attention feature map encoding module includes 1 downsampling layers that are cascaded; the output of the i th downsampling layer of the sound source orientation attention feature map encoding module is used as the i th in the image encoding module. The input of the feature fusion layer in each coding unit;

The I is an integer greater than or equal to 1, and the i is an integer greater than or equal to 1 and less than or equal to 1.

Optionally, the feature fusion module includes a time sequence pooling layer and a feature fusion layer;

Then, the feature fusion module performs a feature fusion operation on the enhanced feature maps of each image frame to obtain the video features of the video to be processed, including:

The time series pooling layer performs a time series pooling operation on the enhanced feature map of each image frame to obtain a time series pooling feature map of each image frame;

The feature fusion layer fuses the time series pooled feature maps of the respective image frames to obtain the video features of the video to be processed.

A video feature determination device, the device comprising:

an acquisition unit for acquiring the video to be processed;

A sound source orientation attention feature map determining unit, configured to determine the sound source orientation attention feature map corresponding to the image frame for the video frame of the video to be processed;

A video feature determination unit, configured to determine the video feature of the video to be processed based on each image frame and the sound source orientation attention feature map corresponding to each image frame.

Optionally, the sound source orientation attention feature map determination unit includes:

a sound signal acquisition unit, configured to acquire the sound signal corresponding to the image frame;

a sound source orientation determination unit, configured to determine the sound source orientation based on the sound signal corresponding to the image frame;

The sound source orientation attention feature map determination subunit is configured to determine the sound source orientation attention feature map corresponding to the image frame according to the image frame and the sound source orientation.

Optionally, the sound source orientation determination unit includes:

a microphone array element coordinate obtaining unit, configured to obtain the coordinates of at least one microphone array element that collects the sound signal;

The sound source coordinate calculation unit is configured to calculate and obtain sound source coordinates based on the time difference between the sound signal reaching the at least one microphone array element, and the sound source coordinates are used to represent the sound source orientation.

Optionally, the sound source orientation attention feature map determination subunit includes:

an image plane coordinate determination unit, configured to determine the image plane coordinates of the sound source in the image frame based on the sound source coordinates;

A two-dimensional random Gaussian distribution generating unit, used for generating a two-dimensional random Gaussian distribution with the image plane coordinates as the mean value and the random value as the standard deviation;

A normalization processing unit, configured to perform normalization processing on the two-dimensional random Gaussian distribution to obtain an attention feature map of the sound source orientation corresponding to the image frame.

Optionally, the video feature determination unit is used for:

The respective image frames and the sound source orientation attention feature map corresponding to each image frame are input into a feature generation network, and the feature generation network is based on the respective image frames and the sound source orientation attention feature corresponding to each image frame. Figure outputs the video features of the video to be processed;

The feature generation network takes the training image frame and the sound source orientation attention feature map corresponding to the training image frame as the training sample, and takes the preset computer vision processing task result marked with the training image frame as the sample label, and the training result is obtained. of.

Optionally, the feature generation network includes an encoding module and a feature fusion module;

Then, the feature generation network outputs the video features of the video to be processed based on the respective image frames and the sound source orientation attention feature maps corresponding to the respective image frames, including:

For the video frame, the encoding module encodes the image frame and the sound source orientation attention feature map corresponding to the image frame to obtain an enhanced feature map of the image frame;

The feature fusion module performs a feature fusion operation on the enhanced feature maps of each image frame to obtain the video features of the video to be processed.

Optionally, the encoding module includes an image frame encoding module and a sound source orientation attention feature map encoding module, the image frame encoding module includes a cascade of 1 encoding units, each encoding unit includes a convolution layer, a Downsampling layer and a feature fusion layer;

The sound source orientation attention feature map encoding module includes 1 downsampling layers that are cascaded; the output of the i th downsampling layer of the sound source orientation attention feature map encoding module is used as the i th in the image encoding module. The input of the feature fusion layer in each coding unit;

The I is an integer greater than or equal to 1, and the i is an integer greater than or equal to 1 and less than or equal to 1.

Optionally, the feature fusion module includes a time sequence pooling layer and a feature fusion layer;

Then, the feature fusion module performs a feature fusion operation on the enhanced feature maps of each image frame to obtain the video features of the video to be processed, including:

The time series pooling layer performs a time series pooling operation on the enhanced feature map of each image frame to obtain a time series pooling feature map of each image frame;

The feature fusion layer fuses the time series pooled feature maps of the respective image frames to obtain the video features of the video to be processed.

A video feature determination device, including a memory and a processor;

the memory for storing programs;

The processor is configured to execute the program to implement the various steps of the above-mentioned video feature determination method.

A readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements each step of the above-mentioned video feature determination method.

With the above technical solutions, the present application discloses a video feature determination method, a related device and a readable storage medium. For computer vision tasks, the image area of the sound source orientation in the video is often the key information with a higher degree of importance. In this solution, after acquiring the video to be processed, the image frame is determined for the video frame of the video to be processed. The corresponding sound source orientation attention feature map is based on each image frame and the sound source orientation attention feature map corresponding to each image frame to determine the video features of the video to be processed. The video features extracted by this scheme can enhance the ability to express the image area of the sound source azimuth in the video.

Description of drawings

Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are for purposes of illustrating preferred embodiments only and are not to be considered limiting of the application. Also, the same components are denoted by the same reference numerals throughout the drawings. In the attached image:

1 is a schematic flowchart of a method for determining a video feature disclosed in an embodiment of the present application;

2 is an effect diagram of a sound source orientation attention feature map disclosed in an embodiment of the application;

FIG. 3 is a schematic schematic diagram of the principle of determining the orientation of a sound source disclosed in an embodiment of the present application;

4 is a schematic structural diagram of a feature generation network disclosed in an embodiment of the present application;

5 is a schematic structural diagram of a feature generation network disclosed in an embodiment of the present application;

6 is a schematic diagram of a fusion principle of a coding unit feature fusion layer disclosed in an embodiment of the present application;

FIG. 7 is a schematic structural diagram of a video feature determination apparatus disclosed in an embodiment of the present application;

FIG. 8 is a block diagram of a hardware structure of a video feature determination device disclosed in an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

Next, the video feature determination method provided by the present application will be introduced through the following embodiments.

Referring to FIG. 1, FIG. 1 is a schematic flowchart of a method for determining a video feature disclosed in an embodiment of the present application, and the method may include:

Step S101: Acquire the video to be processed.

In this application, the video to be processed can be a video collected by a video capture device (such as a camera, etc.) and stored, or it can be a video collected in real time by a video capture device (such as a camera, etc.), and the duration of the video can be any duration, In this regard, this application does not make any limitation.

The video to be processed may be the original video collected by a video collection device (eg, camera, etc.), or may be the video after preprocessing (eg, noise reduction, lens segmentation, etc.), which is not limited in this application.

Step S102: For the video frame of the video to be processed, determine the sound source orientation attention feature map corresponding to the image frame.

For computer vision tasks, the image area of the sound source orientation in the video is often the key information with a higher degree of importance. Therefore, in this application, after acquiring the video to be processed, for the video frame of the video to be processed, determine the The sound source orientation attention feature map corresponding to the image frame. The sound source azimuth attention feature map corresponding to the image frame is used to represent the position of the sound source in the image frame. For ease of understanding, refer to FIG. 2 , which is a sound source azimuth attention disclosed in an embodiment of the application. The effect diagram of the force feature map, the ring and its inner area in Figure 2 are the positions of the sound source in the image frame. The manner of determining the attention feature map of the sound source orientation corresponding to the image frame will be described in detail in the following embodiments, and will not be described herein again.

It should be noted that, for a video acquisition device, the number of image frames collected per second is fixed. In this application, the corresponding sound source orientation attention feature can be determined for each image frame of the video. It is also possible to determine the corresponding sound source orientation attention feature map for a specific image frame of the video according to the scene requirements, which is not limited in this application.

Step S103: Determine video features of the video to be processed based on each image frame and the sound source orientation attention feature map corresponding to each image frame.

In the present application, for an image frame in a video, the image frame can be fused with the sound source orientation attention feature map corresponding to the image frame to obtain an enhanced feature map of the image frame, and then the enhanced feature of each image frame can be combined. The images are fused to obtain the video features of the video to be processed. The fusion mode can be determined based on the scenario requirements, which is not limited in this application. The specific implementation will be described in detail through the following embodiments, and will not be described here.

This embodiment discloses a video feature determination method. After acquiring the video to be processed, for the video frame of the video to be processed, determine the sound source orientation attention feature map corresponding to the image frame, based on each image frame and each image The attention feature map of the sound source orientation corresponding to the frame determines the video features of the video to be processed, so that the video features extracted by this scheme can enhance the expressive ability of the image area of the video sound source orientation.

In another embodiment of the present application, a specific implementation method for determining the sound source orientation attention feature map corresponding to the image frame is described, and the method may include the following steps:

Step S201: Acquire a sound signal corresponding to the image frame.

In the present application, the sound signal in the video is sampled at the sampling point synchronized with the image frame, so as to obtain the sound signal corresponding to the image frame.

Step S202: Determine the sound source orientation based on the sound signal corresponding to the image frame.

As an implementation manner, the coordinates of at least one microphone element that collects the sound signal can be obtained; based on the time difference between the sound signal reaching the at least one microphone element, the sound source coordinates are calculated, and the sound source coordinates Used to characterize the sound source orientation.

For ease of understanding, referring to FIG. 3 , FIG. 3 is a schematic diagram of the principle of determining the azimuth of a sound source disclosed in an embodiment of the present application.

As shown in Figure 3, there are four microphone array elements for collecting the sound signal, which are represented by S1, S2, S3, and S4 respectively, and these four microphone array elements are respectively deployed at the four sides of the centerline of the lens of the device that collects the video. , taking the center of the lens as the coordinate origin, the coordinates of the four microphone array elements are S1(0,0,0.5H), S2(0.5W,0,0), S3(0,0,-0.5H), S4 (-0.5W,0,0), where H and W are the height and width of the lens, respectively.

The collected sound signal has a total of four channels, corresponding to four microphone array elements respectively. Based on the arrival times of the sound signals of the four channels (ie, r1, r2, r3, and r4 shown in FIG. 3 ), the time difference (Time Difference of Arrival) of the sound signals reaching the four microphone array elements can be determined. The time difference between the sound signal reaching the four microphone array elements can construct an equation system, and by solving the equation system, the sound source coordinates P(X, Y, Z) can be obtained.

Step S203: According to the image frame and the sound source position, determine the sound source position attention feature map corresponding to the image frame.

As a possible implementation manner, the determining the attention feature map of the sound source location corresponding to the image frame according to the image frame and the sound source location includes: determining the sound source based on the sound source coordinates The image plane coordinates in the image frame; taking the image plane coordinates as the mean value and the random value as the standard deviation to generate a two-dimensional random Gaussian distribution; normalizing the two-dimensional random Gaussian distribution to obtain the The sound source orientation attention feature map corresponding to the image frame.

Wherein, the image plane coordinates (u, v) of the sound source in the image frame can be calculated by using the formula [u, v] ^T =M·[X, Y, Z] ^T , where M is the camera matrix and is Intrinsic parameters of the device that captures the video. The normalization process of the two-dimensional random Gaussian distribution can be specifically performed by means of MiiMax.

In this application, based on each image frame and the sound source orientation attention feature map corresponding to each image frame, the step of determining the video features of the video to be processed can be implemented by a neural network model. In this application, you can Pre-built and trained to obtain a feature generation network, then each image frame and the sound source orientation attention feature map corresponding to each image frame are input into the feature generation network, and the feature generation network is based on the respective image frames, and The sound source orientation attention feature map corresponding to each image frame can output the video features of the video to be processed.

It should be noted that the feature generation network takes the training image frame and the sound source orientation attention feature map corresponding to the training image frame as the training sample, and the preset computer vision processing task result marked with the training image frame is: Sample labels, obtained from training. Among them, the preset computer vision processing task can be any one of target detection, video classification, target tracking, real-time human pose estimation, etc. Different computer vision processing tasks have different results. For example, computer vision processing for target detection task, the result of which is the object contained in the video.

The structure of the feature generation network will be described below.

Referring to FIG. 4 , FIG. 4 is a schematic structural diagram of a feature generation network disclosed in an embodiment of the present application, where the feature generation network includes an encoding module and a feature fusion module.

Based on the feature generation network shown in FIG. 4 , the feature generation network outputs the video features of the video to be processed based on the respective image frames and the sound source orientation attention feature maps corresponding to the respective image frames, including: for video frames , the encoding module encodes the image frame and the sound source orientation attention feature map corresponding to the image frame to obtain an enhanced feature map of the image frame; the feature fusion module enhances the feature map of each image frame. A feature fusion operation is performed to obtain the video features of the video to be processed.

It should be noted that the internal structures of the encoding module and the feature fusion module may be implemented by any neural network structure that can meet the above requirements. As an implementation manner, the present application discloses a specific implementation manner.

Referring to FIG. 5, FIG. 5 is a schematic structural diagram of a feature generation network disclosed in an embodiment of the application. The feature generation network includes an encoding module and a feature fusion module, and the encoding module includes an image frame encoding module and a sound source orientation attention feature. The picture coding module, the image frame coding module includes 1 coding units that are cascaded, and each coding unit includes a convolution layer, a downsampling layer and a feature fusion layer; the sound source orientation attention feature map coding module Including 1 downsampling layers of concatenation; The output of the ith downsampling layer of the described sound source orientation attention feature map coding module is used as the input of the feature fusion layer in the ith coding unit in the described image coding module; The I is an integer greater than or equal to 1, and the i is an integer greater than or equal to 1 and less than or equal to 1. The feature fusion module includes a time series pooling layer and a feature fusion layer.

It can be understood that, inputting each image frame and the sound source orientation attention feature map corresponding to each image frame into a feature generation network includes: inputting each image frame into the first encoding module of the image encoder. ; Input the sound source orientation attention feature map corresponding to each image frame into the first downsampling layer of the sound source orientation attention feature map encoder.

It should be noted that the downsampling operation is a commonly used method in image encoders, and the length and width scales of the output features are usually half of the original. The i-th down-sampling layer in the module can adopt the same structure, the down-sampling layer in the i-th coding unit outputs the length and width of the feature map and the sound source orientation attention feature map The i-th down-sampling layer in the encoding module The length and width scales of the output feature maps are the same.

It should be noted that when the feature fusion layer in each coding unit fuses the input image frame feature map and the sound source orientation attention feature map, the fusion method by position multiplication can be used, that is, all channels in the same position are fused. Multiply by the same value, which is taken from the corresponding position of the sound source azimuth attention feature map. Referring to FIG. 6, FIG. 6 is a schematic diagram of a fusion principle of a coding unit feature fusion layer disclosed in an embodiment of the application, a schematic diagram of a fusion principle of a feature fusion layer in the i-th coding unit in FIG. 6, t ⁿ represents an image frame, where n Indicates the number of image frames processed in parallel by the coding unit.

Based on the feature generation network shown in FIG. 5 , the feature fusion module performs a feature fusion operation on the enhanced feature maps of each image frame to obtain the video features of the video to be processed, including: the time series pooling layer performs a feature fusion operation on each image frame. The enhanced feature map is subjected to time series pooling operation to obtain the time series pooling feature map of each image frame; the feature fusion layer fuses the time series pooling feature maps of each image frame to obtain the video features of the video to be processed. .

The video feature determination apparatus disclosed in the embodiments of the present application will be described below, and the video feature determination apparatus described below and the video feature determination method described above may refer to each other correspondingly.

Referring to FIG. 7 , FIG. 7 is a schematic structural diagram of a video feature determination apparatus disclosed in an embodiment of the present application. As shown in Figure 7, the video feature determination device may include:

an acquisition unit 11, for acquiring the video to be processed;

The sound source orientation attention feature map determining unit 12 is used for determining the sound source orientation attention feature map corresponding to the image frame for the video frame of the video to be processed;

The video feature determining unit 13 is configured to determine the video feature of the video to be processed based on each image frame and the sound source orientation attention feature map corresponding to each image frame.

As an embodiment, the sound source orientation attention feature map determining unit includes:

a sound signal acquisition unit, configured to acquire the sound signal corresponding to the image frame;

a sound source orientation determination unit, configured to determine the sound source orientation based on the sound signal corresponding to the image frame;

The sound source orientation attention feature map determination subunit is configured to determine the sound source orientation attention feature map corresponding to the image frame according to the image frame and the sound source orientation.

As an embodiment, the sound source orientation determination unit includes:

a microphone array element coordinate obtaining unit, configured to obtain the coordinates of at least one microphone array element that collects the sound signal;

The sound source coordinate calculation unit is configured to calculate and obtain sound source coordinates based on the time difference between the sound signal reaching the at least one microphone array element, and the sound source coordinates are used to represent the sound source orientation.

As an embodiment, the sound source orientation attention feature map determination subunit includes:

an image plane coordinate determination unit, configured to determine the image plane coordinates of the sound source in the image frame based on the sound source coordinates;

A two-dimensional random Gaussian distribution generating unit, used for generating a two-dimensional random Gaussian distribution with the image plane coordinates as the mean value and the random value as the standard deviation;

A normalization processing unit, configured to perform normalization processing on the two-dimensional random Gaussian distribution to obtain an attention feature map of the sound source orientation corresponding to the image frame.

As a possible implementation manner, the video feature determination unit is used for:

The respective image frames and the sound source orientation attention feature map corresponding to each image frame are input into a feature generation network, and the feature generation network is based on the respective image frames and the sound source orientation attention feature corresponding to each image frame. Figure outputs the video features of the video to be processed;

The feature generation network takes the training image frame and the sound source orientation attention feature map corresponding to the training image frame as the training sample, and takes the preset computer vision processing task result marked with the training image frame as the sample label, and the training result is obtained. of.

As an embodiment, the feature generation network includes an encoding module and a feature fusion module;

Then, the feature generation network outputs the video features of the video to be processed based on the respective image frames and the sound source orientation attention feature maps corresponding to the respective image frames, including:

For the video frame, the encoding module encodes the image frame and the sound source orientation attention feature map corresponding to the image frame to obtain an enhanced feature map of the image frame;

The feature fusion module performs a feature fusion operation on the enhanced feature maps of each image frame to obtain the video features of the video to be processed.

As an embodiment, the encoding module includes an image frame encoding module and a sound source orientation attention feature map encoding module, the image frame encoding module includes a cascade of I encoding units, and each encoding unit includes a convolutional layer, a downsampling layer and a feature fusion layer;

The sound source orientation attention feature map encoding module includes 1 downsampling layers that are cascaded; the output of the i th downsampling layer of the sound source orientation attention feature map encoding module is used as the i th in the image encoding module. The input of the feature fusion layer in each coding unit;

The I is an integer greater than or equal to 1, and the i is an integer greater than or equal to 1 and less than or equal to 1.

As an embodiment, the feature fusion module includes a time sequence pooling layer and a feature fusion layer;

Then, the feature fusion module performs a feature fusion operation on the enhanced feature maps of each image frame to obtain the video features of the video to be processed, including:

The time series pooling layer performs a time series pooling operation on the enhanced feature map of each image frame to obtain a time series pooling feature map of each image frame;

The feature fusion layer fuses the time series pooled feature maps of the respective image frames to obtain the video features of the video to be processed.

Referring to FIG. 8, FIG. 8 is a block diagram of a hardware structure of a video feature determination device provided by an embodiment of the application. Referring to FIG. 8, the hardware structure of the video feature determination device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 complete the communication with each other through the communication bus 4;

The processor 1 may be a central processing unit (CPU), or a specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention, etc.;

The memory 3 may include high-speed RAM memory, and may also include non-volatile memory (ioi-volatile memory), etc., such as at least one disk memory;

Wherein, the memory stores a program, and the processor can call the program stored in the memory, and the program is used for:

Get pending video;

For the video frame of the video to be processed, determine the sound source orientation attention feature map corresponding to the image frame;

Based on each image frame and the sound source orientation attention feature map corresponding to each image frame, video features of the video to be processed are determined.

Optionally, the refinement function and extension function of the program may refer to the above description.

Embodiments of the present application further provide a readable storage medium, where the readable storage medium can store a program suitable for execution by a processor, and the program is used for:

Get pending video;

For the video frame of the video to be processed, determine the sound source orientation attention feature map corresponding to the image frame;

Based on each image frame and the sound source orientation attention feature map corresponding to each image frame, video features of the video to be processed are determined.

Optionally, the refinement function and extension function of the program may refer to the above description.

Finally, it should also be noted that in this document, relational terms such as first and second are used only to distinguish one entity or operation from another, and do not necessarily require or imply these entities or that there is any such actual relationship or sequence between operations. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device that includes a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other.

The above description of the disclosed embodiments enables any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, this application is not intended to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (11)

1. A method for video feature determination, the method comprising:

acquiring a video to be processed;

determining a sound source azimuth attention feature map corresponding to the image frame aiming at the video frame of the video to be processed;

and determining the video characteristics of the video to be processed based on each image frame and the sound source azimuth attention characteristic map corresponding to each image frame.

2. The method according to claim 1, wherein the determining a sound source bearing attention feature map corresponding to the image frame comprises:

acquiring a sound signal corresponding to the image frame;

determining a sound source position based on the sound signals corresponding to the image frames;

and determining a sound source azimuth attention feature map corresponding to the image frame according to the image frame and the sound source azimuth.

3. The method of claim 2, wherein determining the sound source orientation based on the sound signals corresponding to the image frames comprises:

acquiring coordinates of at least one microphone element for acquiring the sound signal;

and calculating to obtain sound source coordinates based on the time difference of the sound signals reaching the at least one microphone element, wherein the sound source coordinates are used for representing the sound source azimuth.

4. The method according to claim 3, wherein the determining a sound source azimuth attention feature map corresponding to the image frame according to the image frame and the sound source azimuth comprises:

determining image plane coordinates of the sound source in the image frame based on the sound source coordinates;

generating two-dimensional random Gaussian distribution by taking the image plane coordinates as a mean value and taking a random value as a standard deviation;

and carrying out normalization processing on the two-dimensional random Gaussian distribution to obtain a sound source azimuth attention feature map corresponding to the image frame.

5. The method according to claim 1, wherein the determining the video feature of the video to be processed based on each image frame and the sound source azimuth attention feature map corresponding to each image frame comprises:

inputting the image frames and the sound source position attention feature maps corresponding to the image frames into a feature generation network, wherein the feature generation network outputs video features of the video to be processed based on the image frames and the sound source position attention feature maps corresponding to the image frames;

the feature generation network is obtained by training by taking a training image frame and a sound source azimuth attention feature map corresponding to the training image frame as training samples and taking a preset computer vision processing task result marked by the training image frame as a sample label.

6. The method of claim 5, wherein the feature generation network comprises an encoding module and a feature fusion module;

then, the feature generation network outputs the video features of the video to be processed based on the image frames and the sound source position attention feature maps corresponding to the image frames, including:

for a video frame, the coding module codes the image frame and a sound source azimuth attention feature map corresponding to the image frame to obtain an enhanced feature map of the image frame;

and the feature fusion module performs feature fusion operation on the enhanced feature map of each image frame to obtain the video features of the video to be processed.

7. The method according to claim 6, wherein the encoding modules comprise an image frame encoding module and a sound source orientation attention feature map encoding module, the image frame encoding module comprises a cascade of I encoding units, each encoding unit comprises a convolution layer, a down-sampling layer and a feature fusion layer;

the sound source azimuth attention feature map coding module comprises I cascaded down-sampling layers; the output of the ith down-sampling layer of the sound source azimuth attention feature map coding module is used as the input of a feature fusion layer in the ith coding unit in the image coding module;

the I is an integer which is greater than or equal to 1, and the I is an integer which is greater than or equal to 1 and less than or equal to I.

8. The method of claim 6, wherein the feature fusion module comprises a temporal pooling layer and a feature fusion layer;

then, the feature fusion module performs feature fusion operation on the enhanced feature map of each image frame to obtain the video features of the video to be processed, including:

the time sequence pooling layer performs time sequence pooling operation on the reinforced characteristic graph of each image frame to obtain a time sequence pooling characteristic graph of each image frame;

and the characteristic fusion layer fuses the time sequence pooling characteristic graphs of the image frames to obtain the video characteristics of the video to be processed.

9. An apparatus for video feature determination, the apparatus comprising:

the acquisition unit is used for acquiring a video to be processed;

the sound source azimuth attention feature map determining unit is used for determining a sound source azimuth attention feature map corresponding to the image frame aiming at the video frame of the video to be processed;

and the video characteristic determining unit is used for determining the video characteristics of the video to be processed based on each image frame and the sound source position attention characteristic map corresponding to each image frame.

10. A video feature determination device comprising a memory and a processor;

the memory is used for storing programs;

the processor, configured to execute the program, implementing the steps of the video feature determination method according to any one of claims 1 to 8.

11. A readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the video feature determination method according to any one of claims 1 to 8.

Images