CN116152289A - A tracking method, related device, equipment and storage medium of a target object - Google Patents

Abstract

The present application discloses a target object tracking method based on artificial intelligence technology, which specifically relates to the field of image recognition. The application scenarios include at least various terminals, such as mobile phones, computers, and vehicle-mounted terminals. This application includes if the target object is detected in the previous image frame, extracting the original detection image from the current image frame; obtaining the target feature map through the feature extraction network; for each feature map, obtaining the anchor according to the anchor point set in the feature map Point frame; based on the features corresponding to the anchor point frame, obtain optional bounding box parameters through the object detection network; determine the T group of bounding box parameters; determine the tracking result for the target object in the current image frame according to the T group of bounding box parameters. The present application also provides devices, equipment and storage media. The present application not only reduces the amount of data processing and improves the efficiency of target tracking, but also achieves better recognition effect in the process of image detection, thereby helping to improve the accuracy of target tracking.

Description

A tracking method, related device, equipment and storage medium of a target object

technical field

The present application relates to computer vision technology, and in particular to a method for tracking a target object, related devices, equipment and storage media.

Background technique

Object tracking is an important research direction in computer vision and has a wide range of applications. With the development of computer technology, more and more scenarios of interaction between users and computers appear. In an interactive scene, it is usually necessary to locate and track the target object, and then perform different interactive actions based on the tracked target object.

At present, traditional tracking algorithms are usually used to track the target object. For example, a kernel correlation filter (kernel correlation filter, KCF) algorithm is used to implement target tracking. The advantage of the KCF algorithm is that it is very fast. Therefore, the algorithm can support running on most mobile terminals, and its inference time can reach the millisecond level.

However, the inventors have found that there are at least the following problems in the existing solutions, the KCF algorithm is difficult to identify the deformation of the object. Assuming that there is a square object in the initial image frame, the object is transformed into a rectangle in the subsequent tracking. At this time, it is difficult for the KCF algorithm to adaptively correct a square into a rectangle. Therefore, the accuracy of target tracking is low. .

Contents of the invention

Embodiments of the present application provide a method for tracking a target object, a related device, equipment, and a storage medium. On the one hand, it reduces the amount of data processing and improves the efficiency of target tracking. On the other hand, a better recognition effect is achieved in the image detection process, which is beneficial to improve the accuracy of target tracking.

In view of this, on the one hand, the present application provides a method for tracking a target object, including:

If the target object is detected in the previous image frame of the current image frame, an original detection image is extracted from the current image frame according to the target bounding box corresponding to the previous image frame, wherein the original detection image includes the target object;

Based on the original detection image, the target feature map is obtained through the feature extraction network, where the target feature map includes K feature points, and the target feature map is divided into N feature maps, each feature map is set with an anchor point, and K is An integer greater than 1, N is an integer greater than or equal to 1 and less than K;

For each feature map, get the anchor point box according to the anchor point set in the feature map;

For each feature map, based on the features corresponding to the anchor box, at least one set of optional bounding box parameters is obtained through the object detection network, wherein each set of optional bounding box parameters includes coordinate parameters and size parameters of the optional bounding box;

Determine T groups of bounding box parameters according to at least one group of optional bounding box parameters of each feature map, wherein each group of bounding box parameters includes coordinate parameters and size parameters of the bounding box, and T is an integer greater than or equal to 1;

Determine the tracking result for the target object in the current image frame according to T groups of bounding box parameters.

Another aspect of the present application provides a method for tracking a target object, including:

If the target object is detected in the previous image frame of the current image frame, an original detection image is extracted from the current image frame according to the target bounding box corresponding to the previous image frame, wherein the original detection image includes the target object;

Based on the original detection image, the target feature map is obtained through the feature extraction network;

Based on the target feature map, the first offset and the second offset are obtained through the object detection network, wherein the first offset is the upper left offset, and the second offset is the lower right offset, or, The first offset is an upper right offset, and the second offset is a lower left offset;

According to the first offset and the second offset, a tracking result for the target object in the current image frame is determined.

Another aspect of the present application provides a target object tracking device, including:

The acquisition module is used to extract the original detection image from the current image frame according to the target bounding box corresponding to the previous image frame if the target object is detected in the previous image frame of the current image frame, wherein the original detection image includes target;

The acquisition module is also used to acquire the target feature map through the feature extraction network based on the original detection image, wherein the target feature map includes K feature points, and the target feature map is divided into N feature maps, and each feature map is set with An anchor point, K is an integer greater than 1, N is an integer greater than or equal to 1 and less than K;

The obtaining module is also used for obtaining an anchor point frame according to the anchor point set in the feature map for each feature map;

The obtaining module is further configured to obtain at least one set of optional bounding box parameters through the object detection network based on the features corresponding to the anchor point boxes for each feature map, wherein each set of optional bounding box parameters includes optional bounding box parameters Coordinate parameters and size parameters;

A determining module, configured to determine T groups of bounding box parameters according to at least one set of optional bounding box parameters of each feature map, wherein each group of bounding box parameters includes coordinate parameters and size parameters of the bounding box, and T is greater than or equal to 1 integer;

A tracking module, configured to determine a tracking result for the target object in the current image frame according to T groups of bounding box parameters.

In a possible design, in an implementation manner of another aspect of the embodiments of the present application, the target object tracking device further includes an identification module;

The recognition module is used to perform object recognition on the current image frame if no target object is detected in the previous image frame of the current image frame;

The recognition module is also used for performing object recognition on the current image frame if the previous image frame of the current image frame is not obtained.

In a possible design, in an implementation manner of another aspect of the embodiments of the present application,

The acquisition module is specifically used to adjust the size of the original detection image to obtain the image to be processed;

If the size of the image to be processed fails to match the preset size, the image to be processed is filled to obtain a target detection image that meets the preset size;

Based on the target detection image, the target feature map is output through the feature extraction network.

In a possible design, in an implementation of another aspect of the embodiments of the present application, each set of optional bounding box parameters further includes a confidence parameter, and at least one set of optional bounding box parameters includes multiple sets of optional bounding box parameters. Bounding box parameters;

The determination module is specifically used for selecting an optional bounding box parameter with the maximum confidence parameter from multiple sets of optional bounding box parameters for each feature map, wherein each set of optional bounding box parameters is used to determine an optional bounding box parameter frame;

For each feature map, determine an initial optional bounding box according to the optional bounding box parameter with the maximum confidence parameter;

For each feature map, traverse the remaining optional bounding boxes. If the overlapping area between the optional bounding box and the initial optional bounding box is greater than or equal to the area threshold, delete the optional bounding box until the remaining optional bounding boxes are obtained. bounding boxes, where the remaining optional bounding boxes represent optional bounding boxes in addition to the initial optional bounding box;

For each feature map, the optional bounding box parameters corresponding to the remaining optional bounding boxes are used as bounding box parameters.

In a possible design, in an implementation manner of another aspect of the embodiments of the present application,

A tracking module, specifically for determining T bounding boxes according to T groups of bounding box parameters;

Determine the leftmost vertex, the rightmost vertex, the uppermost vertex, and the lowermost vertex according to the T bounding boxes;

Determine the coordinates of the upper left vertex, the lower left vertex, the upper right vertex, and the lower right vertex according to the leftmost vertex, the rightmost vertex, the uppermost vertex, and the lowermost vertex;

According to the coordinates of the upper left vertex, the lower left vertex, the upper right vertex and the lower right vertex, the tracking result for the target object is determined.

In a possible design, in an implementation of another aspect of the embodiments of the present application, the target object tracking device further includes a training module;

The obtaining module is also used to obtain image samples, wherein the image samples include the target object;

The acquisition module is also used to obtain the sample feature map through the feature extraction network based on the image sample, wherein the sample feature map includes K feature points, the sample feature map is divided into N feature maps to be trained, and each feature map to be trained set has an anchor;

The obtaining module is also used for obtaining the anchor point frame according to the anchor point set in the feature map to be trained for each feature map to be trained;

The obtaining module is also used to obtain at least one set of optional bounding box parameters through the object detection network based on the features corresponding to the anchor box and the real bounding box parameters for each feature map to be trained, wherein each set of optional bounding boxes The parameters include coordinate parameters and size parameters of the bounding box to be trained;

The determination module is also used to determine the predicted bounding box parameters from at least one set of optional bounding box parameters for each feature map to be trained;

The training module is used for updating the model parameters of the feature extraction network and the object detection network by using a loss function according to the predicted bounding box parameters and the real bounding box parameters for each feature map to be trained.

In a possible design, in an implementation manner of another aspect of the embodiments of the present application, the target object tracking device further includes a detection module;

The detection module is used to detect the current image frame using a preset tracking algorithm to obtain the first center position parameter of the target object, wherein the preset tracking algorithm is a kernel correlation filtering algorithm, an online real-time tracking algorithm, a background perception correlation filtering algorithm or Multi-instance online learning algorithm;

The tracking module is specifically used to determine the second center position parameter of the target object according to the T group of bounding box parameters;

According to the first center position parameter and the second center position parameter, a tracking result for the target object in the current image frame is determined.

In a possible design, in an implementation manner of another aspect of the embodiment of the present application, the target object is a human hand;

The target object tracking device also includes a starting module, a generating module and a triggering module;

The starting module is used to start the camera device of the terminal device in response to the operation on the video shooting control;

The acquisition module is also used to collect M consecutive image frames through the camera device, where M is an integer greater than or equal to 1;

The acquisition module is also used to obtain the tracking results for the human hand in consecutive M image frames after determining the tracking results for the target object in the current image frame according to the T group of bounding box parameters;

A generating module, configured to generate a moving track for the human hand according to the tracking results corresponding to the consecutive M image frames and the tracking results corresponding to the current image frame;

The trigger module is configured to trigger the function corresponding to the preset track if the moving track matches the preset track successfully.

In a possible design, in an implementation manner of another aspect of the embodiment of the present application, the target object is a human hand;

The starting module is also used to start the camera device of the terminal device in response to the operation on the video shooting control;

The acquisition module is also used to collect M consecutive image frames through the camera device, where M is an integer greater than or equal to 1;

The acquisition module is also used to obtain the tracking results for the human hand in consecutive M image frames after determining the tracking results for the target object in the current image frame according to the T group of bounding box parameters;

The obtaining module is also used to obtain the track length according to the tracking results corresponding to the consecutive M image frames and the tracking results corresponding to the current image frame;

The trigger module is also used to trigger the function corresponding to the preset track if the track length is greater than or equal to the length threshold.

Another aspect of the present application provides a target object tracking device, including:

The acquisition module is used to extract the original detection image from the current image frame according to the target bounding box corresponding to the previous image frame if the target object is detected in the previous image frame of the current image frame, wherein the original detection image includes target;

The acquisition module is also used to acquire the target feature map through the feature extraction network based on the original detection image;

The obtaining module is also used to obtain the first offset and the second offset through the object detection network based on the target feature map, wherein the first offset is the upper left offset, and the second offset is the lower right offset, or, the first offset is the upper right offset, and the second offset is the lower left offset;

The tracking module is configured to determine a tracking result for the target object in the current image frame according to the first offset and the second offset.

In a possible design, in an implementation manner of another aspect of the embodiments of the present application, the target object tracking device further includes an identification module;

The recognition module is used to perform object recognition on the current image frame if no target object is detected in the previous image frame of the current image frame;

The recognition module is also used for performing object recognition on the current image frame if the previous image frame of the current image frame is not obtained.

In a possible design, in an implementation manner of another aspect of the embodiments of the present application,

The acquisition module is specifically used to adjust the size of the original detection image to obtain the image to be processed;

If the size of the image to be processed fails to match the preset size, the image to be processed is filled to obtain a target detection image that meets the preset size;

Based on the target detection image, the target feature map is output through the feature extraction network.

In a possible design, in an implementation manner of another aspect of the embodiments of the present application, the first offset is an upper-left offset, and the second offset is a lower-right offset, wherein, The upper left offset includes the horizontal offset and vertical offset from the upper left vertex to the initial upper left vertex, and the lower right offset includes the horizontal offset and vertical offset from the lower right vertex to the initial lower right vertex;

The target object tracking device also includes a training module;

The obtaining module is also used to obtain image samples, wherein the image samples include the target object;

The obtaining module is also used to obtain the sample feature map through the feature extraction network based on the image sample;

The obtaining module is also used to obtain the upper left prediction offset and the lower right prediction offset through the object detection network based on the sample feature map;

The training module is used to use the loss function to model the feature extraction network and the object detection network based on the sample feature map, according to the upper left predicted offset, the lower right predicted offset, the upper left real offset and the lower right real offset The parameters are updated.

In a possible design, in an implementation of another aspect of the embodiments of the present application, the first offset is an upper-right offset, and the second offset is a lower-left offset, where the upper-right The offset includes the horizontal offset and vertical offset from the upper right vertex to the initial upper right vertex, and the lower left offset includes the horizontal offset and vertical offset from the lower left vertex to the initial lower left vertex;

The obtaining module is also used to obtain image samples, wherein the image samples include the target object;

The obtaining module is also used to obtain the sample feature map through the feature extraction network based on the image sample;

The obtaining module is also used to obtain the upper-right prediction offset and the lower-left prediction offset through the object detection network based on the sample feature map;

The training module is also used to use the loss function to modify the model parameters of the feature extraction network and the object detection network based on the sample feature map, according to the upper right predicted offset, the lower left predicted offset, the upper right real offset and the lower left real offset to update.

In a possible design, in an implementation manner of another aspect of the embodiments of the present application, the target object tracking device further includes a detection module;

The detection module is used to detect the current image frame using a preset tracking algorithm to obtain the first center position parameter of the target object, wherein the preset tracking algorithm is a kernel correlation filtering algorithm, an online real-time tracking algorithm, a background perception correlation filtering algorithm or Multi-instance online learning algorithm;

The tracking module is specifically configured to determine the second center position parameter of the target object according to the first offset and the second offset;

According to the first center position parameter and the second center position parameter, a tracking result for the target object in the current image frame is determined.

In a possible design, in an implementation manner of another aspect of the embodiment of the present application, the target object is a human hand;

The target object tracking device also includes a starting module, a generating module and a triggering module;

The starting module is used to start the camera device of the terminal device in response to the operation on the video shooting control;

The acquisition module is also used to collect M consecutive image frames through the camera device, where M is an integer greater than or equal to 1;

The acquisition module is also used to obtain the tracking results for the human hand in consecutive M image frames after determining the tracking results for the target object in the current image frame according to the first offset and the second offset;

A generating module, configured to generate a moving track for the human hand according to the tracking results corresponding to the consecutive M image frames and the tracking results corresponding to the current image frame;

The trigger module is configured to trigger the function corresponding to the preset track if the moving track matches the preset track successfully.

In a possible design, in an implementation manner of another aspect of the embodiment of the present application, the target object is a human hand;

The starting module is also used to start the camera device of the terminal device in response to the operation on the video shooting control;

The acquisition module is also used to collect M consecutive image frames through the camera device, where M is an integer greater than or equal to 1;

The acquisition module is also used to obtain the tracking results for the human hand in consecutive M image frames after determining the tracking results for the target object in the current image frame according to the first offset and the second offset;

The obtaining module is also used to obtain the track length according to the tracking results corresponding to the consecutive M image frames and the tracking results corresponding to the current image frame;

The trigger module is also used to trigger the function corresponding to the preset track if the track length is greater than or equal to the length threshold.

Another aspect of the present application provides a computer device, including: a memory, a processor, and a bus system;

Among them, the memory is used to store programs;

The processor is used to execute the program in the memory, and the processor is used to execute the methods in the above aspects according to the instructions in the program code;

The bus system is used to connect the memory and the processor so that the memory and the processor can communicate.

Another aspect of the present application provides a terminal device, including: a memory, a processor, and a bus system;

Among them, the memory is used to store programs;

The processor is used to execute the program in the memory, and the processor is used to execute the methods in the above aspects according to the instructions in the program code;

The bus system is used to connect the memory and the processor so that the memory and the processor can communicate.

Another aspect of the present application provides a computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the computer-readable storage medium is run on a computer, it causes the computer to execute the methods in the above aspects.

Another aspect of the present application provides a computer program product or computer program, where the computer program product or computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided by the above aspects.

It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

In the embodiment of the present application, a method for tracking a target object is provided. Firstly, the current image frame is obtained, and then it is judged whether the target object is detected in the previous image frame of the current image frame. Bounding box, extract the original detection image from the current image frame. Next, the target feature map of the original detection image is obtained through the feature extraction network, and then the T group of bounding box parameters corresponding to the target feature map can be obtained through the object detection network. Finally, combined with the T group of bounding box parameters to determine the target object in the current image frame tracking results. Through the above method, on the one hand, the original detection image containing the target object is extracted from the current image frame for subsequent detection. Compared with directly detecting the entire current image frame, the amount of data processing is reduced. At the same time, the target The feature map is split into multiple feature maps for detection separately, and tracking can be regarded as a small-scale detection problem, so as to solve the problem of scale change and displacement change of the tracked object and improve the efficiency of target tracking. On the other hand, using the object detection network to detect the bounding box of the original detection image, compared with the traditional tracking algorithm, can achieve better recognition effect in the image detection process based on the characteristics of the learned target object, which is conducive to improving Accuracy of object tracking.

Description of drawings

FIG. 1 is a schematic diagram of a scene of a target object tracking method in an embodiment of the present application;

FIG. 2 is a schematic diagram of another scene of the target object tracking method in the embodiment of the present application;

FIG. 3 is a schematic flow diagram of a target object tracking method in an embodiment of the present application;

FIG. 4 is a schematic diagram of object detection based on the anchor point mechanism in the embodiment of the present application;

FIG. 5 is a schematic diagram of target object tracking based on inter-frame control in an embodiment of the present application;

Fig. 6 is a schematic diagram of adjusting the size of the original detection image in the embodiment of the present application;

FIG. 7 is a schematic diagram of tracking results generated based on the anchor point mechanism in the embodiment of the present application;

Fig. 8 is a schematic diagram of the tracking result obtained by fusion in the embodiment of the present application;

FIG. 9 is a schematic diagram of triggering application functions based on motion trajectories in the embodiment of the present application;

FIG. 10 is a schematic diagram of triggering an application function based on a moving distance in an embodiment of the present application;

FIG. 11 is another schematic flowchart of the target object tracking method in the embodiment of the present application;

FIG. 12 is a schematic diagram of object detection based on a single border in the embodiment of the present application;

FIG. 13 is another schematic diagram of object detection based on a single border in the embodiment of the present application;

FIG. 14 is a schematic diagram of a target object tracking device in an embodiment of the present application;

Fig. 15 is another schematic diagram of the target object tracking device in the embodiment of the present application;

FIG. 16 is a schematic structural diagram of a terminal device in an embodiment of the present application.

Detailed ways

Embodiments of the present application provide a method for tracking a target object, a related device, equipment, and a storage medium. On the one hand, it reduces the amount of data processing and improves the efficiency of target tracking. On the other hand, a better recognition effect is achieved in the image detection process, which is beneficial to improve the accuracy of target tracking.

The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and not necessarily Used to describe a specific sequence or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein, for example, can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "corresponding to" and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product or device comprising a sequence of steps or elements need not be limited to the expressly listed Instead, other steps or elements not explicitly listed or inherent to the process, method, product or apparatus may be included.

Object tracking is an important branch of computer vision (CV), which is an interdisciplinary study of pattern recognition, image processing and machine learning, and has a wide range of applications, such as video surveillance, virtual reality, human-computer interaction, image Understanding and unmanned driving, etc. The present application provides a method for tracking a target object, which can take into account both tracking efficiency and tracking accuracy, and is suitable for mobile terminal applications.

Taking the human hand as the target object to be tracked as an example, as the basis for the mobile terminal application of the human hand correlation algorithm, it includes key point detection of the human hand, gesture recognition of the human hand, and 3D pose estimation of the human hand, etc. In the current short video and small video era, users often have a lot of gesture interaction actions to trigger various special effects pendants and other gameplays when using it. Therefore, it is very important to develop a high-speed and accurate tracking algorithm for mobile terminals. For some application products, there may be a wide distribution of mobile phones that need to be adapted from low-end phones to high-end phones. It is necessary to achieve real-time tracking effects on the lowest-end mobile phones and occupy as little computing resources and resources as possible. Improve the battery life of the mobile phone. Based on this premise, it is necessary to implement a fast and stable tracking algorithm, but also to achieve low latency and low power consumption, so that it can be well embedded into the tracking system of the terminal device and improve the overall stability and user experience. experience.

The present application proposes a method for tracking a target object. The method is applied to a target object tracking system. The target object tracking system includes a terminal device, or the target object tracking system includes a terminal device and a server. This application is described by taking running on a terminal device as an example, and a client is deployed on the terminal device. The client can run on the terminal device in the form of a browser, or can run on the terminal device in the form of an independent application program (APP), etc. The specific display form of the client is not limited here. The terminal device may be a smart phone, a tablet computer, a laptop computer, a handheld computer, a personal computer, a smart TV, a smart watch, a vehicle device, a wearable device, etc., but is not limited thereto.

Exemplarily, target tracking can be realized in a single-person interaction scene. For the convenience of description, please refer to FIG. 1. FIG. 1 is a schematic diagram of a scene of a target object tracking method in an embodiment of the present application. There are interactive prompts displayed, and users can follow the interactive prompts to perform corresponding actions. The terminal device collects the video through the camera, and analyzes the image frames in the video to achieve the purpose of tracking the human hand.

Exemplarily, target tracking can be realized in a multi-person interaction scene. For the convenience of description, please refer to FIG. 2. FIG. 2 is a schematic diagram of another scene of the target object tracking method in the embodiment of the present application. There are interactive prompts displayed on the screen, and user A and user B can respectively follow the interactive prompts to perform corresponding actions. The terminal device collects the video through the camera, and analyzes the image frames in the video to achieve the purpose of tracking the human hand.

The present application proposes a method for tracking a target object, which specifically relates to CV technology and machine learning (machine learning, ML) technology in the field of artificial intelligence (AI). Among them, CV is a science that studies how to make machines "see". Furthermore, it refers to the use of cameras and computers instead of human eyes to identify, track and measure targets, and further graphics processing to make computers It is processed into an image that is more suitable for human observation or sent to the instrument for detection. As a scientific discipline, CV studies related theories and technologies, trying to build AI systems that can obtain information from images or multidimensional data. CV technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content/behavior recognition, 3D object reconstruction, 3D technology, virtual reality , augmented reality, simultaneous positioning and map construction, autonomous driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition. ML is a multi-field interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. Specializes in the study of how computers simulate or implement human learning behaviors to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. ML is the core of AI and the fundamental way to make computers intelligent, and its application pervades all fields of AI. ML and deep learning usually include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching-based learning.

Among them, AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, AI is a synthesis of computer science that attempts to understand the nature of intelligence and produce a new class of intelligent machines that respond in ways similar to human intelligence. AI is to study the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making.

AI technology is a comprehensive subject that involves a wide range of fields, including both hardware-level technology and software-level technology. AI basic technologies generally include technologies such as sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation/interaction systems, and mechatronics. AI software technology mainly includes CV technology, speech processing technology, natural language processing technology, machine learning/deep learning, automatic driving, intelligent transportation and other major directions.

With the research and progress of AI technology, AI technology has been researched and applied in many fields, such as common smart home, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots , smart medical care, smart customer service, Internet of Vehicles, autonomous driving, smart transportation, etc. I believe that with the development of technology, AI technology will be applied in more fields and play an increasingly important role.

In combination with the above introduction, the solution provided by the embodiment of this application involves technologies such as AI CV and ML. The following will introduce the tracking method of the target object in this application. Please refer to Figure 3, an implementation of the target object tracking method in the embodiment of this application Examples include:

110. If the target object is detected in the previous image frame of the current image frame, extract an original detection image from the current image frame according to the target bounding box corresponding to the previous image frame, wherein the original detection image includes the target object;

In one or more embodiments, the target object tracking device acquires the current image frame and the previous image frame of the current image frame, and determines whether a target object (for example, a human hand) is detected in the previous image frame, and if the target object is detected , then the approximate range of the target object in the current image frame can be determined based on the tracking result (ie, the target bounding box) of the previous image frame.

Specifically, assuming that the size of the target bounding box is 500×500, in one implementation, an 800×800 region is taken as the original detection image centered on the center point of the target bounding box. In another implementation, based on the location and size of the target bounding box, combined with the moving speed and moving direction of the target object, an area may be estimated as the original detection image, which is not limited here.

It should be noted that the target object tracking device can be deployed on a terminal device, or deployed in a system composed of a terminal device and a server. This application takes the deployment of the target object tracking device on a terminal device as an example for illustration.

120. Based on the original detection image, obtain a target feature map through a feature extraction network, wherein the target feature map includes K feature points, and the target feature map is divided into N feature maps, and each feature map is set with an anchor point, K is an integer greater than 1, and N is an integer greater than or equal to 1 and less than K;

In one or more embodiments, a trained feature extraction network may be used to perform feature extraction on the original detection image, thereby obtaining a target feature map. Taking the size of the target feature map as 40×40 as an example, the target feature map includes 1600 feature points. The features of the original detection image are extracted through the feature extraction network, and the target feature map is obtained. Divide the target feature map into N feature maps (for example, N is equal to 9), and perform detection on each feature map separately.

130. For each feature map, obtain an anchor frame according to the anchor points set in the feature map;

In one or more embodiments, an anchor point (anchor) is set at the center of each feature map, and an anchor point frame can be obtained according to the anchor point, and N anchor point frames can be obtained.

140. For each feature map, based on the features corresponding to the anchor box, obtain at least one set of optional bounding box parameters through the object detection network, where each set of optional bounding box parameters includes coordinate parameters and dimensions of the optional bounding box parameter;

In one or more embodiments, for each feature map, the selected features of the anchor box are used as the input of the object detection network, thereby obtaining at least one set of optional bounding box parameters (for example, 9 sets of optional Bounding box parameters), where each set of optional bounding box parameters includes coordinate parameters and size parameters of an optional bounding box.

Specifically, for ease of understanding, please refer to FIG. 4. FIG. 4 is a schematic diagram of object detection based on the anchor point mechanism in the embodiment of the present application. As shown in the figure, the target feature map is divided into 9 feature maps (ie, 3 ×3 grid), an anchor point is set at the center of each feature map, and the anchor point box can be obtained according to the anchor point. The features selected by the anchor box are used as the input of the object detection network, and the object detection network outputs at least one set of optional bounding box parameters, for example, output 9 sets of optional bounding box parameters, and each set of optional bounding box parameters includes optional boundaries Coordinate parameters and size parameters of the box.

150. Determine T groups of bounding box parameters according to at least one group of optional bounding box parameters of each feature map, where each group of bounding box parameters includes coordinate parameters and size parameters of the bounding box, and T is an integer greater than or equal to 1;

In one or more embodiments, for each feature map, optional bounding box parameters that meet requirements are selected from at least one set of optional bounding box parameters as output bounding box parameters. Finally, a total of T groups of bounding box parameters are obtained based on the N feature maps. Similarly, each group of bounding box parameters includes the coordinate parameters and size parameters of the bounding box.

160. Determine a tracking result for the target object in the current image frame according to T groups of bounding box parameters.

In one or more embodiments, the position of the target object in the original detection image can be determined according to the T group of bounding box parameters, and then the tracking result of the target object in the current image frame can be determined. Among them, the tracking result can be represented as a bounding box containing the target object. If the target object is not detected in the current image frame, the tracking result may represent that the target object is not tracked.

The feature extraction network and object detection network provided in this application occupy a storage space of about 50 kilobytes (KB) in total, and can achieve 1.4M floating point operations per second (FLOPs). Compared with the ultra-real-time single target tracking network (Siamese Region Proposal Network, Siamese RPN) and other calculations, there are 3 to 4 orders of magnitude acceleration, and the time consumption on low-end machines (for example, A53 machines) can be compressed to within 1 millisecond. It greatly accelerates the tracking algorithm of the mobile terminal, and effectively reserves more power consumption and time for the downstream algorithm.

In an embodiment of the present application, a method for tracking a target object is provided. Through the above method, on the one hand, the original detection image containing the target object is extracted from the current image frame for subsequent detection. Compared with directly detecting the entire current image frame, the amount of data processing is reduced. At the same time, the target The feature map is split into multiple feature maps for detection separately, and tracking can be regarded as a small-scale detection problem, so as to solve the problem of scale change and displacement change of the tracked object and improve the efficiency of target tracking. On the other hand, using the object detection network to detect the bounding box of the original detection image, compared with the traditional tracking algorithm, can achieve better recognition effect in the image detection process based on the characteristics of the learned target object, which is conducive to improving Accuracy of object tracking.

Optionally, on the basis of the various embodiments corresponding to FIG. 3 above, another optional embodiment provided by the embodiment of the present application may further include:

If the target object is not detected in the previous image frame of the current image frame, object recognition is performed on the current image frame;

If the previous image frame of the current image frame is not acquired, object recognition is performed on the current image frame.

In one or more embodiments, a method for implementing target tracking based on inter-frame control is introduced. It can be known from the foregoing embodiments that if the current image frame belongs to the first image frame in the video, a detection algorithm may be used to detect whether there is a target object in the current image frame.

Specifically, for ease of understanding, please refer to FIG. 5. FIG. 5 is a schematic diagram of target object tracking based on inter-frame control in the embodiment of the present application. As shown in the figure, assuming that the current image frame has a previous image frame, in step A1 In , it is judged based on inter-frame control whether the target object is detected in the previous image frame, and if there is a bounding box corresponding to the target object in the previous frame, step A2 is performed, that is, the target object is tracked. If there is a bounding box corresponding to the target object in the previous frame, perform step A3, that is, use a detection algorithm to detect the current image frame to detect the target object, and reset the tracker according to the detected target object. Resetting the tracker refers to clearing the parameters in the tracker and updating the parameters in the tracker based on the detection results.

Wherein, the tracker includes a feature extraction network and an object detection network. In addition, the tracker may also include a tracking algorithm, such as a KCF algorithm.

Secondly, in the embodiment of the present application, a method of realizing target tracking based on inter-frame control is provided. Through the above method, only the image that was empty in the previous frame is detected, and detection of each frame image is avoided, thereby reducing Computing power and resources consumed by image detection.

Optionally, on the basis of the various embodiments corresponding to FIG. 3 above, in another optional embodiment provided by the embodiment of the present application, based on the original detection image, the target feature map is obtained through a feature extraction network, which may specifically include:

Adjust the size of the original detection image to obtain the image to be processed;

If the size of the image to be processed fails to match the preset size, the image to be processed is filled to obtain a target detection image that meets the preset size;

Based on the target detection image, the target feature map is output through the feature extraction network.

In one or more embodiments, a method for performing size transformation on an original detected image is introduced. It can be seen from the foregoing embodiments that the original detection image is a part of the image taken from the current image frame. Based on this, the original detection image can be resized and filled (padding) to obtain a uniform size and fidelity target detection image .

Specifically, please refer to FIG. 6. FIG. 6 is a schematic diagram of adjusting the size of the original detection image in the embodiment of the present application. As shown in the figure, assuming that the size of the original detection image is 450×1200, the length of the original detection image and The width is respectively reduced to 1/3 of the original, that is, the image to be processed is obtained, wherein the image to be processed is 150×400. Assuming that the preset size is 400×400, if the size of the image to be processed fails to match the preset size, the image to be processed will be processed by padding to obtain a target detection image that meets the preset size. The target detection image is used as the input of the feature extraction network, and the target feature map is output through the feature extraction network.

It can be understood that the feature extraction network includes a convolutional layer, a pooling layer, and a fully connected layer, and this application does not limit the number of network layers.

Secondly, in the embodiment of the present application, a method of resizing the original detection image is provided. Through the above method, on the one hand, the size of the original detection image can be reduced, thereby reducing the amount of data processing, which is conducive to improving tracking efficiency and saving terminals. The computing power of the device. On the other hand, the fidelity of the image can be achieved by filling to achieve a better detection effect.

Optionally, on the basis of the various embodiments corresponding to FIG. 3 above, in another optional embodiment provided by the embodiment of the present application, each set of optional bounding box parameters further includes a confidence parameter, and at least one set of optional bounding box parameters Box parameters include multiple sets of optional bounding box parameters;

Determine T groups of bounding box parameters according to at least one set of optional bounding box parameters of each feature map, which may specifically include:

For each feature map, select an optional bounding box parameter with a maximum confidence parameter from multiple sets of optional bounding box parameters, wherein each set of optional bounding box parameters is used to determine an optional bounding box;

For each feature map, determine an initial optional bounding box according to the optional bounding box parameter with the maximum confidence parameter;

For each feature map, traverse the remaining optional bounding boxes. If the overlapping area between the optional bounding box and the initial optional bounding box is greater than or equal to the area threshold, delete the optional bounding box until the remaining optional bounding boxes are obtained. bounding boxes, where the remaining optional bounding boxes represent optional bounding boxes in addition to the initial optional bounding box;

For each feature map, the optional bounding box parameters corresponding to the remaining optional bounding boxes are used as bounding box parameters.

In one or more embodiments, a method of filtering out bounding boxes using non maximum suppression (NMS) is introduced. As can be seen from the foregoing embodiments, for each feature map, multiple sets of optional bounding box parameters may be included, a set of optional bounding box parameters corresponds to an optional bounding box, and the optional bounding box parameters also include confidence parameters .

Specifically, taking any one of the N feature maps as an example, it is assumed that 9 sets of optional bounding box parameters are obtained for the feature map, that is, there are 9 optional bounding boxes. First, the optional bounding box with the largest confidence parameter is taken from the nine optional bounding boxes as the initial optional bounding box, and the remaining eight optional bounding boxes are traversed to determine the difference between each optional bounding box and the initial optional bounding box. Whether the overlapping area (that is, the intersection ratio) between the selected bounding boxes is greater than or equal to the area threshold, for the optional bounding boxes greater than or equal to the area threshold, delete them directly, and keep the remaining optional bounding boxes. Based on this, the optional bounding box with the maximum confidence parameter is selected from the remaining optional bounding boxes as the initial optional bounding box, and then the above process is repeated to finally obtain the remaining optional bounding boxes.

Thus, the remaining optional bounding boxes are used as the bounding boxes of the feature map output, and the corresponding bounding box parameters are obtained.

Again, in the embodiment of the present application, a method of screening out bounding boxes by using NMS is provided. Through the above-mentioned method, the bounding boxes containing the target object can be accurately screened out, thereby improving the reliability of detection.

Optionally, on the basis of the various embodiments corresponding to FIG. 3 above, in another optional embodiment provided by the embodiment of the present application, the tracking result for the target object in the current image frame is determined according to the T group of bounding box parameters, specifically Can include:

Determine T bounding boxes according to T groups of bounding box parameters;

Determine the leftmost vertex, the rightmost vertex, the uppermost vertex, and the lowermost vertex according to the T bounding boxes;

Determine the coordinates of the upper left vertex, the lower left vertex, the upper right vertex, and the lower right vertex according to the leftmost vertex, the rightmost vertex, the uppermost vertex, and the lowermost vertex;

According to the coordinates of the upper left vertex, the lower left vertex, the upper right vertex and the lower right vertex, the tracking result for the target object is determined.

In one or more embodiments, a way of determining tracking results based on a plurality of bounding boxes is presented. It can be seen from the foregoing embodiments that after the target feature map is divided into N feature maps, its corresponding bounding box can be obtained for each feature map, and a final bounding box is constructed by combining each bounding box as the tracking result for the target object.

Specifically, for ease of understanding, please refer to FIG. 7. FIG. 7 is a schematic diagram of the tracking results generated based on the anchor point mechanism in the embodiment of the present application. As shown in the figure, it is assumed that the target feature map is divided into 9 feature maps, each Each feature map detects a bounding box, that is, T is equal to N. Determine the leftmost vertex (i.e., point A in the illustration), the rightmost vertex (i.e., point B in the illustration), and the uppermost vertex (i.e., point C in the illustration) from these T bounding boxes ) and the lowermost vertex (ie, point D in the illustration). Based on this, a rectangular frame can be constructed according to the leftmost vertex, the rightmost vertex, the uppermost vertex, and the lowermost vertex, so as to obtain the coordinates of the upper left vertex (that is, point E in the illustration), and the coordinates of the lower left vertex (that is, point E in the figure) point F in the illustration), coordinates of the upper right vertex (that is, point G in the illustration), and coordinates of the lower right apex (that is, point H in the illustration). Thus, a tracking result for the target object is obtained.

Again, in the embodiment of the present application, a method of determining a tracking result based on multiple bounding boxes is provided. Through the above method, the results of T bounding boxes can be integrated to generate a more accurate bounding box, thereby realizing target tracking.

Optionally, on the basis of the various embodiments corresponding to FIG. 3 above, another optional embodiment provided by the embodiment of the present application may further include:

acquiring an image sample, wherein the image sample includes a target object;

Based on the image sample, a sample feature map is obtained through a feature extraction network, wherein the sample feature map includes K feature points, the sample feature map is divided into N feature maps to be trained, and an anchor point is set in each feature map to be trained;

For each feature map to be trained, an anchor frame is obtained according to the anchor points set in the feature map to be trained;

For each feature map to be trained, based on the features corresponding to the anchor box and the real bounding box parameters, at least one set of optional bounding box parameters is obtained through the object detection network, wherein each set of optional bounding box parameters includes the bounding box to be trained The coordinate parameters and size parameters of ;

For each feature map to be trained, determine predicted bounding box parameters from at least one set of optional bounding box parameters;

For each feature map to be trained, a loss function is used to update the model parameters of the feature extraction network and the object detection network according to the predicted bounding box parameters and the real bounding box parameters.

In one or more embodiments, a way of training an object detection network is presented. It can be seen from the foregoing embodiments that an image sample containing a target object (for example, a human hand) can be obtained, and a frame where the target object is located can be marked in the image sample. The image sample is used as the input of the feature extraction network, and thus the sample feature map is output. Similarly, the sample feature map is divided into N feature maps to be trained (for example, 3×3 feature maps to be trained), and each feature map to be trained is detected separately. An anchor point can be set at the center position of each feature map to be trained, and a corresponding anchor point frame can be obtained, based on the anchor point frame, the feature corresponding to the anchor point frame can be extracted.

In the process of training the object detection network, the method of bounding box regression can be used to regress the anchor box corresponding to each anchor point. Based on this, the features corresponding to the anchor box and the ground truth parameters are used as the input of the object detection network, thereby outputting at least one set of optional bounding box parameters, determined from at least one set of optional bounding box parameters Predict bounding box parameters.

Specifically, assume that the parameters of the anchor box are (Px, Py, Pw, Ph), the parameters of the real bounding box are (Gx, Gy, Gw, Gh), and the parameters of the predicted bounding box are (G'x, G'y, G 'w,G'h), thus, it is necessary to find a mapping f such that:

f(Px,Py,Pw,Ph)=(G'x,G'y,G'w,G'h);

and makes:

(G'x,G'y,G'w,G'h)≈(Gx,Gy,Gw,Gh);

The process of transforming the anchor box into the predicted bounding box is first translated and then scaled. The Smooth L1 loss function or L2 loss function can be used to calculate the loss value between the predicted value and the real value, respectively, where the real value refers to the translation and scaling determined based on the parameters of the real bounding box and the parameters of the anchor box. Finally, combined with the sum of the loss values, the model parameters of the feature extraction network and the object detection network are updated.

Thirdly, in the embodiment of the present application, a method of training the object detection network is provided. Through the above method, the method of regressing the anchor frame is used to train the object detection network, and the training process is relatively simple.

Optionally, on the basis of the various embodiments corresponding to FIG. 3 above, another optional embodiment provided by the embodiment of the present application may further include:

Use a preset tracking algorithm to detect the current image frame to obtain the first center position parameter of the target object, wherein the preset tracking algorithm is a kernel correlation filtering algorithm, an online real-time tracking algorithm, a context-aware correlation filtering algorithm or a multi-instance online learning algorithm ;

Determine the tracking result for the target object in the current image frame according to the T group of bounding box parameters, which may specifically include:

Determine the second center position parameter of the target object according to the T group of bounding box parameters;

According to the first center position parameter and the second center position parameter, a tracking result for the target object in the current image frame is determined.

In one or more embodiments, a manner of implementing fusion positioning combined with a preset tracking algorithm is introduced. It can be known from the foregoing embodiments that the preset tracking algorithm can quickly realize target tracking, and can perform tracking when the target object moves rapidly. However, since the accuracy of the preset tracking algorithm is low, the tracking result obtained by the preset tracking algorithm can be used as the motion compensation of the output result of the object detection network.

Specifically, for ease of understanding, please refer to FIG. 8 . FIG. 8 is a schematic diagram of a tracking result obtained through fusion in an embodiment of the present application. As shown in the figure, F1 is used to indicate a bounding box determined by a preset tracking algorithm. F2 is used to indicate that a preset tracking algorithm is used to determine the center point of the bounding box. F3 is used to indicate the bounding box determined based on the T set of bounding box parameters. F4 is used to indicate to determine the center point of the bounding box based on the T set of bounding box parameters.

Assume that the first center position parameter corresponding to the center point indicated by F2 is (x1, y1), the second center position parameter corresponding to the center point indicated by F4 is (x2, y2), and assume that the weight of the first center position parameter It is set to 0.2, and the weight of the second center position parameter is set to 0.8. Based on this, combined with the first center position parameter and the second center position parameter, the final center position parameter of the bounding box is determined to be (x3, y3), namely:

x3＝0.2*x1+0.8*x2;

y3=0.2*y1+0.8*y2;

It can be seen that the preset tracking algorithm is equivalent to making an offset for the output of the object detection network.

It should be noted that the preset tracking algorithms include but not limited to kernel correlation filter (kernel correlation filter, KCF) algorithm, simple online and realtime tracking (simple online and realtime tracking, SORT) algorithm, background-aware correlation filters (background-aware correlation filters, BACF) algorithm and multi-instance online learning (multi instance learning, MIL) algorithm.

Secondly, in the embodiment of the present application, a method for realizing fusion positioning combined with a preset tracking algorithm is provided. Through the above method, considering that the target object is prone to being lost in the case of fast movement, it is necessary to frequently The detection function is invoked frequently, resulting in higher power consumption of the terminal device. Therefore, combined with the preset tracking algorithm, the position of the target object can be quickly located, the calling frequency of the detection function is reduced, and the power consumption of the terminal device is reduced.

Optionally, on the basis of the various embodiments corresponding to FIG. 3 above, in another optional embodiment provided by the embodiment of the present application, the target object is a human hand;

Can also include:

In response to the operation of the video capture control, activate the camera device of the terminal device;

Collecting M consecutive image frames through the camera device, where M is an integer greater than or equal to 1;

After determining the tracking result for the target object in the current image frame according to the T group of bounding box parameters, it may also include:

Obtain the tracking results for human hands in consecutive M image frames;

According to the tracking results corresponding to the consecutive M image frames and the tracking results corresponding to the current image frame, a moving track for the human hand is generated;

If the moving track matches the preset track successfully, the function corresponding to the preset track is triggered.

In one or more embodiments, a manner of triggering a corresponding function based on a motion trajectory is introduced. It can be seen from the above-mentioned embodiments that the terminal device can collect video through the camera device, and analyze the image frames in the video, and then extract the movement track of the human hand.

Specifically, for ease of understanding, please refer to FIG. 9 . FIG. 9 is a schematic diagram of triggering an application function based on a motion trajectory in an embodiment of the present application. As shown in the figure, the target object is a human hand as an example. When the user clicks on the control of "starting the camera", an operation on the video shooting control is triggered, and then the terminal device starts the camera device and collects the video. Assuming that M consecutive image frames are collected by the camera device, and the tracking result of each frame in the M consecutive image frames can be used to obtain the moving trajectory of the human hand, so that the moving trajectory of the human hand can be matched with the preset trajectory.

If the moving track matches the preset track successfully, the function corresponding to the preset track will be triggered, for example, providing corresponding rewards for users, or displaying corresponding special effects, or enabling a specific function, etc., which will not be done here limited.

Secondly, in the embodiment of the present application, a method of triggering corresponding functions based on the motion trajectory is provided. Through the above method, in products such as short video editing or short video special effects, the motion trajectory can be determined based on the tracking result of the human hand. According to the motion The trace triggers the corresponding function. It can be seen that the solution provided by this application can be well generalized to various mobile object tracking tasks, and it can handle objects with different postures, angles and occlusions more robustly, so as to achieve the purpose of improving user experience.

Optionally, on the basis of the various embodiments corresponding to FIG. 3 above, another optional embodiment provided by the embodiment of the present application may further include:

In response to the operation of the video capture control, activate the camera device of the terminal device;

Collecting M consecutive image frames through the camera device, where M is an integer greater than or equal to 1;

After determining the tracking result for the target object in the current image frame according to the T group of bounding box parameters, it may also include:

Obtain the tracking results for human hands in consecutive M image frames;

Obtain the trajectory length according to the tracking results corresponding to the consecutive M image frames and the tracking results corresponding to the current image frame;

If the track length is greater than or equal to the length threshold, the function corresponding to the preset track is triggered.

In one or more embodiments, a manner of triggering a corresponding function based on a moving distance is introduced. It can be seen from the above-mentioned embodiments that the terminal device can collect video through the camera device, and analyze the image frames in the video, and then extract the movement track of the human hand.

Specifically, for ease of understanding, please refer to FIG. 10 . FIG. 10 is a schematic diagram of triggering an application function based on a moving distance in an embodiment of the present application. As shown in the figure, the target object is a human hand as an example. When the user clicks on the control of "starting the camera", an operation on the video shooting control is triggered, and then the terminal device starts the camera device and collects the video. Assume that M consecutive image frames are collected by the camera device, and a continuous motion trajectory can be determined through the tracking results of each frame in the M consecutive image frames, and thus the trajectory length of the motion trajectory can be calculated.

If the track length is greater than or equal to the length threshold, the function corresponding to the preset track will be triggered, for example, providing corresponding rewards for users, or displaying corresponding special effects, or enabling a specific function, etc., which are not limited here .

Secondly, in the embodiment of the present application, a method of triggering corresponding functions based on the moving distance is provided. Through the above method, in products such as short video editing or short video special effects, the moving distance can be determined based on the tracking result of the human hand. The distance triggers the corresponding function. It can be seen that the solution provided by this application can be well generalized to various mobile object tracking tasks, and it can handle objects with different postures, angles and occlusions more robustly, so as to achieve the purpose of improving user experience.

In combination with the above introduction, the following will introduce the target object tracking method in this application, please refer to Figure 11, another embodiment of the target object tracking method in the embodiment of this application includes:

210. If the target object is detected in the previous image frame of the current image frame, extract an original detection image from the current image frame according to the target bounding box corresponding to the previous image frame, wherein the original detection image includes the target object;

In one or more embodiments, the content described in step 110 in the foregoing embodiments will not be repeated here.

220. Based on the original detection image, the target feature map is obtained through a feature extraction network;

In one or more embodiments, the content described in step 320 in the foregoing embodiments will not be repeated here.

230. Based on the target feature map, obtain a first offset and a second offset through an object detection network, wherein the first offset is an upper left offset, and the second offset is a lower right offset, Or, the first offset is an upper-right offset and the second offset is a lower-left offset;

In one or more embodiments, the target feature map is used as an object detection network, and the object detection network can output the first offset and the second offset.

Exemplarily, in one case, the first offset is the upper left offset, and the second offset is the lower right offset, wherein the upper left offset includes the lateral offset between the upper left vertex and the initial upper left vertex And the vertical offset, the lower right offset includes the horizontal offset and the vertical offset from the lower right vertex to the initial lower right vertex.

Exemplarily, another situation is that the first offset is the upper right offset, and the second offset is the lower left offset, wherein the upper right offset includes the lateral offset between the upper right vertex and the initial upper right vertex And the vertical offset, the lower left offset includes the horizontal offset and the vertical offset from the lower left vertex to the initial lower left vertex.

240. Determine a tracking result for the target object in the current image frame according to the first offset and the second offset.

In one or more embodiments, a set of bounding box parameters (x1, y1, n1, m1) is obtained according to the first offset (x1, y1) and the coordinate parameters (n1, m1) of the initial vertex. Similarly, another set of bounding box parameters (x2, y2, n2, m2) is obtained according to the second offset (x2, y2) and the coordinate parameters (n2, m2) of the initial vertex of the diagonal.

Exemplarily, the two diagonal vertices may be an upper left vertex and a lower right vertex, which will be described below with reference to figures.

Specifically, for ease of understanding, please refer to FIG. 12. FIG. 12 is a schematic diagram of object detection based on a single border in the embodiment of the present application. As shown in the figure, the corresponding target feature map can be obtained based on the original detection image, and the target feature The graph is input to the object detection network, and the offset of point a (ie, the upper left offset) and the offset of point b (ie, the lower right offset) are output through the object detection network. Among them, the upper left offset includes the horizontal offset (ie, x1) and the vertical offset (ie, y1) of the upper left vertex (ie, point a) from the initial upper left vertex (ie, point c), and the lower right offset The amount includes a horizontal offset (ie, x2 ) and a vertical offset (ie, y2 ) from the lower right vertex (ie, point b) to the initial lower right vertex (ie, point d).

Based on this, a two-dimensional coordinate system as shown in Figure 12 can be constructed, and a set of bounding box parameters (x1, y1, n1, m1). Similarly, according to the lower right offset (x2, y2) and the coordinate parameters (n2, m2) of the initial lower right vertex, another set of bounding box parameters (x2, y2, n2, m2) is obtained.

Exemplarily, the two diagonal vertices may be an upper right vertex and a lower left vertex, which will be described below with reference to figures.

Specifically, for ease of understanding, please refer to FIG. 13. FIG. 13 is another schematic diagram of object detection based on a single border in the embodiment of the present application. As shown in the figure, the corresponding target feature map can be obtained based on the original detection image, and the target The feature map is input to the object detection network, and the offset of point a (ie, the lower left offset) and the offset of point b (ie, the upper right offset) are output through the object detection network. Among them, the lower left offset includes the horizontal offset (ie, x1) and vertical offset (ie, y1) from the lower left vertex (ie, point a) to the initial lower left vertex (ie, point c), and the upper right offset It includes a horizontal offset (ie, x2) and a vertical offset (ie, y2) from the upper right vertex (ie, point b) to the initial upper right vertex (ie, point d).

Based on this, a two-dimensional coordinate system as shown in Figure 13 can be constructed, and a set of bounding box parameters (x1, y1, n1, m1). Similarly, according to the upper right offset (x2, y2) and the coordinate parameters (n2, m2) of the initial upper right vertex, another set of bounding box parameters (x2, y2, n2, m2) is obtained.

In an embodiment of the present application, a method for tracking a target object is provided. Through the above method, the position of the bounding box can be determined only by the offset of the diagonal vertices, which can simplify the complexity of model learning. Therefore, the complexity of the model can be further reduced, and the effect of accelerated tracking can be achieved.

Optionally, on the basis of the various embodiments corresponding to FIG. 3 above, in another optional embodiment provided by the embodiment of the present application, the first offset is the upper left offset, and the second offset is the right Lower offset, where the upper left offset includes the horizontal offset and vertical offset from the upper left vertex to the initial upper left vertex, and the lower right offset includes the horizontal offset and vertical offset from the lower right vertex to the initial lower right vertex Offset;

Can also include:

acquiring an image sample, wherein the image sample includes the target object;

Based on the image sample, the sample feature map is obtained through the feature extraction network;

Based on the sample feature map, the upper left prediction offset and the lower right prediction offset are obtained through the object detection network;

Based on the sample feature map, according to the upper left predicted offset, the lower right predicted offset, the upper left real offset and the lower right real offset, the model parameters of the feature extraction network and the object detection network are updated using a loss function.

In one or more embodiments, a way of training an object detection network is presented. It can be known from the foregoing embodiments that the object detection network needs to be trained so that it can fit more accurate results. It can be understood that the feature extraction network and the object detection network adopt a joint training method.

Specifically, an image sample will be used as an example for introduction below. In actual training, other image samples are processed in a similar manner. There is a target object in the image sample, and the bounding box containing the target object can be circled by manual labeling. The image sample is input to the feature extraction network, and the feature extraction network outputs the sample feature map. Based on this, the upper left prediction offset and the lower right prediction offset can be obtained by using the sample feature map as the input of the object detection network. Since the bounding box has been pre-marked in the image sample, the upper-left true offset and the lower-right true offset can be determined in the sample feature map.

Based on this, the loss value of the position of the same vertex (for example, upper left vertex or lower right vertex) can be calculated as follows:

Among them, L represents the loss value. n represents the total number of image samples. y _i represents the horizontal offset or vertical offset in the i-th image sample corresponding to the real offset (the upper left real offset or the lower right real offset). f( _xi ) represents the horizontal offset or the vertical offset in the i-th image sample corresponding to the predicted offset (upper-left predicted offset or lower-right predicted offset).

Finally, the model parameters of the feature extraction network and the object detection network are updated in combination with the loss value.

Again, in the embodiment of the present application, a method of training the object detection network is provided. Through the above method, on the one hand, the L1 loss can be used to return the offset of the upper left vertex and the lower right vertex. The L1 loss is conducive to optimizing small disturbances. And regressing offsets is simpler and more effective than regressing similar sizes. On the other hand, it is easy to construct training samples, and a large number of samples can be generated for training the entire model. This scheme can be generalized to most tracking tasks on mobile terminals.

Optionally, on the basis of the various embodiments corresponding to FIG. 3 above, in another optional embodiment provided by the embodiment of the present application, the first offset is the upper right offset, and the second offset is the lower left An offset, wherein the upper right offset includes a horizontal offset and a vertical offset from the upper right vertex to the initial upper right vertex, and the lower left offset includes a horizontal offset and a vertical offset from the lower left vertex to the initial lower left vertex;

Can also include:

acquiring an image sample, wherein the image sample includes the target object;

Based on the image sample, the sample feature map is obtained through the feature extraction network;

Based on the sample feature map, the upper right prediction offset and the lower left prediction offset are obtained through the object detection network;

Based on the sample feature map, according to the upper right predicted offset, the lower left predicted offset, the upper right real offset and the lower left real offset, the model parameters of the feature extraction network and the object detection network are updated using a loss function.

In one or more embodiments, a way of training an object detection network is presented. It can be known from the foregoing embodiments that there is a target object in the image sample, and the bounding box containing the target object can be circled by manual labeling. The image samples are input to the feature extraction network, and thus the sample feature maps are obtained. Based on this, the upper-right prediction offset and the lower-left prediction offset can be obtained by using the sample feature map as the input of the object detection network. Since the bounding box has been pre-marked in the image sample, the upper-right true offset and the lower-left true offset can be determined in the sample feature map.

Based on this, the loss value of the same vertex (for example, lower left vertex or upper right vertex) in position can be calculated as follows:

Among them, L represents the loss value. n represents the total number of image samples. y _i represents the horizontal offset or the vertical offset in the i-th image sample corresponding to the real offset (lower left real offset or upper right real offset). f( _xi ) represents the horizontal offset or the vertical offset in the i-th image sample corresponding to the predicted offset (lower left predicted offset or upper right predicted offset).

Finally, the model parameters of the feature extraction network and the object detection network are updated in combination with the loss value.

Again, in the embodiment of the present application, a method of training the object detection network is provided. Through the above method, on the one hand, the L1 loss can be used to return the offset of the upper right vertex and the lower left vertex. The L1 loss is conducive to optimizing small disturbances, and Regressing offsets is simpler and more efficient than regressing similar dimensions. On the other hand, it is easy to construct training samples, and a large number of samples can be generated for training the entire model. This scheme can be generalized to most tracking tasks on mobile terminals.

Optionally, on the basis of the above-mentioned embodiments corresponding to FIG. 11 , another optional embodiment provided by the embodiment of the present application may further include:

Use a preset tracking algorithm to detect the current image frame to obtain the first center position parameter of the target object, wherein the preset tracking algorithm is a kernel correlation filtering algorithm, an online real-time tracking algorithm, a context-aware correlation filtering algorithm or a multi-instance online learning algorithm ;

According to the first offset and the second offset, determine the tracking result for the target object in the current image frame, including:

Determine a second center position parameter of the target object according to the first offset and the second offset;

According to the first center position parameter and the second center position parameter, a tracking result for the target object in the current image frame is determined.

In one or more embodiments, a manner of implementing fusion positioning combined with a preset tracking algorithm is introduced. It can be known from the foregoing embodiments that the preset tracking algorithm can quickly realize target tracking, and can perform tracking when the target object moves rapidly. However, since the accuracy of the preset tracking algorithm is low, the tracking result obtained by the preset tracking algorithm can be used as the motion compensation of the output result of the object detection network.

Specifically, for ease of understanding, please refer to FIG. 8 again. As shown in the figure, F1 is used to indicate a bounding box determined by using a preset tracking algorithm. F2 is used to indicate that a preset tracking algorithm is used to determine the center point of the bounding box. F3 is used to indicate the bounding box determined based on the first offset and the second offset. F4 is used to indicate to determine the center point of the bounding box based on the first offset and the second offset.

Assume that the first center position parameter corresponding to the center point indicated by F2 is (x1, y1), the second center position parameter corresponding to the center point indicated by F4 is (x2, y2), and assume that the weight of the first center position parameter It is set to 0.2, and the weight of the second center position parameter is set to 0.8. Based on this, combined with the first center position parameter and the second center position parameter, the final center position parameter of the bounding box is determined to be (x3, y3), namely:

x3＝0.2*x1+0.8*x2;

y3=0.2*y1+0.8*y2;

It can be seen that the preset tracking algorithm is equivalent to making an offset for the output of the object detection network.

It should be noted that the preset tracking algorithms include but not limited to KCF algorithm, SORT algorithm, BACF algorithm and MIL algorithm.

Secondly, in the embodiment of the present application, a method for realizing fusion positioning combined with a preset tracking algorithm is provided. Through the above method, considering that the target object is prone to being lost in the case of fast movement, it is necessary to frequently The detection function is invoked frequently, resulting in higher power consumption of the terminal device. Therefore, combined with the preset tracking algorithm, the position of the target object can be quickly located, the calling frequency of the detection function is reduced, and the power consumption of the terminal device is reduced.

Optionally, on the basis of the above-mentioned embodiments corresponding to FIG. 11 , in another optional embodiment provided by the embodiment of the present application, the target object is a human hand;

Can also include:

In response to the operation of the video capture control, activate the camera device of the terminal device;

Collecting M consecutive image frames through the camera device, where M is an integer greater than or equal to 1;

After determining the tracking result for the target object in the current image frame according to the first offset and the second offset, it may further include:

Obtain the tracking results for human hands in consecutive M image frames;

According to the tracking results corresponding to the consecutive M image frames and the tracking results corresponding to the current image frame, a moving track for the human hand is generated;

If the moving track matches the preset track successfully, the function corresponding to the preset track is triggered.

In one or more embodiments, a manner of triggering a corresponding function based on a motion trajectory is introduced. It can be seen from the above-mentioned embodiments that the terminal device can collect video through the camera device, and analyze the image frames in the video, and then extract the movement track of the human hand.

Specifically, for ease of understanding, please refer to FIG. 9 again. As shown in the figure, the target object is a human hand as an example. When the user clicks on the control of "starting the camera", an operation on the video shooting control is triggered, and then the terminal device starts the camera device and collects the video. Assuming that M consecutive image frames are collected by the camera device, and the tracking result of each frame in the M consecutive image frames can be used to obtain the moving trajectory of the human hand, so that the moving trajectory of the human hand can be matched with the preset trajectory.

If the moving track matches the preset track successfully, the function corresponding to the preset track will be triggered, for example, providing corresponding rewards for users, or displaying corresponding special effects, or enabling a specific function, etc., which will not be done here limited.

Secondly, in the embodiment of the present application, a method of triggering corresponding functions based on the motion trajectory is provided. Through the above method, in products such as short video editing or short video special effects, the motion trajectory can be determined based on the tracking result of the human hand. According to the motion The trace triggers the corresponding function. It can be seen that the solution provided by this application can be well generalized to various mobile object tracking tasks, and it can handle objects with different postures, angles and occlusions more robustly, so as to achieve the purpose of improving user experience.

Optionally, on the basis of the above-mentioned embodiments corresponding to FIG. 11 , another optional embodiment provided by the embodiment of the present application may further include:

In response to the operation of the video capture control, activate the camera device of the terminal device;

Collecting M consecutive image frames through the camera device, where M is an integer greater than or equal to 1;

After determining the tracking result for the target object in the current image frame according to the first offset and the second offset, it may further include:

Obtain the tracking results for human hands in consecutive M image frames;

Obtain the trajectory length according to the tracking results corresponding to the consecutive M image frames and the tracking results corresponding to the current image frame;

If the track length is greater than or equal to the length threshold, the function corresponding to the preset track is triggered.

In one or more embodiments, a manner of triggering a corresponding function based on a moving distance is introduced. It can be seen from the above-mentioned embodiments that the terminal device can collect video through the camera device, and analyze the image frames in the video, and then extract the movement track of the human hand.

Specifically, for ease of understanding, please refer to FIG. 10 again. As shown in the figure, the target object is a human hand as an example. When the user clicks on the control of "starting the camera", an operation on the video shooting control is triggered, and then the terminal device starts the camera device and collects the video. Assume that M consecutive image frames are collected by the camera device, and a continuous motion trajectory can be determined through the tracking results of each frame in the M consecutive image frames, and thus the trajectory length of the motion trajectory can be calculated.

If the track length is greater than or equal to the length threshold, the function corresponding to the preset track will be triggered, for example, providing corresponding rewards for users, or displaying corresponding special effects, or enabling a specific function, etc., which are not limited here .

Secondly, in the embodiment of the present application, a method of triggering corresponding functions based on the moving distance is provided. Through the above method, in products such as short video editing or short video special effects, the moving distance can be determined based on the tracking result of the human hand. The distance triggers the corresponding function. It can be seen that the solution provided by this application can be well generalized to various mobile object tracking tasks, and it can handle objects with different postures, angles and occlusions more robustly, so as to achieve the purpose of improving user experience.

The following is a detailed description of the target object tracking device in this application. Please refer to FIG. 14. FIG. 14 is a schematic diagram of an embodiment of the target object tracking device in the embodiment of the present application. The target object tracking device 30 includes:

The acquiring module 310 is configured to extract an original detection image from the current image frame according to the target bounding box corresponding to the previous image frame if the target object is detected in the previous image frame of the current image frame, wherein the original detection image including target audience;

The obtaining module 310 is also used to obtain the target feature map through the feature extraction network based on the original detection image, wherein the target feature map includes K feature points, and the target feature map is divided into N feature maps, and each feature map is set There is an anchor point, K is an integer greater than 1, N is an integer greater than or equal to 1 and less than K;

The obtaining module 310 is also used for obtaining an anchor point frame according to the anchor points set in the feature map for each feature map;

The acquiring module 310 is further configured to acquire at least one set of optional bounding box parameters through the object detection network based on the features corresponding to the anchor box for each feature map, wherein each set of optional bounding box parameters includes an optional bounding box The coordinate parameters and size parameters of ;

The determining module 320 is configured to determine T groups of bounding box parameters according to at least one set of optional bounding box parameters of each feature map, wherein each group of bounding box parameters includes coordinate parameters and size parameters of the bounding box, and T is greater than or equal to 1 an integer of

The tracking module 330 is configured to determine a tracking result for the target object in the current image frame according to T groups of bounding box parameters.

Optionally, on the basis of the above-mentioned embodiment corresponding to FIG. 14 , in another embodiment of the target object tracking device 30 provided in the embodiment of the present application, the target object tracking device 30 further includes an identification module 340;

The recognition module 340 is used to perform object recognition on the current image frame if no target object is detected in the previous image frame of the current image frame;

The recognition module 340 is further configured to perform object recognition on the current image frame if the previous image frame of the current image frame is not acquired.

Optionally, on the basis of the above-mentioned embodiment corresponding to FIG. 14 , in another embodiment of the target object tracking device 30 provided in the embodiment of the present application,

The acquisition module 310 is specifically used to adjust the size of the original detected image to obtain the image to be processed;

If the size of the image to be processed fails to match the preset size, the image to be processed is filled to obtain a target detection image that meets the preset size;

Based on the target detection image, the target feature map is output through the feature extraction network.

Optionally, on the basis of the above-mentioned embodiment corresponding to FIG. 14 , in another embodiment of the target object tracking device 30 provided in the embodiment of the present application, each set of optional bounding box parameters further includes a confidence parameter, at least one The set of optional bounding box parameters includes multiple sets of optional bounding box parameters;

The determining module 320 is specifically configured to, for each feature map, select an optional bounding box parameter with a maximum confidence parameter from multiple sets of optional bounding box parameters, wherein each set of optional bounding box parameters is used to determine an optional bounding box parameter bounding box;

For each feature map, determine an initial optional bounding box according to the optional bounding box parameter with the maximum confidence parameter;

For each feature map, traverse the remaining optional bounding boxes. If the overlapping area between the optional bounding box and the initial optional bounding box is greater than or equal to the area threshold, delete the optional bounding box until the remaining optional bounding boxes are obtained. bounding boxes, where the remaining optional bounding boxes represent optional bounding boxes in addition to the initial optional bounding box;

For each feature map, the optional bounding box parameters corresponding to the remaining optional bounding boxes are used as bounding box parameters.

Optionally, on the basis of the above-mentioned embodiment corresponding to FIG. 14 , in another embodiment of the target object tracking device 30 provided in the embodiment of the present application,

Tracking module 330, specifically for determining T bounding boxes according to T groups of bounding box parameters;

Determine the leftmost vertex, the rightmost vertex, the uppermost vertex, and the lowermost vertex according to the T bounding boxes;

Determine the coordinates of the upper left vertex, the lower left vertex, the upper right vertex, and the lower right vertex according to the leftmost vertex, the rightmost vertex, the uppermost vertex, and the lowermost vertex;

According to the coordinates of the upper left vertex, the lower left vertex, the upper right vertex and the lower right vertex, the tracking result for the target object is determined.

Optionally, on the basis of the above-mentioned embodiment corresponding to FIG. 14 , in another embodiment of the target object tracking device 30 provided in the embodiment of the present application, the target object tracking device 30 further includes a training module 350;

The acquiring module 310 is further configured to acquire an image sample, where the image sample includes a target object;

The acquisition module 310 is also used to obtain a sample feature map through a feature extraction network based on an image sample, wherein the sample feature map includes K feature points, and the sample feature map is divided into N feature maps to be trained, and each feature map to be trained There is an anchor set in;

The obtaining module 310 is also used for obtaining an anchor point frame according to the anchor points set in the feature map to be trained for each feature map to be trained;

The acquisition module 310 is further configured to obtain at least one set of optional bounding box parameters through the object detection network based on the features corresponding to the anchor box and the real bounding box parameters for each feature map to be trained, wherein each set of optional bounding box parameters The frame parameters include coordinate parameters and size parameters of the bounding box to be trained;

The determination module 320 is further configured to determine the predicted bounding box parameters from at least one set of optional bounding box parameters for each feature map to be trained;

The training module 350 is configured to update the model parameters of the feature extraction network and the object detection network by using a loss function for each feature map to be trained according to the predicted bounding box parameters and the real bounding box parameters.

Optionally, on the basis of the above-mentioned embodiment corresponding to FIG. 14 , in another embodiment of the target object tracking device 30 provided in the embodiment of the present application, the target object tracking device 30 further includes a detection module 360;

The detection module 360 is used to detect the current image frame using a preset tracking algorithm to obtain the first center position parameter of the target object, wherein the preset tracking algorithm is a kernel correlation filtering algorithm, an online real-time tracking algorithm, and a background perception correlation filtering algorithm or multi-instance online learning algorithm;

The tracking module 330 is specifically configured to determine a second center position parameter of the target object according to T groups of bounding box parameters;

According to the first center position parameter and the second center position parameter, a tracking result for the target object in the current image frame is determined.

Optionally, on the basis of the above-mentioned embodiment corresponding to FIG. 14 , in another embodiment of the target object tracking device 30 provided in the embodiment of the present application, the target object is a human hand;

The target object tracking device 30 also includes a starting module 370, a generating module 380 and a triggering module 390;

The starting module 370 is used to start the camera device of the terminal device in response to the operation on the video shooting control;

The acquiring module 310 is further configured to acquire M consecutive image frames through the camera device, where M is an integer greater than or equal to 1;

The obtaining module 310 is also used to obtain the tracking results for the human hand in consecutive M image frames after determining the tracking results for the target object in the current image frame according to the T group of bounding box parameters;

The generation module 380 is used to generate a moving track for the human hand according to the tracking results corresponding to the M consecutive image frames and the tracking results corresponding to the current image frame;

The triggering module 390 is configured to trigger the function corresponding to the preset track if the moving track matches the preset track successfully.

Optionally, on the basis of the above-mentioned embodiment corresponding to FIG. 14 , in another embodiment of the target object tracking device 30 provided in the embodiment of the present application, the target object is a human hand;

The starting module 370 is also configured to start the camera device of the terminal device in response to the operation on the video shooting control;

The acquiring module 310 is further configured to acquire M consecutive image frames through the camera device, where M is an integer greater than or equal to 1;

The obtaining module 310 is also used to obtain the tracking results for the human hand in consecutive M image frames after determining the tracking results for the target object in the current image frame according to the T group of bounding box parameters;

The obtaining module 310 is also used to obtain the track length according to the tracking results corresponding to the consecutive M image frames and the tracking results corresponding to the current image frame;

The trigger module 390 is further configured to trigger the function corresponding to the preset track if the track length is greater than or equal to the length threshold.

The following is a detailed description of the target object tracking device in this application. Please refer to FIG. 15. FIG. 15 is a schematic diagram of another embodiment of the target object tracking device in the embodiment of the present application. The target object tracking device 40 includes:

The acquiring module 410 is configured to extract an original detection image from the current image frame according to the target bounding box corresponding to the previous image frame if the target object is detected in the previous image frame of the current image frame, wherein the original detection image including target audience;

The obtaining module 410 is also used to obtain the target feature map through the feature extraction network based on the original detection image;

The obtaining module 410 is also used to obtain the first offset and the second offset through the object detection network based on the target feature map, wherein the first offset is the upper left offset, and the second offset is the right The lower offset, or, the first offset is the upper right offset, and the second offset is the lower left offset;

The tracking module 420 is configured to determine a tracking result for the target object in the current image frame according to the first offset and the second offset.

Optionally, on the basis of the above-mentioned embodiment corresponding to FIG. 15 , in another embodiment of the target object tracking device 40 provided in the embodiment of the present application, the target object tracking device 40 further includes an identification module 430;

The recognition module 430 is used to perform object recognition on the current image frame if no target object is detected in the previous image frame of the current image frame;

The recognition module 430 is further configured to perform object recognition on the current image frame if the previous image frame of the current image frame is not acquired.

Optionally, on the basis of the above-mentioned embodiment corresponding to FIG. 15 , in another embodiment of the target object tracking device 40 provided in the embodiment of the present application,

The acquisition module 410 is specifically used to adjust the size of the original detected image to obtain the image to be processed;

If the size of the image to be processed fails to match the preset size, the image to be processed is filled to obtain a target detection image that meets the preset size;

Based on the target detection image, the target feature map is output through the feature extraction network.

Optionally, on the basis of the above-mentioned embodiment corresponding to FIG. 15 , in another embodiment of the target object tracking device 40 provided in the embodiment of the present application, the first offset is the upper left offset, and the second offset The offset is the lower right offset, where the upper left offset includes the horizontal offset and the vertical offset from the upper left vertex to the initial upper left vertex, and the lower right offset includes the lateral offset from the lower right vertex to the initial lower right vertex. displacement and longitudinal offset;

The target object tracking device 40 also includes a training module 440;

The acquiring module 410 is further configured to acquire an image sample, where the image sample includes a target object;

The obtaining module 410 is also used to obtain the sample feature map through the feature extraction network based on the image sample;

The obtaining module 410 is also used to obtain the upper left prediction offset and the lower right prediction offset through the object detection network based on the sample feature map;

The training module 440 is configured to use a loss function to perform a test on the feature extraction network and the object detection network based on the sample feature map, according to the upper-left predicted offset, the lower-right predicted offset, the upper-left real offset, and the lower-right real offset. The model parameters are updated.

Optionally, on the basis of the above-mentioned embodiment corresponding to FIG. 15 , in another embodiment of the target object tracking device 40 provided in the embodiment of the present application, the first offset is the upper right offset, and the second offset The offset is the lower left offset, where the upper right offset includes the horizontal offset and vertical offset from the upper right vertex to the initial upper right vertex, and the lower left offset includes the horizontal offset and vertical offset from the lower left vertex to the initial lower left vertex Offset;

The acquiring module 410 is further configured to acquire an image sample, where the image sample includes a target object;

The obtaining module 410 is also used to obtain the sample feature map through the feature extraction network based on the image sample;

The obtaining module 410 is also used to obtain the upper-right prediction offset and the lower-left prediction offset through the object detection network based on the sample feature map;

The training module 440 is also used to use the loss function to model the feature extraction network and the object detection network based on the sample feature map, according to the upper right predicted offset, the lower left predicted offset, the upper right actual offset, and the lower left actual offset. The parameters are updated.

Optionally, on the basis of the above-mentioned embodiment corresponding to FIG. 15 , in another embodiment of the target object tracking device 40 provided in the embodiment of the present application, the target object tracking device 40 further includes a detection module 450;

The detection module 450 is used to detect the current image frame using a preset tracking algorithm to obtain the first center position parameter of the target object, wherein the preset tracking algorithm is a kernel correlation filtering algorithm, an online real-time tracking algorithm, and a background perception correlation filtering algorithm or multi-instance online learning algorithm;

The tracking module 420 is specifically configured to determine a second center position parameter of the target object according to the first offset and the second offset;

According to the first center position parameter and the second center position parameter, a tracking result for the target object in the current image frame is determined.

Optionally, on the basis of the above-mentioned embodiment corresponding to FIG. 15 , in another embodiment of the target object tracking device 40 provided in the embodiment of the present application, the target object is a human hand;

The target object tracking device 40 also includes a starting module 460, a generating module 470 and a triggering module 480;

The starting module 460 is used to start the camera device of the terminal device in response to the operation on the video shooting control;

The acquisition module 410 is further configured to acquire M consecutive image frames through the camera device, where M is an integer greater than or equal to 1;

The acquisition module 410 is further configured to acquire the tracking results for the human hand in consecutive M image frames after determining the tracking results for the target object in the current image frame according to the first offset and the second offset;

The generating module 470 is used to generate a moving track for the human hand according to the tracking results corresponding to the M consecutive image frames and the tracking results corresponding to the current image frame;

The triggering module 480 is configured to trigger the function corresponding to the preset track if the moving track matches the preset track successfully.

Optionally, on the basis of the above-mentioned embodiment corresponding to FIG. 15 , in another embodiment of the target object tracking device 40 provided in the embodiment of the present application, the target object is a human hand;

The starting module 460 is also configured to start the camera device of the terminal device in response to the operation on the video shooting control;

The acquisition module 410 is further configured to acquire M consecutive image frames through the camera device, where M is an integer greater than or equal to 1;

The acquisition module 410 is further configured to acquire the tracking results for the human hand in consecutive M image frames after determining the tracking results for the target object in the current image frame according to the first offset and the second offset;

The obtaining module 410 is also used to obtain the track length according to the tracking results corresponding to the consecutive M image frames and the tracking results corresponding to the current image frame;

The trigger module 480 is further configured to trigger the function corresponding to the preset track if the track length is greater than or equal to the length threshold.

The embodiment of the present application also provides another target object tracking device, as shown in Figure 16, for the convenience of description, only the parts related to the embodiment of the present application are shown, and the specific technical details are not disclosed, please refer to the implementation of this application Example methods section. The terminal device may be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POS), a vehicle-mounted computer, etc., taking the terminal device as a mobile phone as an example:

FIG. 16 is a block diagram showing a partial structure of a mobile phone related to the terminal device provided by the embodiment of the present application. Referring to FIG. 16 , the mobile phone includes: a radio frequency (Radio Frequency, RF) circuit 710, a memory 720, an input unit 730, a display unit 740, a sensor 750, an audio circuit 760, a wireless fidelity (wireless fidelity, WiFi) module 770, and a processor 780 , and power supply 790 and other components. Those skilled in the art can understand that the structure of the mobile phone shown in FIG. 16 does not constitute a limitation to the mobile phone, and may include more or less components than shown in the figure, or combine certain components, or arrange different components.

The following is a specific introduction to each component of the mobile phone in conjunction with Figure 16:

The RF circuit 710 can be used for sending and receiving information or receiving and sending signals during a call. In particular, after receiving the downlink information of the base station, it is processed by the processor 780; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 710 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, and the like. In addition, RF circuitry 710 may also communicate with networks and other devices via wireless communications. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access) , CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (Short Messaging Service, SMS), etc.

The memory 720 can be used to store software programs and modules, and the processor 780 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 720 . The memory 720 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playback function, an image playback function, etc.); Data created by the use of mobile phones (such as audio data, phonebook, etc.), etc. In addition, the memory 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage devices.

The input unit 730 can be used to receive input numbers or character information, and generate key signal input related to user settings and function control of the mobile phone. Specifically, the input unit 730 may include a touch panel 731 and other input devices 732 . The touch panel 731, also referred to as a touch screen, can collect touch operations of the user on or near it (for example, the user uses any suitable object or accessory such as a finger or a stylus on the touch panel 731 or near the touch panel 731). operation), and drive the corresponding connection device according to the preset program. Optionally, the touch panel 731 may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, and detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and sends it to the to the processor 780, and can receive and execute commands sent by the processor 780. In addition, the touch panel 731 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 731 , the input unit 730 may also include other input devices 732 . Specifically, other input devices 732 may include but not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), trackball, mouse, joystick, and the like.

The display unit 740 may be used to display information input by or provided to the user and various menus of the mobile phone. The display unit 740 may include a display panel 741. Optionally, the display panel 741 may be configured in the form of a liquid crystal display (Liquid Crystal Display, LCD), an organic light-emitting diode (Organic Light-Emitting Diode, OLED), or the like. Further, the touch panel 731 may cover the display panel 741, and when the touch panel 731 detects a touch operation on or near it, it transmits to the processor 780 to determine the type of the touch event, and then the processor 780 determines the type of the touch event according to the The type provides a corresponding visual output on the display panel 741 . Although in FIG. 16 , the touch panel 731 and the display panel 741 are used as two independent components to realize the input and input functions of the mobile phone, in some embodiments, the touch panel 731 and the display panel 741 can be integrated and Realize the input and output functions of the mobile phone.

The handset may also include at least one sensor 750, such as a light sensor, motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 741 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 741 and/or when the mobile phone is moved to the ear. or backlight. As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when it is stationary, and can be used for applications that recognize the posture of mobile phones (such as horizontal and vertical screen switching, related Games, magnetometer attitude calibration), vibration recognition related functions (such as pedometer, tap), etc.; as for other sensors such as gyroscope, barometer, hygrometer, thermometer, infrared sensor, etc. repeat.

The audio circuit 760, the speaker 761, and the microphone 762 can provide an audio interface between the user and the mobile phone. The audio circuit 760 can transmit the electrical signal converted from the received audio data to the speaker 761, and the speaker 761 converts it into an audio signal for output; After being received, it is converted into audio data, and then the audio data is processed by the output processor 780, and then sent to another mobile phone through the RF circuit 710, or the audio data is output to the memory 720 for further processing.

WiFi is a short-distance wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages, and access streaming media through the WiFi module 770, which provides users with wireless broadband Internet access. Although FIG. 16 shows a WiFi module 770, it can be understood that it is not an essential component of the mobile phone, and can be completely omitted as required without changing the essence of the invention.

The processor 780 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and/or modules stored in the memory 720, and calling data stored in the memory 720, execution Various functions and processing data of the mobile phone, so as to monitor the mobile phone as a whole. Optionally, the processor 780 may include one or more processing units; optionally, the processor 780 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs etc., the modem processor mainly handles wireless communication. It can be understood that the foregoing modem processor may not be integrated into the processor 780 .

The mobile phone also includes a power supply 790 (such as a battery) for supplying power to various components. Optionally, the power supply can be logically connected to the processor 780 through the power management system, so that functions such as charging, discharging, and power consumption management can be realized through the power management system.

Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be repeated here.

The steps performed by the terminal device in the foregoing embodiments may be based on the structure of the terminal device shown in FIG. 16 .

Embodiments of the present application also provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is run on a computer, the computer executes the methods described in the foregoing embodiments.

Embodiments of the present application also provide a computer program product including a program, which when run on a computer, causes the computer to execute the methods described in the foregoing embodiments.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disk and other various media that can store program codes. .

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still understand the foregoing The technical solutions described in each embodiment are modified, or some of the technical features are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the application.

Claims (20)

1. A tracking method for a target object, comprising: If the target object is detected in the previous image frame of the current image frame, an original detection image is extracted from the current image frame according to the target bounding box corresponding to the previous image frame, wherein the original detection image include the target audience; Based on the original detection image, the target feature map is obtained through a feature extraction network, wherein the target feature map includes K feature points, and the target feature map is divided into N feature maps, and each feature map is set with An anchor point, the K is an integer greater than 1, and the N is an integer greater than or equal to 1 and less than the K; For each of the feature maps, an anchor point frame is obtained according to the anchor points set in the feature map; For each of the feature maps, based on the features corresponding to the anchor box, at least one set of optional bounding box parameters is obtained through the object detection network, wherein each set of optional bounding box parameters includes coordinate parameters and dimensions of the optional bounding box parameter; Determine T groups of bounding box parameters according to at least one group of optional bounding box parameters of each feature map, wherein each group of bounding box parameters includes coordinate parameters and size parameters of the bounding box, and the T is an integer greater than or equal to 1 ; Determine a tracking result for the target object in the current image frame according to the T groups of bounding box parameters. 2. The tracking method according to claim 1, wherein the method further comprises: If the target object is not detected in the previous image frame of the current image frame, performing object recognition on the current image frame; If the previous image frame of the current image frame is not acquired, object recognition is performed on the current image frame. 3. tracking method according to claim 1, is characterized in that, described based on described original detection image, obtains target feature map by feature extraction network, comprises: Adjusting the size of the original detected image to obtain the image to be processed; If the size of the image to be processed fails to match the preset size, filling the image to be processed to obtain a target detection image satisfying the preset size; Based on the target detection image, output the target feature map through the feature extraction network. 4. The tracking method according to claim 1, wherein each set of optional bounding box parameters also includes a confidence parameter, and the at least one set of optional bounding box parameters includes multiple sets of optional bounding box parameters; The determining T groups of bounding box parameters according to at least one set of optional bounding box parameters of each feature map includes: For each feature map, select an optional bounding box parameter with a maximum confidence parameter from the multiple sets of optional bounding box parameters, wherein each set of optional bounding box parameters is used to determine an optional bounding box parameter frame; For each feature map, determine an initial optional bounding box according to the optional bounding box parameter with the maximum confidence parameter; For each feature map, traverse the remaining optional bounding boxes, and if the overlapping area between the optional bounding box and the initial optional bounding box is greater than or equal to an area threshold, delete the optional bounding box, until the remaining optional bounding boxes are obtained, wherein the remaining optional bounding boxes represent optional bounding boxes except for the initial optional bounding box; For each feature map, the optional bounding box parameters corresponding to the remaining optional bounding boxes are used as bounding box parameters. 5. The tracking method according to claim 1, wherein the determining the tracking result for the target object in the current image frame according to the T group of bounding box parameters comprises: Determining T bounding boxes according to the T groups of bounding box parameters; determining a leftmost vertex, a rightmost vertex, an uppermost vertex, and a lowermost vertex according to the T bounding boxes; determining coordinates of an upper left vertex, a lower left vertex, an upper right vertex, and a lower right vertex according to the leftmost vertex, the rightmost vertex, the uppermost vertex, and the lowermost vertex; A tracking result for the target object is determined according to the coordinates of the upper left vertex, the coordinates of the lower left vertex, the coordinates of the upper right vertex, and the coordinates of the lower right vertex. 6. The tracking method according to claim 1, wherein the method further comprises: acquiring an image sample, wherein the image sample includes the target object; Based on the image sample, a sample feature map is obtained through the feature extraction network, wherein the sample feature map includes K feature points, and the sample feature map is divided into N feature maps to be trained, each feature to be trained An anchor point is set in the figure; For each feature map to be trained, an anchor point frame is obtained according to the anchor points set in the feature map to be trained; For each feature map to be trained, based on the features corresponding to the anchor box and the real bounding box parameters, at least one set of optional bounding box parameters is obtained through the object detection network, wherein each set of optional bounding box parameters includes The coordinate parameters and size parameters of the bounding box to be trained; For each of the feature maps to be trained, determining predicted bounding box parameters from at least one set of optional bounding box parameters; For each feature map to be trained, a loss function is used to update the model parameters of the feature extraction network and the object detection network according to the predicted bounding box parameters and the real bounding box parameters. 7. The tracking method according to claim 1, wherein the method further comprises: The current image frame is detected by using a preset tracking algorithm to obtain the first center position parameter of the target object, wherein the preset tracking algorithm is a kernel correlation filtering algorithm, an online real-time tracking algorithm, and a context-aware correlation filtering algorithm or multi-instance online learning algorithm; The determining the tracking result for the target object in the current image frame according to the T group of bounding box parameters includes: Determine a second center position parameter of the target object according to the T group of bounding box parameters; A tracking result for the target object in the current image frame is determined according to the first center position parameter and the second center position parameter. 8. The tracking method according to any one of claims 1 to 7, wherein the target object is a human hand; The method also includes: In response to the operation of the video capture control, activate the camera device of the terminal device; Collecting M consecutive image frames through the camera device, where M is an integer greater than or equal to 1; After determining the tracking result for the target object in the current image frame according to the T group of bounding box parameters, the method further includes: Acquiring the tracking results for the human hand in the M consecutive image frames; Generating a movement track for the human hand according to the tracking results corresponding to the M consecutive image frames and the tracking results corresponding to the current image frame; If the moving track matches the preset track successfully, a function corresponding to the preset track is triggered. 9. The tracking method according to any one of claims 1 to 7, wherein the target object is a human hand; The method also includes: In response to the operation of the video capture control, activate the camera device of the terminal device; Collecting M consecutive image frames through the camera device, where M is an integer greater than or equal to 1; After determining the tracking result for the target object in the current image frame according to the T group of bounding box parameters, the method further includes: Acquiring the tracking results for the human hand in the M consecutive image frames; Acquiring the track length according to the tracking results corresponding to the M consecutive image frames and the tracking results corresponding to the current image frame; If the track length is greater than or equal to the length threshold, the function corresponding to the preset track is triggered. 10. A method for tracking a target object, comprising: If the target object is detected in the previous image frame of the current image frame, an original detection image is extracted from the current image frame according to the target bounding box corresponding to the previous image frame, wherein the original detection image include the target audience; Based on the original detection image, the target feature map is obtained through a feature extraction network; Based on the target feature map, a first offset and a second offset are obtained through the object detection network, wherein the first offset is an upper left offset, and the second offset is a lower-right offset, or, the first offset is an upper-right offset, and the second offset is a lower-left offset; A tracking result for the target object in the current image frame is determined according to the first offset and the second offset. 11. The tracking method according to claim 10, wherein the first offset is an upper-left offset, and the second offset is a lower-right offset, wherein the upper-left offset The amount of displacement includes a horizontal offset and a vertical offset from the upper left vertex to the initial upper left vertex, and the lower right offset includes a horizontal offset and a vertical offset from the lower right vertex to the initial lower right vertex; The method also includes: acquiring an image sample, wherein the image sample includes the target object; Obtaining a sample feature map through the feature extraction network based on the image sample; Based on the sample feature map, the upper left prediction offset and the lower right prediction offset are obtained through the object detection network; Based on the sample feature map, according to the upper-left predicted offset, the lower-right predicted offset, the upper-left real offset and the lower-right real offset, a loss function is used for the feature extraction network and the The model parameters of the object detection network are updated. 12. The tracking method according to claim 10, wherein the first offset is an upper right offset, and the second offset is a lower left offset, wherein the upper right offset The amount includes the horizontal offset and vertical offset from the upper right vertex to the initial upper right vertex, and the lower left offset includes the horizontal offset and vertical offset from the lower left vertex to the initial lower left vertex: The method also includes: acquiring an image sample, wherein the image sample includes the target object; Obtaining a sample feature map through the feature extraction network based on the image sample; Based on the sample feature map, obtaining an upper-right prediction offset and a lower-left prediction offset through the object detection network; Based on the sample feature map, according to the upper-right predicted offset, the lower-left predicted offset, the upper-right true offset, and the lower-left true offset, a loss function is used for the feature extraction network and the object detection The model parameters of the network are updated. 13. The tracking method according to claim 10, further comprising: The current image frame is detected by using a preset tracking algorithm to obtain the first center position parameter of the target object, wherein the preset tracking algorithm is a kernel correlation filtering algorithm, an online real-time tracking algorithm, and a context-aware correlation filtering algorithm or multi-instance online learning algorithm; The determining the tracking result for the target object in the current image frame according to the first offset and the second offset includes: determining a second center position parameter of the target object according to the first offset and the second offset; A tracking result for the target object in the current image frame is determined according to the first center position parameter and the second center position parameter. 14. The tracking method according to any one of claims 10 to 13, wherein the target object is a human hand; The method also includes: In response to the operation of the video capture control, activate the camera device of the terminal device; Collecting M consecutive image frames through the camera device, where M is an integer greater than or equal to 1; After determining the tracking result for the target object in the current image frame according to the first offset and the second offset, the method further includes: Acquiring the tracking results for the human hand in the M consecutive image frames; Generating a movement track for the human hand according to the tracking results corresponding to the M consecutive image frames and the tracking results corresponding to the current image frame; If the moving track matches the preset track successfully, a function corresponding to the preset track is triggered. 15. The tracking method according to any one of claims 10 to 13, wherein the target object is a human hand; The method also includes: In response to the operation of the video capture control, activate the camera device of the terminal device; Collecting M consecutive image frames through the camera device, where M is an integer greater than or equal to 1; After determining the tracking result for the target object in the current image frame according to the first offset and the second offset, the method further includes: Acquiring the tracking results for the human hand in the M consecutive image frames; Acquiring the track length according to the tracking results corresponding to the M consecutive image frames and the tracking results corresponding to the current image frame; If the track length is greater than or equal to the length threshold, the function corresponding to the preset track is triggered. 16. A target object tracking device, comprising: An acquisition module, configured to extract the original detected image from the current image frame according to the target bounding box corresponding to the previous image frame if the target object is detected in the previous image frame of the current image frame, wherein, The original detection image includes the target object; The acquisition module is further configured to acquire a target feature map through a feature extraction network based on the original detection image, wherein the target feature map includes K feature points, and the target feature map is divided into N feature maps , an anchor point is set in each feature map, the K is an integer greater than 1, and the N is an integer greater than or equal to 1 and less than the K; The acquiring module is further configured to acquire an anchor point frame according to the anchor points set in the feature map for each of the feature maps; The obtaining module is further configured to obtain at least one set of optional bounding box parameters through the object detection network based on the features corresponding to the anchor point boxes for each of the feature maps, wherein each set of optional bounding box parameters includes optional Select the coordinate parameters and size parameters of the bounding box; A determining module, configured to determine T groups of bounding box parameters according to at least one group of optional bounding box parameters of each feature map, wherein each group of bounding box parameters includes coordinate parameters and size parameters of the bounding box, and the T is greater than or an integer equal to 1; A tracking module, configured to determine a tracking result for the target object in the current image frame according to the T groups of bounding box parameters. 17. A target object tracking device, comprising: An acquisition module, configured to extract the original detected image from the current image frame according to the target bounding box corresponding to the previous image frame if the target object is detected in the previous image frame of the current image frame, wherein, The original detection image includes the target object; The acquiring module is further configured to acquire the target feature map through a feature extraction network based on the original detection image; The obtaining module is further configured to obtain a first offset and a second offset through the object detection network based on the target feature map, wherein the first offset is an upper-left offset, and The second offset is a lower-right offset, or, the first offset is an upper-right offset, and the second offset is a lower-left offset; A tracking module, configured to determine a tracking result for the target object in the current image frame according to the first offset and the second offset. 18. A terminal device, comprising: a memory, a processor, and a bus system; Wherein, the memory is used to store programs; The processor is used to execute the program in the memory, and the processor is used to execute the tracking method according to any one of claims 1 to 9 according to the instructions in the program code, or to execute any one of claims 10 to 15. one of the tracking methods; The bus system is used to connect the memory and the processor so that the memory and the processor can communicate. 19. A computer-readable storage medium, comprising instructions, which, when run on a computer, cause the computer to execute the tracking method according to any one of claims 1 to 9, or to execute the tracking method described in claims 10 to 15. tracking method. 20. A computer program product, including computer programs and instructions, characterized in that, when the computer program/instructions are executed by a processor, the tracking method according to any one of claims 1 to 9 is realized, or, the tracking method according to any one of claims 1 to 9 is realized, or The tracking methods described in 10 to 15 are required.

Images