KR102796145B1 - Real-time object tracking apparatus and method that can improve processing speed while ensuring the accuracy of object tracking - Google Patents

Abstract

The real-time object tracking device and method according to the present invention performs object tracking on an input image by combining the DeepSORT (Deep Simple Online and Real-time Tracking) algorithm and the SORT (Simple Online and Real-time Tracking) algorithm, thereby ensuring the accuracy of object tracking and improving the processing speed at the same time.

Description

{REAL-TIME OBJECT TRACKING APPARATUS AND METHOD THAT CAN IMPROVE PROCESSING SPEED WHILE ENSURING THE ACCURACY OF OBJECT TRACKING}

The present invention relates to a real-time object tracking device and method capable of improving processing speed while ensuring accuracy of object tracking.

Multi-object tracking technology is a technology that continuously tracks the locations of objects such as people and cars in video footage, and is being used in various fields such as autonomous driving systems and unmanned surveillance systems (CCTV, Closed-Circuit Television).

Multi-object tracking technology can be divided into an object detection step, which first identifies the location of each object in each frame (single image) of the video, and an object tracking step, which identifies the relationship between objects existing in two consecutive frames and tracks the change in location of each object.

Modern object detection technology has made great strides with the development of artificial intelligence. Methods such as R-CNN (Regions with Convolution Neural Network features), Fast R-CNN, and YOLO (You Only Look Once) have been proposed as methods for detecting multiple objects in a single image using deep learning. Each technology has advantages and disadvantages in terms of detection performance and speed. For example, YOLO is a technique that has strengths in detection speed rather than accuracy.

Object tracking technology is also developing at a very fast pace, and representative techniques such as FairMOT (Fair Multi-Object Tracking), SORT (Simple Online and Real-time Tracking), and DeepSORT (Deep Simple Online and Real-time Tracking) have been proposed. For example, SORT has the advantage of being fast because it does not require a large amount of computing operations. This object tracking is performed by detecting objects in each frame using techniques such as YOLO described above, and then using the results to check how similar the objects are in the locations of consecutive frames.

In general, in systems that require real-time object tracking, the accuracy of object tracking must be high, and the computational speed according to object tracking must also be fast. In this regard, when performing object tracking using the DeepSORT algorithm, relatively high accuracy object tracking is possible, but the computational amount increases, so there is a disadvantage that the object tracking speed is slowed down. On the other hand, when performing object tracking using the SORT algorithm, relatively fast object tracking is possible because the computational amount is small compared to the DeepSORT algorithm, but there is a disadvantage that the accuracy of object tracking is low.

Therefore, in real-time object tracking systems, research is needed on a technology that can increase the accuracy of object tracking while improving the speed by appropriately combining the DeepSORT algorithm and the SORT algorithm.

The real-time object tracking device and method according to the present invention perform object tracking on an input image by combining the DeepSORT (Deep Simple Online and Real-time Tracking) algorithm and the SORT (Simple Online and Real-time Tracking) algorithm, thereby ensuring the accuracy of object tracking and supporting an improvement in processing speed.

According to one embodiment of the present invention, a real-time object tracking device includes: a first tracking unit which, when an image is input, performs object tracking based on a DeepSORT (Deep Simple Online and Real-time Tracking) algorithm from a first frame among frames constituting the image to an n-th frame (where n is a value designated in advance among natural numbers having a size of 3 or greater); a second tracking unit which performs object tracking based on a SORT (Simple Online and Real-time Tracking) algorithm on the n-th frame and an n+1-th frame and then redesignates the n+1-th frame as the first frame and applies it to the first tracking unit; and a repetition control unit which controls object tracking according to the first tracking unit and the second tracking unit to be repeatedly performed.

In addition, a real-time object tracking method according to an embodiment of the present invention includes a first tracking step of performing object tracking based on a DeepSORT algorithm from a first frame among frames constituting the image to an n-th frame (where n is a value designated in advance among natural numbers having a size of 3 or greater) when an image is input, a second tracking step of performing object tracking based on the SORT algorithm on the n-th frame and the n+1-th frame and then returning to the first tracking step by redesignating the n+1-th frame as the first frame, and a repetition execution control step of controlling object tracking according to the first tracking step and the second tracking step to be repeatedly performed.

The real-time object tracking device and method according to the present invention performs object tracking on an input image by combining the DeepSORT (Deep Simple Online and Real-time Tracking) algorithm and the SORT (Simple Online and Real-time Tracking) algorithm, thereby ensuring the accuracy of object tracking and improving the processing speed at the same time.

FIG. 1 is a diagram illustrating the structure of a real-time object tracking device according to an embodiment of the present invention.
FIG. 2 is a flowchart illustrating a real-time object tracking method according to an embodiment of the present invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the attached drawings. This description is not intended to limit the present invention to specific embodiments, but should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention. In describing each drawing, similar reference numerals are used for similar components, and unless otherwise defined, all terms used in this specification, including technical or scientific terms, have the same meaning as generally understood by a person of ordinary skill in the art to which the present invention belongs.

In this document, when a part is said to "include" a certain component, this means that other components may be further included, rather than excluding other components, unless otherwise specifically stated. In addition, in various embodiments of the present invention, each of the components, functional blocks or means may be composed of one or more subcomponents, and the electrical, electronic and mechanical functions performed by each component may be implemented by various known elements such as electronic circuits, integrated circuits, ASICs (Application Specific Integrated Circuits), or mechanical elements, and each may be implemented separately or two or more may be integrated into one.

Meanwhile, the blocks of the attached block diagram or the steps of the flow chart may be interpreted as computer program instructions that are loaded into a processor or memory of a device capable of data processing, such as a general-purpose computer, a special-purpose computer, a portable notebook computer, or a network computer, and perform designated functions. Since these computer program instructions may be stored in a memory equipped in a computer device or a computer-readable memory, the functions described in the blocks of the block diagram or the steps of the flow chart may be produced as a manufactured product that includes a command means for performing the same. In addition, each block or each step may represent a module, segment, or part of code that includes one or more executable instructions for performing a specific logical function(s). In addition, it should be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may be performed in a different order from the specified order. For example, two blocks or steps illustrated in succession may be performed substantially simultaneously or in reverse order, and in some cases, some blocks or steps may be performed with some blocks or steps omitted.

FIG. 1 is a diagram illustrating the structure of a real-time object tracking device according to an embodiment of the present invention.

Referring to FIG. 1, a real-time object tracking device (110) according to the present invention includes a first tracking unit (111), a second tracking unit (112), and a repetition execution control unit (113).

When an image is input, the first tracking unit (111) performs object tracking based on the DeepSORT (Deep Simple Online and Real-time Tracking) algorithm, starting from the first frame among the frames constituting the image, to the nth frame (n is a value designated in advance among natural numbers having a size of 3 or greater).

The second tracking unit (112) performs object tracking based on the SORT (Simple Online and Real-time Tracking) algorithm for the nth frame and n+1th frame, and then redesignates the n+1th frame as the first frame and applies it to the first tracking unit (111).

The repetition execution control unit (113) controls object tracking to be performed repeatedly according to the first tracking unit (111) and the second tracking unit (112).

In relation to this, assuming n to be '5', the operation of the real-time object tracking device (110) according to the present invention is described as follows.

First, when an image is input, the first tracking unit (111) can perform object tracking based on the DeepSORT algorithm from the first frame to the fifth frame among the frames constituting the image.

Then, the second tracking unit (112) can perform object tracking based on the SORT algorithm for the fifth and sixth frames, and then redesignate the sixth frame as the first frame and apply it to the first tracking unit (111).

At this time, since the repetition execution control unit (113) controls the object tracking according to the first tracking unit (111) and the second tracking unit (112) to be performed repeatedly, when the 6th frame is redesignated as the first frame by the second tracking unit (112) and applied to the first tracking unit (111), the first tracking unit (111) can perform object tracking based on the DeepSORT algorithm again starting from the 6th frame up to the 10th frame, and the second tracking unit (112) can perform object tracking based on the SORT algorithm on the 10th frame and the 11th frame, and then redesignate the 11th frame as the first frame again and apply it to the first tracking unit (111).

In this way, the first tracking unit (111) can perform object tracking based on the DeepSORT algorithm for every five consecutive frames, and the second tracking unit (112) can perform object tracking based on the SORT algorithm for every 5k-th and 5k+1-th frame, such as the 5th and 6th frames, the 10th and 11th frames, and the 15th and 16th frames.

In this way, the real-time object tracking device (110) according to the present invention performs object tracking based on the DeepSORT algorithm, and performs object tracking based on the SORT algorithm between specific frames, thereby maintaining high accuracy in object tracking and improving the speed of object tracking. At this time, the manager of the real-time object tracking device (110) can adjust the size of n to appropriately adjust the number of frames in which object tracking is performed based on the DeepSORT algorithm in the first tracking unit (111) and the number of frames in which object tracking is performed based on the SORT algorithm in the second tracking unit (112), thereby adjusting the accuracy and speed of object tracking to a desired level. For example, the manager can achieve high accuracy in object tracking by increasing the size of n so that more frames are subject to object tracking based on the DeepSORT algorithm, and can improve the speed of object tracking by decreasing the size of n so that more frames are subject to object tracking based on the SORT algorithm.

According to one embodiment of the present invention, in order to perform object tracking based on the DeepSORT algorithm, if the frame of the current time is the first frame, the first tracking unit (111) detects first objects existing on the first frame through a preset object detector, calculates a first feature vector for location information of the first objects, applies a feature vector for location information of second objects existing on a frame preceding the first frame to a Kalman filter, thereby estimating a second feature vector for location information at which the second objects are predicted to exist on the first frame, and then calculates the Mahalanobis distance and the cosine distance between the first feature vector for each of the first objects and the second feature vector for each of the second objects, and calculates the Mahalanobis distance and the cosine distance between the first feature vector and the second feature vector among the first objects and the second objects, such that the weighted average of the Mahalanobis distance and the cosine distance between the first feature vector and the second feature vector is greater than a preset first reference value. The two resulting objects can be tracked as the same object.

That is, when the current frame corresponding to the current point in time in the image is called the first frame, the first tracking unit (111) can detect first objects existing on the first frame through a preset object detector in order to perform object tracking based on the DeepSORT algorithm between the first frame and the previous frame.

Here, the object detector may be a detector configured based on a deep learning-based object detection model for detecting objects from images, such as YOLO (You Only Look Once).

In this way, when the first objects existing on the first frame are detected, the first tracking unit (111) calculates a first feature vector for location information of the first objects, and then applies a feature vector for location information of the second objects existing on a frame previous to the first frame to a Kalman filter, thereby estimating a second feature vector for location information where the second objects are predicted to exist on the first frame.

Here, the Kalman filter refers to a recursive filter that tracks the state of a linear dynamic system including noise, and this Kalman filter can be used to predict the location of objects that existed in the previous frame in object tracking in the current frame.

In this way, when the second feature vectors for the second objects are estimated, the first tracking unit (111) can calculate the Mahalanobis distance and the cosine distance between the first feature vector for each of the first objects and the second feature vector for each of the second objects.

Then, the first tracking unit (111) can track two objects among the first objects and the second objects, in which the weighted average of the Mahalanobis distance and the cosine distance between the first feature vector and the second feature vector is calculated to be greater than a preset first reference value, as the same object.

For example, if the weighted average of the Mahalanobis distance and the cosine distance between the first feature vector for 'object 1', which is one of the first objects, and the second feature vector for 'object 2', which is one of the second objects, is calculated to be greater than the first reference value, the first tracking unit (111) can track 'object 1' existing in the first frame, which is the frame of the current point in time, as the same object as 'object 2' existing in the previous frame.

According to one embodiment of the present invention, in order to perform object tracking based on the SORT algorithm, the second tracking unit (112) detects third objects existing on the second frame through the object detector when the frame of the current time is the second frame, calculates a third feature vector for location information of the third objects, and applies a feature vector for location information of fourth objects existing on a frame preceding the second frame to a Kalman filter, thereby estimating a fourth feature vector for location information predicted to exist on the second frame of the fourth objects, and then calculates the Mahalanobis distance between the third feature vector for each of the third objects and the fourth feature vector for each of the fourth objects, so that two objects among the third objects and the fourth objects, for which the Mahalanobis distance between the third feature vector and the fourth feature vector is calculated to be greater than a preset second reference value, can be tracked as the same object.

That is, when the current frame corresponding to the current point in time in the image is the second frame, the second tracking unit (112) can detect third objects existing on the second frame through the object detector in order to perform object tracking based on the SORT algorithm between the second frame and the previous frame.

In this way, when the third objects existing on the second frame are detected, the second tracking unit (112) calculates a third feature vector for location information of the third objects, and then applies a feature vector for location information of the fourth objects existing on a frame preceding the second frame to a Kalman filter, thereby estimating a fourth feature vector for location information where the fourth objects are predicted to exist on the second frame.

In this way, when the fourth feature vectors for the fourth objects are estimated, the second tracking unit (112) can calculate the Mahalanobis distance between the third feature vector for each of the third objects and the fourth feature vector for each of the fourth objects.

Then, the second tracking unit (112) can track two objects among the third objects and the fourth objects, in which the Mahalanobis distance between the third feature vector and the fourth feature vector is calculated to be greater than a preset second reference value, as the same object.

For example, if the Mahalanobis distance between the third feature vector for 'object 3', which is one of the third objects, and the fourth feature vector for 'object 4', which is one of the fourth objects, is calculated to be greater than or equal to the second reference value, the second tracking unit (112) can track 'object 3' existing in the second frame, which is the frame of the current point in time, as the same object as 'object 4' existing in the previous frame.

According to one embodiment of the present invention, the real-time object tracking device (110) may further include a performance information display unit (114).

The performance information display unit (114) calculates the number of frames in which object tracking is completed per preset unit time as the object tracking processing speed for the image, calculates the MOTA (Multiple Object Tracking Accuracy) index and the MOTP (Multiple Object Tracking Precision) index as information on the accuracy of object tracking for the image, and then displays the object tracking processing speed, the MOTA index, and the MOTP index on the screen.

In relation to this, when the preset unit time is '1 second', the performance information display unit (114) can calculate the number of frames per second as the object tracking processing speed by measuring the number of frames in which object tracking is completed per second.

And, the MOTA index is an index that can be calculated according to the following mathematical formula 1: represents the number of actual objects present in the frame corresponding to the current time t, represents the number of objects that were not detected (i.e., objects that actually exist but were not detected) by the tracking technique applied to the frame of the current time t. represents the number of false negatives (i.e. detecting non-existent objects), represents the number of mismatches (i.e. objects intersecting each other but not recognizing this).

And, the MOTP index is an index that can be calculated according to the following mathematical formula 2. represents the number of matched objects calculated through the current frame and the previous frame based on the current time t, represents the distance between two objects for the i-th matched object.

In this way, the performance information display unit (114) displays the object tracking processing speed, the MOTA index, and the MOTP index on the screen as information related to the performance of object tracking performed through the real-time object tracking device (110), thereby enabling the manager to appropriately adjust the number of frames to utilize DeepSORT and SORT by referring to this performance information.

According to one embodiment of the present invention, the real-time object tracking device (110) may further include an event generating unit (115), a calculating unit (116), a comparing unit (117), a size adjusting unit (118), and a message display unit (119).

The event generating unit (115) generates an analysis event for analyzing the performance of object tracking at preset intervals. For example, if the interval is '10 seconds', the event generating unit (115) can generate an analysis event for analyzing the performance of object tracking at intervals of '10 seconds'.

When the above analysis event occurs, the calculation unit (116) calculates the average object tracking processing speed, average MOTA index, and average MOTP index during the period from the time of occurrence of the above analysis event.

In relation to this, if the analysis event is said to have occurred at a specific point in time, the calculation unit (116) can calculate the average object tracking processing speed, the average MOTA index, and the average MOTP index for the period of '10 seconds' from the time of occurrence of the analysis event. For example, if the object tracking processing speed is the number of frames per second, the calculation unit (116) can calculate the average of the object tracking processing speed for '10 seconds' as the average object tracking processing speed, and can calculate the average of the MOTA index and the MOTP index calculated for each frame for '10 seconds' as the average MOTA index and the average MOTP index.

The comparison unit (117) calculates the Euclidean norm of a vector having the average MOTA index and the average MOTP index as components, and then compares the Euclidean norm with a preset threshold norm, and compares the average object tracking processing speed with a preset threshold processing speed.

For example, if the average MOTA index is referred to as 'Mp' and the average MOTP index is referred to as 'Mn', the comparison unit (117) can calculate the Euclidean norm of '[Mp Mn]', which is a vector having 'Mp' and 'Mn' as components, and then compare the Euclidean norm with a preset threshold norm.

The size adjustment unit (118) determines, as a result of the comparison by the comparison unit (117), that the Euclidean norm exceeds the threshold norm and that the average object tracking processing speed does not exceed the threshold processing speed, that the accuracy of object tracking through the real-time object tracking device (110) is high but the speed is very slow. Therefore, the size of n, which is the number of frames for which object tracking must be performed through the DeepSORT algorithm in the first tracking unit (111), can be reduced by a preset size, and object tracking for the image can be performed according to the reduced size of n.

On the other hand, if the size adjustment unit (118) determines, as a result of the comparison by the comparison unit (117), that the Euclidean norm does not exceed the threshold norm and that the average object tracking processing speed exceeds the threshold processing speed, it can be seen that the speed of object tracking through the real-time object tracking device (110) is fast but the accuracy is very low, and therefore, the size of n, which is the number of frames on which object tracking must be performed through the DeepSORT algorithm in the first tracking unit (111), can be increased by the preset size, and object tracking for the image can be performed according to the increased size of n.

Finally, if the size adjustment unit (118) determines that the Euclidean norm exceeds the threshold norm and the average object tracking processing speed exceeds the threshold processing speed as a result of the comparison by the comparison unit (117), it can be seen that both the accuracy and speed of object tracking through the real-time object tracking device (110) are at a good level, and therefore, object tracking can be performed in the current state without performing any adjustment for the size of n.

The message display unit (119) can generate and display on the screen a message indicating that there is an error in the settings of the real-time object tracking device (110) without performing any adjustment to the size of n, since it can be seen that both the accuracy and the speed of object tracking through the real-time object tracking device (110) fall below the reference level if it is confirmed as a result of the comparison by the comparison unit (117) that the Euclidean norm does not exceed the threshold norm and the average object tracking processing speed does not exceed the threshold processing speed. Through this, the administrator can take measures to improve the accuracy and speed of object tracking by adjusting the settings of the real-time object tracking device (110).

FIG. 2 is a flowchart illustrating a real-time object tracking method according to an embodiment of the present invention.

In the first tracking step (S210), when an image is input, object tracking is performed based on the DeepSORT algorithm from the first frame among the frames constituting the image to the nth frame (where n is a pre-designated value among natural numbers having a size of 3 or greater).

The second tracking step (S220) performs object tracking based on the SORT algorithm for the nth frame and the n+1th frame, and then redesignates the n+1th frame as the first frame to return to the first tracking step.

The repeat execution control step controls the object tracking according to the first tracking step and the second tracking step to be repeatedly performed.

At this time, according to one embodiment of the present invention, in order to perform object tracking based on the DeepSORT algorithm, the first tracking step detects first objects existing on the first frame through a preset object detector when the frame of the current time is the first frame, calculates a first feature vector for location information of the first objects, and applies a feature vector for location information of second objects existing on a frame preceding the first frame to a Kalman filter, thereby estimating a second feature vector for location information at which the second objects are predicted to exist on the first frame, and then calculates the Mahalanobis distance and the cosine distance between the first feature vector for each of the first objects and the second feature vector for each of the second objects, such that two objects, among the first objects and the second objects, in which the weighted average of the Mahalanobis distance and the cosine distance between the first feature vector and the second feature vector is calculated to be greater than or equal to a preset first reference value, can be tracked as the same object.

In addition, according to one embodiment of the present invention, in order to perform object tracking based on the SORT algorithm, when the frame of the current time is the second frame, the second tracking step detects third objects existing on the second frame through the object detector, calculates a third feature vector for location information of the third objects, and applies a feature vector for location information of fourth objects existing on a frame preceding the second frame to a Kalman filter, thereby estimating a fourth feature vector for location information predicted to exist of the fourth objects on the second frame, and then calculates the Mahalanobis distance between the third feature vector for each of the third objects and the fourth feature vector for each of the fourth objects, so that two objects among the third objects and the fourth objects, for which the Mahalanobis distance between the third feature vector and the fourth feature vector is calculated to be greater than a preset second reference value, can be tracked as the same object.

In addition, according to one embodiment of the present invention, the real-time object tracking method may further include a performance information display step of calculating the number of frames in which object tracking is completed per preset unit time as an object tracking processing speed for the image, calculating a MOTA index and a MOTP index as information on the accuracy of object tracking for the image, and then displaying the object tracking processing speed, the MOTA index, and the MOTP index on a screen.

At this time, according to one embodiment of the present invention, the real-time object tracking method comprises an event generation step of generating an analysis event for analyzing the performance of object tracking at preset periodic intervals, a calculation step of calculating, when the analysis event is generated, an average object tracking processing speed, an average MOTA index, and an average MOTP index during the period from the time of occurrence of the analysis event, a comparison step of calculating a Euclidean norm of a vector having the average MOTA index and the average MOTP index as components, and then comparing the Euclidean norm with a preset threshold norm, and comparing the average object tracking processing speed with the preset threshold processing speed, if it is confirmed that the Euclidean norm exceeds the threshold norm and the average object tracking processing speed does not exceed the threshold processing speed, controlling the size of n to be reduced by a preset size so that object tracking for the image is performed according to the reduced size of n, and if it is confirmed that the Euclidean norm does not exceed the threshold norm and the average object tracking processing speed exceeds the threshold processing speed, controlling the size of n to be reduced by the preset size. The method may further include a size adjustment step of controlling the object tracking for the image to be performed according to the increased size of n by increasing the size, and not performing adjustment for the size of n if it is confirmed that the Euclidean norm exceeds the threshold norm and the average object tracking processing speed exceeds the threshold processing speed, and a message display step of generating a message notifying that there is an abnormality in the settings of the real-time object tracking device and displaying it on a screen if it is confirmed that the Euclidean norm does not exceed the threshold norm and the average object tracking processing speed does not exceed the threshold processing speed, without performing adjustment for the size of n.

Above, a real-time object tracking method according to an embodiment of the present invention has been described with reference to FIG. 2. Here, the real-time object tracking method according to an embodiment of the present invention can correspond to the configuration of the operation of the real-time object tracking device (110) described using FIG. 1, and therefore, a more detailed description thereof will be omitted.

A real-time object tracking method according to one embodiment of the present invention can be implemented as a computer program stored in a storage medium for execution through combination with a computer.

In addition, the real-time object tracking method according to an embodiment of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the present invention or may be those known to and usable by those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

Although the present invention has been described with specific details such as specific components and limited examples and drawings, these have been provided only to help a more general understanding of the present invention, and the present invention is not limited to the above examples, and those with common knowledge in the field to which the present invention pertains can make various modifications and variations from this description.

Therefore, the idea of the present invention should not be limited to the described embodiments, and all things that are equivalent or equivalent to the claims described below as well as the claims are included in the scope of the idea of the present invention.

110: Real-time object tracking device
111: 1st Pursuit Unit 112: 2nd Pursuit Unit
113: Repeat execution control unit 114: Performance information display unit
115: Event generation section 116: Output section
117: Comparison section 118: Resizing section
119: Message display section

Claims (12)

When an image is input, a first tracking unit performs object tracking based on the DeepSORT (Deep Simple Online and Real-time Tracking) algorithm from the first frame among the frames constituting the image to the nth frame (where n is a pre-designated value among natural numbers having a size of 3 or greater);
A second tracking unit that performs object tracking based on the SORT (Simple Online and Real-time Tracking) algorithm for the nth frame and the n+1th frame, and then redesignates the n+1th frame as the first frame and applies it to the first tracking unit; and
A repeating execution control unit that controls object tracking according to the first tracking unit and the second tracking unit to be performed repeatedly.
A real-time object tracking device comprising: In the first paragraph,
The above first tracking unit
A real-time object tracking device characterized in that, in order to perform object tracking based on the DeepSORT algorithm, when the frame of the current time is the first frame, first objects existing on the first frame are detected through a preset object detector, a first feature vector for location information of the first objects is calculated, and a feature vector for location information of second objects existing on a frame preceding the first frame is applied to a Kalman filter, thereby estimating a second feature vector for location information at which the second objects are predicted to exist on the first frame, and then calculating the Mahalanobis distance and the cosine distance between the first feature vector for each of the first objects and the second feature vector for each of the second objects, and tracking two objects among the first objects and the second objects, wherein the weighted average of the Mahalanobis distance and the cosine distance between the first feature vector and the second feature vector is calculated to be greater than a preset first reference value, as the same object. In the second paragraph,
The second tracking unit above
A real-time object tracking device characterized in that, in order to perform object tracking based on the SORT algorithm, if the frame of the current time is the second frame, third objects existing on the second frame are detected through the object detector, a third feature vector for location information of the third objects is calculated, and a feature vector for location information of fourth objects existing on a frame preceding the second frame is applied to a Kalman filter, thereby estimating a fourth feature vector for location information predicted to exist of the fourth objects on the second frame, and then calculating the Mahalanobis distance between the third feature vector for each of the third objects and the fourth feature vector for each of the fourth objects, and tracking two objects among the third objects and the fourth objects, for which the Mahalanobis distance between the third feature vector and the fourth feature vector is calculated to be greater than a preset second reference value, as the same object. In the first paragraph,
The number of frames in which object tracking is completed per preset unit time is calculated as an object tracking processing speed for the image, and as information on the accuracy of object tracking for the image, a MOTA (Multiple Object Tracking Accuracy) index and a MOTP (Multiple Object Tracking Precision) index are calculated, and then a performance information display section displays the object tracking processing speed, the MOTA index, and the MOTP index on a screen.
A real-time object tracking device further comprising: In paragraph 4,
An event generator that generates analysis events for analyzing the performance of object tracking at preset periodic intervals;
When the above analysis event occurs, a calculation unit for calculating an average object tracking processing speed, an average MOTA index, and an average MOTP index during the period from the time of occurrence of the above analysis event;
A comparison unit that calculates the Euclidean norm of a vector having the average MOTA index and the average MOTP index as components, and then compares the Euclidean norm with a preset threshold norm, and compares the average object tracking processing speed with a preset threshold processing speed;
a size adjustment unit that reduces the size of n by a preset size when it is confirmed that the Euclidean norm exceeds the threshold norm and the average object tracking processing speed does not exceed the threshold processing speed, and controls object tracking for the image to be performed according to the reduced size of n; and increases the size of n by the preset size when it is confirmed that the Euclidean norm exceeds the threshold norm and the average object tracking processing speed exceeds the threshold processing speed, and controls object tracking for the image to be performed according to the increased size of n; and does not perform adjustment for the size of n when it is confirmed that the Euclidean norm exceeds the threshold norm and the average object tracking processing speed exceeds the threshold processing speed; and
If it is confirmed that the above Euclidean norm does not exceed the above critical norm and that the above average object tracking processing speed does not exceed the above critical processing speed, a message display unit that generates a message indicating that there is an abnormality in the settings of the real-time object tracking device and displays it on the screen without performing any adjustment to the size of n
A real-time object tracking device further comprising: When an image is input, a first tracking step is performed to track an object based on the DeepSORT (Deep Simple Online and Real-time Tracking) algorithm, starting from the first frame among the frames constituting the image to the nth frame (where n is a pre-designated value among natural numbers having a size of 3 or greater);
A second tracking step of performing object tracking based on the SORT (Simple Online and Real-time Tracking) algorithm for the nth frame and the n+1th frame, and then returning to the first tracking step by redesignating the n+1th frame as the first frame; and
A repeat execution control step for controlling the object tracking according to the first tracking step and the second tracking step to be repeatedly performed.
A real-time object tracking method comprising: In Article 6,
The above first tracking step
A real-time object tracking method characterized in that, in order to perform object tracking based on the above DeepSORT algorithm, when the frame of the current time is the first frame, first objects existing on the first frame are detected through a preset object detector, a first feature vector for location information of the first objects is calculated, and a feature vector for location information of second objects existing on a frame preceding the first frame is applied to a Kalman filter, thereby estimating a second feature vector for location information predicted to exist of the second objects on the first frame, and then calculating the Mahalanobis distance and the cosine distance between the first feature vector for each of the first objects and the second feature vector for each of the second objects, and tracking two objects among the first objects and the second objects, wherein the weighted average of the Mahalanobis distance and the cosine distance between the first feature vector and the second feature vector is calculated to be greater than a preset first reference value, as the same object. In Article 7,
The second tracking step above
A real-time object tracking method characterized in that, in order to perform object tracking based on the above SORT algorithm, if the frame of the current time is the second frame, third objects existing on the second frame are detected through the object detector, a third feature vector for location information of the third objects is calculated, and a feature vector for location information of fourth objects existing on a frame preceding the second frame is applied to a Kalman filter, thereby estimating a fourth feature vector for location information predicted to exist of the fourth objects on the second frame, and then calculating the Mahalanobis distance between the third feature vector for each of the third objects and the fourth feature vector for each of the fourth objects, and tracking two objects among the third objects and the fourth objects, for which the Mahalanobis distance between the third feature vector and the fourth feature vector is calculated to be greater than a preset second reference value, as the same object. In Article 6,
A performance information display step in which the number of frames in which object tracking is completed per preset unit time is calculated as an object tracking processing speed for the image, and a Multiple Object Tracking Accuracy (MOTA) index and a Multiple Object Tracking Precision (MOTP) index are calculated as information on the accuracy of object tracking for the image, and then the object tracking processing speed, the MOTA index, and the MOTP index are displayed on a screen.
A real-time object tracking method further comprising: In Article 9,
An event generation step that generates analysis events for analyzing the performance of object tracking at preset periodic intervals;
When the above analysis event occurs, a calculation step for calculating an average object tracking processing speed, an average MOTA index, and an average MOTP index during the period from the time of occurrence of the above analysis event;
A comparison step of calculating the Euclidean norm of a vector having the average MOTA index and the average MOTP index as components, and then comparing the Euclidean norm with a preset threshold norm, and comparing the average object tracking processing speed with a preset threshold processing speed;
a size adjustment step of reducing the size of n by a preset size so that object tracking for the image is performed according to the reduced size of n when it is confirmed that the Euclidean norm exceeds the threshold norm and the average object tracking processing speed does not exceed the threshold processing speed, and controlling the size of n to be increased by the preset size so that object tracking for the image is performed according to the increased size of n when it is confirmed that the Euclidean norm exceeds the threshold norm and the average object tracking processing speed exceeds the threshold processing speed, and not performing an adjustment for the size of n when it is confirmed that the Euclidean norm exceeds the threshold norm and the average object tracking processing speed exceeds the threshold processing speed; and
If it is confirmed that the above Euclidean norm does not exceed the above critical norm and that the average object tracking processing speed does not exceed the above critical processing speed, a message display step for generating and displaying a message on the screen indicating that there is an abnormality in the settings of the real-time object tracking device without performing any adjustment to the size of n
A real-time object tracking method further comprising: A computer-readable recording medium storing a computer program for executing the method of any one of claims 6 to 10 through combination with a computer. A computer program stored in a storage medium for executing the method of any one of claims 6 to 10 through combination with a computer.

Images