TWI718437B - Object location estimating method with timestamp alignment function and related object location estimating device - Google Patents

Abstract

An object location estimating method with a timestamp alignment function is applied to an object location estimating device. The object location estimating method includes acquiring a plurality of first frames from a first camera, setting a first predetermined point of time, defining a first previous frame and a first next frame closest to the first predetermined point of time from the plurality of first frames, acquiring a first previous coordinate value about an object within the first previous frame and a first next coordinate value about the object within the first next frame, and utilizing the first previous coordinate value and the first next coordinate value to compute a first estimation coordinate value of the object at the first predetermined point of time.

Description

Object position estimation method with time stamp alignment function and object position estimation device

The present invention provides an object position estimation method and an object position estimation device, in particular an object position estimation method with a time stamp alignment function and an object position estimation device.

With the advancement of technology, multi-camera surveillance systems are widely used in a wide range of areas or environments. The field of view of one camera is not enough to cover the entire surveillance area. Therefore, the traditional multi-camera surveillance system points each camera to different areas. When tracking objects, it can judge the movement of objects in the surveillance area based on the surveillance images of different cameras. Trajectory. In order to obtain the precise movement trajectory of the object, the traditional multi-camera monitoring system will set the time synchronization for all cameras to ensure that each camera can obtain the monitoring picture at the required time. However, the time point and capture frequency at which each camera obtains the monitoring screen will be affected by the hardware configuration or the network environment, resulting in a gap in metadata timestamps between different cameras. The greater the time stamp gap, the worse the accuracy of object tracking. Therefore, how to overcome this defect and design an object tracking method that can accurately align the object metadata of multiple cameras is one of the key development goals of the surveillance industry.

The present invention provides an object position estimation method and an object position estimation device with a time stamp alignment function to solve the above-mentioned problems.

The scope of patent application of the present invention discloses a method for estimating the position of an object with a time stamp alignment function, which includes obtaining a plurality of first frames of a first camera, setting a first preset time point, and starting from the plurality of first frames The screen fetches a first previous screen and a first later screen that are closest to the first preset time point, and obtains a first previous coordinate value of an object in the first previous screen and the first later screen A first later coordinate value of an object, and a first estimated coordinate value of the object at the first preset time point is calculated by using the first previous coordinate value and the first later coordinate value.

The patent application scope of the present invention further discloses an object position estimation device with a time stamp alignment function, which includes a receiver and a processor. The receiver obtains a picture generated by at least one camera. The processor is electrically connected to the receiver for obtaining a plurality of first frames of a first camera, setting a first preset time point, and fetching the plurality of first frames closest to the first preset time point A first previous screen and a first later screen, obtaining a first previous coordinate value of an object in the first previous screen and a first later coordinate value of the object in the first later screen, and Using the first previous coordinate value and the first later coordinate value, a first estimated coordinate value of the object at the first preset time point is calculated.

The field of view of the multiple cameras of the present invention can be partially overlapped or not overlapped at all, and the time synchronization setting is completed in advance. The object position estimation method and the object position estimation device of the present invention set a number of preset time points according to a specific frequency, and obtain each piece of object metadata of each camera before and after the preset time point by interpolation or other calculation functions Obtain the metadata of the object at this preset time point, and estimate the metadata of the object after time alignment. The present invention can overcome the time stamp gap of metadata between different cameras, and effectively improve the accuracy of object tracking.

Please refer to FIG. 1, which is a functional block diagram of the object position estimation device 10 according to an embodiment of the present invention. When multiple cameras capture images of the same monitoring range from different perspectives, the operating systems of these cameras have completed the time synchronization setting, but each frame of the images obtained by different cameras will still have a slight time difference. The object position estimation device 10 has a time stamp alignment function, which is used to align the object metadata in the frame obtained by multiple cameras to improve the accuracy of object tracking. The object position estimating device 10 can be a server, which collects the screen data of multiple cameras for time stamp alignment; the object position estimating device 10 can also be a camera with special functions, which can receive the monitoring screens of other cameras and compare it with your own monitoring screens. Perform time stamp alignment.

Please refer to Figures 1 to 4. Figure 2 is a flowchart of an object position estimation method according to an embodiment of the present invention, Figure 3 is a schematic diagram of images obtained by a camera at different timings according to an embodiment of the present invention, and Figure 4 is A schematic diagram of an object position estimation device 10 and a camera according to an embodiment of the present invention. The object position estimation device 10 may include a receiver 12 and a processor 14 electrically connected together. The receiver 12 obtains images produced by other cameras. The object position estimation method executed by the processor 14 is applicable to the object position estimation device 10 shown in FIG. 1. Although the time synchronization settings of the first camera 16 and the second camera 18 have been completed, due to the variability of the camera’s system performance and network connection quality, the number of frames per second (FPS) captured by each camera will follow Time changes, as shown in Figure 3.

Regarding the object position estimation method, step S200 is first executed, and the processor 14 may determine the frequency of time stamp alignment. For example, when the captured frames of the first camera 16 and the second camera 18 are 60 fps, the time stamp alignment frequency of the object position estimation method It can be 60fps or 30fps, or other values. Next, steps S202 and S204 are executed, and the processor 14 obtains a plurality of first frames Is1 and a plurality of second frames Is2 of the first camera 16 and the second camera 18 through the receiver 12, and sets the first preset according to the timestamp alignment frequency. Set time point T1. Then step S206 is performed to find the first previous picture Ip1 and the first later picture In1 that are closest to the first preset time point T1 from the first picture Is1, and find the first picture that is closest to the first preset from the second picture Is2 The second previous picture Ip2 and the second later picture In2 at the time point T1.

Then, step S208 is executed to analyze the objects O and O'in the previous screens Ip1, Ip2 and later screens In1, Is2 using the object tracking technology, and obtain the first previous coordinate value Cp1 of the object O in the first previous screen Ip1 and the first previous coordinate value Cp1 in the first previous screen Ip1. The first later coordinate value Cn1 of a later screen In1 and the second previous coordinate value Cp2 of the object O'in the second previous screen Ip2 and the second later coordinate value Cn2 of the second later screen In2. Later, step S210 is executed to calculate the first estimated coordinate value Ce1 of the object O at the first preset time point T1 by using the first previous coordinate value Cp1 and the first later coordinate value Cn1, and use the second previous coordinate value Cp2 Calculate the second estimated coordinate value Ce2 of the object O'at the first preset time point T1 with the second later coordinate value Cn2.

After the first estimated coordinate value Ce1 and the second estimated coordinate value Ce2 are obtained, steps S212 and S214 are performed to select one of the plurality of first frames Is1 and the plurality of second frames Is2 to be spliced to generate a spliced frame I', The first estimated coordinate value Ce1 of the object O and the second estimated coordinate value Ce2 of the object O'are displayed on the splicing screen I'. In this way, the object position estimation device 10 can form the traveling trajectories of the objects O and O'respectively tracked by the two cameras 16 and 18 on the splicing screen I', which is convenient for the user to observe or provide for other computing applications. In this embodiment, the first previous picture Ip1 and the second later picture In2 are preferably used for splicing, but the actual application is not limited to this.

In particular, the first estimated coordinate value Ce1 belongs to the predicted amount of the object O at the first preset time point T1, and the second estimated coordinate value Ce2 belongs to the predicted amount of the object O'at the first preset time point T1, so The first estimated coordinate value Ce1 and the second estimated coordinate value Ce2 are preferably regarded as substantially the same in time sequence; regardless of whether there are slight errors in the actual situation, the object position estimation method of the present invention can reduce the errors of the two to the extreme Small value, so it can still be defined as substantially the same.

Please refer to Figures 3 to 7. Figures 5 and 6 are schematic diagrams of converting the previous and later coordinate values into estimated coordinate values according to the embodiment of the present invention. Figure 7 is the stitching screen of the embodiment of the present invention. The schematic diagram. The object position estimation method of the present invention can obtain the first estimated coordinate value Ce1(x3, x3, y2) between the first previous coordinate value Cp1(x1, y1) and the first later coordinate value Cn1(x2, y2) by using an interpolation method. y3). Among them, the first previous picture Ip1 is obtained at the time point Tp1, and the first later picture In1 is also obtained at the time point Tn1. The coordinate value x3=x1+(x2-x1)*(T1-Tp1)/(Tn1-Tp1), And the coordinate value y3= y1+(y2-y1)*(T1-Tp1)/(Tn1-Tp1). The calculation method for converting the second previous coordinate value Cp2 and the second later coordinate value Cn2 into the second estimated coordinate value Ce2 is the same as the first estimated coordinate value Ce1, which will not be repeated here. In addition, the calculation method of the estimated coordinate value is not limited to the interpolation method, and may be obtained through other calculation functions.

The object position estimation method can additionally set one or more preset time points later than the first preset time point T1, and obtain the estimated coordinate values of the objects O and O'at this preset time point. Taking the second preset time point T2 as an example, the second preset time point T2 can be determined according to the timestamp alignment frequency contained in step S200. The first picture Is1 has a third previous picture Ip3 and a third later picture In3 that are closest to the predetermined time point before and after the second predetermined time point T2. The position of the object O in the third previous screen Ip3 is the third previous coordinate value (not shown in the figure), and the position in the third later screen In3 is the third later coordinate value (not shown in the figure). The third previous coordinate value and the third later coordinate value may use interpolation or other calculation functions to generate a third estimated coordinate value (not shown in the figure). The previous coordinate value and the later coordinate value of the fourth previous frame Ip4 and the fourth later frame In4 that are closest to the second preset time point T2 in the second frame Is2 of the object O'can also be converted to generate another estimated coordinate value. The third previous picture Ip3 can be spliced with the fourth later picture In4 to generate a spliced picture I", and the third estimated coordinate value of the object O and another estimated coordinate value of the object O'are displayed on the spliced picture I".

If there are one or more unused frames Iu between the first later frame In1 and the third previous frame Ip3, the object position estimation method of the present invention can directly discard the data of the object O in the unused frame Iu, which means that Using the coordinate value of the object O in the screen Iu will not be used to calculate the third estimated coordinate value. The unused picture Iu is defined as any picture that is not used for object position estimation between two preset time points T1 and T2. In addition, the unused picture Iu or the third previous picture Ip3 and the third later picture In3 can be used to calculate the estimated coordinate value of the object O at the second preset time point T2 with a specific calculation function; for example, the unused picture Iu can be compared Low weight, the third previous picture Ip3 and the third later picture In3 are given higher weights, and all three frames are used to calculate estimated coordinate values. Or, the object position estimation method can analyze and compare the unused frame Iu with the third previous frame Ip3, and choose one of the two to use the third subsequent frame In3 to estimate the object position.

In the embodiment of the present invention, if the object position estimation device 10 is a server electrically connected to multiple cameras, its time stamp alignment frequency can be set by itself, and its estimated coordinate value can be calculated stably at each preset time point. If the object position estimation device 10 is a camera that receives screen data from other cameras, it can perform screen capture and object position estimation alternately, for example, first calculate the estimated coordinate value of the previous preset time point, and then continue to calculate the next prediction after it is completed. Set the estimated coordinate value of the time point, so the interval of the preset time point may be slightly different. The setting of the timestamp alignment frequency and the preset time point is determined according to the hardware equipment of the object position estimating device 10 and its computing performance, and there may be many kinds of changes, which will not be described separately here.

In summary, the field of view of multiple cameras can be partially overlapped or not overlapped at all, and the time synchronization setting is completed in advance. The object position estimation method and the object position estimation device of the present invention set a number of preset time points according to a specific frequency, and obtain each piece of object metadata of each camera before and after the preset time point by interpolation or other calculation functions Obtain the metadata of the object at this preset time point, and estimate the metadata of the object after time alignment. Compared with the prior art, the present invention can overcome the time stamp gap of metadata between different cameras, and effectively improve the accuracy of object tracking. The above are only preferred embodiments of the present invention, and all equivalent changes and modifications made in accordance with the scope of the patent application of the present invention shall fall within the scope of the present invention.

10: Object position estimation device 12: Receiver 14: Processor 16: First camera 18: Second camera Is1: First picture Is2: Second picture Ip1: First previous picture In1: First later picture Ip2: No. Two previous picture In2: second later picture Ip3: third previous picture In3: third later picture Ip4: fourth previous picture In4: fourth later picture Iu: unused picture I', I”: splicing picture O , O': object Cp1: first previous coordinate value Cn1: first later coordinate value Ce1: first estimated coordinate value Cp2: second previous coordinate value Cn2: second later coordinate value Ce2: second estimated coordinate value T1 : First preset time point T2: second preset time point S200, S202, S204, S206, S208, S210, S212, S214: steps

Figure 1 is a functional block diagram of an object position estimation device according to an embodiment of the present invention. Figure 2 is a flowchart of an object position estimation method according to an embodiment of the present invention. FIG. 3 is a schematic diagram of the images obtained by the camera at different timings according to the embodiment of the present invention. Figure 4 is a schematic diagram of an object position estimation device and a camera according to an embodiment of the present invention. Figures 5 and 6 are schematic diagrams of the conversion of the previous coordinate value and the later coordinate value to the estimated coordinate value according to the embodiment of the present invention. Figure 7 is a schematic diagram of a splicing screen according to an embodiment of the present invention.

S200, S202, S204, S206, S208, S210, S212, S214: steps

Claims (9)

A method for estimating the position of an object with a time stamp alignment function, which includes: obtaining a plurality of first frames of a first camera; setting a first preset time point; taking out the plurality of first frames closest to the first frame A first previous frame and a first later frame at a preset time point; obtaining a first previous coordinate value of an object in the first previous frame and a first later frame of the object in the first later frame Coordinate value; and using the first previous coordinate value and the first later coordinate value to calculate a first estimated coordinate value of the object at the first preset time point. According to the method for estimating the position of an object according to claim 1, it uses an interpolation method to calculate the first estimated coordinate value between the first previous coordinate value and the first later coordinate value. The method for estimating the position of an object as described in claim 1, further comprising: obtaining a plurality of second frames of a second camera; and extracting a second previous one that is closest to the first preset time point from the plurality of second frames Frame and a second later frame; obtain a second previous coordinate value of another object in the second previous frame and a second later coordinate value of the other object in the second later frame; and use the The second previous coordinate value and the second later coordinate value are calculated to obtain a second estimated coordinate value of the other object at the first preset time point; wherein the first estimated coordinate value and the second estimated coordinate value The timing is essentially the same. The method for estimating the position of an object according to claim 3, further comprising: generating a spliced screen based on the first previous frame and the second later frame; and the first estimated coordinate value of the object and the other object The second estimated coordinate value is displayed on the splicing screen. The object position estimation method according to claim 3, wherein the first camera and the second camera have completed time synchronization settings. The method for estimating the position of an object according to claim 1, further comprising: setting a second preset time point later than the first preset time point; taking out the plurality of first frames closest to the second preset time point A third previous screen and a third later screen at a time point; to obtain a third previous coordinate value of the object in the third previous screen and a third later coordinate value of the object in the third later screen And using the third previous coordinate value and the third later coordinate value to calculate a third estimated coordinate value of the object at the second preset time point. The object position estimation method according to claim 6, further comprising: determining whether there is at least one unused picture between the first later picture and the third previous picture; and discarding the at least one unused picture The data of the object. The object position estimation method according to claim 6, wherein the second preset time point is determined based on the calculation time of the first estimated coordinate value. A device for estimating the position of an object with a time stamp alignment function, comprising: a receiver, which obtains a picture generated by at least one camera; and a processor, which is electrically connected to the receiver, and is used to execute the request items 1 to 8 Any of the described object position estimation methods.

Images