WO2022225819A1

WO2022225819A1 - System and method for processing an incoming signal

Info

Publication number: WO2022225819A1
Application number: PCT/US2022/025052
Authority: WO
Inventors: Dennis Braunreiter; Dave Douglas; Anibal MORALES
Original assignee: Raytheon Company
Priority date: 2021-04-23
Filing date: 2022-04-15
Publication date: 2022-10-27
Also published as: WO2022225819A9

Abstract

A method and system for carrying out a method to process a signal includes determining a coverage range for a Fast Fourier Transform (FFT). A number of FFT dimensions are defined for the FFT. Each FFT dimension is swept for searching prime power factors (PPF) and powers of two (PO2) mathematically to yield optimal sized FFTs for all sizes within the coverage range and creating a FFT plan optimizing execution instructions and memory for each combination of PPF and PO2. An index is created having a look-up table mapping each combination of PPF and PO2 to data structures for optimal FFTs, wherein the index includes memory locations for data structures for each optimal FFT. An optimal FFT is selected from the look-up table for the incoming signal.

Description

SYSTEM AND METHOD FOR PROCESSING AN INCOMING SIGNAL

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to and the benefit of U.S. Application No. 17/587,677, filed on January 28, 2022 and entitled “TARGET TRACKING SYSTEM”, U.S. Provisional Patent Application No. 63/178,753, filed on April 23, 2021 and entitled “TARGET TRACKING SYSTEM AND PROCESSING ALGORITHM”, and U.S. Provisional Patent Application No. 63/178,758, filed on April 23, 2021 and entitled “TARGET TRACKING SYSTEM”, the contents of which are incorporated herein by reference as though fully set forth herein.

FIELD

[0002] The subject disclosure relates to signal processing, and more particularly, to efficiency processing a signal with Fast Fourier Transforms (FFT) in real time. The signal processing can occur in environments including audio processing, radar signal processing, and image processing including medical image processing. In one example described herein, the methods disclosed are carried out within a target targeting system for a high energy laser (HEL) system.

BACKGROUND

[0003] High energy laser (HEL) weapons systems require a fine tracking system to point the laser beam accurately on a target. A wide area medium infrared camera system is required to acquire and track the target either autonomously or assisted by the operator to cue and point the fine tracking system by closing a track loop around a gimbaled sensor. The wide area camera field of view contains other targets of interest, masked by background clutter if of a smaller size (as in the case of mortars and UAS), and possess varying illumination conditions. The wide area tracking system must continuously track during the HEL engagement for re-acquisition, battle damage assessment, and acquisition of other targets once the primary target is destroyed. Targets under track will maneuver, emit HEL beam heating effects, have varying illumination, and break apart. [0004] Current infrared (IR) and visible tracking algorithms and systems are designed for either open loop or closed loop tracking of targets but not optimal for HEL mission requirements. For example, while the target is heating, image artifacts appear and obscure the target shape affecting computer vision and/or traditional correlation tracker techniques. Although these trackers provide track on multiple targets, they do not track multiple pieces of the target that become spatially separated in case of target break lock during destruction that move separately. This makes it impossible to accurately assess damage to a target. As such, these trackers are designed to track a target generally, but are not equipped to track the target in a way that allows for an indication of battle damage or target kill.

[0005] Further, engaging fast moving targets such as rockets and mortars requires the HEL weapon system to automatically acquire, track, and provide a target aimpoint to point the HEL beam. The target under track will maneuver, be relatively stationary or moving in a cluttered background, emit HEL beam heating effects, and have varying illumination passively or from an active illuminator, all of which impact a correlation tracker performance by integration of negative effects into the target signature used for correlation causing track break lock. In the case of mortars and rockets tracking the target can be quite small in a given pixel dimension which limits the use of traditional computer vision techniques that require larger target dimensions. Most significantly, as the target heats up from HEL effects, the hot spot of the target will cause traditional tracking techniques to lose lock on target features and track the hot spot created by the HEL beam causing track drift and eventual break lock on the target aimpoint.

[0006] Current semi-active or passive target imaging trackers are based on frame-to- frame correlation to an integrating or fixed target reference and will break track lock in cluttered backgrounds and target aspect change. Break lock occurs from feeding back corrupted target information to the integrated target reference that corrupts the correlation measurement between the target and the reference. Some approaches to mitigate the negative effects in the target reference update include SNR based thresholding of the target prior to integration in the reference as well as methods to estimate the rotation and scale of the target from the image data using computer vision (CV) techniques. Most CV are not suitable though for small targets and dynamic intensity variations, and cause track break lock and corrupted track estimates. [0007] Further, break lock occurs from the hot spot of the target from HEL heating biasing the track target position and causing the tracker to lose lock aimpoint features, which eventually drifts the tracked position off of the target. Others have attempted to solve this problem by eliminating the hot spot from the image or finding a track position on the target that is not near the expected heating point on the target. The approaches that eliminate a hot spot on the image do not account for temporal effects such as hot spot size growth and intensity increase or dynamic range compression and saturation due to intensity increases, all of which cause tracked point corruption and break lock. Approaches that utilize a target feature offset from the hot spot are limited and can be inconsistent in location and SNR further causing break lock during HEL heating.

[0008] HEL weapon systems require precision and very high-speed target aimpoint tracking. Precision tracking approaches including centroid based tracking and correlator-based tracking are known for performance and robustness in clutter and noise. A correlation tracker estimates the shift between a reference map of the target and the current input image. These shifts provide pointing to filtering algorithms for a variety of system laser beam pointing and tracking functions. The most efficient computational implementation for estimating the cross correlation of a target and reference map is done with a 2D (Fast Fourier Transform) FFT. For state-of-the-art HEL tracking applications, the target area is typically separated from the background to reduce background clutter from corrupting the estimate. The size and shape of the target can change over the duration of the engagement. The 2D FFT sizes used in implementation must be adapted to the target size to minimize computation time and latency for high-speed image tracking and subsequent laser beam pointing. The FFT correlator needs to execute the 2D FFT operation in real time in a processing-constrained real time device at very high execution (frame) rates. Extra execution time impacts the algorithm processing timeline and prevents the tracking mission.

[0009] Prior attempts have accepted the execution time penalty of creating a size changing 2D FFT plan in real-time as an unavoidable cost or selected a non-optimal predicted plan size to minimize the plan creation cost. Other approaches create oversized FFTs that will cover all the incoming signal sizes, the results being extracted after execution. Still other attempts have created a series of plans based on conjecture, and selected the closest one, without guarantees for optimal size or execution. Finally, approaches have reduced the computational load of the correlator using binary or threshold representations of the targets which are susceptible to noise and clutter.

SUMMARY

[0010] In at least one aspect, the subject technology relates to a target tracking system for tracking a target. The system includes at least one imaging system configured to continuously capture a plurality of images of the target. The system includes a centroid tracker configured to run a first tracker mode to filter clutter and noise from the images, determine a centroid of the target, and generate a plurality of recursive reference images of the target based on the images. The system includes a correlator tracker configured to run a second tracker mode to determine a shift between successive images of the target and generate an offset for pointing the target tracking system at the target. The correlator tracker utilizes the recursive references images when the first tracker mode is running. The system is configured to track the target by running the first tracker mode and the second tracker mode simultaneously and monitoring the tracker modes for a failure. When a failure is detected in one of the tracker modes, the system runs only the tracker mode where no failure was detected.

[0011] In some embodiments, the system includes a high energy laser (HEL) adjustment module. The HEL adjustment module is configured to run two stages. A first stage detects changes in intensity within a spatial pattern of the target within the image due to heating effects to identify a HEL affected area. After the change in intensity is detected, a second stage turns a HEL adjustment on to remove the heating effects within the spatial pattern of the target within the image. In some cases, when the HEL adjustment is on, the system is further configured to track the target using only areas of the target not within the HEL affected area. In some embodiments, the system is further configured to update the recursive reference image based on the HEL affected area. In some cases, when the HEL adjustment module is on, the system is configured to continuously monitor a peak intensity of the target in the image and update parameters of the system based on the peak intensity of the target in the image. In some embodiments, when the HEL adjustment module is on, the imaging system is configured to adjust sensor gains based on the peak intensity of the target in the image to keep the peak intensity of the target in the image within a predetermined range.

[0012] In some embodiments, the correlation tracker runs at a relatively high speed with respect to a speed of the centroid tracker. In some cases, the system includes a target state estimator configured to estimate a current state of the target by fusing data from the correlator tracker and centroid tracker. The centroid tracker and the correlator tracker can be configured to subsequently rely on the estimated current state of the target.

[0013] In some embodiments, the system includes a line-of-sight manager configured to check for irregular motion of the target and determine an aimpoint for the target. When the first tracker mode and the second tracker mode are operational, the aimpoint is based on the centroid location from the centroid tracker adjusted by the shift from the correlator tracker and the update from the HEL module. The system can also include a HEL and the line-of-sight manager can be configured to determine a second offset to the aimpoint for pointing the HEL. The system can then point the HEL at the target based on the second offset and the aimpoint.

[0014] In some embodiments, the system utilizes gates outlining boundaries within the image within which the system looks for the target. The system can continuously resize the gates based on a shift between successive recursive references images. In some cases, the gates comprise two concentric areas. The first concentric area is a buffer around an estimated extent of the target size used to determine the target position. The second concentric area is an annulus around the first concentric area. The system can estimate background based on the second concentric area.

[0015] In at least one aspect, the subject technology relates to a method of tracking a target with a target tracking system. The method includes continuously capturing, with at least one imaging system, a plurality of images of the target. The method includes running a first tracker mode using a centroid tracker, the first tracker mode including filtering clutter and noise from the images, determining a centroid of the target, and generating a plurality of recursive reference images of the target based on the images. The method also includes running a second tracker mode using a correlator tracker, the second tracker mode including determining a shift between successive images of the target and generating an offset for pointing the target tracking system at the target, wherein the correlator tracker utilizes the recursive references images when the first tracker mode is running. Finally, the method includes tracking the target with the system by running the first tracker mode and the second tracker mode simultaneously and monitoring the tracker modes for a failure, and, when a failure is detected in one of the tracker modes, running only the tracker mode where no failure was detected.

[0016] In some embodiments, the method includes running a high energy laser (HEL) adjustment module in two stages. A first stage includes detecting changes in intensity within a spatial pattern of the target within the image due to heating effects to identify a HEL affected area. After the change in intensity is detected, a second stage includes turning a HEL adjustment on to remove the heating effects within the spatial pattern of the target within the image. In some cases, after turning the HEL adjustment on, the method includes tracking the target using only areas of the target not within the HEL affected area. The recursive reference image is then updated based on the HEL affected area. In some embodiments, after turning the HEL adjustment on, the method includes continuously monitoring a peak intensity of the target in the image and updating parameters of the system based on the peak intensity of the target in the image. In some cases, after turning the HEL adjustment on, the method includes adjusting sensor gains of the imaging system based on the peak intensity of the target in the image to keep the peak intensity of the target in the image within a predetermined range.

[0017] In some embodiments, the correlation tracker runs at a relatively high speed with respect to a speed of the centroid tracker. The method can include estimating, with a target state estimator, a current state of the target by fusing data from the correlator tracker and centroid tracker. The centroid tracker and the correlator tracker can also be configured to subsequently rely on the estimated current state of the target.

[0018] In some embodiments, the method includes checking for irregular motion of the target and determining an aimpoint for the target with a line-of-sight manager. When the first tracker mode and the second tracker mode are operational, the aimpoint is based on the centroid location from the centroid tracker adjusted by the offset from the correlator tracker and the update from the HEL module. The method can further include determining a second offset to the aimpoint for aiming a HEL and aiming the HEL at the target based on the second offset and the aimpoint.

[0019] In some embodiments, the method includes determining a plurality of gates outlining boundaries within the image within which the system looks for the target. The gates are then continuously resized based on a shift between successive recursive references images. In some cases, the gates include two concentric areas. The first concentric area is a buffer around an estimated extent of the target size used to determine the target position. The second concentric area is an annulus around the first concentric area. The system can estimate background based on the second concentric area.

[0020] In at least one aspect, the subject technology relates to a target tracking system for tracking a target with an Acquisition Track Sensor (ATS) and fine track sensor (FTS). The ATS includes a wide field of view (FOV) camera system continuously capturing an image of the target. The ATS is configured to identify the target for tracking and cue the target for tracking. The FTS includes a narrow FOV camera system. The FTS continuously re-centers the narrow FOV camera system around the target after the target is cued by the ATS. The ATS and FTS include independent steering mechanisms for respective camera systems.

[0021] In some embodiments, the target tracking system is a high energy laser (HEL) tracking system. The ATS can be a Mid Wave Infralnfra-Red (MWIR) tracker and the FTS can be a Short Wave Infralnfra-Red (SWIR) tracker. The SWIR can include a centroid tracker configured to filter clutter and noise from the images, determine a centroid of the target, and generate a plurality of recursive reference images of the target based on the images. The SWIR can also include a correlator tracker to determine a shift between successive recursive reference images of the target and generate an offset for an aimpoint for target. In some embodiments, the system can include a target state estimator. The target state estimator can be configured to estimate a current state of the target by fusing data from the correlator tracker, the centroid tracker, and the MWIR, and the recursive reference images can be updated based on the estimated current state of the target. In some cases, the SWIR tracker includes a high-rate SWIR camera system and a low rate SWIR camera system and the MWIR camera system includes a low rate MWIR camera system. [0022] In some embodiments, the FTS is configured to track a plurality of features of the target, including a plurality of segments of the target. The system can be a HEL tracking system and include a HEL beam. The system can then point the HEL beam to the cued target based on the FTS.

[0023] In some embodiments, the system is configured to run a battle damage assessment mode. In the battle assessment mode, the system detects when, after the target is struck by the HEL beam, kinematics of the target deviates from expected dynamics of the target. In the battle assessment mode, the system also detects when one or more of the segments of the target become spatially separated from the target and determines target damage. In the battle assessment mode, the system then provides a target damage assessment indicator based on the target damage determined. In some cases, the system is configured to reassess the target after the HEL beam is turned off to determine the target damage, the target damage being based in part on a distance between spatially separated segments of the target. In some embodiments, the ATS is configured to cue the target for tracking when the target features meet a plurality of predetermined criteria, including: target size, target shape, target velocity, signal-to-noise ratio, and an amount of time appearing.

[0024] In at least one aspect, the subject technology relates to a method of tracking a target with a target tracking system. The method includes operating an ATS including a wide field of view (FOV) camera system to continuously capture an image of the target, identify the target for tracking, and cue the target for tracking. The method also includes operating an FTS including a narrow FOV camera system to continuously re-center the narrow FOV around the target after the target is cued by the ATS, wherein the ATS and FTS include independent steering mechanisms for respective camera systems.

[0025] In some embodiments, the target tracking system is a high energy laser HEL tracking system, the ATS is an MWIR tracker, and the FTS is a SWIR tracker. In some cases, operating the FTS includes operating a centroid tracker to filter clutter and noise from the images, determine a centroid of the target, and generate a plurality of recursive reference images of the target based on the images. Operating the FTS can also include operating a correlator tracker to determine a shift between successive recursive reference images of the target and generate an offset for an aimpoint for target. In some embodiments, the method includes estimating, with a target state estimator, a current state of the target by fusing data from the correlator tracker, the centroid tracker, and the MWIR. The recursive reference images can then be updated are based on the estimated current state of the target. In some cases, the SWIR tracker includes a high-rate SWIR camera system and a low rate SWIR camera system, and the MWIR camera system includes a low rate MWIR camera system.

[0026] In some embodiments, the method includes tracking a plurality of features of the target with the FTS, including a plurality of segments of the target. In some cases, the system is a HEL tracking system including a HEL beam. The method can include pointing the HEL beam at the target, after the target is cued, based on the SWIR tracker. In some embodiments, the method includes running the system in a battle damage assessment mode. In the battle damage assessment mode, the method includes detecting when, after the target is struck by the HEL beam, kinematics of the target deviates from expected dynamics of the target. In the battle damage assessment mode, the method can include detecting when one or more of the segments of the target become spatially separated from the target. In the battle damage assessment mode, the method includes determining target damage and providing a target damage assessment indicator based on the target damage determined.

[0027] In some embodiments, the method includes turning the HEL beam off. After the HEL beam is turned off, the target can be reassessed to determine the target damage, the target damage based in part on a distance between spatially separated segments of the target. In some embodiments, the ATS cues the target for tracking when the target features meet a plurality of predetermined criteria, including: target size, target shape, target velocity, signal-to-noise ratio, and an amount of time appearing.

[0028] In at least one aspect, the subject technology relates to a target tracking system for tracking a target. At least one imaging system is configured to continuously capture a plurality of images of the target. A centroid tracker is configured to run a first tracker mode to generate a plurality of recursive reference images of the target based on the images. A correlator tracker is configured to run a second tracker mode to determine a shift between successive images of the target and generate an offset for pointing the target tracking system at the target. The correlator tracker utilizes the recursive references images when the first tracker mode is running. At least one module configured to execute an algorithm including the following steps: (a) defining a coverage range for a Fast Fourier Transform (FFT) based on an expected minimum size and an expected maximum size of the images; (b) defining a number of FFT dimensions for the FFT; (c) sweeping each FFT dimension and searching prime power factors (PPF) and powers of two (P02) mathematically to yield optimal sized FFTs for all sizes within the coverage range, and creating a FFT plan optimizing execution instructions and memory for each combination of PPF and P02; and (d) creating an index comprising a look-up table mapping each combination of PPF and P02 to data structures for optimal FFTs, wherein the index includes memory locations for the data structures. The system is configured to track the target by running the first tracker mode and the second tracker mode simultaneously. The system selects an optimal FFT from the look-up table for the images. Finally, the system executes the optimal FFT to determine a cross correlation between the reference image and the current image.

[0029] In some embodiments, selecting an optimal FFT is carried out by the correlator tracker as part of the second tracking mode, wherein the second tracking mode is a high-rate execution loop. In some embodiments, steps (a)-(d) are carried out by the centroid tracker as part of the first tracker mode. In some cases, the data structures include an execution wisdom plan for each optimal FFT.

[0030] In at least one aspect, the subject technology relates to a method which includes the following steps: (a) determining an expected minimum size and an expected maximum size to define a coverage range for a Fast Fourier Transform (FFT); (b) defining a number of FFT dimensions for the FFT; (c) sweeping each FFT dimension and searching prime power factors (PPF) and powers of two (P02) mathematically to yield optimal sized FFTs for all sizes within the coverage range and creating a FFT plan optimizing execution instructions and memory for each combination of PPF and P02; (d) creating an index comprising a look-up table mapping each combination of PPF and P02 to data structures for optimal FFTs, wherein the index includes memory locations for data structures for each optimal FFT; and (e) receiving an incoming signal having a signal size, and if the signal size is within the coverage range, selecting an optimal FFT from the look-up table for the incoming signal based on the signal size. [0031] In some embodiments, during step (a), the expected minimum size and expected maximum size are determined based on a received input. In some cases, the method further includes executing the optimal FFT for processing the incoming signal. In some embodiments, during step (c), all combinations of PPF and P02 within the coverage range are searched. The number of FFT dimensions in the method can be two or at least three.

[0032] In some embodiments, the data structures include an execution wisdom plan for each FFT. In some cases, steps (a)-(d) are performed initially as part of an out of loop process. Step (e) is then repeatedly performed as part of a high-rate execution loop as the incoming signal is received, the incoming signal varying over time. In some embodiments, the method further includes step (f) of executing an out of loop deconstruction plan to deconstruct the FFT plans.

BRIEF DESCRIPTION OF THE DRAWINGS

[0033] So that those having ordinary skill in the art to which the disclosed system pertains will more readily understand how to make and use the same, reference may be had to the following drawings.

[0034] FIG. l is a block diagram of a target tracking environment including a target tracking system in accordance with the subject technology.

[0035] FIG. 2 is a block diagram of the tracking architecture for a target tracking system in accordance with the subject technology.

[0036] FIG. 3 is a block diagram of functions carried out by a fine track sensor (FTS) tracker as part of the tracking system of FIG. 2.

[0037] FIG. 4 is an exemplary image generated by the FTS tracker in accordance with the subject technology.

[0038] FIG 5 is a block diagram of a process of adaptively sizing and executing an FFT within the target tracking system in accordance with the subject technology.

[0039] FIG. 6 is a block diagram of functions carried out by the ATS tracker as part of the tracking system of FIG. 2. [0040] FIGs. 7a-7f are block diagrams of exemplary memory layouts for processing a signal in accordance with the subject technology.

[0041] FIG. 8 is a block diagram showing the functional relationship of components for an algorithm in accordance with the subject technology.

PET ATT, ED DESCRIPTION

[0042] The subject technology overcomes many of the prior art problems associated with signal processing in real time. In particular, the subject technology provides a way to efficiently utilize an optimized FFT in real time. The advantages, and other features of the systems and methods disclosed herein, will become more readily apparent to those having ordinary skill in the art from the following detailed description of certain preferred embodiments taken in conjunction with the drawings which set forth representative embodiments of the present disclosure. Like reference numerals are used herein to denote like parts. Further, words denoting orientation such as “upper”, “lower”, “distal”, and “proximate” are merely used to help describe the location of components with respect to one another. For example, an “upper” surface of a part is merely meant to describe a surface that is separate from the “lower” surface of that same part. No words denoting orientation are used to describe an absolute orientation (i.e. where an “upper” part must always at a higher elevation).

[0043] In addition to providing autocuing track functions in the ATS to cue the FTS narrow field of view camera, the tracker disclosed herein includes a multi-object track file incorporating laser on detection in the FTS, multiple feature trackers within the FTS tracker to create a consistent and robust track associated with the same target, and a target state estimator fusing both FTS and ATS track outputs for accuracy and track consistency. The system also includes a mode selection that turns on HEL resistant tracker features in the ATS, including gate size lock, gain control on system and algorithm processing. An example would be to switch to a correlation only track or centroid track once the target features are obscured from HEL heating effects. The third feature is a battle damage assessment mode that detects when the target kinematics are erroneous from the expected dynamics as well as features that are measured that displace in an unexpected manner. The FTS tracker also runs continuously in the background and re-centers the field of view of the narrow tracking sensor when track is lost. This is also an indicator for battle damage assessment. The system can track various segments of the target and determine damage to the target in part based on a distance between separated segments of the target. Notably, where used herein, the ATS may be a short-wave infrared tracker (SWIR) while the FTS may be a medium wave infrared tracker (MWIR). However, in some cases, the ATS and FTS can generally operate in an EM waveband of optical sensors outside of a SWIR or MWIR.

[0044] The system includes an intra-target tracker that tracks multiple features and dynamics of the target instead of multiple targets to assess battle damage impact, a target state estimator that fuses the wide field and view and narrow field of view track sensor outputs to assess track performance and provide indicators for battle damage assessment, an adaptive learning system that selects the tracker type depending on state of the HEL effects during lase including the ability to switch on and off different track features that are more resistant to HEL effects on the tracker output, and adaptive clutter filtering that removes background for both stationary and moving targets.

[0045] The tracker disclosed herein uses both correlation and centroid tracking algorithms with regression to any single mode if one fails with both modes running simultaneously. The centroid tracker has both clutter and noise filtering to aid target segmentation for recursively updating the reference template for the correlation tracker, preventing correlation tracker drift. The correlation tracker then provides the high-speed offset for laser beam pointing using the recursive image updated by the centroid tracker. The recursive image update adapts for HEL heating effect mitigation, and illumination variations. Automatic aimpoint selection is done based on generalized estimates of target shape from a library of aimpoint locations using a computer.

[0046] The tracker system disclosed herein includes interleaved and parallel centroid, target reference, and high-speed correlation trackers to prevent track drift and provide robustness to track loss from artifacts (clutter and noise) and can default to either mode. The system tracks small, large, and overfilled targets in the field of view. The system utilizes clutter and noise filtering using bandpass and spatial noise adaptive filters, background statistical estimates for adaptive thresholding, position feedback from predicted target dynamics, and dynamically updated spatial target gates. The system addresses operation with zero velocity and moving targets. A recursive target reference is dynamically updated for correlation tracking and aimpoint for target maneuvers and/or change aspects. The system uses automatic gain learning for the target mask reference integration used in correlation to compensate for intensity variations from illumination and HEL heating effects. A line-of-sight manager control optimally blends centroid and correlation track positions, integrates aimpoint offsets, and detects errors in track position. Aimpoint estimation is carried out, morphologically filtering target feature estimates without requiring stored target models of target aspect changes. The target reference tracker keeps the target reference position centered to keep the laser on boresight and minimize residual tracker drifts.

[0047] The tracker disclosed herein also accounts for the HEL heating effects, applying an algorithm which utilizes two stages. The first is a “HEL ON” detector (or a HEL detection module) that detects intensity changes from heating effects and sends a flag to the system and the tracker algorithm that the HEL is ON. Once the HEL is detected as ON (i.e. HEL effects are identified on the target), the HEL ON detector notifies the imaging system and the imaging system can use this information to adjust sensor gains to ensure that saturation effects are minimized by keeping the peak intensity and the target within the range of A/D for as long as possible. The second part of the algorithm then cores out the HEL affected area on the target and uses the non-HEL heated target for tracking including the recursive reference. The peak intensity of the target after filtering while tracked is continuously monitored and the tracker algorithm parameters adaptively update to mitigate any residual heating effects on the tracker.

[0048] Therefore, the subject technology includes a method that integrates (either at a system level and/or an algorithm level) a process and/or algorithm that detects the presence of HEL heating effects and adaptively adjusts the tracker algorithm and system parameters to be resistant to those effects. Additionally, the system disclosed herein provides adaptive spatial filtering techniques that provide the removal of the spatial target heating effects in the target spatial pattern to maintain track without drift and track lock by tracking on target features not affected. The adaptive spatial filtering techniques and temporal detection of the HEL beam integrate together for a complete closed loop HEL heating effects mitigation algorithm solution for tracking. [0049] Overall, by tracking targets using an interleaved and parallel centroid, target reference, and high-speed correlation tracking, the tracking systems described herein can prevent track drift and provide robustness to track loss from artifacts (clutter and noise), and can default to either track mode, tracking small, large, and overfilled targets in the field of view.

[0050] Further, optimized FFT libraries require an a priori computational path, i.e. a plan that contains execution profile, memory layout, and sizes of transforms, so that optimal paths are followed for the particular FFT operation and size. The planning step is process intensive and creating plans in a real time loop is prohibitive. This negative impact increases when FFT sizes change to adapt to FFT correlator needs on a frame-to-frame basis, requiring different plans for each size change. Every plan created also needs to be deconstructed, imposing additional time penalties. The system disclosed herein solves the problem of creating and using optimal 2D FFT plans for dynamical transform sizes with minimal impact in the execution time of the correlator, by using an algorithm that exhaustively searches for optimal sizes, automatically selecting the FFT sizes most appropriate out of the main execution loop.

[0051] To that end, the subject technology includes an algorithm that exhaustively searches prime power factors (PPF) and powers of 2 (P02) mathematically yielding optimal sized FFT for all sizes. The algorithm automatically selects the 2D FFT sizes most appropriate for the 2 dimensions of the input signal that is out of the main execution loop and creates all the necessary data structures for use during real time. The algorithm can also be tuned and optimized for any 2D FFT implementation including hardware, software and firmware. Further, the algorithm can be extended to 3D, 4D and ND where the image or signal representation extends in multiple dimensions. Further, the algorithm for automatically selecting 2D FFTs can also be applied as part of other known processes which employ 2D FFTs, as are known. This algorithm can also be used in other environments where FFTs are applied.

[0052] Referring now to FIG. 1, a target tracking environment is shown. A target 100 is being launched from a target launch area 102. The target 100 can be a military target, such as a mortar, missile, drone or the like, or a non-military projectile. A HEL target tracking system 104 in accordance with the teachings herein seeks to acquire the target 100 during flight, and damage or destroy the target 100 using a HEL beam. The tracking blocks shown herein and described below represent steps of tracking the target 100 during the target’s trajectory.

[0053] The system 104 includes a target cueing system 106 and abeam control system 108 working in concert. The target cue at any given time is the presentation of internal position “truth” as given to the tracker system 104 from external sources. This can be a one-time update event or can be provided continuously to the system 104. Therefore, target cues are used as “truth” input (measurement updates) relied on to build target tracks, as will be discussed in more detail herein. Targets can be cued after a number of features are observed and meet predetermined criteria, including a predetermined target size, target shape, target velocity, signal -to-noise ratio, and amount of time (i.e. time appearing in the images). The target cueing system 106 can utilize an ATS (e.g. a MWIR) tracking system while the beam control system 108 utilizes a FTS (e.g. a SWIR) tracking system, as discussed in more detail below. The target cueing system 106 initially seeks out the target 100, rejecting clutter and classifying the target 100 to initiate the target track with a radar track 110 at block 112. After a radar handover to the beam control system 108, the beam control system 108 begins a low bandwidth gimbal track 114 and detects the target 100 with a wide field of view (WFOV) acquisition and track sensor at 116. The beam control system 108 then pulls in and centers the target at block 118. At block 120, the acquisition and track sensor then determines edges and size features for tracking.

[0054] The beam control system 108 then begins a high bandwidth precision track 122. During the precision track 122, the system 108 uses a narrow FOV track using a fine tracking sensor (FTS). The beam control system 108 classifies the target using FTS image features, kinematics, an acquisition and track signature, and line of sight (LOS) geometry, at block 124.

At block 128, the FTS centers the target, determines a target ID, and selects an aimpoint on the target. Optionally, at block 130, a beacon illuminator (BIL) atmospheric compensations can be applied. In the final block 132, the beam control system 108 can engage the target 100 with a HEL based on the aimpoint determined. Further, FTS aimpoint maintenance can be carried out at block 132, as the HEL engages the target 100, including target jerk detection, FTS shape change detection, TS debris field separation, and compensation for HEL effects on the target 100. The functions of the system 104 to effectively track the target 100 are described in more detail herein. Referring now to FIG. 2, a block diagram showing the architecture of a tracking system 200 in accordance with the subject technology is shown. The tracking system 200 can be used to track a target for engaging with a HEL, in accordance with the system 104 discussed previously. The system 200 includes an ATS track system 202 which initially looks for, and finds the target, similar to the target cueing system 106. Once the target is located in the ATS tracker, the ATS system 202 centers the target at a boresight of the system using a gimbal controller 224. Tracking and closed loop tracking sensor pointing is passed to an FTS, or SWIR, tracker for high speed, precision tracking, over a relatively small field of view, since the general location of the target is already known. Fine sensor pointing is controlled by the Fast Steering Mirror (FSM) controller 246. At a high-level inputs are provided to the FSM controller 246, which then manages the line of sight at high rates (tens of kilo Hertz). The FSM controller 246 provides target angle positions through the high-speed FTS controller. The system 200 can include a mission processor unit 248 which includes a processing designed to control the actions of the system 200 based on a current operation objective (e.g. target elimination) and other beam director states and modes needed for a particular engagements. The MPU 248 can include a processor and memory configured to execute instructions to carry out the processing tasks of the ATS system 202 and FTS system 204 described herein.

[0055] In particular, the ATS system 202 cues the target using a detector to generate an image filtered based on various criteria. The detector can be a Robinson filter, or another other target filter or machine learned target shape detector. Detections in the image are screened based on expected size, cue gating, and persistence in time until enough confidence is established to start a track on the target like object. Potential targets can also be screened based on correlation to cue expectations given kinematic features.

[0056] To that end, the ATS system 202 generates a ATS video 208, which can be between 60-120 Hz. The ATS system 202 can process the video 208 with a spatial filter 206, a moving target indicator (MTI) clutter suppression module 212, and a low SNR target integration module 214. The spatial filter 206 is designed to apply adaptive spatial filtering to remove spatial scene clutter that affects the ability to detect the target. This allows the ATS system 202 to maintain a target track without drift and track lock by tracking target features not affected. The spatial filter 206 integrates up the signal to boost SNR as necessary to detect low SNR targets. A number of detections in the image are identified at block 216, and selectively cued as the target based on an association with expected target parameters such as position and velocity of the detection compared to expected position and velocity of targets of interest, at block 218. To identify targets from background, a default size for a target can be set, with the Robinson detector rejecting detections that are too small, for example, or other spatial filter or machine learned target detectors. The ATS system 202 can include a Kalman filter 220 to filter the image position estimates of the target position for a smoothed out tracked target location in the image. After a target is identified and determined to meet expected criteria, by the ATS system 202, the target track can be initialized and the gimbal is commanded to close loop track around the ATS system 202 target track. The tracked target location in the ATS system 202 is used as a cue to the FTS system 204 to begin high speed fine target tracking.

[0057] Referring now to FIG. 6, a flowchart 600 shows the functions of the ATS system 202, from target detection 602 to selection of most likely target for engaging with the HEL 612. The ATS system 202 screens detections for target like objects over a predefined interval of time during the track acquisition stage, so that target tracks can be initialized. At block 602, a number of detections are identified in a current image. At block 604 the detections in block 602 are assembled in time together using a consistency metric to track the position of the target in the image. Each detection in block 602 is then compared to a previous detection and the most likely match is used to update the tracked object positions in block 606. Block 606 is used to drop unwanted target tracks, including target tracks that are false from clutter or of targets that disappear from the field of view. Block 610 updates the kinematic parameters of the target tracks (position, velocity, and time) with estimation filtering with the Kalman filter.

[0058] The ATS system 202 can use Kalman-J filter with a kinematic dynamic model in pixel space for time and measurement updates. At block 604, time updates are provided for existing tracks in accordance with:

[0059] In the equations herein, “F” is a state transition matrix and “x” is a 6 state estimate of the track. P_k|k is Joseph’s form for numerical stability. [0060] At block 606, the algorithm prunes target tracks when information is outdated. New detections are associated with existing current tracks at block 608. Detections can be matched with tracks based on a feature distance measured between all pairs (not necessarily

Euclidean distance. For detections

to z_k^ the distance between all pairs can be measured in accordance with:

[0061] A measurement update is run at block 610. The measurement update can be carried out in accordance with:

[0062] Finally, at block 612, the most likely track for a target is selected by comparing target tracks to the target cue, with targets being selected based on proximity to the cue. Mahalanobis distance is used to find the number of standard deviations from the centroid of a track to the centroid of a cue, in accordance with:

D (x cue ^x track) H (X, cue ^x track) Eq. 9

[0063] The target track that minimizes the distance metric D is selected for a given cue. When a target track has sufficient likelihood of being associated with a target cue (based on the strength of correlation to a cue), the detection is declared a target. If no target is declared, the system defaults to a cue track, where targets are still be acquired and no target has been declared. If there is no cue, target confirmation is insufficient due to missing measurements, and the last known target track is propagated until measurements are available. [0064] With the ATS system 202, detections are provided with azimuth in elevation coordinates in pixel space relative to a focal plan array of the imaging system, in accordance with: q_a°^άn = (x — detector size column) * iFov Eq. 10

@e °^dy = (y — detector size row) * iFov Eq. 11

[0065] Using navigation data, the tracker can track in a coordinate frame such as ECEF (earth centered, earth fixed). The unit line-of-sight can be rotated to ECEF in accordance with:

[0066] Pixel coordinates can be replaced with Earth-Centered Earth-Fixed (ECEF) angles, as follows, and passed to the FTS system 204:

[0067] Referring again to FIG. 2, the FTS system 204 disclosed herein is a high-speed target tracker responsible for steering a high-speed mirror to the target aimpoint while the target is maneuvering and/or in apparent line of sight motion due to atmospheric and platform jitter. The FSM controller 246 directs the high-speed mirror commanded by the FTS system 204 which guides the HEL to provide lethal effects on the target, similar to beam control system 108. The high-speed FTS system 204 shown provides modes to acquire the target from a cue or operator designated image location, engage in track once acquired, automatically determine an aimpoint (e.g. block 238), and then provide an offset 244 for a the laser beam location around the automatically determined aimpoint through a line of sight (LOS) pointing manager utilized by the FSM controller 246 for pointing to the target.

[0068] The tracking architecture for the combined ATS and FTS tracking subsystem is broken down in detail in figure 2. Block 226 shows the high-speed FTS tracking flow of processing from the FTS image to generating a command to the FSM for pointing the HEL beam. The functions within the figure 2. are detailed in figure 3.

[0069] The tracking modes of the FTS system 204 are broken down in detail in FIG 3. In brief overview, the FTS system 204 generate a video feed of captured images which can be between 400-800 Hz, for example. The FTS system 204 uses correlation tracking algorithms (path 231) and centroid tracking algorithms (path 233) with regression to any single mode if one fails, with both modes normally running simultaneously. For example, the system 200 may switch to a correlation only track or centroid track once the target features are obscured from HEL heating effects. Actions on the centroid tracking path 233 are described herein as carried out by a centroid tracker, while actions on the lower path are described as carried out by a correlator tracker. The centroid tracker path 233 filters the captured image with clutter filter 228. The centroid tracker path 233 can make further changes using a HEL adjustment module 230 when HEL effects are detected on the target. This is advantageous, as the heat from the HEL causes the target to heat up significantly after it is engaged, which causes high intensity outputs and large illumination variations in the detected images, making it difficult to continue tracking the target. Making adjustments to account for the HEL effects allows the HEL to effectively engage the target until the target is eliminated.

[0070] There are a number of circumstances that represent possible failures which can be observed, resulting in regression to a single tracker mode. Failure modes in the correlation tracker 300 (discussed in more detail below) that can occur are correlation drift due to target signal corruption due to noise and signature variations. In this case, the centroid tracker will properly threshold the target and segment the target from the background providing an absolute position reference on the target location. Failure modes in the centroid tracker 302 (discussed in more detail below) include missed target detections in the clutter filtering and target detection blocks in block 236 in figure 2. In this case the correlation tracker will continue to correlate against the previously stored target signature reference enabling continued track updates. Failure mode examples include: (1) HEL effect providing positive feedback to Tracker therefore Tracker tracks HEL effect instead of target (2) High speed correlator decorrelates, therefore actual track drifts away from target (3) Centroid aimpoint tracking becomes highly unstable therefore track is broken.

[0071] Within HEL adjustment module 230, the detector determines whether the target is experiencing HEL effects at block 232, based on increased intensity and illumination variations within an image. If the target is experiencing HEL effects, the HEL ON detector (block 232) triggers the system 200 to make further modifications to account for the HEL effects, within HEL ON module 234. In block 236, the integrated target signature used for correlation is detected with a threshold and segmented. The detected integrated target signature is used by the aimpoint algorithms in block 238 for determining the location to point the HEL beam for lethality. In particular, once the HEL ON module 234 is activated, the imaging system of the system 200 can adjust sensor gains to ensure saturation effects are minimized by keeping the peak intensity and the target within the range of A/D for as long as possible. At block 240, a centroid estimate is provided to provide an absolute reference of the target location in the image which is used for target aimpoint and integrating the recursive image used by the high-speed correlator. The recursive reference image can be relied on in subsequent tracking and is used as a reference template by the correlator 229 as well as the template in block 236 for target aimpoint determination. Further processing can be done, at block 242, to eliminate unwanted background noise, clutter, jitter, or the like, using an Alpha-Beta filter, Kalman Filter, or other filter. The updated recursive reference image used for the aimpoint provides a high-speed offset for the HEL, at block 244, which can be relied on by the FSM controller 246 for offset pointing a HEL to the target aimpoint added to frame-to-frame shifts provided by the centroid and correlation trackers to engage the target.

[0072] A target state estimator 222 can fuse data from the FTS system 204 (e.g. from correlator tracker and centroid tracker) and from the ATS system 202 to ensure accuracy and consistency. The system 200 can then rely on the fused data for subsequent tracks. The system 200 continuously tracks the targets and dynamically updates the recursive target reference for correlation tracking and aimpoint as described above, ensuring the system 200 updates for target maneuvers and/or changes.

[0073] The combination of the ATS system 202 and FTS system 204 allows for multiple targets to be cued, simultaneously, and allows for a split object track to be opened up with multiple tracks associated with the same target. This allows separate pieces of a target to be tracked even as the target is engaged by a HEL and destroyed, causing the target to break into multiple pieces. The system 200 can assess battle damage to the target when the kinematics of the target are erroneous from expected dynamics, and/or when features of a target are displaced in an unexpected manner. The system 200 can then provide an indicator of battle damage based on the battle damage assessment.

[0074] Notably, while parts of the system 200 are shown and described herein as functional blocks or modules, it should be understood that the functions disclosed therein can be carried out using a specifically configured system designed to carry out the functions described.

[0075] Referring now to FIG. 3, the functions of the FTS system 204 are shown and described in more detail. The tracking functions of the FTS system 204 are further broken into high speed and low speed target tracking paths 300, 302. These paths 300, 302 generally correspond to the FTS paths 231, 233, respectively. The high-speed tracker path 300 provides target position updates at the current frame rate to the line of sight (LOS) pointing manager 314. The high-speed tracker path 300 is the main path to provide LOS updates to a high-speed mirror directed by FSM controller 246 when a target is being tracked. The low-speed tracker path 302 provides a periodic update to the target aimpoint, temporally integrates an image centered target reference for correlation for the high-speed path, and detects, segments and updates and initializes the target tracking gates.

[0076] Exemplary target tracking gates can be seen in FIG. 4. Referring now to FIG. 4, an image 400 of a target 406 obtained by the FTS system 204 is shown. The image 400 includes two concentric areas 402, 404 representing target gates. The first is a target area 402, which includes a buffer around the estimated extent of the target size. The second is a background estimate area 404, which is an annulus around the target area 402 used to estimate the background. The background estimation is determined during the low-speed track path 302 of FIG. 3. Tracker gates 402, 404 are one way in which the tracking system 200 effectively isolates the area where the target 406 is contained in the image 400 to maximize detection, provide efficient location to search in the image space, and reject background contributions. The tracker gates 402, 404 continuously update size and location in both the high-speed and low speed interleaved tracker paths 300, 302. When no cue information is initially present, the tracker gates 402, 404 can initially be sized to include the entire image the entire image 400. Alternatively, the tracker gates 402, 404 can initially be sized from target location and/or size information if present from the ATS tracker 202 or other system functions.

[0077] The FTS tracker 204 also uses feature state estimators for tracking and centering of the HEL on image feature points. To that end, the FTS tracker 204 includes a number of state estimators. In the truest sense, ‘state’ implies velocity, acceleration, and position of the feature tracked. Since these features are in image coordinates, these feature state estimators provide target feature states in image coordinates and not in inertial coordinates. The target feature states estimators include the following, as will be discussed in more detail below: a correlation tracker module 308, a centroid tracker 324, an aimpoint tracker 312, and a recursive reference image tracker 326.

[0078] Referring again to FIG. 3, the low-speed tracker 302 is illustrated in the lower path, while the high-speed tracker 300 is illustrated in the upper path, but both can be part of the FTS tracker 204. After a target is initially acquired, subsequent tracking can be carried out with both the high and low speed trackers 300, 302. Generally, the components of path 300 can serve the role of the correlation tracker, while the components of path 302 can serve the role of the centroid tracker. An image interleave 330 can be used between the two trackers 300, 302. The image interleave is a mode where the low-speed tracker 302 updates its track position, centroid, and target reference every Nth frame, where N is a variable set by the tracker 302 prior to operation. The output of the tracker 302 of the target position is normally the high-speed tracker 300 correlation output for the subsequent N-l frames, and Nth frame is either to centroid tracker location, correlation tracker target estimated location, or a blended combination of the two tracker target locations. [0079] In the lower speed tracker 302, the image is spatially filtered at block 316. The filter image block 316 removes background and noise and enhances the target signal for further processing. The filter image block 316 operates on each frame of the image and represents the first part in the processing chain. During this step, background statistics in the area around the target, not including the target, are determined to set pixel detection thresholds. Next, the signal is processed by an estimate background function 318. The estimate background process computes the statistics in an annulus around the target location in the case of track, or in a larger boundary in the case of acquisition (i.e. as targets are still being identified).

[0080] Next, the tracker executes a threshold and segment target function 320. The threshold and segmentation function 320 takes as input each of the images processed in the low- speed centroid tracker 324, applies a threshold based on the estimate background function 318, and then detects pixels that are greater than the threshold (both positive and negative). The pixels exceeding the threshold are grouped using 8-way connectivity and then transformed into objects along with their statistics such as size, area, centroid, among others.

[0081] Next, a find and select target function 322 is performed. The find and select target function 322 has two modes of operation; a target acquisition mode as targets are identified and a track mode to track identified targets. In target acquisition, the function 322 takes as input the segmented objects from the threshold and segment target’s function 320 and selects the segmented object closest to the acquisition criteria. In track, the find and select target function 322 selects the segmented object closest to the estimated track position. The find and select target function 322 also computes the extent of the target location and its position for estimating the target and background (or constant false alarm CFAR) gates (e.g. 402, 402).

[0082] Further processing is then carried out by the centroid target tracker 324. The centroid target tracker 324 implements a function which computes the centroid for each target selected in the find and select target function 322 and filters the position to develop rate and position estimates of its current position, and future position on the next time update. Track filtering has a variety of track filters implementation options including Kalman filters, alpha- beta, and other known filters. [0083] An update recursive reference function 326 is then applied to shift the segmented and selected target object into a reference image coordinate frame used for track point and closed loop line-of-sight pointing. Each segmented object is integrated into the reference coordinate frame to enhance signal to noise ratio. This recursive reference picture supports two functions. First, the recursive reference picture is the reference image used to estimate each input image translation in the correlation tracker module 308 in the high-speed tracker (i.e. path 300) for closed loop tracking about the track reference point. Second, a target aimpoint 312 is estimated. Input frames are temporally weighted and designed to match target shape changes due to motion and dynamics to ensure close alignment between the reference and the current input frame, while maximizing the reference image signal to noise ratio. The ‘reference image track state estimator’ is embedded in block 326 keeps track of the target centric center location for pointing in the line of sight pointing manager 314.

[0084] A propagate gates function 328 then updates the gated area from which image pixels are grabbed to compute track points and target features on the next frame based on the current segmented target size and the centroid tracker predicted target position on the next frame. The gates define the target area (e.g. gate 402) and a concentric area around the target (e.g. gate 404) for background estimation in the estimate background function 318 on the next frame, when in track.

[0085] An aimpoint estimate function 312 then uses the integrated target reference image in the update recursive reference function 326 to compute the target aimpoint using computer vision-based techniques to estimate shape, rotation axis, and orientation, or any other computer vision or machine learned technique. The aimpoint is determined about the rotation axis with a predefined location on the rotation axis, or user adjustable location, or user adjustable added to the predefined location. The aimpoint estimate 312 provides the LOS pointing manager 314 an offset pointing location or lase location to the aimpoint, and a shift to the center of the recursive reference for offset track. The aimpoint location is filtered and tracked similarly as the centroid target tracker 324 as discussed previously.

[0086] The high-speed tracker 300 functions simultaneously with the low-speed tracker 302. This high-speed tracker 300 is the main tracking function used for instantaneous pointing of the LOS of the FTS system 204. In the high-speed tracker 300 path, the input image is initially filtered at block 304, which functions similarly to the filter image block 316 described above (e.g. operating on each frame and removing background and noise to enhance the target signal).

[0087] A correlate against reference function 306 is applied to compute the current image shift relative to the reference image estimated in the update recursive image function 326 in the low-speed tracker 302 path. The correlated area is adapted to the size of the target gates computed in the find and select target function 322. The correlate against reference function determines the cross-correlation between the reference image and the current input image using an adaptively size 2D FFT and 2D Inverse FFT, sized to the current gated target area, to perform the cross correlation. This process of adaptively sizing the FFTs is shown in FIG. 5 and discussed in more detail below. The current image shift relative to the recursive image is the detected peak location of the correlation output and interpolated to a subpixel location with a 2D peak interpolation method. The correlate against reference function 306 includes a pixel threshold that clips the gated image intensities and the reference image intensities to create a semi-binary image. The semi-binary image protects against intensity variations across the target shape while retaining the key target shape attributes for cross correlation. Pre-determined pixels are searched from the current image.

[0088] A shift is then carried out by a correlation tracker function 308. The correlation tracker function 308 one of the target state estimates discussed previously. The correlation tracker function 308 uses the output of the image shift between the current frame and the reference frame.

[0089] When correlation mode is operational, the correlation gated area size position updates, at update track gates block 310. The size of the correlation area updates with the interleaved centroid tracker find and select target function 322 discussed previously.

[0090] The LOS pointing manager 314 is responsible for providing closed loop line of sight target track location to the angle controller of the system for pointing. The FSM controller 246 can respond to the pointing location. When the tracking system 200 is in acquisition mode, the centroid location of the detected target in the find and select target function 322 points the line of sight. In track mode, when the correlator is operational, the correlator image shifts relative to the reference image centered location. When the interleaved centroid mode is operational, the centroid location (e.g. centroid target tracker 324) points the line of sight as one option or blended with the correlation tracker or correlation tracker only output. The LOS pointing manager 314 also checks for irregular motion of the pointing and limits spurious pointing commands and adds the aimpoint estimate for pointing offset computed in block 312.

[0091] Referring again to FIG. 4, the image 400 of the target 406 includes a correlation track 408, the track state estimates smoothed target position 410, and the automatically generated aimpoint 412.

[0092] Referring now to FIG. 5, the system 200 utilizes an algorithm 500 that exhaustively searches prime power factors (PPF) and powers of 2 (P02), mathematically yielding optimal FFT size. The algorithm 500 builds an optimal selector that choses the FFT sizes most appropriate for the dimensions of the input signal. It does this outside of the main execution loop, preparing all the necessary data structures for real time FFT and iFFT transforms. While the system 200 utilizes 2D FFTs, in some instances, the algorithm can also be carried out for 3D (or higher dimension) FFTs. The algorithm 500 can be implemented by one or more modules as part of the tracking system, for example, within a computer-readable medium on which instructions are encoded for carrying out the algorithm 500.

[0093] Traditional FFT library usage relies on library wisdom to generate optimal code for pre-determined sizes. Any size deviation from this selected wisdom will result in library -determined execution instructions and memory accesses in a suboptimal fashion. In general, using mismatched wisdom to dynamic incoming FFT sizes cancel out any pre-determined or manually prescribed solutions. Generating optimal wisdom relies on benchmarking on the target platform. Benchmarking and selection of optimal solution is computationally expensive. Performing this process on the fly for dynamically changing FFT sizes is prohibitive and guarantees that the application will use suboptimal solutions to reduce setup time. Real time execution of FFT-based algorithms depends both on optimally generating the FFT algorithm instructions in the target platform, and also in optimally generating memory access patterns to the internal working or scratch spaces. The algorithm 500 described herein is prescribed for applications burdened with dynamically-changing FFT sizes. [0094] At step 502, the algorithm starts by defining a coverage range for FFT sizes. The coverage range can be user input based on the expected size of signals in both dimensions (minS, maxS). The algorithm 500 then generates a solution space of optimal size within plan creation engine 504. This starts by module 506 sweeping each characteristic dimension of the FFT (scalable from 1 to N dimensions) and searching all prime power factors (PPF) and powers of two (P02) factors of the given dimension range MxN. Within module 508, for each combination of factors, a 2D FFT plan is created which will be execution optimal and memory optimal for those sizes. A look up algorithm converts each set of incoming dimensions into the minimum power factors greater or equal to the dimensions. A look-up table is created, within module 510, that will map every combination of factors into an index for the storage spaces of the FFT and inverse FFT (IFFT) for the combination, which is guaranteed optimal for the algorithms used in the FFT calculation. This generates an exhaustive space for every combination of PPF from each dimension and allows the libraries to utilize deep benchmarking to select optimal instructions and memory spaces for that particular combination. A set of lookup tables will index the location of these data structures, which will allow for immediate selection during the fast loop (i.e. during the dynamically changing FFTs sizes) of the application. Any incoming size covered in the original range will be assigned an optimal FFT solution by the selection process in the algorithm 500.

[0095] The plan created in plan creation engine 504 can then be utilized by the correlator 229 to apply the FFT. In particular, input 512 from the tracking system 200 is provided to the algorithm 500. This includes input from a sensor subsystem 518 of the system 200, incoming 2D sizes from the image acquisition system 516, and correlator dynamic signal sizes 514. The FFT for the correlator 229 can then be executed within high-rate execution loop 520. During the high-rate execution loop 520, the pre-generation of mapping tables and functions guarantees that the selection of the FFT plan will meet real-time requirements, without incurring penalties for in-the- loop plan creation. A real time optimal plan selection module 522 then selects the optimal plan from the plan creation engine 504. An FFT correlator algorithm 524 then optimally executes a 2D FFT (optimal execution module 526) to determine the cross-correlation between the reference image and the current input image using the adaptively sized FFT. The results 528 are utilized by the correlator 229 to perform the cross correlation. A plan creation deconstruction engine 530 provides out-of-the-loop plan deconstruction, ensuring that memory deallocation execution penalties are not incurred inside the high-rate algorithm loop. [0096] Overall, while the algorithm 500 is described above for use within a tracking system as disclosed herein, it should be understood that the algorithm can also be used in other applications where the user needs to ensure maximum execution speed. Therefore, utilizing sub-optimal FFTs, or generating on-the-fly wisdom, is to be avoided. This is particularly pertinent for dynamically changing FFTs. By trading off the additional memory space required to setup the solution space and lookup tables during initialization, the algorithm 500 ensures coverage of the required range of FFTs. If a small set of FFT sizes are needed, the algorithm 500 still ensures optimal solutions for the small set, therefore alleviating the manual burden of defining, benchmarking, management, and selection during execution. Other typical applications relate to Field-Programmable Gate Array (FPGA) designs, where the optimal path can be pre-determined and programmed into the array by the bitstream synthesis process. The runtime part of the algorithm 500 therefore selects from an available set of optimal core array processing elements.

[0097] The algorithm 500 can also be used within other applications within systems executing one-dimensional FFTs, as sometimes seen in audio processing and radar signal processing. In audio processing, the algorithm 500 can be utilized for spectral content extraction of digital audio signals. The algorithm 500 can be implemented as part of processing or filtering of audio signals by applying low-pass, high-pass, band-pass, notch filters, and/or gains, to an incoming signal spectrum. Multiple FFT sizes can be used to vary the quality of the processing, larger sizes will produce higher resolution spectral content allowing finer tuning of the processing results, and the signal reconstruction after filtering.

[0098] The algorithm 500 can also be used as part of a radar signal processing system. FFTs are used to process transmission and reception of multiple direction signals in a radar beamforming application, using electronically steerable arrays. This application assumes a fixed number of points (FFT size). Having multiple number of points (multiple FFT sizes) available could open up the use of the steerable arrays in different ways by partitioning the array in many combinations of processing points, allowing different simultaneous beam shapes in different directions, at different resolutions.

[0099] The algorithm 500 is also usable in higher dimensional applications, for 2D, 3D, or ND (higher than 3D) FFT applications, including medical and general image processing, for example. The algorithm 500 can be used as part of a system for processing and filtering of two- dimensional signals in the form of digital images. The algorithm 500 can be implemented to apply low-pass, high-pass, band-pass, notch filters, and/or gains, edge detectors, or morphological operators. Multiple FFTs sizes can be used to rapidly process different image sizes, or subsets of an original size, at optimally high speeds for each FFT size. This is a similar application domain to use of the algorithm 500 within the tracking system described herein. Further, in medical image processing (and other environments) three-dimensional FFTs are used as an extension of image processing applications to volumetric data sets. The same optimization techniques are applicable therein. The characteristic dimensions used in the PPF search will now include these higher dimensions, and the combinational space and lookup tables reflect these additional combinations.

[00100] Referring now to FIGs. 7a-7f, block diagrams visually show how memory layouts are optimized using the algorithm 500 described herein. In each case, at least one signal 702 (or filter of interest) of dimension N x M is received and processed using an FFT. Filler or “padding” space 704 exists within the memory layout to align the signal 702 to the FFT process buffer. Layout 700a shows filler 704 within the dimensions for a P02 dimension 706 for an FFT. The filler 704 is then reduced for the PPF dimension in PPF optimal combination space 708 in layout 700b. By reducing the filler space 704 in the PPF optimal combination space 708, the FFT algorithm can perform faster by skipping over less “dummy” data

[00101] FIG. 7d shows a memory layout 700d including signals 702 (or filters of interest) of dimensions N x M surround by filler 704 within a P02 dimension 706 for an FFT. The signals 702 are shown centered in layout 700c of FIG. 7c. The traditional FFT shift operation, used to align filters or signals 702 with a centered frequency content, benefits clearly from optimal “tighter” padding spaces (filler 704) around the signal 702 of interest. This results in fewer multiplications by zero elements and less data to move around in memory when executing the filtering (FFT multiplication) or IFFT operations. Similar to the relationship of layout 700b to layout 700a, layout 700e represents the layout 700c after reduction for PPF dimensions in a PPF optimal combination space 708. Similarly, layout 700f represents the layout 700d after reduction for PPF dimensions in a PPF optimal combination space 708. [00102] Referring now to FIG. 8, a block diagram 800 illustrates the functional relationship of portions of the algorithm 500. The algorithm 500 keeps multiple memory scratch spaces 804 and wisdom plans 802 for every combination of PPF in each characteristic dimension. A lookup table 806 provides an index which includes a location within the memory space 804 for each optimal PPF. Selection of an optimal FFT is carried out in block 808. A quick selector algorithm 812 returns which PPF is optimal for a given dimension. An execution index selector 810 return which entry in the lookup tables 806 to use based on the optimal PPF. Based on the index selection, the lookup table 806 can be used to identify the location within the memory space 804 of the best execution wisdom plan 802.

[00103] All orientations and arrangements of the components shown herein are used by way of example only. Further, it will be appreciated by those of ordinary skill in the pertinent art that the functions of several elements may, in alternative embodiments, be carried out by fewer elements or a single element. Similarly, in some embodiments, any functional element may perform fewer, or different, operations than those described with respect to the illustrated embodiment. Also, functional elements shown as distinct for purposes of illustration may be incorporated within other functional elements in a particular implementation.

[00104] While the subject technology has been described with respect to preferred embodiments, those skilled in the art will readily appreciate that various changes and/or modifications can be made to the subject technology without departing from the spirit or scope of the subject technology. For example, each claim may depend from any or all claims in a multiple dependent manner even though such has not been originally claimed.

Claims

WHAT IS CLAIMED IS:

1. A target tracking system for tracking a target comprising: at least one imaging system configured to continuously capture a plurality of images of the target; a centroid tracker configured to run a first tracker mode to generate a plurality of recursive reference images of the target based on the images; a correlator tracker configured to run a second tracker mode to determine a shift between successive images of the target and generate an offset for pointing the target tracking system at the target, wherein the correlator tracker utilizes the recursive references images when the first tracker mode is running; at least one module configured to execute an algorithm to:

(a) define a coverage range for a Fast Fourier Transform (FFT) based on an expected minimum size and an expected maximum size of the images;

(b) define a number of FFT dimensions for the FFT;

(c) sweep each FFT dimension and search prime power factors (PPF) and powers of two (P02) mathematically to yield optimal sized FFTs for all sizes within the coverage range and create a FFT plan optimizing execution instructions and memory for each combination of PPF and P02; and

(d) create an index comprising a look-up table mapping each combination of PPF and P02 to data structures for optimal FFTs, wherein the index includes memory locations for the data structures; wherein, the system is configured to track the target by: running the first tracker mode and the second tracker mode simultaneously; selecting an optimal FFT from the look-up table for the images; and executing the optimal FFT to determine a cross-correlation between the reference image and the current image.

2. The system of claim 1, wherein selecting an optimal FFT is carried out by the correlator tracker as part of the second tracking mode, wherein the second tracking mode is a high-rate execution loop.

3. The system of claim 1, wherein steps (a)-(d) are carried out by the centroid tracker as part of the first tracker mode.

4. The system of claim 1, wherein the data structures include an execution wisdom plan for each optimal FFT.

5. A method comprising:

(a) determining an expected minimum size and an expected maximum size to define a coverage range for a Fast Fourier Transform (FFT);

(b) defining a number of FFT dimensions for the FFT;

(c) sweeping each FFT dimension and searching prime power factors (PPF) and powers of two (P02) mathematically to yield optimal sized FFTs for all sizes within the coverage range and creating a FFT plan optimizing execution instructions and memory for each combination of PPF and P02;

(d) creating an index comprising a look-up table mapping each combination of PPF and P02 to data structures for optimal FFTs, wherein the index includes memory locations for data structures for each optimal FFT; and (e) receiving an incoming signal having a signal size, and if the signal size is within the coverage range, selecting an optimal FFT from the look-up table for the incoming signal based on the signal size.

6. The method of claim 5, wherein, during step (a), the expected minimum size and expected maximum size are determined based on a received input.

7. The method of claim 5, further comprising executing the optimal FFT for processing the incoming signal.

8. The method of claim 5, wherein during step (c), all combinations of PPF and P02 within the coverage range are searched.

9. The method of claim 5, wherein the number of FFT dimensions is two.

10. The method of claim 5, wherein the number of FFT dimensions is at least three.

11. The method of claim 5, wherein the data structures include an execution wisdom plan for each FFT.

12. The method of claim 5, wherein: steps (a)-(d) are performed initially as part of an out of loop process; and step (e) is repeatedly performed as part of a high-rate execution loop as the incoming signal is received, the incoming signal varying over time.

13. The method of claim 12, further comprising:

(f) executing an out of loop deconstruction plan to deconstruct the FFT plans.

14. A system including a computer-readable medium on which instructions are encoded for carrying out a process, the process including:

(a) determining an expected minimum size and an expected maximum size to define a coverage range for a Fast Fourier Transform (FFT); (b) defining a number of FFT dimensions for the FFT;

(d) creating an index comprising a look-up table mapping each combination of PPF and P02 to data structures for optimal FFTs, wherein the index includes memory locations for data structures for each optimal FFT; and

(e) receiving an incoming signal having a signal size, and if the signal size is within the coverage range, selecting an optimal FFT from the look-up table for the incoming signal based on the signal size.

15. The system of claim 14, wherein, during step (a), the expected minimum size and expected maximum size are determined based on a received input.

16. The system of claim 14, wherein the process further includes executing the optimal FFT for processing the incoming signal.

17. The system of claim 14, wherein during step (c), all combinations of PPF and P02 within the coverage range are searched.

18. The system of claim 14, wherein the data structures include an execution wisdom plan for each FFT.

19. The system of claim 14, wherein: steps (a)-(d) are performed initially as part of an out of loop process; and step (e) is repeatedly performed as part of a high-rate execution loop as the incoming signal is received, the incoming signal varying over time. 20 The system of claim 19, wherein the process further includes:

(f) executing an out of loop deconstruction plan to deconstruct the FFT plans.