JP2025530567A - Systems and methods for improving marksmanship through machine learning - Google Patents

Abstract

The physical movement improvement system is configured to receive the video data, track one or more physical landmarks associated with the movement to determine the movement of the physical landmarks during the action, correlate the movement of the physical landmarks with a score for the action, and determine movements that are detrimental to the score for the action using machine learning techniques. The system can also recommend drills to the participant to improve the score for the action.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
This application claims the benefit of U.S. Provisional Patent Application No. 63/406,245, entitled "SYSTEMS AND METHODS FOR MARKSMANSHIP IMPROVEMENT THROUGH MACHINE LEARNING," filed September 13, 2022, and U.S. Provisional Patent Application No. 63/406,208, entitled "SYSTEMS AND METHODS FOR AUTOMATED TARGET IDENTIFICATION, CLASSIFICATION, AND SCORING," filed September 13, 2022, and U.S. Provisional Patent Application No. 63/406,241, entitled "SYSTEMS AND METHODS FOR MARKSMANSHIP DIGITIZING AND ANALYZING," filed September 13, 2022, the contents of which are incorporated herein by reference in their entireties.

Target shooting is enjoyed by millions of people annually, and by many reports, the number of people who regularly practice target shooting has increased over the past decade and continues to grow. In the United States alone, it is estimated that over 52 million people regularly shoot targets. There are many different types of recreational shooting activities, ranging from simple plinking with a handgun or rifle at paper or steel targets, to skilled long-range rifle shooting competitions, which require advanced training and skill, to fun, fast-paced pistol shooting at popping or stationary targets, or shotgun shooting at skeet, trap, sporting clays, etc. Apart from recreational shooting, an increasing number of target shooters practice as part of their professions, such as law enforcement, the military, and security guards.

While many participants are satisfied with their current skill level, there is an increasing number of shooters who want to improve their skills. However, in some cases, many participants do not know how to improve. The number of people at shooting ranges continues to grow, and these participants, whether for sport, recreation, self-defense, or public defense, want to improve their skills. However, without individual shooting instruction, and often even if individual instruction is provided, it can be difficult to quantify improvement and issues affecting accuracy. Furthermore, many participants do not know how to improve.

Therefore, there is a need for a system and method that can analyze participants' habits and provide feedback and recommendations for improvement. Furthermore, there is a need for a system that can provide the aforementioned benefits in near real time using consumer-grade devices. These and other advantages will be readily apparent from the disclosure that follows.

One or more computer systems may be configured to perform specific operations or actions by installing software, firmware, hardware, or a combination thereof on the system that causes the system to perform the actions during operation. One or more computer programs may be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform the actions. One general aspect includes a method for improving shooting performance. The method also includes receiving video data of a shooter, determining one or more body landmarks of the shooter, tracking the one or more body landmarks during a shot to generate shot motion data, determining a score for the shot, associating the shot motion data with the score, and generating recommendations for altering the motion data for subsequent shots. Other embodiments of this aspect include corresponding computer systems, devices, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

Implementations may include one or more of the following features. In this method, determining one or more body landmarks of the shooter may include generating a wireframe model by connecting the body landmarks. Associating the shot motion data with the score may include executing a classification and regression tree machine learning model to identify a causal relationship between the shot motion data and the score. The method may include determining the shooter's grip through image analysis of the shooter's video data. The method may include analyzing the shooter's grip and providing, on a display screen, a grip recommendation for changing the grip. Determining the score of the shot may include receiving target video data, performing image analysis on the received target video data, determining a hit on the target, and determining a score for the hit. Receiving video data includes capturing the video data with a mobile phone. Determining one or more body landmarks includes determining 17 body landmarks. Tracking the one or more body landmarks includes generating a bounding box around each of the one or more body landmarks. The method may include executing a machine learning model to correlate the shot motion data with the score. The machine learning model is configured to determine motion data that results in off-center hits on the target. Implementations of the described techniques may include hardware, methods or processes, or computer software on a computer-accessible medium.

One general aspect includes a method for improving the causal outcome of a physical movement. The method also includes receiving video data of the physical movement, determining one or more body landmarks visible in the video data of the physical movement, tracking the one or more body landmarks during the action, generating motion data based at least in part on tracking the one or more body landmarks, determining a score associated with the motion data, associating the motion data with the score, and generating a recommendation for altering the motion data in a subsequent action. Other embodiments of this aspect include corresponding computer systems, apparatuses, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the method.

Implementations may include one or more of the following features. In the method, receiving video data includes capturing the video data with a mobile computing device. The method may include executing a machine learning model to correlate the behavioral data with a score. The machine learning model is configured to determine behavioral data that results in a reduced score. The method may include predicting, by the machine learning model, a predicted score based on the behavioral data. The method may include comparing the predicted score to the score. Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-accessible medium.

The accompanying drawings are part of this disclosure and are incorporated herein. The drawings illustrate examples of embodiments of the present disclosure and, together with the description and claims, serve to explain, at least in part, various principles, features, or aspects of the present disclosure. Specific embodiments of the present disclosure are described more fully below with reference to the accompanying drawings. However, various aspects of the present disclosure may be implemented in many different forms and should not be construed as limited to the implementations set forth herein. Like numbers refer to similar, but not necessarily the same or identical, elements throughout.
1 shows an image of a shooter captured by an image capture device, according to some embodiments. 1B illustrates a wireframe model of the shooter of FIG. 1A, according to some embodiments. 1 illustrates motion data associated with multiple body landmarks, according to some embodiments. 1 illustrates motion data associated with multiple body landmarks and automatic shot detection based on the motion data, according to some embodiments. 1 illustrates analyzing motion data associated with multiple body landmarks and identifying shooting errors based on the motion data, according to some embodiments. 1 illustrates analyzing motion data associated with multiple body landmarks and identifying shooting errors based on the motion data, according to some embodiments. 1 illustrates analysis of motion data associated with multiple body landmarks according to some embodiments. 1 illustrates the analysis of motion data associated with multiple body landmarks and identifying pose changes that result in the least improvement in performance, according to some embodiments. 1 shows an image of a shooter captured with an imaging device, according to some embodiments. 8B illustrates a computer-generated wireframe model of the shooter of FIG. 8A with identified body landmarks, according to some embodiments. 1 illustrates motion data associated with multiple body landmarks and identifying physical motions that lead to poor performance, according to some embodiments. 10A and 10B show a diagnostic target that identifies shooter behavior based on shot patterns, according to some embodiments. 1 illustrates an image of a shooter's hands and a computer-generated wireframe model associated with body landmarks that identify and analyze the shooter's grip, according to some embodiments. FIG. 1 is a diagram of a computer software program user interface for capturing video images, tracking body landmark movements, and analyzing shooter pose and motion data, according to some embodiments. FIG. 1 is a diagram of a computer software program user interface for analyzing a shooter's performance and automatically scoring the performance, according to some embodiments. 1 illustrates a system for digitizing and analyzing marksmanship, according to some embodiments. 1 illustrates a system for digitizing and analyzing marksmanship, according to some embodiments. 1 illustrates a system for digitizing and analyzing marksmanship, according to some embodiments. 1 illustrates a system for digitizing and analyzing marksmanship, according to some embodiments. 1 illustrates the logic of a machine learning algorithm for quantifying shot samples, according to some embodiments. 1 illustrates a sample decision tree machine learning model that correlates body landmark motion data with shot performance, according to some embodiments. 1 illustrates an annotated decision tree machine learning model that correlates body landmark motion data with shot performance, according to some embodiments. 1 illustrates a pruned decision tree machine learning model that correlates body landmark motion data with shot performance, according to some embodiments. 1 illustrates a system for capturing video data of a shooter and a target, according to some embodiments. 10 illustrates image data of a left side view of a shooter captured by an imaging device, according to some embodiments. 10 illustrates image data of a right side view of a shooter captured by an imaging device, according to some embodiments. 1 illustrates image data of an overhead view of a shooter captured by an imaging device, according to some embodiments. 1 illustrates image data of a target captured by an imaging device, according to some embodiments. 1 illustrates a sample user interface of a computer application configured to receive, analyze, and score marksmanship performance, according to some embodiments. 1 illustrates a sample user interface of a computer application configured to receive, analyze, and score marksmanship performance, showing shot placement and automatic scoring, according to some embodiments. 1 illustrates a sample user interface of a computer application configured to receive, analyze, and score a shooting skill performance, allowing selection of a body landmark and showing motion data associated with the selected body landmark, according to some embodiments. Some of the key technologies enabled by the embodiments described herein are presented along with the features they facilitate. 10A-10C illustrate sample user interfaces of a computer application configured to receive, analyze, and score shooting skill performance, illustrating a system that uses multiple source collaboration and machine learning to detect fired shots, analyze shooting position and provide recommendations for improvement, and analyze grip position and provide recommendations for improvement, according to some embodiments. 1 is a process flow for capturing operational data, analyzing the operational data, and generating recommendations for improvement, according to some embodiments. 1 is a process flow for correlating firearms with individual shooters, according to some embodiments. 1 illustrates a sample system for determining an acoustic signature, according to some embodiments. 1 shows a sample flowchart for pre-training a DNN, according to some embodiments. 1 shows a sample flowchart for deriving spectrogram weights according to some embodiments. 1 shows a sample flowchart for classifying and determining an acoustic signature. 1 illustrates a system configured for automated scoring of shooting targets, according to some embodiments. 1 shows a sample process flow for classifying and scoring targets, according to some embodiments. 1 illustrates a sample process flow for identifying and classifying targets, according to some embodiments. 1 illustrates a sample process flow for registering targets and determining scoring hits, according to some embodiments. 35A, 35B, and 35C show a method for initializing a target scoring system to identify a target, according to some embodiments. 1 illustrates a sample process flow for detecting impact on a target, according to some embodiments. 1 illustrates a sample process flow for scoring target impacts, according to some embodiments. 1 illustrates a sample user interface for automated target scoring in a software application, according to some embodiments.

Body landmarks and pose estimation

According to some embodiments, a system is described that uses computer vision and machine learning to significantly reduce the manual and labor-intensive nature of current training techniques, thus providing a system that continuously learns and adapts. According to some embodiments, the system includes a machine vision and machine learning system that can track and estimate a participant's pose and determine positive and negative factors that affect the quality of participation, such as pose, movement, anticipation, recoil, grip, and stance, among others. This may be primarily performed by a computer vision system that can simultaneously track several body landmarks and, in some cases, correlate the movement of one or more body landmarks with the accuracy of the shooting skill. By way of example, the system may identify and track any number of body landmarks, such as three, five, eleven, seventeen, twenty-one, twenty-five, thirty, or more body landmarks. Because the system tracks the location of landmarks, which may be in two or three dimensions, the system can correlate the movement of the landmarks with bullets fired downrange and the scoring of individual bullets. Detection of a bullet fired downrange may be determined by the movement of one or more suitable markers, such as the movement of a participant's hand or wrist (or other body marker) in response to recoil, the sound of a gunshot, a pressure wave associated with a muzzle blast, a target hit, or some other marker.

The system may further monitor the participant from one, two, three, or more viewpoints, analyze the movement of each body landmark, and further monitor the accuracy of the shot and correlate the movement of the body landmark with the accuracy. Based on the accuracy, the system may further provide an analysis of the body movements that contribute to the less-than-perfect accuracy and may further suggest ways to improve the body movements to increase accuracy.

Motion capture may be performed generally with one or more cameras pointed at the participant and one or more cameras pointed at a target. In some cases, one or more of the cameras are associated with mobile computing devices such as smartphones, tablets, laptops, personal digital assistants, and wearable devices (e.g., watches, glasses, body cams, smart hats, etc.). In some cases, the wearable devices may include sensors such as accelerometers, vibration sensors, motion sensors, or other sensors to provide motion data to the system. In some embodiments, the system tracks body marker positions over time and generates a motion plot.

1A , one or more cameras may capture one or more views of a shooter 100. The cameras may capture video data of the shooter as he or she draws, aims, fires, reloads, and/or adjusts position. A computer system may receive the video data, analyze the video data, and create a model associated with the shooter, such as that in FIG. 1B . In some cases, the computer system may identify body landmarks and connect them to a wireframe model 102 that tracks the shooter's pose and movement of the body landmarks. In some instances, the body landmarks may include one or more of the nose, left ear, left eye, left hip, left knee 103, right ear, right eye, right hip, left ankle, left elbow, left wrist, right knee 104, right ankle, right elbow 106, right wrist 108, left shoulder, and right shoulder 110. Of course, other body landmarks are possible, but for efficiency, we will focus on these 17 body landmarks throughout this disclosure. In some embodiments, a single camera may capture two-dimensional motion data associated with one or more of the body landmarks. In some examples, two or more cameras may be used to capture three-dimensional motion data of one or more of the body landmarks.

Body landmarks may be tracked over time, such as during a shooting string (e.g., a shooting session), and the movement of one or more of the body landmarks may be tracked during this time. In some cases, two-dimensional movement is tracked in the x and y directions, corresponding to side-to-side and vertical movement. In some cases, three-dimensional movement of a body landmark is tracked in the x, y, and z directions.

Referring to FIG. 2, a graph of selected body landmarks 200 is shown. The body landmarks are user-selectable by the user to focus on individual body landmarks or combinations of body landmarks for review. For example, the top line 202 represents the right wrist of a right-handed shooter in the horizontal direction, and the third line 203 represents the vertical direction of the right wrist over time. As can be seen, wrist movement moves up and down during the motion capture process. In some cases, the system may correlate the movement of one or more body landmarks with events or stages during the shooting string.

For example, during the first stage 204, the right wrist is in a relatively low position, and the system may correlate this position and movement with a state of readiness to launch. The second stage 206, showing the right wrist moving upward in a very short interval, may correlate with the act of drawing a pistol from a holster. The third stage 208 shows the right wrist remaining fairly stable in the vertical plane, but with sharp peaks 207a-207c in movement, which may correlate with a shot being fired from the pistol.

During the fourth stage 210, the right wrist moves downward and returns to the firing position. The system may correlate this movement with a reloading motion. In some cases, the system is trained with training data, which may be supervised learning, to correlate similar movements with various stages.

During the fifth stage 212, the shooter's right wrist initially moves upward as the shooter takes aim, then settles on the target, followed by peaks of movement 213a-213c, which may correlate with the shot being fired downrange.

In the sixth stage 214, the right wrist again moves downward to its initial position, which may correlate with the act of holstering the pistol.

While this example focuses on the shooter's right wrist in the vertical direction, it is clear that any body landmark can be viewed and analyzed, and movements or combinations of movements can be correlated with events or actions by the shooter that involve groups of body landmarks.

Referring to FIG. 3, a close-up view of the shooting stage 300 is depicted, showing sharp peaks in the vertical movement of the right wrist 302 and the right elbow 304. The system can analyze the motion data and automatically determine when a shot was fired. The system can be configured to correlate sharp peaks in the vertical movement of the wrist and/or elbow with fired shots. As shown in FIG. 3, each of the arrows 306 may correspond to a fired shot. Additionally, audio data can be correlated with the motion data to provide additional cues as to when a shot was fired. In some cases, audio data can be combined and/or synchronized with the motion data to provide additional detail about the fired shot. The motion data can be used to infer additional information about the shooter, the shooter's habits, the shooter's posture, and other cues that may catch the shooter's attention as part of an effort to improve the shooter's accuracy.

Referring to FIG. 4 , motion data 400 is displayed for a shooter's right wrist 402 and right elbow 404. In some cases, the system may apply a trend curve 406 to the motion data, which may represent normalized motion data. Additionally, the system may make inferences and/or decisions based on the motion data 400. For example, as shown in FIG. 4 , at 408, a shooter aims a firearm at a target and attempts to hold the firearm steady; at 410, the shooter lowers their wrist and elbow, and shortly thereafter, a shot is fired 412. The system may recognize this pattern and determine that the lowering of the wrist and/or elbow immediately prior to the shot is evidence that the shooter is anticipating and preparing for the recoil of the firearm. In many cases, the shooter moves the firearm away from the target in anticipation of the recoil that will occur when the shot is fired, dramatically reducing the accuracy of the shot. Similar examples of behavioral data that may reduce a shooter's accuracy include flinching, pre-ignition pushing, trigger jerking, and closing the eyes, among others.

In some cases, the system may provide the shooter with information regarding recoil anticipation, information that may include one or more drills or practice sessions as part of an effort to improve the shooter's performance data regarding recoil anticipation. For example, the system may identify practice regimens that may include dry firing, skipping reloads (e.g., mixing live ammunition with a magazine of dummy ammunition), or other skill-building drills.

Figure 4 also shows drift in the shooter's posture. For example, before the first shot 412, the shooter's right wrist and right arm are at a first vertical height 414, and before the second shot 416, the shooter's right wrist and right elbow are at a second vertical height 418, higher than the first vertical height. This data indicates that the shooter did not return to the same position between the first and second shots, resulting in a slightly different sight picture for the shooter, which may reduce the accuracy of subsequent shots.

Referring to FIG. 5, motion data 500 further illustrates drift. Motion data associated with a right-handed shooter's right wrist 502 and right elbow 504 not only shows recoil anticipation, indicating a downward movement of body parts immediately prior to the shot, but also shows trend lines 506 demonstrating a continuous upward drift of the shooter's wrist and elbow during the firing string. If the shooter does not return to the same position between shots, this can dramatically reduce the accuracy and precision of shots fired within the string.

In some cases, the system may recognize drift in one or more of the body landmarks and provide this information to the shooter. In some cases, the system provides information on a display screen associated with a mobile computing device. For example, the system may be implemented on a mobile computing device associated with the shooter, and the display screen on the mobile computing device may provide information, instructions, or practice drills to the shooter to improve drift and resulting accuracy issues.

Similarly, the system may correlate the movement of body landmarks with other correctable imperfections in the shooter's position or posture. For example, referring to FIGS. 6 and 7, which illustrate body landmark motion data 600, the motion data 600 illustrates the movement of multiple body landmarks during a shooting string. The bottom line graph illustrates right ankle motion data 602, demonstrating that the shooter changed the position of their right foot. Changing position during a shooting string likely impacts the sight picture, accuracy, precision, and other metrics associated with shooting. The system may determine whether the change in foot position was positive or negative with respect to the scoring of the shot and may provide recommendations to the shooter based on this change in posture. The system may look at the shooting accuracy and/or precision of shots 604a-604e fired both before and after the foot repositioning to determine whether moving the foot had a positive or negative impact on shooting performance.

In some cases, the system can correlate target scoring (e.g., shot accuracy and/or precision) with body landmarks and movements to show the shooter which positions of individual body landmarks influenced the shooter's shooting performance, for better or worse.

Through machine learning, the system can be configured to associate specific poses, movements, and combinations with shooting performance. In some cases, body landmark movements and movement combinations may be associated with improved shooting performance, while others may be associated with decreased shooting performance.

The system may normalize the motion data to generate normalized coordinates for each body part's position during every session. The score and its running average may be represented by a signal, for example, by displaying it on a user interface. The motion data and/or score may be analyzed in near real time or stored in a data file that can be saved for later analysis.

Motion data may be associated with shot pattern data, such as the x and y coordinates of each shot, and the shot coordinates may be temporally associated with the motion data occurring at the time the shot was fired. Additionally, a score may be assigned to the shot and stored with the shot data.

One or more machine learning techniques may be applied to the motion and shot data to generate correlations between motion and shot accuracy. For example, in some embodiments, a convolutional deep neural network (CNN) may be designed to locate features in the motion and shot data collection. Other deep learning models for classification purposes may also be used to correlate motion and shot data to identify patterns that lead to improved or decreased accuracy. Transformations may also be used that correlate any set of attributes with any other set of attributes.

FIG. 8A shows a sample camera angle of a shooter 800, and FIG. 8B shows the resulting wireframe model 802 of the shooter, which allows the system to track the shooter's body movements, including selected body landmarks. As can be seen, the system can determine the shooter's pose, stance, and movements throughout the firing string. The wireframe model may include representations of each major joint and body artifact, which may include, among other things, one or more of the shooter's nose or chin 804, right shoulder 806, right elbow 808, right wrist 810, hip 812, left femur 814, left knee 816, left lower leg 818, left ankle 820, right femur 822, right knee 824, right lower leg 826, and right ankle 828. In some cases, the system may be trained with motion data from various shooters and their past performances correlated with the motion data. In this case, the system can determine the impact of specific poses and/or movements on shooting performance. In some cases, the system may rely on performance data from a single shooter to assist that shooter in making adjustments and/or conducting further training exercises to improve the shooter's performance.

Figure 9 shows x-axis motion data 900 associated with the shooter's head 902 and nose 904. As can be seen, the shooter's head moves backward after each shot 906a-906d during the firing string, which may correlate with shooting performance. That is, the system may determine that the shooter is moving their head after each shot, for example to look at the target, and further that they may have drifted during the firing string and not returned to the exact same spot, causing the shooter's performance to fall below a threshold.

In some cases, the system may be configured with logic to determine probable cause and effect based on either the shooter's movements and/or the target's scoring. Referring to Figures 10A and 10B, Figures 10A and 10B show diagnostic targets 1000 for left-handed and right-handed shooters, respectively. Because the right-handed and left-handed targets are mirror images of each other, only one target will be described below.

Assuming the shooter is able to hold the pistol at the target, movements associated with the shooter may be the cause of off-center hits. If the off-center hits are regularly clustered to one side of the target, there are several possible issues that could be causing the off-center hits. For example, if a group of shots lands at the 12:00 position, the system may determine that the shooter is bending their wrist upward 1002 (e.g., riding the recoil), which often occurs in anticipation of the recoil.

If the group of shots lands at the 1:30 mark 1004, this may indicate a grip error known as heeling, where the heel of the hand pushes the pistol butt to the left in anticipation of the shot, which forces the muzzle to the right.

If a group of shots lands at the 3:00 position 1006, this may be an indication that the thumb of the shooting hand is applying too much pressure, pushing the side of the pistol to the right and forcing the muzzle to the right.

If the group of shots lands at the 4:30 position 1008, this may be an indication that the shooter is gripping the gun too tightly while firing (e.g., lobstering). Pulling the trigger will lower the front sight, causing the shot to drift low and to the right for a right-handed shooter.

If a group of shots lands at the 6:00 position 1010, this may be an indication of the wrist bending downward and forward, often in an unconscious effort to control recoil and prevent muzzle lift.

If a group of shots lands at the 7:00 position 1012, this may indicate a trigger jerk or slap. This may indicate the shooter is about to pull the trigger just as the sights are on target.

If a group of shots lands at the 8:00 position 1014, this may be an indication that the shooter is tightening their grip too much during the shot.

If a group of shots lands at the 9:00 position 1016, this may be an indication that the finger is too light on the trigger. This typically causes the shooter to pull the trigger at an angle when finally pulling it back, tending to push the gun to the left.

If a group of shots lands at the 10:30 position 1018, this may be an indication that the shooter is pushing in anticipation of recoil and not following through well.

The system can be programmed to verify target hits during the firing string and, in combination with body landmark movements, determine whether the shooter is engaging in recoil anticipation, trigger control errors, and/or grip errors. The system can make this determination for each individual shooter and recommend practice exercises and practice strings to address their specific shooting issues.

In some cases, the system may have access to DOPE (Data On Previous Engagement) associated with the shooter, which may include previous shooting sessions, records, scores, and analysis.

Complete shooting sessions can be recorded and stored, along with real-time scoring, so users can review them along with decisions from the machine learning system that identify and/or highlight flaws, mistakes, and changes in the shooter's posture or technique that improve or detract from the shooter's performance.

As described herein, synergies arise from synchronizing all available information sources.

Continuing with reference to Figures 10A and 10B, which show diagnostic targets, a basic conventional ML model maps a target shot pattern φ∈Φ to a shooter behavior β∈B, which is denoted as f _T Φ→B. The behavior can be a four-dimensional (4D) phenomenon involving three-dimensional (3D) motion data plus time. The shot pattern Φ is a two-dimensional (2D) projection of the evolving 3D phenomenon (2D plus time). This model can be based on causal hypotheses.

An expert can examine a much larger set of complex shooter behaviors, B ^* ⊃ B, and map them to a larger set of shot patterns Φ ^* , denoted as f _XP : B ^* → Φ. The behaviors β∈B ^* can be a 4D phenomenon, and the shot patterns φ∈Φ ^* are 2D phenomena. Since the expert adapts the model according to his or her understanding of the shooter's physiological kinematics and psychology, as well as the physics of shooting, this can be inferred to be a causal model in principle.

A relatively simple iterative model for expert learning may involve predicting shot patterns from observed shooter behavior, denoted as f _XP :B ^* →Φ. The model may then evaluate the difference between the predicted shot pattern and the actual shot pattern, denoted as δ(φ',φ). The model may then adapt a causal predictive model f _XP based on the difference δ(φ',φ). This may be repeated for each n-shot drill.

This type of learning model treats the shooter behavior β∈B ^* as a 3D phenomenon and the shot pattern φ∈Φ ^* as a 2D phenomenon. In some cases, the model is enhanced to treat B ^* as a 4D phenomenon, Φ ^* as a 3D phenomenon, or both. According to some embodiments, pose analysis methods represent the 4D shooter behavior B ^* by a 3D projection (2D plus time) of B ^** .

The model can objectively distinguish good shot patterns (e.g., clustered around the bull's-eye) from the remaining "bad" shot patterns by implementing a decision function _fG :Φ→D, where D is a binary variable. The model can also be configured to relate the shooter behavior E ^* to how (and why) a shot pattern φ is "bad."

In some embodiments, the sampled time signal for the shooter behavior can be reduced for a single drill as a single line in the dataset. A bounding box can be derived around the set of X-Y coordinates of each of the body landmarks in the drill. The bounding box can be represented as shooter behavior B. Thus, there can be a shooter behavior for each of the body landmarks during a period of time. In some cases, shooter behavior B and shot pattern Φ are represented in polar coordinates as a radius and an angle. Shooter behavior B (which in some examples is 17) is 2D, with (x, y) coordinates or an (r, θ) representation. Shot pattern φ∈Φ is a causal consequence of shooter behavior β∈B, and the method can treat shot pattern Φ as a proxy for the unobservable third-dimensional shooter behavior β. The method treats shooter behavior β∈B and shot pattern φ∈Φ as features and score s∈S as a target in the ML model, which is represented as _fML :BxΦ→S.

The model may be analyzed for patterns (β, φ, s)∈Ψ _L ,Ψ _H that result in low and high scores. For this, implicit or explicit clustering techniques may be used. In some cases, a model f _ML :BxΦ→S can be built for multiple drills for a particular shooter and drills for multiple shooters. In some cases, given a particular drill (β, φ, s) and a model f _ML :BxΦ→S, model quality can be evaluated by comparing the actual score s∈S with the predicted score s′∈S. From these comparisons, several models may be updated sequentially.

In some embodiments, a low-scoring drill (β, φ, s) can be evaluated against Ψ to determine which of the 17 components of the observable shooter behavior β and the shot pattern φ, as proxies for unobservable shooter behavior, are likely responsible for the low score. This may be easiest for an explainable model. Unobservable behavior may be explained by the usual labels in the diagnostic target, as described above. In some cases, the shooter diagnostic application may also distinguish low scores due to aim misalignment, perhaps easiest understood as shots clustered around a center of gravity other than the target bullseye. In some cases, the system may determine that a reduced score was obtained and generate recommendations to improve the score. As used herein, the term "reduced score" is used to mean a score that is less than a target score. The target score may be a perfect score, a participant's best previous score, a participant's average score, or some other metric. For example, in shooting competitions, the highest score for a given shot is often "10." In this case, a reduced score is a score below "10." Similarly, in other sports, a score that is less than a desired score may be assigned. For example, in soccer, a penalty kick could have a binary outcome of a reduced score if the goal is missed compared to a goal. In terms of golf shots, a golfer may average 275 yards with their driver. A reduced score could be due to a driver shot that travels 250 yards, which is less than the golfer's average driver distance, and the system could observe the behavior and determine which behavior(s) resulted in the reduced score.

According to some embodiments, the system relies on artificial intelligence (AI) and/or machine learning (ML) for two decisions: detecting shooter movement and shot landing locations, and analyzing shooter behavior for how it leads to shot landing locations. While detecting movement has been discussed, there are multiple ways to analyze how shooter behavior leads to shot landing locations. First, an expert-based approach utilizes comparing individual shooter behavior for each drill with expert assessments of what is believed to result in a good shot. Second, a data-based approach is implemented by building a model from repetitions of shooting behaviors ("drills") by a single shooter or multiple shooters. This can be considered an AI/ML-based discovery strategy that determines which behaviors correlate with good shots.

In some cases, AI/ML can be used to automate and augment expert-based analysis in the sense that, when the "ideal" behavior of a prototype is known a priori, models of these a priori known behaviors can be fitted to data from a single shooter or multiple shooters.

From that perspective, Transformer ML models can mimic automated and augmented expert-based analysis by combining pre-trained database models for inferred relationships between language fragments (similar to inferring relationships between shooter behavior and shot impact locations) with an additional stage of database adaptation.

Referring to FIG. 11 , the system determines key points related to the shooter's hand 1100. The key points may correspond to each mobile joint in the wrist, hand, and fingers, and their respective locations and positions relative to one another. The system may connect the key points to a wireframe model 1102 of the shooter's hand, allowing for accurate monitoring of pose and movement. FIG. 11 illustrates multiple key points associated with the shooter's hand that can be used to determine the grip the shooter is using. For example, by referencing the key points on the shooter's hand, the system may determine that the shooter is using a thumbs-forward, thumbs-over, cup-and-saucer, wrist grip, trigger-guard gamer, or another style of grip. Different grips and pressure applied by the hand may impart movement to the pistol, and the system may determine that a different or modified grip would result in better performance.

The system receives signals associated with the shooter's position and movement, processes the signals, and detects errors in the shooter's stance and grip. Additionally, by combining single-frame analysis (e.g., analysis over time, finding insights from the relative positions of different body parts at a particular moment), it is possible to identify errors caused by changes in the shooter's position.

Additionally, the system can focus on the position of the hand on the pistol. A hand tracking machine learning model can be used to track the position of each bone in each finger within the video frames. The system receives signals associated with the position of each finger over time.

12A and 12B, an application program that may execute on a mobile computing device and receive image data from one or more imaging sensors associated with the mobile computing device is shown. Similar to any of the embodiments described herein, methods and processes may be programmed into an application program or set of instructions that may be executed on a computing device. In some cases, the application program may execute on the mobile computing device, and audio/video capture, analysis, scoring, recommendations, training exercises, and other feedback may be performed using the mobile computing device. In some cases, the system receives video and/or audio from multiple video capture devices that capture different views of the same shooter. The system may use the multiple different views of the shooter for analysis and feedback to the shooter to improve performance.

As shown in FIG. 12A , a system running as an application on a mobile computing device 1200 captures video frames of a shooter 1202, establishes body landmarks for tracking over time, creates a wireframe model 1203 of the shooter, tracks fired shots 1204, and provides feedback to the shooter 1202. As shown, the system may identify the shooter's stance, in the illustrated example the "Weaver" stance, track the number of shots 1204, and provide feedback 1206 regarding each shot. For example, a shot fired 7 seconds after the start of the string indicates a center-of-mass hit. The system identifies a reload motion 1208 at 8 seconds, lasting 1.2 seconds, followed by a shot at 11 seconds, indicating the shooter anticipated recoil and delivered an off-center shot. In some cases, this functionality of the system may be viewed as a drill instructor, tracking shots, providing feedback, and making suggestions for improving posture, grip, stance, trigger pull, etc., to improve the shooter's performance.

FIG. 12B shows an additional screen 1210 that may be displayed by the system in the Spotter modality, in which a mobile computing device may aim its video capture device at a target. In some cases, the mobile computing device may use an internal or external lens to get a better view of the target; the mobile computing device may be coupled to a spotting scope or other type of optical or electronic telephoto zoom lens to get a better target image.

The system can be toggled between different modes, such as Spotter, Drill Instructor, DOPE, Locker (which can store information about different firearms owned by the shooter), and program settings. Spotter mode may show an additional screen 1210 with a target view where target hits can be viewed. The system may also display other information, such as the firearm 1212a and ammunition 1212b combination used in the shot, distance to the target 1214, target type 1216, firing string time 1218, number of shots fired, number of target hits 1220, and score 1222, among others. An image of the target 1224 may also be shown, and hits 1226 may be further highlighted on the image of the target 1224. It should be understood that Spotter mode may show information in real time, including elapsed time, hits, score, and number of shots. Additionally, the system may store data associated with the firing string for later playback, review, and analysis. For example, the Drill Instructor mode may review the DOPEs associated with the shooter and the firearm and ammunition combination, provide a detailed analysis of the shooter's misses with a particular firearm, and provide exercises or suggestions for improving the misses to improve the shooter's performance.

With reference to Figures 13A, 13B, and 13C, an exemplary architecture for data capture 1300 is shown. According to some embodiments, a shooting range 1301 may be equipped with one or more sensors, such as image sensors, projectors, and computing devices. Figure 13A shows the shooting range 1301 from the perspective of a shooter 1302 looking downrange. Figure 13B shows the shooting range 1301 from a side view, and Figure 13C shows the shooting range 1301 from a top view.

One or more cameras 1304 may be mounted within the shooting range to capture video of the shooter from multiple angles. The cameras 1304 may be any suitable type of video capture device, including, but not limited to, a CCD camera, a thermal camera, a dual thermal camera 1305 (e.g., a capture device with both thermal and optical capture capabilities), among others. The cameras 1304 may be mounted in any suitable location within the range, such as to the sides of the shooters 1304a, 1304b, facing the front of the shooter 1304c, or overhead 1304d, among others.

In some cases, the camera 1304 may be mounted on a gantry system that provides a portable structure to support one or more cameras 1304 to capture multiple angles of the shooter.

In some embodiments, a projector 1306 may be provided to project an image onto a screen 1308. In some cases, the projector may project an arbitrary target image onto the screen or shooting wall, allowing the shooter to practice dry shots (e.g., without firing live ammunition) and the system to register hits on the projected image. For example, the image may show a target on the screen, and the shooter may dry-fire at the target. The system may register the aim point at the time the trigger is pulled and display the hit on the projected target image. Compatible devices exist that fire a laser through the barrel of a firearm when the trigger is pulled to indicate where the shot would have landed on the target. The thermal imaging camera 1304 may detect the laser hit location, and the system may be configured to display the hit on the projected target, which may then register and score the hit. In this way, shooters can practice using a shooting simulator with their own firearms without having to travel to a dedicated shooting range. The system may encourage dry-fire practice to improve poor shooting habits.

Figure 14 illustrates a system configured for multi-source synchronization. In some embodiments, a shooter 1402 is captured from different angles by multiple video capture devices 1404. The system obtains features such as stance, pose, and movement from the shooter 1406. The system may analyze the captured audio signal 1408 to complement the video shot detection. That is, the system may analyze both the video and audio and detect shots fired by the shooter, for example, by correlating spikes in the audio waveform with a sudden muzzle lift of the firearm.

The system may be configured to detect shooting stages 1410, such as first firing string, reload, second firing string, etc., including detection of fired shots. The system may additionally identify and determine shooting errors 1412, identify errors, and drills and exercises to address shooting errors. Error detection may be iterated and further error corrections 1414 may be suggested. The system may also incorporate shot detection 1416, as described herein, including correlation of shot detection with audio data.

The system may further capture images 1418, such as video images of the target, and register hits on the target, which may be correlated with motion data before and during the trigger pull. As described herein, the system may determine shot location 1420 and ultimately determine a score 1422 of one or more of the shots fired. Some embodiments of the system thus provide an intelligent tutroing/adaptive training solution that transforms a time-intensive, non-scalable live training environment into an automated, adaptive, virtual scenario-based training solution.

According to some embodiments, the model is based on instructions executed by one or more processors that cause the processors to perform various operations. For example, the method may include collecting (as many as possible) pose analysis files, which may be stored as comma-separated value (CSV) files.

Figure 15 shows the shot bounding box 1500 determined by the system. The "drillparser.py" script can be configured to combine the pose analysis files. The system can then find the shot in the CSV file for each of the 17 pose landmarks, derive a bounding box 1500 oriented around the x-y coordinates in a 1.0-second time window up to and including the shot, and then use the angle φ 1502, dimensions l 1504, and w 1506 of the bounding box 1500 as elements in a decision tree model for scoring. Because the "allshots" method processes data on a CSV file basis, a bounding box 1500 is drawn around all shots as a single row in the resulting dataset. Because the "oneshot" and "oneshotd" methods can treat each shot individually, the shot bounding box 1500 can be an infinitesimal box around a single shot; therefore, each row in the dataset is one shot. The "oneshotd" method additionally orients the bounding box around the 17 pose landmarks in time order, while the "oneshot" method ignores time.

The system can therefore derive an oriented bounding box around each shot, which can be oriented in the direction of the box around the body landmark.

3) These datasets are processed in BigML as follows:

1. Upload to BigML as a Source object

2. Convert a Source object to a Dataset object

3. Split the Dataset object into Training (80%) and Test (20%) datasets.

4. Build a tree model using elements selected from the training dataset.

5. Create BatchPredictions using the appropriate model and test dataset.

6. Download the model as a JSON PML object for the next step.

4) The "modelannotator.py" script annotates the model PML file as needed for explanation operations. In some cases, this annotation consists simply of adding a "target" attribute to each node that is a list of all target (score) values reachable in the tree from that node.

5) The "itemexplainer.py" script uses the (hypothetical) pruned annotated model to "explain" which pose elements for a new drill result in an unsatisfactory score prediction.

6) Of course, the model can be tuned by repeating model training through machine learning using instances where the predicted scores differ significantly from the actual scores.

The elements derived from the shot sample may include one or more of the following elements: shot_phi with bounding box rotation (-π≦φ≦π), shot_l - bounding box length, shot_w - bounding box width, shot_lw - bounding box area, shot_theta - bounding box center rotation (-π≦φ≦π), shot_r - bounding box center radius. A bounding box 1500 is drawn around the shot sample.

A very similar technique can be used to derive elements from body landmark samples. As body landmarks move over time, each body landmark can similarly be used to create a bounding box with length, width, area, and rotation that can be correlated to shot elements.

In some cases, the system is configured to use one or more cameras for machine vision of the shooter, determine one or more body landmarks (possibly up to 17 or more), and track the movement of each of these landmarks over time during a shooting drill. The system can correlate the timed body landmark movement with scored shots to determine whether a shot is good or bad. A good shot may be relative to the DOPE for the shooter, firearm, and ammunition combination. For example, if a single shot is closer to the aimpoint than the shooter's average shot, it may be classified as a good shot. Furthermore, by examining several shots over time, the system can correlate shooter behavior with good or bad shots. Furthermore, by analyzing shooter behavior (e.g., body landmark movement), the system can predict whether a shot is good or bad without seeing the target outcome. For example, a good shot is considered a shot within the 9 or 10 ring on the target, while a bad shot is one outside the 9 ring. The definition of a good or bad shot may vary depending on the shooter's expertise. As an example, for a highly skilled shooter, anything outside the 10 ring may be considered a bad shot.

Finally, by correlating shooter behavior with target accuracy, the system can provide the shooter with output regarding behaviors that cause accuracy degradation. Additionally, the system can recommend one or more specific drills to the shooter to address the behaviors that cause accuracy degradation.

Figure 16 shows some embodiments of a decision tree model 1600 for correlating body landmark factors with shot samples. A decision tree algorithm is a machine learning algorithm that uses decision trees to make predictions. It follows a tree-like model of decisions and their possible outcomes. In some cases, the algorithm works by recursively splitting the data into subsets based on the most important features at each node of the tree.

From the body landmark data, various features are extracted to represent the kinematic characteristics of body movement. These features include body landmark positions over time, bounding boxes describing the movement of body landmarks during times including before and during the shot, relative distances between landmarks, and time derivatives capturing the dynamic aspects of the movement. Feature scaling is performed to facilitate model convergence.

In some embodiments, the decision tree shown in Figure 16 serves as the core of the model. It is selected for its ability to unravel complex nonlinear relationships in the data while maintaining interpretability. The depth of the tree is optimized to minimize overfitting through techniques such as cross-validation or a predetermined maximum depth. The choice of separation criterion, whether based on Gini impurity or information gain, depends on the specific problem. Pruning methods, such as enforcing a minimum number of samples per leaf, are applied to control excessive branching.

Training a model may involve the recursive construction of a decision tree. The tree has nodes that correlate with body landmarks. For example, left_ankle_phi node 1602 may branch to right_wrist_lw node 1604 and right_knee_phi node 1606, which may indicate how the rotation of the bounding box of the right ankle motion data, combined with the rotation of the bounding box area of the right wrist or the bounding box associated with the right knee, affects the resulting shot landing location. Similarly, a decision tree may correlate shot landing location with a combination of features. At each node, the training data is partitioned based on feature values to optimize a selected loss function, such as mean squared error or cross-entropy. Hyperparameters can be fine-tuned via cross-validation, and various metrics are used to evaluate the model's performance.

During real-time operation, the decision tree model is continuously updated as new body landmark data becomes available for each shot fired. For each input feature vector derived from live landmark data, the model traverses the decision tree and arrives at a leaf node. The label assigned to the leaf node (indicating a positive or negative outcome) is used as the model's prediction. In the illustrated example, right shoulder bounding box lengths between values of 0.80 and 0.82 match the predicted outcome of the shooter's performance, shown in leaf nodes 1608 and 1610.

In the context of this model, a positive outcome denotes successful execution of a particular movement or achievement of the desired shot landing location. Conversely, a negative outcome indicates an inaccurate movement, deviation from the desired position, or an off-center shot landing location.

Model performance is rigorously evaluated through metrics including, but not limited to, accuracy, precision, recall, F1 score, and area under the receiver operating characteristic (ROC) curve. A confusion matrix is employed to quantify the model's proficiency in classifying positive and negative outcomes.

Figure 17 shows an annotated model 1700 showing various values at several nodes. For example, during the firing string, the bounding box area for the right wrist 1702 shows values of 0.47, 0.51, 0.57, 0.59, and 0.61. This indicates that the shooter moved their right wrist within the area defined by the displayed values during the firing string when shots were being fired. Following the branch below the right wrist 1702, for values below the average of the values, the system can identify that the left knee 1704 moved in a specific pattern and correlate this with either good or bad shots, looking for a causal relationship in the combination of right wrist movement and left knee movement. Other features can be similarly annotated in the model. The annotated features can be stored in a feature vector and correlated with each shot fired for analysis of how different features combine to result in a particular shot performance.

FIG. 18 shows a pruned model 1800 that allows for a deeper dive into various features and their interrelationships with one another. For example, reviewing the right_knee_phi 1802 feature (e.g., the bounding box rotation associated with the right knee's movement during the shooting string) reveals values of 0.8, 0.82, and 0.9. Given these values, we can see that a right_knee_phi value of 0.9 results in one outcome at the first result node 1806. We can see that a right_knee_phi 1802 of 0.8 or 0.82 and a right_shoulder_l bounding box length 1804 of 0.09 results in a second outcome at the second result node 1808 and the third result node 1810. In some cases, the second result node 1808 is associated with poor shooting performance, and the third node 1810 is associated with good shooting performance. These nonlinear causal relationships can be determined by machine learning models, allowing the system to determine which movements result in better and worse shooting outcomes by running one or more machine learning algorithms.

In some cases, an isolation forest encodes a dataset of trees such that leaf instances per leaf are identical. Node splits may be chosen randomly rather than to reduce impurity in child target values. In some cases, a node is a leaf when all training instances at the node have the same target value. In some examples, rare instances in the training dataset reach the leaf sooner than less rare instances. According to some embodiments, new anomalous instances have a shorter path relative to the tree depth for all trees in the forest, making anomaly identification more efficient.

In some cases, a classification and regression tree (CART) is a predictive model that describes how an outcome variable is predicted based on other values. A CART-style decision tree is one in which each fork is a partition of the predictor variable and each end node contains a prediction for the outcome variable. It is a recursive algorithm that builds a binary tree to partition feature space into segments that are homogeneous with respect to the target variable, performing binary partitions on input features based on specific criteria to create a tree-like structure. CART-style decision trees can be useful for predicting the outcome of a shot based on motion data of one or more body landmarks. CART-style decision trees are multi-category classifiers (e.g., N > 1), while isolation trees are single-category classifiers (e.g., "abnormal or not"). In some cases, a CART-style decision tree can be made into an isolation tree by designating M < N categories as "not abnormal" and N - M as "abnormal," and pruning branches that reach leaves with N - M "abnormal" categories all the way to the root node. This can be analogous to training a separating tree with M<N "non-anomalous" instances, which allows for impurity reduction splits and impure "non-anomalous" leaves.

FIG. 19A shows a gantry 1900 system that can provide a portable mounting structure for housing one or more imaging devices, including one or more video cameras. In some cases, the structure includes one or more support columns 1902 and one or more crossbars 1904. The gantry structure 1900 can be positioned around the shooter or, in some cases, downrange from the shooter. For example, the gantry 1900 can have the crossbar 1904 positioned approximately 1 foot (≒3 m) to 8 feet (≒2.4 m) above the shooter and between 1 foot (≒3 m) and 15 feet (≒4.5 m) in front of the shooter. In some examples, a camera is positioned on each support column and on the crossbar. Thus, in some embodiments, two, three, or more cameras can be positioned on the gantry, with some of the cameras pointed toward the shooter and one or more additional cameras pointed downrange toward the target.

Figures 19B, 19C, and 19D show various views captured by cameras mounted on the gantry 1900. A first camera 1908 may be mounted on the crossbar 1904 or support 1902 and positioned to capture a left side view of the shooter (Figure 19B). The camera may be configured to capture the shooter's entire body or the shooter's upper body and head.

A second camera 1910 may be mounted on the crossbar 1904 or support 1902 and configured to capture a right side view (FIG. 19C) of the shooter. The camera may be configured to capture the shooter's entire body, or the shooter's upper body and head. In some cases, the first camera 1908 and the second camera 1910 utilize different fields of view, such that one camera captures the shooter's entire body and the other camera captures only a portion of the shooter's body.

A third camera 1912 may be positioned on the crossbar 1904 and configured to capture an overhead view 1914 (FIG. 19D) of the shooter. By positioning the cameras to capture the shooter's movements from various angles, the system can correlate the video data from each camera and determine three-dimensional motion data for selected body landmarks.

A fourth camera may be positioned to capture video data of the target 1916 (FIG. 19E) and the impact of the shots on the target 1918. The video data from each of the cameras may be synchronized and analyzed to determine when shots were fired and to correlate the fired shots with registered hits or misses on the target.

FIG. 20 illustrates a computer program user interface 2000 usable with the systems and methods described herein. For example, the computer program may be configured to receive video data from one or more cameras, synchronize the video data, determine when shots were fired, and register and score hits or misses on the target. The user interface 2000 may include a start record button 2002 that enables the shooter to begin video capture. In some cases, the shooting string may be time, and the start record button may further start a timer. The user interface may further include a timer 2004 associated with the session. The user interface may be presented on a mobile computing device associated with the user or on a mobile computing device associated with the facility. For example, a gantry system may be installed at the facility, and a computing device associated with the facility may be connected to the gantry system and configured to receive video data and provide performance feedback for the embodiments described herein.

The user interface may be provided on any suitable display, such as a television, touchscreen display, tablet screen, smartphone screen, or any other visual computer interface. Further referring to FIG. 21 , the user interface 2000 may display indicia associated with the shooting string, such as video of the shooting string and movements during the shooting string 2102. The video may be displayed in a playback window, providing controls 2104 for playing, pausing, adjusting the volume, and scrubbing the video. Additionally, there may be controls for selecting different views 2106, i.e., controls that allow the viewer to select video clips captured by different cameras during the shooting string to view different views of the shooting string, either individually or in an integrated view, such as a side-by-side view. These views may be synchronized to allow the viewer to view different views of the same event simultaneously.

The user interface 2000 may further show a target 2110 and may identify hits 2112 registered by the system on the target. The user interface 2000 may further display the score 2114 of the most recent shot, along with the average score 2116 for the string. Of course, other information may be displayed as desired by the user, including training tips, motion data that may account for off-center shots, or other mistakes during the shooting string.

FIG. 22 shows an additional view of a user interface 2000 that allows a user to specify a body landmark selection 2202. In addition, the user interface 2000 provides a selection for signal settings 2204, allowing the user to specify details of Y coordinate movement or X coordinate movement. In response to a user selection, the user interface 2000 may display motion data 2206 associated with that selection. This type of review and analysis allows the shooter to very specifically identify the movement of individual body landmarks during the shot string, and additionally, to specifically identify horizontal movement, vertical movement, or both for review.

FIG. 23 illustrates some of the features and techniques employed by embodiments of the described systems and methods. In many embodiments, the disclosed systems utilize machine vision (e.g., computer vision) 2302 to track the body landmarks of a participant (e.g., the shooter) and track target changes to provide automatic target scoring 2304. The systems and methods may also utilize audio processing 2306 to enable shot time detection 2308, which may also be combined with computer vision techniques. The disclosed systems and methods may also utilize signal processing 2310 to provide shooter position correction 2312, including pose, posture, movement, grip, trigger pull, etc. The systems and methods described herein also apply machine learning 2314 to determine the causality of off-center shots, which may include analysis of the shooter's pose, grip 2316, trigger pull, stance, and body landmark movement, among other factors.

Figure 24 shows a method 2400 according to an embodiment described herein. The system may receive video data and optionally audio data. The system may be configured to detect 2402 and score shots through image processing on the video data of the target. The system may process the video data to determine fired shots and a shooter pose analysis 2404. Additionally, the system may analyze audio data for shot detection 2406.

Shot detection 2402 provides 2408 coordinates (e.g., x, y coordinates) of the shot within the target. Shooter pose analysis 2404 provides 2410 coordinates (e.g., x, y coordinates and optionally z coordinates) of body landmarks during the shooting process. Audio shot detection 2406 provides 2412 the precise time of the shot. Shooting analysis may include one or more machine learning algorithms that receive the shot and body data and, through the machine learning algorithms, detect and predict the causality of the shooter's shot performance with the firearm and ammunition combination, which may be referred to as performing shooting analysis 2414. For example, an embodiment of the system may generate one or more of a session score 2416, shooting position recommendations 2418, shot misses, and grip analysis 2420, among others.

Figure 25 shows a flow diagram of a sample process 2500 for using machine vision and machine learning to track a participant's body landmarks and generate recommendations for improvement. As noted herein, the systems and methods described herein can be used in any competition, such as a sporting event, that benefits from repeatability and accurate body kinematics. In addition to shooting, some such competitions include archery, golf, bowling, darts, running, swimming, pole vaulting, football, baseball, basketball, hockey, and many other types of sports. In any case, the system is configured to track a participant's body landmarks and determine how to modify their body movements to improve performance.

At block 2502, the system receives video data of the participants, which may be obtained from a single image capture device, or two image capture devices, or three or more image capture devices. The image capture devices may be any suitable imaging devices configured to capture continuous images of the participants, and may include any consumer or professional video camera, including cameras typically built into mobile computing devices.

At block 2504, the system determines one or more body landmarks for the participant. A body landmark may be associated with any joint, body part, limb, or location associated with a joint, limb, or body part. In some cases, the system generates a wireframe model based on one or more body landmarks and may not use all body landmarks in generating the wireframe model.

In block 2506, body landmarks are tracked during performance of the activity to generate motion data. As a non-limiting example, body landmarks may be created for a golfer's hands, wrists, arms, head, shoulders, torso, waist, knees, ankles, and feet, which may be tracked during a golf club swing.

At block 2508, a score is determined and associated with the performance. As described, in activities involving a projectile, the score may be associated with the trajectory or destination of the projectile. In the case of golf, for example, the score may be based on distance, direction, or proximity to the target, or a metric associated with the participant's average or past performance. In short, any metric may be used to evaluate the quality of the performance results.

In block 2510, the score is associated with the performance. That is, the performance may be linked to the determined score and stored for later analysis to determine trends in performance over time or to compare one performer to another.

In block 2512, the system generates recommendations for altering movements in subsequent performances to improve results. In some cases, the recommendations may involve the hands, including gripping a firearm, golf club, bat, stick, etc. The recommendations may also include changes or shifts in weight distribution. The recommendations may include movements of the hands, head, shoulders, body, legs, feet, or other body parts. Often, the recommendations include suggestions for altering the movement of one or more body landmarks to improve the performance score in subsequent attempts.

Acoustic Small Arms Signature

According to some embodiments, a system is described that can quickly calculate the acoustic sound signature of a shooter's firearm firing a specific ammunition in a given environment using consumer audio recording equipment (e.g., iPhone®, tablet, phone, video camera). In some cases, the system can receive an input audio or audio/visual file and calculate the firearm signature based on the sounds in the captured recording. The recording can be captured by any suitable audio and/or video capture device, such as, but not limited to, a security camera, a traffic camera, a video camera, a television camera, a mobile device recorder such as a smartphone or tablet, and other capture devices. In some cases, the capture device is a readily available consumer recording device. The acoustic signature can further include determining the make, model, silencer, ammunition type, ammunition manufacturer, and other characteristics of the firearm based on the acoustic signature from the fired firearm.

This capability can be used to detect and separate a shooter's shots from those of other shooters in a typical shooting range, which may be useful for automated scoring, etc. A version of this capability could be used to identify specific firearms and ammunition from recordings.

In some cases, the described solutions operate in near real time on a single consumer recording device, such as a mobile phone, using only a modest amount of training data. As used herein, the terms "real time" or "near real time" are broad terms that, in the context of this disclosure, relate to receiving input data, processing the input data, and outputting the results of data analysis with little or no human-perceived latency. In other words, a system such as that described herein that outputs analyzed data within less than one second is considered near real time. A system that operates in real time or near real time may limit the amount of computation for machine learning, or at least training models, that the method can use to characterize a particular firearm that fires the appropriate ammunition. Additionally, there are methods that do not use ML techniques, meaning that they do not adapt ("train") the model to a large number of samples. Some of these other methods may perform well but may also be more computationally intensive or shift the computational load to shot identification time, for example, by searching a large dictionary of signatures for "matches."

Previous approaches that utilize artificial intelligence (AI) to analyze gunfire present a simple approach of using deep neural networks (DNNs) to classify gunfire sounds by firearm and ammunition type. In previous approaches, the DNNs had to be pre-trained with numerous gunfire sounds for each firearm and appropriate ammunition type. These sounds may not have been captured on the same recording device or under the same firing conditions. This potentially improves the generalization ability of the trained DNN instance to classify firearm and ammunition types regardless of the environment and recording device. However, it reduces the ability to identify gunfire sounds for a specific firearm and ammunition type in any environment using any recording device.

Previous techniques have described a two-stage approach to classification. For example, some previous techniques use a pre-trained instance of a relatively general pre-trained DNN instance as an approximate classifier (predictor), and may use an additional trainable step on the classifier output to improve the DNN prediction. In some cases, the DNN treats finite-length time segments of the time-varying spectrogram of gunfire sounds as images, distinguishing pooled images of each firearm and ammunition pair from pooled images of other firearm and ammunition pairs. This approach has several drawbacks, including the use of a general DNN instance that is not particularly adept at classifying sounds from different environments or different ammunition.

According to some embodiments, the described system superimposes a trainable weighting mask onto an image created from a time segment of a time-varying spectrogram, providing a simple training method for adjusting the mask using a small number of instances of a shooter's small arms and ammunition fire recorded in a particular environment with a particular device. Training only the weighting mask is much less computationally intensive than training the DNN itself. Thus, according to some embodiments, the DNN may not be trained or may be trained to a much lesser extent than traditional methods, while the weighting mask is trained. This approach has several advantages.

For example, abstractly, the adjusted mask can be viewed as a signature for the combination of firearm, ammunition, environment, and recording device. This signature is not used to search a catalog of that combination signature of elements, but rather is used to adjust the spectrogram input to improve the DNN's classification performance for firearms and ammunition in a particular environment using a given recording device. The enhanced DNN classification can then be used to improve detection of the shooter's shots and distinction from those of other shooters.

This approach offers significant advantages in the computational intensity required, allowing for much faster analysis and can be used for a large number of firearms in many different environments. For example, the training data set for a particular shooter is small, allowing the weighting mask to be updated online using stochastic gradient descent or offline using gradient descent. In other words, the training data can be for a specific shooter using a specific firearm and ammunition combination in a given environment, resulting in a much smaller data set than would be possible if data points were aggregated from a large number of shooters in different environments.

In addition, both algorithms may estimate the gradient of the function computed by the DNN at each iteration of the weight matrix update. While this is not computationally prohibitive, it may turn out that just the sign of the gradient or a coarsely quantized version of it is sufficient. The simplification mainly depends on whether the DNN computes a monotonically increasing or monotonically decreasing function at each input. There is a large literature investigating the properties of functions computed by DNNs. However, finding a simple answer to this question is not easy. In some cases, a DNN does this when all of its internal layers use linear or affine activation functions and the output layer uses a monotonically non-decreasing activation function.

Thus, in some embodiments, the disclosed system can quickly fingerprint a firearm and ammunition pair in a particular environment using a specific recording device. In other words, the systems and methods described herein can very quickly determine the acoustic signature of a firearm and ammunition combination in an environment. This allows the system to distinguish the analyzed acoustic signature from other firearm and ammunition combinations. This can be particularly useful, for example, in crowded shooting ranges where distinguishing one shooter from another is important, in combination with a system performing automated target scoring. By being able to distinguish between firearm and ammunition pairs, the scoring system can determine that a particular firearm and ammunition combination is utilized and that it is likely to coincide in time with a hit on the target, thereby reducing the number of false positives and missed shots. In some examples, the system can be executed on a mobile computing device and used at a shooting range. If the mobile computing device has a microphone pointed in the general direction of the shooter of interest, shots fired by the shooter of interest will generally have an audio file dominated by shots fired by the shooter of interest, which may help determine whether the shots fired are from the shooter of interest. In some cases, the recording device (e.g., the mobile computing device) has a microphone pointed downrange, such as when the mobile computing device has a camera pointed at the target for automatic target scoring, in which case the volume of shots fired by the shooter of interest will be more difficult to distinguish from other shooters at the range. In these cases, the described embodiments may use training of machine learning algorithms or training of weight masks to quickly distinguish shots fired by the intended shooter from all other shooters at the range.

In some cases, classifiers rely on feature extraction in the form of time-frequency spectrograms. Mel-frequency cepstrum coefficient (MFCC) vectors can be used as an alternative raw power spectral density vector (PSD) for the frequency representation of a spectrogram. Mel-frequency cepstrum (MFC) was originally developed for audio processing and the way information is thought to be encoded in audio waveforms. In some cases, MFC can be applied for firearm fingerprinting purposes. As used throughout this disclosure, "firearm fingerprint" is used to refer to an acoustic signature, or a specific sound or sound image, that identifies a particular firearm and ammunition combination.

In some cases, the MFCC may be generated by performing a series of steps, including, but not limited to, i) windowing a signal segment and computing a Fast Fourier Transform (FFT) of the signal segment; ii) combining the linear FFT coefficients with MEL frequency filter bank coefficients; iii) taking the logarithm of those coefficients; and iv) computing a Discrete Cosine Transform (DCT) of the logarithmic MEL filter bank coefficients. The FFT of a signal segment generally produces a peak at the applied frequency, as well as other peaks, typically called sidelobes, on either side of the peak frequency. The DCT optionally represents a finite sequence of data points in terms of a sum of cosine functions oscillating at different frequencies. In some cases, fewer or more steps than those disclosed may be implemented to arrive at a firearm fingerprint based on gunfire sounds. For example, in some cases, only steps i), ii), and iii) above may be used for gunfire sounds.

According to some embodiments, the MFCC or PSD coefficients may be used as input to a DNN trained to classify the time-frequency spectrogram into several independent categories. For example, the MFCC and/or PSD coefficients may input two categories as proposed, or several categories corresponding to pairs (firearms, ammunition), or a larger number of categories corresponding to tuples of k>2 attributes.

In some cases, the DNN may place a new gunshot sound "near" whatever class is represented by the largest classifier output among all classifier outputs. For example, a gunshot sound may initially be classified through a nearest neighbor technique, such as a k-NN algorithm. Subsequent analysis may further classify the gunshot sound.

The inner layers of the DNN can represent different sets of input attributes. These can also be used for gunfire sound classification and training.

From an optimization theory perspective, training a DNN can be thought of as essentially defining a surface with multiple local optima, and the DNN directing new inputs to the most appropriate local optima. In some embodiments, placing one or more trainable layers after a pre-trained DNN can be thought of as extracting a different set of more optimal attributes for gunfire sounds.

In some cases, placing a trainable layer, possibly including weighting coefficients, in front of the pre-trained DNN can be thought of as tuning the pre-trained DNN so that the shooter's shot of interest is the most positive example of all shots that the pre-trained DNN places "near" whatever class the maximum classifier output represents among all classifier outputs. In some cases, the decision threshold can be tuned to optimize a confusion matrix for the training dataset used to tune the trainable input layer or another test dataset for some useful criterion.

According to some embodiments, the FFT of a windowed segment of shot audio can be computed in O(n log n) time. Computing the MFCC has the same time order, but involves computing two O(n log n) operations. In some cases, the MFCC is not significantly better than the FFT and may be omitted.

According to some embodiments, the MEL frequency log spectrum (omitting the discrete cosine transform (DCT) to obtain the cepstrum) provides an improvement over the raw FFT, with only an extra computational cost of O(n). Therefore, in some examples, the MEL frequency log spectrum is used rather than the DCT to obtain the cepstrum.

In general, there is no fast form for computing functions computed by DNNs (composed of layers of convolutional neural networks (CNNs)), such as FFTs. FFTs exploit regularities in the FFT kernel that are inherently absent in any CNN architecture. However, the examples described herein take an FFT of the input time signal, and the pre-trained DNNs composed of CNNs can be implemented in the frequency domain.

As a non-limiting example, the following DNN-based rapidly trainable category recognizer can be implemented to rapidly determine the acoustic signature of a firearm and ammunition combination:

Let f:RM→RK represent the analog transformation of a real-valued M-dimensional input vector into a K-dimensional vector of class probabilities by a trained deep neural network. The final discrete output mapping Ψ:RK→N≦K selects the most likely of the K classes.

The system may be configured to extend the trained DNN into an enhanced binary classifier for class k. In some cases, the system may implement a rapidly trainable input stage based on the Hadamard product of a weighting matrix W and an input vector x.
In some cases, the Hadamard product is a binary operation that takes two matrices, such as a weighting matrix W and an input vector x, and returns a matrix with corresponding elements multiplied together. The system can then perform a selector function on the DNN class probability vector output.

Γ: RK×N≦K→R provides a single class probability for class k. An extended DNN may implement the following function:

An additional set of input vectors that are all instances of the same class k
Suppose we have: ζ = 1, ζ = 1, ζ = 2 ...

where 0<α<1 is an adaptive constant that weights the relative contribution of ζ to W. For all vectors in ζ, the target value is d(n) = 1.0. Iterations are performed for each z in ζ until convergence. Here, relu[...] denotes element-wise relu() of the argument matrix.

Firing a small arms fire produces multiple acoustic events, such as the muzzle blast caused by the expansion of gases in the chamber and exiting through the barrel, and the ballistic shock wave generated by the projectile, which are often supersonic but may also be subsonic in some cases. The acoustic events are the result of variables that produce the firearm signature, which may include, among others, the type, make, and model of the firearm, barrel length, ammunition type, powder quantity and identity, projectile weight, and projectile shape.

Figure 26 shows a sample process 2600 for correlating firearms with a particular shooter, according to some embodiments. When a shooter visits a shooting range to practice, there may be other shooters at the range firing firearms as well. In some cases, a mixed range may have 10, 20, or even 30 or more shooters all firing at the same time. It may be very difficult for an acoustic system to register shots fired by a shooter of interest amid the clutter of firearm fire. In some cases, the system is configured to distinguish between the firearm of the shooter of interest and the firearms of other shooters at the range.

For example, in block 2602, the system receives video data of the shooter, which also includes audio data. This may be received, for example, via a multi-camera system, a dedicated microphone, or a consumer-grade audio/video capture device such as a mobile computing device.

In block 2604, the system may determine the shooter's body landmarks.

In block 2606, the system may track one or more body landmarks while the shooter fires the shot and generate motion data associated with the body landmarks during the shot.

In block 2608, the system correlates the audio data with the motion data to determine that a shot was fired by the shooter of interest. In some cases, the system may receive audio data indicative of a firearm being fired, which may be represented as a spike in the sound wave file. This may be correlated with motion data, such as the shooter's wrist, indicating that recoil from the firearm has displaced the shooter's hand and therefore that a shot has been fired. In some cases, the system is trained to distinguish the shooter's firearm from other firearms at the range. In this case, the system may be trained to distinguish between shots fired by the shooter of interest and shots fired by other shooters at the range.

In block 2610, the system can determine a score for the shot and associate the score with the action data that led to the score.

In block 2612, the system may analyze the performance data in combination with the score to determine any mistakes made by the shooter and provide suggestions for identifying mistakes and/or how to address future mistakes. The system may also provide training exercises to enable the shooter to address mistakes and improve their shooting performance.

Referring to FIG. 27, which illustrates an exemplary system 2700 using online learning, audio 2702 is received and converted into an incremental spectrogram 2704. The spectrogram 2704 is framed (2706), for example, by a time window and used to determine the MFCC, for example, by determining the FFT of the windowed signal, combining the linear FFT coefficients with MEL frequency filter bank coefficients, determining the logarithm of the coefficients, and determining the DCT of the logarithmic MEL filter bank coefficients. The determined MFCC can be input to a rapidly trainable input stage 2708 and then delivered to a DNN multi-class classifier 2710. A selector 2734 determines the classifier output k with the highest class probability to classify the shot. The rapidly trainable input stage 2708 may be referred to as a trainable weighting mask, or simply a mask. The mask can be tuned using instances of the shooter's firearm and ammunition fire as recorded in a particular environment. In some cases, training only the weighting mask is much less computationally intensive than training a DNN. The trained (e.g., tuned) weighting mask may represent the signature of a combination of firearm, ammunition, environment, and recording device. This signature is not necessarily used to search a catalog of signatures, but rather is used to tune the spectrogram input to improve the DNN's classification performance for firearms and ammunition in a particular environment using a given recording device. In some cases, the training data set for a particular shooter (e.g., a particular firearm and ammunition combination) is small, so the weighting mask can be updated using online methods, such as by using stochastic gradient descent, or offline methods, such as by using gradient descent techniques.

Inner layers of the DNN classifier 2710 may represent different sets of attributes of the input. In some cases, the DNN is trained to define a surface with multiple local optima, and the DNN can operate to guide new inputs to the most appropriate local optima. The enhanced DNN classification can then be used to improve detection of shots from one firearm and distinction from those of other shooters.

The weighting mask W at the input stage 2708 to the pre-trained DNN classifier 2710 can be adapted by an online learning loop 2712 that optimizes the weighting mask W.

Online training loop 2712 includes copies 2714, 2716, and 2718 of shot classifier input stages 2708, 2710, and 2734. When a new shot n is detected, subtractor 2720 in online training loop 2712 compares the result of the selected output k from the copies of shot classifiers 2714, 2716, and 2718 with the expected classification d(n) = 1.0 to calculate a classification error term.

Block 2730 of the adaptation loop calculates the vector sign of the gradient of the DNN output with respect to the spectrogram input for the current output spectrogram x(n) from the framing block 2706.

The raw incremental adjustment to the current weighting vector W i (n) is then calculated by the Hadamard multiplier 2732 as the product of the current shot spectrogram from the framer 2706 and the code vector of the classifier gradient 2730. The vector multiplier 2722 then scales the raw incremental adjustment to the weighting vector W i (n) by an arbitrary value α.

Finally, vector adder 2724 calculates preliminary updated weights by adding the current incremental adjustment to the current W i (n). These preliminary weights are converted into updated weight vector W i+1(n) by vector relu[ ] operation 126.

The weight adaptation loop 2712 described above is repeated until the weight vectors converge, as indicated by the delta operator 2728, e.g., until |Wi(n) - Wi+1(n)| ≤ δ. The resulting Wi+1(n) is then selected as the weight vector W(n+1) for classifying the next shot.

Offline learning can be used, such as when ζ is small. In some cases, offline learning can begin by setting W=I and updating it using gradient descent as follows:

In some cases, offline learning can require roughly the same amount of computation as online learning. However, online learning has the advantage over offline learning that offline learning requires access to the entire training dataset ζ at every iteration of the update function, while stochastic gradient descent does not. Offline learning offers the advantage of finding a local optimum, whereas online learning may only approximate it.

The current argument
The gradient ∇xf evaluated at is not overly costly to compute, but it may be sufficient to pre-compute sgn(∇xf(1)) or sgn(∇xf(1 β)), where 1 is the unit vector if the DNN normalizes the input. In some cases, this can greatly simplify offline learning in this situation, at the expense of only approximating a local optimum, as in online learning.

While some systems aim to enhance DNNs to classify shots, they do so by adding layers to the output of a pre-trained network to customize it for a specific task. The pre-trained network extracts many levels of increasingly abstract features, such as from an image, and additional trainable layers are used to focus on the problem of interest. In contrast, many of the systems and methods described herein work in a much different way, achieving a more efficient, much faster, and more accurate system. The systems described herein often only add a single layer to the input of a pre-trained network. In some use cases, the camera is pointed at the target rather than focused on the shooter, and the camera and microphone pick up other shots without being able to natively determine that the gunfire came from the shooter of interest. Acoustic signatures are generated quickly and are often performed on a mobile device (e.g., a smartphone) that includes the camera and microphone. The mobile device may execute instructions (e.g., an application) that include the components and systems described herein so that classification and acoustic signature determination are performed on the mobile device. In some cases, the pre-trained network may be trained on a relatively comprehensive universe of shots. The trainable input layer may pre-distort the input data to achieve high-probability recognition by the pre-trained network of a particular firearm in the munitions and environment. This may increase the likelihood of detecting shots of interest and rejecting all others.

According to some embodiments, the systems and methods described herein converge to a useful weighting mask where the gradient of the function computed by the DNN is monotonically non-decreasing or monotonically non-increasing at each input.

Referring to FIG. 28, which illustrates pre-training a DNN, process 2800 begins 2802, where an audio file is opened in block 2804, which may be the first audio file or the next or subsequent audio file. Using the audio file, the system captures blocks of samples framing the first and/or next shot in step 2806. In other words, each shot is windowed in time-bounded samples. In block 2808, spectrograms associated with the samples are generated and labeled with attributes.

In block 2810, the system determines whether the most recent sample is associated with the last shot. If not, the system returns to block 2806 to capture a block of samples associated with a subsequent shot. If so, the system proceeds to block 2812 to determine whether the labeled spectrogram created in block 2808 is the last file. If not, the system returns to block 2804 to open or capture the next audio file. If the system determines that the most recent file is the last file, the system proceeds to block 2814, where the labeled spectrograms are aggregated. In block 2816, a DNN is trained with the labeled spectrograms. The system stops at block 2818 with a trained DNN.

Referring to FIG. 29, which illustrates deriving spectrogram weights, process 2900 begins at 2902, where a set of shots are captured at block 2904. The shots may be captured by an audio and/or video recording device or may involve opening a file associated with one or more shots. At block 2906, the system captures a block of samples framing the first and/or next shot. In other words, each shot is windowed in time-bounded samples. At block 2908, spectrograms associated with the samples are created and labeled with attributes.

In block 2910, the system determines whether the most recent spectrogram is associated with the last shot. If not, the system returns to block 2906 to capture a block of samples associated with a subsequent shot. If so, the system proceeds to block 2912 to determine whether the labeled spectrogram created in block 2908 is the last set. If not, the system returns to block 2904 to capture the next set of shots. If the system determines that the most recent file is the last file, the system proceeds to block 2914, where the labeled spectrograms are aggregated. In block 2916, the system updates W (weighting) until the value converges. The system stops at block 2918 with the spectrogram weightings obtained.

Referring to FIG. 30, a process for classifying a shot 3000 is shown. The process begins at block 3002, where in block 3004 the system captures a block of samples framing the shot. In block 3006, the system determines a spectrogram associated with the block of samples. In block 3008, the system determines the Hadamard product of the spectrogram and a weight matrix. In block 3010, the Hadamard product of the spectrogram and the weight matrix is applied to a DNN. In block 3012, the system makes a binary decision as to whether the shot is associated with the signature of the firearm in question. In some cases, the system can identify the type of firearm and the ammunition fired via the firearm. For example, upon receiving an audio sample, the system can determine, without prior knowledge of the firearm in question, that the acoustic signature of the firearm in the audio sample corresponds to a 230-grain round-nosed projectile fired from a Beretta .45 ACP. At block 3014, the process stops.

The system may include one or more processors and one or more computer-readable media that may store various modules, applications, programs, or other data. The computer-readable media may include instructions that, when executed by one or more processors, cause the processors to perform the operations described herein for the system.

In some implementations, the processor(s) may include a central processing unit (CPU), a graphical processing unit (GPU), both a CPU and a GPU, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or additionally, the functions described herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SoCs), complex programmable logic devices (CPLDs), and the like. Additionally, each of the processor(s) may possess its own local memory, which may also store program modules, program data, and/or one or more operating systems. One or more control systems, computer controllers, and remote controls may include one or more cores.

Automated Target Scoring

In some cases, the described systems operate in near real time on a single consumer recording device, such as a mobile phone, using only a modest amount of training data. As used herein, the terms "real time" or "near real time" are broad terms that, in the context of this disclosure, relate to receiving input data, processing the input data, and outputting the results of data analysis with little or no human-perceived latency. In other words, a system such as that described herein that outputs analyzed data within less than one second is considered near real time. A system that operates in real time or near real time may limit the amount of computation for machine learning, or at least training models, that the method can use to characterize specific target acquisition, classification, and scoring.

Previous approaches to automated target scoring have relied on acoustic triangulation, optical triangulation, and piezoelectric sensor triangulation. Acoustic triangulation has been attempted by using acoustic chamber targets, which use the projectile's Mach waves to determine the projectile's location as it passes through the target. Acoustic triangulation automated scoring systems operate by using microphones to measure the projectile's sound waves as it passes through the target. The sound of the projectile passing through the target from multiple audio sensors (e.g., microphones) can then be used to determine the location where the projectile passed through the target.

Optical triangulation automated scoring systems use three or more lasers, such as infrared lasers, to triangulate the projectile's position as it passes through the target. Piezoelectric sensor triangulation systems rely on an array of piezoelectric sensors on a plate that sense vibrations caused by the projectile impacting the target.

FIG. 31 illustrates a system 3100 configured to automatically identify targets, classify targets, determine target impacts, and score firing strings. The system 3100 may include a computing resource 3102, which may be a mobile computing device associated with a shooting range participant and may include any one or more of several mobile computing devices, such as a smartphone, tablet computer, laptop computer, or other suitable computing device. The computing resource 3102 typically includes one or more processors 3104 and a memory 3106 that stores one or more modules 3108. The modules 3108 may store instructions that, when executed, cause the one or more processors 3104 to perform various operations. The computing resource 3102 may further include data storage, which may be remote storage, such as a remote server or cloud-based storage system, or local storage, or a combination thereof. Data storage can store DOPE (Data On Previous Engagement), which can enable data tracking over time, as well as comparative data between different shooters, different firearms, different ammunition, different targets, different environments, etc.

The storage system can further enable historical trend analysis, which can be used to show a shooter's performance over time, including tracking improvements. Data storage can also be analyzed to provide performance predictions, rankings, and social features, among other benefits.

The system may incorporate one or more imaging sensors 3110, such as any suitable video camera. In some cases, the imaging sensor 3110 may be associated with a computing resource 3102. For example, in some embodiments, the computing resource 3102 may be a smartphone with a built-in camera 3110.

The camera 3110 may be aimed to capture an image of the target 3112. The target may be located at any distance from the shooter, and the camera 3110 may be aimed and/or zoomed to capture an image of the target. In some embodiments, the camera may be coupled to a lens, such as a spotting scope or camera lens, allowing the camera to capture images closer to the target through optical or digital zoom.

The computing resources 3102 may include, among other things, instructions (e.g., module 3108) that enable the computing device to initialize 3118 targets, detect 3114 target impacts, and score 3116 target impacts.

Figure 32 shows a decision tree 3200 configured to detect, identify, and classify targets. According to some embodiments, the system does not know the type of target before the system begins searching for the target. For example, in some conventional systems, the scoring system may be pre-programmed with the target against which the shooter will aim. This allows the system to easily understand the size and shape of the target, as well as the location and boundaries of each scoring ring or zone. In the illustrated embodiment, the system is configured to automatically determine and determine the scoring rings and zones without a priori data regarding the type of target. For example, the system may have one or more video capture devices that may be integrated into one or more mobile computing devices. As used herein, a mobile computing device may be one or more of a mobile phone, smartphone, tablet, laptop, personal digital assistant, smart glasses, body cam, wearable computing device, or any other computing device that a user may carry to a shooting range.

The mobile device may activate the camera and capture one or more frames of the target 3202. The computing device may have instructions to analyze the one or more frames and identify the target within the one or more frames by using any suitable image analysis algorithm. If the target is detected and classified in block 3204, the target is registered with the system 3206, and a scoring ring and area are determined. The system may capture additional image frames containing the target and look for differences between one frame and the next that may correlate with impact on the target. The frames may be compared, and in block 3208, a moving average may be generated. Moving averages are fundamental mathematical and statistical techniques applied in image analysis and machine learning for a variety of purposes, including noise reduction, feature extraction, and trend analysis. They involve calculating the average value of pixel intensities or other data points within a moving window or kernel across an image or dataset. Moving averages can be used to extract meaningful features from images. For example, important information can be highlighted by sliding a small window over the image and calculating the average pixel value within that window. For example, in edge detection, a moving average can highlight areas with abrupt changes in pixel intensity, helping to identify the edges or boundaries of the target and scoring area. Edge detection can also be used to identify bullet impacts on the target.

In some instances, moving averages are used in time series data analysis. As one example, moving averages can be used to establish a baseline behavior of a system in order to detect anomalies. Data points that deviate significantly from this baseline can be flagged as anomalies or outliers. These anomalies can be further analyzed to determine target impact.

Once the sequential moving averages are generated, they may be combined into a long-term moving average. In block 3210, the moving average image may be compared to the long-term moving average to determine differences from one frame to a subsequent frame that indicate changes to the target that are most likely associated with impact on the target.

In block 3212, the impacts are selected and classified. For example, the system determines the boundaries of a scoring ring, determines the location of each impact, and associates the location of each impact with a score for the impact.

Returning to block 3204, if a target has not previously been detected and classified, for example, if the shooter is initializing the system or has replaced a target, the system determines whether a target has been detected in block 3214. If not, in block 3216, the system proceeds to detect the target. If a target has been detected, the system classifies the target in block 3218, for example, by identifying the target boundaries, scoring ring boundaries, and scoring ring values.

If the system does not detect the target, the system may capture one or more additional image frames and analyze the one or more additional image frames to determine that the target is located within the field of view of the imaging device. Once the target is detected, the system may classify the target and determine its size, as well as the relative location and size of the scoring ring or region.

Figure 33 further describes the initial steps a system may take to identify and classify a target 3300 by analyzing one or more image frames. Object detection is a computer vision technique that involves identifying and locating multiple objects within an image or video stream. Unlike image classification, which determines the presence of a single object class in an entire image, object detection provides a more granular understanding by not only recognizing objects but also specifying their location through bounding boxes. In some embodiments, object detection algorithms typically output bounding boxes that enclose detected objects. These bounding boxes consist of the coordinates (x, y) of the object's upper-left corner and dimensions (width and height) that define the object's spatial extent within the image.

In block 3302, the system applies object detection to one or more images of the target to search for the target. In some embodiments, the object detection model is generic with respect to the target, allowing the system to detect any target regardless of size or shape. In block 3304, if the target is found in the same location in subsequent images (e.g., two or more images, three or more images, four or more images, etc.), the system assumes the target has been located and defines a bounding box around the target. In some cases, finding the target in the same location in subsequent images includes determining a moving average of the images to determine the target's location, size, and shape.

At block 3306, the targets are optionally classified. In addition to locating objects, the system may be configured to detect objects and classify each detected object into a predetermined class or category. This allows the system to distinguish between different object types, such as circular targets, oval targets, rectangular targets, silhouette targets, etc.

A target classifier can be applied to the images within the bounding box. Thus, the system determines which reference target image to apply.

In block 3308, the system registers the target to the reference target image. In some cases, this involves applying a contrast adjustment to the image. This may also involve iteratively modifying the initial bounding box, such as by adjusting its corners and then projecting the adjusted bounding box onto the reference target image. The difference between the two may be applied as a score, and a hill-climbing technique may be applied to find the optimal corner, which can be correlated with the initial position in the image. Hill-climbing is an optimization algorithm used to find the local maximum (or minimum) of a given objective function. By iteratively taking small steps in the direction of higher values, the algorithm determines the highest and lowest values, which can then be used to determine the boundary of the target. In some cases, object detection is combined with semantic segmentation to provide pixel-level object masks. This allows for a more accurate understanding of object boundaries in the image, such as target boundaries and scoring ring boundaries.

In block 3310, the system initializes and registers the target and begins looking for impacts over the subsequent moving average.

Figure 34 illustrates a process 3400 for registering a target 3300 and determining target impacts. In block 3402, the target may be re-registered, such as by performing a hill-climbing technique to search for a new set of best corners for the target. In some cases, the hill-climbing search uses mean-squared distance in a perceptual space technique. For example, mean-squared distance, also known as mean-squared perceptual error, is a metric used to measure the similarity or dissimilarity between two data points that contain perceptual data, such as target corners. In some cases, for each data point, relevant perceptual features (in this case, target corners, edges, scoring rings, etc.) are extracted. Features can be visual descriptors that can be represented as vectors of perceptual features. These feature vectors capture relevant information about each data point in a more concise and informative manner.

The mean squared distance between two points (represented as their respective feature vectors) is generated by determining the squared difference between corresponding features and calculating the average of these squared differences. The resulting mean squared distance provides a quantitative measure of the dissimilarity between two data points in perceptual space.

In block 3404, the system may apply a transformation matrix that may be used to map the set of corners to an image. In some cases, the image to which the coordinates are mapped has dimensions of 160 pixels, and in other cases, it may be less than 160 pixels.

In block 3406, the moving average image is updated, with the long-term moving average being approximately 10 seconds or more and the short-term moving average being approximately 0.1 seconds. In some cases, the video camera may capture more than 30 frames per second. For the short-term moving average, this equates to averaging approximately 3 frames to determine the short-term moving average.

In block 3408, the system determines the difference between the moving averages. For example, a long-term moving average may be associated with a static target that has not changed for 10 seconds or so and compared to a short-term moving average that reflects changes in the image. Thus, the difference between the short-term and long-term moving averages highlights changes to the image, such as a bullet hitting the target. The system may convolve any difference image with a simple impact kernel, which may be a 5x5 uniform weight square kernel in some cases, to find the location of the largest block in the difference image. A kernel typically refers to a convolution filter that may be used to process and modify pixel values for feature extraction, etc. The square kernel may be convolved (or moved) across the image, and at each location, the value of the kernel may be multiplied by the pixel values in the corresponding neighborhood, and the results may be summed to generate a new pixel value in the output image. Naturally, the size of the kernel may be varied to adjust the range of the neighborhood considered during the convolution, and may include any of the full set of aperture kernels with non-uniform weights, and may have any suitable size.

The convolution returns a set of potential hits on the target. The set of potential hits may be further filtered, such as by using simple statistics of a window surrounding the flagged difference. In some cases, the window is chosen to be a 16x16 window with the difference in the center of the window. Of course, other window sizes are entirely reasonable, and the pixel values described herein are merely illustrative of some embodiments. The system may also apply some business rules to the windowed differences; for example, the system should not detect multiple hits in the exact same location.

At block 3410, a hit on the target is determined. Optionally, this is accomplished by passing the filtered set of differences to a hit classifier for scoring. If the difference score exceeds a threshold, the location is marked as a hit, and another window may be placed around the hit. Optionally, a 10x10 window is placed around the hit location in the long-term average, and the short-term average is used over 5-10 subsequent frames. This ensures that the same hit is not detected again. Thus, the difference is windowed with a first window, and if the difference exceeds a threshold score, the difference is windowed with a second window smaller than the first. The windowed difference associated with the short-term moving average may be added to a long-term moving average of at least 5 frames, or at least 6 frames, or at least 10 frames, or at least 12 frames, or at least 15 frames or more. Optionally, if the difference score falls below a threshold, the difference is marked as a false hit, and the system does not need to evaluate and classify it again.

According to some embodiments, the system may receive audio data associated with a shot being fired and determine that a shot has been fired based on the audio data. In some cases, the audio data is correlated with a target image, and the system may convolve a differential target image in response to audio data indicating that a shot has been fired. In some cases, the system may not need to continuously convolve a differential image. In this case, the system may determine that a shot has been fired through the audio data, then update a short-term running average and convolve the differential image to look for the shot. In some cases, the system is configured to distinguish shots fired by a user aiming at a target from other shooters on the shooting range. In this way, the system can know when a shooter of interest is about to fire a shot, even if other active shooters are on the range.

In some cases, audio data can be used in impact detection, such as by correlating the audio of a fired shot with impacts appearing on a target image.

Figures 35A-35C illustrate and describe initializing the scoring system by identifying and classifying a target. In some cases, the system can automatically determine the target's boundaries, but in some embodiments, user input may define the target's boundaries. For example, using a human-computer interface (e.g., a touchscreen, mouse, stylus, touchpad, etc.), a human can draw a boundary around the target to help the system identify it. However, in many embodiments, the system uses machine vision to identify the target and its boundaries. Figure 35A shows an image 3500 captured by a camera associated with the system. The image may include a target stand 3502, a target 3504, a target fixation clip 3506, and other features within the field of view. The system may determine an initial bounding box 3508 around the identified target, such as by a trained target detection model. In some embodiments, a user may define the initial bounding box, such as by drawing it on a computer display using the human-computer interface. The human-computer interface may be any suitable interface, possibly a touchscreen, pen, mouse, trackball, etc. The initial bounding box may not exactly match the edges and corners of the target, especially if the bounding box is user-defined. The initial bounding box and target image may be referred to as an initialization frame. The initialization frame may be converted to Lab color space, which includes components for lightness, a green-to-red axis, and a blue-to-yellow axis, to produce perceptual uniformity. In some cases, the luminance channel is equalized via contrast-limited adaptive histogram equalization (CLAHE).

Figure 35B shows a target whose coordinates have been determined as described above, often representing a quadrilateral that can be projected onto a reference target image 3510. The reference target image 3510 can also be converted to Lab color space, and a squared difference (in Lab space) can be generated between the projection and the target. A hill-climbing algorithm can be applied to the coordinates, where a possible delta is a small change to the coordinate, and a better solution can be determined by the squared perceptual difference.

Figure 35C shows the best coordinates, such as determined by minimum difference over several random restarts of the hill climbing algorithm. These coordinates can then be used to apply an updated bounding box 3512. Thus, even if the target image is distorted, such as when the viewing angle from the camera makes the target appear as a parallelogram rather than a rectangle, the initial bounding box can be modified to match the shape of the target as presented in the image captured by the camera.

In some embodiments, the system may define the edges of the target through image analysis. However, in some cases, the edges of the target are irrelevant; only the scoring rings are important. Thus, in some cases, the system is configured to identify the scoring rings and is not concerned with the target boundary. Additionally, the system only needs to identify the scoring rings, rather than classifying the target. For example, the system may determine through one or more machine learning models that a target represents a central bullseye target with a series of scoring rings. The system may assign a score value to each ring, such as 10 points for the bullseye, 9 points for the next larger ring, and so on. Similarly, the system may identify a target with five bullseye-sized circles spaced throughout the target and assign a value of 10 points to each of these scoring rings. One or more of the multiple bullseye-sized rings may have radially spaced larger scoring rings that may be assigned a smaller value than the bullseye-sized rings. Thus, the system need omit the step of classifying the target and only focus on the size and location of the scoring rings.

Figure 36 illustrates and describes impact detection and scoring of detected impacts. A machine learning model may be implemented to determine whether a difference between image frames is likely to be a projectile impact on the target 3504. The target 3504 may be re-registered, such as by applying a hill-climbing technique using possible corner coordinates, as in the target initialization step. The differences are windowed (3602a, 3602b, 3602c) by comparing the short-term moving average difference with the long-term moving average to generate a difference image 3612 between the current target and the long-term target average.

The difference image may be convolved with an impact kernel (e.g., a windowed kernel that scans the image difference). Any point that is several standard deviations away from the long-term exponential moving average (EMA) of the maximum convolution value is flagged as a possible impact 3604a, 3604b, 3604c.

The potential hits 3604a-3604c are fed to a machine learning model (e.g., a classifier) 3606 that determines whether the difference is likely to be an actual hit. If the difference is above a threshold, the system marks the difference as an actual hit 3608. However, if the difference is below the threshold, the system marks the difference as a false hit 3610.

FIG. 37 illustrates and describes scoring of impacts on a target 3504. Different scoring zones may be determined by the system based on computer vision, by referencing registered targets from previous shooting sessions, by retrieving stored target models from a known target database, or in some other manner. Scoring zones 3702 on the target may be represented as connected areas of one or more simple shapes (e.g., ellipses, rectangles, circles, triangles, etc.). Coordinates of detected impacts 3704 may be normalized and transformed to axes implied by a reference image. In other words, the impacts may be overlaid on a reference image, which can be used for the impact's coordinates. The coordinates may be Cartesian coordinates expressed as x, y values. In some cases, the coordinates may be radial coordinates, representing the impact as an angle and distance from, for example, the center of the target. The system can then determine whether the impact is entirely within a single scoring zone or penetrates a scoring zone boundary, allowing the system to accurately score the impact. The system can use simple geometric shapes to determine whether a significant portion of a given impact falls within any of the simple shapes in each target zone.

In some embodiments, the system uses the coordinates of the shots for further analysis. For example, by generating and storing the coordinates for a given firing string, grouping can be quantified and used as a measure of improvement over time. Similarly, a shooter's MOA (moment of angle) is a measure of group size in inches and arc minutes and can be determined from center to center and edge to edge. Additionally, grouping may be used to define pose, grip, or motion errors during a firing string. Grouping may be quantified, including group size, group rotation, or other metrics.

Scoring can be quantified by any suitable metric. In some cases, scoring is point-based, with points awarded to each zone on the target being added to or subtracted from an initial amount. In some cases, missed shots or extra shots fired are scored as negative or higher values, depending on the scoring type. In some cases, timed scoring is used, where total time is reflected in the score, with time being increased as a penalty for misses. In some cases, group size is used to determine scoring, with extra or missed shots penalizing group size. Of course, other metrics and combinations of metrics may be determined by the system to score a particular firing string.

The system may be configured to return the coordinates of impact and the time each shot occurred for each shot in a firing string, allowing the firing string to be ranged using some metric combining position and time. For example, the time between shots may be measured, or shots after a buzzer or other initiation signal may be tracked and stored along with accuracy metrics.

In some cases, once impacts are identified, the system may draw a bounding box around one or more of the impacts. When the firing string ends, the system may draw a bounding box containing each of the shots in the group and determine a metric based on the bounding boxes to determine the score.

As shown in FIG. 38, which illustrates a user interface 3800 of a system developed and operable in accordance with some of the embodiments described herein, the system may be configured to determine a bounding box 3802 that may pass through the center of the outermost impact or along the edge of the impact. The system may determine any of several metrics, such as, but not limited to, group size 3804, group overall width 3806, group height 3808, bounding box rotation angle, MOA, elevation offset 3810, windage offset 3812, and may further determine a shot distance 3814, which may be manually entered or determined based on the detected projectile's time of flight.

For example, the system may be configured to register the sound of gunfire, shock waves from projectile or gunpowder explosions, firearm or shooter movement, or some other indicator that a shot has been fired. The system may then detect when impact on the target occurs, determine the ammunition's time of flight, and determine target distance based on the firearm, ammunition, and/or propellant. This process can be performed in near real time by a simple consumer-grade mobile computing device. In some cases, the mobile computing device may utilize the zoom capabilities of an on-board image capture device. In some cases, an external zoom lens may be used to capture image frames by the mobile computing device. For example, a mobile phone may be coupled to a spotting scope, which provides optical zoom through the spotting scope, allowing the mobile computing device to capture clearer images of targets that may be downrange. Some mobile computing devices may rely on digital zoom to capture one or more images of targets located downrange.

The system may further determine and display the number of shots fired in the current shooting string 3816, the average split time for each shot 3818, which may be useful in timed shooting competitions. The system may further display the score 3820 associated with each shot and the cumulative score for the shooting string 3822.

Some embodiments further provide an automated scoring system capable of quickly identifying targets, classifying targets, including identifying target scoring rings, and scoring target hits at a shooting range. In some cases, the system is stored and executed on a consumer mobile computing device (e.g., an iPhone, tablet, phone, video camera). In some cases, the system includes a video camera device pointed at a target of interest, and the system is configured to identify the target, classify the target, determine shot impacts on the target, and score the target impacts. In some cases, the system is configured to prompt the shooter regarding the shooting stage. For example, the system may be configured for use in a CMP high-powered rifle competition, and the system may prompt the user that the current stage requires firing 20 shots downrange from an off-hand position within a 20-minute time frame. In some cases, the system knows the number of shots expected during the shooting stage (called a firing sequence) and may prompt the user with information related to the current shooting stage, such as the number of shots, the time frame, and the shooting location. In some cases, the shooter may input information associated with the shooting stage, such as the number of shots the system should expect, the firearm to be used, and the distance to the target, among other things. In some cases, the system is started and stopped manually, identifying shots on target only during the time the system is started.

Comprehensive Training System

As described above, in some embodiments, the system utilizes a gantry-style arrangement that may use multiple recording devices to capture audio and video of participants and/or targets. In some cases, an imaging device may be aimed at the target, which may have a zoom lens, digital zoom capabilities, or rely on an external lens, such as a camera mounted on a spotting scope, to capture images of targets located downrange. In some cases, the system is configured to synchronize multiple sources, such as one or more video frames and/or audio data from one or more audio capture devices. In some cases, the system is configured to synchronize multiple video frames and audio data from one or more audio/video capture devices.

The described system therefore provides a comprehensive firearms training solution that allows shooters, even in mixed ranges, to track body movements, identify errors in technique, correlate scores with errors in technique, automatically score targets, and distinguish shots fired from a participant's firearm from those from other firearms.

The system may utilize one or more machine learning models for synchronization, prediction, and validation, and may be further trained to analyze scoring data and associate it with motion data to determine correlations between specific motion data (e.g., behaviors) and scoring trends. As an example, the system may correlate a shooter's wrist pivoting downward before a shot with typically scoring outside and below the ten-ring and determine that the shooter is anticipating firearm recoil before the shot. The system may then provide feedback to the user with information related to the motion/score correlation, as well as one or more exercises or drills to enable the user to recognize and address behaviors that result in reduced scores. The system may be further trained to distinguish firearm fires associated with a shooter of interest, even when in a mixed range with multiple shooters. As described elsewhere herein, a similar process may be used for any motion data from any activity or sport.

In some embodiments utilizing multiple video capture devices, the system may track body motion in two or three dimensions from multiple angles. The two or three dimensional body motion data may be correlated, synchronized, and analyzed to determine two or three dimensional motion data that may be further correlated with the resulting score.

While the described system embodiments are described in the context of a shooter firing a series of shots, it should be understood that the systems and methods described herein are applicable to capturing any type of physical motion, as well as other sports where physical motion can lead to performance metrics. For example, the system embodiments described herein can record the motion of a basketball free throw, a golf swing, a figure skating element, archery, soccer, a baseball swing, or a series of observable physical landmarks over time, and can be used to track, critique, and improve physical motion, such as any other sport or movement, where the motion has any observable causal consequences.

The system may include one or more processors and one or more computer-readable media that may store various modules, applications, programs, or other data. The computer-readable media may include instructions that, when executed by one or more processors, cause the processors to perform the operations described herein for the system.

In some implementations, the processor(s) may include a central processing unit (CPU), a graphical processing unit (GPU), both a CPU and a GPU, a microprocessor, a digital signal processor, or other processing units or components known in the art. Alternatively or additionally, the functions described herein may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SoCs), complex programmable logic devices (CPLDs), and the like. Additionally, each of the processor(s) may possess its own local memory, which may also store program modules, program data, and/or one or more operating systems. One or more control systems, computer controllers, and remote controls may include one or more cores.

Embodiments may be provided as a computer program product including a non-transitory machine-readable storage medium on which are stored instructions (in compressed or uncompressed format) used to program a computer (or other electronic device) to perform the processes or methods described herein. Computer-readable media may include volatile and/or non-volatile memory, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Machine-readable storage media include, but are not limited to, hard drives, floppy disks, optical disks, CD-ROMs, DVDs, read-only memory (ROM), random-access memory (RAM), EPROMs, EEPROMs, flash memory, magnetic or optical cards, solid-state memory devices, or other types of media/machine-readable media suitable for storing electronic instructions. Furthermore, embodiments may be provided as a computer program product including a transitory machine-readable signal (in compressed or uncompressed format). Examples of machine-readable signals include, but are not limited to, signals, whether or not modulated using a carrier wave, that can be configured to be accessed by a computer system or machine that hosts or executes a computer program (including signals downloaded over the Internet or other network).

Those skilled in the art will recognize that any process or method disclosed herein may be modified in many ways. The process parameters and order of steps described and/or illustrated herein are given by way of example only and can be changed as desired. For example, although the steps illustrated and/or described herein may be shown or described in a particular order, these steps do not necessarily have to be performed in the order shown or described.

The various exemplary methods described and/or illustrated herein may omit one or more of the steps described or illustrated herein or may include additional steps in addition to those disclosed. Furthermore, the steps of any method as disclosed herein may be combined with any one or more steps of any other method as disclosed herein.

This disclosure describes exemplary embodiments and is therefore not intended to limit in any way the scope of the embodiments of the present disclosure and the appended claims. The embodiments are described above using functional components that illustrate implementations of specified functions and their relationships. The boundaries of these functional components have been arbitrarily defined herein for convenience of description. Alternate boundaries may be defined to the extent that the specified functions and their relationships are appropriately performed.

The foregoing description of specific embodiments sufficiently clarifies the general nature of the disclosed embodiments so that others, by applying the knowledge of those skilled in the art, can readily modify and/or adapt such specific embodiments for various applications without undue experimentation and without departing from the general concepts of the disclosed embodiments. Therefore, such adaptations and modifications are intended to be within the meaning and range of equivalents of the disclosed embodiments, based on the teaching and guidance presented herein. The phrases or terminology used herein are for the purpose of description, not limitation, and should be interpreted by those skilled in the art in light of the teaching and guidance presented herein.

The breadth and scope of the embodiments of the present disclosure should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.

In particular, conditional language such as "can," "could," "might," or "may," unless expressly stated otherwise or understood otherwise within the context in which it is used, is generally intended to convey that certain implementations include certain features, elements, and/or operations, but may not include them in other implementations. Thus, such conditional language is generally not intended to imply that features, elements, and/or operations are somehow required in one or more implementations, or that one or more implementations necessarily include logic for determining whether those features, elements, and/or operations should be included or performed in any particular implementation, with or without user input or prompting.

Unless otherwise noted, when used herein, the terms "connected to" and "coupled to" (and their derivatives) should be interpreted as allowing for both direct and indirect (i.e., via other elements or components) connections. Additionally, when used herein, the terms "a" or "an" should be interpreted as meaning "at least one of." Finally, for ease of use, when used herein, the terms "including" and "having" (and their derivatives) should be interpreted as open-ended and not excluding additional components.

This specification and the accompanying drawings disclose examples of systems, apparatus, devices, and techniques that may provide systems and methods for determining the acoustic signature of a fired firearm. It is, of course, not possible to describe every conceivable combination of elements and/or methodologies for purposes of describing various features of the present disclosure, but those skilled in the art will recognize that many further combinations and permutations of the disclosed features are possible. Accordingly, various modifications can be made to the present disclosure without departing from the scope or spirit thereof. Moreover, other embodiments of the present disclosure will be apparent from consideration of the specification and accompanying drawings, as well as from practice of the disclosed embodiments presented herein. The examples presented in this specification and the accompanying drawings are to be considered in all respects as illustrative and not restrictive. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

Those skilled in the art will understand that in some implementations, the functionality provided by the processes and systems described above may be provided in alternative ways, such as divided among more software programs or routines or integrated into fewer programs or routines. Similarly, in some implementations, the illustrated processes and systems may provide more or less functionality than described, such as when other illustrated processes, respectively, lack or include such functionality, or when the amount of functionality provided is varied. Additionally, while various operations may be shown as being performed in a particular way (e.g., sequentially or simultaneously) and/or in a particular order, those skilled in the art will understand that in other implementations, these operations may be performed in other orders and in other ways. Those skilled in the art will also understand that the data structures described above may be structured differently, such as by dividing a single data structure into multiple data structures or by combining multiple data structures into a single data structure. Similarly, in some implementations, the illustrated data structures may store more or less information than described, such as when other illustrated data structures, respectively, lack or include such information, or when the amount or type of information stored is varied. The various methods and systems shown in the figures and described herein represent example implementations. In other implementations, the methods and systems may be implemented in software, hardware, or a combination thereof. Similarly, in other implementations, the order of any method may be changed, and various elements may be added, rearranged, combined, omitted, modified, etc.

From the foregoing, it will be understood that, although specific implementations have been described herein for illustrative purposes, various modifications may be made without departing from the spirit and scope of the appended claims and the elements described therein. In addition, while certain aspects are presented below in certain claim forms, the inventors contemplate various aspects in any available claim form. For example, while only some aspects may currently be described as being embodied in a particular configuration, other aspects may likewise be so embodied. Various modifications and changes may be made, as would be apparent to one skilled in the art having the benefit of this disclosure. All such modifications and changes are intended to be encompassed, and therefore the above description should be regarded in an illustrative rather than a limiting sense.

Claims (17)

1. A method for improving shooting performance, comprising:
receiving video data of the shooter;
determining one or more body landmarks of the shooter;
tracking the one or more body landmarks during a shot to generate shot motion data;
determining a score for said shot;
Associating the shot motion data with the score;
generating recommendations for altering the motion data in subsequent shots. The method of claim 1, wherein determining one or more body landmarks of the shooter includes generating a wireframe model by connecting the body landmarks. The method of claim 1, wherein associating the shot motion data with the score includes executing a classification and regression tree machine learning model to identify a causal relationship between the shot motion data and the score. The method of claim 1, further comprising determining the shooter's grip through image analysis of the video data of the shooter. The method of claim 4, further comprising analyzing the shooter's grip and providing, on a display screen, grip recommendations for changing the grip. Determining the score for the shot comprises:
receiving target video data;
performing image analysis on the received target video data;
determining a hit on the target;
and determining a score for the hit. The method of claim 1, wherein receiving the video data includes capturing the video data with a mobile phone. The method of claim 1, wherein determining one or more body landmarks includes determining 17 body landmarks. The method of claim 1, wherein tracking the one or more body landmarks includes generating a bounding box around each of the one or more body landmarks. The method of claim 1, further comprising running a machine learning model to correlate the shot motion data with the score. The method of claim 10, wherein the machine learning model is configured to determine the motion data that results in an off-center hit on the target. 1. A method for improving the causal consequences of bodily movement, comprising:
receiving video data of the body movement;
determining one or more body landmarks visible in the video data of the body movement;
tracking the one or more body landmarks during an action;
generating motion data based at least in part on tracking the one or more body landmarks;
determining a score associated with the motion data;
associating the performance data with the score;
generating recommendations for altering the operational data in subsequent actions. The method of claim 12, wherein receiving the video data includes capturing the video data with a mobile computing device. The method of claim 12, further comprising running a machine learning model to correlate the behavioral data with the score. The method of claim 14, wherein the machine learning model is configured to determine the behavioral data that results in a reduced score. The method of claim 15, further comprising predicting a prediction score based on the behavioral data using the machine learning model. The method of claim 16 , further comprising comparing the predicted score to the score.