CN120658945B - A shooting adjustment method and shooting robot based on AI intelligent optimization - Google Patents
A shooting adjustment method and shooting robot based on AI intelligent optimizationInfo
- Publication number
- CN120658945B CN120658945B CN202511157257.2A CN202511157257A CN120658945B CN 120658945 B CN120658945 B CN 120658945B CN 202511157257 A CN202511157257 A CN 202511157257A CN 120658945 B CN120658945 B CN 120658945B
- Authority
- CN
- China
- Prior art keywords
- shooting
- frame
- picture
- motion
- route
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/682—Vibration or motion blur correction
- H04N23/684—Vibration or motion blur correction performed by controlling the image sensor readout, e.g. by controlling the integration time
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/64—Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/681—Motion detection
- H04N23/6811—Motion detection based on the image signal
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/681—Motion detection
- H04N23/6812—Motion detection based on additional sensors, e.g. acceleration sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
- H04N7/0127—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
Abstract
本发明公开了一种基于AI智能优化拍摄调整方法及拍摄机器人，涉及智能拍摄技术领域，包括：对高速运动的拍摄主体自动跟随拍摄过程中，获取帧率变动视频段；提取帧率变动视频段的首帧拍摄图片和尾帧拍摄图片，在首帧拍摄图片和尾帧拍摄图片中分别提取拍摄主体的首帧相对位置数据和尾帧相对位置数据；基于AI模型生成拟合运动路线；在帧率变动视频段内划分若干段视频调整段，基于AI模型生成若干段视频调整段对应的子拟合路线；整合每段子拟合路线和拟合运动路线进行分析，在每段视频调整段内插入一张以上的补偿帧图片。通过生成运动轨迹拟合路线，结合多段视频调整段的子拟合路线进行动态插帧补偿，提升高速运动场景下视频流畅度与画面质量。
The present invention discloses a shooting adjustment method and a shooting robot based on AI intelligent optimization, which relate to the field of intelligent shooting technology, including: obtaining a frame rate change video segment during the automatic follow-up shooting process of a high-speed moving shooting subject; extracting the first frame shot picture and the last frame shot picture of the frame rate change video segment, and extracting the first frame relative position data and the last frame relative position data of the shooting subject in the first frame shot picture and the last frame shot picture respectively; generating a fitting motion route based on an AI model; dividing the frame rate change video segment into several video adjustment segments, and generating sub-fitting routes corresponding to the several video adjustment segments based on the AI model; integrating each sub-fitting route and the fitting motion route for analysis, and inserting one or more compensation frame pictures in each video adjustment segment. By generating a motion trajectory fitting route and combining the sub-fitting routes of multiple video adjustment segments for dynamic frame insertion compensation, the video smoothness and picture quality in high-speed motion scenes are improved.
Description
Technical Field
The invention relates to the technical field of intelligent shooting, in particular to an AI-based intelligent optimization shooting adjustment method and a shooting robot.
Background
The existing sports shooting video especially relates to shooting of sports ball games, and shooting equipment is required to have shooting effects of high precision and high stability. Because the shooting capture to the player and the sphere is required to be maintained in the shooting process, the shooting equipment is required to move along with the player or the sphere to shoot, so that the condition of frame dropping easily exists in the shooting process, and the shooting effect is influenced. Particularly, under a fast moving scene, the traditional shooting equipment is difficult to capture the complete track of a moving main body in real time, and the problems of picture jamming, blurring and the like are easily caused. In addition, the processing mode of the prior art for frame rate variation is single, intelligent compensation cannot be performed according to the actual change condition of a moving main body, and the quality of the finally output video is uneven. Meanwhile, the existing shooting robot lacks the capability of real-time detection and self-adaptive adjustment of frame rate change, and is difficult to meet the actual requirements of professional-level high-speed motion high-quality real-time shooting.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, provides an intelligent optimization shooting adjustment method based on AI and a shooting robot, and the intelligent optimization shooting adjustment method based on AI and the shooting robot have the advantages of improving video smoothness and picture quality in a high-speed moving scene by detecting frame rate fluctuation video segments in real time and generating a motion track fitting route and combining sub-fitting routes of a plurality of video adjustment segments to dynamically perform frame interpolation compensation.
The invention provides an AI-based intelligent optimization shooting adjustment method, which comprises the following steps:
a. In the process of automatically following a shooting main body moving at a high speed, detecting the frame rate change of a shooting picture in real time, and acquiring a frame rate change video segment;
b. Extracting a first frame shooting picture and a last frame shooting picture of the frame rate variation video segment, and respectively extracting first frame relative position data and last frame relative position data of a shooting subject from the first frame shooting picture and the last frame shooting picture;
c. generating a fitting motion route by combining the relative position data of the first frame and the relative position data of the tail frame based on the AI model;
d. Dividing a plurality of video adjustment sections in the frame rate variation video section, and generating sub-fitting routes corresponding to the plurality of video adjustment sections based on an AI model;
e. and integrating each sub-fitting route and each fitting movement route for analysis, and inserting more than one compensation frame picture into each video adjustment section.
Further, the step a includes:
setting a detection sliding window to carry out sliding window detection on a real-time shooting picture in the shooting process, and obtaining detection data;
Performing standard deviation calculation on the frame rate of the detection data to obtain a variation index value;
And when the variation index value is larger than a preset detection threshold value, marking the detection data as a frame rate variation video segment.
Further, the step b includes:
b1, selecting a panoramic background picture according to shooting scenes related to the frame rate variation video segment;
b2, extracting a shooting subject from the first frame shooting picture, and acquiring first frame relative position data of the shooting subject in the panoramic background picture;
And b3, picking up the picture at the tail frame to extract the shooting subject, and acquiring the relative position data of the tail frame of the shooting subject in the panoramic background picture.
Further, the step c includes:
c1, analyzing the motion trend of the shooting subject according to the relative position data of the first frame and the relative data of the last frame through an AI model;
c2, inserting a plurality of blank frame pictures between the first frame shot picture and the last frame shot picture according to a preset shooting frame rate, and inserting a shooting main body prediction position point in each blank frame picture by combining the motion trend of the shooting main body;
And c3, constructing a fitting motion route according to the predicted position points of the blank frame pictures.
Further, the step c2 includes:
Taking the panoramic background picture as the image content of a blank frame, and carrying out shooting body position prediction of the next frame picture by combining the motion trend of the shooting body and the relative position data of the shooting body in the current frame picture through an AI model to obtain a shooting body prediction position point of the next frame picture.
Further, the step d includes:
d1, extracting a plurality of shooting reference objects of a frame rate variation video segment, and extracting a plurality of process shooting pictures in the frame rate variation video segment according to the distribution of the shooting reference objects;
d2, dividing the frame rate variation video segments into a plurality of video adjustment segments according to the shooting time sequence of the pictures in a plurality of processes;
d3, acquiring relative position change data of a shooting reference object and a shooting subject of each video adjustment section, and generating a sub-fitting route according to the relative position change data through an AI model.
Further, the step e includes:
e1, dividing the fitted movement route into a plurality of fitted route segments, wherein the fitted route segments correspond to the fitted sub-routes one by one;
e2, determining a frame inserting compensation position point according to the comparison analysis of the fitting route segment and the corresponding fitting sub-route;
and e3, generating a motion estimation value according to the fitted route segment, generating motion sub-data according to the fitted sub-route, and outputting a compensation frame picture according to the motion estimation value and the motion sub-data through an AI model.
Further, the step e3 includes:
analyzing the motion parameters of a shooting subject and a shooting reference object according to the shooting content of each fitted route segment;
comparing the motion parameters with a preset threshold value, and dividing the motion level of the complementary frame;
and selecting a corresponding frame supplementing mode according to the frame supplementing motion level, and generating a compensating frame picture.
Further, the dividing operation of the frame compensating motion level is as follows:
If it is And (2) andDividing the frame-supplementing motion level into high-difficulty frame-supplementing;
If it is And (2) andDividing the frame-supplementing motion level into low-difficulty frame-supplementing;
wherein, the 、Is a coefficient of proportionality and is used for the control of the power supply,In order to capture the rate of movement of the subject,In order to capture the rate of movement of the reference object,In order to photograph the movement direction angle of the subject,To capture the direction angle of motion of the reference object.
The invention also provides a shooting robot based on the AI intelligent optimization shooting adjustment, which is characterized in that the shooting robot is used for executing the AI intelligent optimization shooting adjustment method, and the shooting robot comprises:
The shooting assembly is used for shooting in real time and transmitting shooting pictures to the frame rate processing assembly in real time;
the frame rate processing component is used for detecting the frame rate change of a shot picture and acquiring a frame rate change video segment;
and the frame rate compensation component is used for carrying out frame picture analysis on the frame rate variation video segment to generate a compensation frame picture.
The invention provides an AI intelligent optimization shooting adjustment method and a shooting robot based on the same, which are used for automatically carrying out automatic follow shooting on a shooting main body moving at high speed in the process of automatic follow shooting by capturing states of the shooting main body and a shooting reference object, and carrying out AI intelligent frame supplementing operation of a frame dropping video segment by combining the shooting reference object, so that high-quality shooting video can be output, and a high-efficiency video shooting optimization adjustment effect is realized.
Drawings
FIG. 1 is a flowchart of an AI-based intelligent optimization shooting adjustment method in an embodiment of the invention;
FIG. 2 is a flow chart of a method of step b in an embodiment of the invention;
FIG. 3 is a flow chart of a method of step c in an embodiment of the invention;
FIG. 4 is a flow chart of a method of step d in an embodiment of the invention;
FIG. 5 is a flow chart of the method of step e in an embodiment of the present invention;
Fig. 6 is a schematic diagram of a working system of a shooting robot based on AI intelligent optimization shooting adjustment in an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Embodiment one:
Fig. 1 shows a flowchart of an AI-based intelligent optimization shooting adjustment method in an embodiment of the present invention, where the adjustment method includes steps a to e.
Step a, in the process of automatically following a shooting subject moving at a high speed, detecting the frame rate change of a shooting picture in real time, and acquiring a frame rate change video segment;
setting a detection sliding window to carry out sliding window detection on a real-time shooting picture in the shooting process, and obtaining detection data;
Performing standard deviation calculation on the frame rate of the detection data to obtain a variation index value;
And when the variation index value is larger than a preset detection threshold value, marking the detection data as a frame rate variation video segment.
The window length of the detection sliding window can be 5 to 15 frames, the window sliding step length can be 1 to 3 frames, and the window coverage range is dynamically adjusted to adapt to detection requirements under different movement speeds. In the standard deviation calculation process, the discrete degree of the frame rate data in the window is converted into a continuous numerical index through a mathematical formula, for example, a sample standard deviation formula is adopted to calculate and obtain a variation index value. The setting of the detection threshold can be dynamically adjusted according to the historical frame rate data, for example, 1.5 to 2.5 times of the average value of the standard deviation of the historical frame rate is taken as a threshold reference. The combination of window sliding and standard deviation calculation enables the space-time continuity of frame rate fluctuation to be effectively captured, and detection dead zones caused by fixed interval sampling are avoided.
Specifically, in the real-time shooting process, the sliding window is detected to slide along the time axis in a preset step length, and frame rate data of all frames in the window are extracted after each sliding. Discrete frame rate fluctuations are converted to quantization indices by calculating the standard deviation of the frame rate within a window, for example, when the window contains 10 frames of data, the standard deviation calculation may reflect the frame rate stability within the 10 frames. When the variation index value exceeds a dynamically set detection threshold, abnormal fluctuation of the video segment corresponding to the current window is judged, for example, when the frame rate suddenly drops from 60fps to 45fps due to sudden acceleration of a player in shooting basketball games, the standard deviation in the window is obviously increased, and a mark is triggered. Through the combination of continuous sliding window detection and statistical quantitative analysis, the risk of single sampling missing transient fluctuation is avoided, conventional fluctuation noise is filtered through quantitative indexes, and accurate positioning of abnormal fragments is ensured.
And b, extracting a first frame shooting picture and a last frame shooting picture of the frame rate variation video segment, and respectively extracting first frame relative position data and last frame relative position data of a shooting subject from the first frame shooting picture and the last frame shooting picture. Step b is specifically implemented by steps b1-b 3.
B1, selecting a panoramic background picture according to shooting scenes related to the frame rate variation video segment;
When the shooting main body is subjected to motion shooting, the shooting unit based on the fixed position carries out panoramic shooting on the motion scene where the shooting main body is located, so that the motion shooting scene of the shooting main body is obtained, and panoramic background pictures corresponding to the shooting main body in the high-speed motion process can be extracted from picture data of the panoramic shooting.
B2, extracting a shooting subject from the first frame shooting picture, and acquiring first frame relative position data of the shooting subject in the panoramic background picture;
Specifically, fig. 2 shows a flowchart of a method in step b in the embodiment of the present invention, and the identification of the shooting subjects of the first frame image and the last frame image may be implemented by an image identification algorithm, such as an image algorithm of Fast R-CNN model (Fast Region-based Convolutional Neural Networks), SSD model (Single Shot MultiBox Detector), YOLO model (You Only Look Once), and the like, which is not limited herein.
And b3, picking up the picture at the tail frame to extract the shooting subject, and acquiring the relative position data of the tail frame of the shooting subject in the panoramic background picture.
Further, the shooting subject is also extracted from the tail frame shooting picture, and the relative position data of the tail frame of the shooting subject in the panoramic background picture is obtained, in this way, the position data of the head and tail frames are both based on the same panoramic background coordinate system, the spatial consistency is ensured, a unified panoramic background reference system is established based on the panoramic background picture, and the problem of inconsistent coordinate references in a dynamic scene is effectively solved. By mapping the subject position to the panoramic background coordinate system, reference drift due to lens movement or view angle variation is eliminated. The double-frame coordinate mapping method based on panoramic background isolates local picture interference, provides accurate space-time reference for subsequent motion route fitting, and improves accuracy of motion route fitting.
Specifically, in the dynamic shooting process, after a frame rate variation video segment is detected, a pre-stored panoramic background picture is called as a space reference based on video content characteristics. And (3) aligning the first frame picture with the panoramic background through an image registration technology, positioning a shooting main body boundary frame by using a target detection model, calculating the coordinate value of the center point of the main body boundary frame in a panoramic coordinate system, and storing the coordinate value as a starting position. And multiplexing the same panoramic background during tail frame processing, tracking the displacement track of the main body by adopting an optical flow method, and finally outputting termination position data based on a unified coordinate system. The process controls the head-to-tail position data error within 0.5 pixel by eliminating the local picture visual angle difference, and provides accurate time-space reference for the subsequent motion route fitting. For example, in shooting football matches, even if a camera moves along with a player to cause background deviation, displacement data of the player in a standard field coordinate system can still be accurately acquired through a panoramic field map.
And c, generating a fitting motion route based on the AI model and combining the relative position data of the first frame and the relative position data of the last frame. Specifically, fig. 3 shows a flowchart of a method of step c in an embodiment of the present invention, including steps c1-c3.
C1, analyzing the motion trend of the shooting subject according to the relative position data of the first frame and the relative data of the last frame through an AI model;
furthermore, the AI model can be obtained based on neural network model training, and the AI model is trained through different sample data of various shooting subjects, so that the AI model can generate a movement route of the shooting subject according to the first frame shooting picture and the last frame shooting picture.
C2, inserting a plurality of blank frame pictures between the first frame shot picture and the last frame shot picture according to a preset shooting frame rate, and inserting a shooting main body prediction position point in each blank frame picture by combining the motion trend of the shooting main body;
And c3, constructing a fitting motion route according to the predicted position points of the blank frame pictures.
Specifically, the step c2 includes taking a panoramic background picture as the image content of a blank frame, and carrying out shooting subject position prediction of a next frame picture by combining an AI model with the motion trend of the shooting subject and the relative position data of the shooting subject in the current frame picture to obtain a shooting subject prediction position point of the next frame picture.
Specifically, the AI model is configured to analyze nonlinear motion features, such as acceleration or abrupt changes in direction, in the end-to-end frame position data, and extract potential patterns of motion trajectories through a deep learning algorithm. The number of blank frame pictures is set by the actual frame rate of the original video segment, for example, when the target frame rate is 60fps, 58 blank frames need to be inserted between the first frame shot picture and the last frame shot picture, so that the AI model can perform compensation frame insertion according to the actually required shooting frame rate, and the AI model can simulate the movement route of the shooting subject at the preset shooting frame rate.
Further, a time sequence prediction model is adopted for generating the predicted position points, the position coordinates of the middle moment are output by taking the head and tail frame positions as input, and a loss function used in model training can comprise a speed continuity constraint and an acceleration smoothness constraint so as to generate a movement route of a shooting subject in continuous frame pictures.
Specifically, when inserting blank frames between the first frame and the last frame, the time stamps of each blank frame are uniformly distributed according to a preset frame rate, for example, 60 equidistant time points are inserted in 1 second intervals. The AI model captures time-dependent features by analyzing the difference in position of the head and tail frames, in combination with kinematic equations or neural networks to predict the position coordinates of the intermediate moments, for example, using LSTM networks. In the generation process of the predicted position point, an error correction mechanism can be introduced, for example, when the speed change of the adjacent predicted point exceeds a threshold value, the generation parameters of the subsequent predicted point can be automatically adjusted. The predicted position points in the blank frame picture are converted into specific pixel positions in the image through a coordinate mapping algorithm, for example, three-dimensional space coordinates are projected to a two-dimensional image plane.
The construction of the fitting motion route adopts a spline interpolation algorithm, discrete predicted position points are connected into a continuous track, wherein parameters of the interpolation algorithm are dynamically adjusted according to motion trend, for example, bezier curve fitting is adopted in a high-speed rotation scene. Therefore, the precision and the instantaneity of motion compensation are effectively improved through processing the head and tail frame data in stages, dynamically generating the middle predicted point and optimizing the track fitting algorithm.
Further, the panoramic background picture is generated by 360-degree scene acquisition of the scene environment in a pre-shooting stage, the resolution of the panoramic background picture is consistent with that of a real-time shooting picture, the AI model adopts a convolutional neural network architecture, an input end receives a data packet containing a motion trend vector and a current frame relative position coordinate, wherein the motion trend vector consists of a speed component and a direction angle, and the current frame relative position coordinate is based on a panoramic background picture coordinate system.
In the prediction process, the AI model updates the position reference data in real time at the operation frequency of 30 frames processed per second, and the fusion weight of the motion trend vector and the position coordinate is dynamically adjusted according to the scene complexity.
And d, dividing a plurality of video adjustment sections in the frame rate variation video section, and generating sub-fitting routes corresponding to the video adjustment sections based on the AI model. Specifically, fig. 4 shows a flowchart of a method of step d in an embodiment of the present invention, specifically including steps d1-d3.
D1, extracting a plurality of shooting references of the frame rate variation video segment, and extracting a plurality of process shooting pictures in the frame rate variation video segment according to the distribution of the shooting references, wherein the extraction of the shooting references can be realized by adopting an algorithm based on edge detection or characteristic point matching, for example, static objects in a scene are identified as references through a SIFT algorithm.
And d2, dividing the frame rate variable video segments into a plurality of video adjustment segments according to the shooting time sequence of the plurality of process shooting pictures, wherein the process shooting picture extraction can set sampling intervals based on the reference object distribution density, for example, when the reference object distribution density exceeds 5 characteristic points per square pixel, the process shooting pictures are extracted at the rate of 10 frames per second. The division of the video adjustment segments requires combining the temporal sequence with the reference distribution characteristics, for example, combining consecutive segments of the adjacent process taken pictures with a time interval less than 0.5 seconds into one video adjustment segment.
D3, acquiring relative position change data of a shooting reference object and a shooting subject of each video adjustment section, and generating a sub-fitting route according to the relative position change data through an AI model.
The relative position change data can be obtained by calculating displacement vectors of the shooting subject and the reference object through an optical flow method, and the AI model can analyze time sequence change rules of the displacement vectors by adopting an LSTM network to generate a sub-fitting route matched with the fitting movement route.
Specifically, in a frame rate varying video segment, an object having a stable spatial position in a scene is first identified as a shooting reference, such as a billboard or an audience mount structure in a stadium, by a computer vision algorithm. Based on the spatial distribution density of the reference, a picture is taken in a process of extracting the complete distribution characteristics of the reference at a preset sampling frequency, for example, a frame of picture is taken every 0.3 seconds. Then dividing the video segments corresponding to the pictures shot in the continuous process into video adjustment segments according to the time sequence, wherein the time length of each video adjustment segment can be controlled between 0.5 and 2 seconds. For each video adjustment segment, a dynamic dataset containing three-dimensional coordinate changes is generated by tracking the relative displacement of the subject of the shot and the reference. The AI model predicts the motion trail of the shooting subject in the sub-time period by analyzing the direction, speed and acceleration parameters of the displacement vector in the data set, and generates a sub-fitting route which is spatially continuous with the whole fitting motion route. Therefore, the division of the video adjustment section is extended from a simple time dimension to a space feature dimension, so that the sub-fitting route can dynamically reflect the cooperative motion rule of the shooting main body and the scene reference object, and an accurate space position reference is provided for the compensation frame insertion.
And e, integrating each sub-fitting route and the fitting movement route for analysis, and inserting more than one compensation frame picture into each video adjustment section. Specifically, FIG. 5 shows a flow chart of a method of step e in an embodiment of the present invention, including steps e1-e3.
E1, dividing the fitted movement route into a plurality of fitted route segments, wherein the fitted route segments correspond to the fitted sub-routes one by one;
e2, determining a frame inserting compensation position point according to the comparison analysis of the fitting route segment and the corresponding fitting sub-route;
and e3, generating a motion estimation value according to the fitted route segment, generating motion sub-data according to the fitted sub-route, and outputting a compensation frame picture according to the motion estimation value and the motion sub-data through an AI model.
The step e3 specifically comprises analyzing the motion parameters of the shooting subject and the shooting reference object according to the shooting content of each fitted route segment, and specifically, tracking the position changes of the shooting subject and the reference object in continuous frames through a computer vision algorithm so as to acquire the parameters such as the motion speed, the acceleration, the direction and the like. For example, optical flow or object tracking algorithms may be employed to implement this process.
And comparing the motion parameters with preset thresholds, dividing the motion levels of the complementary frames, and setting a plurality of thresholds to define different motion levels. For example, a motion speed of less than 5 pixels/frame may be defined as low-speed motion, 5-15 pixels/frame as medium-speed motion, and greater than 15 pixels/frame as high-speed motion.
And selecting a corresponding frame supplementing mode according to the frame supplementing motion level, and generating a compensating frame picture.
The athletic parameters may include at least one of speed, direction angle, and the preset threshold may be dynamically adjusted based on different scenarios, such as setting the speed threshold to 5m/s and the direction angle difference threshold to 30 degrees during a sporting event. The division of the frame interpolation motion level can be achieved by a linear weighting mode, for example, the weighted sum of rates of a shooting subject and a shooting reference object is compared with a preset threshold value, and the frame interpolation motion level is divided into high-difficulty frame interpolation levels when the weighted sum exceeds the threshold value. The selection of the frame interpolation mode may include switching the type of the frame interpolation algorithm or the generation logic, for example, the optical flow method is used to generate the compensation frame by combining the bidirectional motion estimation at a high difficulty level, and the linear interpolation algorithm is used at a low difficulty level.
Specifically, after integrating the main route and the sub-route, the motion parameters of the shooting subject and the reference object are extracted by analyzing the shooting content in each route, for example, the displacement of the shooting subject between continuous frames is calculated by feature point matching, and the motion track of the reference object is obtained by background optical flow analysis. When the motion parameters are compared with a preset threshold, a multidimensional evaluation model is adopted, for example, the rate and the direction angle are respectively given to different weight coefficients, then comprehensive scoring is carried out, and the scoring result is mapped into three complementary frame grades of high, medium and low. When selecting the frame supplementing mode according to the level, invoking a pre-trained high-precision motion prediction model to generate a compensation frame in a high-difficulty scene, wherein the number of the compensation frame can be dynamically increased to 1.5 times of the number of the original frame, and generating the compensation frame by adopting a lightweight algorithm based on position interpolation in a low-difficulty scene. By dynamically matching the frame compensation strategy and the motion complexity, the matching error of the compensation frame and the actual motion trail can be reduced to be within 3 pixels, so that the continuity and stability of the video picture in the high-speed motion scene are ensured.
The dividing operation of the frame compensating motion level comprises the following steps:
If it is And (2) andDividing the frame-supplementing motion level into high-difficulty frame-supplementing;
If it is And (2) andDividing the frame-supplementing motion level into low-difficulty frame-supplementing;
wherein, the 、Is a coefficient of proportionality and is used for the control of the power supply,In order to capture the rate of movement of the subject,In order to capture the rate of movement of the reference object,In order to photograph the movement direction angle of the subject,To capture the direction angle of motion of the reference object.
Further, in the generation of the motion change data, the subject motion trajectories extracted by the first frame shot picture and the last frame shot picture are quantized to S1 and θ1 parameters, while the reference motion trajectories extracted by the process shot picture are quantized to S2 and θ2 parameters. When dividing the motion compensation video segments, the alpha and beta coefficients corresponding to each segment of video are dynamically configured according to the scene type.
Specifically, in the motion compensation video segment processing process, firstly, a coordinate change sequence of a shooting subject in continuous frames is extracted through a feature matching algorithm, and meanwhile, the displacement of a fixed reference object in a background is detected.
The subject rate is calculated by dividing the difference in displacement between adjacent frames by the time interval, for example, when a soccer player is detected moving 15 pixels in 0.1 seconds, the rate of motion is 150 pixels/second. And calculating the direction angle through vector included angles formed by three continuous coordinate points, and judging the direction is suddenly changed if the angle change exceeds 45 degrees due to the fact that the basketball player suddenly stops turning.
In the frame-compensating grading stage, the motion parameters of the shooting subject and the reference object are input into a weighted calculation formula, for example, when the athlete speed weight alpha is set to 0.7 and the reference object speed weight beta is set to 0.3, the comprehensive motion index exceeds a preset threshold value of 8.5, namely, the high-difficulty frame-compensating mode is triggered.
Furthermore, the frame supplementing mode selection module calls a corresponding algorithm library according to the grade identification, enables a frame inserting model based on PWC-Net optical flow estimation in a high-difficulty mode, and performs motion compensation by adopting a phase correlation method in a low-difficulty mode. By dynamically adjusting the frame supplementing strategy, the video processing system can flexibly select the frame supplementing mode while ensuring the operation efficiency, and the operation efficiency of the whole system is improved.
Specifically, the shooting adjustment method further includes:
Before shooting activities, shooting scenes of a shooting environment to obtain environment image data;
and carrying out background optimization on each frame of image of the shot picture based on the environment image data.
The scene taking can comprise selecting a plurality of sampling points in a scene to take at multiple angles, the distance between the sampling points can be set to be 0.5-1.2 m, and the times of the scene taking can be 3-5 times. The ambient image data may include illumination intensity, color temperature distribution, and background texture feature parameters.
Furthermore, the background optimization process may include dividing a background area in the real-time picture into 8×8 pixel blocks, comparing features with pre-stored environmental data, triggering noise reduction processing when the matching degree is lower than 85%, and performing optimization processing by means of brightness compensation, contrast correction, noise reduction intensity adjustment and the like, so that the photographed video picture can be clear and smooth.
Specifically, before shooting activities, the shooting environment can be subjected to scene shooting to obtain panoramic background pictures, and background optimization is performed on each frame of image of the shooting picture based on the panoramic background pictures, so that the overall quality of the shooting video is improved.
The scene capturing can be performed by adopting a surrounding type multi-angle capturing or fixed-point scanning capturing mode, for example, 360-degree surrounding flight is performed in a captured scene by an unmanned aerial vehicle carrying a multi-lens array, and image data covering the whole scene area is acquired. The image data is processed by a three-dimensional reconstruction algorithm to generate a panoramic background picture, the resolution can be set to be 4K level, and the size of a single picture is controlled within 8000 multiplied by 6000 pixels. In the background optimization process, a dynamic shooting main body is separated from a static background through an image segmentation algorithm, for example, a U-Net neural network model is adopted to carry out semantic segmentation on each frame of image, the segmented dynamic main body area and the panoramic background are subjected to pixel-level fusion, and transparency parameters are set to be 0.9 during fusion so as to ensure the natural transition of the main body edge.
Specifically, before the shooting activity starts, a target scene is shot at multiple angles by a shooting device equipped with a wide-angle lens, for example, the target scene is shot in a stadium along an audience seat top track, and panoramic images covering a competition field are acquired. The exposure time is set to be 1/1000 second in the acquisition process to eliminate the dynamic blur, and three groups of photos with different exposure parameters are taken in an HDR mode. And combining a plurality of groups of photos into a panoramic background picture through an image stitching algorithm, and storing the panoramic background picture into a PNG format file with a transparency channel. And during subsequent video processing, extracting a motion main body contour by adopting a background difference method aiming at each frame of image, mapping the main body contour to a corresponding coordinate position of a panoramic background, and eliminating a splicing gap through a bilinear interpolation algorithm. When the compensation frame is inserted, the panoramic background is directly called as a bottom layer picture, and a complete compensation frame image is generated by combining the predicted track of the motion main body, so that the spatial consistency of background elements on a time axis is ensured. According to the scheme, the unified background reference library is established, so that video frames at different time points share the same background data, and the background jump phenomenon caused by illumination change or temporary shielding is effectively eliminated.
The core innovation point of the application is that the dynamic frame rate detection and the multi-level motion track analysis are combined, and the self-adaptive generation of the compensation frame in the high-speed motion scene is realized through a dual mechanism of segment fitting and global path integration. The scheme breaks through the limitation that the traditional frame inserting technology relies on a fixed algorithm, utilizes an AI model to intelligently predict the nonlinear motion, effectively eliminates picture blocking while reducing calculation load, and is particularly suitable for shooting optimization of complex motion tracks of sports events and the like.
The application has the working process and principle that in the process of automatically following a shooting main body moving at high speed, firstly, the frame rate change of a shooting picture is detected in real time, and a frame rate change video segment is obtained. In the step, a detection sliding window is set to detect a real-time shooting picture in a sliding window mode, detection data are obtained, standard deviation calculation is conducted on the frame rate of the detection data, and a variation index value is obtained. When the variation index value is larger than a preset detection threshold value, marking the detection data as a frame rate variation video segment.
Next, a first frame shot picture and a last frame shot picture are extracted from the frame rate fluctuation video segment, and first frame relative position data and last frame relative position data of the shooting subject are extracted from the two pictures, respectively. The specific operation comprises the steps of selecting a panoramic background picture according to shooting scenes related to a frame rate variation video segment, extracting a shooting subject from the shooting pictures of a first frame and a last frame, and acquiring relative position data of the shooting subject in the panoramic background picture.
Then, a fitted motion route is generated based on the AI model in combination with the first frame relative position data and the last frame relative position data. The AI model firstly analyzes the motion trend of a shooting subject, inserts a plurality of blank frame pictures between a first frame and a last frame according to a preset shooting frame rate, and inserts a predicted position point of the shooting subject in each blank frame picture. And finally, constructing a fitting motion route according to the predicted position points.
Further, a plurality of video adjustment sections are divided in the frame rate variation video section, and sub-fitting routes corresponding to the plurality of video adjustment sections are generated based on the AI model. The method comprises the steps of extracting a plurality of shooting references of a frame rate variation video segment, extracting a plurality of process shooting pictures according to the distribution of the shooting references, and dividing the video adjustment segment according to the shooting time sequence of the pictures. And for each video adjusting section, acquiring relative position change data of a shooting reference object and a shooting subject, and generating a sub-fitting route through an AI model.
And finally, integrating each sub-fitting route and the fitting movement route for analysis, and inserting more than one compensation frame picture into each video adjustment section. The specific operation comprises the steps of dividing the fitting movement route into a plurality of fitting route segments, and corresponding to the sub-fitting routes one by one. And determining interpolation frame compensation position points through comparative analysis, generating a motion estimation value according to the fitting route section, generating motion sub-data according to the fitting sub-route, and finally outputting compensation frame pictures through an AI model.
Embodiment two:
fig. 6 shows a schematic diagram of a working system of a shooting robot for performing an AI-based intelligent optimization shooting adjustment method based on AI intelligent optimization shooting adjustment in an embodiment of the present invention, the shooting robot including:
The shooting component 10 is used for shooting in real time and transmitting shooting pictures to the frame rate processing component in real time;
The photographing assembly 10 may implement real-time picture acquisition using a CMOS sensor or a CCD sensor and transmit raw picture data to a frame rate processing assembly through an MIPI interface at a transmission rate of not less than 30 fps.
The frame rate processing component 20 is used for detecting the frame rate change of the shot picture and acquiring a frame rate change video segment;
The frame rate processing component 20 may be integrated with an FPGA chip or a dedicated ASIC processor, after receiving the frame data, scan the frame stream at a frequency of once per millisecond by a built-in frame rate detection algorithm, and when detecting that the standard deviation of the interval time between adjacent frames exceeds a preset threshold, immediately trigger a video segment marking mechanism to cache the abnormal video segment into the DDR4 memory module. The frame rate compensation component may piggy-back the GPU acceleration unit.
The frame rate compensation component 30 is configured to perform frame picture analysis on the frame rate variation video segment to generate a compensated frame picture.
And carrying out motion vector analysis on the marked video segment by using an OpenCL frame, calculating the motion trail of the main body and the background in the picture by using an optical flow method, and generating an intermediate compensation frame.
Those of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be implemented by a program for instructing related hardware, and the program may be stored in a computer readable storage medium, and the storage medium may include a Read Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk or an optical disk, etc.
In addition, while the foregoing describes in detail the embodiments of the present invention, specific examples are set forth herein to illustrate the principles and embodiments of the present invention, and are provided to assist in understanding the methods and concepts of the present invention, and to enable one of ordinary skill in the art to make various changes in the detailed description and the application scope of the invention, such that the disclosure is not to be construed as limiting the invention.
Claims (8)
1. An AI-based intelligent optimization shooting adjustment method is characterized by comprising the following steps:
a. in the process of shooting the main body high-speed motion, detecting the frame rate change of a shooting picture in real time, and acquiring a frame rate change video segment;
b. Extracting a first frame shooting picture and a last frame shooting picture of the frame rate variation video segment, and respectively extracting first frame relative position data and last frame relative position data of a shooting subject from the first frame shooting picture and the last frame shooting picture;
c. generating a fitting motion route by combining the relative position data of the first frame and the relative position data of the tail frame based on the AI model;
d. Dividing a plurality of video adjustment sections in the frame rate variation video section, and generating sub-fitting routes corresponding to the plurality of video adjustment sections based on an AI model;
e. integrating each section of sub-fitting route and fitting motion route for analysis, and inserting more than one compensation frame picture into each section of video adjustment section;
the step e comprises the following steps:
e1, dividing the fitted movement route into a plurality of fitted route segments, wherein the fitted route segments correspond to the fitted sub-routes one by one;
e2, determining a frame inserting compensation position point according to the comparison analysis of the fitting route segment and the corresponding fitting sub-route;
e3, generating a motion estimation value according to the fitted route segment, generating motion sub-data according to the fitted sub-route, and outputting a compensation frame picture according to the motion estimation value and the motion sub-data through an AI model;
The step e3 includes:
analyzing the motion parameters of a shooting subject and a shooting reference object according to the shooting content of each fitted route segment;
comparing the motion parameters with a preset threshold value, and dividing the motion level of the complementary frame;
and selecting a corresponding frame supplementing mode according to the frame supplementing motion level, and generating a compensating frame picture.
2. The AI-based intelligent optimization shooting adjustment method according to claim 1, wherein the step a includes:
setting a detection sliding window to carry out sliding window detection on a real-time shooting picture in the shooting process, and obtaining detection data;
Performing standard deviation calculation on the frame rate of the detection data to obtain a variation index value;
And when the variation index value is larger than a preset detection threshold value, marking the detection data as a frame rate variation video segment.
3. The AI-based intelligent optimization shooting adjustment method according to claim 1, wherein the step b includes:
b1, selecting a panoramic background picture according to shooting scenes related to the frame rate variation video segment;
b2, extracting a shooting subject from the first frame shooting picture, and acquiring first frame relative position data of the shooting subject in the panoramic background picture;
And b3, picking up the picture at the tail frame to extract the shooting subject, and acquiring the relative position data of the tail frame of the shooting subject in the panoramic background picture.
4. The AI-based intelligent optimization shooting adjustment method according to claim 1, wherein the step c includes:
c1, analyzing the motion trend of the shooting subject according to the relative position data of the first frame and the relative data of the last frame through an AI model;
c2, inserting a plurality of blank frame pictures between the first frame shot picture and the last frame shot picture according to a preset shooting frame rate, and inserting a shooting main body prediction position point in each blank frame picture by combining the motion trend of the shooting main body;
And c3, constructing a fitting motion route according to the predicted position points of the blank frame pictures.
5. The AI-based intelligent optimization shooting adjustment method according to claim 4, wherein the step c2 includes:
Taking the panoramic background picture as the image content of a blank frame, and carrying out shooting body position prediction of the next frame picture by combining the motion trend of the shooting body and the relative position data of the shooting body in the current frame picture through an AI model to obtain a shooting body prediction position point of the next frame picture.
6. The AI-based intelligent optimization shooting adjustment method according to claim 1, wherein the step d includes:
d1, extracting a plurality of shooting reference objects of a frame rate variation video segment, and extracting a plurality of process shooting pictures in the frame rate variation video segment according to the distribution of the shooting reference objects;
d2, dividing the frame rate variation video segments into a plurality of video adjustment segments according to the shooting time sequence of the pictures in a plurality of processes;
d3, acquiring relative position change data of a shooting reference object and a shooting subject of each video adjustment section, and generating a sub-fitting route according to the relative position change data through an AI model.
7. The AI-based intelligent optimization shooting adjustment method according to claim 1, wherein the dividing operation of the frame interpolation motion level is as follows:
If it is And (2) andDividing the frame-supplementing motion level into high-difficulty frame-supplementing;
If it is And (2) andDividing the frame-supplementing motion level into low-difficulty frame-supplementing;
wherein, the 、Is a coefficient of proportionality and is used for the control of the power supply,In order to capture the rate of movement of the subject,In order to capture the rate of movement of the reference object,In order to photograph the movement direction angle of the subject,To capture the direction angle of motion of the reference object.
8. Shooting robot based on AI intelligent optimization shooting adjustment, characterized in that it is used to execute the AI intelligent optimization shooting adjustment method based on any one of claims 1 to 7, and comprises:
The shooting assembly is used for shooting in real time and transmitting shooting pictures to the frame rate processing assembly in real time;
the frame rate processing component is used for detecting the frame rate change of a shot picture and acquiring a frame rate change video segment;
and the frame rate compensation component is used for carrying out frame picture analysis on the frame rate variation video segment to generate a compensation frame picture.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202511157257.2A CN120658945B (en) | 2025-08-19 | 2025-08-19 | A shooting adjustment method and shooting robot based on AI intelligent optimization |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202511157257.2A CN120658945B (en) | 2025-08-19 | 2025-08-19 | A shooting adjustment method and shooting robot based on AI intelligent optimization |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN120658945A CN120658945A (en) | 2025-09-16 |
| CN120658945B true CN120658945B (en) | 2025-10-17 |
Family
ID=96996477
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202511157257.2A Active CN120658945B (en) | 2025-08-19 | 2025-08-19 | A shooting adjustment method and shooting robot based on AI intelligent optimization |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN120658945B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117201841A (en) * | 2023-09-11 | 2023-12-08 | 中国银行股份有限公司 | Picture frame supplementing method, device, equipment and readable storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6438275B1 (en) * | 1999-04-21 | 2002-08-20 | Intel Corporation | Method for motion compensated frame rate upsampling based on piecewise affine warping |
| US8384787B2 (en) * | 2011-02-24 | 2013-02-26 | Eastman Kodak Company | Method for providing a stabilized video sequence |
| US10003768B2 (en) * | 2016-09-28 | 2018-06-19 | Gopro, Inc. | Apparatus and methods for frame interpolation based on spatial considerations |
| CN113837136B (en) * | 2021-09-29 | 2022-12-23 | 深圳市慧鲤科技有限公司 | Video frame insertion method and device, electronic equipment and storage medium |
-
2025
- 2025-08-19 CN CN202511157257.2A patent/CN120658945B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117201841A (en) * | 2023-09-11 | 2023-12-08 | 中国银行股份有限公司 | Picture frame supplementing method, device, equipment and readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN120658945A (en) | 2025-09-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110692083B (en) | Block-matched optical flow and stereo vision for dynamic vision sensors | |
| CN115100744A (en) | Badminton game human body posture estimation and ball path tracking method | |
| US6665342B1 (en) | System and method for producing a still image representation of a motion video | |
| KR100224752B1 (en) | Target tracking method and apparatus | |
| CN113838098A (en) | An intelligent tracking and shooting system for long-distance high-speed moving targets | |
| KR20090006068A (en) | How to modify video sequence and its device | |
| US11361448B2 (en) | Image processing apparatus, method of controlling image processing apparatus, and storage medium | |
| CN112906475B (en) | Artificial intelligence-based rolling shutter imaging method and system for urban surveying and mapping unmanned aerial vehicle | |
| CN113836980A (en) | Face recognition method, electronic device and storage medium | |
| CN117036404A (en) | A method and system for simultaneous positioning and mapping of monocular thermal imaging | |
| Qi et al. | Alpine skiing tracking method based on deep learning and correlation filter | |
| Kalafatić et al. | Multiple object tracking for football game analysis | |
| CN107333031B (en) | Multi-channel video automatic editing method suitable for campus football match | |
| CN117812275A (en) | Image-optimized communication method for volleyball assisted training | |
| CN120658945B (en) | A shooting adjustment method and shooting robot based on AI intelligent optimization | |
| CN110910489A (en) | Monocular vision based intelligent court sports information acquisition system and method | |
| CN119763197A (en) | Human motion optimization method, system, terminal and storage medium for multi-lens video | |
| Bastanfard et al. | Improving tracking soccer players in shaded playfield video | |
| JP2825863B2 (en) | Moving object detection device | |
| CN111563912B (en) | Pedestrian tracking system and method | |
| CN120293108B (en) | Theodolite stability tracking method based on multi-frame information fusion | |
| CN119672267B (en) | Data processing method for AR intelligent camera device | |
| CN121053171B (en) | A Badminton Shuttlecock Tracking Method and System Based on 3D Vision | |
| Tang | Research on the application of computer vision technology in the adversarial judgment system in competitive sports competitions | |
| CN121010510B (en) | A dual-light image fusion method for intelligent cameras and a dual-light integrated intelligent camera |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |