CN102148934B - A multi-mode real-time electronic image stabilization system - Google Patents
A multi-mode real-time electronic image stabilization system Download PDFInfo
- Publication number
- CN102148934B CN102148934B CN 201110084163 CN201110084163A CN102148934B CN 102148934 B CN102148934 B CN 102148934B CN 201110084163 CN201110084163 CN 201110084163 CN 201110084163 A CN201110084163 A CN 201110084163A CN 102148934 B CN102148934 B CN 102148934B
- Authority
- CN
- China
- Prior art keywords
- motion
- image
- module
- picture
- filtering
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Landscapes
- Image Processing (AREA)
Abstract
The invention relates to a multi-mode real-time electronic image stabilizing system, belonging to the image processing field. The system mainly comprises a digital model and preprocessing module, an interframe image stabilizing module and an intraframe image stabilizing module. The interframe image stabilizing module comprises an interest region based global motion estimation unit, a motion vector filtering unit, a motion determining unit and a motion compensation unit; the digital model and preprocessing module is connected with the interest region based global motion estimation unit in the intraframe image stabilizing module, the interest region based global motion estimation unit is respectively sequentially connected with the motion vector filtering unit and the intraframe image stabilizing module, the motion vector filtering unit, the motion determining unit and the motion compensation unit in order, and the motion compensation unit is respectively connected with the intraframe image stabilizing module and external DSP (digital signal processing) parallel processing. The system provided by the invention has the characteristics of multiple modes, high accuracy, instantaneity, high frequency vibration resistance and intellectualization and also has the advantages of simple structure, convenience in integration and easiness in operation.
Description
Technical field
The present invention relates to a kind of multi-mode real-time electronic image stabilizing system, the high accuracy, the real-time that are used for finishing video image under different platform and the environment are processed, and belong to image processing field.
Background technology
Since 21 century, electronic image stabilizing is easy to operate with it, high accuracy, volume is little and price is low, energy consumption is little, the characteristics such as intelligent, is widely used in military field, civilian aerial survey and camera chain.But there are some difficult problems in traditional electronic steady image theory and technology: 1) motion platform environment and feature are complicated, and image-forming condition is changeable, and the Mathematical Models between various motions and the image is very difficult; 2) complexity of actual photographed scenery and diversity have caused the versatility of steady picture algorithm, real-time poor; 3) because mode of operation and imaging theory do not have essence to break through for many years, and present electronic image stabilizing still has many deficiencies, such as the fuzzy problem that high dither brings, cause the definition deficiency of imaging and surely look like precision low.
For above difficult point, the research worker has proposed a series of Electronic Image Stabilization and realization technology both at home and abroad, such as a kind of vectorial track algorithm that in large-scale search window, carries out estimation, referring to M.Mattavelli, " Vector-tracing algorithm formotion estimation in large search windows ", IEEE Trans.Circuits Syst.Video Techno, 2000, Vol.10,1426-1437; Based on the Electronic Image Stabilization of representative point coupling, referring to Zhong Ping, " application of a kind of improved representative point match algorithm in steady picture technology ", optical technology, 2005, Vol.31, No.5,742-745; Based on the global motion vector algorithm for estimating of gradient information, referring to T.Marius, A.Sakari, V.Markku, " Method of Motion Estimation forImage Stabilization ", IEEE ICASSP, 2006, Vol.2,277-280; Based on vehicle-mounted electronic image stabilizing, referring to Hsu Sheng-Che, Liang Sheng-Fu, Fan Kang-Wei, Lin Chin-Teng, " A Robust in-car digital imagestabilization technique ", IEEE Transactions on Systems, Man and Cybernetics, 2007, Vol.37, No.2,234-247.The Electronic Image Stabilization research and comparison is many, but, these methods have all been ignored the relation between platform and the image, ignored the impact that also there are motion in background and target in the image, be only applicable to the more single image sequence of process information, especially image information is complicated and changeable, and these methods limitation separately makes it all can't satisfy real-time and high-precision requirement.
Domestic research to electronic image stabilizing is started late, and still is in the junior stage at present, and most researchers is the research of the Electronic Image Stabilization under the single research special exercise platform, such as the steady picture such as airborne, vehicle-mounted or boat-carrying algorithm.The stabilized platform that the electronic image stabilizing that engineering is used mainly utilizes gyrosensor and servo-control system to consist of carries out steady picture, and the shortcoming such as the intrinsic volume of servo system is large, cost is high, consumed power is large often is difficult to satisfy the needs that military and civilian equipment further develops in a lot of occasions.
Summary of the invention
Technology of the present invention is dealt with problems and is: overcome the deficiencies in the prior art, a kind of multi-mode real-time electronic image stabilizing system is provided, this system has multi-mode, high accuracy, real-time, anti-dither and intelligentized characteristics, and simple in structure, be convenient to the advantages such as integrated, easy operating.
Technical solution of the present invention is: a kind of multi-mode real-time electronic image stabilizing system comprises that mainly Mathematical Modeling and pretreatment module, interframe are surely as steady picture module in module and the frame.Wherein interframe surely comprises ROI-based overall motion estimation unit, motion vector filtering unit, motion determining means and motion compensation units as module; Mathematical Modeling surely is connected as the ROI-based overall motion estimation unit in the module with interframe with pretreatment module, ROI-based overall motion estimation unit is connected surely with the motion vector filtering unit respectively, and the picture module is connected with frame, motion vector filtering unit, motion determining means and motion compensation units are linked in sequence successively, and motion compensation units surely is connected with the DSP parallel processing of being connected as module respectively with in the frame;
Wherein, Mathematical Modeling and pretreatment module are set up the multi-mode electronics surely as Mathematical Modeling and finish the judgement of input video type of sports; Interframe is surely finished as the ROI-based overall motion estimation unit in the module area-of-interest is carried out overall motion estimation, image processed estimate present frame with respect to the global motion side-play amount of reference frame, namely by camera parameters or change in location and the variation of the whole image that causes, then by the motion vector filtering unit image is carried out filtering, and filtered image is sent to the motion determining means; The motion determining means is analyzed the global motion vector that detects, and judges that this motion is that randomized jitter by the shooting carrier causes, still belong to video camera normal scan motion, or both has concurrently; Motion compensating module is according to being not intended to component motion as parameter to be compensated, directly makes pixel on the image do equivalent in the other direction according to the motion excursion that is detected and moves, to realize the row, column restructuring of image; In the frame surely as module for fuzzy in the picture frame that is caused by dither, carry out the image restoration of the spatially-variable motion blur of I picture.
Motion compensation units adopts the panoramic picture compensation method based on Kalman filtering to compensate, at first carry out the parameter Estimation of global motion, next calculates component motion, then utilize the recursive Kalman filtering device to the accumulation global motion vector sequence of original video sequence, carry out the Sequential filter processing according to the time sequencing that observed quantity arrives, extract again corresponding jittering component with the compensation present image, the boundary information of last combining image splicing reconstructing lost, the integrality of assurance image.
Mathematical Modeling and pretreatment module adopt Rigid model to translational motion with around the rotatablely moving of optical axis; For the translation under the degree of freedom, Rotation and Zoom motion, set up respectively similarity transformation model and affine Transform Model; For translation, level sweep, vertically sweep, rotation, lens zoom motion, adopt Perspective transformation model.
ROI-based overall motion estimation unit at first finishes the area-of-interest characteristic block and point extracts, further estimate amount of exercise by tracking characteristics point, by setting up the gray distribution features function, determine effective characteristic area according to gray distribution features numerical value in the zone, and extraction characteristic quantity, this characteristic quantity mates in reference picture, obtains the motion change value of characteristic quantity, these local amount of motion is brought into the global motion amount that can fit the picture of publishing picture in the image overall motion model.
Before calculating the global motion amount, at first local amount of motion is carried out error analysis, and the low local amount of motion of precision is removed, guarantee surely to reach inferior pixel as precision.
Adopt FPGA to finish the parallel processing of view data in conjunction with many DSP miniaturization digital picture real-time processor, FPGA finishes image preliminary treatment and electronic steady image Mathematical Models, three DSP are parallel to finish respectively Image Restoration Algorithm and motion compensation based on overall motion estimation and digital filtering, spatially-variable motion blur, realizes the real-time processing of video image.
Principle of the present invention: in the steady picture of video sequence is processed, because the global motion of image relates to the motion of motion, video camera carrier and video camera self of image and the motion of target subject, three kinds of motions are interrelated, influence each other, therefore, in order to determine the motion of image, the movement characteristic of necessary analysis and research platform and video camera is judged type of sports according to inputted video image, and motion vector between analysis frame, the electronic steady image Mathematical Modeling of Erecting and improving.For the interframe movement in the video sequence, adopt motion prediction new theory and the new technology of ROI-based, and a kind of panoramic picture compensation method based on Kalman filtering proposed, realize the Real-Time Filtering processing of video sequence, keep stable scanning component to extract unstable jittering component; For intraframe motion, adopt the image restoration technology of spatially-variable motion blur.For finishing in real time above algorithm, realize the electronic steady image task, specialized designs centered by large-scale FPGA in conjunction with many DSP of high-performance miniaturization digital picture real-time processor, take full advantage of FPGA and DSP simultaneous resource and treatment technology, FPGA finishes image preliminary treatment and electronic steady image Mathematical Models, three DSP are parallel to finish respectively Image Restoration Algorithm and motion compensation based on overall motion estimation and digital filtering, spatially-variable motion blur, realizes the real-time processing of video image.
The present invention's advantage compared with prior art is: 1) the present invention is directed to different motion platform and complicated imaging circumstances, but the multi-mode electronics of having set up high real-time and Project Realization surely looks like Mathematical Modeling; 2) the present invention improves the precision of electronic steady image according to the scene changes information design goes out the precision height, the parallel electronic of being convenient to real-time implementation surely looks like algorithm, particularly ROI-based motion prediction new theory and new technology; The image restoration new technology of spatially-variable motion blur, the definition of raising electronic steady image; The recursive Kalman filtering processing method guarantees the processing of sequence of video images Real-Time Filtering fast; The combination of this several method and the optimization of parallel algorithm break through one of key issue of multi-mode real-time electronic image stabilizing technology, finish the theory innovation of essence; 3) effectively utilize FPGA and DSP parallel processing technique, fully excavate parallel characteristics and the computational resource of these processors, break through the key technologies such as multi-mode real-time electronic image stabilizing system Comprehensive Signal Processing, realize the real-time processing of video image.
Description of drawings
Fig. 1 is a kind of multi-mode real-time electronic image stabilizing system structured flowchart of the present invention;
Fig. 2 is the perspective projection schematic diagram that the present invention analyzes imaging process;
Fig. 3 is the flow chart that the present invention judges the interframe movement type;
Fig. 4 is for the present invention is based on area-of-interest motion estimation module technology path block diagram;
Fig. 5 is the panoramic picture compensation flow chart that the present invention is based on Kalman filtering;
Fig. 6 is for the present invention is based on spatially-variable motion blur image restoration flow chart;
The electronic steady image system principle diagram that Fig. 7 designs for the present invention;
Fig. 8 realizes schematic diagram for the FPGA of 3 * 3 Gaussian filters that the present invention designs.
Embodiment
Below in conjunction with embodiment and accompanying drawing the present invention is described in further details.
As shown in Figure 1, main three modules of the present invention form: surely as steady picture module in module and the frame, wherein interframe surely comprises ROI-based overall motion estimation unit, motion vector filtering unit, motion determining means and motion compensation units as module for Mathematical Modeling and pretreatment module, interframe.
1. the present invention's motion of platform motion being caused video camera is divided into following several situation to the Mathematical Modeling of image change:
1) translational motion and around the rotatablely moving of optical axis, adopt Rigid model:
Wherein θ is the anglec of rotation of image, m
2And m
5Be translational movement, the rigid transformation matrix has 3 degrees of freedom.
2) translation, rotation, convergent-divergent motion, adopt the similarity transformation model:
This model can not twist original shape of object, but the size of object can change.The similarity transformation matrix has 4 degrees of freedom.
3) translation, rotation, convergent-divergent motion, adopt affine Transform Model:
Still be parallel lines after the parallel lines process affine transformation in the image.Affine transformation matrix has 6 degrees of freedom.
The motions such as 4) translation, level sweep, vertically sweep, rotation, lens zoom, adopt Perspective transformation model:
Perspective transform has more generally form, and wherein rigid transformation, similarity transformation and affine change all are the special cases of Perspective transformation model, and perspective transformation matrix has 8 degrees of freedom.
The present invention is the movement characteristic of analysis and research platform and video camera at first, judges type of sports according to inputted video image, and motion vector between analysis frame, the electronic steady image Mathematical Modeling of Erecting and improving.Concrete interframe movement type decision flow chart judges at first whether motion vector mostly is zero as shown in Figure 3, gets rid of after the frozen frozen mass; Judge again whether motion vector has the direction consistency, and distinguish horizontal motion or movement in vertical direction; Then judge whether to exist the expansion focus area, and the motion of differentiation convergent-divergent rotatablely moves still; At last, all in the ungratified situation, this moment, interframe movement was random motion at all situations.The present invention intends choosing two width of cloth images and carries out piecemeal (sub-block is of a size of 8 * 8) processing, and each sub-block is detected its motion vector, establishes MV
IjThe motion vector of the horizontal and vertical direction of=(u, v) expression sub-block (i, j) is then drawn motion vector distribution figure, can distinguish the interframe movement type according to some statistical properties.In decision process, get two characteristic value: direction v/u and the amplitude of motion vector
2. utilize the motion prediction new theory of ROI-based and recurrence Kalman Sequential filter processing method fast, and the image restoration technology of spatially-variable motion blur, realize that multi-mode electronics parallel, that optimize surely looks like algorithm
Motion in the video sequence generally is divided into intraframe motion and two kinds of situations of interframe movement.Rock between the picture frame that is caused by low-frequency vibration, although each two field picture is clearly, between frame and the frame variation has occured, namely image sequence is by time ambiguity.This just need to process interframe, namely obtain the motion vector of image sequence interframe by motion estimation algorithm, again the pixel on the image being done equivalent in the other direction according to the motion excursion that is detected moves, realize the ranks restructuring of image, so that the second frame output image overlaps with the first frame output image, thereby reach the stable purpose of image compensation.Fuzzy in the picture frame that is caused by dither, because the motion of a pixel occurs to surpass pixel within the time for exposure, then each two field picture all blurs, and namely the whole space of image function is all by fuzzy.This just need to process carrying out deblurring in the picture frame, can adopt the image restoration technology of spatially-variable motion blur to solve.
2.1 the motion forecast method of ROI-based
1) motion estimation unit
Estimation is by various motion estimation algorithm image to be processed, and estimates present frame with respect to the global motion side-play amount of reference frame, namely by camera parameters or change in location and the variation of the whole image that causes.Because background noise, enter these external interference such as visual field such as movement, the motion of target itself, the foreign matter of wisp in the background and the local motion that produces, all can affect the precision of overall motion estimation.Therefore, the key technology of motion estimation module is exactly overall motion estimation: improve the precision of overall motion estimation, also will guarantee simultaneously the ability of processing in real time.Can say that this module has determined speed and the precision of whole image stabilization system.
The present invention adopts motion prediction new theory and the new technology of ROI-based, realizes motion estimation module, and main technological route is at first finished area-of-interest characteristic block and some extraction as shown in Figure 4, further estimates amount of exercise by tracking characteristics point.The speed of extracting in order to improve characteristic block and point, requirement of real time needs to optimize the characteristic quantity recognition methods of area-of-interest.Each area-of-interest is carried out the characteristic validity analysis, by setting up the gray distribution features function, determine effective characteristic area according to gray distribution features numerical value in the zone, and extract characteristic quantity.Characteristic quantity mates in reference picture, obtains the motion change value of characteristic quantity, and it has reflected the local amount of motion of each effective coverage of image.These local amount of motion are brought into the global motion amount that can fit the picture of publishing picture in the image overall motion model, the computational accuracy that this shows the global motion amount of image is the precision that depends on each local motion component of image, therefore, before calculating the global motion amount, at first to carry out error analysis to local amount of motion, and the local amount of motion removal low to precision, the precision of assurance computed image global motion amount surely reaches inferior pixel as precision.
2) motion determines and motion compensation units
The motion determining means is that the global motion vector that detects is analyzed, and judges that this motion is that randomized jitter by the shooting carrier causes, still belong to video camera normal scan motion, or both has concurrently.The key of motion determining means is motion vector filtering.Owing to exist the accuracy of detection that many factors can affect motion vector, motion, the motion of target itself, foreign matter such as wisp in the background enter visual field etc., how the motion vector of estimating is done further processing to improve its precision, this problem is solved by the motion determining means.
Motion compensating module is to be not intended to component motion as parameter to be compensated according to what the motion decision module provided, directly making pixel on the image do equivalent in the other direction according to the motion excursion that is detected moves, to realize the row, column restructuring of image, reach the stable purpose of image compensation, make monitor export clearly sequence of video images.The present invention proposes a kind of panoramic picture compensation method based on Kalman filtering, sequence of video images is carried out Real-Time Filtering process, and namely is not intended to motion to extract unstable jittering component, namely has a mind to move and keep stable scanning component.Effectively reduce the interframe shake of sequence of video images, be suitable for real-time application, its flow process as shown in Figure 5, at first carry out the parameter Estimation of global motion, next calculates component motion, then utilize fast the recursive Kalman filtering device to the accumulation global motion vector sequence of original video sequence, carry out the Sequential filter processing according to the time sequencing that observed quantity arrives, extract again corresponding jittering component with the compensation present image, the boundary information of last combining image splicing reconstructing lost is to guarantee the integrality of image.
2.2 the image recovery method of the spatially-variable motion blur in the frame in the steady picture module
Application region of the present invention partitioning is carried out the image restoration of the spatially-variable motion blur of I picture, its basic procedure as shown in Figure 6, image segmentation is become a lot of tiny areas, each zone can be approximated to be space invariance motion blur zone, thus the constant motion blur image restoration method in application space.For each micro rectangle image block, use diamond search (ds) and can calculate the motion excursion amount, calculate the motion vector of each rectangular image piece in the image, thereby obtain motion blur yardstick and blur direction.Indirect calculation goes out the Linear Fuzzy kernel function (PSF) in each rectangular image piece.To use the blindly restoring image algorithm to the rectangular image piece, and with the PSF that calculates as suction parameter because PSF has been the better approximate of exact value, therefore this blind restoration algorithm only needs iterations seldom just can finish image blurring recovery.Utilize the method to guarantee the of overall importance of motion detection, improve the precision of steady picture, make the integral body of system surely reach 1/4 pixel as precision.
3. utilize FPGA and DSP parallel processing technique, excavate parallel characteristics and the computational resource of these processors
The present invention has designed centered by large-scale FPGA in conjunction with many DSP of high-performance miniaturization digital picture real-time processor, take full advantage of FPGA and DSP simultaneous resource and treatment technology, FPGA finishes image preliminary treatment and electronic steady image Mathematical Models, three DSP are parallel to finish respectively Image Restoration Algorithm and motion compensation based on overall motion estimation and digital filtering, spatially-variable motion blur, realizes the real-time processing of video image.Propose the electronic steady image system principle diagram of meter as shown in Figure 7.
The present invention is according to the multi-mode real-time electronic image stabilizing algorithm of studying, take following electronic steady image system parallel signal to process key technology: the configuration of FPGA system resource and image preliminary treatment realization, the basic thought of parallel processing is that functional module is divided into a plurality of submodules, submodule is carried out parallel processing, and multichannel is selected corresponding output.The method can make the area change of circuit, and power consumption increases.The saving of module whole power consumption comes from reducing by half of submodule clock frequency, and the power that frequency is saved still can make total power consumption significantly decrease after offsetting the power consumption that is produced by the hardware increase.Its shortcoming is that hardware spending increases, clock generating distortion (Clock Skew).Image Pretreatment Algorithm, usually all be to the single channel data add and subtract, the operational processes such as filtering, and Virtex-4SX35 has 192 DSP modules, can satisfy the parallel processing of multiplex data stream fully.In algorithm optimization, take into full account FPGA with the IP kernel resource.Since IP kernel be a kind of predefined, and through the sophisticated functions module of checking, generally realized the optimization of performance and resource, can be when realizing same function, conserve system resources.In Virtex-4FPGA, the power consumption of each CLB reduces half, and quiescent dissipation just reduces 40%, and dynamic power consumption then reduces 50%.
The concurrency characteristics that the present invention has according to size and the FPGA of filter, design 3 * 3 Gaussian filters as shown in Figure 8, at first utilize the block RAM of FPGA inside to consist of two dual port RAMs, be used for respectively storing n-1 and the capable input of n-2 data, distribute simultaneously 9 registers to be used for storing the ephemeral data that participates in filtering operation.When capable k-2 the data of n write register R33, read at first respectively k-2 capable data of n-1 and n-2 and deposit register register R23 and R13 in from RAM, then R33 will deposit respectively the position of originally storing R23 and R13 value among the RAM in R23 successively.In each clock cycle 9 elements of R11 to R33 are carried out computing according to Gauss's window, and operation result is exported as the filtering result.After finishing a filtering, the value of register R3i and R2i is moved into respectively (i=1,2,3) among R2i and the R1i.Carry out successively according to this flow process, can finish the filtering to the capable data of n-1.
Parallel DSP system and parallel Real-time Electronic Image Stabilizing Algorithm realize: the parallel system that the present invention is designed, the data of memory block are surely looked like the processing of algorithm, namely the choosing of characteristic point, mate, the asking for and compensating of tracking, motion vector; Utilizing DAT COPY () function to deliver to display-memory to the data of processing shows, forwards simultaneously the processing of proceeding the next frame image to.The transplanting of completion code on DSP, the code that next will be transplanted to exactly on the DSP is tested to reach expected result under CCS running environment.
Parallel real time electronic steady image systematic function test and appraisal speed-up ratio refers to certain specific application, uses the execution speed of parallel algorithm with respect to using the fast multiple of serial algorithm execution speed.The efficient of parallel system refers to the ratio of speed-up ratio and processor number.Amdahal law, Gustafson-Barsis law and Sun ﹠amp; The Ni law is applicable to respectively fixation problem scale, set time and the fixedly speed-up ratio analysis of these 3 kinds of performance models of utilance.Here consider fixation problem scale performance model, according to the Amdahal law, the steady job load of supposing problem is W, and the percentage that wherein must sequentially carry out is α, and when disregarding all expenses, speed-up ratio S is:
Wherein, n is the processor number.By following formula as seen, when n → ∞, S → 1/ α, namely speed-up ratio can improve along with the increase of processor number, but has the limit, and this limit is to be determined by problem itself.In fact, along with the increase of processor number, overhead can be increasing.If overhead is W
0, then (8) formula should be revised as:
When n → ∞, S → 1/ (α+W/W
0), namely overhead so that the limit of speed-up ratio reduced.
Obviously, the efficient realization of the high-precision Electronic Image Stabilization of multi-mode in system, fully parallel characteristics and the computational resource of these processors of excavation only in this way just can reach the set goal.
Claims (5)
1. multi-mode real-time electronic image stabilizing system, this system comprises that mainly Mathematical Modeling and pretreatment module, interframe are surely as steady picture module in module and the frame; Wherein interframe surely comprises ROI-based overall motion estimation unit, motion vector filtering unit, motion determining means and motion compensation units as module; Mathematical Modeling surely is connected as the ROI-based overall motion estimation unit in the module with interframe with pretreatment module, ROI-based overall motion estimation unit is connected surely with the motion vector filtering unit respectively, and the picture module is connected with frame, motion vector filtering unit, motion determining means and motion compensation units are linked in sequence successively, and motion compensation units surely is connected with the DSP parallel processing of being connected as module respectively with in the frame; It is characterized in that:
Mathematical Modeling and pretreatment module are set up the multi-mode electronics surely as Mathematical Modeling and finish the judgement of input video type of sports; Interframe is surely finished as the ROI-based overall motion estimation unit in the module area-of-interest is carried out overall motion estimation, image processed estimate present frame with respect to the global motion side-play amount of reference frame, namely by camera parameters or change in location and the variation of the whole image that causes, then by the motion vector filtering unit image is carried out filtering, and filtered image is sent to the motion determining means; The motion determining means is analyzed the global motion vector that detects, and judges that this motion is that randomized jitter by the shooting carrier causes, still belong to video camera normal scan motion, or both has concurrently; Motion compensating module is according to being not intended to component motion as parameter to be compensated, directly makes pixel on the image do equivalent in the other direction according to the motion excursion that is detected and moves, to realize the row, column restructuring of image; In the frame surely as module for fuzzy in the picture frame that is caused by dither, carry out the image restoration of the spatially-variable motion blur of I picture;
Wherein, adopt regional partitioning to carry out the image restoration of the spatially-variable motion blur of I picture, be specially: image segmentation is become a lot of tiny areas, and each zone is space invariance motion blur zone; For each micro rectangle image block, use diamond search (ds) and calculate the motion excursion amount, calculate the motion vector of each rectangular image piece in the image, thereby obtain motion blur yardstick and blur direction; And then calculate Linear Fuzzy kernel function in each rectangular image piece; To use the blindly restoring image algorithm to the rectangular image piece, and with the Linear Fuzzy kernel function calculated as suction parameter, iteration is finished image blurring recovery;
Adopt FPGA to finish the parallel processing of view data in conjunction with many DSP miniaturization digital picture real-time processor, FPGA finishes image preliminary treatment and electronic steady image Mathematical Models, three DSP are parallel to finish respectively Image Restoration Algorithm and motion compensation based on overall motion estimation and digital filtering, spatially-variable motion blur, realizes the real-time processing of video image;
Wherein, digital filtering adopts 3 * 3 Gaussian filters, at first utilizes the block RAM of FPGA inside to consist of two dual port RAMs, is used for respectively storing n-1 and the capable input of n-2 data, distributes simultaneously 9 registers to be used for storing the ephemeral data that participates in filtering operation; When capable k-2 the data of n write register R33, read at first respectively k-2 capable data of n-1 and n-2 and deposit register register R23 and R13 in from RAM, then R33 will deposit respectively the position of originally storing R23 and R13 value among the RAM in R23 successively; In each clock cycle 9 elements of R11 to R33 are carried out computing according to Gauss's window, and operation result is exported as the filtering result; After finishing a filtering, the value of register R3i and R2i is moved into respectively (i=1,2,3) among R2i and the R1i; Carry out successively according to this flow process, can finish the filtering to the capable data of n-1.
2. a kind of multi-mode real-time electronic image stabilizing system as claimed in claim 1, it is characterized in that: described motion compensation units adopts the panoramic picture compensation method based on Kalman filtering to compensate, at first carry out the parameter Estimation of global motion, next calculates component motion, then utilize the recursive Kalman filtering device to the accumulation global motion vector sequence of original video sequence, carry out the Sequential filter processing according to the time sequencing that observed quantity arrives, extract again corresponding jittering component with the compensation present image, the boundary information of last combining image splicing reconstructing lost, the integrality of assurance image.
3. a kind of multi-mode real-time electronic image stabilizing system as claimed in claim 2 is characterized in that: Mathematical Modeling and pretreatment module adopt Rigid model to translational motion with around the rotatablely moving of optical axis; For the translation under the degree of freedom, Rotation and Zoom motion, set up respectively similarity transformation model and affine Transform Model; For translation, level sweep, vertically sweep, rotation, lens zoom motion, adopt Perspective transformation model.
4. a kind of multi-mode real-time electronic image stabilizing system as claimed in claim 3, it is characterized in that: ROI-based overall motion estimation unit at first finishes the area-of-interest characteristic block and point extracts, further estimate amount of exercise by tracking characteristics point, by setting up the gray distribution features function, determine effective characteristic area according to gray distribution features numerical value in the zone, and extraction characteristic quantity, this characteristic quantity mates in reference picture, obtain the motion change value of characteristic quantity, these local amount of motion are brought into the global motion amount that can fit the picture of publishing picture in the image overall motion model.
5. a kind of multi-mode real-time electronic image stabilizing system as claimed in claim 4 is characterized in that: at first local amount of motion was carried out error analysis before calculating the global motion amount, and the low local amount of motion of precision is removed, guarantee surely to reach inferior pixel as precision.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110084163 CN102148934B (en) | 2011-04-02 | 2011-04-02 | A multi-mode real-time electronic image stabilization system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110084163 CN102148934B (en) | 2011-04-02 | 2011-04-02 | A multi-mode real-time electronic image stabilization system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN102148934A CN102148934A (en) | 2011-08-10 |
| CN102148934B true CN102148934B (en) | 2013-02-06 |
Family
ID=44422902
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201110084163 Expired - Fee Related CN102148934B (en) | 2011-04-02 | 2011-04-02 | A multi-mode real-time electronic image stabilization system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102148934B (en) |
Families Citing this family (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102427505B (en) * | 2011-09-29 | 2013-11-13 | 深圳万兴信息科技股份有限公司 | Video image stabilization method and system on the basis of Harris Corner |
| CN102395043B (en) * | 2011-11-11 | 2013-09-11 | 北京声迅电子股份有限公司 | Video quality diagnosing method |
| CN102497494A (en) * | 2011-12-09 | 2012-06-13 | 首都师范大学 | FPGA (Field Programmable Gate Array)-based motion estimation module in high-speed image stabilizing system |
| CN103813056B (en) * | 2012-11-15 | 2016-03-16 | 浙江大华技术股份有限公司 | A kind of digital image stabilization method and device |
| CN103841297B (en) * | 2012-11-23 | 2016-12-07 | 中国航天科工集团第三研究院第八三五七研究所 | A kind of electronic image stabilization method being applicable to resultant motion shooting carrier |
| CN103177455B (en) * | 2013-03-20 | 2016-04-20 | 南京理工大学 | Based on the implementation method of the KLT Moving Target Tracking Algorithm of multi-core DSP |
| KR102072014B1 (en) * | 2013-09-16 | 2020-01-31 | 에스케이 텔레콤주식회사 | Method and Apparatus for Using Image Stabilization |
| CN104125470B (en) * | 2014-08-07 | 2017-06-06 | 成都瑞博慧窗信息技术有限公司 | A kind of method of transmitting video data |
| CN104125471B (en) * | 2014-08-07 | 2016-01-20 | 成都瑞博慧窗信息技术有限公司 | A kind of video image compressing method |
| CN106357958B (en) * | 2016-10-10 | 2019-04-16 | 山东大学 | A kind of swift electron digital image stabilization method based on Region Matching |
| US11240407B2 (en) * | 2016-10-31 | 2022-02-01 | Eizo Corporation | Image processing device, image display device, and program |
| WO2018133077A1 (en) * | 2017-01-22 | 2018-07-26 | 四川金瑞麒智能科学技术有限公司 | Environmental information acquisition and feedback system and method for intelligent wheelchair |
| CN107197121B (en) * | 2017-06-14 | 2019-07-26 | 长春欧意光电技术有限公司 | A kind of electronic image stabilization method based on on-board equipment |
| US10534837B2 (en) * | 2017-11-13 | 2020-01-14 | Samsung Electronics Co., Ltd | Apparatus and method of low complexity optimization solver for path smoothing with constraint variation |
| CN108111760B (en) * | 2017-12-26 | 2019-09-10 | 北京理工大学 | A kind of electronic image stabilization method and system |
| CN110677578A (en) * | 2019-08-14 | 2020-01-10 | 北京理工大学 | Mixed image stabilization method and device based on bionic eye platform |
| CN113326722B (en) * | 2020-02-29 | 2023-06-02 | 湖南超能机器人技术有限公司 | Image blurring detection method and device based on sequence mode |
| CN114485473B (en) * | 2022-02-21 | 2024-01-30 | 上海电机学院 | Laser interference phase demodulation method based on component synthesis and gradient projection |
| CN115801973B (en) * | 2022-11-09 | 2024-04-12 | 北京航空航天大学 | A real-time electronic image stabilization system and method for waterfall video data based on FPGA |
| CN117440248B (en) * | 2023-12-21 | 2024-05-03 | 西安松果电子科技有限公司 | Method and system for realizing target servo intelligent control based on axial image stabilization technology |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101511024A (en) * | 2009-04-01 | 2009-08-19 | 北京航空航天大学 | Movement compensation method of real time electronic steady image based on motion state recognition |
| CN101924874A (en) * | 2010-08-20 | 2010-12-22 | 北京航空航天大学 | A Real-Time Electronic Image Stabilization Method Based on Matching Block Grading |
| CN101951463A (en) * | 2010-05-19 | 2011-01-19 | 上海稳像信息技术有限公司 | Real time video image stabilization method based on simple fast global motion parameter estimation |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8121409B2 (en) * | 2008-02-26 | 2012-02-21 | Cyberlink Corp. | Method for handling static text and logos in stabilized images |
| JP5284048B2 (en) * | 2008-11-12 | 2013-09-11 | キヤノン株式会社 | Image processing apparatus, imaging apparatus, and image processing method |
-
2011
- 2011-04-02 CN CN 201110084163 patent/CN102148934B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101511024A (en) * | 2009-04-01 | 2009-08-19 | 北京航空航天大学 | Movement compensation method of real time electronic steady image based on motion state recognition |
| CN101951463A (en) * | 2010-05-19 | 2011-01-19 | 上海稳像信息技术有限公司 | Real time video image stabilization method based on simple fast global motion parameter estimation |
| CN101924874A (en) * | 2010-08-20 | 2010-12-22 | 北京航空航天大学 | A Real-Time Electronic Image Stabilization Method Based on Matching Block Grading |
Non-Patent Citations (1)
| Title |
|---|
| 朱娟娟.电子稳像理论及其应用研究.《西安电子科技大学博士学位论文》.2009,(第7期), * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN102148934A (en) | 2011-08-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102148934B (en) | A multi-mode real-time electronic image stabilization system | |
| CN110378264B (en) | Target tracking method and device | |
| Barranco et al. | Parallel architecture for hierarchical optical flow estimation based on FPGA | |
| Marzat | Real-time dense and accurate parallel optical flow using CUDA | |
| CN111539872A (en) | Real-time electronic image stabilization method for video image under random jitter interference | |
| CN102509071B (en) | Optical flow computation system and method | |
| CN111899278B (en) | Unmanned aerial vehicle image rapid target tracking method based on mobile terminal | |
| CN116433728A (en) | DeepSORT target tracking method for shake blur scene | |
| Kunz et al. | An FPGA-optimized architecture of horn and schunck optical flow algorithm for real-time applications | |
| CN114584785A (en) | Real-time image stabilizing method and device for video image | |
| Palaniappan et al. | Parallel flux tensor analysis for efficient moving object detection | |
| Tomasi et al. | High-performance optical-flow architecture based on a multi-scale, multi-orientation phase-based model | |
| Senst et al. | Robust modified L 2 local optical flow estimation and feature tracking | |
| CN108282597A (en) | A kind of real-time target tracing system and method based on FPGA | |
| Hu et al. | High temporal resolution-based temporal iterative tracking for high framerate and ultra-low delay dynamic tracking system | |
| CN109523573A (en) | The tracking and device of target object | |
| Chen et al. | A fast and energy efficient FPGA-based system for real-time object tracking | |
| Xie et al. | Hierarchical quadtree feature optical flow tracking based sparse pose-graph visual-inertial SLAM | |
| Mahmoudi et al. | Real-time GPU-based motion detection and tracking using full HD videos | |
| CN117237417A (en) | System for realizing optical flow tracking based on image and imu data hardware | |
| CN108492308B (en) | A method and system for determining variational optical flow based on mutual structure-guided filtering | |
| Ham et al. | Developing a smart camera for gesture recognition in HCI applications | |
| CN116193103A (en) | Video picture jitter level assessment method | |
| Czuni et al. | Motion segmentation and tracking with edge relaxation and optimization using fully parallel methods in the cellular nonlinear network architecture | |
| Priyanka et al. | Implementation of Sobel Edge Detection with Image Processing on FPGA |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C17 | Cessation of patent right | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20130206 Termination date: 20140402 |