CN107223275A - The method and system of multichannel sensing data fusion - Google Patents
The method and system of multichannel sensing data fusion Download PDFInfo
- Publication number
- CN107223275A CN107223275A CN201680004510.3A CN201680004510A CN107223275A CN 107223275 A CN107223275 A CN 107223275A CN 201680004510 A CN201680004510 A CN 201680004510A CN 107223275 A CN107223275 A CN 107223275A
- Authority
- CN
- China
- Prior art keywords
- state
- imaging device
- movable equipment
- determined
- observer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 129
- 230000004927 fusion Effects 0.000 title claims description 15
- 238000003384 imaging method Methods 0.000 claims description 487
- 230000000007 visual effect Effects 0.000 claims description 93
- 238000012545 processing Methods 0.000 claims description 42
- 238000005259 measurement Methods 0.000 claims description 29
- 238000005070 sampling Methods 0.000 claims description 9
- 230000005055 memory storage Effects 0.000 claims description 3
- 230000008859 change Effects 0.000 description 27
- 230000033001 locomotion Effects 0.000 description 21
- 238000010586 diagram Methods 0.000 description 20
- 230000001133 acceleration Effects 0.000 description 15
- 230000007613 environmental effect Effects 0.000 description 14
- 238000004891 communication Methods 0.000 description 12
- 238000009826 distribution Methods 0.000 description 12
- 239000011159 matrix material Substances 0.000 description 11
- 238000003860 storage Methods 0.000 description 9
- 241001269238 Data Species 0.000 description 8
- 230000006870 function Effects 0.000 description 8
- 238000001914 filtration Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 238000012795 verification Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 238000012360 testing method Methods 0.000 description 5
- 230000004888 barrier function Effects 0.000 description 4
- 238000010168 coupling process Methods 0.000 description 4
- 238000005859 coupling reaction Methods 0.000 description 4
- 238000006073 displacement reaction Methods 0.000 description 4
- 238000002474 experimental method Methods 0.000 description 4
- 239000013598 vector Substances 0.000 description 4
- 238000005267 amalgamation Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000005286 illumination Methods 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000005183 dynamical system Methods 0.000 description 2
- 230000003447 ipsilateral effect Effects 0.000 description 2
- 230000011664 signaling Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 2
- 241001465754 Metazoa Species 0.000 description 1
- NBIIXXVUZAFLBC-UHFFFAOYSA-N Phosphoric acid Chemical compound OP(O)(O)=O NBIIXXVUZAFLBC-UHFFFAOYSA-N 0.000 description 1
- 238000010521 absorption reaction Methods 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000002411 adverse Effects 0.000 description 1
- 239000000969 carrier Substances 0.000 description 1
- JZVJCTVXALSTOA-XMPIZRASSA-N chembl1288988 Chemical compound C1=CC(OC)=CC=C1C[C@H](N(C)C(=O)[C@H](CO)NC(=O)[C@@H](C)NC(=O)[C@@H](N(C1=O)C)C2)C(=O)N[C@@H](C)C(=O)N(C)[C@H]1CC(C=C1)=CC=C1OC1=CC2=CC=C1O JZVJCTVXALSTOA-XMPIZRASSA-N 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000000151 deposition Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 230000004064 dysfunction Effects 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000010304 firing Methods 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000005036 potential barrier Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000000452 restraining effect Effects 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 230000009329 sexual behaviour Effects 0.000 description 1
- 241000894007 species Species 0.000 description 1
- 230000002463 transducing effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01D—MEASURING NOT SPECIALLY ADAPTED FOR A SPECIFIC VARIABLE; ARRANGEMENTS FOR MEASURING TWO OR MORE VARIABLES NOT COVERED IN A SINGLE OTHER SUBCLASS; TARIFF METERING APPARATUS; MEASURING OR TESTING NOT OTHERWISE PROVIDED FOR
- G01D21/00—Measuring or testing not otherwise provided for
- G01D21/02—Measuring two or more variables by means not covered by a single other subclass
-
- G—PHYSICS
- G08—SIGNALLING
- G08C—TRANSMISSION SYSTEMS FOR MEASURED VALUES, CONTROL OR SIMILAR SIGNALS
- G08C17/00—Arrangements for transmitting signals characterised by the use of a wireless electrical link
- G08C17/02—Arrangements for transmitting signals characterised by the use of a wireless electrical link using a radio link
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Studio Devices (AREA)
- Testing Or Calibration Of Command Recording Devices (AREA)
- Position Fixing By Use Of Radio Waves (AREA)
- Navigation (AREA)
Abstract
The method and system merged there is provided a kind of multichannel sensing data.Wherein it is determined that the method for the state of movable equipment includes：The second sensing data got in second sensor system is unavailable or in period for not updating, the first sensing data got according to first sensor system determines the predicted state of the movable equipment；When it is determined that second sensing data that the second sensor system is got can use or update, according to second sensing data, the first observer state of the movable equipment is determined；According to the first deviation between first observer state and the predicted state, it is determined whether update the state of the movable equipment according to first observer state, wherein, first deviation is used to indicate whether second sensing data can use.
Description
Copyright notice
This patent document disclosure includes material protected by copyright.The copyright is all for copyright holder.Copyright
Owner does not oppose that anyone replicates the patent document in the presence of the proce's-verbal of Patent＆Trademark Office and archives or should
Patent is disclosed.
Technical field
The present embodiments relate to navigation field, and relate more specifically to the method for multichannel sensing data fusion and be
System.
Background technology
Unmanned vehicle, for example, unmanned vehicle (Unmanned Aerial Vehicle, referred to as " UAV "), also referred to as
Unmanned plane, unpiloted automobile etc. are in an increasingly wide range of applications in many fields such as search and rescue, exploration.Generally using long-range
Controller is controlled and navigated to unmanned vehicle.But in some scenes, unmanned vehicle can be carried out certainly according to environmental information
Main remote sensing and navigation.
The positional information and movable information of unmanned vehicle can be obtained by multiple sensors to realize the control of unmanned vehicle
System and navigation.But the sensor-based system of unmanned vehicle is not too preferable at present.For example, most sensor-based system is all to use
Single line decision-making mode does not take into full account redundancy nargin, and single line decision-making mode is because specific environment is (for example, indoor and outdoor, height
Empty, low latitude) when causing one or more sensor faults or getting inaccurate data, it is impossible to select available sensor
And/or sensing data.Thus the security performance of unmanned vehicle can be reduced.
Therefore, it is necessary to provide a kind of method of multichannel sensing data fusion, the security performance of unmanned vehicle is improved.
The content of the invention
The embodiment of the present invention provides the method and system of multichannel sensing data fusion, can be by movable equipment
The data that multiple sensor is got are merged, and improve the security performance of movable equipment.
First aspect includes there is provided a kind of method for being used to determine the state of movable equipment, the movable equipment
First sensor system and second sensor system, the data of the first sensor system and the second sensor system are adopted
Sample frequency is different, and methods described includes：The second sensing data got in the second sensor system is unavailable or not more
In the new period, the first sensing data got according to the first sensor system determines the pre- of the movable equipment
Survey state；When it is determined that second sensing data that the second sensor system is got can use or update, according to described
Second sensing data, determines the first observer state of the movable equipment；According to first observer state and the prediction
The first deviation between state, it is determined whether the state of the movable equipment is updated according to first observer state, wherein,
First deviation is used to indicate whether second sensing data can use.
The embodiment of the present invention according to first sensor system it is determined that during the state of movable equipment, get
First sensing data determines the predicted state of movable equipment, and the second sensing data got according to second sensor system
The observer state of movable equipment is determined, passes through the deviation between predicted state and observer state, it is determined whether according to observation shape
State updates the state of movable equipment.Thereby, it is possible to by being verified to each sensing system, in multichannel sensor system
Suitable sensing system is selected, and the sensing data got according to the suitable sensing system selected updates removable
The state of dynamic equipment, it is possible to increase the security performance of movable equipment.
Second aspect is set there is provided a kind of method for being used to select imaging device in movable equipment, described may move
It is standby to be above provided with multiple imaging devices, the multiple imaging device include at least one first imaging device and at least one second
Imaging device, first imaging device is operated under multi-vision visual pattern, and second imaging device is operated in monocular vision
Under pattern, methods described includes：Determine of each imaging device relative to other imaging devices in the multiple imaging device
One relative position, and each imaging device is relative to the second relative position of the movable equipment；According to selection information,
The selection target imaging device from the multiple imaging device, wherein, the selection information includes at least one in following message
Kind：Each imaging device relative to the imaging device object within the vision or ground distance, pass through described at least one
The parallax and the work of the multiple imaging device of match point at least frame stereo-picture that individual first imaging device is got
Make environment, wherein, the distance is according to first relative position and second relative position determination；Using the mesh
Mark imaging device and obtain view data.
The method for being used to select imaging device in movable equipment of the embodiment of the present invention, according to selection information, from many
Selection target imaging device in individual imaging device, thus, it is possible to select more particularly suitable imaging device to obtain more accurate figure
As data, to be merged with the data that other sensors system is got in the view data for getting imaging device
When, it is determined that the state of more accurate movable equipment, improves the security performance of movable equipment.
The third aspect is used to determine to be used for the availability of the imaging device of visual sensing on movable equipment there is provided a kind of
Method, including：The view data got according to multiple imaging devices for visual sensing, determines the movable equipment
Multiple first observer states；The sensing data got according to Inertial Measurement Unit IMU, determines many of the movable equipment
Individual predicted state；According to the first deviation and the first predetermined threshold value between predicted state and the first observer state, it is determined that each use
In the availability of the imaging device of visual sensing.
The side for being used to determine to be used on movable equipment the availability of the imaging device of visual sensing of the embodiment of the present invention
Method, the view data that the sensing data got by Inertial Measurement Unit is got to multiple imaging devices is verified, really
Determine the availability of imaging device.Thereby, it is possible to get reliable view data by available imaging device, inciting somebody to action
When the view data that available imaging device is got is merged with the data that other sensors system is got, it is determined that more
The state of accurate movable equipment, improves the security performance of movable equipment.
There is provided a kind of system for being used to determine the state of movable equipment for fourth aspect.Including：Memory, for depositing
Storage program, at least one processor, by performing the program in the memory, is either individually or collectively used for：Obtain and institute
The sensing data that the associated multiple sensors of movable equipment are obtained is stated, wherein, the multiple sensor includes the first sensing
The data sampling frequency of device system and second sensor system, the first sensor system and the second sensor system is not
Together；The second sensing data got in the second sensor system is unavailable or in period for not updating, according to described
The first sensing data that one sensing system is got, determines the predicted state of the movable equipment；When it is determined that described second
When second sensing data that sensing system is got is available or updates, according to second sensing data, it is determined that described
First observer state of movable equipment；According to the first deviation between first observer state and the predicted state, really
The fixed state that the movable equipment whether is updated according to first observer state, wherein, first deviation is used to indicate
Whether second sensing data can use.
There is provided a kind of system for being used to select imaging device in movable equipment in terms of 5th, including：Memory,
For storage program, at least one processor, by performing the program of memory storage, is either individually or collectively used for：It is determined that
Each imaging device is relative to the first relative position of other imaging devices, and each imaging device in multiple imaging devices
Relative to the second relative position of the movable equipment, wherein, the multiple imaging device is arranged on the movable equipment
On, the multiple imaging device includes at least one first imaging device and at least one second imaging device, described the first one-tenth
As equipment is operated under multi-vision visual pattern, second imaging device is operated under monocular vision pattern；According to selection information,
The selection target imaging device from the multiple imaging device, wherein, the selection information includes at least one in following message
Kind：Each imaging device relative to the imaging device object within the vision or ground distance, pass through described at least one
The parallax and the work of the multiple imaging device of match point at least frame stereo-picture that individual first imaging device is got
Make environment, wherein, the distance is according to first relative position and second relative position determination；Control the mesh
Mark imaging device and obtain view data.
6th aspect is used to determine to be used for the availability of the imaging device of visual sensing on movable equipment there is provided a kind of
System, including：Memory, for storage program；At least one processor, by performing the program of memory, individually or
It is commonly used for：The view data got according to multiple imaging devices for visual sensing, determines many of the unmanned plane
Individual first observer state；The sensing data got according to Inertial Measurement Unit IMU, determines the multiple pre- of the movable equipment
Survey state；According to the first deviation and the first predetermined threshold value between predicted state and the first observer state, it is determined that each be used to regard
Feel the availability of the imaging device of sensing.
There is provided a kind of system for being used to determine the state of movable equipment in terms of 7th, including：Acquisition module, is used for
The sensing data that the multiple sensors associated with the movable equipment are obtained is obtained, wherein, the multiple sensor includes
First sensor system and second sensor system, the data of the first sensor system and the second sensor system are adopted
Sample frequency is different；Determining module, the second sensing data for being got in the second sensor system is unavailable or not more
In the new period, the first sensing data got according to the first sensor system determines the pre- of the movable equipment
Survey state；The determining module, is additionally operable to when second sensing data that the determination second sensor system is got can
With or update when, according to second sensing data, determine the first observer state of the movable equipment；The determination mould
Block, is additionally operable to according to the first deviation between first observer state and the predicted state, it is determined whether according to described
One observer state updates the state of the movable equipment, wherein, first deviation is used to indicate second sensing data
Whether can use.
Eighth aspect there is provided it is a kind of be used in movable equipment select imaging device system, including：Determine mould
Block, for determining the first relative position of each imaging device in multiple imaging devices relative to other imaging devices, and it is described
Each imaging device relative to the movable equipment the second relative position, wherein, the multiple imaging device is arranged on institute
State on movable equipment, the multiple imaging device includes at least one first imaging device and at least one second imaging is set
Standby, first imaging device is operated under multi-vision visual pattern, and second imaging device is operated under monocular vision pattern；
Processing module, for according to selection information, the selection target imaging device from the multiple imaging device, wherein, the selection
Information includes at least one of following message：Each imaging device relative to the imaging device object within the vision or
The distance on ground, match point at least frame stereo-picture got by least one described first imaging device are regarded
The working environment of difference and the multiple imaging device, wherein, the distance is according to first relative position and described second
What relative position was determined；The processing module, is additionally operable to control the target imaging equipment to obtain view data.
9th aspect is used to determine to be used for the availability of the imaging device of visual sensing on movable equipment there is provided a kind of
System, including：First processing module, for the view data got according to multiple imaging devices for visual sensing,
Determine multiple first observer states of the movable equipment；Second processing module, for being obtained according to Inertial Measurement Unit IMU
The sensing data arrived, determines multiple predicted states of the movable equipment；3rd processing module, for according to predicted state with
The first deviation and the first predetermined threshold value between first observer state, it is determined that being each used for the available of the imaging device of visual sensing
Property.
Tenth aspect is there is provided a kind of storage medium, the storage medium store instruction, and the instruction can be used for performing
Method in first aspect.
Tenth one side is there is provided a kind of storage medium, and the storage medium store instruction, the instruction can be used for holding
Method in row second aspect.
12nd aspect is there is provided a kind of storage medium, and the storage medium store instruction, the instruction can be used for holding
Method in the row third aspect.
Therefore, in embodiments of the present invention, verified by the data to multichannel sensor system, in multiple sensor
In select suitable sensing system, and will be melted by the data for getting the suitable sensing system selected
Close, determine the more accurate state of movable equipment, it is possible to increase the security performance of movable equipment.
Brief description of the drawings
Fig. 1 is the schematic diagram of the movable equipment with multiple sensing systems according to embodiments of the present invention；
Fig. 2 (a) and (b) are that sensor controller according to embodiments of the present invention is communicated with multiple sensing systems
Schematic diagram；
Fig. 3 is the indicative flowchart of the method for the state of determination movable equipment according to embodiments of the present invention；
Fig. 4 is the schematic diagram of the different cycles sample frequency of two sensing systems shown in Fig. 3；
Fig. 5 is another indicative flowchart of the method for the state of determination movable equipment according to embodiments of the present invention；
Fig. 6 is the indicative flowchart of the method for the state of determination movable equipment according to another embodiment of the present invention；
Fig. 7 is the indicative flowchart of the method for selection imaging device according to embodiments of the present invention；
Fig. 8 is the schematic diagram of hand and eye calibrating method according to embodiments of the present invention；
Fig. 9 is the schematic diagram of the method according to embodiments of the present invention that vision sensor is selected according to pre-determined distance threshold value；
Figure 10 is the schematic diagram of the method according to embodiments of the present invention that vision sensor is selected according to preset height threshold value；
Figure 11 is the schematic diagram of binocular camera according to embodiments of the present invention；
Figure 12 is the schematic diagram of the visual sensing scope of movable equipment according to embodiments of the present invention；
Figure 13 is the indicative flowchart of the method for the availability of determination imaging device according to embodiments of the present invention；
Figure 14 is the redundancy decision-making for being used to select sensor and/or data at different conditions according to embodiments of the present invention
The indicative flowchart of method；
Figure 15 is the schematic block diagram for being used to determine the system of the state of movable equipment according to embodiments of the present invention；
Figure 16 is the schematic of the system for being used to select imaging device in movable equipment according to embodiments of the present invention
Block diagram；
Figure 17 is according to embodiments of the present invention for determining to be used for the imaging device of visual sensing on movable equipment
The schematic block diagram of the system of availability.
Figure 18 is the schematic block diagram of movable equipment according to another embodiment of the present invention.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present disclosure, the technical scheme in the embodiment of the present disclosure is carried out clear, complete
Site preparation is described.
It should be understood that specific example herein is intended merely to help those skilled in the art to more fully understand that the disclosure is real
Apply example, and the scope of the unrestricted embodiment of the present disclosure.
The embodiment of the present invention can apply to various types of movable equipments.Movable equipment in the embodiment of the present invention
Can be moved under any suitable environment, for example, in air (for example, determine wing aircraft, gyroplane, or both do not determined the wing or
There is no the aircraft of rotor), in water (for example, steamer or submarine), land (for example, automobile or train), space is (for example, too
Empty aircraft, satellite or detector), and any of the above environment any combinations.Movable equipment can be aircraft, for example nobody
Machine (Unmanned Aerial Vehicle, referred to as " UAV ").In certain embodiments, movable equipment can carry life
Body, for example, human or animal.
In embodiments of the present invention, movable equipment includes one or more sensing systems, for obtaining all kinds
Data.For example, the letter for obtaining the object in the information related to the state of movable equipment, environmental information or environment
Breath.Sensor in the embodiment of the present invention includes position sensor (for example, global positioning system (Global Positioning
System, referred to as " GPS ") sensor, triangulation network alignment sensor), vision sensor (for example, visible ray can be detected, it is red
The imaging device of outer light or ultraviolet light, for example, video camera), proximity transducer or range sensor be (for example, supersonic sensing
Device, radar, flight time or depth camera), inertial sensor is (for example, accelerometer, gyroscope, Inertial Measurement Unit
(Inertial Measurement Units, referred to as " IMU ")), height sensor, attitude transducer (for example：Compass),
Pressure sensor (for example, barometer), audio sensor (for example, microphone) or magnetic field sensor are (for example, magnetometer, electromagnetism
Sensor).Movable equipment can include the above-mentioned sensing of any suitable number (for example, one, two, three or more)
The combination of device or the sensor.Further, it is possible to receive data by different types of sensor.
It is understood that different types of sensor can obtain different types of by different metering system measurements
Signal or information (for example, positional information, azimuth information, velocity information, acceleration information, range information, pressure information etc.).Example
Such as, the sensor in movable equipment includes multiple active sensors and multiple passive sensors.For another example, some sensors
Can provide in global coordinate system absolute measurement data (for example, the position data provided by GPS sensor, by compass or
Magnetometer provide attitude data), and some sensors can provide in local coordinate relative measurement data (for example, by
Gyroscope provide relative angular speed, provided by accelerometer relative movement acceleration, by vision sensor provide it is relative
Attitude information, the relative distance information provided by ultrasonic sensor, radar or time-of-flight camera).Under many circumstances, originally
Ground coordinate system is generally defined as the body coordinate system relative to UAV.
In embodiments of the present invention, the status information of movable equipment can include be used for identify movable equipment relative to
The position of three-dimensional (Three Dimensional, referred to as " 3D ") locus of one fixed reference frame or mobile reference frame
Information.For example, positional information includes location information (for example, height, longitude and/or latitude) and azimuth information (for example, rolling
Angle, the angle of pitch and/or yaw angle).Status information can also include movable information, and movable information includes movable equipment 6
Translational speed and/or angular speed and acceleration under one or more of free degree free degree.And one can be passed through
Or multiple sensing systems determine locus and/or motion of the movable equipment under 6DOF (for example, under 3DOF
Position and/or displacement, direction and/or rotation under 3DOF).It can be determined by one or more sensing systems
Movable equipment relative to one or more equipment (for example, remote control, barrier, ground, target object etc.) distance and/or
Relative motion.
In embodiments of the present invention, the data got by sensing system can reflect environmental information.For example, sensing
Data can reflect the type of the environment residing for movable equipment, for example, indoor environment, outdoor environment, low latitude environment, high idle loop
Border.And sensing data can further reflect current environmental condition, described current environmental condition includes weather (for example, fine
It is bright, rain, snow), visibility conditions, wind speed, time of day etc..Further, the environmental information that sensor is got can be with
Include the information of other objects (for example, the barrier being mentioned above) in environment.Alternatively, the information of barrier includes number
Measure information, density information, geometry information and/or spatial positional information.
In embodiments of the present invention, testing result is to enter line sensor by the sensing data for getting multiple sensors
What fusion treatment was obtained.For example, using sensor amalgamation mode by different types of sensor (for example, GPS sensor, inertia
Sensor, vision sensor, radar, sonac etc.) sensing data that gets merged.And it is possible to using biography
Polytype sensing data, these polytype sensing datas can be included absolute measurement data by sensor amalgamation mode
(for example, gps data) and relative measurement data (for example, visual sensing data, radar data or supersonic sensing data).By
This, the limitation or inaccuracy of single sensor can be made up by sensor amalgamation mode, so as to improve testing result
Accuracy and reliability.
In embodiments of the present invention, sensor system control can handle the sensing number that multiple sensing systems are got
According to, and the sensing system and/or sensing data of the state for determining movable equipment can be selected.Sensing system control
Device processed can be arranged on movable equipment, can also be not arranged on movable equipment.Obtained by the sensing system selected
The sensing data got will be transmitted to flight controller.Flight controller can pass through one or more according to sensing data
Velocity of electrons control unit, one or more power units control the motion of movable equipment.For example, by the sensor selected
The sensing data that system is got can be used locus, speed and/or the direction of control movable equipment (for example, logical
Cross a suitable processing unit and/or control module).In addition, sensing system can be provided for determining may move
The sensing data of the surrounding enviroment of equipment, these sensing datas can include weather condition, the distance of potential barrier, geography
Feature locations, position of man-made structures etc..
In existing correlation technique, the performance of Method for Single Sensor System is undesirable.For example, when movable equipment is in severe day
When under the conditions of gas, under indoor environment or near building, GPS sensor system can be restricted.Although difference global is positioned
System (Differential Global Positioning System, referred to as " DGPS ") and real time differential (Real Time
Kinematic, referred to as " RTK ") GPS compared to traditional GPS have higher precision, but these technologies have it is various
Restraining factors have impact on their application.For example, visual sensing system needs substantial amounts of calculating to estimate, and visual sensing system
Precision can be influenceed by picture quality (for example, low image resolution, image blurring), image fault equally can also reduce vision
The performance of sensor-based system.For another example distance sensor system can be influenceed by sensor accuracy and using scope, such as grow away from
From sensor because too conference causes to use in some scenes.In addition, radar sensor property under high light conditions
It is able to can reduce.
In order to reduce in existing correlation technique due to measurement error, this hair caused by the latent defect of single sensing system
The sensing got in bright embodiment by the sensing system that can obtain effective or accurate sensing data selected
Data determine the state of movable equipment.Sensor system control in the embodiment of the present invention can be by will be multiple different
The sensing data that sensing system is got mutually is verified, and determines whether to carry out sensing data by the result of verification
The mode of fusion determines the state of movable equipment.Sensor system control can be different environmental form selection activation
And/or the different sensing system of operation.Further, sensor system control can be according to the validity of sensing data
And/or environment residing for movable equipment it is smooth be switched to another sensing system from a sensing system.
Thus, the method for the multichannel sensing data fusion of the embodiment of the present invention, can utilize the excellent of the sensor selected
Gesture avoids the measurement error and failure of single sensing system.The method of multichannel sensing data fusion can be only by sensing system
Part sensing data in the sensing data got is merged.Thereby, it is possible to ignore insufficient or insecure sensing number
According to, and then the motion of the movable equipment determined under circumstances and/or the accuracy of position can be improved.
In embodiments of the present invention, the sensing system of any suitable quantity and species can be merged.For example, can
So that GPS sensor system, IMU sensing systems and visual sensor system to be merged.Or can be by GPG sensors system
System and INU sensing systems are merged.Or GPS sensor system and visual sensor system can be merged.And
And the sensing data that multiple sensing systems are got can be merged according to any appropriate order, for example, can first by
GPS sensing datas are merged with IMU sensing datas, afterwards by visual sensing data and GPG sensing datas and IMU sensing datas
Merged.
Fig. 1 shows movable equipment 100 according to embodiments of the present invention.Movable equipment 100 can be unmanned plane.And
And multiple sensing systems are provided with movable equipment 100.As shown in Figure 1, multiple sensor-based systems include IMU
110th, GPS sensor 120, and/or multiple vision sensors 130.The sensing data got by multiple sensing systems can
For obtaining position and/or movable information, it is possible thereby to which movable equipment is controlled and/or navigated according to these information.
Multiple sensing systems are communicated with a sensor system control being arranged on movable equipment 140.Alternatively, pass
Sensor controller 140 can also be not arranged on movable equipment 100.Sensor controller 140 includes one or more processing
Device.Sensor controller 140 using redundancy decision mode determine to select under different conditions which sensing system and/or which
A little sensing datas.Here different conditions include sensor fault, inaccurate sensing data or sensing data deviation, may move
Running environment of equipment 100 etc..
Wherein, IMU110 include one or more accelerometers, one or more gyroscopes, one or more magnetometers,
Or the combination of said apparatus.For example, IMU110 can include 3 accelerometers, this 3 accelerometers are used to measure removable
The linear acceleration that equipment is moved on 3-dimensional direction.And including 3 gyroscopes, this 3 gyroscopes are used to measure removable
The angular acceleration that equipment is rotated on 3-dimensional direction.Because IMU110 is rigidly connected on movable equipment, movable equipment
Motion it is consistent with IMU motion.Alternatively, IMU110 can be moved along 6 frees degree relative to movable equipment.
IMU110 can be connected directly between on movable equipment, or IMU110 can be connected to the support on movable equipment
On part.IMU110 forever or can be removably attachable on movable equipment.IMU110 can be provided for indicate that can
The signal of the motion of mobile device, motion here for example can be position, direction, speed and/or acceleration.For example,
IMU110 can obtain the signal for the acceleration for representing movable equipment, and removable set is obtained by carrying out once integration to signal
Standby velocity information, position and/or the directional information of movable equipment can be obtained by integrating twice.
GPS sensor 120 can obtain gps data signal 124 by being communicated with one or more gps satellites 122.GPS
Sensor 120 is rigidly connected on movable equipment, therefore position and the position consistency of movable equipment of GPS sensor 120.
Alternatively, GPS sensor 120 can be moved along 6 frees degree relative to movable equipment.GPS sensor 120 can be direct
It is connected on movable equipment, or GPS sensor 120 can be connected in the support member on movable equipment.
Support member can include load, such as carrier (Carrier) or pay(useful) load (Payload).GPS sensor 120 can be forever
Long or be removably attachable on movable equipment.GPS sensor 120 can be one of the payload of movable equipment
Point.
The gps signal that GPS sensor 120 is got is used to determine position of the movable equipment relative to global coordinate system
(for example, height, longitude, latitude), and can be used for determining that the translational speed and/or acceleration of movable equipment.GPS is passed
Sensor 120 can use any appropriate GPS technology, such as DGPS and RTK-GPS.GPS sensor can be used in arbitrarily closing
The position of movable equipment is determined under suitable precision.Here precision be, for example, meter accuracy (for example, within 10, within 5 meters, 2 meters
Within etc.) or centimetre class precision (for example, within 500cm, within 200cm, within 100cm, within 50cm etc.).
Vision sensor 130 can be it is any can be by obtaining surrounding's ring of target device (for example, target device 102)
The optical signalling in border, and according to the equipment of the optical signalling generation view data got.It is suitable that movable equipment can include
The vision sensor of quantity.Vision sensor in the embodiment of the present invention can be alternately as video camera or imaging device.Can
Selection of land a, vision sensor can be an optical component of a camera or an imaging device.Vision sensor can
To be a part for the different imaging devices that can be worked under various modes.For example, vision sensor can be one or many
One part of individual monocular camera and/or many mesh cameras.
In embodiments of the present invention, imaging device includes at least one and is operated in imaging device under monocular pattern and at least
One imaging device for being operated under many mesh patterns.Wherein, many mesh patterns include bimodulus pattern.As shown in Figure 1, it is imaged
Equipment can include binocular camera 132-1 and binocular camera 132-2, and each binocular camera includes a pair of vision sensors (in figure
It is not shown).A pair of vision sensors are spaced laterally apart on movable equipment, and thus the two vision sensors can be never
Same visual angle provides image, is consequently formed 3 D visual image.For example, two vision sensors can be spaced laterally apart 1m, 500cm
Deng.Binocular camera may be mounted on the same side of movable equipment, or the relative side of movable equipment.One or many
Individual binocular camera may be mounted at the forward and backward, upper and lower of movable equipment or side.Binocular camera is rigid be connected to it is removable
In equipment, therefore the positional information got by binocular camera is consistent with the positional information of movable equipment.Alternatively, binocular
Camera removably can be connected on movable equipment by one or more carriers, so that binocular camera being capable of edge
6 frees degree to move relative to movable equipment.
In embodiments of the present invention, the monocular camera 134 that imaging device includes includes a vision sensor, monocular camera
134 removably can be connected on movable equipment by carrier, so that monocular camera can be along 6 free degree phases
For movable equipment motion.Alternatively, monocular camera can be directly installed on movable equipment, or may be coupled to one
In the individual support member being installed on movable equipment.Monocular camera can be a part for the pay(useful) load of movable equipment.
And monocular camera 134 can obtain the image of the target device 102 in environment.
In embodiments of the present invention, vision sensor 130 can obtain image simultaneously under specific frequency, thus, it is possible to
Produce the time series of view data.Using suitable method (for example：Machine vision algorithm) to the time series of view data
Handled, and determine according to the information obtained after processing the position, direction and/or speed of movable equipment.For example, can lead to
Crossing machine vision algorithm determines one or more characteristic points on each image (for example, the center of the edge of object, object, two
Plant the border of the object of different colours).The combination of any suitable method and method can be for recognizing and provide characteristic point
Numeral expression, for example, accelerating sections testing algorithm, binary robust independence essential characteristic algorithm.Then view data is carried out
Match to identify a series of general characteristics points appeared in the image that two vision sensors are obtained.And then can root
According to general characteristics point determine the movable information of movable equipment, vision sensor relative to movable equipment locus with
And the relative tertiary location between vision sensor.
Although not shown in Fig. 1, the movable equipment 100 of the embodiment of the present invention can also include range sensor system
System, distance sensor system is used for the positional information of movable equipment.Range sensor can any can obtain removable
Equipment and the range sensor of the distance between one or more periphery objects.For example, distance sensor system can include surpassing
Sonic sensor and/or radar sensor.In embodiments of the present invention, range sensor can be by rotating (such as 360 degree)
Obtain the distance and positional information of the multiple objects on movable equipment periphery.Also, the multiple objects on movable equipment periphery
Distance and position can be used for determining locus and/or the movable information of movable equipment.
Fig. 2 (a) and (b) show sensor system control 140 according to embodiments of the present invention and multiple sensor systems
The schematic diagram that system is communicated.Sensor system control 140 removably can be connected with any amount of sensing system.
For example, shown in Fig. 2 (a), sensor system control 140 is communicated with three sensing systems, in such as Fig. 2 (b)
Shown, sensor system control 140 is communicated with N number of sensing system, wherein, N is the integer more than 3.Sensing
Device system controller 140 includes one or more processors, and one or more of processors are connected to removable for acquisition
The sensing data that multiple sensing systems in equipment are got.And sensor system control 140 can be according to getting
Sensing data determine the state of movable equipment.The state of movable equipment can be physical state, and physical state can lead to
Positional information and/or movable information is crossed to characterize.The positional information of movable equipment includes location information and/or directional information.
The movable information of movable equipment includes linear velocity, angular speed, linear acceleration and/or angular acceleration.
In embodiments of the present invention, sensor system control 140 uses redundancy decision making algorithm to multiple different sensors
The validity of the data of system is mutually verified, thus select the sensor that can use in different environments and/or
Sensing data.Sensing data is carried out mutually to verify and confirms to bring extra redundancy, thus, it is possible to improve movable equipment
Safe operation coefficient.
As shown in Fig. 2 (a), multiple sensors described above include a first sensor system 210, one
Individual second sensor system 220 and a 3rd sensor system 230.First sensor system 210 includes IMU, the IMU bags
Include at least one accelerometer and/or gyroscope.Second sensor system 220 includes gps receiver.Alternatively, the second sensing
Device system 220 includes one or more vision sensors, and when including multiple vision sensors, multiple sensors are connected to removable
The different piece of dynamic equipment.When second sensor system 220 includes a gps receiver, 3rd sensor system 230 includes
One or more vision sensors.When second sensor system 220 includes one or more sensors, 3rd sensor system
230 include a gps receiver.Alternatively, vision sensor can include at least one binocular camera and/or at least one list
Mesh camera.
Sensing system 210,220 and 230 has different sample frequencys.For example, first sensor system 210 have than
Second sensor system 220 and the higher sample frequency of 3rd sensor system 230.For example, first sensor system 210
Sample frequency could be arranged to 100-1000Hz, the sample frequency of second sensor system 220 could be arranged to 10-30Hz,
The sample frequency of 3rd sensor system 230 could be arranged to 20-60Hz.Alternatively, IMU gathers the frequency of inertia sensing data
About
200Hz, the frequency of gps receiver collection gps data is about 20Hz, and vision sensor gathers the frequency of view data
Rate is about 50Hz.Because second sensor system 220 and 3rd sensor system 230 are with lower than first sensor system
Sample frequency, therefore occur that first sensor system 210 gets latest data, and second sensor system 220 and
Three sensing systems 230 do not get the period of latest data.
Sensor system control 140 can carry out mutual school to the validity of the sensing data of multiple sensing systems
Test.For example, sensor system control 140 can calculate the deviation between the sensing data that different sensors system is got.
In embodiments of the present invention, sensing system control 140 can calculate the sensing data got by first sensor system 210
And the first deviation of the sensing data got by second sensor system 220, and calculating are obtained by second sensor system 220
The second deviation between the sensing data got and the sensing data got by 3rd sensor system 230.And further
Ground, sensor system control 140 calculates the sensing data that is got by first sensor system 210 and by 3rd sensor system
The 3rd deviation between the sensing data that system 230 is got.It is understood that mutually verification can expand to N number of sensor
The system sensing data different with N groups.For example, shown in fig. 2, mutually verification can 1 sensing system with it is N number of
(1 between sensing system:N), or verification can between N number of sensing system and N number of sensing system (N:N), or
Mutually verification can between x sensing system and y sensing system (x:Y), wherein, the value phase that x value can be with y
Together, x value can also be differed with y value.
In embodiments of the present invention, the first deviation can serve to indicate that the sensing data got by second sensor system
It is whether effective relative to first sensor system.Obtained likewise, the second deviation can serve to indicate that by 3rd sensor system
Sensing data it is whether effective relative to second sensor system and first sensor system.Likewise, the 3rd deviation is used for
Indicate whether the sensing data got by 3rd sensor system is effective relative to first sensor system.
Fig. 3 shows the schematic flow of the method for the physical state of determination movable equipment according to embodiments of the present invention
Figure.The physical state of movable equipment can include position and/or movable information.Method 300 can be by sensing system control
Device is performed, wherein, sensor system control be used to obtaining the first sensing data for being got by first sensor system and by
The second sensing data that second sensor system is got.First sensor system has the first sample frequency, second sensor
System has the second sample frequency, and alternatively, the first sample frequency is higher than the second sample frequency.As shown in figure 3, method
300 include：
S301, in the period that the second sensing data got in second sensor system cannot or not update, according to
The first sensing data that the first sensor system is got, determines the predicted state of the movable equipment；
S302, when it is determined that second sensing data that the second sensor system is got can use or update, root
According to second sensing data, the first observer state of the movable equipment is determined；
For example, as shown in Figure 4, in t1<t<T2 and t2<t<In t3 period, second sensor system
Second sensing data is unavailable or does not update.Within the above-mentioned period, sensor system control can be passed according to by first
Multiple priori physical states that the first data that sensor system is got are determined, determine the predicted state of movable equipment.
As shown in Figure 4, the second sensing data of second sensor system specific at the time of (for example, t1, t2,
T3 etc.) available or renewal.Sensor system control obtains what is got by second sensor system above-mentioned specific at the time of
Second sensing data, and determine according to the second sensing data got the first observer state of movable equipment.
S303, according to the first deviation between first observer state and the predicted state, it is determined whether according to institute
The state that the first observer state updates the movable equipment is stated, wherein, first deviation is used to indicate second sensing
Whether data can use.
In embodiments of the present invention, alternatively, second sensor system includes GPS sensor, when GPS sensor is from two
Or multiple satellite receptions to gps signal when, gps data is available or updates.
In embodiments of the present invention, alternatively, second sensor system includes one or more vision sensors, works as vision
When sensor gets image, visual sensing data are available or update.
It is understood that being passed every time when the second sensing data that second sensor system is got is available or updates
Sensor system controller can all be verified to the second sensing data.Specifically include, determine predicted state and the first observer state
Between the first deviation, the first deviation is one of the deviation between the first observer state and predicted state measurement, the first deviation
Availability for indicating the second sensing data that second sensor system is got.
In embodiments of the present invention, alternatively, the first deviation can be determined by the method for one or more data statistics.
For example, the first deviation can be the mahalanobis distance (Mahalanobis Distance) between predicted state and the first observer state
Or Euclidean distance.Mahalanobis distance is obtained by the way that the distribution of the first observer state and predicted state is carried out into comparing calculation.Predict shape
The distribution of state includes one group of predicted state determined according to priori prediction state, wherein, priori prediction state is in the second sensing
Determined in the period that second sensing data of device system cannot or not update.As shown in Figure 4, it can be passed second
Mahalanobis distance is calculated when the second sensing data that sensor system is got is available or updates.For example, at the t1 moment, mahalanobis distance is
First observer state (at the z of position) and the average value mu of the distribution including multiple predicted state μmBetween measurement distance, it is above-mentioned
Distribution be according in t1<t<What the priori prediction state S determined in the t2 periods was determined, μmWith movable equipment at the t2 moment
Predicted state is corresponding.The observation shape that position z is determined with the second data got at the t2 moment according to second sensor system
State is corresponding.When position z is in μmDuring place, mahalanobis distance is 0, and mahalanobis distance is as z is relative to μmOffset increase and
Increase.
Also, the mahalanobis distance and covariance matrix between predicted state and observer state can be with second sensor systems
The second sensing data got of uniting is available or updates and real-time update.For example, at the t3 moment, mahalanobis distance be another first
Observer state (at position z ' places) and the average value mu of the distribution including multiple predicted state μ 'm' between measurement distance.It is above-mentioned
Distribution be according in t2<t<What the priori prediction state S ' determined in the t3 periods was determined.Or it is optional, above-mentioned distribution is
According in t1<t<What the priori prediction state determined in the t3 periods was determined.
In embodiments of the present invention, mahalanobis distance can be represented by formula (1)：
Wherein, zkIt is corresponding with the first observer state, CkIt is a constant,It is the average value of predicted state μ distribution,It is the covariance matrix related to distribution, k is that the second sensing data that second sensor system is got can use or update
One or more time points.
In embodiments of the present invention, alternatively, in S303, when the first deviation is less than or equal to the first predetermined threshold value,
Sensor system control determines to determine the state of movable equipment according to predicted state and the first observer state.Specifically, may be used
To carry out predicted state and the first observer state to merge the state for determining movable equipment, the method merged can be card
Kalman Filtering (Kalman Filter), extended BHF approach, Unscented kalman filtering etc..
In embodiments of the present invention, alternatively, sensor system control can be abandoned is obtained by second sensor system
The second sensing data arrived.For example, when the first deviation is more than the first predetermined threshold value, sensor system control can not basis
First observer state determines the state of movable equipment.In this case, sensor system control can be only according to first
The predicted state that the first sensing data that sensing system is obtained is determined determines the state of movable equipment.For example, can select
One predicted state is defined as the state of movable equipment.
In embodiments of the present invention, alternatively, when the first deviation (for example, mahalanobis distance, Euclidean distance) is pre- more than first
If during threshold value, it is believed that second sensor system breaks down.For example, the sensor in second sensor system is relative to first
Beginning calibrating position drifts about.For example, the first predetermined threshold value could be arranged to deviate 1,2,3 of average value of distribution or
The numerical value of 4 standard deviations, or the first predetermined threshold value can be one relative to all the sensors in second sensor system
A value being all suitable for, or the first predetermined threshold value can be a fixed value or can be a variable value.
It should be noted that the above-mentioned description to the first predetermined threshold value go in the embodiment of the present invention other are pre-
If threshold value.
Below will be using first sensor system as IMU, second sensor system is exemplified by GPS sensor, description is according to this
The method that invention determines the state of movable equipment.GPS sensor can safeguard the state [p of itselfx,py,vx,vy]T.By
It is inaccurate in the z directions of GPS sensor, the amount in z directions is not estimated here.In order that with IMU data and GPS numbers
According to being merged, it is input to the acceleration information in IMU as a variable in sensor system control, GPS is sensed
The position data and speed data that device is obtained are used as observed quantity.Obtained continuous system equation and observational equation is expressed as
Formula (2) and (3)：
Z=Cx+ δ (3)
In formula (2) and (3), state vector x=[px,py,vx,vy]T, [px,py]TIt is the horizontal position of movable equipment
Put, [vx,vy]TIt is the horizontal velocity of movable equipment, dominant vector
G is acceleration of gravity,The rotation from IMU reference frame to world coordinate system is represented,
It can be obtained using compass from IMU.The acceleration information of horizontal direction is used, the acceleration without considering vertical direction
Spend information.GPS observation vector x=[px_gps,py_gps,vx_gps,vy_gps]T, sytem matrix A, input matrix B and observing matrix C
It is expressed as：
Determine field continuous system criterion to above-mentioned matrix A, B and C progress discretization according to linear, because discretization is mainly pair
For the state equation for describing system dynamic characteristic, therefore observational equation is to keep not after static algebraic equation, its discretization
Become, be after system equation discretization：
xk=Gxk-1+Huk；
Wherein, T is the sampling time, for example, the sample frequency of the accelerometer in IMU is 1000Hz, T=here
1000s.It can be seen from description above, IMU sample frequency is generally higher than the sample frequency of GPS sensor.For example, IMU
Sample frequency can 1 order of magnitude higher than the use frequency of GPS sensor, 2 orders of magnitude or more an order of magnitude.In GPS
Before data are available, using the state and covariance of IMU data prediction movable equipments, until gps data arrives and quality can
By when, just update primary system state.
It should be noted that when merging IMU data and gps data using Kalman filtering mode, gps data is worked as every time
During arrival, it is required for verifying gps data using the above method of calibration.
Below will be using first sensor system as IMU, second sensor system is included exemplified by multiple vision sensors, description
According to present invention determine that the method for the state of movable equipment.Multiple vision sensors include the difference installed in movable equipment
The binocular camera of (forward and backward, upper and lower and both sides) at position.IMU sample frequency is higher than the sample frequency of vision sensor.
Before the data of vision sensor are available, using the state of IMU data prediction movable equipments, it is expressed as：
System mode x=[px,py,pz,vx,vy,vz]T；
Observation of the output of vision sensor both for initial key frame：
Z1=[I3×3 03×3][P V]T；
Z2=[I3×3 03×3][P V]T；
Z3=[I3×3 03×3][P V]T。
If movable equipment includes N roads binocular camera, it is necessary to carry out n times renewal to system.Only wait until visual sensing
When the data arrival of device and reliable in quality, the state of movable equipment is just updated according to the observed result of vision sensor.
Explanation is needed also exist for, when merging IMU data and vision sensor data using Kalman filtering mode, often
It is secondary when vision sensor data are reached, be required for using the above method of calibration to vision sensor data carry out school
Test.
In embodiments of the present invention, alternatively, as shown in figure 5, methods described also includes：
S304, obtains the 3rd sensing data that 3rd sensor system is got.
Alternatively, in S304, the sample frequency of 3rd sensor system is less than the sample frequency of first sensor system,
For example, the sample frequency of 3rd sensor system 1 order of magnitude lower than the sample frequency of first sensor system, 2 orders of magnitude
Or more the order of magnitude.The sample frequency of 3rd sensor system and the sample frequency of second sensor system can be with identical or not
Together.
In embodiments of the present invention, alternatively, first sensor system includes IMU, and second sensor system is passed including GPS
Sensor, 3rd sensor system includes one or more vision sensors.
Specifically, sensor system control obtains the first sensing data that first sensor system is got.Exist afterwards
When the sensing data of second sensor system and 3rd sensor system is unavailable or does not update, sensor system control according to
First sensing data determines the predicted state of movable equipment.Sensor system control is got in second sensor system
When second sensing data is available or updates, second sensing data is obtained, and the first observation is determined according to the second sensing data
State.Equally when the 3rd sensing data that 3rd sensor system is got can use or update, the 3rd sensing number is obtained
According to, and the 3rd observer state is determined according to the 3rd sensing data.
In embodiments of the present invention, alternatively, the first observer state is the observer state determined according to gps data, second
Observer state is the observer state determined according to vision data.When it is determined that the first deviation is less than or equal to the first predetermined threshold value,
Effectively, and gps data can be for verifying to vision data for gps data.For example, obtaining present frame respectively to key frame
Vision calculate displacement and GPS and calculate displacement, afterwards, calculate the second deviation of the two displacements (for example, Euclidean distance or geneva
Distance), if Euclidean distance or mahalanobis distance are less than or equal to the second predetermined threshold value, then it is assumed that vision data is effective.On the contrary,
If Euclidean distance or mahalanobis distance are more than the second predetermined threshold value, it is believed that vision data is invalid.
Afterwards, visual sensor system controller determines whether to update removable according to the second observer state according to the second deviation
The state of dynamic equipment.Visual sensor system controller according to the result for being compared the second deviation and the second predetermined threshold value,
Determine whether the state according to the second observer state renewal movable equipment.If the second deviation is less than or equal to the second default threshold
Value, sensor system control determines the state of movable equipment according to predicted state and the second observer state.
In embodiments of the present invention, alternatively, sensor system control can be abandoned is obtained by 3rd sensor system
The 3rd sensing data arrived.For example, when the second deviation is more than the second predetermined threshold value, sensor system control can not basis
Second observer state determines the state of movable equipment.In this case, preset if the first deviation is less than or equal to first
Threshold value, sensor system control can determine the state of movable equipment according to predicted state and the first observer state.
Fig. 6 shows the schematic flow of the method for the state of determination movable equipment according to another embodiment of the present invention
Figure.As shown in fig. 6, method 600 includes：
S601, obtains the first sensing data that first sensor system is obtained；
S602, predicted state is determined according to the first sensing data；
S603, obtains the second sensing data that second sensor system is obtained；
S604, the first observer state is determined according to the second sensing data；
S605, obtains the 3rd sensing data that 3rd sensor system is obtained；
S606, the second observer state is determined according to the 3rd sensing data；
S607, determines the first deviation D between predicted state and the first predicted state12；
S608, determines the second deviation D between predicted state and the second predicted state13；
S609, determines D12Whether the first predetermined threshold value T is less than or equal to12；
S610, determines D13Whether the second predetermined threshold value T is less than or equal to13；
S611, when it is determined that D12More than T12, and D13More than T13When, it is determined that only determining movable equipment according to predicted state
State；
S612, when it is determined that D12Less than or equal to T12, and D13More than T13When, it is determined that according to predicted state and the first observation shape
State determines the state of the movable equipment；
S613, when it is determined that D12More than T12, and D13Less than or equal to T13When, it is determined that according to predicted state and the second observation shape
State updates the state of the movable equipment；
S614, when it is determined that D12Less than or equal to T12, and D13Less than or equal to T13When, determine the first observer state and second
The 3rd deviation D between observer state23；
S615, determines D23Whether the 3rd predetermined threshold value T is less than or equal to23；
S616, when it is determined that D23Less than or equal to T23When, it is determined that according to predicted state and the first observer state and/or second
Observer state determines the state of the movable equipment；
S617, when it is determined that D23Less than or equal to T23When, it is determined that determining institute according to the first observer state and the second observer state
State the state of movable equipment；
It is understood that S616 and S617 are alternatives arranged side by side, sensor system control can select to perform
S616, or selection perform S617.
S618, when it is determined that D23More than T23, and D12Less than D23When, it is determined that being determined according to predicted state and the first observer state
The state of the movable equipment；
S619, when it is determined that D23More than T23, and D12More than D23When, it is determined that being determined according to predicted state and the second observer state
The state of the movable equipment.
It is understood that S618 and S619 are alternatives arranged side by side, sensor system control can select to perform
S618, or selection perform S619.
Specifically, sensor system control obtains the first sensing data obtained by first sensor system, afterwards
The sensing data got in second sensor system and 3rd sensor system is unavailable or in period for not updating, passes
Sensor system controller determines the predicted state of movable equipment according to the first sensing data.When second sensor system is got
The second sensing data it is available or when updating, sensor system control obtains second sensing data, afterwards sensor system
System controller determines the first observer state of movable equipment according to the second sensing data.It is same when 3rd sensor system is obtained
When the 3rd sensing data got is available or updates, sensor system control obtains the 3rd sensing data, afterwards basis
3rd sensing data determines the second observer state of movable equipment.
It is understood that when the second sensing data that second sensor system is got is available or updates, sensor
System controller can all be verified to the second sensing data, specifically, and sensor system control calculates predicted state and the
The first deviation D between one observer state12.Similarly, the 3rd sensing data got when 3rd sensor system it is available or
During renewal, sensor system control can all be verified to the 3rd sensing data, specifically, and sensor system control is calculated
The second deviation D between predicted state and the second observer state13。
Sensor system control is by D afterwards12With the first predetermined threshold value T12Contrasted, and by D13With the second default threshold
Value T13Contrasted.When sensor system control determines D12More than T12, and D13More than T13When, only updated according to predicted state
The state of movable equipment.For example, when first sensor system is IMU, sensor system control is only true according to IMU data
Determine the state of movable equipment.Also, further, sensor system control can abandon the second sensing data and the 3rd biography
Feel data.
Or, when sensor system control determines D12Less than or equal to T12, and D13More than T13When, sensing system control
Device processed determines the state of movable equipment according to predicted state and the first observer state.For example, when first sensor system is
IMU, when second sensor system is GPS, sensor system control determines movable equipment according to IMU data and gps data
State.Also, further, sensor system control can abandon the 3rd sensing data, for example, when 3rd sensor system
When uniting as vision sensor, sensing system control system discards the vision data got.
Or, when sensor system control determines D12More than T12, and D13Less than or equal to T13When, sensing system control
Device processed determines the state of movable equipment according to predicted state and the second observer state.For example, when first sensor system is
IMU, when 3rd sensor system is vision sensor, sensor system control is determined according to IMU data and vision data can
The state of mobile device.Also, further, sensor system control can abandon the second sensing data, for example, when second
When sensing system is GPS, sensor system control discards the gps data got.
Seen from the above description, the first sensing data (for example, IMU data) can be to the second sensing data (for example, GPS
Data) validity or accuracy verified, likewise, the first sensing data can be to the 3rd sensing data (for example, vision
Data) validity or accuracy verified.
In embodiments of the present invention, alternatively, when sensor system control determines D12Less than or equal to T12, and D13It is small
In or equal to T13When, the second sensing data can mutually be verified with the 3rd sensing data.Specifically, sensing system is controlled
Device calculates the 3rd deviation D between the first predicted state and the second predicted state23, afterwards by D23With the 3rd predetermined threshold value T23Enter
Row compares.When sensor system control determines D23Less than or equal to T23When, sensor system control according to predicted state and
First observer state and/or the second observer state determine the state of movable equipment, or sensor system control according to
One observer state and the second observer state determine the state of movable equipment.For example, when first sensor system is IMU, second
Sensing system is GPS, and when 3rd sensor system is vision sensor, sensor system control is according to IMU data and GPS
Data and/or vision data determine the state of movable equipment, or determine movable equipment according to gps data and vision data
State.
Alternatively, when sensor system control determines D23More than T23When, sensor system control is according to D12And D13's
Size determines the data of the state for determining movable equipment.Specifically, when sensor system control determines D12Less than D13
When, sensor system control determines the state of movable equipment according to predicted state and the first observer state.When sensor system
System controller determines D12More than D13When, sensor system control determines removable according to predicted state and the second observer state
The state of equipment.
It should be noted that sensor system control is according to predicted state, the first observer state, and/or the second observation
State determines the state of movable equipment, can be by the way that predicted state, the first observer state, and/or the second observer state are entered
Row fusion treatment, obtains the state of movable equipment.The method for carrying out fusion treatment use can be Kalman filtering (Kalman
Filter), extended BHF approach, Unscented kalman filtering etc..
In embodiments of the present invention, alternatively, predetermined threshold value above is (for example, the first predetermined threshold value, the second default threshold
Value and the 3rd predetermined threshold value) can be obtained according to experiment, or predetermined threshold value can be a span or constant.Tool
For body, predetermined threshold value can be at least one determination in following condition：(1) environment of movable equipment operation；
(2) one or more kinetic characteristics of movable equipment；(3) location information of movable equipment；(4) height of movable equipment
Degree.For example, predetermined threshold value can be changed with the change of the environment residing for movable equipment, or predetermined threshold value can be with
Change with the change of one or more kinetic characteristics of movable equipment, or predetermined threshold value can be with movable equipment
The change of location information and change, or predetermined threshold value can change with the change of the height of movable equipment.
Also, the first predicted state and the second predicted state depend on the environmental form that movable equipment is run.Different
There can be at least one following distinguishing characteristics between environmental form：(1) weather condition；(2) density of object and distribution；(3)
The vision or physical characteristic of object.Further, the first predicted state and the second predicted state additionally depend on second sensor system
One or more service conditions of system and 3rd sensor system.Above-mentioned service condition include signal intensity, sensor type,
Dysfunction, power grade, sensing accuracy and/or calibration grade.Wherein, signal intensity depends on one or more sensors
The quantity of signal amplitude, the signal received.For example, a sensing system includes GPS sensor, the signal of GPS sensor
Intensity depends on the quantity or the amplitude of the gps signal received for the transducing signal that GPS sensor is received.Under normal circumstances,
Indoors when environment, atrocious weather, gps receiver failure, the intensity of gps signal is very weak.In outdoor environment, sunny
Weather, very high height, when there is no failure, the intensity of gps signal is very strong.In some cases, movable equipment is very
With relatively low altitude between more high building, now high building can stop or weaken satellite-signal, now can
Cause the decrease of gps signal or even disappear.
In embodiments of the present invention, it is alternatively possible to be determined according to the magnitude relationship of the first deviation and the first predetermined threshold value
Availability and service condition of the second sensor system under a specific environment.For example, when second sensor system is in the first kind
When running and/or run in the desired manner in the environment of type, the first deviation is less than or equal to the first predetermined threshold value, sensor system
System controller determines that second sensor system is available under first kind environment.Otherwise, when second sensor system is
When being run in the environment of one type, the first deviation is more than the first predetermined threshold value, and sensor system control determines second sensor
System is unavailable in the environment of the first kind or breaks down.
In embodiments of the present invention, alternatively, sensor system control can be controlled removable by redundancy decision mode
Multiple vision sensors in dynamic equipment.Sensor system control can detect the running status of each vision sensor.Example
Such as, sensor system control can detect whether First look sensor breaks down or produce inaccurate visual sensing
Data, when it is determined that First look sensor breaks down or produces inaccurate visual sensing data, are passed from First look
Sensor is switched to other vision sensors to ensure to take over seamlessly the acquisition with data.
Alternatively, as an example, multiple vision sensors can be mounted in the different piece of movable equipment
Multiple imaging devices.Multiple imaging devices include binocular camera and/or monocular camera.At least one imaging device can work
Under many mesh patterns, and at least one imaging device can be operated under monocular pattern.Or at least one imaging device was both
Use under monocular pattern, which can be operated in, can be operated under many mesh patterns.Also, many mesh patterns include binocular pattern.
In embodiments of the present invention, alternatively, multiple imaging devices are connected on movable equipment, and including (1) extremely
Lack first imaging device being operated under many mesh patterns and (2) at least one second imaging being operated under monocular pattern is set
It is standby.For example, multiple imaging devices include multiple first imaging devices, the multiple imaging device is detachably arranged in removable
On the not ipsilateral of equipment, the first imaging device includes binocular camera.For example, the first binocular camera is arranged on movable equipment
Above, the second binocular camera is arranged on behind movable equipment, and the 3rd binocular camera is arranged on the left surface of movable equipment,
4th binocular camera is arranged on the right flank of movable equipment, and the 5th binocular camera is arranged on above movable equipment, and the 6th
Binocular camera is arranged on below movable equipment.Or, one or more cameras may be mounted at the same of movable equipment
On side.
The second above-mentioned imaging device can be removably attachable on movable equipment by carrier.And the second imaging
Equipment rotates relative to movable equipment along at least one direction.
Fig. 7 shows the method for being used in movable equipment select imaging device according to embodiments of the present invention.Such as Fig. 7 institutes
Show, method 700 includes：
S701, determines the first relative position of each imaging device in multiple imaging devices relative to other imaging devices,
The second relative position with each imaging device relative to the movable equipment.
In embodiments of the present invention, sensor system control can obtain each imaging device and be set relative to other imagings
The locus of standby and movable equipment.If multiple imaging devices that movable equipment includes, which have, prolongs multiple directions extension
Optical axis, the position relationship of each imaging device and the IMU on movable equipment can be determined out.Because IMU is generally right
Translational motion is insensitive, thus if the size and location of imaging device is known, the position between IMU and each imaging device
The relation of putting directly can be determined out.Each the angular relationship between imaging device and IMU can pass through hand and eye calibrating method
Calculate.
Fig. 8 is the schematic diagram of hand and eye calibrating method according to embodiments of the present invention.Because IMU data need and visual sensing
Data are merged, it is therefore desirable to know position and angular relationship of each imaging device (for example, camera) with IMU.As in Fig. 8
It is shown, the rotation A of the camera at two positions can be calculated according to image information, and read from IMU data removable
The rotation B of dynamic equipment, it is possible thereby to calibrate IMU to the rotation R of imaging device by calculating, can pass through sensing system control
Device processed determines the position between multiple imaging devices and IMU and angular relationship simultaneously.
Wherein, AXBTXT=I → AX=XB → AX-XB=0；By minimizing min | | AX-XB | | it is suitable to find
The rotation of imaging device on X, i.e. all directions to IMU.
In embodiments of the present invention, camera can be calibrated with respect to IMU.It can specifically be pushed away by the time in camera
Multiple image is obtained during shifting and estimates that the change in location of itself realizes the calibration of camera by camera.Method for self-calibrating is similar to
Different cameras are calibrated by considering two different camera α and β in the two field pictures that different time point i and i' are obtained
Method.Likewise, method for self-calibrating goes for IMU calibration.It is assumed that A and B represent camera and IMU own coordinate respectively
Change.Subscript i is represented in moment i=1, during 2 ..., n, AiAnd BiCoordinate system mapping.Moment 2 reflects relative to the moment 1
Firing table is shown as：WithX represents the mapping between camera and IMU.It can be seen from hand and eye calibrating algorithm
AX=XB, wherein, A, B and X are the normalized mappings with following form：
Further, RARX=RXRB(RA-I)tX=RXtB-tA.According to the characteristic of above-mentioned equation and spin matrix, there is many
The mode of kind can solve RXAnd tX.In order to ensure the uniqueness of solution, n value is more than or equal to 3.
S702, according to selection information, the selection target imaging device from the multiple imaging device, wherein, the selection
Information includes at least one of following message：Each imaging device relative to the imaging device object within the vision or
The distance on ground, match point at least frame stereo-picture got by least one described first imaging device are regarded
The working environment of difference and the multiple imaging device, wherein, the distance is according to first relative position and described second
What relative position was determined.
In embodiments of the present invention, sensor system control can select visual imaging pattern for each imaging device.
Sensor system control can determine at least one of following parameters：(a) at least one imaging device is relative to the imaging
The object within the vision or the distance on ground of equipment；(b) at least frame stereogram got by the first imaging device
The parallax of match point as in；(c) working environment of multiple imaging devices.Sensor system control is according in above-mentioned parameter
At least one determines that at least one first imaging device obtains image under many mesh patterns, or determine at least one second
Imaging device obtains image under monocular pattern.
In embodiments of the present invention, alternatively, sensor system control is determined by one or more range sensors
Within the vision object or the distance on ground of at least one imaging device relative to the imaging device.The visual field of imaging device
In the range of object can be a target object in environment, movable equipment can get target object image or
The target object is tracked.Here ground surface or the plane of reference that ground can refer to, or an object table
Face.Involved range sensor can be ultrasonic sensor, time-of-flight camera etc..And barometric surveying can be passed through
The height of movable equipment.The 3-dimensional that can also be determined by the stereo-picture got by least one first imaging device is deep
Degree information, positional information and/or movable information determine at least one imaging device relative to the imaging device within sweep of the eye
Object or ground distance.Or can also be got according to IMU positional information and/or movable information, by second imaging
The image and the second imaging device that equipment is got determine at least one imaging device relative to the locus of movable equipment
Relative to the object within the vision or the distance on ground of the imaging device.
S730, image is obtained using the target imaging equipment.
Specifically, Fig. 9 shows the method according to embodiments of the present invention according to predetermined threshold value selection vision sensor
Schematic diagram.As shown in figure 9, multiple imaging devices 130 are installed on movable equipment 100.Imaging device 130 includes at least one
The first imaging device 132 being operated under many mesh patterns is operated in the second imaging device 134 under monocular pattern with least one.
For example, showing two first imaging devices 132-1 and 132-2 in Fig. 9, the first imaging device 132-1 may be mounted at removable
Before equipment, the first imaging device 132-2 may be mounted at behind movable equipment.Second imaging device 134 is by carrying
Body 104 is removably connected with movable equipment.Carrier 104 enable to the second imaging device relative to movable equipment along
At least one axle is rotated.
As shown in Figure 9, movable equipment includes sensor system control 140, and sensor controller 140 can
Each imaging device is obtained relative to other imaging devices and IMU110 relative tertiary location by hand and eye calibrating method.
Movable equipment 100 is under a running environment, and target object 102 is an object under running environment.Mesh
Mark object 102 can be the object or an object that can be moved of static one motion of object.Pass
Stereo-picture that sensor system controller 140 can be got by range sensor, by the first imaging device etc. determines removable
The distance between dynamic equipment and target object 102 d.For example, sensor system control 140 determines movable equipment and object
Initial distance between body 102 is d1.
And then, sensor system control 140 selects suitable by comparing d and pre-determined distance threshold value D size in real time
Visual sensing pattern.When d is less than or equal to D, sensor system control 140 selects the first one-tenth be operated under many mesh patterns
As equipment 132-1 is used to obtain view data.Corresponding, when d is more than D, selection is operated in the second one-tenth under monocular pattern
As equipment 134 is used to obtain view data.
In embodiments of the present invention, above-mentioned pre-determined distance threshold value can be determined according to experiment.Pre-determined distance threshold value
It can be an a distance range either constant.Or pre-determined distance threshold value can be with the operation ring of movable equipment
Border, the change of the position of movable equipment or height and change.
In embodiments of the present invention, alternatively, sensor system control 140 can determine by the first imaging device (example
Such as, 132-1) match point in one or more stereo-pictures for getting parallax.Sensor system control 140 can root
According to the parallax of match point suitable visual sensing pattern is selected with the magnitude relationship of default parallax threshold value.Default parallax threshold value dp
Can be：Dp=c*f/H, wherein, c is baseline constant, and f is the focal length of the first imaging device, and H is preset height threshold value.
Alternatively, as an example, when parallax is more than default parallax threshold value, sensor system control 140 is selected
The first imaging device 132-1 being operated under many mesh patterns is used to obtain image.Otherwise, the be operated under monocular pattern is chosen
Two imaging devices 134 are used to obtain image.Default parallax threshold value can be determined by experiment, or default parallax threshold value can
To be a scope.
In embodiments of the present invention, alternatively, sensor system control 140 as shown in Figure 10 can by comparing
The height h and preset height threshold value H of mobile device magnitude relationship select suitable visual sensing pattern.As shown in Figure 10, h is worked as
(for example, h1<When H) being less than or equal to H, sensor system control 140 selects the be operated under many mesh patterns first imaging to set
Standby 132-1 is used to obtain image, otherwise (for example, h2>H) the second imaging device 134 that selection is operated under monocular pattern is used for
Obtain image.
In the above-described embodiments, alternatively, preset height threshold value H can be the experiment according to the movable equipment got
The value that data are determined.For example, experimental data indicates that the quality of binocular image data is low when the height of movable equipment is higher than 8m
In the quality that can be received, and experimental data instruction is when movable equipment is higher than 8m is less than, the quality of binocular image data
It is acceptable, then preset height threshold value can be set to 8m.
Preset height threshold value H above can be scope either constant, and preset height threshold value can be with
The change of the running environment of movable equipment and change, or preset height threshold value can be with the day gas bar in running environment
Part changes and changed, and either preset height threshold value can change or default height with the change of the height of movable equipment
Degree threshold value can change with the density of the object in environment and the change of distribution, or preset height threshold value can be with environment
In vision or physical characteristic change and change.
In embodiments of the present invention, alternatively, sensor system control 140 can be less in binocular ranging number, average
Parallax is smaller, movable equipment height is more than preset height threshold value, movable equipment and is more than relative to the distance of target object
Second is selected during preset height threshold value, and/or when movable equipment is more than pre-determined distance threshold value relative to the distance of target object
Imaging device is used to obtain view data.Under normal circumstances, it is double when scene lacks texture (for example, the water surface or pure color desktop)
Mesh coupling number can be fewer.Can optical flow match and zero-mean crosscorrelation detect by way of determine binocular ranging etc.
Level.
In embodiments of the present invention, alternatively, the image got according to the first imaging device and/or the second imaging device
Data can calculate one or more motion features of movable equipment.Specifically, it can be obtained according to the first imaging device
To the depth information of stereo-picture determine the kinetic characteristic of movable equipment.Or can be got according to the second imaging device
Continuous two images between conversion determine movable equipment kinetic characteristic.
In this embodiment of the present invention, it is alternatively possible to obtain the different piece in a scene using different cameras
View data.Can be according to for certain a part of relative signal quality and/or relative signal accuracy, it is determined that using which
Individual camera obtains the view data of the part.The quality and accuracy of view data depend on the proprietary spy of each vision sensor
Property, and can be because change, change of weather of scene etc. are varied from.For example, dual mode camera is in short distance internal ratio monocular phase
Machine has higher accuracy.Or, can be according to the selection of the applicable sensing scope of camera using which camera acquisition picture number
According to.
In embodiments of the present invention, alternatively, visual sensing is made up by way of Distance-sensing and visual sensing are combined
Deficiency, and then improve visual sensing reliability.For example, although camera can produce the colour with high-resolution
Image.But when using many mesh cameras, it is difficult to pass through the accurate depth data of image data acquisition.Also, vision sensor
The view data suited the requirements when light intensity is very strong or reflective or can not be obtained under adverse circumstances.Likewise, ultrasonic wave is passed
Sensor and other range sensors can not detect the object with small reflecting surface or have the object of absorption, can not divide
Discern the distance of multiple objects in complex scene.The Distance-sensing number that can be got due to vision sensor in range sensor
According to it is bad when get believable data, visual sensing data and Distance-sensing data can be combined.
Figure 11 shows binocular camera 900 according to embodiments of the present invention.As shown in figure 11, binocular camera 900 includes one
Individual left vision sensor 902 and a right vision sensor 904.The focal length of camera is f, and the size of optical sensor is l, two
The distance between vision sensor is b, the characteristic point of a pair of matchings on the image that left and right vision sensor is gotWithBetween 3-dimensional coordinate bePixel distanceWithAfter being multiplied with pixel size
Obtain space lengthWithIt therefore, it can according to formula：Determine 3-dimensional coordinateWith the distance between vision sensor and parameter D.According to the inner parameter matrix K and D of camera, it may be determined that go out a littlePrediction 3-dimensional coordinate
According to frame-frame matching and the Stereo matching mode of characteristic point, it may be determined that go out the 3-dimensional coordinate pair of each characteristic pointAnd can by analyze characteristic point motion determine camera speed.For example, the n that given time t is obtained
Individual coordinate pair c1,c2,…,cn, matrixOne, which can be expressed as, includes the matrix of three row vectorsCamera
Inner parameter matrix can be expressed as：
Accordingly, each characteristic point existsWhen predicted position motion or change can be obtained by solution formula (4).
Because the position motion of prediction is mainly carried out by the view data for getting vision sensor 902 and 904
Frame matching is determined, so the accuracy and precision of prediction can be influenceed by n value.
Figure 12 is the schematic diagram of the visual sensing scope of movable equipment according to embodiments of the present invention.Wherein, Figure 12 (a)
It is the top view seen downwardly over from movable equipment, Figure 12 (b) is from the side view in terms of the side of movable equipment, Figure 12
(c) it is a 3-dimensional view.Movable equipment in Figure 12 for example can be unmanned plane.
Multiple imaging devices can be installed on the not ipsilateral of movable equipment in Figure 12.The visual angle of each imaging device
For α, the maximum visual sensing scope of movable equipment can be according to the chi of the imaging sensor in visual angle α and each imaging device
It is very little to determine.Visual sensing scope can be expressed as circle 1060 and 1070 or ball 1080.It is understood that visual sensing scope
Any shape and/or size can be defined as, for example, visual sensing scope can be defined as shape (cube, the circle of rule
Post, circular cone) or irregular shape.
Alternatively, as an example, the visual field of adjacent imaging device may be overlapping, thus, it is possible to guarantee to obtain
Take image data point enough in scene.Or the visual field of adjacent imaging device can not be overlapping.And it is possible to according to acquisition
The image data point arrived sets up the environmental map with certain degree of accuracy.
Also, multiple imaging devices can obtain many mesh, binocular or the monocular image of the scene around movable equipment.It is many
Individual imaging device can obtain view data with same or different time interval.And can be according to binocular or many mesh images
Determine the 3-dimensional depth map of environment.Multiple imaging devices can provide the visual field of a n degree, for example, n can for 90 °, 100 °,
110°、120°、130°、140°、150°、160°、170°、180°、190°、200°、210°、220°、230°、240°、250°、
260 °, 270 °, 280 °, 290 °, 300 °, 310 °, 320 °, 330 °, 340 °, 350 ° or 360 °.When n is 360 °, it can realize
Loopful is around visual sensing.
Alternatively, as an example, visual sensing scope is defined as into one has the tool centered on movable equipment
There is the ball of pre-set radius.The scope of pre-set radius can be from several meters to hundreds of meters, or pre-set radius can be less than 1m or big
In 500m.The complexity for the environment that visual sensing scope can be run by movable equipment is influenceed, for example, working as movable equipment
, it is necessary to increase visual sensing scope or improve visual sensing when being run in an environment with multiple objects or barrier
Susceptibility.Conversely, can then lower the susceptibility of visual sensing scope or visual sensing.
In embodiments of the present invention, image sensing apparatus can obtain multiple images simultaneously, or obtain in certain sequence
Multiple images, or obtain multiple images at different time points.The multiple images got can be for setting up 3D scenes, 3D
Virtual scene, 3D maps or 3D models.Also, carried out by the stereo video data got to one or more imaging devices
Analysis can obtain environmental information.Environmental information includes environmental map, or environmental information includes topological map or measurement map.
Specifically, it can comprise the following steps to carry out analysis to stereo video data：(1) school is carried out to imaging device
It is accurate；(2) Stereo matching is carried out to picture frame；(3) depth map is calculated.Wherein, carrying out calibration to imaging device is included to imaging
The inner parameter and external parameter of equipment are calibrated.Carrying out Stereo matching to picture frame includes：(1) it is substantial or approximate real time
The each binocular image of extraction in each monocular image characteristic point；(2) motion feature of characteristic point is calculated；(3) based on spy
Motion feature a little is levied, the corresponding characteristic point extracted from picture frame is matched；(4) unmatched feature is excluded
Point.Calculating depth map includes：(1) characteristic point based on matching, calculates the disparity map based on pixel；(2) according to binocular camera
External parameter calculate depth map.
In embodiments of the present invention, alternatively, what sensor system control was installed for assessing on movable equipment is more
The availability of individual imaging device, and the imaging device according to the available Sexual behavior mode for the imaging device determined for visual sensing.
For example, the view data that sensor system control is got according to multiple imaging devices determines multiple first observer states, and
The data got according to IMU determine multiple predicted states.Afterwards, sensor system control according to the first observer state with it is pre-
The first deviation between survey state determines the availability of imaging device.
Figure 13 shows that the imaging for being used to determine to be used on movable equipment visual sensing according to embodiments of the present invention is set
The method of standby availability.As shown in figure 13, method 1300 includes：
S1301, the view data got according to multiple imaging devices for visual sensing determines that described may move sets
Standby multiple first observer states；
S1302, the sensing data got according to Inertial Measurement Unit IMU determines the multiple pre- of the movable equipment
Survey state；
S1303, according to the first deviation and the first predetermined threshold value between predicted state and the first observer state, it is determined that each
Availability for the imaging device of visual sensing.
Specifically, when corresponding first deviation of an imaging device is less than or equal to the first predetermined threshold value, the imaging is set
It is standby to can be used for visual sensing, conversely, the imaging device is not suitable for visual sensing.Sensor system control can be determined
First imaging device set and the second imaging device set, wherein, the imaging device in the first imaging device set can be used for
Visual sensing, the imaging device portion in the second imaging device set is suitable for visual sensing.Further, sensing system control
Corresponding first observer state of imaging device of the device processed in the first imaging device set and corresponding predicted state, it is determined that
The state of movable equipment.Further, the imaging device in second sensor set may be because of corresponding first deviation
Change and be made available by.
When table 1 shows that movable equipment is moved in different environment, sensor system control is according to the first deviation d
(1) availability of the imaging device at the position of front, rear, left and right 4 of the movable equipment determined.T1 is first default in table 1
Threshold value.
Table 1
It is understood that the different availabilities of imaging device are that environment on the different directions of movable equipment is poor
Caused by different and/or object type.For example, weather condition, illumination condition, object on movable equipment different directions is close
Degree, surface texture etc. are likely to difference.For example, when the sun sets, if movable equipment faces west, it may move
The intensity of illumination of equipment forward portion is higher than the intensity of illumination of rear portion.
In embodiments of the present invention, alternatively, the gps data that sensor system control is got according to GPS sensor,
Determine the second observer state of movable equipment.Sensor system control according to the second observer state and the first observer state it
Between the second deviation and the second predetermined threshold value between magnitude relationship, determine the availability of imaging device.Likewise, the second sensing
Imaging device in device set may should be the change of corresponding second deviation and be made available by.
Alternatively, the change of above-mentioned the first deviation and the second deviation is by the first observer state and the second observer state
Caused by changing.The change of first observer state and the second observer state can be due to the residing environment of movable equipment
Caused by changing.
Figure 14 is the redundancy decision-making for being used to select sensor and/or data at different conditions according to embodiments of the present invention
The indicative flowchart of method.Movable equipment includes multiple imaging devices, IMU and GPS sensor.Sensor system control
It can be communicated with imaging device, IMU and GPS sensor.As shown in figure 14, method 1400 includes：
S1401, sensor system control obtains the IMU data got by IMU；
S1402, sensor system control determines predicted state according to IMU data；
S1403, sensor system control determines whether the gps data that GPS sensor is got can use or update；
S1404, sensor system control determines the first prediction shape when gps data can use or update according to gps data
State；
S1405, sensor system control obtains the first imaging device to N imaging devices and gets view data, and N is
Integer more than 2；
S1406, sensor system control determines whether the imaging device meets following 3 bars for each imaging device
Part：(1) movable equipment is more than or equal to preset height threshold value relative to the height of reference planes；(2) parallax of match point is small
In or equal to default parallax threshold value；(3) the distance between movable equipment and target object are more than pre-determined distance threshold value；
S1407, when sensor system control determines that imaging device meets 3 conditions in S1406, control imaging is set
It is standby to be operated under monocular pattern；
S1408, when sensor system control determines that imaging device is unsatisfactory for one or more of S1406 conditions,
Control imaging device is operated under many mesh patterns；
S1409, the view data that sensor system control is got according to each imaging device determines the second observation shape
State；
S1410, sensor system control determines the first deviation D between predicted state and the first observer state12,
S1411, sensor system control determines the second deviation D between predicted state and the second observer state13；
S1412, sensor system control determines D12Whether the first predetermined threshold value T is less than or equal to12；
S1413, sensor system control determines D13Whether the second predetermined threshold value T is less than or equal to13；
S1414, when sensor system control determines D12More than T12, and D13More than T13When, it is determined that only according to IMU data
Update the state of movable equipment；
S1415, when sensor system control determines D12Less than or equal to T12, and D13More than T13When, by IMU data with
Gps data is merged, and determines the state of movable equipment；
S1416, when sensor system control determines D12More than T12, and D13Less than or equal to T13When, it is determined that by IMU numbers
Merged according to view data, determine the state of movable equipment；
S1417, when sensor system control determines D12Less than or equal to T12, and D13Less than or equal to T13When, determine
The 3rd deviation D between one observer state and the second observer state23；
S1418, when sensor system control determines D23Whether the 3rd predetermined threshold value T is less than or equal to23；
S1419, when sensor system control determines D23Less than T23When, by by IMU data, gps data and/or figure
As data are merged, the state of movable equipment is determined；
S1420, when sensor device controller determines D23Less than or equal to T23When, by by gps data and view data
Merged, determine the state of movable equipment；
It is understood that S1419 and S1420 are alternatives arranged side by side, sensor system control can select to hold
Row S1419, or selection perform S1420.
S1421, when sensor system control determines D23More than T23, and D12Less than D23When, by by IMU data and GPS
Data are merged, and determine the state of movable equipment；
S1422, when sensor system control determines D23More than T23, and D12More than D23When, by by IMU data with figure
As data are merged, the state of movable equipment is determined.
It is understood that S1421 and S1422 are alternatives arranged side by side, sensor system control can select to hold
Row S1421, or selection perform S1422.
It should be noted that (the first predetermined threshold value, the second predetermined threshold value and the 3rd are default for the predetermined threshold value in method 1400
Threshold value) determination mode and characteristic it is identical with method 600, for avoid repeat, will not be repeated here.
Similarly, what sensor system control was merged to different data implements and the side of fusion above
Formula is identical, to avoid repeating, will not be repeated here.
Thus, by the way that different data are merged, movable equipment can accurately be navigated so that removable
Dynamic equipment being capable of avoiding obstacles, the security of raising movable equipment and flexibility.
Method according to embodiments of the present invention is described in detail above in association with Fig. 1 to Figure 14, below in conjunction with Figure 15 to figure
17 are described in detail the system for performing the method in the embodiment of the present invention.
Figure 15 is the schematic block diagram for being used to determine the system of the state of movable equipment according to embodiments of the present invention, such as
Shown in Figure 15, system 1500 includes：
Acquisition module 1501, for obtaining the sensing number that the multiple sensors associated with the movable equipment are obtained
According to, wherein, the multiple sensor include first sensor system and second sensor system, the first sensor system with
The data sampling frequency of the second sensor system is different；
Determining module 1502, the second sensing data for being got in the second sensor system is unavailable or not more
In the new period, the first sensing data got according to the first sensor system determines the pre- of the movable equipment
Survey state；
The determining module 1502, is additionally operable to the second sensing number got when the determination second sensor system
During according to available or renewal, according to second sensing data, the first observer state of the movable equipment is determined；
The determining module 1502, is additionally operable to inclined according to first between first observer state and the predicted state
Difference, it is determined whether the state of the movable equipment is updated according to first observer state, wherein, first deviation is used for
Indicate whether second sensing data can use.
Therefore, the system for being used to determine the state of movable equipment according to embodiments of the present invention, it is determined that removable set
During standby state, the first sensing data got according to first sensor system determines the prediction shape of movable equipment
State, and the second sensing data got according to second sensor system determines the observer state of movable equipment, passes through prediction
Deviation between state and observer state, it is determined whether the state of movable equipment is updated according to observer state.Thereby, it is possible to logical
Cross and each sensing system is verified, suitable sensing system is selected in multichannel sensor system, and according to selection
The sensing data that the suitable sensing system gone out is got updates the state of movable equipment, it is possible to increase movable equipment
Security performance.
In embodiments of the present invention, alternatively, first sensing data includes first group of position data and first group of fortune
Dynamic data, second sensing data includes second group of position data and second group of exercise data.
In embodiments of the present invention, alternatively, the first sensor system includes Inertial Measurement Unit IMU.
In embodiments of the present invention, alternatively, the second sensor system includes global positioning system GPS receiver.
In embodiments of the present invention, alternatively, the second sensor system includes one or more vision sensors.
In embodiments of the present invention, alternatively, the acquisition module 1501 is additionally operable to：According to the movable equipment extremely
A few priori prediction state, determines first sensing data.
In embodiments of the present invention, alternatively, the determining module 1502 specifically for：According to first deviation and
One predetermined threshold value, it is determined whether the state of the movable equipment is updated using first observer state.
In embodiments of the present invention, alternatively, the determining module 1502 specifically for：When it is determined that first deviation
During less than or equal to first predetermined threshold value, it is determined that updating the shape of the movable equipment according to first observer state
State；When it is determined that first deviation is more than first predetermined threshold value, it is determined that updating institute without using first observer state
State the state of movable equipment.
In embodiments of the present invention, alternatively, first deviation is first observer state and the predicted state
Between mahalanobis distance or Euclidean distance.
In embodiments of the present invention, alternatively, when it is determined that updating the movable equipment according to first observer state
State when, the determining module 1502 is additionally operable to：Can according to updating the predicted state and first observer state
The state of mobile device.
In embodiments of the present invention, alternatively, the determining module 1502 is additionally operable to：When it is determined that being seen without using described first
When survey state updates the state of the movable equipment, the predicted state is defined as to the state of the movable equipment.
In embodiments of the present invention, alternatively, the determining module 1502 is additionally operable to：At least one in following message
Plant and determine first predetermined threshold value：The running environment information of the movable equipment, the motion feature of the movable equipment
The elevation information of information, the positional information of the movable equipment and the movable equipment.
In embodiments of the present invention, alternatively, the movable equipment also includes 3rd sensor system, and the described 3rd passes
The data sampling frequency of sensor system is different from the first sensor system and the second sensor system, the determination mould
Block 1502 is additionally operable to：When it is determined that the 3rd sensing data that the 3rd sensor system is got can use or update, according to institute
The 3rd sensing data is stated, the second observer state of the movable equipment is determined；When it is determined that second sensing data it is available or
During renewal, according to the second deviation between first observer state and second observer state, it is determined whether according to described
Second observer state updates the state of the movable equipment, wherein, second deviation is used to indicate the 3rd sensing number
According to whether can use.
In embodiments of the present invention, alternatively, the determining module 1502 specifically for：According to second deviation and
Two predetermined threshold values, it is determined whether the state of the movable equipment is updated using second observer state.
In embodiments of the present invention, alternatively, the determining module 1502 specifically for：When it is determined that second deviation
During less than or equal to second predetermined threshold value, it is determined that updating the shape of the movable equipment according to second observer state
State；When it is determined that second deviation is more than second predetermined threshold value, it is determined that being updated without using second observer state
The state of the movable equipment.
In embodiments of the present invention, alternatively, the first sensor system includes IMU, the second sensor system
Including gps receiver, the 3rd sensor system includes one or more vision sensors.
In embodiments of the present invention, alternatively, when it is determined that updating the movable equipment according to second observer state
State when, the determining module 1502 is additionally operable to：Can according to updating the predicted state and second observer state
The state of mobile device；Or, according to updating the predicted state, first observer state and second observer state
The state of movable equipment.
In embodiments of the present invention, alternatively, set when it is determined that not updating described may move according to second observer state
During standby state, the determining module 1502 is additionally operable to：According to updating the predicted state and first observer state
The state of movable equipment.
In embodiments of the present invention, alternatively, the determining module 1502 is additionally operable to：At least one in following message
Plant and determine second predetermined threshold value：The running environment information of the movable equipment, the motion feature of the movable equipment
The elevation information of information, the positional information of the movable equipment and the movable equipment.
Each unit and other above-mentioned operation and/or functions in system 1500 according to embodiments of the present invention is respectively for reality
The corresponding flow of existing method 300, for sake of simplicity, will not be repeated here.
Figure 16 is the schematic of the system for being used to select imaging device in movable equipment according to embodiments of the present invention
Block diagram, as shown in figure 16, system 1600 include：
Determining module 1601, for determining of each imaging device relative to other imaging devices in multiple imaging devices
One relative position, and each imaging device is relative to the second relative position of the movable equipment, wherein, it is the multiple
Imaging device is arranged on the movable equipment, and the multiple imaging device is including at least one first imaging device and at least
One the second imaging device, first imaging device is operated under multi-vision visual pattern, and second imaging device is operated in
Under monocular vision pattern；
Processing module 1602, for according to selection information, the selection target imaging device from the multiple imaging device, its
In, the selection information includes at least one of following message：Visual field model of each imaging device relative to the imaging device
In the distance of object or ground in enclosing, at least frame stereo-picture got by least one described first imaging device
Match point parallax and the working environment of the multiple imaging device, wherein, the distance is according to the described first relative position
Put what is determined with second relative position；
The processing module 1602, is additionally operable to control the target imaging equipment to obtain view data.
Therefore, the system for being used to select imaging device in movable equipment according to embodiments of the present invention, according to selection
Information, the selection target imaging device from multiple imaging devices, thus, it is possible to select more particularly suitable imaging device to obtain more
For accurate view data, to cause the number got in the view data for getting imaging device and other sensors system
During according to being merged, it is determined that the state of more accurate movable equipment, improves the security performance of movable equipment.
In embodiments of the present invention, alternatively, at least one described first imaging device is arranged on the movable equipment
Multiple positions at, direction at least two of the multiple position relative to the movable equipment.
In embodiments of the present invention, alternatively, at least one described second imaging device is arranged on the movable equipment
Carrier on, at least one described second imaging device can be revolved along at least one direction relative to the movable equipment
Turn.
In embodiments of the present invention, alternatively, the multi-vision visual pattern includes binocular vision pattern, when described the first one-tenth
When being operated in as equipment under binocular vision pattern, the view data that first imaging device is got includes video data,
The video data can be encoded using many mesh combined coding modes.
In embodiments of the present invention, alternatively, the processing module 1602 is additionally operable to：Determined by range sensor described
Distance of each imaging device relative to the object within the vision of the imaging device；Or, pass through range sensor and/or gas
Pressure meter determines distance of each imaging device relative to ground.
In embodiments of the present invention, alternatively, the processing module 1602 specifically for：According to the selection information and in advance
If distance threshold, the selection target imaging device from the multiple imaging device.
In embodiments of the present invention, alternatively, the processing module 1602 specifically for：When each imaging device phase
When the distance of object within the vision or ground for the imaging device is less than or equal to the pre-determined distance threshold value, by institute
State at least one first imaging device and be defined as the target imaging equipment；Or, when each imaging device relative to this into
When being more than the pre-determined distance threshold value as the object within the vision of equipment or the distance on ground, will it is described at least one second
Imaging device is defined as the target imaging equipment.
In embodiments of the present invention, alternatively, the processing module 1602 specifically for：According to the selection information and in advance
If parallax threshold value, the selection target imaging device from the multiple imaging device.
In embodiments of the present invention, alternatively, the processing module 1602 specifically for：When by it is described at least one
When the parallax of match point at least frame stereo-picture that one imaging device is got is more than or equal to the default parallax value,
At least one described first imaging device is defined as the target imaging equipment；Or, when by described at least one the first one-tenth
When the difference of match point at least frame stereo-picture got as equipment is less than the default parallax value, by described at least
One the second imaging device is defined as the target imaging equipment.
In embodiments of the present invention, alternatively, the processing module 1602 is additionally operable to：Institute is determined according to described image data
State at least one kinetic characteristic of movable equipment.
Each unit and other above-mentioned operation and/or functions in system 1600 according to embodiments of the present invention is respectively for reality
The corresponding flow of existing method 700, for sake of simplicity, will not be repeated here.
Figure 17 is that the imaging according to another embodiment of the present invention for being used to determine to be used for visual sensing on movable equipment is set
The schematic block diagram of the system of standby availability, as shown in figure 17, system 1700 include：
First processing module 1701, for the view data got according to multiple imaging devices for visual sensing,
Determine multiple first observer states of the movable equipment；
Second processing module 1702, for the sensing data got according to Inertial Measurement Unit IMU, is determined described removable
Multiple predicted states of dynamic equipment；
3rd processing module 1703, for pre- according to the first deviation between predicted state and the first observer state and first
If threshold value, it is determined that being each used for the availability of the imaging device of visual sensing.
Therefore, it is according to embodiments of the present invention be used to determine on movable equipment for visual sensing imaging device can
With the system of property, the view data that the sensing data got by Inertial Measurement Unit is got to multiple imaging devices is carried out
Verification, determines the availability of imaging device.Thereby, it is possible to get reliable view data by available imaging device, with
So that when the view data for getting available imaging device is merged with the data that other sensors system is got,
It is determined that the state of more accurate movable equipment, improves the security performance of movable equipment.
In embodiments of the present invention, alternatively, the multiple imaging device for visual sensing includes multiple the first one-tenth
As equipment and the second imaging device, wherein, the multiple first imaging device is arranged on the different directions of the movable equipment
On, second imaging device is connected by carrier with the movable equipment.
In embodiments of the present invention, alternatively, second imaging device can be along at least one direction relative to described
Movable equipment is rotated.
In embodiments of the present invention, alternatively, the 3rd processing module 1703 specifically for：According to predicted state and the
The first deviation and the first predetermined threshold value between one observer state, determine the first imaging device set, wherein, first imaging
Imaging device in cluster tool can use.
In embodiments of the present invention, alternatively, the 3rd processing module 1703 is additionally operable to：Will be according to the described first imaging
The first observer state and target prediction state that the view data that imaging device in cluster tool is got is determined are merged
Processing, wherein, the target prediction state be the multiple predicted state in the imaging in the first imaging device set
The corresponding predicted state of the first observer state that view data that equipment is got is determined.
In embodiments of the present invention, alternatively, the 3rd processing module 1703 specifically for：According to preset state and the
The first deviation and the first predetermined threshold value between one observer state, determine the second imaging device set, wherein, second imaging
Imaging device in cluster tool is unavailable.
In embodiments of the present invention, alternatively, the 3rd processing module 1703 is additionally operable to：Second imaging is abandoned to set
The view data that imaging device in standby set is got.
In embodiments of the present invention, alternatively, the first processing module 1701 is additionally operable to：According to global positioning system
The sensing data that GPS is got, determines the second observer state；Determine second between second observer state and predicted state
Deviation is less than or equal to the second predetermined threshold value.
Each unit and other above-mentioned operation and/or functions in system 1700 according to embodiments of the present invention is respectively for reality
The corresponding flow of existing method 1300, for sake of simplicity, will not be repeated here.
Figure 18 is the schematic block diagram of movable equipment according to embodiments of the present invention.As shown in figure 18, movable equipment
1800 include carrier 18 10 and load 1820.Movable equipment is described as unmanned plane just for the sake of in terms of description in Figure 18.
Load 1820 can be connected on movable equipment with obstructed carrier 1810 of crossing.Movable equipment 1800 can also include dynamical system
1830th, sensor-based system 1840 and communication system 1850.
Dynamical system 1830 can include electron speed regulator (referred to as electricity adjust), one or more propellers and with one
Or the corresponding one or more motors of multiple propellers.Motor and propeller are arranged on corresponding horn；Electron speed regulator
Drive signal for receiving flight controller generation, and driving current is provided to motor according to drive signal, with controlled motor
Rotating speed and/or steering.Motor is used to drive propeller to rotate, so as to provide power for UAV flight, the power causes UAV
The motion of one or more frees degree can be realized.In certain embodiments, UAV can be around the rotation of one or more rotary shafts
Turn.For example, above-mentioned rotary shaft can include roll axle, translation shaft and pitch axis.It should be understood that motor can be direct current generator,
Can be with alternating current generator.In addition, motor can be brushless electric machine, it is possibility to have brush motor.
Sensor-based system 1840 is used for the attitude information for measuring UAV, i.e. UAV is in the positional information and status information in space, example
Such as, three-dimensional position, three-dimensional perspective, three-dimensional velocity, three-dimensional acceleration and three-dimensional angular velocity etc..Sensor-based system can for example include top
Spiral shell instrument, electronic compass, Inertial Measurement Unit (Inertial Measurement Unit, referred to as " IMU "), vision sensor,
At least one in the sensor such as global positioning system (Global Positioning System, referred to as " GPS ") and barometer
Kind.Flight controller is used for the flight for controlling UAV, for example, the attitude information control UAV's that can be measured according to sensor-based system flies
OK.It should be understood that flight controller can be controlled according to the programmed instruction finished in advance to UAV, can also by respond come
UAV is controlled from one or more control instructions of commanding apparatus.
Communication system 1850 can pass through wireless signal 1880 with a terminal device 1860 with communication system 1870
Communicated.Communication system 1850 and communication system 1870 can include it is multiple for the emitter of radio communication, receiver and/
Or transceiver.Here radio communication can be one-way communication, for example, can only be movable equipment 1800 to terminal device 1860
Send data.Or radio communication can be two-way communication, data are that can be sent to terminal device from movable equipment 1800
1860, movable equipment 1800 can also be sent to by terminal device 1060.
Alternatively, terminal device 1860, which can be provided, is directed to one or more movable equipments 1800, the and of carrier 18 10
The control data of load 1820, and the information that movable equipment 1800, carrier 18 10 and load 1820 are sent can be received.Terminal is set
Standby 1860 control datas provided can be used in controlling one or more movable equipments 1800, carrier 18 10 and load 1820
State.Alternatively, carrier 18 10 and load 1020 include the communication module for being used to be communicated with terminal device 1860.
It is understood that movable equipment illustrated in fig. 18 can be including the system 1500 shown in Figure 15, Tu16Suo
The system 1700 shown in system 1600 and Figure 17 shown, and method 300,700 and 1300 is able to carry out, for sake of simplicity, herein not
Repeat again.
It should be understood that in various embodiments of the present invention, the size of the sequence number of above-mentioned each process is not meant to that execution is suitable
The priority of sequence, the execution sequence of each process should be determined with its function and internal logic, without the implementation of the reply embodiment of the present invention
Process constitutes any limit.
Those of ordinary skill in the art are it is to be appreciated that the list of each example described with reference to the embodiments described herein
Member and algorithm steps, can be realized with the combination of electronic hardware or computer software and electronic hardware.These functions are actually
Performed with hardware or software mode, depending on the application-specific and design constraint of technical scheme.Professional and technical personnel
Described function can be realized using distinct methods to each specific application, but this realization is it is not considered that exceed
The scope of the present invention.
It is apparent to those skilled in the art that, for convenience and simplicity of description, the system of foregoing description,
The specific work process of device and unit, may be referred to the corresponding process in preceding method embodiment, will not be repeated here.
, can be with several embodiments provided herein, it should be understood that disclosed systems, devices and methods
Realize by another way.For example, device embodiment described above is only schematical, for example, the unit
Divide, only a kind of division of logic function there can be other dividing mode when actually realizing, such as multiple units or component
Another system can be combined or be desirably integrated into, or some features can be ignored, or do not perform.It is another, it is shown or
The coupling each other discussed or direct-coupling or communication connection can be the indirect couplings of device or unit by some interfaces
Close or communicate to connect, can be electrical, machinery or other forms.
The unit illustrated as separating component can be or may not be it is physically separate, it is aobvious as unit
The part shown can be or may not be physical location, you can with positioned at a place, or can also be distributed to multiple
On NE.Some or all of unit therein can be selected to realize the mesh of this embodiment scheme according to the actual needs
's.
In addition, each functional unit in each embodiment of the invention can be integrated in a processing unit, can also
That unit is individually physically present, can also two or more units it is integrated in a unit.
If the function is realized using in the form of SFU software functional unit and is used as independent production marketing or in use, can be with
It is stored in a computer read/write memory medium.Understood based on such, technical scheme is substantially in other words
The part contributed to prior art or the part of the technical scheme can be embodied in the form of software product, the meter
Calculation machine software product is stored in a storage medium, including some instructions are to cause a computer equipment (can be individual
People's computer, server, or network equipment etc.) perform all or part of step of each of the invention embodiment methods described.
And foregoing storage medium includes：USB flash disk, mobile hard disk, read-only storage (ROM, Read-Only Memory), arbitrary access are deposited
Reservoir (RAM, Random Access Memory), magnetic disc or CD etc. are various can be with the medium of store program codes.
The foregoing is only a specific embodiment of the invention, but protection scope of the present invention is not limited thereto, any
Those familiar with the art the invention discloses technical scope in, change or replacement can be readily occurred in, should all be contained
Cover within protection scope of the present invention.Therefore, protection scope of the present invention should be based on the protection scope of the described claims.
Claims (111)
1. a kind of method for being used to determine the state of movable equipment, it is characterised in that the movable equipment includes first and passed
Sensor system and second sensor system, the data sampling frequency of the first sensor system and the second sensor system
Difference, methods described includes：
The second sensing data got in the second sensor system is unavailable or in period for not updating, according to described
The first sensing data that one sensing system is got, determines the predicted state of the movable equipment；
When it is determined that second sensing data that the second sensor system is got can use or update, according to described second
Sensing data, determines the first observer state of the movable equipment；
According to the first deviation between first observer state and the predicted state, it is determined whether according to the described first observation
State updates the state of the movable equipment, wherein, first deviation is used to indicate that second sensing data whether may be used
With.
2. according to the method described in claim 1, it is characterised in that first sensing data include first group of position data and
First group of exercise data, second sensing data includes second group of position data and second group of exercise data.
3. method according to claim 1 or 2, it is characterised in that the first sensor system includes inertia measurement list
First IMU.
4. according to the method in any one of claims 1 to 3, it is characterised in that the second sensor system includes complete
Ball position system GPS receiver.
5. according to the method in any one of claims 1 to 3, it is characterised in that the second sensor system includes one
Individual or multiple vision sensors.
6. method according to any one of claim 1 to 5, it is characterised in that methods described also includes：
According at least one priori prediction state of the movable equipment, first sensing data is determined.
7. method according to any one of claim 1 to 6, it is characterised in that described according to first observer state
The first deviation between the predicted state, it is determined whether update the movable equipment using first observer state
State, including：
According to first deviation and the first predetermined threshold value, it is determined whether update described removable using first observer state
The state of equipment.
8. method according to claim 7, it is characterised in that described according to first deviation and the first predetermined threshold value,
Determine whether the state using first observer state renewal movable equipment, including：
When it is determined that first deviation is less than or equal to first predetermined threshold value, it is determined that according to first observer state more
The state of the new movable equipment；
When it is determined that first deviation is more than first predetermined threshold value, it is determined that updating institute without using first observer state
State the state of movable equipment.
9. method according to claim 8, it is characterised in that first deviation be first observer state with it is described
Mahalanobis distance or Euclidean distance between predicted state.
10. method according to claim 8 or claim 9, it is characterised in that when it is determined that updating institute according to first observer state
When stating the state of movable equipment, methods described also includes：
The state of the movable equipment is updated according to the predicted state and first observer state.
11. method according to claim 8 or claim 9, it is characterised in that methods described also includes：
When it is determined that updating the state of the movable equipment without using first observer state, the predicted state is determined
For the state of the movable equipment.
12. the method according to any one of claim 7 to 11, it is characterised in that methods described also includes：
First predetermined threshold value is determined according at least one of following message：The running environment letter of the movable equipment
The height of breath, the body dynamics information of the movable equipment, the positional information of the movable equipment and the movable equipment
Spend information.
13. method according to claim 1 or 2, it is characterised in that the movable equipment also includes 3rd sensor system
System, the data sampling frequency of the 3rd sensor system and the first sensor system and the second sensor system are not
Together, methods described also includes：
When it is determined that the 3rd sensing data that the 3rd sensor system is got can use or update, according to the described 3rd sensing
Data, determine the second observer state of the movable equipment；
When it is determined that second sensing data can use or update, according to first observer state and second observer state
Between the second deviation, it is determined whether the state of the movable equipment is updated according to second observer state, wherein, it is described
Second deviation is used to indicate whether the 3rd sensing data can use.
14. method according to claim 13, it is characterised in that described according to first observer state and described second
The second deviation between observer state, it is determined whether the state of the movable equipment is updated according to second observer state,
Including：
According to second deviation and the second predetermined threshold value, it is determined whether update described removable using second observer state
The state of equipment.
15. method according to claim 14, it is characterised in that described according to second deviation and the second default threshold
Value, it is determined whether the state of the movable equipment is updated using second observer state, including：
When it is determined that second deviation is less than or equal to second predetermined threshold value, it is determined that according to second observer state
Update the state of the movable equipment；
When it is determined that second deviation is more than second predetermined threshold value, it is determined that being updated without using second observer state
The state of the movable equipment.
16. method according to claim 15, it is characterised in that the first sensor system includes IMU, described second
Sensing system includes gps receiver, and the 3rd sensor system includes one or more vision sensors.
17. the method according to claim 15 or 16, it is characterised in that when it is determined that being updated according to second observer state
During the state of the movable equipment, methods described also includes：
The state of the movable equipment is updated according to the predicted state and second observer state；Or,
The shape of the movable equipment is updated according to the predicted state, first observer state and second observer state
State.
18. the method according to claim 15 or 16, it is characterised in that when it is determined that not according to second observer state more
During the state of the new movable equipment, methods described includes：
The state of the movable equipment is updated according to the predicted state and first observer state.
19. the method according to any one of claim 13 to 18, it is characterised in that methods described also includes：
Second predetermined threshold value is determined according at least one of following message：The running environment letter of the movable equipment
The height of breath, the body dynamics information of the movable equipment, the positional information of the movable equipment and the movable equipment
Spend information.
20. a kind of method for being used to select imaging device in movable equipment, it is characterised in that set on the movable equipment
Multiple imaging devices are equipped with, the multiple imaging device includes at least one first imaging device and at least one second imaging is set
Standby, first imaging device is operated under multi-vision visual pattern, and second imaging device is operated under monocular vision pattern,
Methods described includes：
The first relative position of each imaging device in the multiple imaging device relative to other imaging devices is determined, and it is described
Second relative position of each imaging device relative to the movable equipment；
According to selection information, the selection target imaging device from the multiple imaging device, wherein, under the selection information includes
At least one of column information：Each imaging device relative to the imaging device object within the vision or ground away from
Parallax and institute from the match point in, at least frame stereo-picture that is got by least one described first imaging device
The working environment of multiple imaging devices is stated, wherein, the distance is according to first relative position and the second relative position
Put determination；
View data is obtained using the target imaging equipment.
21. method according to claim 20, it is characterised in that at least one described first imaging device is arranged on described
At multiple positions of movable equipment, direction at least two of the multiple position relative to the movable equipment.
22. the method according to claim 20 or 21, it is characterised in that at least one described second imaging device is arranged on
On the carrier of the movable equipment, at least one described second imaging device can along at least one direction relative to it is described can
Mobile device is rotated.
23. the method according to any one of claim 20 to 22, it is characterised in that the multi-vision visual pattern includes double
Mesh visual pattern, when first imaging device is operated under binocular vision pattern, what first imaging device was got
View data includes video data, and the video data can be encoded using many mesh combined coding modes.
24. the method according to any one of claim 20 to 23, it is characterised in that methods described also includes：
By range sensor determine each imaging device relative to the object within the vision of the imaging device away from
From；Or,
Distance of each imaging device relative to ground is determined by range sensor and/or barometer.
25. the method according to any one of claim 20 to 24, it is characterised in that described according to selection information, from institute
Selection target imaging device in multiple imaging devices is stated, including：
According to the selection information and pre-determined distance threshold value, the selection target imaging device from the multiple imaging device.
26. method according to claim 25, it is characterised in that described according to the selection information and pre-determined distance threshold
Value, the selection target imaging device from the multiple imaging device, including：
When each imaging device is less than or waited relative to the object within the vision of the imaging device or the distance on ground
When the pre-determined distance threshold value, at least one described first imaging device is defined as the target imaging equipment；Or,
When each imaging device relative to the object within the vision of the imaging device or the distance on ground more than described
During pre-determined distance threshold value, at least one described second imaging device is defined as the target imaging equipment.
27. the method according to any one of claim 20 to 24, it is characterised in that described according to selection information, from institute
Selection target imaging device in multiple imaging devices is stated, including：
According to the selection information and default parallax threshold value, the selection target imaging device from the multiple imaging device.
28. method according to claim 27, it is characterised in that described according to the selection information and default parallax threshold
Value, the selection target imaging device from the multiple imaging device, including：
When the parallax of the match point at least frame stereo-picture got by least one described first imaging device is big
When the default parallax value, at least one described first imaging device is defined as the target imaging equipment；Or,
When the difference of the match point at least frame stereo-picture got by least one described first imaging device is small
When the default parallax value, at least one described second imaging device is defined as the target imaging equipment.
29. the method according to any one of claim 20 to 28, it is characterised in that methods described also includes：
At least one kinetic characteristic of the movable equipment is determined according to described image data.
30. a kind of be used to determine to be used for the method for the availability of the imaging device of visual sensing on movable equipment, its feature exists
In, including：
The view data got according to multiple imaging devices for visual sensing, determines multiple the of the movable equipment
One observer state；
The sensing data got according to Inertial Measurement Unit IMU, determines multiple predicted states of the movable equipment；
According to the first deviation and the first predetermined threshold value between predicted state and the first observer state, passed it is determined that being each used for vision
The availability of the imaging device of sense.
31. method according to claim 30, it is characterised in that the multiple imaging device for visual sensing includes
Multiple first imaging devices and the second imaging device, wherein, the multiple first imaging device is arranged on the movable equipment
Different directions on, second imaging device is connected by carrier with the movable equipment.
32. method according to claim 31, it is characterised in that second imaging device can be along at least one direction
Rotated relative to the movable equipment.
33. the method according to any one of claim 30 to 32, it is characterised in that described according to predicted state and first
The first deviation and the first predetermined threshold value between observer state, it is determined that each it is used for the availability of the imaging device of visual sensing,
Including：
According to the first deviation and the first predetermined threshold value between predicted state and the first observer state, the first imaging device collection is determined
Close, wherein, the imaging device in the first imaging device set can use.
34. method according to claim 33, it is characterised in that methods described also includes：
The first observer state that the view data that imaging device in the first imaging device set is got is determined
Fusion treatment is carried out with target prediction state, wherein, the target prediction state is with described the in the multiple predicted state
The corresponding predicted state of the first observer state that view data that imaging device in one imaging device set is got is determined.
35. the method according to any one of claim 30 to 32, it is characterised in that described according to predicted state and first
The first deviation and the first predetermined threshold value between observer state, it is determined that each it is used for the availability of the imaging device of visual sensing,
Including：
According to the first deviation and the first predetermined threshold value between predicted state and the first observer state, the second imaging device collection is determined
Close, wherein, the imaging device in the second imaging device set is unavailable.
36. method according to claim 35, it is characterised in that methods described also includes：
Abandon the view data that the imaging device in the second imaging device set is got.
37. the method according to any one of claim 30 to 36, it is characterised in that seen according to predicted state with first
The first deviation and the first predetermined threshold value between survey state, it is determined that each for visual sensing imaging device availability it
Before, methods described also includes：
The sensing data got according to global position system GPS, determines the second observer state；
Determine that the second deviation between second observer state and predicted state is less than or equal to the second predetermined threshold value.
38. a kind of system for being used to determine the state of movable equipment, it is characterised in that including：
Memory, for storage program；
At least one processor, by performing the program in the memory, is either individually or collectively used for：
The sensing data that the multiple sensors associated with the movable equipment are obtained is obtained, wherein, the multiple sensor
Including first sensor system and second sensor system, the number of the first sensor system and the second sensor system
It is different according to sample frequency；
The second sensing data got in the second sensor system is unavailable or in period for not updating, according to described
The first sensing data that one sensing system is got, determines the predicted state of the movable equipment；
When it is determined that second sensing data that the second sensor system is got can use or update, according to described second
Sensing data, determines the first observer state of the movable equipment；
According to the first deviation between first observer state and the predicted state, it is determined whether according to the described first observation
State updates the state of the movable equipment, wherein, first deviation is used to indicate that second sensing data whether may be used
With.
39. the system according to claim 38, it is characterised in that first sensing data includes first group of position data
With first group of exercise data, second sensing data includes second group of position data and second group of exercise data.
40. the system according to claim 38 or 39, it is characterised in that the first sensor system includes inertia measurement
Unit IMU.
41. the system according to any one of claim 38 to 40, it is characterised in that the second sensor system includes
Global positioning system GPS receiver.
42. the system according to any one of claim 38 to 40, it is characterised in that the second sensor system includes
One or more vision sensors.
43. the system according to any one of claim 38 to 42, it is characterised in that the processor is additionally operable to：
According at least one priori prediction state of the movable equipment, first sensing data is determined.
44. the system according to any one of claim 38 to 43, it is characterised in that the processor specifically for：
According to first deviation and the first predetermined threshold value, it is determined whether update described removable using first observer state
The state of equipment.
45. system according to claim 44, it is characterised in that the processor specifically for：
When it is determined that first deviation is less than or equal to first predetermined threshold value, it is determined that according to first observer state more
The state of the new movable equipment；
When it is determined that first deviation is more than first predetermined threshold value, it is determined that updating institute without using first observer state
State the state of movable equipment.
46. system according to claim 45, it is characterised in that first deviation is first observer state and institute
State the mahalanobis distance or Euclidean distance between predicted state.
47. the system according to claim 45 or 46, it is characterised in that when it is determined that being updated according to first observer state
During the state of the movable equipment, the processor is additionally operable to：
The state of the movable equipment is updated according to the predicted state and first observer state.
48. the system according to claim 45 or 46, it is characterised in that the processor is additionally operable to：
When it is determined that updating the state of the movable equipment without using first observer state, the predicted state is determined
For the state of the movable equipment.
49. the system according to any one of claim 44 to 48, it is characterised in that the processor is additionally operable to：
First predetermined threshold value is determined according at least one of following message：The running environment letter of the movable equipment
The height of breath, the body dynamics information of the movable equipment, the positional information of the movable equipment and the movable equipment
Spend information.
50. the system according to claim 38 or 39, it is characterised in that the movable equipment also includes 3rd sensor
System, the data sampling frequency of the 3rd sensor system and the first sensor system and the second sensor system
Difference, the processor is additionally operable to：
When it is determined that the 3rd sensing data that the 3rd sensor system is got can use or update, according to the described 3rd sensing
Data, determine the second observer state of the movable equipment；
When it is determined that second sensing data can use or update, according to first observer state and second observer state
Between the second deviation, it is determined whether the state of the movable equipment is updated according to second observer state, wherein, it is described
Second deviation is used to indicate whether the 3rd sensing data can use.
51. system according to claim 50, it is characterised in that the processor specifically for：
According to second deviation and the second predetermined threshold value, it is determined whether update described removable using second observer state
The state of equipment.
52. system according to claim 51, it is characterised in that the processor specifically for：
When it is determined that second deviation is less than or equal to second predetermined threshold value, it is determined that according to second observer state
Update the state of the movable equipment；
When it is determined that second deviation is more than second predetermined threshold value, it is determined that being updated without using second observer state
The state of the movable equipment.
53. system according to claim 52, it is characterised in that the first sensor system includes IMU, described second
Sensing system includes gps receiver, and the 3rd sensor system includes one or more vision sensors.
54. the system according to claim 52 or 53, it is characterised in that when it is determined that being updated according to second observer state
During the state of the movable equipment, the processor is additionally operable to：
The state of the movable equipment is updated according to the predicted state and second observer state；Or, according to described pre-
Survey state, first observer state and second observer state update the state of the movable equipment.
55. the system according to claim 52 or 53, it is characterised in that when it is determined that not according to second observer state more
During the state of the new movable equipment, the processor is additionally operable to：
The state of the movable equipment is updated according to the predicted state and first observer state.
56. the system according to any one of claim 50 to 55, it is characterised in that the processor is additionally operable to：
Second predetermined threshold value is determined according at least one of following message：The running environment letter of the movable equipment
The height of breath, the body dynamics information of the movable equipment, the positional information of the movable equipment and the movable equipment
Spend information.
57. a kind of system for being used to select imaging device in movable equipment, it is characterised in that including：
Memory, for storage program；
At least one processor, by performing the program of memory storage, is either individually or collectively used for：
The first relative position of each imaging device in multiple imaging devices relative to other imaging devices is determined, and it is described each
Imaging device relative to the movable equipment the second relative position, wherein, the multiple imaging device be arranged on it is described can
On mobile device, the multiple imaging device includes at least one first imaging device and at least one second imaging device, institute
State the first imaging device to be operated under multi-vision visual pattern, second imaging device is operated under monocular vision pattern；
According to selection information, the selection target imaging device from the multiple imaging device, wherein, under the selection information includes
At least one of column information：Each imaging device relative to the imaging device object within the vision or ground away from
Parallax and institute from the match point in, at least frame stereo-picture that is got by least one described first imaging device
The working environment of multiple imaging devices is stated, wherein, the distance is according to first relative position and the second relative position
Put determination；
The target imaging equipment is controlled to obtain view data.
58. system according to claim 57, it is characterised in that at least one described first imaging device is arranged on described
At multiple positions of movable equipment, direction at least two of the multiple position relative to the movable equipment.
59. the system according to claim 57 or 58, it is characterised in that at least one described second imaging device is arranged on
On the carrier of the movable equipment, at least one described second imaging device can along at least one direction relative to it is described can
Mobile device is rotated.
60. the system according to any one of claim 57 to 59, it is characterised in that the multi-vision visual pattern includes double
Mesh visual pattern, when first imaging device is operated under binocular vision pattern, what first imaging device was got
View data includes video data, and the video data can be encoded using many mesh combined coding modes.
61. the system according to any one of claim 57 to 60, it is characterised in that the processor is additionally operable to：
By range sensor determine each imaging device relative to the object within the vision of the imaging device away from
From；Or,
Distance of each imaging device relative to ground is determined by range sensor and/or barometer.
62. the system according to any one of claim 57 to 61, it is characterised in that the processor specifically for：
According to the selection information and pre-determined distance threshold value, the selection target imaging device from the multiple imaging device.
63. system according to claim 62, it is characterised in that the processor specifically for：
When each imaging device is less than or waited relative to the object within the vision of the imaging device or the distance on ground
When the pre-determined distance threshold value, at least one described first imaging device is defined as the target imaging equipment；Or,
When each imaging device relative to the object within the vision of the imaging device or the distance on ground more than described
During pre-determined distance threshold value, at least one described second imaging device is defined as the target imaging equipment.
64. the system according to any one of claim 57 to 61, it is characterised in that the processor specifically for：
According to the selection information and default parallax threshold value, the selection target imaging device from the multiple imaging device.
65. system according to claim 64, it is characterised in that the processor specifically for：
When the parallax of the match point at least frame stereo-picture got by least one described first imaging device is big
When the default parallax value, at least one described first imaging device is defined as the target imaging equipment；Or,
When the difference of the match point at least frame stereo-picture got by least one described first imaging device is small
When the default parallax value, at least one described second imaging device is defined as the target imaging equipment.
66. the system according to any one of claim 57 to 65, it is characterised in that the processor is additionally operable to：
At least one kinetic characteristic of the movable equipment is determined according to described image data.
67. a kind of be used to determine to be used for the system of the availability of the imaging device of visual sensing on movable equipment, its feature exists
In, including：
Memory, for storage program；
At least one processor, by performing the program of memory storage, is either individually or collectively used for：
The view data got according to multiple imaging devices for visual sensing, determines that multiple the first of the unmanned plane are seen
Survey state；
The sensing data got according to Inertial Measurement Unit IMU, determines multiple predicted states of the movable equipment；
According to the first deviation and the first predetermined threshold value between predicted state and the first observer state, passed it is determined that being each used for vision
The availability of the imaging device of sense.
68. system according to claim 67, it is characterised in that the multiple imaging device for visual sensing includes
Multiple first imaging devices and the second imaging device, wherein, the multiple first imaging device is arranged on the movable equipment
Different directions on, second imaging device is connected by carrier with the movable equipment.
69. system according to claim 68, it is characterised in that second imaging device can be along at least one direction
Rotated relative to the movable equipment.
70. the system according to any one of claim 67 to 69, it is characterised in that the processor specifically for：
According to the first deviation and the first predetermined threshold value between predicted state and the first observer state, the first imaging device collection is determined
Close, wherein, the imaging device in the first imaging device set can use.
71. system according to claim 70, it is characterised in that the processor is additionally operable to：
The first observer state that the view data that imaging device in the first imaging device set is got is determined
Fusion treatment is carried out with target prediction state, wherein, the target prediction state is with described the in the multiple predicted state
The corresponding predicted state of the first observer state that view data that imaging device in one imaging device set is got is determined.
72. the system according to any one of claim 67 to 69, it is characterised in that the processor is additionally operable to：
According to the first deviation and the first predetermined threshold value between predicted state and the first observer state, the second imaging device collection is determined
Close, wherein, the imaging device in the second imaging device set is unavailable.
73. the system according to claim 72, it is characterised in that the processor is additionally operable to：
Abandon the view data that the imaging device in the second imaging device set is got.
74. the system according to any one of claim 67 to 73, it is characterised in that the processor is additionally operable to：
The sensing data got according to global position system GPS, determines the second observer state；
Determine that the second deviation between second observer state and predicted state is less than or equal to the second predetermined threshold value.
75. a kind of system for being used to determine the state of movable equipment, it is characterised in that including：
Acquisition module, for obtaining the sensing data that the multiple sensors associated with the movable equipment are obtained, wherein, institute
Stating multiple sensors includes first sensor system and second sensor system, and the first sensor system is passed with described second
The data sampling frequency of sensor system is different；
Determining module, the second sensing data for being got in the second sensor system is unavailable or period for not updating
The first sensing data that is interior, being got according to the first sensor system, determines the predicted state of the movable equipment；
The determining module, be additionally operable to when determine second sensing data that the second sensor system is got it is available or
During renewal, according to second sensing data, the first observer state of the movable equipment is determined；
The determining module, is additionally operable to according to the first deviation between first observer state and the predicted state, it is determined that
Whether the state of the movable equipment is updated according to first observer state, wherein, first deviation is used to indicate institute
State whether the second sensing data can use.
76. the system according to claim 75, it is characterised in that first sensing data includes first group of position data
With first group of exercise data, second sensing data includes second group of position data and second group of exercise data.
77. the system according to claim 75 or 76, it is characterised in that the first sensor system includes inertia measurement
Unit IMU.
78. the system according to any one of claim 75 to 77, it is characterised in that the second sensor system includes
Global positioning system GPS receiver.
79. the system according to any one of claim 75 to 77, it is characterised in that the second sensor system includes
One or more vision sensors.
80. the system according to any one of claim 75 to 79, it is characterised in that the acquisition module is additionally operable to：
According at least one priori prediction state of the movable equipment, first sensing data is determined.
81. the system according to any one of claim 75 to 80, it is characterised in that the determining module specifically for：
According to first deviation and the first predetermined threshold value, it is determined whether update described removable using first observer state
The state of equipment.
82. the system according to claim 81, it is characterised in that the determining module specifically for：
When it is determined that first deviation is less than or equal to first predetermined threshold value, it is determined that according to first observer state more
The state of the new movable equipment；
When it is determined that first deviation is more than first predetermined threshold value, it is determined that updating institute without using first observer state
State the state of movable equipment.
83. the system according to claim 82, it is characterised in that first deviation is first observer state and institute
State the mahalanobis distance or Euclidean distance between predicted state.
84. the system according to claim 82 or 83, it is characterised in that when it is determined that being updated according to first observer state
During the state of the movable equipment, the determining module is additionally operable to：
The state of the movable equipment is updated according to the predicted state and first observer state.
85. the system according to claim 82 or 83, it is characterised in that the determining module is additionally operable to：
When it is determined that updating the state of the movable equipment without using first observer state, the predicted state is determined
For the state of the movable equipment.
86. the system according to any one of claim 81 to 85, it is characterised in that the determining module is additionally operable to：
First predetermined threshold value is determined according at least one of following message：The running environment letter of the movable equipment
The height of breath, the body dynamics information of the movable equipment, the positional information of the movable equipment and the movable equipment
Spend information.
87. the system according to claim 75 or 76, it is characterised in that the movable equipment also includes 3rd sensor
System, the data sampling frequency of the 3rd sensor system and the first sensor system and the second sensor system
Difference, the determining module is additionally operable to：
When it is determined that the 3rd sensing data that the 3rd sensor system is got can use or update, according to the described 3rd sensing
Data, determine the second observer state of the movable equipment；
When it is determined that second sensing data can use or update, according to first observer state and second observer state
Between the second deviation, it is determined whether the state of the movable equipment is updated according to second observer state, wherein, it is described
Second deviation is used to indicate whether the 3rd sensing data can use.
88. the system according to claim 87, it is characterised in that the determining module specifically for：
According to second deviation and the second predetermined threshold value, it is determined whether update described removable using second observer state
The state of equipment.
89. the system according to claim 88, it is characterised in that the determining module specifically for：
When it is determined that second deviation is less than or equal to second predetermined threshold value, it is determined that according to second observer state
Update the state of the movable equipment；
When it is determined that second deviation is more than second predetermined threshold value, it is determined that being updated without using second observer state
The state of the movable equipment.
90. the system according to claim 89, it is characterised in that the first sensor system includes IMU, described second
Sensing system includes gps receiver, and the 3rd sensor system includes one or more vision sensors.
91. the system according to claim 89 or 90, it is characterised in that when it is determined that being updated according to second observer state
During the state of the movable equipment, the determining module is additionally operable to：
The state of the movable equipment is updated according to the predicted state and second observer state；Or, according to described pre-
Survey state, first observer state and second observer state update the state of the movable equipment.
92. the system according to claim 89 or 90, it is characterised in that when it is determined that not according to second observer state more
During the state of the new movable equipment, the determining module is additionally operable to：
The state of the movable equipment is updated according to the predicted state and first observer state.
93. the system according to any one of claim 87 to 92, it is characterised in that the determining module is additionally operable to：
Second predetermined threshold value is determined according at least one of following message：The running environment letter of the movable equipment
The height of breath, the body dynamics information of the movable equipment, the positional information of the movable equipment and the movable equipment
Spend information.
94. a kind of system for being used to select imaging device in movable equipment, it is characterised in that including：
Determining module, for determining first relative position of each imaging device relative to other imaging devices in multiple imaging devices
Put, and each imaging device is relative to the second relative position of the movable equipment, wherein, the multiple imaging device
Be arranged on the movable equipment, the multiple imaging device include at least one first imaging device and at least one second
Imaging device, first imaging device is operated under multi-vision visual pattern, and second imaging device is operated in monocular vision
Under pattern；
Processing module, for according to selection information, the selection target imaging device from the multiple imaging device, wherein, it is described
Information is selected to include at least one of following message：Within the vision thing of each imaging device relative to the imaging device
Match point in the distance on body or ground, at least frame stereo-picture got by least one described first imaging device
Parallax and the multiple imaging device working environment, wherein, the distance is according to first relative position and described
What the second relative position was determined；
The processing module, is additionally operable to control the target imaging equipment to obtain view data.
95. the system according to claim 94, it is characterised in that at least one described first imaging device is arranged on described
At multiple positions of movable equipment, direction at least two of the multiple position relative to the movable equipment.
96. the system according to claim 94 or 95, it is characterised in that at least one described second imaging device is arranged on
On the carrier of the movable equipment, at least one described second imaging device can along at least one direction relative to it is described can
Mobile device is rotated.
97. the system according to any one of claim 94 to 96, it is characterised in that the multi-vision visual pattern includes double
Mesh visual pattern, when first imaging device is operated under binocular vision pattern, what first imaging device was got
View data includes video data, and the video data can be encoded using many mesh combined coding modes.
98. the system according to any one of claim 94 to 97, it is characterised in that the processing module is additionally operable to：
By range sensor determine each imaging device relative to the object within the vision of the imaging device away from
From；Or,
Distance of each imaging device relative to ground is determined by range sensor and/or barometer.
99. the system according to any one of claim 94 to 98, it is characterised in that the processing module specifically for：
According to the selection information and pre-determined distance threshold value, the selection target imaging device from the multiple imaging device.
100. the system according to claim 99, it is characterised in that the processing module specifically for：
When each imaging device is less than or waited relative to the object within the vision of the imaging device or the distance on ground
When the pre-determined distance threshold value, at least one described first imaging device is defined as the target imaging equipment；Or,
When each imaging device relative to the object within the vision of the imaging device or the distance on ground more than described
During pre-determined distance threshold value, at least one described second imaging device is defined as the target imaging equipment.
101. the system according to any one of claim 94 to 98, it is characterised in that the processing module specifically for：
According to the selection information and default parallax threshold value, the selection target imaging device from the multiple imaging device.
102. the system according to claim 101, it is characterised in that the processing module specifically for：
When the parallax of the match point at least frame stereo-picture got by least one described first imaging device is big
When the default parallax value, at least one described first imaging device is defined as the target imaging equipment；Or,
When the difference of the match point at least frame stereo-picture got by least one described first imaging device is small
When the default parallax value, at least one described second imaging device is defined as the target imaging equipment.
103. the system according to any one of claim 94 to 102, it is characterised in that the processing module is additionally operable to：
At least one kinetic characteristic of the movable equipment is determined according to described image data.
104. a kind of be used to determine to be used for the system of the availability of the imaging device of visual sensing on movable equipment, its feature exists
In, including：
First processing module, for the view data got according to multiple imaging devices for visual sensing, it is determined that described
Multiple first observer states of movable equipment；
Second processing module, for the sensing data got according to Inertial Measurement Unit IMU, determines the movable equipment
Multiple predicted states；
3rd processing module, for according to the first deviation and the first predetermined threshold value between predicted state and the first observer state,
It is determined that being each used for the availability of the imaging device of visual sensing.
105. the system according to claim 104, it is characterised in that the multiple imaging device bag for visual sensing
Multiple first imaging devices and the second imaging device are included, wherein, the multiple first imaging device is arranged on described may move and set
On standby different directions, second imaging device is connected by carrier with the movable equipment.
106. the system according to claim 105, it is characterised in that second imaging device can be along at least one side
Rotated to relative to the movable equipment.
107. the system according to any one of claim 104 to 106, it is characterised in that the 3rd processing module tool
Body is used for：
According to the first deviation and the first predetermined threshold value between predicted state and the first observer state, the first imaging device collection is determined
Close, wherein, the imaging device in the first imaging device set can use.
108. the system according to claim 107, it is characterised in that the 3rd processing module is additionally operable to：
The first observer state that the view data that imaging device in the first imaging device set is got is determined
Fusion treatment is carried out with target prediction state, wherein, the target prediction state is with described the in the multiple predicted state
The corresponding predicted state of the first observer state that view data that imaging device in one imaging device set is got is determined.
109. the system according to any one of claim 104 to 106, it is characterised in that the 3rd processing module tool
Body is used for：
According to the first deviation and the first predetermined threshold value between preset state and the first observer state, the second imaging device collection is determined
Close, wherein, the imaging device in the second imaging device set is unavailable.
110. the system according to claim 109, it is characterised in that the 3rd processing module is additionally operable to：
Abandon the view data that the imaging device in the second imaging device set is got.
111. the system according to any one of claim 104 to 110, it is characterised in that the first processing module is also
For：
The sensing data got according to global position system GPS, determines the second observer state；
Determine that the second deviation between second observer state and predicted state is less than or equal to the second predetermined threshold value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110511494.XA CN113223286A (en) | 2016-11-14 | 2016-11-14 | Method and system for fusing multi-channel sensing data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/CN2016/105747 WO2018086122A1 (en) | 2016-11-14 | 2016-11-14 | Method and system for fusion of multiple paths of sensing data |
Related Child Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110511494.XA Division CN113223286A (en) | 2016-11-14 | 2016-11-14 | Method and system for fusing multi-channel sensing data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107223275A true CN107223275A (en) | 2017-09-29 |
| CN107223275B CN107223275B (en) | 2021-05-28 |
Family
ID=59927633
Family Applications (2)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110511494.XA Pending CN113223286A (en) | 2016-11-14 | 2016-11-14 | Method and system for fusing multi-channel sensing data |
| CN201680004510.3A Expired - Fee Related CN107223275B (en) | 2016-11-14 | 2016-11-14 | Method and system for fusing multi-channel sensing data |
Family Applications Before (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110511494.XA Pending CN113223286A (en) | 2016-11-14 | 2016-11-14 | Method and system for fusing multi-channel sensing data |
Country Status (2)
| Country | Link |
|---|---|
| CN (2) | CN113223286A (en) |
| WO (1) | WO2018086122A1 (en) |
Cited By (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108106614A (en) * | 2017-12-22 | 2018-06-01 | 北京轻威科技有限责任公司 | A Data Fusion Algorithm for Inertial Sensor and Visual Sensor |
| CN108563235A (en) * | 2018-05-24 | 2018-09-21 | 南方科技大学 | Multi-rotor unmanned aerial vehicle, method, device and equipment for grabbing target object |
| CN108616838A (en) * | 2018-04-29 | 2018-10-02 | 山东省计算中心（国家超级计算济南中心） | Agricultural greenhouse Data Fusion method based on Kalman filtering algorithm |
| CN108646739A (en) * | 2018-05-14 | 2018-10-12 | 北京智行者科技有限公司 | A kind of sensor information fusion method |
| CN108733076A (en) * | 2018-05-24 | 2018-11-02 | 南方科技大学 | Method and device for grabbing target object by unmanned aerial vehicle and electronic equipment |
| CN108961716A (en) * | 2018-07-25 | 2018-12-07 | 安徽江淮汽车集团股份有限公司 | A kind of wireless kilowatt meter reading-out system based on ZigBee |
| CN109073390A (en) * | 2018-07-23 | 2018-12-21 | 深圳前海达闼云端智能科技有限公司 | A kind of localization method and device, electronic equipment and readable storage medium storing program for executing |
| CN109682371A (en) * | 2017-10-18 | 2019-04-26 | 苏州宝时得电动工具有限公司 | Automatic running device and its localization method and device |
| CN109737981A (en) * | 2019-01-11 | 2019-05-10 | 西安电子科技大学 | Target search device and method for unmanned vehicle based on multi-sensor |
| CN109781163A (en) * | 2018-12-18 | 2019-05-21 | 北京百度网讯科技有限公司 | Calibrating parameters validity check method, apparatus, equipment and storage medium |
| WO2019140649A1 (en) * | 2018-01-19 | 2019-07-25 | 深圳市大疆创新科技有限公司 | Visual module integration module and unmanned aerial vehicle |
| CN110146055A (en) * | 2019-05-21 | 2019-08-20 | 深圳市道通智能航空技术有限公司 | A kind of the super voice abnormality detection method, device and electronic equipment |
| CN110174105A (en) * | 2019-06-14 | 2019-08-27 | 西南科技大学 | Intelligent body Autonomous Navigation Algorithm and system under a kind of complex environment |
| CN110667598A (en) * | 2018-06-18 | 2020-01-10 | 百度（美国）有限责任公司 | Method for handling sensor faults in autonomous vehicles |
| CN110926484A (en) * | 2020-02-10 | 2020-03-27 | 中智行科技有限公司 | Vehicle position obtaining method and device and intelligent vehicle |
| CN110967064A (en) * | 2018-09-28 | 2020-04-07 | 罗伯特·博世有限公司 | Method, device and sensor system for environmental detection of a vehicle |
| CN111079607A (en) * | 2019-12-05 | 2020-04-28 | 灵动科技（北京）有限公司 | Autonomous Driving System with Tracking |
| CN112346040A (en) * | 2019-08-09 | 2021-02-09 | 大众汽车股份公司 | Method and device for determining parallax problem in sensor data of two sensors |
| CN112389456A (en) * | 2019-08-15 | 2021-02-23 | 大众汽车股份公司 | Method and device for checking the calibration of an environmental sensor |
| CN113436261A (en) * | 2021-06-24 | 2021-09-24 | 湖南大学 | Monocular vision inertial positioning method for automatic driving of closed park |
| CN113597590A (en) * | 2020-04-27 | 2021-11-02 | 深圳市大疆创新科技有限公司 | Movable platform and obstacle avoidance method and device thereof |
| CN113762001A (en) * | 2020-10-10 | 2021-12-07 | 北京京东乾石科技有限公司 | Target detection method and device, electronic equipment and storage medium |
| CN114485760A (en) * | 2022-01-26 | 2022-05-13 | 震坤行工业超市(上海)有限公司 | Sensor calibration method, electronic device, medium, and system |
| CN110780610B (en) * | 2019-10-30 | 2022-10-18 | 航宇救生装备有限公司 | Method for fusing and processing health management signals of ejection lifesaving system |
| TWI830363B (en) * | 2022-05-19 | 2024-01-21 | 鈺立微電子股份有限公司 | Sensing device for providing three dimensional information |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3736597B1 (en) | 2019-05-06 | 2025-02-12 | Easymile | Method and system for modifying the speed of a vehicle depending on the validation of its calculated localization field |
| CN112309131B (en) * | 2019-07-30 | 2022-12-27 | 北京京东振世信息技术有限公司 | Parking space state monitoring method and device, storage medium and electronic equipment |
| CN112747754B (en) * | 2019-10-30 | 2025-01-14 | 北京魔门塔科技有限公司 | A method, device and system for fusion of multi-sensor data |
| CN111310627B (en) * | 2020-02-07 | 2024-01-30 | 广州视源电子科技股份有限公司 | Detection method and device of sensing device and electronic equipment |
| CN112753213A (en) * | 2020-04-26 | 2021-05-04 | 深圳市大疆创新科技有限公司 | Target positioning method, movable platform and storage medium |
| CN112790786A (en) * | 2020-12-30 | 2021-05-14 | 无锡祥生医疗科技股份有限公司 | Point cloud data registration method and device, ultrasonic equipment and storage medium |
| CN115167508B (en) * | 2022-07-05 | 2023-05-09 | 南京邮电大学 | A multi-rotor UAV sensor fail-safe flight control system and method |
| WO2024247452A1 (en) * | 2023-05-30 | 2024-12-05 | 株式会社堀場エステック | Computation device, pressure measurement device, computation method, and computation program |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001224064A (en) * | 2000-02-07 | 2001-08-17 | Nippon Telegraph & Telephone East Corp | Portable information terminal and network service system using portable information terminal |
| CN102789233A (en) * | 2012-06-12 | 2012-11-21 | 湖北三江航天红峰控制有限公司 | Vision-based combined navigation robot and navigation method |
| CN103529692A (en) * | 2013-10-30 | 2014-01-22 | 中国航天空气动力技术研究院 | Fault reconstruction method for simple redundancy flight control system of long-endurance unmanned aerial vehicle |
| CN105022397A (en) * | 2014-04-25 | 2015-11-04 | 谷歌公司 | Methods and systems for object detection using multiple sensors |
| CN105700550A (en) * | 2016-01-26 | 2016-06-22 | 深圳市大疆创新科技有限公司 | Unmanned plane and flight control method and system therefor |
| CN105814875A (en) * | 2013-12-06 | 2016-07-27 | 谷歌公司 | Selecting camera pairs for stereoscopic imaging |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4386021B2 (en) * | 2005-10-11 | 2009-12-16 | ソニー株式会社 | Imaging device |
| CN101995741B (en) * | 2009-08-27 | 2013-08-28 | 鸿富锦精密工业（深圳）有限公司 | Camera switching system and method |
| KR20120071857A (en) * | 2010-12-23 | 2012-07-03 | 팅크웨어(주) | Electronic device and mehtod of shooting pictures thereof |
| CN102243765A (en) * | 2011-05-06 | 2011-11-16 | 大连民族学院 | Multi-camera-based multi-objective positioning tracking method and system |
| US9354488B2 (en) * | 2014-05-23 | 2016-05-31 | Howard Preston | Methods, apparatuses, systems and software for focusing a camera |
| CN105989737B (en) * | 2015-02-05 | 2019-04-05 | 辽宁省交通高等专科学校 | A kind of parking induction method |
| CN105763774B (en) * | 2016-02-29 | 2019-07-26 | 联想(北京)有限公司 | A kind of camera module, electronic equipment and image-pickup method |
-
2016
- 2016-11-14 CN CN202110511494.XA patent/CN113223286A/en active Pending
- 2016-11-14 CN CN201680004510.3A patent/CN107223275B/en not_active Expired - Fee Related
- 2016-11-14 WO PCT/CN2016/105747 patent/WO2018086122A1/en active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001224064A (en) * | 2000-02-07 | 2001-08-17 | Nippon Telegraph & Telephone East Corp | Portable information terminal and network service system using portable information terminal |
| CN102789233A (en) * | 2012-06-12 | 2012-11-21 | 湖北三江航天红峰控制有限公司 | Vision-based combined navigation robot and navigation method |
| CN103529692A (en) * | 2013-10-30 | 2014-01-22 | 中国航天空气动力技术研究院 | Fault reconstruction method for simple redundancy flight control system of long-endurance unmanned aerial vehicle |
| CN105814875A (en) * | 2013-12-06 | 2016-07-27 | 谷歌公司 | Selecting camera pairs for stereoscopic imaging |
| CN105022397A (en) * | 2014-04-25 | 2015-11-04 | 谷歌公司 | Methods and systems for object detection using multiple sensors |
| CN105700550A (en) * | 2016-01-26 | 2016-06-22 | 深圳市大疆创新科技有限公司 | Unmanned plane and flight control method and system therefor |
Cited By (39)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109682371A (en) * | 2017-10-18 | 2019-04-26 | 苏州宝时得电动工具有限公司 | Automatic running device and its localization method and device |
| CN108106614A (en) * | 2017-12-22 | 2018-06-01 | 北京轻威科技有限责任公司 | A Data Fusion Algorithm for Inertial Sensor and Visual Sensor |
| CN110313170A (en) * | 2018-01-19 | 2019-10-08 | 深圳市大疆创新科技有限公司 | Vision mould group integration module and unmanned plane |
| US11115642B2 (en) | 2018-01-19 | 2021-09-07 | SZ DJI Technology Co., Ltd. | Integrated vision module component and unmanned aerial vehicle |
| CN110313170B (en) * | 2018-01-19 | 2021-08-31 | 深圳市大疆创新科技有限公司 | Visual module integrated module and unmanned aerial vehicle |
| WO2019140649A1 (en) * | 2018-01-19 | 2019-07-25 | 深圳市大疆创新科技有限公司 | Visual module integration module and unmanned aerial vehicle |
| CN108616838A (en) * | 2018-04-29 | 2018-10-02 | 山东省计算中心（国家超级计算济南中心） | Agricultural greenhouse Data Fusion method based on Kalman filtering algorithm |
| CN108646739A (en) * | 2018-05-14 | 2018-10-12 | 北京智行者科技有限公司 | A kind of sensor information fusion method |
| CN108563235A (en) * | 2018-05-24 | 2018-09-21 | 南方科技大学 | Multi-rotor unmanned aerial vehicle, method, device and equipment for grabbing target object |
| CN108733076A (en) * | 2018-05-24 | 2018-11-02 | 南方科技大学 | Method and device for grabbing target object by unmanned aerial vehicle and electronic equipment |
| CN108563235B (en) * | 2018-05-24 | 2022-02-08 | 南方科技大学 | A multi-rotor unmanned aerial vehicle, method, device and equipment for grabbing a target object |
| CN110667598A (en) * | 2018-06-18 | 2020-01-10 | 百度（美国）有限责任公司 | Method for handling sensor faults in autonomous vehicles |
| CN110667598B (en) * | 2018-06-18 | 2023-03-24 | 百度（美国）有限责任公司 | Method for handling sensor faults in autonomous vehicles |
| CN109073390A (en) * | 2018-07-23 | 2018-12-21 | 深圳前海达闼云端智能科技有限公司 | A kind of localization method and device, electronic equipment and readable storage medium storing program for executing |
| CN109073390B (en) * | 2018-07-23 | 2022-10-04 | 达闼机器人股份有限公司 | Positioning method and device, electronic equipment and readable storage medium |
| CN108961716A (en) * | 2018-07-25 | 2018-12-07 | 安徽江淮汽车集团股份有限公司 | A kind of wireless kilowatt meter reading-out system based on ZigBee |
| CN110967064A (en) * | 2018-09-28 | 2020-04-07 | 罗伯特·博世有限公司 | Method, device and sensor system for environmental detection of a vehicle |
| CN110967064B (en) * | 2018-09-28 | 2023-09-05 | 罗伯特·博世有限公司 | Method, device and sensor system for environmental detection of a vehicle |
| CN109781163B (en) * | 2018-12-18 | 2021-08-03 | 北京百度网讯科技有限公司 | Calibration parameter validity checking method, device, equipment and storage medium |
| CN109781163A (en) * | 2018-12-18 | 2019-05-21 | 北京百度网讯科技有限公司 | Calibrating parameters validity check method, apparatus, equipment and storage medium |
| CN109737981A (en) * | 2019-01-11 | 2019-05-10 | 西安电子科技大学 | Target search device and method for unmanned vehicle based on multi-sensor |
| CN110146055A (en) * | 2019-05-21 | 2019-08-20 | 深圳市道通智能航空技术有限公司 | A kind of the super voice abnormality detection method, device and electronic equipment |
| CN110174105A (en) * | 2019-06-14 | 2019-08-27 | 西南科技大学 | Intelligent body Autonomous Navigation Algorithm and system under a kind of complex environment |
| CN110174105B (en) * | 2019-06-14 | 2022-02-11 | 西南科技大学 | Intelligent agent autonomous navigation algorithm and system in complex environment |
| CN112346040A (en) * | 2019-08-09 | 2021-02-09 | 大众汽车股份公司 | Method and device for determining parallax problem in sensor data of two sensors |
| CN112389456A (en) * | 2019-08-15 | 2021-02-23 | 大众汽车股份公司 | Method and device for checking the calibration of an environmental sensor |
| CN110780610B (en) * | 2019-10-30 | 2022-10-18 | 航宇救生装备有限公司 | Method for fusing and processing health management signals of ejection lifesaving system |
| CN111079607A (en) * | 2019-12-05 | 2020-04-28 | 灵动科技（北京）有限公司 | Autonomous Driving System with Tracking |
| CN110926484B (en) * | 2020-02-10 | 2020-05-19 | 中智行科技有限公司 | Vehicle position obtaining method and device and intelligent vehicle |
| CN110926484A (en) * | 2020-02-10 | 2020-03-27 | 中智行科技有限公司 | Vehicle position obtaining method and device and intelligent vehicle |
| WO2021217334A1 (en) * | 2020-04-27 | 2021-11-04 | 深圳市大疆创新科技有限公司 | Moveable platform, and obstacle avoidance method and device therefor |
| CN113597590A (en) * | 2020-04-27 | 2021-11-02 | 深圳市大疆创新科技有限公司 | Movable platform and obstacle avoidance method and device thereof |
| CN113762001A (en) * | 2020-10-10 | 2021-12-07 | 北京京东乾石科技有限公司 | Target detection method and device, electronic equipment and storage medium |
| CN113762001B (en) * | 2020-10-10 | 2024-04-19 | 北京京东乾石科技有限公司 | Target detection method and device, electronic equipment and storage medium |
| CN113436261B (en) * | 2021-06-24 | 2022-04-29 | 湖南大学 | Monocular vision inertial positioning method for automatic driving of closed park |
| CN113436261A (en) * | 2021-06-24 | 2021-09-24 | 湖南大学 | Monocular vision inertial positioning method for automatic driving of closed park |
| CN114485760A (en) * | 2022-01-26 | 2022-05-13 | 震坤行工业超市(上海)有限公司 | Sensor calibration method, electronic device, medium, and system |
| CN114485760B (en) * | 2022-01-26 | 2023-10-31 | 震坤行工业超市(上海)有限公司 | Sensor calibration method, electronic device, medium and system |
| TWI830363B (en) * | 2022-05-19 | 2024-01-21 | 鈺立微電子股份有限公司 | Sensing device for providing three dimensional information |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113223286A (en) | 2021-08-06 |
| CN107223275B (en) | 2021-05-28 |
| WO2018086122A1 (en) | 2018-05-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107223275A (en) | The method and system of multichannel sensing data fusion | |
| US20220124303A1 (en) | Methods and systems for selective sensor fusion | |
| EP3903164B1 (en) | Collision avoidance system, depth imaging system, vehicle, map generator, amd methods thereof | |
| US11032527B2 (en) | Unmanned aerial vehicle surface projection | |
| KR102001728B1 (en) | Method and system for acquiring three dimentional position coordinates in non-control points using stereo camera drone | |
| CN105928498B (en) | Method, the geodetic mapping and survey system, storage medium of information about object are provided | |
| CN107850901B (en) | Sensor fusion using inertial and image sensors | |
| CN107850436B (en) | Sensor fusion using inertial and image sensors | |
| CN105492985B (en) | A system and method for controlling movable objects within an environment | |
| KR101220527B1 (en) | Sensor system, and system and method for preparing environment map using the same | |
| EP3061065B1 (en) | Augmented image display using a camera and a position and orientation sensor unit | |
| McLeod et al. | Using video acquired from an unmanned aerial vehicle (UAV) to measure fracture orientation in an open-pit mine | |
| Saadatseresht et al. | UAV photogrammetry: a practical solution for challenging mapping projects | |
| KR20110082904A (en) | Photographing aerial photographs using model aircraft and GPS and method of producing 3D terrain information | |
| CN106767785B (en) | Navigation method and device of double-loop unmanned aerial vehicle | |
| CN110319772A (en) | Visual large-span distance measurement method based on unmanned aerial vehicle | |
| KR102466007B1 (en) | Multidimensional spatial information generation system and method | |
| JP7153306B2 (en) | Detection target localization system | |
| EP2523062B1 (en) | Time phased imagery for an artificial point of view | |
| CN109564434A (en) | System and method for positioning loose impediment | |
| JP6001914B2 (en) | Target position specifying device, target position specifying system, and target position specifying method | |
| KR101349148B1 (en) | Air shooting unit by using auto shooting system | |
| CN113759365B (en) | Binocular vision three-dimensional optical image and foundation radar data fusion method and system | |
| US10521663B1 (en) | Iterative image position determination | |
| Hosseinpoor et al. | Pricise target geolocation based on integeration of thermal video imagery and rtk GPS in UAVS |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20210528 |