KR102555269B1 - Posture estimation fusion method and system using omnidirectional image sensor and inertial measurement sensor - Google Patents

Abstract

A posture estimation fusion method and system of an omnidirectional image sensor and an inertial measurement sensor are disclosed. A posture estimation method performed by a posture estimation system includes predicting position data of feature points estimated in image information through fusion of an image sensor and an inertial measurement sensor; and adjusting an external parameter for estimating relative attitude information between the image sensor and the inertial measurement sensor based on the positional data of the predicted feature point.

Description

Posture estimation fusion method and system of omnidirectional image sensor and inertial measurement sensor

The following description relates to a technique for estimating a posture using image information.

Conventional image-based driving motion estimation technology (Visual Odometry, Visual SLAM) uses a monocular or stereo camera to obtain corresponding points in the image in space and time, and to obtain a six-degree-of-freedom posture that best describes the geometric consistency between them It is a technology that simultaneously estimates the 3D points of the surrounding static environment through optimization. However, the image-based driving motion estimation technology cannot estimate absolute distance information of the real environment in the case of a monocular camera, and in the case of a stereo camera system, it is difficult for the sensing area to be greater than 180 degrees due to limitations in terms of hardware configuration. Because of this, the posture estimation accuracy of the visual SLAM technology, which is vulnerable to dynamic objects in Yeongsan, is very poor in a congested environment such as a city center.

Driving motion estimation technology using a laser sensor (LiDAR SLAM) is based on an optimization method for estimating the 6-DOF posture that best describes the structural consistency between 3D points obtained using a very high-accuracy distance measurement sensor such as LiDAR. It is a skill. The technology for estimating driving motion using the laser sensor has a limitation in that stability is poor due to physical limitations of sensor hardware design and a sensing method that is affected by interference, and significant performance degradation is observed when estimating fast motion due to a small amount of information obtained per unit time.

It is possible to provide a method and system for tracking feature point position prediction in a fisheye image through inertial measurement sensor fusion.

It is possible to provide a method and system for simultaneously optimizing attitude and environment maps including external parameters between multiple cameras and inertial sensor measurement sensors.

A posture estimation method performed by a posture estimation system includes predicting position data of feature points estimated in image information through fusion of an image sensor and an inertial measurement sensor; and adjusting an external parameter for estimating relative attitude information between the image sensor and the inertial measurement sensor based on the positional data of the predicted feature point.

The predicting may include calculating a relative motion by accumulating inertial measurement sensor values observed between a previous image of the current image and the current image on a time axis based on the current image.

In the predicting step, as the relative motion is calculated, if there is a 3D feature point observed in the previous image, the 3D feature point is re-projected as the calculated relative motion, and the position data of the 3D feature point in the current image is re-projected. It may include predicting.

The predicting step may include predicting position data of a 3D feature point in the current image by applying only the relative rotational movement of the inertial measurement sensor when the 3D feature point does not exist in the previous image as the relative motion is calculated. can include

The predicting may include correcting the location data of the 3D feature points tracked in the current image by using the predicted location data of the 3D feature points as an initial value of the tracker.

The image sensor includes an omnidirectional image sensor composed of a plurality of cameras, and the attitude estimation system includes an omnidirectional image sensor composed of the plurality of cameras and an inertial measurement sensor, and the omnidirectional image sensor composed of the plurality of cameras. An inertial measurement sensor may be located around the image sensor.

The adjusting of the parameter may include measuring an error of an external parameter between an image sensor and an inertial measurement sensor using observation values of an inertial measurement sensor selected to have a reprojection error of less than or equal to a specific value of a reference threshold among the positional data of the predicted feature point. The method may include adjusting an external parameter by repeatedly performing a minimization method and using the adjusted external parameter to predict position data of feature points estimated in the image information.

The adjusting of the parameter may include selecting an observation value of an inertial measurement sensor having a reprojection error of less than twice a reference threshold among the position data of the predicted feature point, and using the selected observation value to obtain motion information and a 3D feature point. , optimizing the external parameters in a manner that minimizes the geometric error of the external parameters between the image sensor and the inertial measurement sensor.

In the step of adjusting the parameters, the optimization process is re-performed by using the selected observation values to minimize geometrical errors of motion information, 3D feature points, and external parameters between the image sensor and the inertial measurement sensor. It may include re-selecting observation values of the inertial measurement sensor that are 1.5 times or less of .

In the adjusting of the parameters, the optimization process is performed again in such a manner as to minimize geometrical errors of motion information, 3D feature points, and external parameters between the image sensor and the inertial measurement sensor using the re-selected observation values, and the reference threshold It may include re-selecting observation values of the following inertial measurement sensors and obtaining adjusted optimized motion information, 3D feature points, and external parameters through the re-selection.

A computer program stored in a non-transitory computer readable recording medium may be included to execute a posture estimation method in the posture estimation system.

The posture estimation system includes a position prediction unit that predicts position data of feature points estimated in image information through fusion of an image sensor and an inertial measurement sensor; and an optimization unit adjusting an external parameter for estimating relative attitude information between the image sensor and the inertial measurement sensor based on the positional data of the predicted feature point.

The position predictor calculates relative motion by accumulating inertial measurement sensor values observed between the previous video of the current video and the current video on the time axis based on the current video, and calculating the relative motion, within the previous video. If there is a 3D feature point observed in , the position data of the 3D feature point in the current image is predicted by re-projecting the 3D feature point with the calculated relative motion, and as the relative motion is calculated, the feature point in the previous image is predicted. When there is no 3D feature point in , the position data of the 3D feature point in the current image is predicted by applying only the relative rotational motion of the inertial measurement sensor, and the position data of the predicted 3D feature point is used as the initial value of the tracker. Position data of 3D feature points tracked in the current image may be corrected.

The optimization unit minimizes the error of external parameters between the image sensor and the inertial measurement sensor by using the observed values of the inertial measurement sensor whose reprojection error is selected to be less than or equal to a specific value of a reference threshold among the positional data of the predicted feature point. As it is repeatedly performed, the external parameter may be adjusted, and the adjusted external parameter may be used to predict position data of feature points estimated in the image information.

The optimizer selects an observation value of an inertial measurement sensor whose reprojection error is less than twice the reference threshold among the position data of the predicted feature point, and uses the selected observation value to obtain motion information, a 3D feature point, and an image sensor. Optimizing external parameters by minimizing geometric errors of external parameters between inertial measurement sensors, and minimizing geometric errors of external parameters between motion information, 3D feature points, image sensors and inertial measurement sensors using the selected observation values As the optimization process is re-executed in this way, the observed values of the inertial measurement sensor that are less than 1.5 times the reference threshold are re-selected, and the re-selected observation values are used to obtain motion information, 3D feature points, image sensors, and inertial measurement sensors. The optimization process is re-progressed in a way that minimizes the geometric error of the external parameters of the liver, the observed values of the inertial measurement sensor below the reference threshold are re-selected, and the optimized motion information, 3D feature points and external parameters adjusted through the re-selection are re-selected. can be obtained.

Even in a situation where the sharpness of the image is degraded due to motion blur, it enables posture estimation with a robust feature point tracking and optimization method, enabling autonomous driving even in fast-speed robots. In addition, new technologies can be developed in connection with various surrounding environment recognition technologies.

Through the convergence of the image sensor and the inertial measurement sensor, it is possible to estimate the exact posture by simultaneously optimizing external parameters whose initial values may be incorrect due to physical factors as well as measuring the position of feature points in the image information.

1 is an example for explaining multiple cameras and an inertial measurement sensor configured in a posture estimation system according to an embodiment.
2 is an example showing qualitative results in an outdoor environment of a posture estimation method according to an embodiment.
3 is an example showing qualitative results in an indoor environment of a posture estimation method according to an embodiment.
4 is a diagram for explaining quantitative results and effects in an indoor environment according to an embodiment.
5 is a diagram for explaining a physical correlation between multiple cameras configured in a posture estimation system and an inertial measurement sensor according to an embodiment.
6 is a block diagram illustrating a configuration of a posture estimation system according to an exemplary embodiment.
7 is a flowchart illustrating a posture estimation method in a posture estimation system according to an exemplary embodiment.
8 is a diagram for explaining a posture estimation operation in a posture estimation system according to an exemplary embodiment.
9 is a diagram for explaining an external parameter optimization operation in a posture estimation system according to an exemplary embodiment.

Hereinafter, an embodiment will be described in detail with reference to the accompanying drawings.

1 is an example for explaining multiple cameras and an inertial measurement sensor configured in a posture estimation system according to an embodiment.

The posture estimation system may include a plurality of cameras and at least one inertial measurement sensor. At this time, the camera may photograph a specific range or all directions, and various types of cameras may be used. An inertial measurement sensor means a device that measures the speed, direction, gravity, and acceleration of a moving object, and means that it operates in a sensor-based manner. The inertial measurement sensor can recognize the motion situation of a pedestrian and a moving object by using an accelerometer, an angular speedometer, a geomagnetic machine, and an altimeter.

For example, four wideFOV fisheye cameras and one inertial measurement sensor can be attached to a rig. The yellow dotted circle in FIG. 1 indicates the position of the inertial measurement sensor. The camera can capture very challenging data sets with images with heavy motion blur. Specifically, data can be captured by using a square shaped rig (0.3 X 0.3 m) with an Xsens Mti-10 inertial measurement sensor and four 220 degree wideFOV fisheye cameras.

Referring to FIG. 5 , it is a diagram for explaining a physical correlation between multiple cameras configured in a posture estimation system and an inertial measurement sensor. Four wideFOV fisheye cameras can be used for motion estimation. A wideFOV fisheye camera can be equipped with a FOV lens with a 220 degree field of view to maximize the overlapping area for stereo matching of the tracked features.

에서 가속도와 가속도를 측정한 값을 나타낸 것이다. 여기서, 센서 노이즈뿐만 아니라 가속도계와 자이로스코프가 변화하는 편향

,

은 센서 정보에 영향을 미친다. 더욱이, 중력 은 움직임을 계산할 때, 효과를 제거하기 위해 원시 가속도계 출력에서 감산되어야 한다. 두 개의 연속적인 키 프레임 사이의 관성측정 센서의 움직임은 모든 측정에서 사전 통합

,

,

및 관성측정 센서의 방향

, 위치

, 속도

의 관점에서 정의될 수 있다.Referring to FIG. 5, the inertial measurement sensor b is at regular time intervals.

It shows the measured value of acceleration and acceleration in . Here, not only sensor noise but also accelerometer and gyroscope shifting bias

,

affects the sensor information. Moreover, gravity should be subtracted from the raw accelerometer output to remove the effect when calculating motion. The movement of the inertial measurement sensor between two successive key frames is pre-integrated in all measurements.

,

,

and the direction of the inertial measurement sensor

, location

, speed

can be defined in terms of

와

는 사전 통합을 명시적으로 재계산하지 않고 편향을 변경하는 효과의 1차 근사치를 설명한다. Here, Jacobian

and

describes a first-order approximation of the effect of changing the bias without explicitly recomputing the pre-integration.

로 변환하고, 여기서 R(r)은 회전자에 대한 3X3 회전 행렬이다.

는 변환의 구성을 나타내는 반면, -1은 역변환이다. 도 5와 같이, 세 개의 좌표계, 월드(w), 바디(b), 카메라(c)가 사용될 수 있다. 필요한 경우, 좌표계는 변환의 왼쪽에 표시되며,

의 경우처럼, 바디(body)에서 월드 좌표계 또는

로부터 강체 변환을 원드 좌표계의 한 지점으로 나타낸다. 실시예에서는 바디 좌표계가 관성측정 센서의 좌표계와 일치한다고 가정하기로 한다. 시간은 오른쪽 아래 첨자를 사용하여 지정될 수 있다. 예를 들면, 시간 t에서 월드 점 X의 카메라 좌표를 다음과 같이 나타낼 수 있다.A rigid transformation T can be parameterized with a rotation vector and a translation vector t in R ³ . Basically, the 3D point X

, where R(r) is the 3X3 rotation matrix for the rotor.

indicates the configuration of the transform, while -1 is the inverse transform. As shown in FIG. 5, three coordinate systems, world (w), body (b), and camera (c) may be used. If necessary, the coordinate system is displayed to the left of the transformation,

As in the case of , in the body, the world coordinate system or

represents the rigid body transformation as a point in the Wand coordinate system. In the embodiment, it is assumed that the coordinate system of the body coincides with the coordinate system of the inertial measurement sensor. Times can be specified using right subscripts. For example, the camera coordinates of the world point X at time t can be expressed as follows.

와 픽셀 좌표 x 사이의 매핑을 결정할 수 있다. 예를 들면, 투영 함수

은 카메라의 고유 파라미터를 나타낸다. WideFOV 카메라 때문에 평면 대신 단위 구가 3차원 점의 광선(ray)을 계산하는데 사용될 수 있다.

이 단위 구에 대한 투영을 나타내도록 한다고 하면,

은 단위 길이 광선 점 X이며 투영된 점의 정규화된 영상 좌표 역할을 한다. 파라미터

는 최적화에 사용되는 Cauchy robust norm이다. 따라서, 최적화 θ의 상태 벡터는 다음과 같이 정의될 수 있다.The intrinsic parameters of the camera are three-dimensional points in the image.

and the pixel coordinate x can be determined. For example, the projection function

represents a unique parameter of the camera. Because of the WideFOV camera, a unit sphere instead of a plane can be used to compute the ray of a 3D point.

Let us represent the projection on this unit sphere,

is the unit length ray point X and serves as the normalized image coordinates of the projected point. parameter

is the Cauchy robust norm used for optimization. Therefore, the state vector of optimization θ can be defined as:

와

는 가속도계와 자이로 편향, X는 관측된 3D 랜드마크의 위치를 나타낸다. Here, T and v represent the body posture and velocity in the world coordinate system,

and

is the accelerometer and gyro deflection, and X is the position of the observed 3D landmark.

2 is an example showing qualitative results in an outdoor environment of a posture estimation method according to an embodiment.

The blue line shows the qualitative results in the outdoor environment of the posture estimation method (ROVINS) proposed in the example, the red line shows the qualitative results in the outdoor environment of the image-based method for comparison, and the black line shows the results in the outdoor environment. It represents the ground truth. It can be confirmed that the result according to the posture estimation method proposed in the embodiment is similar to the actual trajectory.

3 is an example showing qualitative results in an indoor environment of a posture estimation method according to an embodiment.

Referring to FIG. 3 , it can be confirmed that the ROVINS method proposed in the embodiment shows a smooth trajectory estimation like a ground truth regardless of camera movement. You can see that it works in a precise and powerful way even in difficult situations.

4 is a diagram for explaining quantitative results and effects in an indoor environment according to an embodiment.

An evaluation performed on feature tracking will be described. A comparison of average inliers and ATEtrans between an algorithm with feature tracking of an inertial measurement sensor (Example) and an algorithm without feature tracking of an inertial measurement sensor (prior art). According to the results of FIG. 4, the algorithm with feature tracking of the inertial measurement sensor showed better performance than the algorithm without feature tracking of the inertial measurement sensor for both measurements, and the number of inliers that help increase the overall performance of the system was improved. It can be seen that the number has improved.

6 is a block diagram illustrating a configuration of a posture estimation system according to an exemplary embodiment, and FIG. 7 is a flowchart illustrating a posture estimation method in the posture estimation system according to an exemplary embodiment.

The processor of the posture estimation system 100 may include a position predictor 610 and an optimizer 620 . Components of the processor may be representations of different functions performed by the processor according to control instructions provided by program codes stored in the posture estimation system. The processor and components of the processor may control the posture estimation system to perform steps 710 to 720 included in the posture estimation method of FIG. 7 . In this case, the processor and components of the processor may be implemented to execute instructions according to the code of an operating system included in the memory and the code of at least one program.

The processor may load a program code stored in a program file for a posture estimation method into a memory. For example, when a program is executed in the posture estimation system, the processor may control the posture estimation system to load a program code from a program file into a memory under the control of an operating system. At this time, each of the position predictor 610 and the optimizer 620 may be different functional representations of the processor for executing subsequent steps 710 to 720 by executing instructions of corresponding portions of the program code loaded into the memory. can

In step 710, the position predictor 610 may predict position data of feature points estimated in the image information through fusion of the image sensor and the inertial measurement sensor. The position predictor 610 may calculate relative motion by accumulating inertial measurement sensor values observed between the current image and previous images of the current image on the time axis based on the current image. As the position prediction unit 610 calculates the relative motion, if there is a 3D feature point observed in the previous image, the position data of the feature point in the current image can be predicted by re-projecting the 3D feature point with the calculated relative motion. . As the position predictor 610 calculates the relative motion, if the 3D feature point does not exist in the previous image, position data of the 3D feature point in the current image can be predicted by applying only the relative rotational movement of the inertial measurement sensor. . The location predictor 610 may use the predicted location data of the 3D feature point as an initial value of the tracker to correct the location data of the 3D feature point tracked in the current image.

In step 720, the optimizer 720 may adjust an external parameter for estimating relative attitude information between the image sensor and the inertial measurement sensor based on the position data of the predicted feature point. The optimizer 720 minimizes the error of the external parameter between the image sensor and the inertial measurement sensor by using the measured values of the inertial measurement sensor selected to have a reprojection error of less than a specific value of a reference threshold among the positional data of the predicted feature point. It is possible to adjust the external parameter by repeatedly performing and use the adjusted external parameter to predict the location data of the feature point estimated in the image information. The optimizer 720 selects an observation value of an inertial measurement sensor whose reprojection error is less than twice the reference threshold among the position data of the predicted feature point, and uses the selected observation value to obtain motion information, a 3D feature point, and an image sensor. External parameters can be optimized in a way that minimizes the geometric error of external parameters between inertial measurement sensors. The optimizer 720 re-runs the optimization process in such a way as to minimize the geometrical error of motion information, 3D feature points, and external parameters between the image sensor and the inertial measurement sensor using the selected observation values, thereby increasing the value by 1.5 times the reference threshold. Observed values of the following inertial measurement sensors can be re-selected. The optimization unit 720 re-runs the optimization process by minimizing the geometric error of motion information, 3D feature points, and external parameters between the image sensor and the inertial measurement sensor using the re-selected observation values, and the inertia below the reference threshold Observation values of the measurement sensor may be re-selected, and optimized motion information adjusted through the re-selection, 3D feature points, and external parameters may be obtained.

8 is a diagram for explaining a posture estimation operation in a posture estimation system according to an exemplary embodiment.

In the embodiment, it is assumed that intrinsic parameters of the camera and external parameters of the inertial measurement sensor and the camera are calibrated and provided. All cameras can capture video information simultaneously with data from inertial measurement sensors synchronized in time with the camera. First, the raw input image is made into a hybrid projection image, and the motion of the inertial measurement sensor data can be propagated through midpoint pre-integration. Next, features can be detected in the hybrid projection image, and feature tracking can be performed in the inertial measurement sensor. Rotational data from the inertial measurement sensor can be input to the inertial measurement sensor supported feature tracker to predict feature positions in the current frame. Stereo feature matching between views may follow to find feature correspondence between cameras. When the data processing process is completed, it can be confirmed whether the camera and the inertial measurement sensor have been initialized. A vision-only Structure-From-Motion (SFM) may be performed to handle visual inertial alignment if the camera and inertial measurement sensor are not initialized. Then, it can be initialized to access the attitude estimation system using non-linear optimization.

The attitude estimation system can track the position prediction of the feature point in the fisheye image through the fusion of the inertial measurement sensor. In the case of a conventional feature point location tracking algorithm in an image, there is a limitation in that performance is significantly deteriorated when motion blur or the like due to fast movement occurs because only image information is used. Therefore, in the embodiment, the relative attitude information between the continuously observed young mountains is estimated through the sensor value of the inertial measurement sensor, and the position data of the feature points observed in the previous image in the current image is predicted using the estimated relative attitude information as the initial value. An operation for improving the accuracy of feature point location tracking using the above method will be described. The location data of the estimated feature points contributes to performance improvement in the methodology for simultaneous optimization of posture and environment maps.

In detail, the posture estimation system may receive sensor data. In this case, the sensor data may refer to image information obtained from a camera (image sensor) and data obtained from an inertial measurement sensor. The attitude estimation system may estimate attitude information through convergence of image information obtained from a camera and data acquired from an inertial measurement sensor.

The attitude estimation system can perform a hybrid projection process and a pre-integration process. A raw input image is converted into a hybrid projection image, which can be used for feature extraction, tracking, and matching. Utilizing hybrido projection imaging requires minimizing distortion and maximizing feature matching and tracking across views. Directional FAST and rotational BRIFF (ORB) features can be extracted as input data for intra-view tracking and inter-view matching in hybrid projection images. At the same time, the motion of the inertial measurement sensor data can be propagated using a midpoint pre-integration process. The pre-integration process calculates the uncertainty of the relative pose change and the pose covariance matrix in the previous image frame. When one image information is obtained from the camera, 10 pieces of data may be obtained from the inertial measurement sensor. Through this, a rough posture between cameras can be estimated. After both the hybrid projection process and the pre-integration process have been processed, the motion of the pre-integrated inertial measurement sensor can be utilized to improve feature tracking performance, and stereo features can be matched between views.

The posture estimation system may track feature information through prediction. The KLT tracking technique is used to find a point where features between images match. The KLT tracker searches for feature locations for the current image using photo coherence, starting from the feature locations of the previous image. When processing large motion, the KLT tracker can use image parameters where the coarse position is calculated first and then the fine motion is updated. Finding a good initial position in a new frame is important for accurate and fast feature tracking. In an embodiment, the predicted feature position may be calculated using attitude information of the pre-integrated inertial measurement sensor before being initialized at the predicted position in the KLT tracker. Accurate posture and feature depth are needed to predict the exact feature location.

When the posture estimation system can use the 3D feature point, it can predict the feature position by re-projecting the 3D feature point onto the plane of the current image using the motion information of the inertial measurement sensor. At this time, when the feature information is not yet registered, the feature position may be predicted considering only the rotation of the inertial measurement sensor. In addition, feature point matching may be performed not only at different viewpoints but also at the same viewpoint.

In other words, the posture estimation system may calculate relative motion by accumulating inertial measurement sensor values observed between the previous image and the current image on the time axis. The posture estimation system may predict the position of the feature point in the current image by re-projecting the 3D feature point with the calculated relative motion, if there is a 3D feature point observed in the previous image. When the 3D feature point does not exist in the previous image, the posture estimation system may predict the position of the feature point in the current image by applying only the relative rotational motion of the inertial measurement sensor. The posture estimation system may correct the exact position of the tracked feature point in the current image by using the position of the predicted feature point as an initial value of the KLT tracker.

The posture estimation system may perform an initialization process. The initialization process refers to the process of integrating two very different measurements: the direction of gravity and the deflection of the inertial measurement sensor. Since these parameters are initially unknown, they must be bootstrapped with visual and inertial measurements.

The attitude estimation system can simultaneously optimize attitude and environment maps including external parameters between multiple cameras and inertial measurement sensors. By using the geometrical assumption that the feature point in the image is the same as the feature point tracked in the next image, overall attitude information can be obtained and noise of the inertial measurement sensor can be captured.

In the embodiment, since two types of heterogeneous sensors are used, there is a limit in that it is difficult to accurately measure a relative posture using a camera and an inertial measurement sensor. In addition, fluctuations occur due to physical factors such as shaking caused by movement of the posture estimation system, making it difficult to accurately trust the initial value and deteriorating the performance of the entire system. In the embodiment, an optimization process may be performed to secure the stability of the entire system by simultaneously optimizing the relative posture between the camera and the inertial measurement sensor, that is, an external parameter, in addition to movement and environmental maps. There is an LBA (Local Bundle Adjustment) that simultaneously estimates posture information and 3D feature points using existing geometric matching. However, since optimization of external parameters between the multiple cameras and the inertial measurement sensor is not performed at the same time, it is not applicable to the embodiment.

The posture estimation system may select an observed value of an inertial measurement sensor having a reprojection error of less than twice a reference threshold among 3D feature point information in the observed image information. The attitude estimation system can simultaneously optimize motion information (posture, speed, sensor deflection), 3D feature points, and external parameters between the camera and the inertial measurement sensor in a way that minimizes geometric errors using selected observation values. The attitude estimation system proceeds with the optimization process again using the selected observation values, re-selects the observation values that are 1.5 times or less of the reference threshold value, and proceeds with the optimization process using the re-selection observation values. Observations below the threshold can be selected, and the optimized camera posture, 3D feature points, and external parameters can be used as the final output. Also, the final product can be reused in the attitude estimation system.

More specifically, visual SFM and visual inertial alignment will be described. For example, SFM can work well for initialization that depends on visual SFM results. It fully utilizes the omnidirectional multi-view setup to achieve high robustness in dynamic scenes and textureless indoor environments. Since all forward directions are observed, there is little possibility that the initial movement will be regressed. Failure cases can be filtered out by checking the posture normality count (>50). As long as the initial SFM succeeds and the rig moves a sufficient distance, it can be assumed to return a reasonably accurate attitude. After first monitoring that enough movement has been created (15 keyframes in the current setup), visual inertial alignment can be performed. In addition, visual inertial alignment allows measurement scales to be observed directly in an omnidirectional multi-view stereo setup, combining measurements from inertial measurement sensors and cameras without worrying about initial scale estimation and scale updates.

The posture estimation system can measure an optimization-based visual inertial driving record. The trajectory of the posture tracking system can be calculated by continuously estimating the body posture, velocity, and deflection of the inertial measurement sensor at a frame rate. When initialization is complete, the current frame attitude is updated to the pre-integration attitude of the inertial measurement sensor, after which outlier features can be removed based on reprojection errors or tangential errors of unit rays in the ultra-wide FOV setting. After removing the outliers, the state vector θ of the current frame and the key frame of the active local window W can be optimized under the constraints of the visual and inertial sensors. Optimization problems can be calculated using the Ceres solver.

는 가중치 파라미터를 나타낸다.

는 상대적인 3D 랜드마크가 정상치로 확인된 카운트 수에 비례한다. Here, E _vis is the visual constraint, E _imu is the constraint of the inertial measurement sensor,

represents a weight parameter.

is proportional to the number of counts in which the relative 3D landmark is identified as normal.

The visual constraints can be calculated based on the reprojection error of the feature points, given the body posture, extrinsic and map points. Visual constraints are widely used as geometrical error conditions for many feature-based visual-based systems. The reprojection error E _vis can be given as

에 이르는 외적 지점과 시간

에서 바디 자세를 취하고, 월드 좌표에서 i와 j번째 카메라

가 관측한 랜드마크 위치를 사용할 수 있다. jth camera on the body

extrinsic point and time to reach

Take the body pose at , and the i and jth cameras in world coordinates.

can use the observed landmark location.

For the inertial measurement sensor constraints, the relative inertial measurement sensor constraints may be obtained by pre-integrating the inertial side sensor values between two consecutive key frames. The pre-integration error E _imu can be defined as:

는 사전 통합 및 편향 랜덤 워크(random walk)에 대한

의 정보 매트릭스이다. here,

for pre-integration and biased random walks.

is the information matrix of

9 is a diagram for explaining an external parameter optimization operation in a posture estimation system according to an exemplary embodiment.

Referring to FIG. 9 , observation information and parameters are shown. External parameters, attitude information, speed, deflection, etc. can be optimized by simultaneously adjusting the 3D points generated from 2D information, the attitude of the key frame or frame, and the attitude values accumulated from the inertial measurement sensor.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. can be embodied in Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (15)

In the posture estimation method performed by the posture estimation system,
Predicting positional data of feature points estimated in image information through fusion of an image sensor and an inertial measurement sensor; and
Adjusting an external parameter for estimating relative attitude information between the image sensor and the inertial measurement sensor based on the positional data of the predicted feature point
including,
The predicting step is
Based on the current image, relative motion is calculated by accumulating inertial measurement sensor values observed between the current image and the previous image of the current image on the time axis, and as the relative motion is calculated, the three-dimensional data observed in the previous image If there is a feature point, predicting position data of the 3D feature point in the current image by re-projecting the 3D feature point with the calculated relative motion
Posture estimation method comprising a. delete delete According to claim 1,
The predicting step is
Predicting positional data of a 3D feature point in the current image by applying only the relative rotational movement of the inertial measurement sensor when the 3D feature point does not exist in the previous image as the relative motion is calculated.
Posture estimation method comprising a. According to any one of claims 1 or 4,
The predicting step is
Correcting the location data of the 3D feature points tracked in the current image by using the predicted 3D feature point location data as an initial value of the tracker.
Posture estimation method comprising a. According to claim 1,
The image sensor includes an omnidirectional image sensor composed of a plurality of cameras,
The posture estimation system includes an omnidirectional image sensor and an inertial measurement sensor composed of the plurality of cameras, and the inertial measurement sensor is located around the omnidirectional image sensor composed of the plurality of cameras. method. According to claim 1,
Adjusting the parameters,
Among the positional data of the predicted feature point, a method of minimizing the error of external parameters between the image sensor and the inertial measurement sensor is repeatedly performed using the observed values of the inertial measurement sensor whose reprojection error is selected to be less than a specific value of the reference threshold. Adjusting an external parameter according to the image information, and using the adjusted external parameter to predict position data of feature points estimated in the image information.
Posture estimation method comprising a. In the posture estimation method performed by the posture estimation system,
Predicting positional data of feature points estimated in image information through fusion of an image sensor and an inertial measurement sensor; and
Adjusting an external parameter for estimating relative attitude information between the image sensor and the inertial measurement sensor based on the positional data of the predicted feature point
including,
Adjusting the parameters,
Among the positional data of the predicted feature point, a method of minimizing the error of external parameters between the image sensor and the inertial measurement sensor is repeatedly performed using the observed values of the inertial measurement sensor whose reprojection error is selected to be less than a specific value of the reference threshold. Inertial measurement in which an external parameter is adjusted according to the image information, the adjusted external parameter is used to predict position data of feature points estimated in the image information, and a reprojection error among the predicted feature point position data is less than twice the reference threshold. Selecting observation values of sensors and optimizing external parameters by using the selected observation values to minimize geometric errors of motion information, 3D feature points, and external parameters between an image sensor and an inertial measurement sensor.
Posture estimation method comprising a. According to claim 8,
Adjusting the parameters,
By using the selected observation values, the optimization process is re-progressed in such a way as to minimize the geometrical errors of motion information, 3D feature points, and external parameters between the image sensor and the inertial measurement sensor, so that the inertial measurement sensor is less than 1.5 times the reference threshold. step of re-screening observations of
Posture estimation method comprising a. According to claim 9,
Adjusting the parameters,
Using the re-selected observation values, the optimization process is re-progressed in such a way as to minimize the geometrical errors of motion information, 3D feature points, and external parameters between the image sensor and the inertial measurement sensor, and the observed value of the inertial measurement sensor below the reference threshold. re-screening and acquiring optimized motion information, 3D points, and external parameters adjusted through the re-screening
Posture estimation method comprising a. A computer program stored in a non-transitory computer readable recording medium to execute the posture estimation method of any one of claims 1, 4, 6, 7, and 8 in the posture estimation system. In the posture estimation system,
a position prediction unit that predicts position data of feature points estimated in image information through fusion of an image sensor and an inertial measurement sensor; and
An optimization unit for adjusting an external parameter for estimating relative attitude information between the image sensor and the inertial measurement sensor based on the position data of the predicted feature point.
including,
The position prediction unit,
Based on the current image, relative motion is calculated by accumulating inertial measurement sensor values observed between the current image and the previous image of the current image on the time axis, and as the relative motion is calculated, the three-dimensional data observed in the previous image When a feature point exists, the 3D feature point is re-projected with the calculated relative motion to predict the location data of the feature point in the current image, and when the 3D feature point does not exist in the previous image as the relative motion is calculated. , Predicting the location data of 3D feature points in the current image by applying only the relative rotational motion of the inertial measurement sensor, and using the position data of the predicted 3D feature points as the initial value of the tracker to track the 3D feature points in the current image to calibrate the positional data of
Posture estimation system. delete According to claim 12,
The optimization unit,
Among the positional data of the predicted feature point, a method of minimizing the error of external parameters between the image sensor and the inertial measurement sensor is repeatedly performed using the observed values of the inertial measurement sensor whose reprojection error is selected to be less than a specific value of the reference threshold. Adjusting the external parameter according to the image information, and using the adjusted external parameter to predict the position data of the feature point estimated in the image information
Characterized in that the posture estimation system. In the posture estimation system,
a position prediction unit that predicts position data of feature points estimated in image information through fusion of an image sensor and an inertial measurement sensor; and
An optimization unit for adjusting an external parameter for estimating relative attitude information between the image sensor and the inertial measurement sensor based on the position data of the predicted feature point.
including,
The optimization unit,
Among the positional data of the predicted feature point, a method of minimizing the error of external parameters between the image sensor and the inertial measurement sensor is repeatedly performed using the observed values of the inertial measurement sensor whose reprojection error is selected to be less than a specific value of the reference threshold. Inertial measurement in which an external parameter is adjusted according to the image information, the adjusted external parameter is used to predict position data of feature points estimated in the image information, and a reprojection error among the predicted feature point position data is less than twice the reference threshold. Observation values of the sensor are selected, motion information, 3D feature points, and external parameters are optimized in such a manner as to minimize the geometric error of the external parameters between the image sensor and the inertial measurement sensor using the selected observation values, and the selected observations are performed. As the optimization process is re-progressed in a way that minimizes the geometric error of motion information, 3D feature points, and external parameters between the image sensor and the inertial measurement sensor using the value, the observed value of the inertial measurement sensor that is 1.5 times or less of the reference threshold is obtained. Re-selection, and using the re-selected observation values, the optimization process is re-progressed by minimizing the geometric error of motion information, 3D feature points, and external parameters between the image sensor and the inertial measurement sensor, and the inertial measurement below the reference threshold is performed. Obtaining optimized motion information, 3D feature points, and external parameters adjusted through re-selection by re-selecting the observed values of the sensor
Characterized in that the posture estimation system.

Images