CN114630266B

CN114630266B - Multi-mode data fusion indoor positioning system based on neural network

Info

Publication number: CN114630266B
Application number: CN202011466281.1A
Authority: CN
Inventors: 陈彦如; 刘诗佳; 倪振心; 陈良银; 杨彦兵; 王伟; 张媛媛; 郭敏; 胡顺仿; 王浩
Original assignee: Chengdu Yishuqiao Technology Co ltd
Current assignee: Chengdu Yishuqiao Technology Co ltd
Priority date: 2020-12-13
Filing date: 2020-12-13
Publication date: 2024-06-28
Anticipated expiration: 2040-12-13
Also published as: CN114630266A

Abstract

The invention discloses a multi-mode data fusion indoor positioning system based on a neural network. The invention combines the geomagnetic field intensity value, wi-Fi signal intensity value and travelling direction value into the multi-mode fingerprint, and has higher uniqueness in an open space. The invention adopts a Mi-shaped grid to divide the main path in the indoor space, then designs a fixed-length fingerprint dividing method based on sequence extreme points to divide the fingerprint of each main path, takes the extreme points as marks for dividing and aligning the segmented fingerprint sequence, and restores complex and various tracks by a segmentation recombination idea while solving the problem of segmentation fingerprint matching dislocation. In order to more accurately identify the multi-modal fingerprint formed by combining a plurality of segmented fingerprints according to time sequence, the invention provides a multi-modal fingerprint classification model based on a convolution and long-short-term memory network, which can accurately match diversified fingerprint sequences and improve the positioning accuracy of a system in an open space.

Description

Multi-mode data fusion indoor positioning system based on neural network

Technical Field

The invention belongs to the field of indoor positioning, and relates to a multi-mode indoor positioning system based on deep learning and an implementation method thereof.

Background

In recent years, in an outdoor environment, application programs Based on Location-Based services (LBS) by means of a global positioning system (Global Positioning System, GPS) are continuously emerging, penetrating into the aspects of people's clothing and eating. However, in indoor environments, the GPS positioning effect cannot meet the needs of people for indoor location services (Indoor Location-Based Service, ILBS) due to signal blocking and multipath effects. Accordingly, indoor positioning technology has been rapidly developed, and various signals are used in indoor positioning technology, such as Wi-Fi (Wireless-Fidelity), bluetooth (Bluetooth), radio frequency identification technology (Radio Frequency Identification, RFID), infrared, ultra Wide Band (UWB), geomagnetism, visible light, ultrasonic, frequency modulation (Frequency Modulation, FM), inertial navigation system (Inertial Navigation System, INS), image and multi-modal fusion technologies, etc. The indoor positioning scheme needs to consider the deployment and maintenance cost of the scheme while considering the positioning precision. In contrast, the geomagnetic field-based positioning technology has stable signals and high positioning accuracy, does not need to deploy additional equipment, and gradually becomes a focus of research.

The existing geomagnetic positioning scheme is mostly aimed at unidirectional one-dimensional positioning, but in open indoor spaces like airport terminal buildings, stations, shopping centers and the like, the indoor environment range span is large, complex and changeable, the walking track of pedestrians is also changed from one dimension to two dimensions and is difficult to enumerate one by one, and a one-dimensional positioning algorithm is greatly limited in such environments. The existing research has the following defects: 1) The fingerprint acquisition cost is higher: the existing research requires staff to survey at a positioning place, manually collect fingerprint data in a space, mark positions and establish a mapping relation between fingerprints and the positions. 2) The difficulty in establishing the mapping relation between the complex track and the fingerprint sequence is high: the open indoor space is larger in area and more in paths, and because of unpredictability of the walking direction of pedestrians, the tracks become complex and various, and the mapping relation between various fingerprint sequences and corresponding tracks is difficult to establish. 3) Fingerprint sequence matching dislocation problem: the accuracy of positioning can be ensured only by aligning the segmented fingerprints when the segmented fingerprints are matched, the calculation cost is increased by using a sliding window for alignment, and the segmented fingerprints are positioned by using a regression method.

Disclosure of Invention

The invention aims to solve the problems of high fingerprint acquisition cost, high difficulty in establishing a mapping relationship between a complex track and a fingerprint sequence, mismatch of fingerprint sequence matching and the like of the traditional geomagnetic indoor positioning algorithm. The fingerprint classification model is trained through a deep learning method, and a mapping relation between the position coordinate points and geomagnetic fingerprints is established, so that a geomagnetic indoor positioning system with low cost, high efficiency and accuracy is realized.

The core technical thought of the invention is to realize positioning by matching and identifying geomagnetic fingerprints through a deep learning model, and the system comprises two stages of offline training and online positioning. 1) Offline training stage: geomagnetic fingerprint information in a space is collected, a geomagnetic fingerprint classification model is trained after processing, and a mapping relation between magnetic fingerprints and positions is established. 2) On-line positioning: after receiving a positioning request of a user, calculating a positioning result through a fingerprint classification model and a related algorithm, and feeding back the positioning result to the user.

The core technology for solving the problems comprises a multi-modal fingerprint, a segmentation combined fingerprint mapping method and a multi-modal fingerprint classification model.

(1) Multimodal fingerprints. In order to improve the information quantity of the fingerprint, geomagnetism, wi-Fi and direction values are combined into the multi-mode fingerprint. The Wi-Fi is utilized to provide the characteristic of coarser granularity positioning, and is complementary with geomagnetism and directions, so that the problem that geomagnetic fingerprints are similar in different places in an open environment is avoided.

(2) A segmented combined fingerprint mapping method. In order to ensure that the track of the pedestrian and fingerprint information are mapped with each other in a refined way and simultaneously reduce the fingerprint acquisition cost, the invention provides a concept of sectioning and recombination. The invention provides a fixed-length fingerprint segmentation method based on sequence extreme points, which takes the extreme points as marks for segmentation and alignment of segmented fingerprint sequences, improves segmentation granularity of segmented fingerprints and restores complex and diverse tracks while solving the problem of matching dislocation of the segmented fingerprints.

(3) Multimodal fingerprint classification model. In order to more accurately identify multi-modal fingerprints formed by combining a plurality of segmented fingerprints according to time sequence, the invention provides a multi-modal fingerprint classification model based on a Convolutional Neural Network (CNN) and a long-short-term memory network (LSTM). In the process of extracting the fingerprint features, the information can be reduced or blurred no matter the information is maximally pooled or averagely pooled, and the cavity convolution can enlarge the receptive field while the fingerprint information is not lost, so that the feature extraction is carried out on each section of fingerprint in the walking process by using the cavity convolution, the information of a plurality of sectional fingerprints in time sequence is transmitted through LSTM, and the model generated by training can accurately match diversified fingerprint sequences, thereby improving the positioning accuracy in two-dimensional space.

Drawings

FIG. 1 is a block diagram of a multi-modal data fusion indoor positioning system based on convolution and long-short term memory network according to the present invention.

Fig. 2 is a main path division diagram of an open space according to the present invention.

Fig. 3 is a diagram illustrating a fixed-length fingerprint segmentation based on sequence extreme points according to the present invention.

Fig. 4 is a diagram of walking track and corresponding segment fingerprint examples according to the present invention.

Fig. 5 is a diagram of a multi-modal fingerprint classification model in accordance with the present invention.

Detailed Description

The invention is further described below with reference to the accompanying drawings, and the system structure is shown in fig. 1, and mainly comprises four parts of multi-mode data acquisition, multi-mode fingerprint processing, multi-mode fingerprint classification model training and real-time position estimation. It should be noted that the description of these embodiments is only for aiding in understanding the present invention, and is not intended to limit the present invention.

1. Multimodal data acquisition

Because the walking paths in the open space are complex and various, fingerprint sequences of all possible paths are difficult to acquire in a field acquisition mode, and the difficulty of establishing the mapping relation between the complex track and the fingerprint sequences is greatly increased. In order to ensure that the acquired fingerprint data set can cover the whole indoor space, provide good positioning accuracy and not cause too high cost of acquiring data, the invention adopts a Mi-shaped grid to divide the map, and uses eight continuous sectional paths in main directions to approximately represent all paths in the space. The whole map is covered by the Mi-shaped grid, and then the whole map is regarded as a whole, wherein each straight line is a main path. Fingerprint collection is carried out on the main paths, and all positions and paths where pedestrians walk in a two-dimensional space are covered as far as possible.

2. Multimodal fingerprint processing

The invention provides a curve difference matrix, wherein the absolute value is replaced by the difference between sampling points to describe the shape of a curve formed by a fingerprint sequence, so that a neural network can better feel the high-dimensional space characteristics of data. The degree of discretization differentiation between sequences is preserved compared to the normalization process. Converting the segmented fingerprint into a curve difference matrix, and assuming that the segmented geomagnetic fingerprint sequence is S _xyz＝{g₁,g₂,…,g_n }, wherein the curve difference matrix D is shown in formula 1:

After conversion, a two-dimensional image with richer content is formed, different types of curves correspond to different curve difference matrix diagrams, and the image types are clear. The horizontal and vertical coordinates of the matrix represent the coordinates of sampling points of the sequence, and the difference between the sampling points has high dimensional characteristics in space, so that the condition of classifying by using an image processing mode is satisfied, and the geomagnetic fingerprint sequence classification can be realized by adopting a convolutional neural network to perform characteristic extraction.

Through the built-in Wi-Fi module, the mobile phone can not only acquire the name of a nearby Wi-Fi AP (Access Point), but also acquire an MAC (MEDIA ACCESS Control Address) Address and a RSS (Received Signal Strength) value. The MAC address is the unique identification of the Wi-Fi AP, so that the condition that the AP name is renamed can be effectively avoided. RSS values represent signal strength and are negative, with larger values representing less signal attenuation. The RSS value is a main basis for evaluating the distance between the mobile phone and the Wi-Fi AP. The single AP information can be expressed as w= (MAC, RSS), the position of the mobile phone is determined according to the fingerprint information of multiple APs, the fingerprint collected by the pedestrian in the walking process can form a Wi-Fi fingerprint sequence, and finally the position in the actual environment and the Wi-Fi fingerprint sequence are related to form a Wi-Fi fingerprint model. The Wi-Fi fingerprint sequence can thus be expressed as:

Where m represents the total number of Wi-Fi APs that can be scanned and n represents the number of samples of the fingerprint sequence. Wi-Fi is sampled at the same place, the acquired RSS value can have certain fluctuation, and sampling values of different types of mobile phones at the same place can have certain difference. If the collected data is directly used for training the model, the model is difficult to converge due to the fact that the numerical value is large and deviation exists, so that the Wi-Fi RSS value needs to be normalized, as shown in a formula 3:

The dimension is cancelled, the influence caused by fluctuation of data and equipment difference is reduced, and model convergence is accelerated.

The fixed-length geomagnetic fingerprint segmentation method based on the sequence extreme points firstly needs to mark all effective extreme points of a fingerprint sequence before segmentation. The acquired fingerprint sequences have different extreme points in intervals with different sizes, the heights and the widths of waveforms corresponding to the different extreme points are different, and the acquired fingerprint sequences comprise pseudo wave crests (wave troughs) formed by the outside, and the pseudo wave crests cannot be used as mark points for aligning the sequences due to larger variation and poor reliability. In order to reject these spurious peaks and select valid extremal points, the spacing G between peaks and the peak H need to be limited. The fingerprint sequence presents fine fluctuation due to the fact that the magnetic field itself has fine fluctuation and is interfered by external natural or artificial factors. To mitigate the interference of these noises, the fingerprint sequence is first subjected to a sliding average filtering. In addition, in order to ensure that a stable peak and trough are selected, the peak value needs to meet H _max > mu, and the trough value needs to meet H _min < mu, wherein mu is the mean value of geomagnetic fingerprints of a main path. In order to reduce the amount of fingerprint redundancy, inspired by DCGIL algorithm overlap cut, the spacing between the specified peaks should satisfy G > SSN/2. After the parameter conditions are met, all qualified extreme points can be marked by traversing the whole fingerprint sequence. After the screening rule and the segmentation length of the effective extreme points of the geomagnetic fingerprint sequence are determined, the geomagnetic fingerprint sequence can be segmented. The method comprises the core ideas of traversing geomagnetic fingerprint sequences of a section of main path, marking all effective extreme points, traversing every two adjacent extreme points in the travelling direction, dividing the sampling number between the two extreme points by the number of segments between the two points after rounding SSN, sequentially segmenting from the two extreme points to the center, and finally segmenting according to the number of segments which are satisfied by the distance between the head and tail extreme points and the head and tail of the sequence, wherein all segmented sequence segments are marked with position labels. The segmentation method not only ensures the balance of the walking length required by positioning and reduces the positioning cost, but also solves the dislocation problem during sequence matching by describing the path with smaller segments, thereby improving the overall positioning accuracy as shown in figure 3.

After the geomagnetic fingerprint sequences of all main paths in all directions are segmented by the method, all segmented geomagnetic fingerprint sequences in the indoor space are obtained. And as Wi-Fi data acquired on one section of the main path is also a sequence, after sampling frequency conversion, the Wi-Fi fingerprint sequence is segmented with the same length, and Wi-Fi fingerprints of each segment are obtained. After each segmented geomagnetic fingerprint sequence is converted into a corresponding curve difference matrix D, the normalized Wi-Fi fingerprint W corresponding to the segmented geomagnetic fingerprint sequence and the discretized travelling direction value O jointly form a multi-mode segmented fingerprint MSFP= (D, W, O). And after a plurality of adjacent multi-mode segmented fingerprints are spliced in sequence, the multi-mode fingerprint MFP= { MSFP ₁,…,MSFP_n } of a walking path is formed.

In the open indoor space, because the track of the pedestrian is complex and various, the track length is also not fixed, the track fingerprint can be formed by connecting various sectional fingerprints in series, and various fingerprint sequences with different lengths (the types and the number of MSFP (multiple-service provider) forming the MFP (multi-service provider) are required to be prepared as a data set for training the fingerprint classification model. If the sampling cost is high by the actual walking of a person, a Random WayPoint (RWP) model is adopted to automatically simulate the walking process, so that a plurality of possible two-dimensional tracks are generated. As shown in fig. 4, the dashed line in the figure is an example of a walking track simulated by RWP, the line segment (MSFP) is an example of a fingerprint path segment on the walking path, and the crosses are extreme points for alignment.

3. Multimodal fingerprint classification model training

The specific structure of the multi-modal fingerprint classification model of the invention is shown in the following figure 5, and is mainly formed by combining cavity convolution and LSTM, and the input of the model is divided into a curve difference matrix of a segmented geomagnetic fingerprint sequence, a Wi-Fi fingerprint sequence W and a direction sequence O. The method comprises the steps of generating a one-dimensional tensor after the characteristics of a segmented geomagnetic fingerprint sequence are extracted through hole convolution, splicing W and O, sending the tensor together as fusion data of the segmented fingerprint to an LSTM, wherein the input of the LSTM is the multi-mode fusion data tensor of a current segment and the LSTM output value of a last segment, updating the state of a neuron after processing of each gate of the LSTM, and finally outputting the confidence coefficient of the current segmented fingerprint sequence at each position coordinate after passing through a full connection layer (FC). Conv represents a convolution layer, cavity convolution is still selected, and the expansion rate is 1,2, 5, 1 and 2 of saw tooth structures. The LSTM has a time_step of 10, which is complemented with 0 if the input sequence length is insufficient, and a Masking layer is added in front of the LSTM to filter out the time_step complemented with 0 in order to reduce the effect of noise addition on the model. All Conv were followed by one BN, preventing the gradient from disappearing and speeding up training. ReLU is used as the activation function except for the last layer. The activation function of the last layer FC adopts the Softmax function because of the multi-classification task. A Keras deep learning frame is selected as a model frame, tensorFlow is used at the back end, an Adam optimization algorithm is selected by a neural network optimization method, and categorical _ crossentropy is selected by a loss function.

4. Real-time position estimation

When the real-time positioning is carried out in a real scene, in order to ensure that the dislocation problem does not occur in the segmented fingerprint sequence fed into the classification model, the real-time fingerprint sequence needs to be segmented in advance by a fixed-length fingerprint segmentation method based on sequence extreme points, and the position points are predicted after effective segmented fingerprints are screened out to be combined in time sequence. Firstly, carrying out moving average value filtering processing on the geomagnetic fingerprint sequence and the acceleration sequence, and filtering noise in original data. And then mapping the direction change points in the acquired direction sequence to geomagnetic, wi-Fi and acceleration sequences, and dividing the sequences into subsequences of all directions respectively. Sequentially processing fingerprint sequences in different directions, finding out all extreme points meeting the requirements of the distance G and the height H in the current fingerprint sequence, calculating whether the distance from the starting point to the first extreme point is greater than FSL, if so, segmenting the fingerprint with the length of FSL before the first extreme point, converting geomagnetic fingerprints into curve difference matrixes, normalizing Wi-Fi fingerprints, and adding a direction value to obtain the MSFP. And then sequentially taking all the extreme points as basic points, carrying out fingerprint segmentation of the FSL length forwards, stopping segmentation of the current extreme point if the segmentation sequence comprises the next extreme point, and after segmentation, also needing data conversion and processing. The last segment needs to calculate the longitudinal difference ld to the end of the whole sequence. After the circulation is finished, all the obtained MSFPs are spliced into MPFs in time sequence. And finally, sending the MPF into a multi-mode fingerprint classification model, predicting the MPF to obtain confidence degrees of all categories, and calculating a final positioning result of the positioning algorithm through the position and the ld of the highest confidence degree.

The user of the present invention uses the scenario example:

in some large building open indoor spaces, the path of travel is complex, where pedestrians often get lost, requiring real-time acquisition of location-based services, i.e., knowing where themselves or other people or objects are located in the indoor space. In the scene, the method has better effect, and compared with the existing method, the method has higher positioning accuracy and lower equipment deployment cost and positioning walking cost.

Claims

1. A multi-mode data fusion indoor positioning system based on a neural network is characterized in that: the system comprises a mobile terminal and a service terminal; the mobile terminal is divided into a fingerprint acquisition module and a real-time positioning module, wherein the fingerprint acquisition module of the mobile terminal is used for dividing a map, eight continuous sectional paths in the main direction are used for representing all paths in a space approximately, the whole map is covered by the rice-shaped grid and then is regarded as a whole, each straight line is a main path, and fingerprint acquisition is carried out on the main paths; firstly, carrying out sliding average value filtering on a fingerprint sequence, wherein the crest value needs to meet H _max & gtmu, the trough value needs to meet H _min & ltmu, mu is the average value of geomagnetic fingerprints of a main path, the interval between the specified peaks needs to meet G & gtSSN/2, and traversing the whole fingerprint sequence to mark all qualified extremum points; Traversing the geomagnetic fingerprint sequence of a section of main path to mark all effective extremum points, traversing every two adjacent extremum points in the travelling direction, dividing the sampling number between the two extremum points by the number of segments between the two extremum points after rounding SSN, sequentially cutting from the two extremum points to the center, finally cutting according to the number of segments which are satisfied by the distance between the head and tail of the two extremum points and the head and tail of the sequence, and marking all the cut sequence segments with position labels to obtain all the segmented geomagnetic fingerprint sequences in the indoor space; dividing the Wi-Fi fingerprint sequence by the same length to obtain Wi-Fi fingerprints of each segment; After each segmented geomagnetic fingerprint sequence is converted into a corresponding curve difference matrix D, the normalized Wi-Fi fingerprint W corresponding to the segmented geomagnetic fingerprint sequence and the discretized travelling direction value 0 jointly form a multi-mode segmented fingerprint MSFP= (D, W, O); after a plurality of adjacent multi-mode segmented fingerprints are spliced in sequence, a multi-mode fingerprint MFP= { MSFP ₁,…,MSFP_n } of a walking path is formed; the sampling cost is high by the actual walking of a person, and the walking process is automatically simulated by adopting a random road point model, so that a plurality of possible two-dimensional tracks are generated; The multi-mode fingerprint classification model is mainly formed by combining a cavity volume and an LSTM, and the input of the model is divided into a curve difference matrix of a segmented geomagnetic fingerprint sequence, a Wi-Fi fingerprint sequence W and a direction sequence 0; The method comprises the steps of generating one-dimensional tensor after extracting features of a segmented geomagnetic fingerprint sequence by hole convolution, splicing W and O, sending the tensor into LSTM as fusion data of the segmented fingerprint, wherein the input of the LSTM is the multi-mode fusion data tensor of the current segment and the LSTM output value of the last segment, updating the state of neurons after processing by each gate of the LSTM, finally outputting the confidence coefficient of the current segmented fingerprint sequence at each position coordinate after passing through a full connection layer, selecting the hole convolution, selecting the saw tooth structure of 1,2, 5, 1 and 2 for expansion rate, supplementing 0 for the time_step of the LSTM if the length of the input sequence is insufficient, adding a Masking layer in front of the LSTM, filtering out time_step with 0 complement, immediately following all Cony, adopting a ReLU as an activation function except the last layer, adopting a Softmax function as an activation function of the last layer FC, selecting a Keras deep learning frame as a model frame, using TensorFlow at the rear end, selecting an Adam optimization algorithm by a neural network optimization method, and selecting categorical _ crossentropy as a loss function; The real-time fingerprint sequence is segmented in advance by a fixed-length fingerprint segmentation method based on sequence extreme points, effective segmented fingerprints are screened out to be combined in time sequence, then position points are predicted, firstly geomagnetic fingerprint sequences and acceleration sequences are subjected to sliding average value filtering processing, then direction change points in the collected direction sequences are mapped to geomagnetic sequences, wi-Fi sequences and acceleration sequences, the sequences are divided into subsequences in all directions respectively, fingerprint sequences in different directions are sequentially processed, after all extreme points meeting the requirements of spacing G and height H in the current fingerprint sequence are found out, whether the distance from a starting point to the first extreme point is larger than FSL is calculated, if the difference value is larger than the threshold value, the fingerprint with the FSL length before the first extreme point is segmented, geomagnetic fingerprint is converted into a curve difference value matrix, wi-Fi fingerprint is normalized, the curve difference value matrix is added with a direction value to form MSFP, then all extreme points are sequentially taken as base points, the fingerprint with the FSL length is segmented forward, if the segmentation sequence comprises the next extreme point, the segmentation of the current extreme point is stopped, the data conversion and processing are required after the segmentation, the longitudinal difference value ld between the last segment and the end point of the whole sequence is calculated, all MSFP obtained after the circulation is ended are spliced into MPF in time sequence, and finally the MPF is sent into a multi-mode fingerprint classification model, And predicting by the model to obtain confidence degrees of all categories, calculating a final positioning result of the positioning algorithm by the position and ld of the highest confidence degree, and returning the result to the mobile terminal of the user after position correction is finally carried out.