CN118628608A - Panoramic image multi-source information fusion system - Google Patents

Abstract

The present invention discloses a panoramic image multi-source information fusion system, including a scene establishment module and an application module. When establishing a panoramic map, the present invention extracts the SIFT feature vectors of image feature points and establishes a geometric transformation relationship between images, thereby eliminating the discontinuity, mismatch or overlap problems that occur during image splicing, eliminating the splicing seams and ghosting generated thereto, fusing multiple images to obtain a panoramic map, and simultaneously performing multi-source fusion of global text information, picture information, audio information, map information and panoramic information, which can not only display a panoramic map of a region to a user, but also provide audio explanations and promotional film shooting of locations according to regional characteristics, bind audio and video links to locations for users to retrieve and view, ensure the integrity of the information system, provide users with an immersive virtual experience, and provide users with a real scene with a 720-degree panoramic viewing effect.

Description

Panoramic image multi-source information fusion system

Technical Field

The invention relates to the technical field of digital maps, in particular to a panoramic image multi-source information fusion system.

Background Art

Panoramic image information is a virtual reality technology generated based on image processing technology. It stitches images of a place or an object from multiple angles and is a three-dimensional virtual image that can display real scenes.

In the prior art, when a panoramic image is created, information from different sources will have an impact on the system, and the existing multi-source information processing cannot combine sound, animation and video images, resulting in an incomplete information system. At the same time, in the process of panoramic photography and panoramic image production, since the panoramic picture is horizontally rotated and decomposed, it is necessary to stitch the panoramic pictures in different directions, resulting in overlapping stitching during the production of the panoramic picture, which affects the construction effect of the panoramic image.

Summary of the invention

The purpose of the present invention is to provide a panoramic image multi-source information fusion system to solve the problem that the multi-source information cannot be processed in the above-mentioned background technology, resulting in incomplete information, and there are splicing and overlapping when constructing a panoramic image, which affects the panoramic surround effect.

To achieve the above-mentioned object, the present invention provides the following technical solutions: A panoramic image multi-source information fusion system, comprising a scene establishment module and an application module;

The scene building module is used to collect cultural resource information, including text and image information, and build a panoramic digital map;

The scene establishment module includes an information collection module, an information processing module, a panoramic image stitching module, an information annotation module and a multi-source fusion module;

The information collection module is used to collect text and image information of the panoramic map;

The information processing module is used to process the collected text information to obtain text, audio and video information;

The panoramic image stitching module is used to find matching content in two adjacent images, determine the relative positions of the images, stitch the images, and obtain a panoramic image;

The information annotation module is used to generate information annotations of different locations, add travel routes for users, add relevant location names, and plan hot spots;

The multi-source fusion module is used to fuse text information, audio information, map information and panoramic information to generate a complete smart map;

The application module is used to display, retrieve and browse map attractions on the smart map.

Preferably, the information collection module includes a text information extraction module and a picture information extraction module;

The text information extraction module is used to retrieve text information related to the location and collect the text information in a unified manner;

The image information extraction module is used to use camera equipment and a low-altitude drone system to take fisheye images and plan views of points to generate image information.

Preferably, the information processing module includes a commentary writing module, an explanation audio recording module and a commentary video recording module;

The description writing module is used to write a description for the location based on the retrieved location related information;

The explanation audio recording module is used to perform post-dubbing for the written commentary and obtain the location explanation audio:

The commentary video recording module is used to shoot a special promotional video related to the video production location.

Preferably, the panoramic image stitching module includes an image registration module, an image alignment module and an image fusion module;

The image registration module is used to filter and compensate for illumination of the image, extract the SIFT feature vector of each image feature point for stitching, match it with the feature vectors of adjacent images, and establish a geometric transformation relationship between the images based on the positional relationship of the matching points;

The image alignment module is used to map multiple images according to the transformation relationship between the images, eliminate the discontinuity, mismatch or overlap problems that occur when the images are stitched, and form a complete image;

The image fusion module is used to eliminate the stitching seams and ghosting produced after alignment, and uses a hyperbolic fusion algorithm to fuse multiple images.

Preferably, the calculation formula of the SIFT feature vector of the image feature point is as follows:

The direction of the eigenvector is:

θ(x,y)＝arctan[((L(x,y+1)-L(x,y-1))/(L(x+1,y)-L(x-1,y))) ];

The magnitude of the eigenvector is:

Among them, (a, b) represents feature points, N represents the number of feature points, and the SIFT feature vector is an N× matrix.

Preferably, the calculation formula of the hyperbolic fusion algorithm is as follows:

Among them, d _x represents the Euclidean distance from a point on the fused image to a point on the x-th image to be stitched, l _x represents the result obtained after calculating the gradient of the x-th image to be stitched using the Laplace operator, λ _x represents the weight coefficient of the pixel value intensity of the x-th image to be stitched in the overlapping area, U _x represents the area of the image; w _x (i, j) represents the weight of the image in the overlapping area; Ω represents the four neighborhoods of the pixel point: above, below, left, and right.

Preferably, the information annotation module includes a route setting module, a name adding module, a positioning flashing module and a music configuration module;

The route planning module is used to establish the best travel route for a location and provide travel recommendation services;

The name adding module is used to label the point names of the established panoramic image locations for users to select and search for access to the global panoramic image of each point;

The positioning and flashing module is used to locate the user's current position and flash to prompt the user's location;

The music configuration module is used to configure commentary, promotional video links and background music for the full-domain panorama.

Preferably, in the multi-source fusion module, fusing text information, audio information, map information and panoramic information to generate a complete smart map includes the following steps:

S1. Integrate text information, audio information, map information and panoramic information from different sources and convert them into a unified format, extract targets and perform feature discrimination;

S2. Extract feature information from different data sources, analyze and process it, predict the current state based on Kalman filtering, and estimate the prediction error. Dynamically adjust the state estimation and error estimation through Kalman gain, fuse the feature information, and obtain a more comprehensive and rich data representation.

S3. Set n decision makers, n ≥ 1, make target decisions on the fused feature information, integrate the decision results from the decision makers, and generate a complete smart map.

Preferably, the formula of the Kalman filter is as follows:

The formula for predicting the status is:

x' _k =Mx _k-1 +Ny _k-1 ;

The prediction error covariance matrix formula is:

_P'k ＝MPk _- ^1Mγ +Q；

The Kalman gain formula is:

K _k =P' _k H ^T (HP' _k H ^T +A);

The formula for updating the status is:

x _k =x' _k +K _k (z _k -Hx' _k );

The formula for updating the error covariance matrix is:

P _k =(IK _k H)P'_k;

Among them, M represents the state transfer matrix, N represents the control coefficient matrix of external input, y represents the external input vector, Q represents the covariance matrix of process noise, H represents the observation matrix, and A represents the covariance matrix of observation noise.

Preferably, the application module includes a panoramic display module, a classification retrieval module, a scene switching module and a scene explanation module;

The panoramic display module is used to display the panoramic view of the smart map to the user, and all information of the display area can be viewed;

The classification search module is used to set the map search area, and the location can be entered for scene search, and all the information of the search scene can be jumped out;

The scene switching module is used to control scene movement and viewpoint switching, select a viewpoint at will, and zoom in and out on the scene, so that you can look around, look down, and look up as if you were there in person;

The scene explanation module is used to provide voice and video explanations of the scene retrieved by the user, and the user can browse and roam the scene in a panoramic view.

Compared with the prior art, the present invention has the following beneficial effects:

When establishing a panoramic map, the present invention extracts the SIFT feature vectors of image feature points and establishes a geometric transformation relationship between images. This can eliminate the problems of discontinuity, mismatch or overlap in the image stitching process, eliminate the stitching seams and ghosting caused thereto, fuse multiple images to obtain a panoramic map, and simultaneously perform multi-source fusion of global text information, picture information, audio information, map information and panoramic information. This can not only display a panoramic map of a region to users, but also provide audio explanations and promotional videos of locations based on regional characteristics, bind audio and video links to locations for users to retrieve and view, thereby ensuring the integrity of the information system and improving the integrity of smart map construction. This can provide users with an immersive virtual experience and a real scene with a 720-degree panoramic viewing effect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a system block diagram of a panoramic image multi-source information fusion system according to the present invention;

FIG2 is a block diagram of an information acquisition module of a panoramic image multi-source information fusion system according to the present invention;

FIG3 is a block diagram of an information processing module of a panoramic image multi-source information fusion system according to the present invention;

FIG4 is a block diagram of a panoramic image stitching module of a panoramic image multi-source information fusion system according to the present invention;

FIG5 is a block diagram of an information annotation module of a panoramic image multi-source information fusion system according to the present invention;

FIG6 is a histogram of the map completeness rate of the panoramic image multi-source information fusion system of the present invention.

In the figure:

1. Scene establishment module; 11. Information collection module; 111. Text information extraction module; 112. Image information extraction module; 12. Information processing module; 121. Commentary writing module; 122. Commentary audio recording module; 123. Commentary video recording module; 13. Panoramic image stitching module; 131. Image registration module; 132. Image alignment module; 133. Image fusion module; 14. Information annotation module; 141. Route planning module; 142. Name adding module; 143. Positioning flashing module; 144. Music configuration module; 15. Multi-source fusion module; 2. Application module; 21. Panoramic display module; 22. Classification retrieval module; 23. Scene switching module; 24. Scene explanation module.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Example 1

Referring to FIGS. 1-6 , this embodiment provides a panoramic image multi-source information fusion system, including a scene establishment module 1 and an application module 2;

The scene establishment module 1 is used to collect cultural resource information, including text and image information, and establish a panoramic digital map;

The scene establishment module 1 includes an information collection module 11, an information processing module 12, a panoramic image stitching module 13, an information annotation module 14 and a multi-source fusion module 15;

The information collection module 11 is used to collect text and image information of the panoramic map;

The information collection module 11 includes a text information extraction module 111 and a picture information extraction module 112;

The text information extraction module 111 is used to retrieve text information related to the location and collect the text information in a unified manner;

The image information extraction module 112 is used to use camera equipment and a low-altitude drone system to take fisheye and plan views of points to generate image information.

The information processing module 12 is used to process the collected text information to obtain text, audio and video information;

The information processing module 12 includes a commentary writing module 121, an explanation audio recording module 122 and an explanation video recording module 123;

The description writing module 121 is used to write a description for the location based on the retrieved location related information;

The explanation audio recording module 122 is used to perform post-dubbing for the written explanation words and obtain the location explanation audio:

The explanation video recording module 123 is used to shoot a special promotional video related to the video production location.

The panoramic image stitching module 13 is used to find matching content in two adjacent images, determine the relative positions of the images, stitch the images, and obtain a panoramic image;

The panoramic image stitching module 13 includes an image registration module 131, an image alignment module 132 and an image fusion module 133;

The image registration module 131 is used to filter and compensate for illumination of the image, extract the SIFT feature vector of the feature points of each image used for stitching, match it with the feature vectors of adjacent images, and establish a geometric transformation relationship between the images based on the positional relationship of the matching points;

The calculation formula of SIFT feature vector of image feature points is as follows:

The direction of the eigenvector is:

θ(x,y)＝arctan[((L(x,y+1)-L(x,y-1))/(L(x+1,y)-L(x-1,y))) ];

The magnitude of the eigenvector is:

Where (a, b) represents feature points, N represents the number of feature points, and the SIFT feature vector is a matrix of N×128;

The calculation formula of the hyperbolic fusion algorithm is as follows:

Among them, d _x represents the Euclidean distance from a point on the fused image to a point on the x-th image to be stitched, l _x represents the result obtained after calculating the gradient of the x-th image to be stitched using the Laplace operator, λ _x represents the weight coefficient of the pixel value intensity of the x-th image to be stitched in the overlapping area, U _x represents the area of the image; w _x (i, j) represents the weight of the image in the overlapping area; Ω represents the four neighborhoods of the pixel point: above, below, left, and right.

The image alignment module 132 is used to map multiple images according to the transformation relationship between the images, eliminate the discontinuity, mismatch or overlap problems that occur when the images are stitched, and form a complete image;

The image fusion module 133 is used to eliminate the stitching seams and ghosting produced after alignment, and fuses multiple images using a hyperbolic fusion algorithm.

The information annotation module 14 is used to generate information annotations of different locations, add travel routes for users, add relevant location names, and plan hot spots;

The information marking module 14 includes a route making module 141, a name adding module 142, a positioning flashing module 143 and a music configuration module 144;

The route planning module 141 is used to establish the best travel route for a location and provide travel recommendation services;

The name adding module 142 is used to label the location of the established panoramic image with a point name, so that the user can select and search to access the global panoramic image of each point;

The positioning flashing module 143 is used to locate the user's current position and flash to prompt the user's location;

The music configuration module 144 is used to configure commentary, promotional video links and background music for the full-domain panorama;

The multi-source fusion module 15 is used to fuse text information, audio information, map information and panoramic information to generate a complete smart map;

In the multi-source fusion module 15, text information, audio information, map information and panoramic information are integrated to generate a complete smart map, including the following steps:

S1. Integrate text information, audio information, map information and panoramic information from different sources and convert them into a unified format, extract targets and perform feature discrimination;

S2. Extract feature information from different data sources, analyze and process it, predict the current state based on Kalman filtering, and estimate the prediction error. Dynamically adjust the state estimation and error estimation through Kalman gain, fuse the feature information, and obtain a more comprehensive and rich data representation.

S3, set n decision makers, where the value of n is 3, make target decisions on the fused feature information, integrate the decision results from multiple decision makers, and generate a complete smart map;

The formula of Kalman filter is as follows:

The formula for predicting the status is:

x' _k =Mx _k-1 +Ny _k-1 ;

The prediction error covariance matrix formula is:

_P'k ＝MPk _- ^1Mγ +Q；

The Kalman gain formula is:

K _k =P' _k H ^T (HP' _k H ^T +A);

The formula for updating the status is:

x _k =x' _k +K _k (z _k -Hx' _k );

The formula for updating the error covariance matrix is:

P _k =(IK _k H)P'_k;

Among them, M represents the state transfer matrix, N represents the control coefficient matrix of external input, y represents the external input vector, Q represents the covariance matrix of process noise, H represents the observation matrix, and A represents the covariance matrix of observation noise.

Application module 2 is used to display, retrieve and browse map attractions on the smart map;

The application module 2 includes a panoramic display module 21, a classification search module 22, a scene switching module 23 and a scene explanation module 24;

The panoramic display module 21 is used to display the panoramic view of the smart map to the user, and all information of the display area can be viewed;

The classification search module 22 is used to set the map search area, and the location can be entered for scene search, and all the information of the search scene can be jumped out;

The scene switching module 23 is used to control scene movement and viewpoint switching, select a viewpoint at will, and zoom in and out the scene, so that you can look around, look down, and look up as if you were there in person;

The scene explanation module 24 is used to provide voice and video explanations of the scene retrieved by the user, and the user can browse and roam around the scene.

In this embodiment, after collecting the field pictures, the SIFT feature vector of the pictures is obtained, and the multiple images are fused using the hyperbola fusion algorithm to splice them into a panoramic map. At the same time, relevant text, audio and map annotation information are produced, and all text information, audio information, map information and panoramic information are fused. The Kalman filter is used to extract the feature information of different data sources, and the fused feature information is judged by three decision makers to obtain a smart map, and the completeness of the map is judged, so that a complete smart map can be obtained in the shortest time.

Example 2

Referring to FIGS. 1-6 , this embodiment provides a panoramic image multi-source information fusion system, including a scene establishment module 1 and an application module 2;

The scene establishment module 1 is used to collect cultural resource information, including text and image information, and establish a panoramic digital map;

The scene establishment module 1 includes an information collection module 11, an information processing module 12, a panoramic image stitching module 13, an information annotation module 14 and a multi-source fusion module 15;

The information collection module 11 is used to collect text and image information of the panoramic map;

The information collection module 11 includes a text information extraction module 111 and a picture information extraction module 112;

The text information extraction module 111 is used to retrieve text information related to the location and collect the text information in a unified manner;

The image information extraction module 112 is used to use camera equipment and a low-altitude drone system to take fisheye and plan views of points to generate image information.

The information processing module 12 is used to process the collected text information to obtain text, audio and video information;

The information processing module 12 includes a commentary writing module 121, an explanation audio recording module 122 and an explanation video recording module 123;

The description writing module 121 is used to write a description for the location based on the retrieved location related information;

The explanation audio recording module 122 is used to perform post-dubbing for the written explanation words and obtain the location explanation audio:

The explanation video recording module 123 is used to shoot a special promotional video related to the video production location.

The panoramic image stitching module 13 is used to find matching content in two adjacent images, determine the relative positions of the images, stitch the images, and obtain a panoramic image;

The panoramic image stitching module 13 includes an image registration module 131, an image alignment module 132 and an image fusion module 133;

The image registration module 131 is used to filter and compensate for illumination of the image, extract the SIFT feature vector of the feature points of each image used for stitching, match it with the feature vectors of adjacent images, and establish a geometric transformation relationship between the images based on the positional relationship of the matching points;

The calculation formula of SIFT feature vector of image feature points is as follows:

The direction of the eigenvector is:

θ(x,y)＝arctan[((L(x,y+1)-L(x,y-1))/(L(x+1,y)-L(x-1,y))) ];

The magnitude of the eigenvector is:

Where (a, b) represents feature points, N represents the number of feature points, and the SIFT feature vector is a matrix of N×128;

The calculation formula of the hyperbolic fusion algorithm is as follows:

Among them, d _x represents the Euclidean distance from a point on the fused image to a point on the x-th image to be stitched, l _x represents the result obtained after calculating the gradient of the x-th image to be stitched using the Laplace operator, λ _x represents the weight coefficient of the pixel value intensity of the x-th image to be stitched in the overlapping area, U _x represents the area of the image; w _x (i, j) represents the weight of the image in the overlapping area; Ω represents the four neighborhoods of the pixel point: above, below, left, and right.

The image alignment module 132 is used to map multiple images according to the transformation relationship between the images, eliminate the discontinuity, mismatch or overlap problems that occur when the images are stitched, and form a complete image;

The image fusion module 133 is used to eliminate the stitching seams and ghosting produced after alignment, and fuses multiple images using a hyperbolic fusion algorithm.

The information annotation module 14 is used to generate information annotations of different locations, add travel routes for users, add relevant location names, and plan hot spots;

The information marking module 14 includes a route making module 141, a name adding module 142, a positioning flashing module 143 and a music configuration module 144;

The route planning module 141 is used to establish the best travel route for a location and provide travel recommendation services;

The name adding module 142 is used to label the location of the established panoramic image with a point name, so that the user can select and search to access the global panoramic image of each point;

The positioning flashing module 143 is used to locate the user's current position and flash to prompt the user's location;

The music configuration module 144 is used to configure commentary, promotional video links and background music for the full-domain panorama;

The multi-source fusion module 15 is used to fuse text information, audio information, map information and panoramic information to generate a complete smart map;

In the multi-source fusion module 15, text information, audio information, map information and panoramic information are integrated to generate a complete smart map, including the following steps:

Step 1: Integrate text information, audio information, map information and panoramic information from different sources and convert them into a unified format, extract targets and perform feature identification;

Step 2: Extract feature information from different data sources, analyze and process it, predict the current state based on Kalman filtering, and estimate the prediction error. Dynamically adjust the state estimation and error estimation through Kalman gain, fuse the feature information, and obtain a more comprehensive and rich data representation.

Step 3: Set n decision makers, where the value of n is 1, to directly make target decisions based on the fused feature information and generate a complete smart map;

The formula of Kalman filter is as follows:

The formula for predicting the status is:

x' _k =Mx _k-1 +Ny _k-1 ;

The prediction error covariance matrix formula is:

_P'k ＝MPk _- ^1Mγ +Q；

The Kalman gain formula is:

K _k =P' _k H ^T (HP' _k H ^T +A);

The formula for updating the status is:

x _k =x' _k +K _k (z _k -Hx' _k );

The formula for updating the error covariance matrix is:

P _k =(IK _k H)P'_k;

Among them, M represents the state transfer matrix, N represents the control coefficient matrix of external input, y represents the external input vector, Q represents the covariance matrix of process noise, H represents the observation matrix, and A represents the covariance matrix of observation noise.

Application module 2 is used to display, retrieve and browse map attractions on the smart map;

The application module 2 includes a panoramic display module 21, a classification search module 22, a scene switching module 23 and a scene explanation module 24;

The panoramic display module 21 is used to display the panoramic view of the smart map to the user, and all information of the display area can be viewed;

The classification search module 22 is used to set the map search area, and the location can be entered for scene search, and all the information of the search scene can be jumped out;

The scene switching module 23 is used to control scene movement and viewpoint switching, select a viewpoint at will, and zoom in and out the scene, so that you can look around, look down, and look up as if you were there in person;

The scene explanation module 24 is used to provide voice and video explanations of the scene retrieved by the user, and the user can browse and roam around the scene.

In this embodiment, after collecting the field picture, the SIFT feature vector of the picture is obtained, and the multiple images are fused using the hyperbola fusion algorithm to splice them into a panoramic map. At the same time, relevant text, audio and map annotation information are produced, and all text information, audio information, map information and panoramic information are fused. The Kalman filter is used to extract the feature information of different data sources, and the fused feature information is judged by a single decision maker to obtain a smart map and judge the completeness of the map. The error rate of a single decision maker is high, and the completeness of the map is affected.

Example 3

Referring to FIGS. 1-6 , this embodiment provides a panoramic image multi-source information fusion system, including a scene establishment module 1 and an application module 2;

The scene establishment module 1 is used to collect cultural resource information, including text and image information, and establish a panoramic digital map;

The scene establishment module 1 includes an information collection module 11, an information processing module 12, a panoramic image stitching module 13, an information annotation module 14 and a multi-source fusion module 15;

The information collection module 11 is used to collect text and image information of the panoramic map;

The information collection module 11 includes a text information extraction module 111 and a picture information extraction module 112;

The text information extraction module 111 is used to retrieve text information related to the location and collect the text information in a unified manner;

The image information extraction module 112 is used to use camera equipment and a low-altitude drone system to take fisheye and plan views of points to generate image information.

The information processing module 12 is used to process the collected text information to obtain text, audio and video information;

The information processing module 12 includes a commentary writing module 121, an explanation audio recording module 122 and an explanation video recording module 123;

The description writing module 121 is used to write a description for the location based on the retrieved location related information;

The explanation audio recording module 122 is used to perform post-dubbing for the written explanation words and obtain the location explanation audio:

The explanation video recording module 123 is used to shoot a special promotional video related to the video production location.

The panoramic image stitching module 13 is used to find matching content in two adjacent images, determine the relative positions of the images, stitch the images, and obtain a panoramic image;

The panoramic image stitching module 13 includes an image registration module 131, an image alignment module 132 and an image fusion module 133;

The image registration module 131 is used to filter and compensate for illumination of the image, extract the SIFT feature vector of the feature points of each image used for stitching, match it with the feature vectors of adjacent images, and establish a geometric transformation relationship between the images based on the positional relationship of the matching points;

The calculation formula of SIFT feature vector of image feature points is as follows:

The direction of the eigenvector is:

θ(x,y)＝arctan[((L(x,y+1)-L(x,y-1))/(L(x+1,y)-L(x-1,y))) ];

The magnitude of the eigenvector is:

Where (a, b) represents feature points, N represents the number of feature points, and the SIFT feature vector is a matrix of N×128;

The calculation formula of the hyperbolic fusion algorithm is as follows:

Among them, d _x represents the Euclidean distance from a point on the fused image to a point on the x-th image to be stitched, l _x represents the result obtained after calculating the gradient of the x-th image to be stitched using the Laplace operator, λ _x represents the weight coefficient of the pixel value intensity of the x-th image to be stitched in the overlapping area, U _x represents the area of the image; w _x (i, j) represents the weight of the image in the overlapping area; Ω represents the four neighborhoods of the pixel point: above, below, left, and right.

The image alignment module 132 is used to map multiple images according to the transformation relationship between the images, eliminate the discontinuity, mismatch or overlap problems that occur when the images are stitched, and form a complete image;

The image fusion module 133 is used to eliminate the stitching seams and ghosting produced after alignment, and fuses multiple images using a hyperbolic fusion algorithm.

The information annotation module 14 is used to generate information annotations of different locations, add travel routes for users, add relevant location names, and plan hot spots;

The information marking module 14 includes a route making module 141, a name adding module 142, a positioning flashing module 143 and a music configuration module 144;

The route planning module 141 is used to establish the best travel route for a location and provide travel recommendation services;

The name adding module 142 is used to label the location of the established panoramic image with a point name, so that the user can select and search to access the global panoramic image of each point;

The positioning flashing module 143 is used to locate the user's current position and flash to prompt the user's location;

The music configuration module 144 is used to configure commentary, promotional video links and background music for the full-domain panorama;

The multi-source fusion module 15 is used to fuse text information, audio information, map information and panoramic information to generate a complete smart map;

In the multi-source fusion module 15, text information, audio information, map information and panoramic information are integrated to generate a complete smart map, including the following steps:

Step 1: Integrate text information, audio information, map information and panoramic information from different sources and convert them into a unified format, extract targets and perform feature identification;

Step 2: Extract feature information from different data sources, analyze and process it, predict the current state based on Kalman filtering, and estimate the prediction error. Dynamically adjust the state estimation and error estimation through Kalman gain, fuse the feature information, and obtain a more comprehensive and rich data representation.

Step 3: Set n decision makers, where the value of n is 6, to directly make target decisions based on the fused feature information and generate a complete smart map.

The formula of Kalman filter is as follows:

The formula for predicting the status is:

x' _k =Mx _k-1 +Ny _k-1 ;

The prediction error covariance matrix formula is:

_P'k ＝MPk _- ^1Mγ +Q；

The Kalman gain formula is:

K _k =P' _k H ^T (HP' _k H ^T +A);

The formula for updating the status is:

x _k =x' _k +K _k (z _k -Hx' _k );

The formula for updating the error covariance matrix is:

P _k =(IK _k H)P'_k;

Among them, M represents the state transfer matrix, N represents the control coefficient matrix of external input, y represents the external input vector, Q represents the covariance matrix of process noise, H represents the observation matrix, and A represents the covariance matrix of observation noise.

Application module 2 is used to display, retrieve and browse map attractions on the smart map;

The application module 2 includes a panoramic display module 21, a classification search module 22, a scene switching module 23 and a scene explanation module 24;

The panoramic display module 21 is used to display the panoramic view of the smart map to the user, and all information of the display area can be viewed;

The classification search module 22 is used to set the map search area, and the location can be entered for scene search, and all the information of the search scene can be jumped out;

The scene switching module 23 is used to control scene movement and viewpoint switching, select a viewpoint at will, and zoom in and out the scene, so that you can look around, look down, and look up as if you were there in person;

The scene explanation module 24 is used to provide voice and video explanations of the scene retrieved by the user, and the user can browse and roam around the scene.

In this embodiment, after collecting the field picture, the SIFT feature vector of the picture is obtained, and the multiple images are fused using the hyperbola fusion algorithm to splice them into a panoramic map. At the same time, relevant text, audio and map annotation information are produced, and all text information, audio information, map information and panoramic information are fused. The Kalman filter is used to extract the feature information of different data sources, and the fused feature information is judged by six decision makers to obtain a smart map and judge the completeness of the map. The excessive number of decision makers increases the system operation time and affects the efficiency of the judgment.

Comparative Example 1

Please refer to Figures 1-6. This embodiment provides a panoramic image multi-source information fusion system, which is roughly the same as Example 1, with the main difference being that in the image fusion module 133, an Alpha fusion algorithm is used to stitch and fuse images.

Comparative Example 2

Please refer to Figures 1-6, this embodiment provides a panoramic image multi-source information fusion system, which is roughly the same as Example 1, and the main difference is that: in the multi-source fusion module 15, a genetic algorithm is used to generate a smart map.

Test and result analysis:

The panoramic image multi-source information fusion system used in Examples 1-3 is recorded as Example Group 1-3, and the panoramic image multi-source information fusion system used in Comparative Example 1-2 is recorded as Comparative Group 1-2. The stitching success rate, running time and map completeness rate of the panoramic image multi-source information fusion system of Example Group 1-3 and Comparative Group 1-2 are tested, and the relevant test data are recorded in Table 1.

Table 1: Test data record table

Group Splicing success rate (%) Running time(s) Map completeness rate (%) Example 1 Group 100 8.18 100 Example 2 Group 100 8.20 90 Example 3 Group 100 25.37 96 Comparison group 1 85 36.75 80 Comparison of 2 groups 90 80.25 89

It can be seen from Table 1 that the stitching success rate of the panoramic image multi-source information fusion system used by the implementation groups 1-3 is 100%, which is much higher than the stitching success rate of the comparison group 1-2. The running time of the panoramic image multi-source information fusion system used by the implementation groups 1-3 is much shorter than the running time of the comparison group 1-2. The final integrated map completeness rate of the panoramic image multi-source information fusion system used by the implementation groups 1-3 is much higher than the map completeness rate of the comparison group 1-2. Therefore, the panoramic image multi-source information fusion system used in Example 1 has a higher image stitching success rate and an integrated map completeness rate. In Example 2, only the panoramic image multi-source information fusion system used in Example 1 is used. Using a single decision maker to judge the target features may result in misjudgment of the results, which affects the completeness of map fusion. In Example 3, six decision makers are used to judge the target features. Due to the excessive number of decision makers used, the calculation time and storage space increase, and too many judgment results need to be fused, which affects the running time of the system. Example 1 uses three decision makers to judge the target features and integrates the judgment results at the same time, which reduces the misjudgment rate of a single decision maker and improves the reliability of the decision. At the same time, it has no impact on the running time of the system and can well build and generate a complete smart map.

The present invention sets a scene establishment module 1 and an application module 2. The application module 2 is set at the user's mobile phone end. When establishing a panoramic map, the SIFT feature vectors of the image feature points are extracted to establish a geometric transformation relationship between images, so as to eliminate the discontinuity, mismatch or overlap problems in the image stitching process, eliminate the stitching seams and ghosting generated thereafter, fuse multiple images, improve the effect of image fusion, obtain a panoramic map, and simultaneously obtain relevant text information of the retrieved location, write explanation words and explanation audio for the location, and shoot a special promotional video related to the location. After the point is labeled with a name, the global panoramic image of each point is accessed for user search, and the audio and video links are bound to the location. When the user searches for the relevant location in the map, the bound audio and video information can be jumped out, and the hot Flashing signs are set at door points to remind users. By integrating text information, picture information, audio information, map information and panoramic information from all sources, not only can the panoramic map of the area be displayed to users, but audio explanations and promotional videos can be shot for the locations based on regional characteristics. Audio and video links are bound to the locations for users to retrieve and view, ensuring the integrity of the information system. By utilizing the technology of the panoramic image multi-source information fusion system, the geographical landscapes, intangible cultural heritage resources, relics and other cultural tourism resources related to scenic spots and tourist areas can be visualized for an immersive virtual experience, providing users with an immersive virtual experience and a real-life scene with a 720-degree panoramic viewing effect. Users can realize virtual travel without leaving their homes and obtain the same virtual travel effect as the visual cognition obtained from actual personal experience.

Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims (10)

1. A panoramic image multi-source information fusion system, characterized in that it comprises a scene establishment module (1) and an application module (2); The scene establishment module (1) is used to collect cultural resource information, including text and image information, and establish a panoramic digital map; The scene establishment module (1) comprises an information collection module (11), an information processing module (12), a panoramic image stitching module (13), an information annotation module (14) and a multi-source fusion module (15); The information collection module (11) is used to collect text and image information of the panoramic map; The information processing module (12) is used to process the collected text information to obtain text, audio and video information; The panoramic image stitching module (13) is used to search for matching content in two adjacent images, determine the relative positions of the images, stitch the images, and obtain a panoramic image; The information annotation module (14) is used to generate information annotations of different locations, add travel routes for users, add relevant location names, and plan hot spots; The multi-source fusion module (15) is used to fuse text information, audio information, map information and panoramic information to generate a complete smart map; The application module (2) is used to display the smart map, and to retrieve and browse map attractions. 2. The panoramic image multi-source information fusion system according to claim 1, characterized in that: the information acquisition module (11) comprises a text information extraction module (111) and a picture information extraction module (112); The text information extraction module (111) is used to retrieve text information related to the location and collect the text information in a unified manner; The image information extraction module (112) is used to use a camera device and a low-altitude drone system to take fisheye and plan views of the points to generate image information. 3. The panoramic image multi-source information fusion system according to claim 1, characterized in that: the information processing module (12) includes a commentary writing module (121), an explanation audio recording module (122) and a commentary video recording module (123); The commentary writing module (121) is used to write a commentary for the location based on the retrieved location-related information; The explanation audio recording module (122) is used to perform post-dubbing for the written explanation words to obtain the location explanation audio: The explanation video recording module (123) is used to shoot a special promotional video related to the video production location. 4. The panoramic image multi-source information fusion system according to claim 1, characterized in that: the panoramic image stitching module (13) comprises an image registration module (131), an image alignment module (132) and an image fusion module (133); The image registration module (131) is used to filter and compensate for illumination of images, extract SIFT feature vectors of feature points of each image used for splicing, match them with feature vectors of adjacent images, and establish a geometric transformation relationship between images based on the positional relationship of matching points; The image alignment module (132) is used to map multiple images according to the transformation relationship between the obtained images, eliminate the discontinuity, mismatch or overlap problems that occur when the images are spliced, and form a complete image; The image fusion module (133) is used to eliminate the splicing seams and ghosting produced after alignment, and fuses multiple images using a hyperbolic fusion algorithm. 5. The panoramic image multi-source information fusion system according to claim 4, characterized in that: the calculation formula of the SIFT feature vector of the image feature point is as follows: The direction of the eigenvector is: θ(x,y)＝arctan[((L(x,y+1)-L(x,y-1))/(L(x+1,y)-L(x-1,y))) ]; The magnitude of the eigenvector is: Among them, (a, b) represents feature points, N represents the number of feature points, and the SIFT feature vector is an N×128 matrix. 6. The panoramic image multi-source information fusion system according to claim 4, characterized in that: the calculation formula of the hyperbolic fusion algorithm is as follows: Among them, d _x represents the Euclidean distance from a point on the fused image to a point on the x-th image to be stitched, l _x represents the result obtained after calculating the gradient of the x-th image to be stitched using the Laplace operator, λ _x represents the weight coefficient of the pixel value intensity of the x-th image to be stitched in the overlapping area, U _x represents the area of the image; w _x (i, j) represents the weight of the image in the overlapping area; Ω represents the four neighborhoods of the pixel point: above, below, left, and right. 7. The panoramic image multi-source information fusion system according to claim 1, characterized in that: the information annotation module (14) includes a route formulation module (141), a name adding module (142), a positioning flashing module (143) and a music configuration module (144); The route planning module (141) is used to establish the best travel route for a location and provide travel recommendation services; The name adding module (142) is used to label the point names of the established panoramic image locations for users to select and search for access to the global panoramic image of each point; The positioning and flashing module (143) is used to locate the current position of the user and flash to prompt the user where he is; The music configuration module (144) is used to configure commentary, promotional video links and background music for the full-domain panorama. 8. The panoramic image multi-source information fusion system according to claim 1 is characterized in that: in the multi-source fusion module (15), the text information, audio information, map information and panoramic information are fused to generate a complete smart map, including the following steps: S1. Integrate text information, audio information, map information and panoramic information from different sources and convert them into a unified format, extract targets and perform feature discrimination; S2. Extract feature information from different data sources, analyze and process it, predict the current state based on Kalman filtering, and estimate the prediction error. Dynamically adjust the state estimation and error estimation through Kalman gain, fuse the feature information, and obtain a more comprehensive and rich data representation. S3. Set n decision makers, n ≥ 1, make target decisions on the fused feature information, integrate the decision results from the decision makers, and generate a complete smart map. 9. The panoramic image multi-source information fusion system according to claim 8, characterized in that: the formula of the Kalman filter is as follows: The formula for predicting the status is: x' _k =Mx _k-1 +Ny _k-1 ; The prediction error covariance matrix formula is: P _k '＝MP _k-1 M ^γ +Q; The Kalman gain formula is: K _k =P _k 'H ^T (HP _k 'H ^T +A); The formula for updating the status is: x _k =x' _k +K _k (z _k -Hx' _k ); The formula for updating the error covariance matrix is: P _k =(IK _k H)P _k '; Among them, M represents the state transfer matrix, N represents the control coefficient matrix of external input, y represents the external input vector, Q represents the covariance matrix of process noise, H represents the observation matrix, and A represents the covariance matrix of observation noise. 10. The panoramic image multi-source information fusion system according to claim 1, characterized in that: the application module (2) includes a panoramic display module (21), a classification retrieval module (22), a scene switching module (23) and a scene explanation module (24); The panoramic display module (21) is used to display the panoramic view of the smart map to the user, and all information of the display area can be viewed; The classification search module (22) is used to set a map search area, and a location can be input to perform scene search, and all information of the search scene can be jumped out; The scene switching module (23) is used to control scene movement and view angle switching, arbitrarily select a view angle, and zoom in and out the scene, so that the user can look around, look down, and look up as if they were at the scene in person; The scene explanation module (24) is used to provide voice and video explanations for the scene retrieved by the user, and the user can browse and roam in a panoramic view.