CN115827881A - Multi-mode tourism information positioning type retrieval method based on tourism knowledge map - Google Patents

Abstract

A multi-modal tourism information positioning retrieval method based on tourism knowledge graph, constructs a weighted tourism knowledge graph based on multi-modal data in the mixed database of graphic travel notes and travel videos, and saves it during the construction and update process The entity and the relationship between the entities index the semantic position of the data source. When the user performs a text search, the text extracts the search entity and the relationship between the search entity, and maps it to a subgraph of the knowledge graph. After the subgraph is enhanced and expanded, it is returned according to the corresponding index. Search Results. The results returned by the present invention to the retrieved text are also multi-modal, and point to semantically corresponding positions. For the travel data in the database, return the text and pictures corresponding to the enhanced sub-picture and the travel notes; for the travel video data in the database, return the video segment and the whole video corresponding to the enhanced sub-picture. The invention solves the problems that it is difficult to effectively manage the multi-modal data, and it is difficult to locate the target semantic unit in tourism data retrieval.

Description

A positioning retrieval method for multimodal tourism information based on tourism knowledge graph

technical field

The invention belongs to the field of multimedia computing, relates to semantic analysis of text, pictures and videos, and specifically relates to a multi-modal travel information positioning retrieval method based on a travel knowledge graph.

Background technique

The knowledge map can describe the concepts, entities, events and the relationship between them in the objective world, thus supporting applications such as information retrieval and intelligent question answering. Tourism big data has complex sources, huge volume, and diverse modes, making it difficult to effectively acquire and manage it. Therefore, it is difficult to accurately retrieve multi-modal tourism data. Using multi-modal tourism big data to build a multi-modal knowledge graph can effectively enhance the ability to manage and utilize tourism data.

Most of today's tourism information retrieval applications rely on text search and tag matching, and it is difficult to give accurate retrieval results when the retrieval requirements are complex. Tourism knowledge graphs can support more complex retrieval requirements, but the existing tourism knowledge graphs have a single mode, mostly text data, and few image data and video data. However, with the development of mobile terminals, today's Internet is flooded with massive pictures and video data. People often take a lot of pictures and videos when traveling. Graphic travel notes and travel vlogs have become popular ways to share travel experiences. . In addition, the text-based retrieval method is difficult to locate the semantically corresponding position for unlabeled pictures and videos, and the user still needs a second manual screening and positioning, making retrieval difficult. Therefore, traditional retrieval methods and single-modal tourism knowledge graphs cannot support the positioning retrieval of current multi-modal tourism big data.

Contents of the invention

The problem to be solved by the present invention is: for the retrieval and positioning of multi-modal tourism information, the semantic positioning of the retrieval of multi-modal tourism big data can be realized by constructing a multi-modal tourism knowledge map, and more satisfactory retrieval results can be obtained.

The technical solution of the present invention is: a multi-modal tourism information positioning search method based on the tourism knowledge map, based on the multi-modal tourism database mixed with graphic travel data and travel video data, to construct a tourism knowledge map with weights , and save the semantic position index of the entity and the relationship between entities to the data source during the construction and update process. When the user performs text search, the text is extracted from the search entity and the relationship between the search entity, and mapped to a subgraph of the knowledge map. After the subgraph is enhanced and expanded, the retrieval result is returned according to the corresponding index, and the returned retrieval result is the multimodal data corresponding to the subgraph in the multimodal tourism database.

As an optimal way, constructing a tourism knowledge graph with weights is specifically:

1) Construct an ontology library according to the tourism vertical website, and define entity types, including cities, scenic spots, places, time, activities, and other entities;

2) Obtain multi-modal data from vertical tourism websites and video websites as a multi-modal tourism database, including semi-structured city, scenic spot data and unstructured travel data from vertical tourism websites, and unstructured data from video websites. personalized travel videos;

3) Preprocess the multimodal data, perform word segmentation, part-of-speech analysis and dependency analysis on the text in the travel data, perform object recognition on the pictures in the travel data, perform object tracking recognition and scene text recognition on the video, and Word segmentation, part-of-speech analysis and dependency analysis of scene text;

4) Extract semantic entities from the analyzed text of travel data, the scene text of video recognition, the objects recognized by travel data, and the objects recognized by video tracking, combined with semi-structured data;

5) Mining the relationship between the extracted entities to form a knowledge map, the calculation method of the relationship weight between entities is:

w(h,r,t)=P((r,t)|h),

where w(h, r, t) represents the weight of the relationship (h, r, t) between the head entity h and the tail entity t, and P((r, t)|h) represents the entity relationship in the head entity The probability of occurrence of an event whose relationship is r and tail entity is t under the condition that h appears.

As a preferred method, the retrieval method is as follows:

1) Construct a tourism knowledge map with weights based on multi-modal tourism data;

2) In the process of constructing the tourism knowledge map, the semantic unit location index of the data source corresponding to the entity and entity relationship in the knowledge map is saved, and the entity semantic positioning of the travel text data source is expressed as <document id, chapter id, section id, paragraph id, sentence id, word id>, the entity semantic positioning representation of the travel picture data source is <document id, chapter id, section id, segment id, picture sentence id, bounding box>, the entity semantic positioning representation in the video data source image is <video id, shot id, 0, 0, frame id, bounding box>, and the entity semantic positioning in the text identified by the video data source is expressed as <video id, shot id, 0, 0, sentence id, word id>, The entity relationship semantic positioning of the data source is expressed as <head entity positioning, tail entity positioning>;

3) Extract entities and entity relationships from the input retrieval text;

4) Map the entity and entity relationship obtained in step 3) to the knowledge graph constructed in step 1), and obtain a subgraph thereof;

5) For the subgraph obtained in step 4), expand the associated entity along the entity relationship for each entity according to the extension threshold set, and add the expanded entity and entity relationship into the subgraph to obtain an enhanced subgraph;

6) Return retrieval data according to the source data semantic positions corresponding to the entities and entity relationships in the enhanced subgraph in step 5).

The results returned by the present invention to the retrieved text are also multi-modal, and point to semantically corresponding positions. For the travel data in the database, return the text and pictures corresponding to the enhanced sub-picture and the travel notes; for the travel video data in the database, return the video segment and the entire video corresponding to the enhanced sub-picture.

Further, the present invention realizes the construction of a weighted multimodal tourism knowledge map and the retrieval of text subgraph mapping retrieval. The present invention constructs a weighted multimodal knowledge map for multimodal tourism big data, providing multimodal The location-based retrieval of tourism information provides a solution. After the retrieval subgraph is constructed for the retrieval text, it is mapped to the knowledge graph subgraph. According to the semantic position index of the source data in the knowledge graph subgraph and the entity relationship, the source that meets the retrieval requirements is returned. Data and their corresponding semantic locations.

The present invention first uses technologies such as text analysis, picture object recognition, video scene text recognition and video object tracking to construct a weighted multi-modal tourism knowledge graph. Different from a single text modal tourism knowledge map, this knowledge map can complement and restrict the knowledge extracted from images and videos and the support extracted from text, and provide more abundant and accurate entities and entity relationships. The invention uses the constructed multimodal tourism knowledge map with weights to support the positioning retrieval of multimodal tourism information. It effectively solves the problem that traditional text retrieval and label retrieval cannot support complex semantic retrieval requirements, and solves the problem that a single text modal knowledge graph cannot perform semantic retrieval on pictures and videos. At the same time, the present invention uses a positioning search, which can help users find a search target with more accurate positioning, and does not need to manually search and understand the returned data. Especially for long video data sources, the effect of reducing labor costs is more obvious.

The effective benefit of the present invention is that it provides a solution for multi-modal tourism information positioning search, and enhances the retrieval of complex semantic requirements of multi-modal tourism big data by constructing a weighted multi-modal tourism knowledge map Ability, through the knowledge map to index the semantic position of the data source, it can return more accurate retrieval results, reduce the cost of manual secondary search and understanding, and has good universality and practicability.

Description of drawings

Fig. 1 is a schematic diagram of the retrieval principle of the present invention.

Fig. 2 is the construction process of the multimodal tourism knowledge map of the present invention.

Detailed ways

The present invention proposes a multimodal tourism information positioning retrieval method based on tourism knowledge map, the principle is shown in Figure 1, and multimodal tourism knowledge is constructed according to the multimodal data in the mixed database of graphic travel data and tourism video data Graph, and save the entity and the relationship between the entity and the semantic position index of the data source during the construction and update process. When the user searches for text, the text extracts the search entity and the relationship between the search entity, and maps it to a subgraph of the knowledge graph. After the subgraph is enhanced and expanded, the retrieval result is returned according to the corresponding index. The results returned by the present invention to the retrieved text are also multi-modal, and point to semantically corresponding positions. For the travel data in the database, return the text and pictures corresponding to the enhanced sub-picture and the travel notes; for the travel video data in the database, return the video segment and the entire video corresponding to the enhanced sub-picture.

In the present invention, the construction of weighted multimodal tourism knowledge map and the realization of retrieval text subgraph mapping retrieval include:

1) As shown in Figure 2, construct a tourism knowledge map with weights based on multi-modal tourism data;

1.1) Shot segmentation is performed on the video using the lens segmentation software ShotDetect;

1.2) Get frames every 0.5 seconds for each shot of step 1.1) segmentation, and use the text recognition software PaddleOCR to recognize the frame scene text;

1.3) Deduplicate the text identified in step 1.2) in each shot, and save it in units of shots;

1.4) Use the tracker CenterTrack to perform multi-category and multi-object tracking on the video;

1.5) to step 1.4) the tracking result saves the object category and the object bounding box of each frame;

1.6) Use Mask R-CNN for object recognition on travel pictures;

1.7) to step 1.6) the object bounding box of the category of recognition result preservation object;

1.8) Make sentences for each section of each chapter of the travel notes;

1.9) word segmentation is carried out to step 1.8) sentence result;

1.10) carry out part-of-speech analysis based on step 1.9) part-of-speech result;

1.11) carry out named entity recognition based on step 1.9) participle result;

1.12) carry out dependency syntax analysis based on step 1.9) participle result;

1.13) segment the text of each shot of the video into sentences;

1.14) carry out participle to step 1.13) sentence result;

1.15) carry out part-of-speech analysis based on step 1.13) part-of-speech result;

1.16) carry out named entity recognition based on step 1.13) participle result;

1.17) Based on the result of step 1.13) word segmentation, the dependency syntax analysis is carried out.

1.18) Construct a mapping relationship between cities and corresponding scenic spots;

1.19) According to the order of the travel notes, the pictures and the sentences of the travel notes are taken as semantic units;

1.20) from step 1.19) in each sentence, part of speech is the named entity of the place name, selects the named entity that can correspond to the city scenic spot mapping and extracts as city entity and scenic spot entity, and is recorded as the nearest city or the nearest scenic spot;

1.21) from step 1.19) in each sentence, the part of speech is selected as the named entity of the place name, and the named entity corresponding to the urban scenic spot mapping is selected to be extracted as the location entity;

1.22) from step 1.19) in each sentence, select the near time word combination to extract as time entity;

1.23) from step 1.19) in each sentence, select verbs to be extracted as active entities;

1.24) From step 1.19) in each sentence, select non-location nouns and objects in pictures that have dependencies with verbs and prepositions and extract them as other entities;

1.25) According to the time sequence of the video shots, the video shots and the text recognition sentences of the video are used as semantic units;

1.26) from step 1.25) in each sentence, part of speech is the named entity of the place name, selects the named entity that can correspond to the urban scenic spot mapping and extracts it as a city entity and a scenic spot entity, and records it as the nearest city or the nearest scenic spot;

1.27) from step 1.25) the part of speech in each sentence is the named entity of the place name, and the named entity that fails to map corresponding to the city scenic spot is selected as the place entity;

1.28) from step 1.25) in each sentence, select the near time word combination to extract as time entity;

1.29) from step 1.25) in each sentence, select verbs to be extracted as active entities;

1.30) From each sentence in step 1.25), select non-location nouns that have dependencies with verbs and prepositions and objects tracked in the camera lens to be extracted as other entities;

1.31) Calculate the Levin-Stan ratio for entities extracted from step 1.19) to step 1.30) and merge entities of the same category.

1.32) Construct the ownership relationship between the extracted scenic spot entity and the nearest city entity;

1.33) Construct the relationship between the extracted location entity and the nearest city entity;

1.34) Construct the ownership relationship between the extracted location entity and the nearest scenic spot entity;

1.35) The relationship between the extracted activity entity and the nearest attraction entity is constructed;

1.36) Construct the happened-in relationship for the extracted activity entity and location entity according to the dependency relationship;

1.37) Construct the time-of-occurrence relationship for the extracted activity entity and time entity according to the dependency relationship;

1.38) Construct the time-of-occurrence relationship for the extracted activity entity and time entity according to the dependency relationship;

1.39) Constructing a location proximity relationship for the extracted scenic spot entity and location entity according to keywords and dependencies;

1.40) Construct utilization arrival and utilization departure relations for other extracted entities and locations, scenic spots, and city entities according to keywords and dependencies;

1.41) Construct the belonging relationship between other extracted entities according to the dependency relationship or semantic order;

1.42) The calculation method for the relationship weight between entities extracted from step 3.32) to step 3.41) is:

w(h,r,t)=P((r,t)|h),

Where w(h, r, t) represents the weight of the relationship (h, r, t) between the head entity h and the tail entity t, and P((r, t)|h) represents the entity relationship in the head entity The probability of occurrence of an event whose relationship is r and tail entity is t under the condition that h appears.

2) In the process of constructing the tourism knowledge map in step 1), the semantic unit position index of the source data corresponding to the entity and entity relationship in the knowledge map is saved, and the entity semantic positioning of the travel text data source is expressed as <document id, chapter id, section id, Segment id, sentence id, word id>, entity semantic positioning of travel picture data source is expressed as <document id, chapter id, section id, segment id, picture sentence id, bounding box>, entity semantic positioning in video data source image Expressed as <video id, shot id, 0, 0, frame id, bounding box>, entity semantic positioning in the text identified by the video data source is represented as <video id, shot id, 0, 0, sentence id, word id> , the entity relationship semantic location of the data source is expressed as <head entity location, tail entity location>;

3) Extract entities and entity relationships for a given search text:

3.1) Perform text recognition and analysis on the retrieved text;

3.2) extracting semantic entities according to part-of-speech and syntactic dependencies from the analyzed data obtained in step 3.1);

3.3) From the analyzed data obtained in step 3.1), the relationship between entities is extracted according to the syntactic dependency relationship.

4) Map the retrieved text entities and entity relationships obtained in step 3) to a subgraph in the knowledge graph constructed in step 1):

4.1) Construct a retrieval subgraph for the entities and entity relationships in the retrieval text;

4.2) Map the retrieval subgraph constructed in step 4.1) to a retrieval subgraph of the weighted tourism knowledge graph constructed in step 1).

5) For the mapped retrieval subgraph obtained in step 4), extend each entity along the entity relationship to the associated entity according to the threshold, and add the expanded entity and entity relationship to the subgraph to obtain an enhanced subgraph:

5.1) For the edge entity h, the subgraph will not be extended for the time being. For the edge entity relationship (h, r, t), the entity t is used as the edge entity. Now the subgraph only has edge entities but not edge entity relationships;

如果存在非子图关系

关系且

大于等于阈值α，将非子图实体

和非子图关系

添加为扩展子图实体

和扩展子图实体关系

同样地，如果存在非子图关系

关系且

大于等于阈值α，将非子图实体

和非子图关系

关添加为扩展子图实体

和扩展子图实体关系

5.2) For the subgraph obtained in step 5.1), for the edge entity h and the non-subgraph entity

If there is a non-subgraph relationship

relationship and

Greater than or equal to the threshold α, the non-subgraph entity

and non-subgraph relations

Added as an extended subgraph entity

and Extended Subgraph Entity Relationships

Likewise, if there is a non-subgraph relation

relationship and

Greater than or equal to the threshold α, the non-subgraph entity

and non-subgraph relations

Close Add as Extended Subgraph Entity

and Extended Subgraph Entity Relationships

如果存在非子图关系

且

大于等于阈值α，将非子图实体

和非子图关系

添加为扩展子图实体

和扩展子图实体关系

同样地，如果存在非子图关系

且

大于等于阈值α，将非子图实体

和非子图关系

添加为扩展子图实体

和扩展子图实体关系

由此类推直到对非子图实体

的权值乘积均小于阈值α，则对于原边缘实体h的扩展结束；5.3) For the extended subgraph obtained in step 5.2), for the original edge entity h, and the non-subgraph entity

If there is a non-subgraph relationship

and

Greater than or equal to the threshold α, the non-subgraph entity

and non-subgraph relations

Added as an extended subgraph entity

and Extended Subgraph Entity Relationships

Likewise, if there is a non-subgraph relation

and

Greater than or equal to the threshold α, the non-subgraph entity

and non-subgraph relations

Added as an extended subgraph entity

and Extended Subgraph Entity Relationships

And so on until for non-subgraph entities

The weight products of all are less than the threshold α, then the expansion of the original edge entity h ends;

5.4) When all edge entity expansion ends, the subgraph expansion ends.

6) Enhancing the entity and entity relationship in the sub-graph according to step 5), correspondingly querying the semantic position of the source data in the multimodal tourism database, and returning the retrieved data, specifically:

6.1) To enhance the entity and entity relationship in the subgraph, obtain the source data mapping index;

6.2) Query the multimodal tourism database according to the mapping index. For the travel data in the database, return the text and pictures corresponding to the index and the travel notes; for the travel video data in the database, return the video clips and the entire video corresponding to the index.

Claims (14)

1. A multi-mode tourism information positioning type retrieval method based on a tourism knowledge map is characterized in that a multi-mode tourism database with a weight is built based on a mixed image-text travel note data and tourism video data, semantic position indexes of entities and the relationships among the entities to a data source are stored in the building and updating processes, when a user conducts text search, the relationships between a search entity and the search entity are extracted from a text and are mapped to a sub-graph of the knowledge map, a retrieval result is returned according to the corresponding indexes after the sub-graph is enhanced and expanded, and the returned retrieval result is multi-mode data corresponding to the sub-graph in the multi-mode tourism database.

2. The multi-modal tourist information positioning type retrieval method based on the tourist knowledge map as claimed in claim 1, wherein the construction of the tourist knowledge map with the weight specifically comprises:

1) Constructing an ontology base according to the tourism vertical website, and defining entity types including cities, scenic spots, places, time, activities and other entities;

2) Obtaining multi-mode data from a travel vertical website and a video website as a multi-mode travel database, wherein the multi-mode data comprises semi-structured city and scenic spot data and unstructured travel note data obtained from the travel vertical website, and unstructured travel videos obtained from the video website;

3) Preprocessing multi-modal data, performing word segmentation, part-of-speech analysis and dependency relationship analysis on texts in the travel record data, performing object identification on pictures in the travel record data, performing object tracking identification and scene character identification on videos, and performing word segmentation, part-of-speech analysis and dependency relationship analysis on scene characters;

4) Extracting semantic entities from analyzed texts of travel note data, scene text recognized by videos, objects recognized by the travel note data and objects recognized by video tracking by combining with semi-structured data;

5) The relation between the entities extracted is mined to form a knowledge graph, and the calculation method of the relation weight between the entities comprises the following steps:

w(h,r,t)＝P((r,t)|h)，

where w (h, r, t) represents a weight of a relationship (h, r, t) between the head entity h and the tail entity t, and P ((r, t) | h) represents a probability that an event in which the relationship is r and the tail entity is t occurs under the condition that the head entity h occurs.

3. The multi-modal travel information positioning type retrieval method based on the travel knowledge graph as claimed in claim 2, wherein the picture object recognition is carried out on the travel record data as follows:

1) Performing object recognition on the travel record picture by using Mask R-CNN;

2) And saving the object type and the object surrounding frame for the object identification result.

4. The multi-modal travel information positioning type retrieval method based on the travel knowledge graph as claimed in claim 2, wherein the acquisition of scene text of the video specifically comprises:

1) Shot segmentation is carried out on the video by using shotDetect;

2) Framing every 0.5 second for each divided shot, and recognizing a frame scene text by using PaddleOCR;

3) And removing the duplication of the scene text identified in each shot, and storing the scene text by taking the shot as a unit to obtain the scene text.

5. The multi-modal tourist information positioning type retrieval method based on the tourist knowledge map as claimed in claim 2, wherein the object recognition and tracking of the video specifically comprises:

1) Performing multi-class multi-object tracking on the video by using a CenterTrack;

2) And saving the object type and the object enclosing frame of each frame for the tracking result.

6. The multi-modal travel information location-based retrieval method based on the travel knowledge graph as claimed in claim 2, wherein the word segmentation, part of speech analysis and dependency relationship analysis are specifically as follows:

1) Dividing sentences of each section of text of each chapter of the travel note text or dividing sentences of the scene character text;

2) Performing word segmentation on the sentence segmentation result;

3) Performing part-of-speech analysis based on the word segmentation result;

4) Carrying out named entity recognition based on the word segmentation result;

5) And performing dependency syntax analysis based on the word segmentation result.

7. The multi-modal travel information positioning type retrieval method based on the travel knowledge graph as claimed in claim 2, wherein in the entity extraction, the semantic entity extraction in the travel record data is specifically as follows:

1) Constructing a mapping relation between the obtained city and the corresponding scenic spot according to the semi-structured data;

2) The sentence of the picture and the text is taken as a semantic unit according to the line text sequence of the travel note data;

3) Selecting named entities which can be mapped and corresponding to cities and scenic spots from the named entities with the part of speech as the place name in each sentence of the travel note data text, extracting the named entities into city entities and scenic spot entities, and recording the city entities and the scenic spot entities as the nearest cities or the nearest scenic spots;

4) Selecting named entities which cannot be mapped with urban scenic spots from the named entities with the part of speech as the place name in each sentence of the travel note data text, and extracting the named entities as place entities;

5) Selecting a word combination of the adjacent time from each sentence of the travel memory data text and extracting the word combination as a time entity;

6) Selecting verbs from each sentence of the travel note data text and extracting the verbs as active entities;

7) Selecting non-place nouns which have dependency relationship with verbs and prepositions from each sentence of the travel-memory data text and objects identified in the travel-memory data picture, and extracting the non-place nouns and the objects as other entities;

8) Calculating the Levensan distance of the entities extracted from the step 3) to the step 7), and combining the similar entities according to a set threshold value.

8. The multi-modal travel information positioning type retrieval method based on the travel knowledge graph as claimed in claim 2, wherein in the entity extraction, the semantic entity extraction in the video is specifically as follows:

1) Constructing a mapping relation between the obtained city and the corresponding scenic spot according to the semi-structured data;

2) Taking the shots of the video and the recognition sentences in the corresponding shots as semantic units according to the time sequence of the video shots;

3) Selecting a named entity which can be mapped and corresponds to the urban scenic spot from the named entities with the part of speech of each recognition sentence as a place name, extracting the named entities into an urban entity and a scenic spot entity, and recording the urban entity and the scenic spot entity as the nearest city or the nearest scenic spot;

4) Selecting named entities which cannot be mapped and corresponding to urban scenic spots from the named entities of which the part of speech of each recognition sentence is a place name, and extracting the named entities as place entities;

5) Selecting an adjacent time word combination from each recognition sentence and extracting the adjacent time word combination as a time entity;

6) Selecting verbs from each recognition sentence and extracting the verbs as active entities;

7) Selecting non-place nouns having dependency relations with verbs and prepositions and objects tracked in the shot from each recognition sentence, and extracting the non-place nouns and the objects as other entities;

8) Calculating the Levensan distance of the entities extracted from the step 3) to the step 7), and combining the similar entities according to a set threshold value.

9. The knowledge graph construction method based on multi-mode tourism big data according to claim 7 or 8, characterized in that entity relationship mining is heterogeneous relationship mining, and specifically comprises the following steps:

1) Constructing a relationship between the extracted scenery spot entities and the nearest city entity;

2) Constructing a relationship between the extracted place entity and the nearest city entity;

3) Constructing a relationship between the extracted location entity and the nearest scenery spot entity;

4) Establishing occurrence relationship between the extracted activity entity and the nearest scenery spot entity;

5) Constructing an occurrence relation between the extracted active entities and the extracted place entities according to the dependency relation;

6) Constructing a time-of-occurrence relation between the extracted active entities and the time entities according to the dependency relation;

7) Constructing a time-of-occurrence relation between the extracted active entities and the time entities according to the dependency relation;

8) Constructing a position proximity relation for the extracted scenery spot entities and the extracted place entities according to the keywords and the dependency relations;

9) Constructing and utilizing arrival and departure relations of other extracted entities, places, scenic spots and city entities according to the keywords and the dependency relations;

10 Based on dependency relationships or semantic order, constructs relationships between other extracted entities.

10. The multi-mode travel information positioning type retrieval method based on the travel knowledge graph as claimed in claim 1 or 2, wherein the retrieval method is specifically as follows:

1) Constructing a tourism knowledge map with a weight value based on multi-modal tourism data;

2) In the process of constructing the travel knowledge map, semantic unit position indexes of data sources corresponding to entities and entity relations in the knowledge map are stored, entity semantic positioning of a travel record text data source is expressed as < document id, chapter id, section id, segment id, sentence id and word id >, entity semantic positioning of a travel record picture data source is expressed as < document id, chapter id, section id, segment id, picture statement id and bounding box >, entity semantic positioning in a video data source image is expressed as < video id, lens id,0, frame id and bounding box >, entity semantic positioning in a text identified by the video data source is expressed as < video id, lens id,0, sentence id and word id >, and entity relation semantic positioning of the data source is expressed as < head entity positioning and tail entity positioning >;

3) Extracting entities and entity relations from the input retrieval text;

4) Mapping the entity and entity relationship obtained in the step 3) to the knowledge graph constructed in the step 1) to obtain one sub-graph;

5) For the subgraph obtained in the step 4), expanding the associated entities along the entity relationship by each entity according to the set expansion threshold, and adding the expanded entities and the entity relationship into the subgraph to obtain an enhanced subgraph;

6) And returning retrieval data according to the semantic position of the source data corresponding to the entity and the entity relation in the enhanced subgraph in the step 5).

11. The multi-modal travel information location-based retrieval method based on the travel knowledge graph as claimed in claim 10, wherein the retrieval text entity extraction and the entity relationship extraction for the input are specifically as follows:

3.1 Performing text recognition analysis on the retrieval text;

3.2 Extracting semantic entities from the text recognition analysis result according to the part of speech and the syntactic dependency relationship;

3.3 Extract relationships between entities from the textual recognition analysis results based on syntactic dependencies.

12. The multi-mode travel information positioning type retrieval method based on the travel knowledge graph as claimed in claim 10, wherein the step 4) of obtaining knowledge graph retrieval subgraphs through mapping is as follows:

4.1 Constructing a retrieval subgraph for the entities and the entity relations in the retrieval text;

4.2 Mapping the retrieval subgraph constructed in the step 4.1) to one retrieval subgraph of the constructed tourism knowledge map with the weight value;

4.3 ) expanding the relation between the entity and the entity according to the threshold value of the relation weight between the edge entity and other entities by using the knowledge graph retrieval subgraph mapped in the step 4.2).

13. The multi-modal travel information positioning type retrieval method based on the travel knowledge graph as claimed in claim 12, wherein the weight threshold of the pair graph is expanded as follows:

4.3.1 For the edge entity h, the subgraph does not extend temporarily, for the edge entity relationship (h, r, t), the entity t is taken as the edge entity, and the current subgraph only has the edge entity but does not have the edge entity relationship;

4.3.2 For subgraphs obtained in step 4.3.1), for edge entities h and non-subgraph entities

If there is a non-sub graph relationship

Are in relation to each other

Greater than or equal to threshold value alpha, and adding non-sub-graph entity

And non-subgraph relationships

Added as an extended sub-graph entity

And extended subgraph entity relationships

Likewise, if there is a non-sub graph relationship

Are in relation to each other

Greater than or equal to threshold value alpha, and adding non-sub-graph entity

And non-subgraph relationships

Adding as an extended sub-graph entity

And extended subgraph entity relationships

4.3.3 For the extended subgraph obtained in step 4.3.2), for the primary edge entity h, and the non-subgraph entity

If there is a non-sub graph relationship

And is

Greater than or equal to threshold value alpha, and adding non-sub-graph entity

And non-subgraph relationships

Added as an extended sub-graph entity

And extended subgraph entity relationships

Likewise, if there is a non-sub graph relationship

And is provided with

Greater than or equal to threshold value alpha, and adding non-sub-graph entity

And non-subgraph relationships

Added as an extended sub-graph entity

And extended subgraph entity relationships

From this analogy to non-sub-graph entities

If the weighted products are all smaller than the threshold value alpha, the expansion of the original edge entity h is ended;

4.3.4 The sub-graph expansion ends when all edge entity expansion ends.

14. The multi-modal travel information location-based retrieval method based on the travel knowledge graph of claim 10, wherein the returned retrieval data is as follows:

1) Acquiring a source data mapping index for an entity and an entity relation in the subgraph;

2) Inquiring a multi-mode travel database according to the mapping index, and returning texts and pictures corresponding to the index and the travel notes in the database for the travel note data in the database; and returning the video clips corresponding to the indexes and the whole video for the travel video data in the database.

Images