CN106484781A - Indonesia's Chinese cross-language retrieval method of fusion association mode and user feedback and system - Google Patents

Abstract

The invention discloses an Indonesian-Chinese cross-language retrieval method and system that integrates association patterns and user feedback, uses a machine translation module to translate Indonesian user queries into Chinese queries and submits them to the search engine module for retrieval to obtain a document set of preliminary inspection results, and utilizes user Click behavior-related feedback information extraction module to obtain the relevant document set of user feedback for initial inspection, and obtain the relevant document database for initial inspection after preprocessing by the document preprocessing module, call the fully weighted association rule mining module to build a fully weighted association rule library, and use cross-language query to expand words The generation module builds an extended thesaurus, and uses the cross-language query extension implementation module to submit the combined new query to the search engine module to obtain the final retrieval result Chinese document, and uses the final result display module to submit the final retrieval result to the machine translation module for translation into Indonesian documents and then returned to the user. The invention effectively improves and improves cross-language retrieval performance, and has good practical application value and popularization prospect.

Description

Indonesian-Chinese cross-language retrieval method and system integrating association model and user feedback

technical field

The invention belongs to the field of text information retrieval, and specifically relates to an Indonesian-Chinese cross-language retrieval method and system that integrates association patterns and user feedback, and is applicable to fields such as cross-language text information retrieval that uses Indonesian language to query and retrieve Chinese documents.

Background technique

Cross-language information retrieval refers to the technology of retrieving information resources in other languages with a query in one language. The Indonesian-Chinese cross-language information retrieval method is to query and retrieve Chinese documents in Indonesian language. The Indonesian language used to express the query is called the source language, and the Chinese language of the retrieved documents is called the target language. With the increasingly close exchanges between China and ASEAN countries, the research on cross-lingual information retrieval methods for ASEAN languages is urgent and important.

Scholars from all over the world have conducted in-depth discussions and research on cross-lingual information retrieval methods and systems from different angles and directions, and have achieved rich results. However, the current problems in cross-lingual information retrieval research have not been completely resolved. One of the problems that have been solved and paid more attention to is the serious query topic drift problem in the process of cross-language information retrieval, which is more serious than single-language retrieval. The problem of word mismatch, these problems often lead to low performance of cross-language retrieval, Not as good as monolingual retrieval performance. In view of the above problems, in recent years, research on cross-lingual information retrieval based on query expansion has received more attention and discussion, and its research mainly focuses on correlation feedback-based (Parton K, GaoJ. Combining Signals for Cross-Lingual Relevance Feedback [C ].Proceedings of 8thAsia Information Retrieval Societies Conference(AIRS 2012),Tianjin,China.Springer-Verlag Berlin Heidelberg2012,LNCS 7675,Information Retrieval Technology.2012:356-365.Lee C J,Croft W B.Cross-Language Informs Pseudo-Relevance Text[C]. Proceedings of 36th European Conference on IR Research (ECIR 2014), Amsterdam, The Netherlands. Advances in Information Retrieval. Springer International Publishing, 2014: 260-272.), latent semantics (Guan Jianting, Su Yidan. Based on latent Cross-language query expansion method based on semantic analysis [J]. Computer Engineering, 2009, 35 (10): 49-53. Ning Jian, Lin Hongfei. Cross-language retrieval based on improved latent semantic analysis [J]. Chinese Journal of Information, 2010, 24(3):105-111.), language model and topic model (Ganguly Debasis and Leveling Johannesand Jones Gareth J.F.Cross-lingual topical relevance models[C].In:24thInternational Conference on Computational Linguistics(COLING 2012),2012 .; WangXuwen, Zhang Qiang, Wang Xiaojie, et al.LDA based pseudo relevance feedback for cross language information retrieval[C].I EEE International Conference on Cloud Computing and Intelligence Systems(CCIS2012).Hangzhou:IEEE,2012:1993-1998.;Xuwen Wang,Qiang Zhang,Xiaojie Wang,et al.Cross-lingual PseudoRelevance Feedback Based on Weak Relevant Topic Alignment.Proceedingsific of the29th Pacific Asia Conference on Language, Information and Computation, PACLIC29, Shanghai, China, 2015:529-534.) and other cross-language information retrieval research, the language objects are mainly English, and most of them are cross-language research on English and other languages. Search questions.

At present, since Nanning, China has been the permanent host of the China-ASEAN Expo, the political, economic, and cultural exchanges between China and ASEAN countries have become more frequent and close, and the research on cross-language information retrieval and cross-language information services for the languages of ASEAN countries has become more urgent , its importance is becoming increasingly prominent.

Contents of the invention

The purpose of the present invention is to solve the above-mentioned problems in the prior art, combine fully weighted association rule mining technology and user-related feedback to Indonesian-Chinese cross-language information retrieval, and provide a Indonesian-Chinese cross-language retrieval that integrates association patterns and user feedback The method and system can improve and improve the performance of Indonesian medium-span language information retrieval, and the Indonesian medium-span language retrieval effect for long queries is better.

In order to realize the above-mentioned purpose of the invention, the present invention has adopted following technical scheme:

An Indonesian-Chinese cross-language retrieval method that integrates association patterns and user feedback, comprising the following steps:

(1) Translate the Indonesian language user query into Chinese query formula through the machine translation module, and submit it to the search engine for preliminary retrieval on the Internet, and obtain the preliminary inspection result document set;

(2) Extract the first r Chinese documents from the cross-language preliminary inspection result document set and submit them to the user;

(3) The user judges the Chinese documents in the cross-language preliminary inspection result document set to obtain the user feedback related document set, and the total number of documents in the document set is set to n;

(4) Preprocessing user feedback related document sets, that is, performing preprocessing operations of Chinese word segmentation, removing stop words, calculating feature word weights and extracting feature words, and constructing a database of relevant documents for initial inspection;

(5) Scan the relevant document database for initial inspection, mine fully weighted feature word 1_candidate item set C ₁ , calculate C ₁ weight w(C ₁ ), and count the maximum weight maxCw _i (!C ₁ of items other than C ₁ ) and C ₁ support count n _c1 , ms is the minimum support threshold, calculate the value of KIWT(1,2), the calculation formula of KIWT(1,2) is: KIWT(1,2)=n×1×ms -n _C1 ×maxCw _i (!C ₁ );

(6) Calculate the support degree FTISup(C ₁ ) of C _1. If FTISup(C ₁ )≧ms, mine 1_frequent itemset L ₁ from 1_candidate item set C ₁ and add it to the fully weighted feature word frequency Itemset set L, the calculation formula of FTISup(C ₁ ) is:

(7) Mining k_itemsets, wherein said k≧2, including steps (7.1) to (7.7):

(7.1) Compare the candidate (k-1)_itemset C _k-1 weight and KIWT(k-1,k) value, and cut off the candidates whose W(C _k-1 )<KIWT(k-1,k) Itemset C _k-1 ;

(7.2) Aproiri connection is performed on the remaining candidate (k-1) item sets C _k-1 to obtain C _k ;

(7.3) When k=2, cut off the candidate 2_itemset not containing the query item;

(7.4) Scan the initial inspection related document database, count the maximum weight maxCw _i (!C _k ) of items other than C _k and the support count n _ck of C _k , and calculate the C _k weight w(C _k ) and KIWT(k -1,k), the calculation formula of KIWT(k-1,k) is: KIWT(k-1,k)=n×k×ms-n _ck ×maxCw _i (!C _k );

(7.5) Cut off the candidate item set C _{k where} n _ck is 0;

(7.6) For the remaining candidate k_itemset C _k , calculate C _k support FTISup(C _k ), if FTISup(C _k )≧ms, mine k_frequent itemset from candidate k_itemset C _k L _k , and added to the fully weighted feature word frequent itemset set L, the calculation formula of FTISup(C _k ) is:

(7.7) If k is greater than the candidate item set length threshold or the candidate k_item set is an empty set, then the mining ends, otherwise, continue the loop steps (7.1) to (7.6);

(8) Mining the fully weighted association rules of the feature words containing the query term from the fully weighted feature word frequent itemset set L, and constructing the fully weighted association rule base;

(9) Extract the cross-lingual expansion words related to the original query from the fully weighted association rule base, and construct the extended thesaurus;

(10) Submitting the original query and the expanded word combination to the search engine to retrieve the final retrieval result Chinese document again;

(11) Submit the Chinese document of the final retrieval result to the machine translation module for translation into a Indonesian language document, and finally return the Chinese document of the final retrieval result and the Indonesian language document of the final retrieval result to the user.

The calculation of the feature word weight described in the above-mentioned steps (4) adopts the tf-idf method, and its calculation formula is: Among them, tf _m,n represents the number of occurrences of the feature word t _m in the document d _n , df _m represents the number of documents containing the feature word t _m , and N represents the total number of documents in the document collection.

The method of above-mentioned step (8) comprises steps (8.1) to (8.4):

(8.1) Extract a certain fully weighted i_frequent itemset tlL _i from the fully weighted feature word frequent itemset set L, and find out all proper subsets of tlL _i ;

(8.2) Take out two proper subsets tlI ₁ and tlI ₂ arbitrarily from the proper subset set of tlL _i , when And tlI ₁ ∪tlI ₂ ＝L _i , if FTARConf(tlI ₁ →tlI ₂ )≧mc, fully weighted feature word strong association rules tlI ₁ →tlI ₂ are mined; if FTARConf(tlI ₂ →tlI ₁ )≧mc, Then dig out the fully weighted feature word strong association rule tlI ₂ →tlI ₁ ; the mc is the minimum confidence threshold, tlI ₁ and tlI ₂ are fully weighted feature word frequent itemsets, which are proper subset itemsets of tlL _i , FTARConf (tlI ₁ →tlI ₂ ) is the confidence degree of the fully weighted feature word association rule tlI ₁ →tlI ₂ , and its calculation formula is:

Among them, FTISup(L _i ) is the support degree of fully weighted frequent itemset L _i , FTISup(tlI ₁ ) is the support degree of fully weighted frequent itemset tlI ₁ ;

(8.3) Perform step (8.2) in a loop until each proper subset in the proper subset set of the fully weighted i_frequent itemset tlL _i is taken out once, and can only be taken out once, then turn to step (8.4);

(8.4) Perform step (8.1) to step (8.3) in a loop. When the itemsets in the frequent itemsets set L of fully weighted feature words are taken out once, and only once, the mining ends.

A retrieval system suitable for the above-mentioned Indonesian-Chinese cross-language retrieval method that integrates the association model and user feedback, including the following 4 modules and 3 databases:

Machine translation module: This module uses the Bing machine translation interface to translate Indonesian user queries into Chinese queries, and translate the final search result Chinese documents into Indonesian documents for submission to users;

Search engine module: this module is a search engine, which is used to search the translated Chinese query formula on the Internet, and obtain the cross-language initial inspection result document set;

Fully weighted association pattern mining and user-related feedback module: used to submit the top r cross-language preliminary inspection result document sets to the user, and the user will judge the relevance of these documents and determine the relevant document database for the initial inspection, and then use the fully weighted association The rule mining technology mines the expansion words related to the query on the relevant document database of the initial inspection, realizes cross-language query expansion, and re-searches the combination of the expansion words and the original query to obtain the final retrieval result Chinese document;

Final result display module: used to submit the Chinese document of the final retrieval result to the machine translation module for translation into an Indonesian document, and return the Chinese document of the final retrieval result and the Indonesian language document of the final retrieval result to the user;

Preliminary inspection of relevant document databases;

Fully weighted association rule base;

Extended thesaurus.

The above-mentioned fully weighted association pattern mining and user-related feedback modules include the following 5 modules:

User click behavior-related feedback extraction module: used to capture the document download behavior generated when the user browses the document set of the preliminary examination result, and extract the document downloaded by the user to construct the relevant document set of user feedback;

Document preprocessing module: used to perform Chinese word segmentation, remove stop words, calculate feature word weights and extract feature words for user feedback related document sets, and build a database of relevant documents for initial inspection;

Fully weighted association rule mining module: used for fully weighted association rule mining on the primary inspection related document database, mining the fully weighted feature term frequent itemsets and association rule patterns containing the original query term, and constructing a fully weighted association rule library;

Cross-lingual query expansion word generation module: used to extract the expansion words related to the original query from the fully weighted association rule base, and build an extended thesaurus;

Cross-language query extension implementation module: used to extract Chinese extended words from the extended thesaurus, combine the extended words and the original query to form a new query, submit it to the search engine for retrieval on the Internet, and obtain the final search result Chinese document.

Compared with the prior art, the present invention has the advantages of:

(1) The present invention combines fully weighted association rule mining technology and user-related feedback to Indonesian-Chinese cross-language information retrieval, and proposes an Indonesian-Chinese cross-language information retrieval method and system that integrates user click download behavior and fully weighted association pattern mining. Compared with the monolingual Chinese text retrieval benchmark MB, the Indonesian cross-language retrieval benchmark CLB and the traditional cross-language information retrieval method CLR_PRF based on pseudo-relevance feedback, the retrieval performance of the method of the present invention has been greatly improved and improved, and the experimental results show that , the present invention obtains very good retrieval results, and its index values are higher than the values of the benchmark CLB and CLR_PRF algorithms, and the retrieval effect of the description type of query subject is also better than that of the title type, and the MAP value of the retrieval results is increased the most.

(2) Experimental results show that the Indonesian-Chinese cross-language information retrieval method and system that integrates fully weighted association pattern mining and user-related feedback proposed by the present invention are effective, and can improve and enhance cross-language information retrieval performance. The main reasons are as follows: In cross-language information retrieval, query translation results have a greater impact on cross-language retrieval results, often resulting in the quality of cross-language initial inspection results being inferior to single-language initial inspection results, that is, the problem of query topic drift. However, the fusion of user click behavior and fully weighted association pattern mining is applied to the Indonesian cross-language information retrieval model, and the most relevant feedback information to the original query can be obtained, and the expansion words related to the original query can be obtained through fully weighted association rule mining to realize cross-language Query expansion avoids the serious topic drift problem in cross-language retrieval and improves the performance of cross-language retrieval in Indonesia.

Description of drawings

Fig. 1 is the block diagram of the Indonesian-Chinese cross-language retrieval method of the present invention fusion association pattern and user's feedback.

Fig. 2 is the overall flow chart of the Indonesian-Chinese cross-language retrieval system of the present invention which integrates the association model and user feedback.

Fig. 3 is a structural block diagram of the Indonesian-Chinese cross-language retrieval system of the present invention which integrates the association model and user feedback.

Fig. 4 is a structural block diagram of the fully weighted association pattern mining and user-related feedback module of the present invention.

detailed description

The technical solution of the present invention will be described in further non-limiting detail below in conjunction with the embodiments and accompanying drawings.

One, in order to better illustrate the technical scheme of the present invention, the relevant concepts involved in the present invention are introduced as follows below:

Assume that the target language (TargetLanguage, TL) first-check related document set obtained by the user query after the initial cross-language search and user-related feedback is TLdoc＝{tld ₁ ,tld ₂ ,…,tld _n }, tld _i (1≦i≦ n) represents the i-th document in the target language document set TLdoc, tld _j = {t ₁ ,t ₂ ,...,t _m ,...,t _p }, t _m (m=1,2,...,p) is called is the Feature-term Item (FTI) of the target language, referred to as a feature item, generally composed of words, words or phrases, and the corresponding feature item weight set W _{i in tld i =} {w _i1 ,w _i2 , …,w _im ,…,w _ip },w _im is the corresponding weight of the mth feature item t _m in the i-th document tld _i , let tlI={t ₁ ,t ₂ ,…,t _k } express The set of all feature items in TLdoc, the subset Y of tlI is called Feature-term Itemsets in TLdoc, that is, item set Y.

For itemsets (tlI ₁ ,tlI ₂ ), and According to the theoretical knowledge of fully weighted association pattern mining (Huang Mingxuan, Yan Xiaowei, Zhang Shichao. Pseudo-relevance feedback query expansion based on matrix weighted association rule mining. Software Journal, Vol.20, No.7, July 2009, pp.1854-1865), give Some basic concepts are given below.

Definition 1 The formula for calculating the fully weighted support (Feature-term Itemsets Support, FTISup) of feature term item set I (I=(tlI ₁ , tlI ₂ )) is shown in formula (1).

in, is the sum of the weights of each document in itemset I in TLdocD, k is the item length (that is, the number of items) of itemset I, and n is the total number of documents in the initial inspection related document set TLdoc.

Definition 2 The fully weighted confidence (Feature-termAssociationRule Confidence, FTARConf) of the inter-word association rule tlI ₁ → tlI ₂ is shown in formula (2).

Among them, FTIsup(tlI ₁ , tlI ₂ ) is the fully weighted support degree of the itemset (tlI ₁ , tlI ₂ ).

Definition 3 assumes that the minimum support threshold is ms, and the minimum confidence threshold is mc. If it satisfies: FTISup(tlI ₁ ,tlI ₂ )≧ms, FTARConf(tlI ₁ →tlI ₂ )≧mc, then it is called feature word item set (tlI ₁ , tlI ₂ ) are frequent itemsets, and the inter-word association rules (tlI ₁ →tlI ₂ ) are strong association rules.

Definition 4 The feature word k_itemset weight threshold (k-Item Weighted Threshold, KIWT) (q<k) that contains q_itemset refers to the weight prediction of subsequent itemsets that contain q_itemset.

Let tlT be a fully weighted q-itemset, and q<k, in the (tlI-tlT) itemset, record the corresponding weights of the first (kq) items with the largest weights as w ₁ , w ₂ ,...w kq , the support count of q-itemset _tlT in TLdoc is SC(tlT), according to k -Theoretical knowledge of weight threshold value, the calculation formula of the weight threshold value of feature word k_itemset including q_itemset is given, as shown in formula (3).

Two, as shown in Figure 1, the Indonesian-Chinese cross-language retrieval method of the fusion association model of the present embodiment and user feedback comprises the following steps:

(1) Translate the Indonesian language user query into Chinese query formula through the machine translation module, and submit it to the search engine for preliminary retrieval on the Internet, and obtain the document set of preliminary inspection results; the machine translation module uses the Bing machine translation interface, namely Microsoft TranslatorAPI; The search engine module can be an existing search engine such as Baidu or Google;

(2) Extract the first r Chinese documents from the cross-language preliminary inspection result document set and submit them to the user;

(3) The user judges the Chinese documents in the cross-language preliminary inspection result document set to obtain the user feedback related document set, and the total number of documents in the document set is set to n;

(4) Preprocessing user feedback related document sets, that is, performing preprocessing operations of Chinese word segmentation, removing stop words, calculating feature word weights and extracting feature words, and constructing a database of relevant documents for initial inspection;

The calculation of the feature word weight adopts the tf-idf method, and its calculation formula is:

Among them, tf _{m, n} represent the number of occurrences of the feature word t _m in the document d _n , df _m represents the number of documents containing the feature word t _m , and N represents the total number of documents in the document collection;

(5) Scan the relevant document database for initial inspection, mine fully weighted feature word 1_candidate item set C ₁ , calculate C ₁ weight w(C ₁ ), and count the maximum weight maxCw _i (!C ₁ of items other than C ₁ ) and C ₁ support count n _c1 , ms is the minimum support threshold, calculate the value of KIWT(1,2), the calculation formula of KIWT(1,2) is: KIWT(1,2)=n×1×ms -n _c1 ×maxCw _i (!C ₁ );

(6) Calculate the support degree FTISup(C ₁ ) of C _1. If FTISup(C ₁ )≧ms, mine 1_frequent itemset L ₁ from 1_candidate item set C ₁ and add it to the fully weighted feature word frequency Itemset set L, the calculation formula of FTISup(C ₁ ) is:

(7) Mining k_itemsets, where k≧2, including steps (7.1) to (7.7):

(7.1) Compare the candidate (k-1)_itemset C _k-1 weight and KIWT(k-1,k) value, and cut off the candidates whose W(C _k-1 )<KIWT(k-1,k) Itemset C _k-1 ;

(7.2) Aproiri connection is performed on the remaining candidate (k-1) item sets C _k-1 to obtain C _k ;

(7.3) When k=2, cut off the candidate 2_itemset not containing the query item;

(7.4) Scan the initial inspection related document database, count the maximum weight maxCw _i (!C _k ) of items other than C _k and the support count n _ck of C _k , and calculate the C _k weight w(C _k ) and KIWT(k -1,k), the calculation formula of KIWT(k-1,k) is: KIWT(k-1,k)=n×k×ms-n _ck ×maxCw _i (!C _k );

(7.5) Cut off the candidate item set C _{k where} n _ck is 0;

(7.6) For the remaining candidate k_itemset C _k , calculate C _k support FTISup(C _k ), if FTISup(C _k )≧ms, mine k_frequent itemset from candidate k_itemset C _k L _k , and added to the fully weighted feature word frequent itemset set L, the calculation formula of FTISup(C _k ) is:

(7.7) If k is greater than the candidate item set length threshold or the candidate k_item set is an empty set, then the mining ends, otherwise, continue the loop steps (7.1) to (7.6);

(8) Mining the fully weighted association rules of the feature words containing the query term from the fully weighted feature word frequent itemset set L, constructing the fully weighted association rule base; the method includes steps (8.1) to (8.4):

(8.1) Extract a certain fully weighted i_frequent itemset tlL _i from the fully weighted feature word frequent itemset set L, and find out all proper subsets of tlL _i ;

(8.2) Take out two proper subsets tlI ₁ and tlI ₂ arbitrarily from the proper subset set of tlL _i , when And tlI ₁ ∪tlI ₂ ＝L _i , if FTARConf(tlI ₁ →tlI ₂ )≧mc, fully weighted feature word strong association rules tlI ₁ →tlI ₂ are mined; if FTARConf(tlI ₂ →tlI ₁ )≧mc, Then dig out the fully weighted feature word strong association rule tlI ₂ →tlI ₁ ; the mc is the minimum confidence threshold, tlI ₁ and tlI ₂ are fully weighted feature word frequent itemsets, which are proper subset itemsets of tlL _i , FTARConf (tlI ₁ →tlI ₂ ) is the confidence degree of the fully weighted feature word association rule tlI ₁ →tlI ₂ , and its calculation formula is:

Among them, FTISup(L _i ) is the support degree of fully weighted frequent itemset L _i , FTISup(tlI ₁ ) is the support degree of fully weighted frequent itemset tlI ₁ ;

(8.3) Perform step (8.2) in a loop until each proper subset in the proper subset set of the fully weighted i_frequent itemset tlL _i is taken out once, and can only be taken out once, then turn to step (8.4);

(8.4) Step (8.1) to step (8.3) is carried out in a loop, when the itemsets in the frequent itemsets set L of fully weighted feature words are all taken out once, and can only be taken out once, then the mining ends;

(9) Extract the cross-lingual expansion words related to the original query from the fully weighted association rule base, and construct the extended thesaurus;

(10) Submitting the original query and the expanded word combination to the search engine to retrieve the final retrieval result Chinese document again;

(11) Submit the Chinese document of the final retrieval result to the machine translation module for translation into a Indonesian language document, and finally return the Chinese document of the final retrieval result and the Indonesian language document of the final retrieval result to the user.

Three, as shown in Figures 2 to 4, the retrieval system applicable to the Indonesian-Chinese cross-language retrieval method of the fusion of the association model and user feedback in this embodiment includes the following 4 modules and 3 databases:

Machine translation module: This module uses the Bing machine translation interface, that is, Microsoft TranslatorAPI, to translate Indonesian user queries into Chinese queries, and translate the final retrieval result Chinese documents into Indonesian documents and submit them to users;

Search engine module: this module is a search engine, which is used to search the translated Chinese query formula on the Internet, and obtain the cross-language initial inspection result document set;

Fully weighted association pattern mining and user-related feedback module: used to submit the top r cross-language preliminary inspection result document sets to the user, and the user will judge the relevance of these documents and determine the relevant document database for the initial inspection, and then use the fully weighted association The rule mining technology mines the expansion words related to the query on the relevant document database of the initial inspection, realizes cross-language query expansion, and re-searches the combination of the expansion words and the original query to obtain the final retrieval result Chinese document;

Final result display module: used to submit the Chinese document of the final retrieval result to the machine translation module for translation into an Indonesian document, and return the Chinese document of the final retrieval result and the Indonesian language document of the final retrieval result to the user;

Preliminary inspection of relevant document databases;

Fully weighted association rule base;

Extended thesaurus.

Wherein, the fully weighted association pattern mining and user-related feedback modules include the following 5 modules:

User click behavior-related feedback extraction module: used to capture the document download behavior generated when the user browses the document set of the preliminary examination result, and extract the document downloaded by the user to construct the relevant document set of user feedback;

Document preprocessing module: used to perform Chinese word segmentation, remove stop words, calculate feature word weights and extract feature words for user feedback related document sets, and build a database of relevant documents for initial inspection;

Fully weighted association rule mining module: used for fully weighted association rule mining on the primary inspection related document database, mining the fully weighted feature term frequent itemsets and association rule patterns containing the original query term, and constructing a fully weighted association rule library;

Cross-lingual query expansion word generation module: used to extract the expansion words related to the original query from the fully weighted association rule base, and build an extended thesaurus;

Cross-language query extension implementation module: used to extract Chinese extended words from the extended thesaurus, combine the extended words and the original query to form a new query, submit it to the search engine for retrieval on the Internet, and obtain the final search result Chinese document.

Four, in conjunction with technical scheme of the present invention, the beneficial effect of the present invention is further described by experiment below:

Because the research scope of search engine is wide and there are many factors to be considered, the present invention is carried out in Indonesian interlanguage retrieval system based on vector space model instead, therefore, this experiment is a simulation experiment. The source program of the method and system of the present invention is written to carry out the experiment of the present invention. The Chinese corpus of NTCIR-5CLIR, a cross-lingual information retrieval standard data test set at the International Evaluation Conference on Multilingual Language Processing hosted by the Japan Institute of Information and Information Technology, is used as the experimental corpus.

NTCIR-5CLIR has a query set, a document test set, and a result set. Among them, the query set has 50 query topics, which are divided into four types: TITLE, DESC, NARR, and CONC. In this paper, the TITLE and DESC types are selected, and the TITLE type query topics A brief description with nouns and noun phrases is a short query, and the DESC type is a brief description of the query subject in the form of a sentence, which is a long query. There are two evaluation criteria for the result set: Rigid and Relax. The Rigid criterion means that the answers are highly relevant or related to the original query. The Relax standard means that the answers are highly relevant, relevant or partially relevant.

In order to conduct the experiment of the cross-lingual information retrieval model in Indonesia, professional translators from translation agencies were invited to manually translate 50 query topics of the Chinese version of NTCIR-5CLIR into Indonesian queries.

In this experiment, the Chinese lexical analysis system ICTCLAS developed by the Institute of Computing Technology, Chinese Academy of Sciences is used to preprocess the Chinese experimental corpus and translated Chinese queries. The traditional tf-idf method is used for feature word weight calculation, and the post-translation query item weight (w _i,q ) calculation formula (from the literature G.Salton,C.Buckley.Term-weighting approaches in automatic text retrieval[J].Information Processing&Management ,1988,24(5):513-523.) as shown in formula (4).

Among them, tf _{i, q} is the initial frequency of the query item appearing in the query text information, N is the total number of relevant documents in the initial inspection, and df _i is the number of relevant documents in the initial inspection containing the i-th query item.

In this experiment, the method of setting the weight of Chinese extended words is as follows: the confidence of the matrix weighted association rules is used as the weight of the extended words. Extended word weights.

The experimental evaluation benchmarks are:

(1) Monolingual Baseline (MB): The retrieval results obtained by directly retrieving Chinese documents with Chinese queries.

(2) Cross-language Baseline (CLB): refers to the first cross-language retrieval result without any relevant feedback, that is, the retrieval result obtained by retrieving Chinese documents after the Indonesian query is translated by the machine translation system.

(3) The traditional cross-language retrieval method CLR_PRF based on pseudo-relevance feedback (Jianfeng Gao, JianyunNie, Jian Zhang, et al, TREC-9CLIR Experiments at MSRCN[C].In:Proc.ofthe9th Text Retrieval Evaluation Conference,2001:343- 353.; Wu Dan, He Daqing, Wang Huilin. Cross-lingual query expansion based on pseudo-correlation [J]. Journal of Information Science, 2010,29(2):232-239.). In this experiment, 20 cross-language first-order documents were extracted to build a first-inspection related document set, and 20 feature words with the first weight (in descending order) were extracted as extended words.

The experimental parameters of the method of the present invention: extract the top 100 documents of the cross-language initial inspection documents and submit them to the user, and the user determines the initial inspection document set after making a correlation judgment. In this experiment, the first 100 initial inspection documents contain related documents in the known result set It is regarded as user-related feedback information, and extracted to construct the relevant document set of the user's initial inspection. Finally, the fully weighted association rule mining technology is used to mine the expansion words of the relevant document set of the initial inspection to realize query expansion.

The source program is written, and the method of the present invention and the benchmark methods MB, CLB and CLR_PRF are used for Indonesian-Chinese cross-language text retrieval on the NTCIR-5CLIR test set, and their cross-language retrieval performance is compared and analyzed.

(1) Benchmark experiment results

Run the experimental source program, submit the title part and description part of 50 query topics of NTCIR-5CLIR for Chinese single-language retrieval, Indonesian-Chinese cross-language retrieval and traditional Indonesian-Chinese cross-language retrieval based on pseudo-correlation feedback, that is, run the benchmark algorithm MB , CLB and CLR_PRF, and the retrieval experiment results of three benchmark methods are shown in Table 1.

Table 1:

The experimental results in Table 1 show that the retrieval results of the Indonesian-Chinese cross-language retrieval benchmark CLB and the traditional CLR_PRF method only achieve about 30% to 60% of the single-language retrieval benchmark MB, and the retrieval effect of long query description is better than that of short query The retrieval effect of the title type is good. For the CLR_PRF algorithm, except for MAP, the index values of the other index values have increased compared with the benchmark CLB by about 5% to 30%, while the MAP value has generally decreased, with a maximum rate of %46. These results show that cross-language retrieval is affected by query translation factors, and the retrieval performance is generally low, which is not as good as its corresponding single-language retrieval performance.

(2) the retrieval performance comparison of the inventive method and benchmark algorithm

Using the title type and description type of 50 query topics of NTCIR-5CLIR, the retrieval performance experiment was carried out when the support degree changed and the confidence degree changed, and it was compared with the Indonesian-Chinese cross-language retrieval benchmark CLB and the traditional CLR_PRF method, as well as a single language Retrieve benchmark MB for retrieval performance comparison. Specific parameters of the experiment: Table 2 shows the comparison of retrieval performance when the support threshold changes, and Table 3 shows the MAP, P@5 and P@15 values of the retrieval results when the confidence threshold changes.

Table 2:

table 3:

As can be seen from the experimental results in Table 2, when the fully weighted support threshold changes, the index values of the retrieval results of the method of the present invention are all higher than the value of the Indonesian-Chinese cross-language retrieval benchmark CLB and the traditional pseudo-correlation cross-language retrieval method CLR_PRF, Both achieve 60% to 102% of the monolingual retrieval benchmark MB. Compared with the benchmark CLB, the maximum improvement is 91.55% (that is, the P@5 value of Rigid type), and the lowest is 36.06% type, and the P@15 value of Relax evaluation). Compared with the CLR_PRF method, its improvement can reach up to 244.97% (that is, the MAP value of description query type and Rigid evaluation), and the lowest is 32.89%. In particular, the MAP value of its description query type and Rigid evaluation has reached and Exceeds 2% of the single language retrieval benchmark MB. In addition, the retrieval effect of the query subject description type is better than that of the title type, and the MAP value of the retrieval results has the largest increase.

The experimental results in Table 3 show that when the confidence threshold changes, the present invention obtains good retrieval results, and its index values are all higher than the values of the benchmark CLB and CLR_PRF algorithms, all reaching 58.07% to 101.2% of the single-language retrieval benchmark MB %, the retrieval effect of the description type of query subject is better than that of the title type, and the MAP value of the retrieval result is increased the most.

To sum up, the present invention has good application value.

Claims (5)

1. a kind of Indonesian Chinese interlingual retrieval method that merges association mode and user feedback, is characterized in that, comprises the steps: (1) Translate the Indonesian language user query into Chinese query formula through the machine translation module, and submit it to the search engine for preliminary retrieval on the Internet, and obtain the preliminary inspection result document set; (2) Extract the first r Chinese documents from the cross-language preliminary inspection result document set and submit them to the user; (3) The user judges the Chinese documents in the cross-language preliminary inspection result document set to obtain the user feedback related document set, and the total number of documents in the document set is set to n; (4) Preprocessing user feedback related document sets, that is, performing preprocessing operations of Chinese word segmentation, removing stop words, calculating feature word weights and extracting feature words, and constructing a database of relevant documents for initial inspection; (5) Scan the relevant document database for initial inspection, mine fully weighted feature word 1_candidate item set C ₁ , calculate C ₁ weight w(C ₁ ), and count the maximum weight maxCw _i (!C ₁ of items other than C ₁ ) and C ₁ support count n _c1 , ms is the minimum support threshold, calculate the value of KIWT(1,2), the calculation formula of KIWT(1,2) is: KIWT(1,2)=n×1×ms -n _C1 ×maxCw _i (!C ₁ ); (6) Calculate the support degree FTISup(C ₁ ) of C _1. If FTISup(C ₁ )≧ms, mine 1_frequent itemset L ₁ from 1_candidate item set C ₁ and add it to the fully weighted feature word frequency Itemset set L, the calculation formula of FTISup(C ₁ ) is: (7) Mining k_itemsets, wherein said k≧2, including steps (7.1) to (7.7): (7.1) Compare the candidate (k-1)_itemset C _k-1 weight and KIWT(k-1,k) value, and cut off the candidates whose W(C _k-1 )<KIWT(k-1,k) Itemset C _k-1 ; (7.2) Aproiri connection is performed on the remaining candidate (k-1) item sets C _k-1 to obtain C _k ; (7.3) When k=2, cut off the candidate 2_itemset not containing the query item; (7.4) Scan the initial inspection related document database, count the maximum weight maxCw _i (!C _k ) of items other than C _k and the support count n _ck of C _k , and calculate the C _k weight w(C _k ) and KIWT(k -1,k), the calculation formula of KIWT(k-1,k) is: KIWT(k-1,k)=n×k×ms-n _ck ×maxCw _i (!C _k ); (7.5) Cut off the candidate item set C _{k where} n _ck is 0; (7.6) For the remaining candidate k_itemset C _k , calculate C _k support FTISup(C _k ), if FTISup(C _k )≧ms, mine k_frequent itemset from candidate k_itemset C _k L _k , and added to the fully weighted feature word frequent itemset set L, the calculation formula of FTISup(C _k ) is: (7.7) If k is greater than the candidate item set length threshold or the candidate k_item set is an empty set, then the mining ends, otherwise, continue the loop steps (7.1) to (7.6); (8) Mining the fully weighted association rules of the feature words containing the query term from the fully weighted feature word frequent itemset set L, and constructing the fully weighted association rule base; (9) Extract the cross-lingual expansion words related to the original query from the fully weighted association rule base, and construct the extended thesaurus; (10) Submitting the original query and the expanded word combination to the search engine to retrieve the final retrieval result Chinese document again; (11) Submit the Chinese document of the final retrieval result to the machine translation module for translation into a Indonesian language document, and finally return the Chinese document of the final retrieval result and the Indonesian language document of the final retrieval result to the user. 2. the Indonesian-Chinese cross-language retrieval method of fusion association pattern according to claim 1 and user feedback, it is characterized in that, the calculation of the feature word weight value described in step (4) adopts tf-idf method, its calculation formula yes: Among them, tf _m,n represents the number of occurrences of the feature word t _m in the document d _n , df _m represents the number of documents containing the feature word t _m , and N represents the total number of documents in the document collection. 3. the Indonesian-Chinese cross-language retrieval method of fusion association mode and user feedback according to claim 1, is characterized in that, the method for step (8) comprises steps (8.1) to (8.4): (8.1) Extract a certain fully weighted i_frequent itemset tlL _i from the fully weighted feature word frequent itemset set L, and find out all proper subsets of tlL _i ; (8.2) Take out two proper subsets tlI ₁ and tlI ₂ arbitrarily from the proper subset set of tlL _i , when And tlI ₁ ∪tlI ₂ ＝L _i , if FTARConf(tlI ₁ →tlI ₂ )≧mc, fully weighted feature word strong association rules tlI ₁ →tlI ₂ are mined; if FTARConf(tlI ₂ →tlI ₁ )≧mc, Then dig out the fully weighted feature word strong association rule tlI ₂ →tlI ₁ ; the mc is the minimum confidence threshold, tlI ₁ and tlI ₂ are fully weighted feature word frequent itemsets, which are proper subset itemsets of tlL _i , FTARConf (tlI ₁ →tlI ₂ ) is the confidence degree of the fully weighted feature word association rule tlI ₁ →tlI ₂ , and its calculation formula is: Among them, FTISup(L _i ) is the support degree of fully weighted frequent itemset L _i , FTISup(tlI ₁ ) is the support degree of fully weighted frequent itemset tlI ₁ ; (8.3) Perform step (8.2) in a loop until each proper subset in the proper subset set of the fully weighted i_frequent itemset tlL _i is taken out once, and can only be taken out once, then turn to step (8.4); (8.4) Perform step (8.1) to step (8.3) in a loop. When the itemsets in the frequent itemsets set L of fully weighted feature words are taken out once, and only once, the mining ends. 4. A retrieval system applicable to the Indonesian-Chinese cross-language retrieval method of fusion association mode and user feedback described in claim 1, is characterized in that: comprise following 4 modules and 3 databases: Machine translation module: This module uses the Bing machine translation interface to translate Indonesian user queries into Chinese queries, and translate the final search result Chinese documents into Indonesian documents for submission to users; Search engine module: this module is a search engine, which is used to search the translated Chinese query formula on the Internet, and obtain the cross-language initial inspection result document set; Fully weighted association pattern mining and user-related feedback module: used to submit the top r cross-language preliminary inspection result document sets to the user, and the user will judge the relevance of these documents and determine the relevant document database for the initial inspection, and then use the fully weighted association The rule mining technology mines the expansion words related to the query on the relevant document database of the initial inspection, realizes cross-language query expansion, and re-searches the combination of the expansion words and the original query to obtain the final retrieval result Chinese document; Final result display module: used to submit the Chinese document of the final retrieval result to the machine translation module for translation into an Indonesian document, and return the Chinese document of the final retrieval result and the Indonesian language document of the final retrieval result to the user; Preliminary inspection of relevant document databases; Fully weighted association rule base; Extended thesaurus. 5. retrieval system according to claim 4, is characterized in that, described fully weighted association pattern mining and user-related feedback module comprise following 5 modules: User click behavior-related feedback extraction module: used to capture the document download behavior generated when the user browses the document set of the preliminary examination result, and extract the document downloaded by the user to construct the relevant document set of user feedback; Document preprocessing module: used to perform Chinese word segmentation, remove stop words, calculate feature word weights and extract feature words for user feedback related document sets, and build a database of relevant documents for initial inspection; Fully weighted association rule mining module: used for fully weighted association rule mining on the primary inspection related document database, mining the fully weighted feature term frequent itemsets and association rule patterns containing the original query term, and constructing a fully weighted association rule library; Cross-lingual query expansion word generation module: used to extract the expansion words related to the original query from the fully weighted association rule base, and build an extended thesaurus; Cross-language query extension implementation module: used to extract Chinese extended words from the extended thesaurus, combine the extended words and the original query to form a new query, submit it to the search engine for retrieval on the Internet, and obtain the final search result Chinese document.