CN106980664B - A bilingual comparable corpus mining method and device - Google Patents

Abstract

The invention discloses a bilingual comparable corpus mining method and device. By grabbing multiple pictures and corresponding text information from databases in different languages in advance, a multimodal knowledge base containing pictures and text information is established; The pictures in the knowledge base are used as query pictures, and pictures are retrieved in the target language knowledge base to find the target pictures that are similar to the query pictures; according to the text information corresponding to the target pictures and the text information corresponding to the query pictures, a bilingual comparable corpus is constructed. This application adopts cross-media information retrieval technology, uses pictures as a medium for communicating the source language and the target language, and then obtains the equivalent or comparable text of the source language at the target end, which provides a new method for mining bilingual comparable resources in the Internet. Solved the problem of scarcity of specific bilingual resources.

Description

A bilingual comparable corpus mining method and device

technical field

The invention relates to the field of computer technology, in particular to a bilingual comparable corpus mining method and device.

Background technique

Bilingual comparable corpora refers to a collection of texts in different languages that represent similar semantics. Large-scale bilingual comparable corpora usually contain rich and diverse bilingual translation units, such as phrase-level, sentence-level translation pairs, and bilingual dictionaries. In small languages or in some limited fields, parallel resources are usually few, but bilingual comparable corpora is relatively easy to obtain. Therefore, bilingual comparable corpora has become an important resource in the fields of machine translation and cross-lingual information retrieval. How to automatically obtain large-scale bilingual comparable corpora has become a basic task in machine translation.

At present, the research methods of bilingual comparable corpus acquisition can be roughly divided into the following three categories: one is a bilingual comparable corpus construction method based on cross-language information retrieval, which extracts keywords from documents in the source language and uses a bilingual dictionary The keywords are translated into the target language, and then used as a retrieval query to retrieve a set of candidate documents in the target language, and finally obtain comparable bilingual documents. The second category is a bilingual comparable corpus construction method based on content and structure similarity, which uses a translation engine (Google Translate or Bing Translate, etc.) to translate the source language document into the target language, and obtain the pseudo-translation result of the source document. And further proceed from the similarity of vocabulary, theme and structure, evaluate the similarity between the pseudo-translation document and the target language document, and sort and select similar documents. The third type of method is to extract bilingual comparable corpora from structured knowledge bases (such as Wikipedia, etc.). Utilize existing taxonomy entries and link information in knowledge bases such as Wikipedia to obtain comparable resources at the document level. Existing methods only build bilingual comparable corpus resources from the perspective of text similarity with the help of bilingual dictionaries or existing knowledge bases. Such methods rely on the performance of a large number of manually annotated dictionaries, knowledge bases or existing translation systems. When targeting small languages or specific fields, the scarce language resources will restrict the generality of such methods.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a bilingual comparable corpus mining method and device, so as to solve the problem that the existing methods only start from the perspective of text similarity and have poor generality to specific scarce speech resources.

In order to solve the above-mentioned technical problems, the present invention provides a bilingual comparable corpus mining method, including:

Capture multiple pictures and corresponding text information from databases in different languages in advance, and establish a multimodal knowledge base containing the pictures and the text information;

Taking the picture in the source language knowledge base as the query picture, performing picture retrieval in the target language knowledge base, and finding out the target picture similar to the query picture;

According to the text information corresponding to the target image and the text information corresponding to the query image, a bilingual comparable corpus is constructed.

Optionally, the pre-fetching of multiple pictures and corresponding text information from databases in different languages includes:

Use a web crawler to grab a picture from a news website, the text information is the subject and/or title information corresponding to the picture, and the picture and the corresponding text information are stored in the multimodal knowledge base as a two-tuple .

Optionally, performing image retrieval in the target language knowledge base, and finding out a target image similar to the query image includes:

Extract the key points of the query picture by using a scale-invariant feature transformation algorithm, and characterize the query picture as a feature vector based on the key points;

Extract the feature vectors of all candidate pictures in the target language knowledge base, and match the query picture and the key points of the candidate pictures;

Calculate the average Euclidean distance between all matching key points as the image similarity between images;

The candidate pictures are sorted according to the picture similarity, and a target picture similar to the query picture is selected.

Optionally, performing image retrieval in the target language knowledge base, and finding out a target image similar to the query image includes:

Determine the subject classification and release time information of the query picture;

Filtering out pictures that do not match the subject classification and release time information in the target language knowledge base;

Image retrieval is performed in the filtered target language knowledge base to find out a target image similar to the query image.

Optionally, according to the text information corresponding to the target picture and the text information corresponding to the query picture, constructing a bilingual comparable corpus includes:

calculating the text similarity between the text information corresponding to the query picture and the text information corresponding to the target picture;

Reordering the target pictures according to the text similarity;

Based on the re-ranked results, a bilingual comparable corpus is constructed.

Optionally, the calculating the text similarity between the text information corresponding to the query picture and the text information corresponding to the target picture includes:

Calculate the content similarity, entity similarity and structural similarity between the text information corresponding to the query picture and the text information corresponding to the target picture;

The content similarity, the entity similarity and the structural similarity are weighted and averaged, and the text similarity of the corresponding text information is calculated.

The present invention also provides a bilingual comparable corpus mining device, comprising:

The knowledge base building module is used to capture multiple pictures and corresponding text information from databases in different languages in advance, and establish a multimodal knowledge base including the pictures and the text information;

a search module, used for taking the picture in the source language knowledge base as a query picture, performing picture retrieval in the target language knowledge base, and finding out a target picture similar to the query picture;

The construction module is configured to construct bilingual comparable corpus according to the text information corresponding to the target image and the text information corresponding to the query image.

Optionally, the search module includes:

an extraction unit, used for extracting the key points of the query picture by adopting a scale-invariant feature conversion algorithm, and characterizing the query picture as a feature vector based on the key points;

a matching unit for extracting the feature vectors of all candidate pictures in the target language knowledge base, and matching the query picture and the key points of the candidate pictures;

The similarity calculation unit is used to calculate the average Euclidean distance between all matching key points as the image similarity between the images;

A selection unit, configured to sort candidate pictures according to the picture similarity, and select a target picture similar to the query picture.

Optionally, the search module further includes:

a determining unit for determining the subject classification and release time information of the query picture;

a filtering unit, configured to filter out pictures in the target language knowledge base that do not match the subject classification and release time information;

A search unit, configured to perform image retrieval in the filtered target language knowledge base to find a target image similar to the query image.

Optionally, the building blocks include:

a text similarity calculation unit, configured to calculate the text similarity of the text information corresponding to the query picture and the text information corresponding to the target picture;

a sorting unit, configured to reorder the target pictures according to the text similarity;

The construction unit is used to construct bilingual comparable corpus according to the reordered results.

The bilingual comparable corpus mining method and device provided by the present invention builds a multimodal knowledge base including pictures and text information by grabbing multiple pictures and corresponding text information from databases in different languages in advance; The pictures in the database are used as query pictures, and pictures are retrieved in the target language knowledge base to find the target pictures similar to the query pictures; according to the text information corresponding to the target pictures and the text information corresponding to the query pictures, a bilingual comparable corpus is constructed. This application adopts cross-media information retrieval technology, uses pictures as a medium for communicating the source language and the target language, and then obtains the equivalent or comparable text of the source language at the target end, which provides a new method for mining bilingual comparable resources in the Internet. Solved the problem of scarcity of specific bilingual resources.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following will briefly introduce the accompanying drawings used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only For some embodiments of the present invention, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

1 is a flow chart of a specific implementation of a bilingual comparable corpus mining method provided by the present invention;

Figure 2 is a schematic diagram of similar pictures and comparable titles mined in news texts;

3 is a schematic diagram of a process of finding a target picture similar to the query picture in an embodiment of the present invention;

4 is a flowchart of another specific implementation of the bilingual comparable corpus mining method provided by the present invention;

Figure 5 is a schematic diagram of the matching result of the key points in the picture;

FIG. 6 is a structural block diagram of a bilingual comparable corpus mining apparatus provided by an embodiment of the present invention.

Detailed ways

In order to make those skilled in the art better understand the solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

A flowchart of a specific implementation of the bilingual comparable corpus mining method provided by the present invention is shown in FIG. 1 , and the method includes:

Step S101: Capture multiple pictures and corresponding text information from databases in different languages in advance, and establish a multimodal knowledge base including the pictures and the text information;

Specifically, in this embodiment of the present invention, a web crawler can be used to capture large-scale pictures and corresponding text information from news websites in different languages. The text information can be the subject and/or title information corresponding to the pictures, and the pictures and The corresponding text information is stored in the multimodal knowledge base as a binary group.

It should be pointed out that the present invention is specifically based on the Cross-Media Information Retrieval (CMIR) technology, which is very similar to the Cross-Language Information Retrieve (CLIR) technology, and the only difference lies in the communication between the source language and the target language. The medium is different. Cross-media information retrieval uses pictures instead of text as a bridge to communicate between the source language and the target language, and then obtains the equivalent or comparable text of the source language at the target end.

As a specific implementation manner, the knowledge base of CMIR may include two parts, a picture and a picture title. The picture title describes the content of the picture and corresponds to the picture one-to-one. Refer to FIG. 2 for an example. CMIR uses a similar image search engine to establish the connection of texts in different languages, that is, the titles corresponding to similar images are considered to be bilingual comparable texts. The CLIR system uses a weak translation system or a bilingual dictionary to translate the source language text into the target language, and uses the translation result as a search query to retrieve similar texts in the target language. To a certain extent, CMIR is easier to use than CLIR. The only problem that CMIR needs to consider is how to improve the quality of retrieval. CLIR needs to additionally consider the quality of bilingual dictionaries and the translation performance of weak translators. Therefore, the bilingual comparable corpus mining method based on the cross-media information retrieval technology proposed in the embodiment of the present invention is the first attempt to mine bilingual comparable corpus by using the similarity of pictures.

Step S102: using the picture in the source language knowledge base as the query picture, perform picture retrieval in the target language knowledge base, and find out the target picture similar to the query picture;

Between multimodal knowledge bases in different languages, a similar image retrieval system is established to retrieve similar images in the target language knowledge base.

Referring to FIG. 3 , in the embodiment of the present invention, image retrieval is performed in the target language knowledge base, and the process of finding a target image similar to the query image may specifically include:

Step S1021: using a scale-invariant feature transformation algorithm to extract key points of the query picture, and characterizing the query picture as a feature vector based on the key points;

Step S1022: extract the feature vectors of all candidate pictures in the target language knowledge base, and match the query picture and the key points of the candidate pictures;

Step S1023: Calculate the average Euclidean distance between all matching key points as the picture similarity between the pictures;

Step S1024: Sort candidate pictures according to the picture similarity, and select a target picture similar to the query picture.

Step S103 : constructing a bilingual comparable corpus according to the text information corresponding to the target picture and the text information corresponding to the query picture.

The bilingual comparable corpus mining method provided by the present invention establishes a multi-modal knowledge base including pictures and text information by grabbing multiple pictures and corresponding text information from databases in different languages in advance; The image is used as the query image, and the image retrieval is performed in the target language knowledge base to find the target image similar to the query image; according to the text information corresponding to the target image and the text information corresponding to the query image, a bilingual comparable corpus is constructed. This application adopts cross-media information retrieval technology, uses pictures as a medium for communicating the source language and the target language, and then obtains the equivalent or comparable text of the source language at the target end, which provides a new method for mining bilingual comparable resources in the Internet. Solved the problem of scarcity of specific bilingual resources.

Based on any of the above embodiments, in order to improve the accuracy and operation efficiency of the similar image retrieval system, the embodiment of the present invention further proposes a model for optimizing retrieval of similar images in combination with news topic information and time tag information. Specifically, in the bilingual comparable corpus mining method provided by the present invention, image retrieval is performed in the target language knowledge base, and the process of finding out the target image similar to the query image can be specifically as follows:

Determine the subject classification and release time information of the query picture;

Filtering out pictures that do not match the subject classification and release time information in the target language knowledge base;

Image retrieval is performed in the filtered target language knowledge base to find out a target image similar to the query image.

Referring to the flowchart of another specific implementation of the bilingual comparable corpus mining method provided by the present invention in FIG. 4 , the specific implementation process of the method will be further elaborated below.

The process includes:

Step S201 : building a multimodal knowledge base, this module uses web crawlers to crawl pictures and corresponding picture titles from news websites in multiple languages, and constructs a multimodal knowledge base by taking the picture and corresponding title pairs as two-tuples.

Specifically, it can be divided into the following steps: according to the division of news website columns, crawling corresponding news pages under different columns or themes; parsing news pages, extracting pictures and picture titles through web page structure analysis, and extracting news release time; and the corresponding image title to form a binary pair, which is stored in the multimodal knowledge base.

Grab the news web pages under the corresponding theme or column according to the theme column of the news website, and store them in the corresponding theme directory respectively. Match images and image titles in web pages according to image tags (eg: img, pic, etc.), and remove web page tags (eg: <a>, </br>, etc.) that are not related to content in the title text. The image and image title are represented as two-tuples as a record in the multimodal knowledge base.

Taking Chinese-English comparable resource mining as an example, the present invention crawls news from mainstream Chinese news websites (such as: Xinhuanet, Fenghuang.com, Sina.com) and English news websites (FOX, CNN, BBC), and gives according to the news website The different columns (such as: military, finance, technology, sports, entertainment, etc.) classify news and themes, and record the time of news release. For crawled news, the embodiment of the present invention performs structural analysis on news web pages, extracts pictures and picture titles in the web pages, and stores them as two-tuples (image, caption) as a record in a multimodal database. In this way, multimodal databases in different languages are established.

Step S202: Determine the subject classification and release time information of the query picture; filter out pictures that do not match the subject classification and release time information in the target language knowledge base;

Manually integrate the existing topic classification and labeling system of news websites. For news websites in different languages, use translation dictionaries or machine translation technology to achieve cross-language topic label mapping. Different languages and different news websites have their own news classification systems, so it is necessary to establish a unified classification standard. For example map "Sports" to "Sports".

Integrate image retrieval optimization with news topics, and mark images with specified tags according to the topic category of the news. When searching for similar images, limit the search scope to the same subject. Comparable news often has the same or similar themes, so when retrieving similar images, the subject of the image source is limited, and candidate images are retrieved and sorted under the same subject to filter out invalid candidate images.

The optimization of image retrieval that integrates time tags, automatically divides news according to the time of news release, and limits the retrieval range to a specified time window when retrieving similar images, and changes the time window threshold to obtain different retrieval results. The occurrence of news events is time-sensitive, resulting in similar news pictures often appearing in a similar time period. Based on this, using the release time of news pictures as a label, different time windows d are set, and similar pictures are retrieved within the corresponding time interval {t-d, t+d}.

This step uses labels such as topic classification and release time of the news where the pictures are located to limit the search space of the similar picture retrieval module to filter dissimilar picture candidates, so that the efficiency and performance of the similar picture retrieval system can be stabilized when facing large-scale data sets promote.

Step S203: Perform image retrieval in the filtered target language knowledge base to find out a target image similar to the query image;

By extracting key point features of pictures, and calculating the similarity between pictures based on the extracted features, it is used to retrieve similar pictures in multimodal knowledge bases of different languages.

Specifically, it can be divided into the following steps: take the picture in the source language knowledge base as the query picture, use the scale-invariant feature transformation algorithm to extract the key points of the picture, and represent the picture as a feature vector based on key points; extract all candidate pictures in the target picture library Its feature vector, and match the key points of the query image and the target candidate image; calculate the average Euclidean distance between all matching key points as the similarity measure between the images; for the query image, according to the similarity of the image, the target image All images in the library are sorted, and Top-N images are selected as the similar image set.

Specifically, for the keypoint x in the retrieved picture, two most similar keypoints y, z are found in the target candidate picture. The similarity between two keypoints is obtained by the Euclidean distance of the feature vectors of the keypoints.

To determine whether the key point x and the key point y of the target image match, the rule is determined according to whether the ratio of the Euclidean distance between the nearest key point and the next nearest key point is less than the specified threshold. That is, if d(x,y)<d(x,z)*θ, it is considered that the key point x and the key point y match, where the threshold θ∈[0,1] is generally set to 0.8. Figure 5 shows the matching results of keypoints in the two pictures.

Calculate the similarity between pictures. Calculate the similarity between the entire image based on all matching keypoints in the image. Specifically, the average Euclidean distance between all matching points is calculated. The formula is as follows:

m represents the number of matching keypoints in the two images, and n represents the dimension of the keypoint feature vector.

According to the image similarity, the target candidate image set is sorted, and the similar images of topN are selected.

Step S204: According to the text information corresponding to the target image and the text information corresponding to the query image, construct a bilingual comparable corpus.

With the help of a similar image retrieval system, similar images are retrieved under news topics related to the target domain, and bilingual comparable texts are extracted. The invention proposes a title comparability measure based on image similarity modeling, and a title comparability measure integrating image similarity and text similarity modeling.

The title comparability measurement method based on image similarity modeling can be divided into the following steps:

Using the similar image retrieval system, the image in the source language is used as input, and the similar image collection is retrieved at the target end, and the similarity between the query image and the retrieval result image is calculated based on the SIFT algorithm.

Use the image similarity to model the comparability of image titles, and use the ranking result of the images as the ranking result of the comparability of image titles. The comparability between titles is measured using the following formula:

where S(s _cap , t _cap ) represents the comparability between the title s _cap and the title t _cap . v(s _img ,t _img ) represents the similarity between the picture s _img and the picture t _img , which is calculated by the SITF algorithm, and b is a constant.

The title comparability measurement method combining image similarity and text similarity can be divided into the following steps:

Using the similar image retrieval system, the image in the source language is used as input, and the similar image collection is retrieved at the target end, and the similarity between the query image and the retrieval result image is calculated based on the SIFT algorithm.

Calculate the text similarity between the title of the query image and the title of the target candidate image. The present invention comprehensively evaluates the text similarity between the query title and the candidate title from three angles of content similarity (fc), entity similarity (fe), and structural similarity (fl). The specific formula is as follows:

S=αfc+βfe+γfl

It further fuses the text similarity score between the image similarity and the image title, and re-ranks the similar image retrieval results (based on the image similarity). Based on the reordered results, construct comparable text pairs. The present invention adopts the following formula to calculate the comprehensive similarity between the query and the candidate:

where S _txt (s _cap ,t _cap ) represents the textual similarity between the query title and the retrieval title, and b is a constant used to control the contribution of image similarity to the final similarity score.

The calculation method of the text similarity between the query image title and the candidate image title is as follows: use the Google online translation system to translate the query title into the target language to retain important information such as word order, sentence structure and named entities of the original text.

The comparability of query title translation and target image title is evaluated based on the following three features:

Content similarity: perform word segmentation, stop word removal, and rooting of the translated text description of the query title to obtain a bag-of-words representation. Based on the vector space model, the cosine similarity (denoted as fc) of the query title and the target title is calculated.

Entity similarity: Use the Stanford named entity recognition tool to identify named entities in image captions, obtain a set of named entity word bags, and calculate the named entity similarity (denoted as fe) between bilingual texts based on a vector space model.

Structural similarity: The ratio of the number of content words (including nouns, verbs, adverbs, adjectives, proper nouns, etc.) in the query title and the target title is used as a comparability evaluation standard (denoted as fl).

A weighted average method is used to fuse the above three features to obtain the comparability score between the query title and the target title.

S=αfc+βfe+γfl

Among them, α=0.8, β=0.15, and γ=0.05 are empirically set according to the contribution degree of each feature to the comparability.

The bilingual comparable corpus mining method based on cross-media information retrieval proposed by the present invention provides a new method for bilingual comparable resource mining in the Internet. Not only a method of constructing a large-scale multimodal database is proposed, but also the mining of comparable resources based on image similarity is proposed. Based on this, two comparability measurement methods are proposed to obtain large-scale comparable texts.

The following describes the bilingual comparable corpus mining device provided by the embodiment of the present invention. The bilingual comparable corpus mining device described below and the bilingual comparable corpus mining method described above may refer to each other correspondingly.

6 is a block diagram of a structure of a bilingual comparable corpus mining device provided by an embodiment of the present invention. Referring to FIG. 6, the bilingual comparable corpus mining device may include:

The knowledge base building module 100 is used for grabbing multiple pictures and corresponding text information from databases in different languages in advance, and establishing a multimodal knowledge base including the pictures and the text information;

The search module 200 is configured to use the picture in the source language knowledge base as a query picture, perform picture retrieval in the target language knowledge base, and find out a target picture similar to the query picture;

The construction module 300 is configured to construct a bilingual comparable corpus according to the text information corresponding to the target picture and the text information corresponding to the query picture.

As a specific embodiment, in the bilingual comparable corpus mining device provided by the present invention, the search module may specifically include:

an extraction unit, used for extracting the key points of the query picture by adopting a scale-invariant feature conversion algorithm, and characterizing the query picture as a feature vector based on the key points;

a matching unit for extracting the feature vectors of all candidate pictures in the target language knowledge base, and matching the query picture and the key points of the candidate pictures;

The similarity calculation unit is used to calculate the average Euclidean distance between all matching key points as the image similarity between the images;

A selection unit, configured to sort candidate pictures according to the picture similarity, and select a target picture similar to the query picture.

As a specific implementation manner, the above-mentioned search module may also include:

a determining unit for determining the subject classification and release time information of the query picture;

a filtering unit, configured to filter out pictures in the target language knowledge base that do not match the subject classification and release time information;

A search unit, configured to perform image retrieval in the filtered target language knowledge base to find a target image similar to the query image.

Specifically, the above-mentioned building modules may specifically include:

a text similarity calculation unit, configured to calculate the text similarity of the text information corresponding to the query picture and the text information corresponding to the target picture;

a sorting unit, configured to reorder the target pictures according to the text similarity;

The construction unit is used to construct bilingual comparable corpus according to the reordered results.

The bilingual comparable corpus mining device of this embodiment is used to realize the aforementioned bilingual comparable corpus mining method, so the specific implementation of the bilingual comparable corpus mining device can be found in the foregoing section of the embodiment of the bilingual comparable corpus mining method, for example , the knowledge base building module 100, the search module 200, and the building module 300 are respectively used to realize the steps S101, S102, and S103 in the above-mentioned bilingual comparable corpus mining method. Therefore, the specific implementation methods can refer to the descriptions of the corresponding partial embodiments. , and will not be repeated here.

The bilingual comparable corpus mining device provided by the present invention builds a multi-modal knowledge base including pictures and text information by grabbing multiple pictures and corresponding text information from databases in different languages in advance; The image is used as the query image, and the image retrieval is performed in the target language knowledge base to find the target image similar to the query image; according to the text information corresponding to the target image and the text information corresponding to the query image, a bilingual comparable corpus is constructed. This application adopts cross-media information retrieval technology, uses pictures as a medium for communicating the source language and the target language, and then obtains the equivalent or comparable text of the source language at the target end, which provides a new method for mining bilingual comparable resources in the Internet. Solved the problem of scarcity of specific bilingual resources.

The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments may be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant part can be referred to the description of the method.

Professionals may further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two, in order to clearly illustrate the possibilities of hardware and software. Interchangeability, the above description has generally described the components and steps of each example in terms of function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of the present invention.

The steps of a method or algorithm described in conjunction with the embodiments disclosed herein may be directly implemented in hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in random access memory (RAM), internal memory, read only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other in the technical field. in any other known form of storage medium.

The method and device for mining a bilingual comparable corpus provided by the present invention are described above in detail. The principles and implementations of the present invention are described herein by using specific examples, and the descriptions of the above embodiments are only used to help understand the method and the core idea of the present invention. It should be pointed out that for those skilled in the art, without departing from the principle of the present invention, several improvements and modifications can also be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims (8)

1. a bilingual comparable corpus mining method, is characterized in that, comprises: Capture multiple pictures and corresponding text information from databases in different languages in advance, and establish a multimodal knowledge base containing the pictures and the text information; Taking the picture in the source language knowledge base as the query picture, performing picture retrieval in the target language knowledge base, and finding out the target picture similar to the query picture; calculating the text similarity between the text information corresponding to the query picture and the text information corresponding to the target picture; The comprehensive similarity is calculated according to the picture similarity and the text similarity between the query picture and the target picture, and the target pictures are reordered according to the comprehensive similarity; wherein, the method for calculating the comprehensive similarity includes: :

S(s _cap ,t _cap ) represents the comprehensive similarity between the query title s _cap and the retrieval title t _cap ; v(s _img ,t _img ) represents the image similarity between the query image s _img and the target image t _img ; S _txt (s _cap , t _cap ) represents the text similarity between the query title s _cap and the retrieval title t _cap , b is a constant, and is used to control the contribution of the image similarity to the final similarity score; wherein, the retrieval title Specifically, the title of the target image; Based on the re-ranked results, a bilingual comparable corpus is constructed. 2. The bilingual comparable corpus mining method as claimed in claim 1, wherein the pre-fetching of multiple pictures and corresponding text information from databases of different languages comprises: A picture is grabbed from a news website by a web crawler, the text information is the title information corresponding to the picture, and the picture and the corresponding text information are stored in the multimodal knowledge base as a two-tuple. 3. The bilingual comparable corpus mining method as claimed in claim 2, wherein the image retrieval is performed in the target language knowledge base, and the target image that is similar to the query image is found to include: Extract the key points of the query picture by using a scale-invariant feature transformation algorithm, and characterize the query picture as a feature vector based on the key points; Extract the feature vectors of all candidate pictures in the target language knowledge base, and match the query picture and the key points of the candidate pictures; Calculate the average Euclidean distance between all matching key points as the image similarity between images; The candidate pictures are sorted according to the picture similarity, and a target picture similar to the query picture is selected. 4. The bilingual comparable corpus mining method according to any one of claims 1 to 3, wherein the image retrieval is performed in the target language knowledge base, and the target image that is similar to the query image is found to include: Determine the subject classification and release time information of the query picture; Filtering out pictures that do not match the subject classification and release time information in the target language knowledge base; Image retrieval is performed in the filtered target language knowledge base to find out a target image similar to the query image. 5. The bilingual comparable corpus mining method according to claim 1, wherein the calculating the text similarity of the text information corresponding to the query picture and the text information corresponding to the target picture comprises: Calculate the content similarity, entity similarity and structural similarity between the text information corresponding to the query picture and the text information corresponding to the target picture; The content similarity, the entity similarity and the structural similarity are weighted and averaged, and the text similarity of the corresponding text information is calculated. 6. A bilingual comparable corpus mining device, characterized in that, comprising: The knowledge base building module is used to capture multiple pictures and corresponding text information from databases in different languages in advance, and establish a multimodal knowledge base including the pictures and the text information; a search module, used for taking the picture in the source language knowledge base as a query picture, performing picture retrieval in the target language knowledge base, and finding out a target picture similar to the query picture; a building module, configured to calculate the text similarity between the text information corresponding to the query image and the text information corresponding to the target image; The comprehensive similarity is calculated according to the picture similarity and the text similarity between the query picture and the target picture, and the target pictures are reordered according to the comprehensive similarity; wherein, the method for calculating the comprehensive similarity includes: :

S(s _cap ,t _cap ) represents the comprehensive similarity between the query title s _cap and the retrieval title t _cap ; v(s _img ,t _img ) represents the image similarity between the query image s _img and the target image t _img ; S _txt (s _cap , t _cap ) represents the text similarity between the query title s _cap and the retrieval title t _cap , b is a constant, and is used to control the contribution of the image similarity to the final similarity score; wherein, the retrieval title Specifically, the title of the target image; Based on the re-ranked results, a bilingual comparable corpus is constructed. 7. The bilingual comparable corpus mining device according to claim 6, wherein the search module comprises: an extraction unit, used for extracting the key points of the query picture by adopting a scale-invariant feature conversion algorithm, and characterizing the query picture as a feature vector based on the key points; a matching unit for extracting the feature vectors of all candidate pictures in the target language knowledge base, and matching the query picture and the key points of the candidate pictures; The similarity calculation unit is used to calculate the average Euclidean distance between all matching key points as the image similarity between the images; A selection unit, configured to sort candidate pictures according to the picture similarity, and select a target picture similar to the query picture. 8. The bilingual comparable corpus mining device according to claim 6 or 7, wherein the search module further comprises: a determining unit for determining the subject classification and release time information of the query picture; a filtering unit, configured to filter out pictures in the target language knowledge base that do not match the subject classification and release time information; A search unit, configured to perform image retrieval in the filtered target language knowledge base to find a target image similar to the query image.

Images