CN102262624A - System and method for realizing cross-language communication based on multi-mode assistance - Google Patents

Abstract

The present invention proposes a cross-language communication system and method based on multi-modal assistance. The method utilizes the foreground interaction module, data management module and semantic association module in the cross-language communication system to analyze conversation content and use natural language processing tools It can automatically extract the central topic and keywords in the conversation, and the semantic association module can automatically search for relevant pictures and video clips based on the detected central topic and keyword information, and provide them to both parties in an appropriate way, so as to promote mutual understanding and communication. Here, the pictures and videos used as aids in understanding can be automatically picked up from the Internet by searching, or directly obtained from a pre-marked multimedia library. Finally, the system generates a multi-modal conversation summary based on the text chat information of the two parties in the conversation and the corresponding picture and video content.

Description

System and method for realizing cross-language communication based on multimodal assistance

technical field

The invention belongs to the fields of multimedia analysis and network communication, and relates to a method for realizing cross-language communication based on multi-mode assistance.

Background technique

With the rapid development of communication technology and Internet technology, a network instant messaging system completely different from traditional communication methods such as mail, telephone, and telegram has emerged, such as MSN and QQ. Traditional emails and telegrams mainly use text, and telephone calls mainly use voice, while instant messaging can not only use text and voice, but also can assist rich multimedia methods such as video and pictures. Through the instant messaging system, people across oceans can realize face-to-face real-time conversation. The whole earth has become a veritable global village.

Language issues remain an insurmountable barrier in instant messaging for interlocutors who speak different languages. In recent years, due to the great progress made in machine translation technology, the language problems existing in the communication between users of different languages have been solved to some extent by the technology of machine translation. But machine translation has two obvious disadvantages. The first is accurate translation between different languages. But machine translation can still only automatically translate some simple conversations. Even for the two most spoken languages in the world: English and Chinese, the accuracy of automatic translation between them still cannot fully meet the needs of daily use. If one considers the numerous minority languages in the world, accurate automatic translation between different languages may still be a problem with a long way to go. The second is that the polysemy of word meaning is another challenging problem encountered in machine translation.

In order to enhance the text-to-image synthesis system for communication, in the prior art, the main content of the input text is represented in the form of pictures. The solution to this problem is to complete the conversion from text to pictures through three optimizations, that is, to maximize the probability of keywords based on the input text, to maximize the probability of corresponding pictures based on the input text and the selected keywords, and to maximize the probability of occurrence of the corresponding pictures based on the input text and selected keywords. Input text, selected keywords and corresponding images to maximize the spatial distribution of text and images. In this way, based on these three optimizations, the conversion from text to pictures is finally completed. But this system has the following three disadvantages:

1). The processing speed of the system is slow. Due to the calculation optimization of this system, it will slow down the conversion speed of pictures to text;

2). The interface of the system is not friendly. Because it is necessary to optimize the input text and the given picture together, the spatial layout is presented to the user. If such a layout with mixed text and pictures is applied to the dialogue between users, it will definitely give users an unfriendly feeling.

3). The system is not easy to use. Since it is terminal software, it is bound to require the user to download the software by himself. The shortcomings of the system that are not easy to use can be solved with the help of web pages.

Contents of the invention

The purpose of the present invention is to solve the technical defects of slow processing speed and difficult use in the prior art, and assist people who use different languages to communicate online smoothly through multi-modal information. The ambiguity and polysemy generated in traditional automatic translation are reduced through multi-modal information such as images and videos, and the semantic understanding of user dialogue content is assisted. Therefore, the present invention provides a cross-language communication based on multi-modal assistance. Methods.

In order to achieve the stated purpose, the first aspect of the present invention provides a cross-language communication system based on multimodal assistance. The technical solution of the system includes: a front-end interaction module, a data management module and a semantic association module, wherein:

The input terminal of the front-end interaction module accepts the text chat content input by the user and preprocesses the content of the user chat, obtains the text information of the user chat, and transmits the processed user text chat through the output end of the front-end interaction module of the front-end interaction module Content; the chat page of the front-end interactive module displays the text content of the conversation between the two parties and the multimedia pictures recommended by the system according to the content of the conversation between the two parties for the user;

The input end of the semantic association module is connected to the output end of the front-end interaction module, which receives and analyzes the text chat content of the user, uses natural language processing tools to extract the main content of the conversation between the two parties, obtains and outputs the text information associated with the translated text and related Corresponding multimedia information, and generating a multimodal summary based on text chat content, translated content and corresponding multimedia information;

The input end of the data management module is connected to the output end of the semantic association module. The data management module needs to store the newly input text chat content, translated content and corresponding multimedia information, and simultaneously store the historical user information together with the new user information. Integrate, generate and display all the text content of the conversation between the two chatting parties and the multimedia picture information recommended by the system according to the content of the conversation between the two parties.

In a preferred embodiment, after the semantic association module in the background receives the text information sent by the user, the semantic association module will translate the results of Google translation in order to help chat users of different languages understand the meaning of the other party's speech from the perspective of the language used. Integrate in; so that in addition to the original user chat information, the translation of the chat content based on Google Translate is also attached.

In a preferred embodiment, the semantic association module extracts the main content of the conversation between the two parties by using these main content as keywords, and using text-based image retrieval to retrieve the corresponding candidate picture set from the image database.

In order to achieve the stated purpose, the second aspect of the present invention provides a method for realizing cross-language communication using a cross-language communication system based on multimodal assistance. The method is based on user dialogue and chat, and analyzes the content of the conversation according to text analysis technology. The result obtained is to provide users with multimedia elements to assist semantic understanding between users who have language barriers or have differences in cultural backgrounds. The implementation steps of the method include the following:

Step S1: The user first sends the text content he wants to chat with the other party through the front-end interface of the semantic chat. The modal analysis method analyzes the content of the user's conversation, and uses natural language processing tools to automatically extract the central topic and keywords in the conversation;

Step S2: According to the central topic and keyword information in the conversation, the semantic association module automatically retrieves relevant picture collections and video clips from the database or the Internet according to the conversation topic and provides them to the conversation parties by using text-based image retrieval;

Step S3: The system generates a multi-modal conversation summary based on the text chat information of the two parties in the conversation and the corresponding pictures and video clips, and finally realizes smooth semantic communication between users of different languages in the form of multimedia; At the same time, the system can generate a multi-modal conversation summary for the two parties based on the text chat history information of the two parties and the corresponding picture and video content.

In a preferred embodiment, the multimodal conversation summary includes text, audio, image and video information, and provides multimedia elements for users to assist semantic understanding between users who have language barriers or have different cultural backgrounds.

In a preferred embodiment, the content of the pictures and video clips is automatically picked up from the network through searching, or directly obtained from a pre-marked multimedia library.

In a preferred embodiment, the multimodal conversation summary is a topic-based summary, using a relational network and detecting topics according to statistics of co-occurrence frequencies of words that appeared in a predefined prediction library in the last conversation.

Beneficial effects of the present invention: the core of the present invention is how to describe text information through multimedia information (image or video). The cross-language communication system based on multimodal assistance proposed by the present invention can provide a friendly and convenient environment for online instant messaging, and has three main features: first, friendliness, due to the use of topic-related image or video search technology to assist text Content understanding, which greatly reduces the ambiguity and ambiguity of translation; second interactivity, which enables the system to better meet the individual needs of users; third ease of use, the proposed system can automatically Generate summaries for multimedia.

In order to assist the communication and understanding between users, the system of the present invention adopts a topic-based cross-modal analysis method. The system generates a multi-modal conversation summary based on the text chat information of the two parties in the conversation and the corresponding pictures and video content. In this way, because this multimodal conversation contains rich content, that is, multimodal auxiliary information such as images, videos, and texts that are very intuitive and easy to understand, it can effectively eliminate the ambiguity that occurs in automatic translation between plain texts and improve It improves the efficiency and quality of language communication, and realizes smooth semantic communication between users of different languages.

Description of drawings

Fig. 1 is the interface block diagram of the cross-language communication system based on multimodal assistance in the present invention;

Fig. 2 is the structural block diagram of the cross-language communication system based on multimodal assistance in the present invention;

Figures 3a and 3b present an example result for ordering a pizza;

Figure 4 is an example of a multimedia summary for a conversation.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

The present invention proposes a cross-language communication system based on multimodal assistance and a method for realizing cross-language communication. The method utilizes a front-end interaction module 1, a data management module 2 and a semantic association module 3 to analyze conversation content and use natural language processing tools It can automatically extract the central topic and keywords in the conversation, and the semantic association module 3 automatically searches for relevant pictures and video clips based on the detected central topic and keyword information, and provides them to both parties in an appropriate way, so as to achieve Promote mutual understanding and communication. Here, the pictures and videos used as aids in understanding can be automatically picked up from the Internet by searching, or directly obtained from a pre-marked multimedia library. Finally, the system generates a multimodal conversation summary based on the text chat information of the two parties in the conversation and the corresponding picture and video content.

Fig. 1 shows the user interface of a multimedia chatting system that assists cross-language communication proposed by the present invention, which can provide a friendly, interactive and timely communication environment for users using different languages to communicate. It mainly includes three aspects of functions: text communication based on timely translation, a picture or video retrieval based on the topic of the conversation, and multimedia summary for the content of the conversation (shown in Figure 4). The top part of Figure 1 is mainly used to display the name of the system and the subject of the user's chat conversation. What follows is the main display area of the system interface, that is, text dialogue and multimedia auxiliary information display, such as asking for directions, buying a car, and ordering a hotel. The right part of Figure 1 is text communication based on instant translation, and the user text chat area: presents the user's basic text chat information and Google-translated text information related to the machine; the left part of Figure 1 is a picture or video based on the topic of the conversation Retrieval, as well as multimedia abstracts for conversation content, multimedia content display area: based on the content of user conversations, relevant multimedia information is presented for users to assist users in semantic understanding.

FIG. 2 shows a structural block diagram of the cross-language communication system based on multi-modal assistance in the present invention. The framework of the cross-language communication system based on multimodal assistance is divided into three components, namely the front-end interaction module 1, the data management module 2 and the semantic association module 3. The front-end design includes two parts: the chat interface and the front-end and back-end interaction. Wherein the foreground interactive module 1 accepts the text chat content of user's input and carries out preprocessing to the content of user's chat, obtains the text information of user's chat; The user's text chat content after is transmitted to semantic association module 3, and the chat page of foreground interactive module 1 shows the text content of the conversation of chatting two sides and the multimedia picture that comes out according to the content system of both sides' conversation for the user display.

The input end of the semantic association module 3 is connected to the output end of the front-end interaction module 1. After receiving and analyzing the content of the user’s text chat, the natural language processing tool is used to extract the main content of the conversation between the two parties, and the text information is obtained and output. text and corresponding multimedia information, and generate a multimodal summary according to text chat content, translated content and corresponding multimedia information; semantic association module 3 outputs text chat content, translated content and corresponding multimedia information to Data management module2.

The input end of the data management module 2 is connected to the output end of the semantic association module 3, and the data management module 2 stores newly input text chat content, translated content and corresponding multimedia information. At the same time, it is necessary to integrate historical user information with new user information, generate and display all the text content of the dialogue between the chatting parties and the multimedia picture information recommended by the system according to the content of the conversation between the two parties; and finally return them to the front-end interaction module 1. Finally, the chat page of the foreground interaction module 1 will display all the information to the user. The workflow of the module is described in detail below.

The user first sends the chat content to the foreground interaction module 1 through the chat interface. See Figure 1. The user's semantic chat interface is divided into two main parts, one part is to display the text content of the traditional conversation between the two parties in the chat, and the other part is to display the list of multimedia pictures recommended by the system according to the content of the conversation between the two parties. At this time, the front-end interface transmits the text information of the text chat entered by the user to the back-end through the front-end and back-end interaction modules built by Ajax. The background framework is divided into two parts, one is the data management module 2, and the other is the semantic association module 3. After the background receives the text information sent by the user, the semantic association module 3 integrates the results of Google translation in order to help chat users of different languages understand the meaning of the other party's speech from the perspective of their own language. In this way, in addition to the original user chat information, a translation of the chat content based on Google Translate is attached. Semantic association module 3 uses natural language processing tools to extract the main content of the conversation between the two parties on the text information. At this time, the semantic association module 3 first uses these main contents as keywords, and uses text-based image retrieval to retrieve corresponding candidate picture sets from the image database. Finally, all user conversations and corresponding multimedia information can be used to generate a multimodal summary. Take a sample result of ordering pizza as an example to illustrate the generated multimedia summary, as shown in Figure 4. From the multimodal summary given in Figure 4, it can be seen that the user made choices about the type of pizza, drink and payment method during the conversation with the goods clerk in the pizzeria. The picture of the pizza of the corresponding pizzeria that the user feeds back through the chat system can better choose according to his own wishes. This multimodal summary is also beneficial for the user to imagine pizza again in the future, and the multimedia information provided by this multimodal summary can help the user to review.

The semantic association mechanism in Fig. 2 is described below. The semantic association mechanism is mainly divided into three parts, namely text communication based on instant translation, video retrieval based on topics and pictures, and finally multimodal summarization based on user text chat content and corresponding multimedia information.

(1). Text communication based on timely translation

Similar to most instant communication systems, the system proposed by the present invention also supports the most basic text communication. However, since the two parties in the conversation may have different language backgrounds. For example, when an English-speaking American talks with a Chinese-speaking Chinese on the Internet, and the American does not understand Chinese, and the Chinese does not understand English, ordinary text conversations cannot enable the two parties to communicate without barriers. For this reason, the system of the present invention integrates a simple machine translation function. When chatting, the speaker's language is automatically translated into the recipient's language and then displayed, so that both sides of the conversation can roughly understand each other's intentions.

(2). Topic-based image and video retrieval

Despite machine translation as a bridge, cross-language communication is still not quite satisfactory. The original intention is that the accuracy of machine translation (the intelligibility of the translated target language) is still low. The translation results between major languages, such as between English and Chinese, are still not up to practical standards. In addition, due to the existence of polysemous words and sentences in everyday language, it is difficult for machine translation technology to meet the needs of reality. The foods shown in Fig. 3a include: seafood, fruit, and meat. Fruits include: bananas, apples, and oranges. For example, the word "apple" can represent either a fruit or an apple company (Fig. 3a). In order to create an easy-to-understand and immersive online communication environment, we designed a topic-based image/video retrieval sub-module to assist users with different language backgrounds to communicate with each other. Among them, topic detection, image retrieval and related feedback are the three main functions.

Topic detection is achieved in two ways. The first is for the user to select a topic from a predefined list of topics. Different topics are associated with different annotated (annotated by hand or learned methods) image/video databases. The second method is to extract subject keywords through extractive text analysis. In a conversation, many entity words representing the content of the conversation can be extracted. According to these entity words, we first build a semantic relationship tree similar to WordNet, which describes the semantic inheritance relationship between words. As shown in Figure 3a, the words "apple", "banana" and "orange" all belong to the category of food Fruit sub-category in , and the word "Apple" shown in Figure 3b may also belong to the category of computer brands together with "Dell" and "Lenovo". Figure 3b shows examples of "Apple" computer brands include: desktop computers mac, tablet ipad and smartphone iphone. The above semantic relationships can be extracted from WordNet, or can be obtained by using the "term frequency-inverse document frequency" weight (TF-IDF) of words in a predefined corpus. Once we extract keywords from the conversation, the system can automatically infer the corresponding potential topics by analyzing the semantic relationship between the keywords.

According to the topic extracted in the dialogue, the system automatically retrieves the corresponding picture information from the network or background database. Using text-based retrieval, we can easily find relevant annotated images based on the conversation topic. However, most of the network pictures are unlabeled. We use the retrieved pictures associated with the labeled text as the training set to learn a topic model, and use this topic model to retrieve a large number of unlabeled pictures. For this reason, topic-based image retrieval needs to build a topic model first, and its goal is to automatically find a latent (implicit) semantic space in order to more accurately model the document information in the retrieval process. Here, the semantic structure of a document includes some potential hidden concepts or themes (they often correspond to a stable and unique co-occurrence pattern between words). Through the weighted combination of latent topics, a document can be represented as a series of latent topics, and its comprehensive combination coefficient can be regarded as a feature representation of the document. This representation has a number of advantages: First, the semantic space tends to have a lower dimensionality than the word space. This not only saves storage space, but also facilitates fast search; secondly, through the conversion from word space to semantic space, it can not only reduce the noise in the word vector, but also solve the above polysemy and ambiguity problems, thereby improving the retrieval performance. For example, the word "apple" can refer to both a fruit and a computer brand (Fig. 3b). Its precise meaning can be deduced from other related keywords on the same topic.

Feedback, as a popular human-computer interaction technique, is widely used in the analysis of visual information in the text domain. Through the user's feedback and evaluation of the system output, the system can be adaptively corrected. Supervision information obtained through user feedback has been proven to be effective in practice. In our system, users can select the correct topic from a candidate list generated by an automatic topic extraction algorithm. The selected topics will be used for the next topic extraction by modeling temporal (current and next) topic relationships. In image retrieval, our system lists some retrieved sample images and invites users to rate related images according to the conversation topic.

(3). Multimodal Summary

Traditional instant communication usually keeps chat records in text mode. In our system, users can use multi-modal methods such as pictures, videos, and texts to express the speaker's intentions. By saving chat information in a multimodal way instead of a single text, you can get more vivid records than ever before.

Text, image and video summarization is a research hotspot in the field of natural language processing and multimedia. It often expresses the original text (picture or video) information in a general way through a more concise and concise text (picture or video). Most of the current related technologies construct abstract content based on salient features, repeated modalities or keywords (frames) and other information. In our system, considering that there are a large amount of image and video information besides text, we adopt a topic-driven summarization method to generate summary information on a specific topic by analyzing the content of conversations between users. This summary information contains relevant text, pictures and video content related to the topic.

The above is only a specific implementation mode in the present invention, but the scope of protection of the present invention is not limited thereto. Anyone familiar with the technology can understand the conceivable transformation or replacement within the technical scope disclosed in the present invention. All should be covered within the scope of protection of the claims of the present invention.

Claims (7)

1. A cross-language communication system based on multimodal assistance, characterized in that the system includes: a front-end interaction module, a data management module and a semantic association module, wherein: The input terminal of the front-end interaction module accepts the text chat content input by the user and preprocesses the content of the user chat, obtains the text information of the user chat, and transmits the processed user text chat through the output end of the front-end interaction module of the front-end interaction module Content; the chat page of the front-end interactive module displays the text content of the conversation between the two parties and the multimedia pictures recommended by the system according to the content of the conversation between the two parties for the user; The input end of the semantic association module is connected to the output end of the front-end interaction module, which receives and analyzes the text chat content of the user, uses natural language processing tools to extract the main content of the conversation between the two parties, obtains and outputs the text information associated with the translated text and related Corresponding multimedia information, and generating a multimodal summary based on text chat content, translated content and corresponding multimedia information; The input end of the data management module is connected to the output end of the semantic association module. The data management module needs to store the newly input text chat content, translated content and corresponding multimedia information, and simultaneously store the historical user information together with the new user information. Integrate, generate and display all the text content of the conversation between the two chatting parties and the multimedia picture information recommended by the system according to the content of the conversation between the two parties. 2. The cross-language communication system based on multi-modal assistance as claimed in claim 1, characterized in that, after the semantic association module in the background receives the text information sent by the user, the semantic association module can help chat users of different languages from using In order to understand the meaning of the other party’s speech from the perspective of their own language, the results of Google translation are integrated; in this way, in addition to the original user chat information, the translation of the chat content based on Google translation is also attached. 3. The cross-language communication system based on multimodal assistance as claimed in claim 1, wherein the main content of the conversation between the two parties is extracted by the semantic association module by using these main content as keywords, and using text-based image retrieval from the image database The corresponding candidate image set is retrieved. 4. A method for realizing cross-language communication using the cross-language communication system based on multimodal assistance described in claim 1, characterized in that, the method is based on the user dialogue and chatting, and is obtained by analyzing the content of the conversation according to the text analysis technology As a result, multimedia elements are provided for users to assist semantic understanding between users who have language barriers or have differences in cultural backgrounds, and the implementation of the method includes the following steps: Step S1: The user first sends the text content he wants to chat with the other party through the front-end interface of the semantic chat. The modal analysis method analyzes the content of the user's conversation, and uses natural language processing tools to automatically extract the central topic and keywords in the conversation; Step S2: According to the central topic and keyword information in the conversation, the semantic association module automatically retrieves relevant picture collections and video clips from the database or the Internet according to the conversation topic and provides them to the conversation parties by using text-based image retrieval; Step S3: The system generates a multi-modal conversation summary based on the text chat information of the two parties in the conversation and the corresponding pictures and video clips, and finally realizes smooth semantic communication between users of different languages in the form of multimedia; At the same time, the system can generate a multi-modal conversation summary for the two parties based on the text chat history information of the two parties and the corresponding picture and video content. 5. The method for realizing cross-language communication as claimed in claim 4, characterized in that, the multi-modal talk summary includes text, audio, image and video information, and provides multimedia elements for users to assist in language communication. Semantic understanding between users with different or cultural backgrounds. 6. The method for realizing cross-language communication as claimed in claim 4, characterized in that, the contents of the pictures and video clips are automatically picked up from the Internet by searching, or directly obtained from a pre-marked multimedia library. 7. The method for realizing cross-language communication as claimed in claim 4, characterized in that, the multi-modal conversation summary is a subject-based summary, using a relational network and appearing in a predefined The co-occurrence frequency of words in the expected library is detected by the subject.

Images