WO2021102754A1 - Data processing method and device and storage medium - Google Patents

Abstract

A data processing method and device and a storage medium. The method comprises: acquiring audio data, and recognizing the audio data to obtain recognition texts (201); extracting at least two pieces of feature data of the recognition texts, and searching, from a knowledge graph database, additional information associated with each of the at least two pieces of feature data (202); and using the searched additional information and the recognition texts to generate a simultaneous interpretation result, and outputting the simultaneous interpretation result (203), the simultaneous interpretation result being used for being presented at a first terminal when the audio data is played.

Description

Data processing method, device and storage medium Technical field

This application relates to simultaneous interpretation technology, in particular to a data processing method, device and storage medium.

Background technique

Machine simultaneous interpretation technology is a speech translation product for conferences, reports and other scenes that has emerged in recent years. It combines automatic speech recognition (ASR, Automatic Speech Recognition) technology and machine translation (MT, Machine Translation) technology to provide speakers The speech content provides multilingual subtitles to display, instead of manual simultaneous interpretation services.

In related machine simultaneous interpretation technology, the speech content is usually translated and displayed through text, but the displayed content cannot enable users to fully and accurately understand the speech content.

Summary of the invention

The embodiments of the present application provide a data processing method, device, and storage medium.

The embodiment of the present application provides a data processing method, including:

Obtain audio data;

Recognizing the audio data to obtain recognized text;

Extracting at least two feature data of the recognized text, and searching a knowledge graph database for additional information associated with each feature data of the at least two feature data;

Use the additional information found and the recognized text to generate simultaneous interpretation results;

The simultaneous interpretation result is output; the simultaneous interpretation result is used for presentation on the first terminal when the audio data is played.

The embodiment of the present application also provides a data processing device, including:

The obtaining unit is configured to obtain audio data;

The first processing unit is configured to recognize the audio data to obtain recognized text; extract at least two feature data of the recognized text, and search for each feature data of the at least two feature data from the knowledge graph database Associated additional information;

The second processing unit is configured to use the found additional information and the recognized text to generate a simultaneous interpretation result;

The output unit is configured to output the simultaneous interpretation result; the simultaneous interpretation result is used for presentation on the first terminal when the audio data is played.

The embodiment of the present application further provides a data processing device, including a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor implements any of the above-mentioned methods when the program is executed. A step of.

The embodiment of the present application also provides a storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the steps of any one of the foregoing methods are implemented.

The data processing method, device, and storage medium provided by the embodiments of the application obtain audio data; recognize the audio data to obtain recognized text; extract at least two feature data of the recognized text, and search and search from the knowledge graph database Additional information associated with each feature data in the at least two feature data; use the additional information found and the recognition text to generate a simultaneous interpretation result; output the simultaneous interpretation result, which is used in the playback When the audio data is presented in the first terminal, the user is provided with additional information associated with the audio data, which can help the user to fully and accurately understand the speech content of the producer of the audio data, and reduce the user’s speech on the speech. Difficulty in understanding the content.

Description of the drawings

Figure 1 is a schematic diagram of the implementation process of simultaneous interpretation in related technologies;

FIG. 2 is a schematic diagram of an implementation process of a data processing method according to an embodiment of the application;

3 is a schematic diagram 1 of the implementation process of extracting at least two feature data of the recognized text by the server according to the embodiment of the application;

4 is a schematic diagram 1 of the implementation process of extracting at least two feature data of the translated text by the server according to the embodiment of the application;

5 is a second schematic diagram of the implementation process of extracting at least two feature data of the recognized text by the server according to the embodiment of the application;

6 is a second schematic diagram of the implementation process of extracting at least two feature data of the translated text by the server according to the embodiment of the application;

FIG. 7 is a schematic diagram of the implementation process of searching the additional information corresponding to the recognized text by the server according to the embodiment of the application;

FIG. 8 is a schematic diagram of the implementation process of searching the additional information corresponding to the translated text by the server according to the embodiment of the application;

9 is a schematic diagram of the implementation process of the server searching for the additional information corresponding to the recognized text according to the embodiment of the application;

FIG. 10 is a schematic diagram of the implementation process of searching the additional information corresponding to the translated text by the server according to the embodiment of the application;

FIG. 11 is a schematic diagram of the simultaneous interpretation result output by the server according to an embodiment of the application;

FIG. 12 is a schematic diagram of an implementation process of the server generating and outputting simultaneous interpretation results according to an embodiment of the application;

FIG. 13 is a schematic diagram of another implementation process of the server generating and outputting simultaneous interpretation results according to an embodiment of the application;

FIG. 14 is a schematic diagram of a structure of a data processing device according to an embodiment of the application;

FIG. 15 is a schematic diagram of another composition structure of the data processing device according to an embodiment of the application.

Detailed ways

Before describing the technical solutions of the embodiments of the present application in detail, first, a brief description of the related technologies will be given.

Among related technologies, machine simultaneous interpretation technology is a voice translation product for conferences, reports and other scenarios that has appeared in recent years. It combines artificial intelligence (AI, Artificial Intelligence) technology, MT, ASR, and speech synthesis (TTS, Text-To- Speech technology to realize Simultaneous Interpretation (SI, Simultaneous Interpretation). The machine simultaneous interpretation may also be referred to as machine simultaneous interpretation, AI simultaneous interpretation, AI simultaneous interpretation, and the like.

In practical applications, the lecturer can give a conference lecture through the client, and project the displayed content to the display screen, and show it to the user through the display screen. Figure 1 is a schematic diagram of the implementation process of simultaneous interpretation in related technologies. As shown in Figure 1, during a conference speech, the client uses a microphone to collect the speaker’s audio and sends the collected audio to the server. The end recognizes the audio data, obtains the recognized text corresponding to the source language, and then performs machine translation on the recognized text to obtain the translation result corresponding to the target language; finally, the voice is displayed on the screen or broadcast through headphones and other devices to show the user the translation result , So as to achieve the translation of the lecturer's speech content into the language required by the user.

Simultaneous interpretation solutions in related technologies can display simultaneous interpretation content in different languages for users, but only perform simultaneous interpretation for the speaker’s verbal content, and do not translate information related to the speaker’s verbal content (such as professional terms, citations, events, etc.). Character profile, etc.), if the user lacks or is not familiar with the above information, it will be difficult to correctly and fully understand the lecture content of the lecturer, which will not effectively reduce the difficulty of the user’s understanding of the lecture content, thereby reducing the user experience. In order to make it more convenient and easier for users to understand the content of the lecturer's speech, the current machine simultaneous interpretation technology uses a string matching method to match the interpreted string with a preset dictionary and display the matching result. But the target application scenario is not a simultaneous interpretation scenario; and for the preset dictionary, a lot of manual development is required, which is time-consuming and labor-intensive. In addition, for situations where the content of the speech cannot be determined in advance, it is even more difficult to pre-set the dictionary, and it is difficult to give a full understanding of the content of the speech by the speaker, and lack flexibility.

Based on this, in various embodiments of the present application, audio data is obtained; the audio data is recognized to obtain the recognized text; at least two feature data of the recognized text are extracted, and the data is searched from the knowledge graph database. Additional information associated with each feature data in at least two feature data (that is, supplementary information that can help users fully and accurately understand the content of the speech); use the additional information found and the recognized text to generate simultaneous interpretation results; output the The result of the simultaneous interpretation; the result of the simultaneous interpretation is used for presentation on the first terminal when the audio data is played.

The application will be further described in detail below in conjunction with the drawings and specific embodiments.

The embodiment of the application provides a data processing method, which is applied to a server. FIG. 2 is a schematic diagram of the implementation process of the data processing method according to the embodiment of the application; as shown in FIG. 2, the method includes:

Step 201: Acquire audio data; recognize the audio data to obtain recognized text;

Here, the audio data is audio generated when a user in a scene where simultaneous interpretation is applied is giving a speech.

The process by which the server obtains the audio data will be described below.

Here, the client may be provided with or connected to a voice collection module, such as a microphone, through which the voice collection module collects the user's speech content in a scene where simultaneous interpretation is applied to obtain the audio data. A communication connection is established between the client and the server and sent to the server through a wireless communication module. The wireless communication module may be a Bluetooth module, a wireless fidelity (WiFi, Wireless Fidelity) module, or the like.

The specific type of the client is not limited in this application. For example, it may be a smart phone, a personal computer, a notebook computer, a tablet computer, and a portable wearable device.

In an embodiment, after the server obtains the audio data, the method further includes:

Using voice recognition technology to perform voice recognition on the audio data to obtain recognized text.

Wherein, the language of the recognized text is consistent with the language of the user who generates the audio data in the scenario where simultaneous interpretation is applied, that is, the source language.

For example, in a conference scenario where simultaneous interpretation is used, a user gives a speech on how terminals in the communication field perform uplink transmission. The speech content contains two terms "unlicensed spectrum" and "LBT type", and the server obtains the After the content of the speech, you can query additional information matching the term "unlicensed spectrum" from the knowledge graph database, such as the definition of "unlicensed spectrum", that is, unlicensed spectrum refers to shared spectrum, in other words, in different communication systems As long as the communication equipment of the country meets the regulatory requirements set by the country or region on the spectrum, the spectrum can be used, and there is no need to apply for a proprietary spectrum authorization from the government; you can also query the knowledge map database to match the term "LBT type" Additional information, such as the definition of "LBT type", that is, the LBT type includes three types. LBT Category1 (Cat1): The communication device does not need to perform channel detection on the unlicensed spectrum, and the switching gap (switching gap) is immediately transmitted after the end; the switching gap No more than 16μs. LBT Category2 (Cat2): The communication device performs channel detection on the unlicensed spectrum; within a single detection period, the channel is idle and the signal can be sent, and the channel is occupied, the signal cannot be sent; the single detection time is 16us or 25us. LBT Category3 (Cat4): The communication device performs channel detection on the unlicensed spectrum; the length of the channel detection needs to be further determined according to the priority of the transmission service.

It should be noted that after the server obtains the recognized text, it can subsequently search the knowledge graph database based on the recognized text for the source language of the producer of the audio data and contribute to the audio data. The recipient of the audio data understands the additional information of the speech content of the producer of the audio data. In this way, the server can not only provide the speech content to the recipient of the audio data, but also provide additional information that helps to understand the speech content. Information and content are richer.

In an embodiment, after the server obtains the audio data, the method further includes:

Use voice recognition technology to perform voice recognition on the audio data to obtain recognized text;

The recognized text is translated using a preset translation model to obtain the translated text.

Wherein, the language of the translated text is consistent with the language of the receiver of the audio data, that is, the target language. The translation model is used to translate a text in a first language into at least one text in a second language; the first language is different from the second language.

For example, in a conference scenario where simultaneous interpretation is used, a user gives a speech on how the terminal in the communication field performs uplink transmission. The speech content contains two terms "NRU" and "LBT Category", and the server obtains the speech content After that, the speech content is translated according to the target language to obtain the translated text, and then the additional information that matches the "unlicensed spectrum" in the translated text can be queried from the knowledge graph database, such as the definition of "unlicensed spectrum", that is, unlicensed spectrum It refers to the shared spectrum. In other words, the communication devices in different communication systems can use the spectrum as long as they meet the regulatory requirements set by the country or region on the spectrum, without applying for a proprietary spectrum authorization from the government; Query additional information that matches the "LBT type" in the translated text from the knowledge graph database, such as the definition of "LBT type", that is, the LBT type includes: Type LBT Category1 (Cat1): Communication equipment does not need to perform channel detection on unlicensed spectrum , The switching gap (switching gap) is immediately transmitted after the end; the switching gap does not exceed 16μs. LBT Category2 (Cat2): The communication device performs channel detection on the unlicensed spectrum; within a single detection period, the channel is idle and the signal can be sent, and the channel is occupied, the signal cannot be sent; the single detection time is 16us or 25us. LBT Category3 (Cat4): The communication device performs channel detection on the unlicensed spectrum; the length of the channel detection needs to be further determined according to the priority of the transmission service.

It should be noted that, after the server translates the recognized text to obtain the translated text, it can subsequently search the knowledge graph database for the target language corresponding to the recipient of the audio data based on the translated text, and The additional information that helps the receiver of the audio data understand the speech content of the producer of the audio data, so that the receiver of the audio data can reduce the difficulty of understanding the speech content.

Step 202: Extract at least two feature data of the recognized text, and search for additional information associated with each feature data of the at least two feature data from the knowledge graph database;

Here, the additional information may refer to supplementary explanatory information that can help the user fully and accurately understand the content of the speech.

In order to be able to expand the search scope, the knowledge graph database supports searching for additional information that helps understand the speech content of the audio data producer based on the recognized text corresponding to the source language of the audio data producer.

In an embodiment, the extracting at least two feature data of the recognized text includes:

Extract at least two keywords in the recognized text;

For each of the at least two keywords, entity recognition is performed on the corresponding keywords to obtain at least two entity words;

The at least two entity words are used as at least two feature data of the recognized text.

Among them, entity recognition refers to the recognition of entity words with specific meanings in the text, such as entity names, place names, time, organization names, event names, and professional terms.

Here, at least two entity phrases with specific meaning can also be extracted, which is not limited here.

Specifically, the server may use keyword extraction technology to extract keywords from the recognized text to obtain at least two keywords; it may also convert the first text into multiple bit data; Search for bit data whose number of bits meets a preset threshold in the data; use the found bit data that meets the preset condition as the keyword.

The specific situation of entity recognition of the at least two keywords will be described below.

In the first case, using preset rules, entity recognition is performed on the at least two keywords to obtain at least two entity words.

For example, a regular expression is used to search for a keyword matching a preset character string from the at least two keywords, and the searched keyword that matches the preset character string is used as an entity word.

In the second case, the classification model is used to perform entity recognition on the at least two key words to obtain at least two entity words with specific meanings.

For example, a neural network model is used to map the input to the output of the at least two key words to obtain a recognition result; when the recognition result indicates that the corresponding keyword has a specific meaning, the corresponding keyword is used as an entity word.

In the third case, a sequence labeling model is used to perform entity recognition on the at least two keywords to obtain at least two entity words.

For example, using semantic analysis technology to analyze the semantics of the at least keywords to obtain semantic information of related keywords, mark the parts of speech of the at least two keywords according to the obtained semantic information, and select from the marked keywords Keywords with specific meanings are regarded as entity words.

In an example, taking entity words as an example, a schematic diagram describing the implementation process of extracting at least two feature data of the recognized text by the server, as shown in FIG. 3, includes:

Step 1: Extract at least two keywords in the recognized text.

Assume that the recognized text is "September 23, 2019, Huawei President Ren Zhengfei's interview with the host of a German TV station in Beijing by the United States regarding the Meng Wanzhou incident".

The at least two keywords may be "2019, Huawei, Ren Zhengfei, the United States, Meng Wanzhou, events, Beijing";

Step 2: Perform entity recognition on the at least two keywords to obtain at least two entity words.

The at least two entity words may be "Ren Zhengfei, Huawei, Meng Wanzhou".

Here, the at least two entity words may be used as the at least two feature data of the recognized text, and the knowledge graph database may be subsequently searched for additional information that helps to understand the speech content of the creator of the audio data To help the receiver of the audio data quickly understand the content of the speech.

In an example, taking entity words as an example, a schematic diagram describing the implementation process of extracting at least two feature data of the translated text by the description server, as shown in FIG. 4, includes:

Step 1: Translate the recognized text to obtain the translated text.

The recognized text is translated according to the target language of the receiver of the audio data to obtain the translated text.

Step 2: Extract at least two keywords in the recognized text.

Assume that the recognized text is "September 23, 2019, Huawei President Ren Zhengfei's interview with the host of a German TV station in Beijing by the United States regarding the Meng Wanzhou incident".

The at least two keywords may be "2019, Huawei, Ren Zhengfei, the United States, Meng Wanzhou, events, Beijing";

Step 3: Perform entity recognition on the at least two keywords to obtain at least two entity words.

The at least two entity words may be "Ren Zhengfei, Huawei, Meng Wanzhou".

Here, the at least two entity words may be used as the at least two feature data of the recognized text, and the knowledge graph database may be subsequently searched for additional information that helps to understand the speech content of the creator of the audio data To help the receiver of the audio data quickly understand the content of the speech.

In an embodiment, when extracting the at least two feature data, the method further includes:

For each entity word in the at least two entity words, based on a preset rule and a preset neural network model, extract event-related information associated with the corresponding entity word in the recognized text to obtain at least two event-related information;

The at least two entity words and at least two event-related information are used as at least two feature data of the recognized text.

Wherein, the event-related information may be associated with one entity word among the at least two entity words, or may be associated with multiple entity words among the at least two entity words.

In actual application, if the number of the at least two entity words is less than the preset threshold, the entity words found can be combined with preset rules, classification models, and sequence labeling models to extract and correspond from the recognized text. Event-related information associated with entity words.

The specific situation of extracting event-related information from the recognized text in combination with the at least two entity words is described below.

In the first case, a preset rule is used to extract at least two event-related information from the recognized text in combination with the at least two entity words.

For example, a regular expression is used to search for a first text containing a preset character string from the recognized text, to determine whether the first text is related to each of the at least two entity words, when When the first text is associated with a certain entity word of the at least two entity words, the text information corresponding to the first text is used as the event-related information associated with the entity word. Among them, the text corresponding to the preset character string may be "event".

In the second case, a classification model is used to extract at least two event-related information from the recognized text in combination with the at least two entity words.

For example, a neural network model is used to map the recognized text from input to output to obtain the recognition result; when the recognition result represents the first text containing a preset character string in the recognized text, it is determined that the first Whether the text is associated with each of the at least two entity words, when the first text is associated with an entity word in the at least two entity words, the first The text information corresponding to the text serves as the event-related information associated with the entity word.

In an example, taking entity words and event-related information as examples, a schematic diagram describing the implementation process of extracting at least two characteristic data of the recognized text by the server, as shown in FIG. 5, includes:

Step 1: Extract at least two keywords in the recognized text.

Assume that the recognized text is "September 23, 2019, Huawei President Ren Zhengfei's interview with the host of a German TV station in Beijing by the United States regarding the Meng Wanzhou incident".

The at least two keywords may be "2019, Huawei, Ren Zhengfei, the United States, Meng Wanzhou, events, Beijing";

Step 2: Perform entity recognition on the at least two keywords to obtain at least two entity words.

The at least two entity words may be "Ren Zhengfei, Huawei, Meng Wanzhou".

Step 3: Use the at least two entity words to extract event-related information associated with the corresponding entity words in the recognized text to obtain at least two event-related information.

For the entity word "Ren Zhengfei", the event-related information extracted from the recognized text is "Meng Wanzhou event"; for the entity word "Huawei", the event-related information extracted from the recognized text is "Meng Wanzhou event" "; For the entity word "Meng Wanzhou", the event-related information extracted from the recognized text is "Meng Wanzhou event".

Here, the at least two entity words and the at least two event-related information may be used as at least two feature data of the recognized text, and subsequent searches may be made from the knowledge graph database to help understand the audio data The additional information of the speech content of the producer helps the receiver of the audio data to understand the background of the speech content.

In an example, taking entity words and event-related information as an example, a schematic diagram describing the implementation process of extracting at least two feature data of the translated text by the server, as shown in FIG. 6, includes:

Step 1: Translate the recognized text to obtain the translated text.

The recognized text is translated according to the target language of the receiver of the audio data to obtain the translated text.

Step 2: Extract at least two keywords in the translated text.

Assume that the translated text is "September 23, 2019, Huawei President Ren Zhengfei's interview with the host of a German TV station in Beijing by the United States regarding the Meng Wanzhou incident".

The at least two keywords may be "2019, Huawei, Ren Zhengfei, the United States, Meng Wanzhou, events, Beijing";

Step 3: Perform entity recognition on the at least two keywords to obtain at least two entity words.

The at least two entity words may be "Ren Zhengfei, Huawei, Meng Wanzhou".

Step 4: Use the at least two entity words to extract event-related information associated with the corresponding entity words in the translated text to obtain at least two event-related information.

In an embodiment, searching for additional information associated with each of the at least two feature data from the knowledge graph database includes:

For each feature data in the at least two feature data, determine the first index identifier corresponding to the corresponding feature data;

Searching for at least two knowledge nodes corresponding to the first index identifier from a knowledge graph database; the knowledge graph database stores the relationship between the index identifier and the knowledge node;

Combining the context of the recognized text, exclude from the at least two knowledge nodes a first knowledge node that meets a preset condition, so as to determine at least two second knowledge nodes that match the context of the recognized text;

Acquiring additional information corresponding to the at least two second knowledge nodes.

Wherein, the context refers to the context of the recognized text.

In an example, at least two entity words are taken as an example to describe a schematic diagram of the implementation process of the server searching for the additional information corresponding to the recognized text, as shown in FIG. 7, including:

Step 1: Determine the first index identifier corresponding to each of the at least two entity words.

Assuming that the at least two entity words are "Ren Zhengfei, Huawei, and Meng Wanzhou", the corresponding first index identifiers are 01, 02, and 03, respectively.

Step 2: Use the first index identifier to search for at least two knowledge nodes corresponding to the at least two entity words from the knowledge graph database.

The node identifiers of the knowledge nodes corresponding to 01 are A and B;

The node identifier of the knowledge node corresponding to 02 is C;

The node identifiers of the knowledge nodes corresponding to 03 are D and E.

Step 3: Combining the context of the recognized text, disambiguate the knowledge information corresponding to the at least two knowledge nodes to obtain at least two second knowledge nodes that match the context of the recognized text.

Disambiguation is performed on the knowledge information corresponding to the at least two knowledge nodes, and the obtained at least two second knowledge nodes are:

The node identifier of the knowledge node corresponding to 01 is B;

The node identifier of the knowledge node corresponding to 02 is C;

The node identifier of the knowledge node corresponding to 03 is E.

Step 4: Acquire knowledge information corresponding to the at least two second knowledge nodes.

The additional information may be knowledge information corresponding to the at least two second knowledge nodes.

In an example, at least two entity words are taken as an example to describe a schematic diagram of the implementation process of the server searching for additional information corresponding to the translated text, as shown in FIG. 8, including:

Step 1: Determine the first index identifier corresponding to each of the at least two entity words.

Assuming that the at least two entity words are "Ren Zhengfei, Huawei, and Meng Wanzhou", the corresponding first index identifiers are 01, 02, and 03, respectively.

Step 2: Use the first index identifier to search for at least two knowledge nodes corresponding to the at least two entity words from the knowledge graph database.

The node identifiers of the knowledge nodes corresponding to 01 are A and B;

The node identifier of the knowledge node corresponding to 02 is C;

The node identifiers of the knowledge nodes corresponding to 03 are D and E.

Step 3: Combining the context of the translated text, disambiguate the knowledge information corresponding to the at least two knowledge nodes to obtain at least two second knowledge nodes that match the context of the translated text.

Disambiguation is performed on the knowledge information corresponding to the at least two knowledge nodes, and the obtained at least two second knowledge nodes are:

The node identifier of the knowledge node corresponding to 01 is B;

The node identifier of the knowledge node corresponding to 02 is C;

The node identifier of the knowledge node corresponding to 03 is E.

Step 4: Acquire knowledge information corresponding to the at least two second knowledge nodes.

The additional information may be knowledge information corresponding to the at least two second knowledge nodes.

In an embodiment, the using the at least two second knowledge nodes to obtain additional information includes:

Obtain at least two pieces of knowledge information by using the at least two second knowledge nodes;

Sorting the at least two pieces of knowledge information according to the importance level to obtain a sorting result;

Selecting knowledge information whose importance level is greater than a preset level threshold from the ranking result;

The selected knowledge information is used as the additional information.

Among them, the importance level can be determined according to the user's historical access times to the knowledge information. For example, if the user's historical access times to a certain knowledge information is between 0 and 1000, the importance level corresponding to the knowledge information is the first Three levels, the lowest level of characterization; if the user’s historical access times to a certain knowledge information is between 1,000 and 2,000, the importance level corresponding to the knowledge information is the second level; if the user’s historical access times to a certain knowledge information Between 2000 and 5000, the importance level corresponding to the knowledge information is the first level, which represents the highest level.

In an example, at least two entity words are taken as an example to describe a schematic diagram of the implementation process of the server searching for the additional information corresponding to the recognized text, as shown in FIG. 9, including:

Step 1: Use the at least two entity words to search for at least two knowledge nodes corresponding to the at least two entity words from the knowledge graph database.

Assuming that the at least two entity words are "Ren Zhengfei, Huawei, and Meng Wanzhou", the corresponding first index identifiers are 01, 02, and 03, respectively. The node ID of the knowledge node corresponding to 01 is A and B; the node ID of the knowledge node corresponding to 02 is C; the node ID of the knowledge node corresponding to 03 is D, E.

Step 2: Combining the context of the recognized text, disambiguate the knowledge information corresponding to the at least two knowledge nodes to obtain at least two second knowledge nodes that match the context of the recognized text.

The at least two second knowledge nodes are: the node identifier of the knowledge node corresponding to 01 is B; the node identifier of the knowledge node corresponding to 02 is C; and the node identifier of the knowledge node corresponding to 03 is E.

Step 3: Sort the knowledge information corresponding to the at least two second knowledge nodes according to the importance level, and use the knowledge information of the knowledge nodes with the importance level above the second level as additional information.

Assuming that the importance level of the knowledge node with the node identification B is the second level, the importance level of the knowledge node with the node identification C is the second level, and the importance level of the knowledge node with the node identification E is the third level, then The knowledge information of nodes whose importance level is above the second level can be identified as knowledge nodes corresponding to B and C as additional information.

In an example, at least two entity words are taken as an example to describe a schematic diagram of the implementation process of the server searching for additional information corresponding to the translated text, as shown in FIG. 10, including:

Step 1: Use the at least two entity words to search for at least two knowledge nodes corresponding to the at least two entity words from the knowledge graph database.

Assuming that the at least two entity words are "Ren Zhengfei, Huawei, and Meng Wanzhou", the corresponding first index identifiers are 01, 02, and 03, respectively. The node ID of the knowledge node corresponding to 01 is A and B; the node ID of the knowledge node corresponding to 02 is C; the node ID of the knowledge node corresponding to 03 is D, E.

Step 2: Combining the context of the translated text, disambiguate the knowledge information corresponding to the at least two knowledge nodes to obtain at least two second knowledge nodes that match the context of the recognized text.

The at least two second knowledge nodes are: the node identifier of the knowledge node corresponding to 01 is B; the node identifier of the knowledge node corresponding to 02 is C; and the node identifier of the knowledge node corresponding to 03 is E.

Step 3: Sort the knowledge information corresponding to the at least two second knowledge nodes according to the importance level, and use the knowledge information of the knowledge nodes with the importance level above the second level as additional information.

Assuming that the importance level of the knowledge node with the node identification B is the second level, the importance level of the knowledge node with the node identification C is the second level, and the importance level of the knowledge node with the node identification E is the third level, then The knowledge information of nodes whose importance level is above the second level can be identified as knowledge nodes corresponding to B and C as additional information.

It should be noted that removing knowledge information with a lower importance level from the knowledge information corresponding to at least two knowledge nodes found can avoid providing redundant additional information to the receiver of the audio data.

The knowledge graph database not only supports searching for additional information that helps to understand the speech content of the producer of the audio data based on the recognized text corresponding to the source language of the producer of the audio data; it also supports searching for additional information based on the source language of the producer of the audio data; The translated text corresponding to the target language of the recipient of the data is searched for additional information that helps to understand the content of the speech of the producer of the audio data.

Here, the process of using the translated text to find the additional information from the knowledge graph database is similar to the process of using the recognized text to find the additional information from the knowledge graph database.

Step 203: Use the found additional information and the recognized text to generate a simultaneous interpretation result; output the simultaneous interpretation result; the simultaneous interpretation result is used for presentation on the first terminal when the audio data is played.

Here, the simultaneous interpretation result is used for presentation on the first terminal when the audio data is played, which may mean that the simultaneous interpretation result is presented while the audio data is being played, that is, the data processing method can be applied to simultaneous interpretation. Scenes.

Here, when the language of the recognized text and the language of the recipient of the audio data belong to the same language, the additional information found and the recognized text are used to generate a simultaneous interpretation result.

Here, when the language of the recognized text and the language of the recipient of the audio data belong to different languages, the recognition result is translated to obtain the translated text; the additional information found and the translated text are used to generate the same Pass the result.

Wherein, the additional information may be searched from the knowledge graph database based on the recognized text, or may be searched from the knowledge graph database based on the translated text.

In an embodiment, when generating simultaneous interpretation results, the method further includes:

Based on a preset filtering rule, exclude the first information that meets a preset condition in the additional information;

The second information except the first information in the additional information and the recognized text are used to generate a simultaneous interpretation result.

Wherein, the preset filtering rule may be that the number of words contained in the additional information is greater than a preset word number threshold, for example, counting the number of words contained in the additional information; when the counted number of words exceeds 100 words, deleting important information in the additional information The first information with a lower sex level ensures that the number of additional information is within 100 words.

In actual application, the server may use the form of audio stream, use the additional information and the translated text, to generate and output the simultaneous interpretation result.

In an embodiment, the outputting the simultaneous interpretation result includes:

Perform speech synthesis on the simultaneous interpretation result to obtain simultaneous interpretation audio data;

The simultaneous interpretation audio data is sent to the first terminal; the simultaneous interpretation audio data is used for playback by the first terminal.

Here, the simultaneous interpretation audio data may be played through the headset of the first terminal, so as to help the user who uses the first terminal understand the speech content of the producer of the audio data.

In actual applications, the server can also use structured formats such as tables, graphics, and web pages to generate simultaneous interpretation results based on the additional information and the translated text.

In an embodiment, when the language of the recognized text and the language of the recipient of the audio data belong to the same language, the output of the simultaneous interpretation result includes:

The simultaneous interpretation result is sent to the display screen associated with the first terminal; the simultaneous interpretation result is used for the first terminal to display the additional information in the first display frame of the display screen, and the The recognized text is displayed in the second display box of the display screen.

In an embodiment, when the language of the recognized text and the language of the recipient of the audio data belong to different languages, the output of the simultaneous interpretation result includes:

The simultaneous interpretation result is sent to the display screen associated with the first terminal; the simultaneous interpretation result is used for the first terminal to display the additional information in the first display frame of the display screen, and the The translated text corresponding to the recognized text is displayed in the second display box of the display screen.

Here, the simultaneous interpretation of the audio data may be performed through the display screen of the first terminal, so as to help the user who uses the first terminal to understand the speech content of the producer who uses the audio data.

FIG. 11 is a schematic diagram of the server outputting the simultaneous interpretation result. As shown in FIG. 11, the first terminal may place the additional information in the simultaneous interpretation result in the first terminal corresponding to the position above the display screen associated with the first terminal. For display in a display box, the top position is directly above the center, top center and right, top center and left, etc.; when the language of the recognized text and the language of the receiver of the audio data belong to the same language, all The first terminal may display the recognized text in the simultaneous interpretation result in a second display box corresponding to a lower position of the display screen, where the lower position is centered directly below, centered below to the right, centered below Lean to the left, etc.; the display mode can include at least one of the following: pictures, multimedia, text boxes, and rich text boxes.

In an example, taking the search for additional information based on the recognized text as an example, a schematic diagram of the implementation process of the server generating and outputting simultaneous interpretation results is described, as shown in Figure 12, including:

Step 1: The client terminal collects the speaker's audio data and sends it to the server.

In a conference scenario where simultaneous interpretation is used, the client uses a microphone to collect the content of the lecturer's speech, obtain audio data, and send it to the server.

Step 2: The server performs voice recognition on the audio data to obtain recognized text corresponding to the source language.

Step 3: The server searches the knowledge graph database for additional information matching the recognized text based on the recognized text.

The server extracts at least two entity words in the recognized text; searches for knowledge nodes corresponding to the at least two entity words from the knowledge graph database; in the search process, combining the context of the recognized text, from the at least The first knowledge node that meets the preset condition is excluded from the two knowledge nodes, so as to determine at least two second knowledge nodes that match the context of the recognized text, and obtain additional corresponding to the at least two second knowledge nodes. information.

Here, the additional information may specifically be entity words/phrases, that is, additional information that supplements difficult-to-understand content. The difficult-to-understand content may include technical terms, person names, quoted event names, etc.; it may also be an entity Explanation of words/phrases, such as definitions of terms, example information, introduction of characters, explanatory information, elements of events, and related impact information, etc.

Step 4: The server generates a simultaneous interpretation result based on the additional information and the recognized text.

When the language of the recognized text and the language of the receiver of the audio data belong to the same language, the additional information found and the recognized text are used to generate a simultaneous interpretation result.

Here, when the language of the recognized text and the language of the recipient of the audio data belong to different languages, the recognition result is translated to obtain the translated text; the additional information found and the translated text are used to generate the same Pass the result.

Step 5: The server performs speech synthesis on the simultaneous interpretation result to obtain simultaneous interpretation audio data.

Here, the server sends the generated simultaneous interpretation audio data to the first terminal, and the first terminal broadcasts the simultaneous interpretation audio data through a headset.

Step 6: The server sends the simultaneous interpretation result to the display screen associated with the first terminal.

When the language of the recognized text and the language of the recipient of the audio data belong to the same language, the first terminal displays the additional information in the first display frame of the display screen, and displays the recognized text The text is displayed in the second display box of the display screen;

When the language of the recognized text and the language of the receiver of the audio data belong to different languages, the first terminal displays the additional information in the first display frame of the display screen, and displays the recognition The translated text corresponding to the text is displayed in the second display box of the display screen.

In an example, taking the search for additional information based on translated text as an example, a schematic diagram of the implementation process of the server generating and outputting simultaneous interpretation results is described, as shown in Figure 13, including:

Step 1: The client terminal collects the speaker's audio data and sends it to the server.

In a conference scenario where simultaneous interpretation is used, the client uses a microphone to collect the content of the lecturer's speech, obtain audio data, and send it to the server.

Step 2: The server performs voice recognition on the audio data to obtain recognized text corresponding to the source language.

Step 3: The server translates the recognized text to obtain the translated text.

Step 4: The server searches the knowledge graph database for additional information matching the recognized text based on the translated text.

The server extracts at least two entity words in the translated text; searches for knowledge nodes corresponding to the at least two entity words from the knowledge graph database; in the search process, combining the context of the recognized text, from the at least The first knowledge node that meets the preset condition is excluded from the two knowledge nodes, so as to determine at least two second knowledge nodes that match the context of the recognized text, and obtain additional corresponding to the at least two second knowledge nodes. information.

Here, the additional information may specifically be entity words/phrases, that is, additional information that supplements difficult-to-understand content. The difficult-to-understand content may include technical terms, person names, quoted event names, etc.; it may also be an entity Explanation of words/phrases, such as definitions of terms, example information, introduction of characters, explanatory information, elements of events, and related impact information, etc.

Step 5: The server generates a simultaneous interpretation result based on the additional information and the translated text.

Step 6: The server performs speech synthesis on the simultaneous interpretation result to obtain simultaneous interpretation audio data.

Here, the server sends the generated simultaneous interpretation audio data to the first terminal, and the first terminal broadcasts the simultaneous interpretation audio data through a headset.

Step 7: The server sends the simultaneous interpretation result to the display screen associated with the first terminal.

The first terminal displays the additional information in a first display frame of the display screen, and displays the translated text in a second display frame of the display screen.

It should be understood that the order of the steps described in the above embodiments does not mean the order of execution. The order of execution of the processes should be determined by their functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

The data processing method, device, and storage medium provided by the embodiments of the application obtain audio data; recognize the audio data to obtain recognized text; extract at least two feature data of the recognized text, and search and search from the knowledge graph database Additional information associated with each feature data in the at least two feature data; use the additional information found and the recognition text to generate a simultaneous interpretation result; output the simultaneous interpretation result, which is used in the playback When the audio data is presented in the first terminal, the user is provided with additional information of the speech content of the producer of the audio data, which can help the user to fully and accurately understand the speech content of the producer of the audio data, and reduce the user’s dissatisfaction. The difficulty of comprehension of the speech content of the producer of the audio data.

In order to implement the data processing method of the embodiment of the present application, the embodiment of the present application also provides a data processing device. FIG. 14 is a schematic diagram of the composition structure of a data processing device according to an embodiment of the application; as shown in FIG. 14, the data processing device includes:

The obtaining unit 141 is configured to obtain audio data;

The first processing unit 142 is configured to recognize the audio data to obtain recognized text; extract at least two feature data of the recognized text, and search for each feature of the at least two feature data from the knowledge graph database. Additional information associated with the data;

The second processing unit 143 is configured to use the found additional information and the recognized text to generate a simultaneous interpretation result;

The output unit 144 is configured to output the simultaneous interpretation result; the simultaneous interpretation result is used for presentation on the first terminal when the audio data is played.

In one embodiment, the first processing unit 142 is configured to extract at least two key words in the recognized text; for each key word in the at least two key words, perform entity recognition on the corresponding key word to obtain At least two entity words; use the at least two entity words as at least two feature data of the recognized text.

In an embodiment, the first processing unit 142 is configured to extract, for each entity word of the at least two entity words, based on a preset rule and a preset neural network model, the recognition text is associated with the corresponding entity word At least two event-related information are obtained from the event-related information of the, and the at least two entity words and the at least two event-related information are used as at least two feature data of the recognized text.

In an embodiment, the first processing unit 142 is configured to translate the recognized text to obtain the translated text.

In an embodiment, the first processing unit 142 is configured to extract at least two key words in the translated text; for each key word in the at least two key words, perform entity recognition on the corresponding key word to obtain At least two entity words; use the at least two entity words as at least two feature data of the recognized text.

In an embodiment, the first processing unit 142 is configured to extract, for each entity word of the at least two entity words, based on a preset rule and a preset neural network model, the translation text is associated with the corresponding entity word At least two event-related information are obtained from the event-related information of the, and the at least two entity words and the at least two event-related information are used as at least two feature data of the recognized text.

In an embodiment, the first processing unit 142 is configured to determine, for each feature data of the at least two feature data, a first index identifier corresponding to the corresponding feature data; and search for the first index identifier corresponding to the knowledge graph database. Identify the corresponding at least two knowledge nodes; the knowledge graph database stores the relationship between the index identifier and the knowledge node; in combination with the context of the recognized text, exclude from the at least two knowledge nodes those that meet preset conditions The first knowledge node determines at least two second knowledge nodes that match the context of the recognized text; and obtains additional information corresponding to the at least two second knowledge nodes.

In an embodiment, the first processing unit 142 is configured to use the at least two second knowledge nodes to obtain at least two pieces of knowledge information; to sort the at least two pieces of knowledge information according to the importance level to obtain the sorting result ; Select knowledge information with an importance level greater than a preset level threshold from the ranking result; use the selected knowledge information as the additional information.

In an embodiment, the first processing unit 142 is configured to exclude first information that meets a preset condition from the additional information based on a preset filtering rule; use the additional information in addition to the first information The second information of and the recognized text generate a simultaneous interpretation result.

In an embodiment, the output unit 144 is configured to perform speech synthesis on the simultaneous interpretation result to obtain simultaneous interpretation audio data; send the simultaneous interpretation audio data to the first terminal; the simultaneous interpretation audio data is used for supply The first terminal plays.

In an embodiment, the output unit 144 is configured to send the simultaneous interpretation result to a display screen associated with the first terminal; the simultaneous interpretation result is used for the first terminal to display the additional information on the The first display box of the display screen is displayed, and the recognized text is displayed in the second display box of the display screen.

In an embodiment, the output unit 144 is configured to send the simultaneous interpretation result to a display screen associated with the first terminal; the simultaneous interpretation result is used for the first terminal to display the additional information on the The first display box of the display screen is displayed, and the translated text corresponding to the recognized text is displayed in the second display box of the display screen.

In practical applications, the acquisition unit 141 and the output unit 144 can be implemented through a communication interface; the first processing unit 142 and the second processing unit 143 can both be implemented by a processor in the device.

It should be noted that when the device provided in the above embodiment performs data processing, only the division of the above-mentioned program modules is used as an example. In practical applications, the above-mentioned processing can be allocated by different program modules as needed, that is, the terminal The internal structure is divided into different program modules to complete all or part of the processing described above. In addition, the device provided in the foregoing embodiment and the data processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

Based on the hardware implementation of the above-mentioned equipment, an embodiment of the present application also provides a data processing apparatus. FIG. 15 is a schematic diagram of the hardware composition structure of the data processing apparatus according to an embodiment of the application. As shown in FIG. 15, the data processing apparatus 150 includes a memory 153. , The processor 152 and a computer program stored on the memory 153 and capable of running on the processor 152; the processor 152 located in the data processing device executes the program to implement the method provided by one or more technical solutions on the data processing device side .

Specifically, when the processor 152 located in the data processing device 150 executes the program, it realizes: acquiring audio data; the audio data is collected by the first terminal; translating the audio data to obtain the recognized text; extracting all the audio data; The at least two feature data of the recognized text are searched for additional information associated with each feature data of the at least two feature data from the knowledge graph database; the simultaneous interpretation result is generated by using the found additional information and the recognized text ; Output the result of the simultaneous interpretation; The result of the simultaneous interpretation is output synchronously with the collection of audio data.

It should be noted that the specific steps implemented when the processor 152 of the data processing device 150 executes the program have been described in detail above, and will not be repeated here.

It can be understood that the data processing device further includes a communication interface 151; various components in the data processing device are coupled together through the bus system 154. It can be understood that the bus system 154 is configured to implement connection and communication between these components. In addition to the data bus, the bus system 154 also includes a power bus, a control bus, and a status signal bus.

It can be understood that the memory 153 in this embodiment may be a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM, Read Only Memory), a programmable read-only memory (PROM, Programmable Read-Only Memory), an erasable programmable read-only memory (EPROM, Erasable Programmable Read- Only Memory, Electrically Erasable Programmable Read-Only Memory (EEPROM), Ferromagnetic Random Access Memory (FRAM), Flash Memory, Magnetic Surface Memory , CD-ROM, or CD-ROM (Compact Disc Read-Only Memory); magnetic surface memory can be magnetic disk storage or tape storage. The volatile memory may be a random access memory (RAM, Random Access Memory), which is used as an external cache. By way of exemplary but not restrictive description, many forms of RAM are available, such as static random access memory (SRAM, Static Random Access Memory), synchronous static random access memory (SSRAM, Synchronous Static Random Access Memory), and dynamic random access memory. Memory (DRAM, Dynamic Random Access Memory), Synchronous Dynamic Random Access Memory (SDRAM, Synchronous Dynamic Random Access Memory), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM, Double Data Rate Synchronous Dynamic Random Access Memory), enhanced Type synchronous dynamic random access memory (ESDRAM, Enhanced Synchronous Dynamic Random Access Memory), synchronous connection dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), direct memory bus random access memory (DRRAM, Direct Rambus Random Access Memory) ). The memories described in the embodiments of the present application are intended to include, but are not limited to, these and any other suitable types of memories.

The method disclosed in the foregoing embodiments of the present application may be applied to the processor 152 or implemented by the processor 152. The processor 152 may be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 152 or instructions in the form of software. The aforementioned processor 152 may be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and the like. The processor 152 may implement or execute the methods, steps, and logical block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or any conventional processor or the like. Combining the steps of the method disclosed in the embodiments of the present application, it may be directly embodied as execution and completion by a hardware decoding processor, or execution and completion by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, and the storage medium is located in a memory. The processor 152 reads the information in the memory and completes the steps of the foregoing method in combination with its hardware.

The embodiment of the present application also provides a storage medium, which is specifically a computer storage medium, and more specifically, a computer-readable storage medium. Stored thereon are computer instructions, that is, a computer program, which is a method provided by one or more technical solutions on the data processing device side when the computer instructions are executed by a processor.

In the several embodiments provided in this application, it should be understood that the disclosed method and smart device can be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division, and there may be other divisions in actual implementation, such as: multiple units or components can be combined, or It can be integrated into another system, or some features can be ignored or not implemented. In addition, the coupling, or direct coupling, or communication connection between the components shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical, mechanical or other forms. of.

The units described above as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the embodiments of the present application may all be integrated into a second processing unit, or each unit may be individually used as a unit, or two or more units may be integrated into one unit; The above-mentioned integrated unit may be implemented in the form of hardware, or may be implemented in the form of hardware plus software functional units.

A person of ordinary skill in the art can understand that all or part of the steps of the above method embodiments can be implemented by a program instructing relevant hardware. The foregoing program can be stored in a computer readable storage medium, and when the program is executed, it is executed. Including the steps of the foregoing method embodiment; and the foregoing storage medium includes: various media that can store program codes, such as a mobile storage device, ROM, RAM, magnetic disk, or optical disk.

Alternatively, if the aforementioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it may also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product in essence or a part that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for A computer device (which may be a personal computer, a data processing device, or a network device, etc.) executes all or part of the methods described in the various embodiments of the present application. The aforementioned storage media include: removable storage devices, ROM, RAM, magnetic disks, or optical disks and other media that can store program codes.

It should be noted that: "first", "second", etc. are used to distinguish similar objects, and not necessarily used to describe a specific sequence or sequence.

In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

The above are only specific implementations of this application, but the scope of protection of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Should be covered within the scope of protection of this application.

Claims (11)

A data processing method, including: Obtain audio data; Recognizing the audio data to obtain recognized text; Extracting at least two feature data of the recognized text, and searching a knowledge graph database for additional information associated with each feature data of the at least two feature data; Use the additional information found and the recognized text to generate simultaneous interpretation results; The simultaneous interpretation result is output; the simultaneous interpretation result is used for presentation on the first terminal when the audio data is played. The method according to claim 1, wherein said extracting at least two characteristic data of said recognized text comprises: Extract at least two keywords in the recognized text; For each keyword in the at least two keyword terms, entity recognition is performed on the corresponding keyword to obtain at least two entity terms with specific meaning; The at least two entity words are used as at least two feature data of the recognized text. The method according to claim 2, wherein when extracting the at least two characteristic data, the method further comprises: For each entity word in the at least two entity words, based on a preset rule and a preset neural network model, extract event-related information associated with the corresponding entity word in the recognized text to obtain at least two event-related information; The at least two entity words and at least two event-related information are used as at least two feature data of the recognized text. The method according to any one of claims 1 to 3, wherein searching for additional information associated with each of the at least two feature data from a knowledge graph database comprises: For each feature data in the at least two feature data, determine the first index identifier corresponding to the corresponding feature data; Searching for at least two knowledge nodes corresponding to the first index identifier from a knowledge graph database; the knowledge graph database stores the relationship between the index identifier and the knowledge node; Combining the context of the recognized text, exclude from the at least two knowledge nodes a first knowledge node that meets a preset condition, so as to determine at least two second knowledge nodes that match the context of the recognized text; Acquiring additional information corresponding to the at least two second knowledge nodes. The method according to claim 4, wherein said acquiring additional information corresponding to said at least two second knowledge nodes comprises: Obtain at least two pieces of knowledge information by using the at least two second knowledge nodes; Sorting the at least two pieces of knowledge information according to the importance level to obtain a sorting result; Selecting knowledge information whose importance level is greater than a preset level threshold from the ranking result; The selected knowledge information is used as the additional information. The method according to claim 4, wherein when generating simultaneous interpretation results, the method further comprises: Based on a preset filtering rule, exclude the first information that meets a preset condition in the additional information; The second information except the first information in the additional information and the recognized text are used to generate a simultaneous interpretation result. The method according to claim 1, wherein said outputting said simultaneous interpretation result comprises: Perform speech synthesis on the simultaneous interpretation result to obtain simultaneous interpretation audio data; The simultaneous interpretation audio data is sent to the first terminal; the simultaneous interpretation audio data is used for playback by the first terminal. The method according to claim 1, wherein said outputting said simultaneous interpretation result comprises: The simultaneous interpretation result is sent to a display screen associated with the first terminal; the simultaneous interpretation result is used for the first terminal to display the additional information in the first display frame of the display screen. A data processing device includes: The obtaining unit is configured to obtain audio data; The first processing unit is configured to recognize the audio data to obtain recognized text; extract at least two feature data of the recognized text, and search for each feature data of the at least two feature data from the knowledge graph database Associated additional information; The second processing unit is configured to use the found additional information and the recognized text to generate a simultaneous interpretation result; The output unit is configured to output the simultaneous interpretation result; the simultaneous interpretation result is used for presentation on the first terminal when the audio data is played. A data processing device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor. The processor implements the steps of the method according to any one of claims 1 to 7 when the processor executes the program . A storage medium having computer instructions stored thereon, which implement the steps of the method described in any one of claims 1 to 8 when the instructions are executed by a processor.

Images