CN118760359A - Multi-modal multi-task intelligent interaction method, device, electronic device and storage medium - Google Patents

Abstract

The present invention relates to a multimodal multitask intelligent interaction method, device, electronic device and storage medium. The method comprises: obtaining multimodal data, and preprocessing and feature extracting the multimodal data to construct a multimodal knowledge base. Receive input multimodal information, and identify the current user and usage scenario based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario. Use multiple tasks as input to the Transformer multitask model to form a corresponding multi-head model of the multiple tasks, and fuse the features of different tasks to establish a multimodal understanding framework to output recognition results corresponding to the multiple tasks. Select a corresponding interaction strategy based on the recognition result, generate corresponding interaction content according to the interaction strategy, and visualize the interaction content. The method introduces more interaction methods, is close to the habits and needs of users, and can complete interaction tasks more quickly and accurately.

Description

Multi-modal multi-task intelligent interaction method, device, electronic device and storage medium

Technical Field

The present invention relates to the field of human-computer interaction technology, and in particular to a multi-modal multi-task intelligent interaction method, device, electronic device and storage medium.

Background Art

With the continuous development of science and technology, people's requirements for intelligent interaction are getting higher and higher. Traditional single-modal intelligent interaction can no longer meet people's needs. There are mainly the following problems:

(1) Limited interaction capabilities and accuracy issues. Due to technical limitations, single-modal intelligent interaction may have difficulty understanding the user's intentions and context, resulting in low accuracy in the interaction content. For example, communication between humans and machines is more easily affected by factors such as the environment and noise, making it difficult to identify the user's intentions.

(2) The interactive content is mostly basic and mechanical dialogue. Most intelligent interactive systems can only provide services within a relatively limited task domain, so they can only conduct simple question-and-answer dialogues for specific issues and tasks.

(3) Low intelligence. For example, most intelligent interactions can only accept text and voice input, which limits the way users interact with the system to a certain extent, making it difficult for the interactive system to learn users' personalized needs and preferences, and thus unable to provide personalized services for different users.

Therefore, traditional intelligent interaction methods are limited to interacting with users through text or voice, which makes the human-computer interaction efficiency low in different interaction scenarios, the system interaction is easily affected by the environment, the interaction accuracy is low, and it is difficult to meet the interaction needs in multimodal interaction scenarios.

Summary of the invention

Based on this, it is necessary to provide a multimodal multi-task intelligent interaction method, device, electronic device and storage medium that can improve human-computer interaction efficiency and interaction accuracy in various interaction scenarios in response to the above technical problems.

The present invention provides a multi-modal multi-task intelligent interaction method, the method comprising:

Acquiring multimodal data, and performing preprocessing and feature extraction on the multimodal data to construct a multimodal knowledge base;

Receiving input multimodal information, and identifying a current user and a usage scenario based on the multimodal information to obtain a plurality of tasks corresponding to the current user and the usage scenario;

The multiple tasks are used as inputs of the Transformer multi-task model to form the multiple tasks into corresponding multi-head models, and the features of different tasks are information-fused to establish a multimodal understanding framework to output recognition results corresponding to the multiple tasks;

Selecting a corresponding interaction strategy based on the recognition result, generating corresponding interaction content according to the interaction strategy and visually displaying the interaction content;

Among them, the multimodal data includes text data, audio data, video data and image data, the preprocessing includes data cleaning and clustering algorithm processing, the multimodal information includes text information, audio information, video information and image information actively or passively input by the user, the multiple tasks include interaction intention recognition tasks, emotion recognition tasks and entity recognition tasks, the interaction strategies include task-based interaction, knowledge-based interaction and open domain chat interaction, and the interaction content includes text, pictures and voice.

In one embodiment, the acquiring of multimodal data, and preprocessing and feature extraction of the multimodal data to construct a multimodal knowledge base includes:

Performing corpus cleaning on the text data, deleting useless data and duplicate data in the text data through clustering annotation, and removing garbled characters and redundant symbols in the text data to obtain preprocessed text data; and

Illegal images in the image data are removed through clustering annotation, and the image data are labeled and annotated, and the image data are adjusted to a uniform size and resolution to obtain preprocessed image data.

In one embodiment, the step of acquiring multimodal data, and preprocessing and extracting features from the multimodal data to construct a multimodal knowledge base further includes:

Performing video segmentation on the video data to segment the video data into a plurality of video segments, and extracting a plurality of video frames from the video segments to perform feature extraction on the video frames; and

The audio data is denoised, and the audio data is segmented into multiple audio segments according to a set rule, so as to extract features from the multiple audio segments.

In one embodiment, the step of acquiring multimodal data, and preprocessing and extracting features from the multimodal data to construct a multimodal knowledge base further includes:

Performing feature extraction on the preprocessed multimodal data by using a convolutional neural network to extract attribute features and feature vectors in the multimodal data, wherein the attribute features and feature vectors are used to perform similarity analysis and data classification on the multimodal data;

The multimodal data is stored in the multimodal knowledge base including a relational database and a graph database according to the similarity analysis results and data classification results of the attribute features and feature vectors, and indexes between different modal data are established.

In one embodiment, the receiving of input multimodal information and identifying a current user and a usage scenario based on the multimodal information to obtain a plurality of tasks corresponding to the current user and the usage scenario includes:

Based on the multimodal information input by different users and different scenarios, a user portrait knowledge base and a scene portrait knowledge base are established as candidate sets, and a portrait matching unit is used to generate a corresponding input representation according to the multimodal information input by the current user;

The matching score corresponding to the multimodal information input by the current user is calculated through the set retrieval matching strategy, and the user portrait matching vector and scene portrait matching vector with the highest score in the candidate set are fed back.

In one embodiment, the multiple tasks are used as inputs of a Transformer multi-task model to form the multiple tasks into corresponding multi-head models, and information fusion is performed on the features of different tasks to establish a multimodal understanding framework to output recognition results corresponding to the multiple tasks, including:

Introducing prompt dialog learning in the decoding layer of the Transformer multi-task model to guide the Transformer multi-task model to generate label results corresponding to different tasks;

The multimodal understanding framework is called to perform recognition processing on different tasks to generate recognition results corresponding to the different tasks, wherein the recognition results have the label results corresponding to each task, and the label results are used to determine the interaction strategies corresponding to the different tasks.

In one embodiment, selecting a corresponding interaction strategy based on the recognition result, generating corresponding interaction content according to the interaction strategy, and visually displaying the interaction content includes:

Acquire a user intention corresponding to the current user based on the recognition result, and feed back corresponding first interaction content in response to the user intention;

It is determined by monitoring whether the first interactive content meets the user needs of the current user, and when the first interactive content does not meet the user needs, the second interactive content is fed back to determine the user intention of the current user.

The present invention also provides a multi-modal multi-task intelligent interaction device, the device comprising:

A knowledge base construction module, used for acquiring multimodal data, and performing preprocessing and feature extraction on the multimodal data to construct a multimodal knowledge base;

An information identification module, used to receive input multimodal information, and identify a current user and a usage scenario based on the multimodal information to obtain a plurality of tasks corresponding to the current user and the usage scenario;

A task identification module is used to use the multiple tasks as inputs of the Transformer multi-task model to form the multiple tasks into corresponding multi-head models, and to fuse information of features of different tasks to establish a multimodal understanding framework to output identification results corresponding to the multiple tasks;

A content interaction module, configured to select a corresponding interaction strategy based on the recognition result, generate corresponding interaction content according to the interaction strategy, and visually display the interaction content;

Among them, the multimodal data includes text data, audio data, video data and image data, the preprocessing includes data cleaning and clustering algorithm processing, the multimodal information includes text information, audio information, video information and image information actively or passively input by the user, the multiple tasks include interaction intention recognition tasks, emotion recognition tasks and entity recognition tasks, the interaction strategies include task-based interaction, knowledge-based interaction and open domain chat interaction, and the interaction content includes text, pictures and voice.

The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the multi-modal multi-task intelligent interaction method as described above is implemented.

The present invention also provides a computer storage medium storing a computer program, wherein when the computer program is executed by a processor, the multi-modal multi-task intelligent interaction method as described above is implemented.

The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the multi-modal multi-task intelligent interaction method as described above is implemented.

The above-mentioned multimodal multi-task intelligent interaction method, device, electronic device and storage medium construct a multimodal knowledge base by acquiring multimodal data, preprocessing and extracting features of the multimodal data. Subsequently, the input multimodal information is received, and the current user and the usage scenario are identified based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario. Multiple tasks are used as the input of the Transformer multi-task model, and the multiple tasks are composed of corresponding multi-head models, and the features of different tasks are information-fused to establish a multimodal understanding framework to output the recognition results corresponding to multiple tasks. Finally, the corresponding interaction strategy is selected based on the recognition result, and the corresponding interaction content is generated according to the interaction strategy and the interaction content is visualized. Multimodal intelligent interaction can introduce more interaction methods, such as images, gestures, touch, facial expressions, eyes, etc., which are closer to the habits and needs of users, while allowing users to complete interaction tasks more quickly and accurately, improving interaction efficiency and user satisfaction. At the same time, multimodal intelligent interaction can also improve the reliability and robustness of interaction. In some noisy environments or inconvenient voice input, users can interact through other methods, avoiding the limitations of existing voice interaction methods and adapting to different interaction scenarios and environments.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

FIG1 is a schematic diagram of a multi-modal multi-task intelligent interaction method provided by the present invention;

FIG2 is a schematic diagram of a multimodal knowledge base construction process of a multimodal multitask intelligent interaction method in a specific embodiment provided by the present invention;

FIG3 is a schematic diagram of a multimodal information recognition process of a multimodal multitask intelligent interaction method in a specific embodiment provided by the present invention;

FIG4 is a schematic diagram of a multi-task learning process of a multi-modal multi-task intelligent interaction method in a specific embodiment of the present invention;

FIG5 is a second flow chart of the multi-modal multi-task intelligent interaction method provided by the present invention;

FIG6 is a third flow chart of the multi-modal multi-task intelligent interaction method provided by the present invention;

FIG7 is a fourth flow chart of the multi-modal multi-task intelligent interaction method provided by the present invention;

FIG8 is a fifth flow chart of the multi-modal multi-task intelligent interaction method provided by the present invention;

FIG9 is a sixth flow chart of the multi-modal multi-task intelligent interaction method provided by the present invention;

FIG10 is a seventh flow chart of the multi-modal multi-task intelligent interaction method provided by the present invention;

FIG11 is a schematic diagram of the structure of a multi-modal multi-task intelligent interaction device provided by the present invention;

FIG. 12 is a diagram showing the internal structure of the electronic device provided by the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

The multi-modal multi-task intelligent interaction method, device, electronic device and storage medium of the present invention are described below in conjunction with Figures 1 to 12.

As shown in FIG1 , in one embodiment, a multi-modal multi-task intelligent interaction method includes the following steps:

Step S110, acquiring multimodal data, and performing preprocessing and feature extraction on the multimodal data to construct a multimodal knowledge base.

Specifically, the server obtains multimodal data such as text data, audio data, video data, and image data, and performs data cleaning and clustering algorithm preprocessing and feature extraction on the multimodal data to complete the construction of the multimodal knowledge base.

Step S120, receiving input multimodal information, and identifying the current user and the usage scenario based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario.

Specifically, the server receives multimodal information such as text information, audio information, video information, and image information actively or passively input by the user, and identifies the current user and usage scenario based on the received multimodal information to obtain multiple tasks corresponding to the current user and usage scenario as input of the Transformer multi-task model.

Step S130, taking multiple tasks as input of the Transformer multi-task model, so as to combine the multiple tasks into a corresponding multi-head model, and fuse the features of different tasks to establish a multimodal understanding framework, so as to output the recognition results corresponding to the multiple tasks.

Specifically, the server takes multiple tasks such as interaction intention recognition tasks, emotion recognition tasks, and entity recognition tasks as the input of the Transformer multi-task model, combines the multiple tasks into a multi-head model, and fuses the features of different tasks to establish a multimodal understanding framework to output the recognition results corresponding to multiple tasks.

Step S140, selecting a corresponding interaction strategy based on the recognition result, generating corresponding interaction content according to the interaction strategy and visually displaying the interaction content.

Specifically, the server selects a corresponding interaction strategy based on the recognition result obtained in step S130, generates corresponding interaction content according to the selected interaction strategy, and visually displays the interaction content to the current user.

Among them, the interaction strategies include task-based interaction, knowledge-based interaction and open-domain chat interaction, and the interaction content includes text, pictures and voice.

The above multimodal multitask intelligent interaction method constructs a multimodal knowledge base by acquiring multimodal data, preprocessing and extracting features of the multimodal data. Subsequently, the input multimodal information is received, and the current user and the usage scenario are identified based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario. Multiple tasks are used as the input of the Transformer multitask model, and the multiple tasks are combined into corresponding multi-head models, and the features of different tasks are information-fused to establish a multimodal understanding framework to output the recognition results corresponding to multiple tasks. Finally, the corresponding interaction strategy is selected based on the recognition results, and the corresponding interaction content is generated according to the interaction strategy and the interaction content is visualized. Multimodal intelligent interaction can introduce more interaction methods, such as images, gestures, touch, facial expressions, eyes, etc., which are closer to the habits and needs of users, while allowing users to complete interaction tasks more quickly and accurately, improving interaction efficiency and user satisfaction. At the same time, multimodal intelligent interaction can also improve the reliability and robustness of interaction. In some noisy environments or inconvenient voice input, users can interact through other methods, avoiding the limitations of existing voice interaction methods and adapting to different interaction scenarios and environments.

As shown in FIG2 , in a specific embodiment, the multimodal multitask intelligent interaction method provided by the present invention constructs a rich and heterogeneous knowledge base, and through multimodal learning, various types of modal data are fused from the vector space to realize cross-modal information fusion analysis and associated retrieval, realize automatic classification of knowledge, and actively discover new knowledge through data association relationships. In the process of knowledge base construction, data needs to be collected first. The multimodal knowledge base needs to contain multiple types of data such as images, videos, and texts. Therefore, data needs to be collected from various sources, such as network pictures, videos, and text materials. Subsequently, the collected multimodal data is preprocessed. To construct a multimodal knowledge base, different types of data need to be preprocessed differently. Relevant materials are automatically clustered through a clustering algorithm, and automatic maintenance is performed to ensure the accuracy, consistency, and comparability of the data, thereby providing a better basis for subsequent analysis and application. The specific operation is to clean and de-duplicate the collected data of different modalities to ensure the accuracy and reliability of the data.

In this embodiment, text data cleaning includes corpus cleaning (clustering and annotation to delete irrelevant data and duplicate data, and remove garbled and redundant symbol data), word segmentation (jieba (an open source Chinese word segmentation tool), LTP (Chinese tool set), etc.), and part-of-speech tagging. Image data cleaning mainly requires image inspection, clustering and annotation to remove illegal images (watermarks, noise, blur, etc.), annotating text labels for images, mapping image text expressions, and adjusting images to uniform size and resolution for better comparison and analysis. Video data cleaning mainly requires video segmentation, that is, segmenting the video and cutting it into a series of short clips for subsequent processing and analysis. It is also necessary to perform frame extraction, that is, extract a certain number of frames from the video for feature extraction and analysis. Audio data mainly requires denoising, that is, denoising the audio data to reduce the interference of noise on subsequent processing. It is also necessary to perform sound segmentation, that is, segment the audio data according to certain rules for further processing and analysis.

After data preprocessing, different features need to be extracted for different types of data. For example, images can be extracted through algorithms such as convolutional neural networks to extract attribute features such as color, texture, and shape, as well as feature vectors of images, for similarity analysis and classification. Text can be extracted through attribute features such as vocabulary and grammar, as well as semantic feature vectors of text, for similarity analysis and classification. Video data can be extracted through algorithms based on deep learning to extract feature vectors of video frames for similarity analysis and classification. Audio data can be extracted through algorithms based on deep learning to extract feature vectors of audio for similarity analysis and classification. After that, the processed data is stored in the knowledge base, and a rich knowledge base system is stored in two ways: relational database or graph database. Relational databases include FAQ question and answer bases, table question and answer bases, etc., and graph databases are multimodal knowledge graphs constructed in business fields.

In this embodiment, an index is established for the data in the knowledge base, wherein a multimodal relationship association is established for different modal knowledge information, and cross-modal association training can be performed based on the CLIP model (multimodal large model), so that different types of data are connected and associated, and multimodal relationships are established. Cross-modal retrieval is implemented using CLIP, and query operations can be conveniently performed. In the subsequent data operation process, a user interface is designed, and convenient tools are provided. The existing knowledge base is optimized through user interaction records, manual service records, etc., and users are supported to add, delete, modify, and query in real time. It has convenient knowledge entry, editing, and review functions, and supports rich media methods such as text, pictures, hyperlinks, videos, emoticons, interactive pages, and interface calls. Knowledge points can be grouped and managed according to business, and the data in the knowledge base is continuously updated, effectively ensuring the practicality and reliability of the knowledge base.

As shown in FIG3 , in the process of inputting multimodal information, the input includes two parts: passive input and active input. Passive input data includes data of multiple modes such as text, voice, picture and video input by the user, and active input includes user face, user portrait and scene portrait information after scene recognition. According to the needs of different groups and different scenes, an industry user portrait knowledge base and a scene portrait knowledge base are established as candidate sets. Then, the user attribute information such as the gender, age, face and the environmental attribute information of the scene are used as the query through the portrait matching module to generate personalized input representations, so as to calculate the matching score through the retrieval matching strategy, and return the highest score that best matches the user, that is, the user portrait matching vector (u) and the scene matching vector (g). Finally, the input is entered into the subsequent interaction strategy identification module to capture the personalized interaction mode.

As shown in Figure 4, entity represents entity, intent represents intent, sentiment represents sentiment library, Decoder represents decoder, Encoder represents encoder, and concatenate is a concatenation function in deep learning. In the multi-task learning process, the following steps are specifically included:

1) Collect multimodal data: Collect multimodal data such as user voice, video, gestures, etc. through cameras, microphones, sensors and other devices. The multimodal data collected in this way is mainly passive input data. In addition, by matching user portraits and scene portrait databases, user portrait data and scene portrait data can be output as active recognition input data.

2) Preprocess the multimodal input data collected in 1), such as denoising audio data, converting speech into text, segmenting text data, extracting frames from videos, etc.

3) Multimodal feature extraction: Extract features from preprocessed multimodal data for subsequent modeling. For example, for speech data, extract MFCC acoustic features (spectrum, Mel cepstral coefficients), for text data, use BERT (autoencoder language model) to extract semantic features, and for image data, use MAE (mean absolute error) to extract convolutional neural network features.

4) Multi-task learning: Learn multiple tasks (such as interaction intent recognition, sentiment judgment, entity recognition, etc.) as a whole so that multiple tasks can be trained simultaneously and features can be shared. Based on the Transformer multi-task model, multiple tasks are combined into a multi-head model, and the Attention Mask is designed to share some parameters to achieve inductive transfer and improve the generalization performance of the model. Multi-task learning introduces Prompt Learning in the decoding layer to guide the model to generate corresponding task label results. The multi-task learning process can fuse information from features of different tasks, so that cross-modal data expressing similar semantics can be mapped into the same space, thereby obtaining a more comprehensive representation of user input information features, improving the recognition accuracy of the model, and realizing a multi-modal understanding framework with both representation and generation capabilities.

5) Output recognition results: Output the label results corresponding to user intention recognition, emotion recognition, and entity recognition. The label results are used for subsequent interactive behavior decision-making.

Afterwards, a comprehensive judgment of the interactive decision is made based on the label results corresponding to the output of user intention recognition, emotion recognition, and entity recognition. The specific steps are as follows:

1) Respond to user intent: Provide corresponding answers, suggestions or actions based on user intent.

2) Check the feedback and whether it meets the user's needs. If it does, trigger the user's exit condition. Otherwise, enter 3).

3) Ask for further information: If you do not fully understand the user’s intent, you need to proactively ask the user for further information to confirm the intent.

4) Optional: Provide relevant information feedback. Based on user intent, proactively provide useful information, such as related materials, links, or services.

5) Exit condition judgment. Based on user feedback, the system determines whether the user's needs are met or the exit condition is triggered. If not, the process from 1) to 4) is repeated until the exit is completed.

In this embodiment, through personalized feedback output, different forms of interaction modes can be provided according to the user's usage scenario and emotions. For example, in the office, more suitable texts and cards are selected for interaction, and in the leisure environment, highly perceived voice and video multimodal interactions are recommended. In terms of interactive content, personalized expressions are generated according to the current context, user portrait, and emotions of the interaction. For example, if the user is depressed or angry, the expression speed and content style are adjusted to better meet the needs of the user. The recognizer sends prompt information or recommended information according to the usage scenario, time, etc., and intelligently decides the triggering time. For example, combined with the weather, adjustments are made when asking about travel methods, and reminders are given for safety, guiding users to perform specific operations, such as prompting users to enter specified information, reminding users to click specific buttons or links, etc. In addition, higher interactivity and feedback mechanisms, such as incentives and feedback, can also be provided to enhance user experience and participation. During the interaction process, the user's intentions and operation preferences are continuously recorded so that the user's intentions and needs can be more accurately identified in the future, and more intelligent services can be provided.

As shown in FIG5 , in one embodiment, the multimodal multitask intelligent interaction method provided by the present invention obtains multimodal data, and preprocesses and extracts features of the multimodal data to construct a multimodal knowledge base, specifically comprising the following steps:

Step S111 , cleaning the text data, deleting useless data and duplicate data in the text data through clustering annotation, and removing garbled characters and redundant symbols in the text data to obtain preprocessed text data.

Step S112, removing illegal images from the image data through clustering annotation, labeling the image data, adjusting the image data to a uniform size and resolution, and obtaining pre-processed image data.

As shown in FIG6 , in one embodiment, the multimodal multitask intelligent interaction method provided by the present invention acquires multimodal data, and preprocesses and extracts features of the multimodal data to construct a multimodal knowledge base, and specifically includes the following steps:

Step S113, performing video segmentation on the video data, dividing the video data into a plurality of video segments, and extracting a plurality of video frames from the video segments to perform feature extraction on the video frames.

Step S114, denoising the audio data, and dividing the audio data into multiple audio segments according to a set rule, so as to extract features from the multiple audio segments.

As shown in FIG. 7 , in one embodiment, the multimodal multitask intelligent interaction method provided by the present invention acquires multimodal data, and preprocesses and extracts features of the multimodal data to construct a multimodal knowledge base, and specifically includes the following steps:

Step S115, performing feature extraction on the preprocessed multimodal data through a convolutional neural network to extract attribute features and feature vectors in the multimodal data, and the attribute features and feature vectors are used to perform similarity analysis and data classification on the multimodal data.

Step S116, storing the multimodal data in a multimodal knowledge base including a relational database and a graph database according to the similarity analysis results of the attribute features and feature vectors and the data classification results, and establishing an index between different modal data.

As shown in FIG8 , in one embodiment, the multimodal multitasking intelligent interaction method provided by the present invention receives input multimodal information, and identifies the current user and the usage scenario based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario, specifically including the following steps:

Step S122, based on the multimodal information input by different users and different scenarios, a user portrait knowledge base and a scene portrait knowledge base are established as candidate sets, and a portrait matching unit is used to generate a corresponding input representation according to the multimodal information input by the current user.

Step S124, calculate the matching score corresponding to the multimodal information input by the current user through the set retrieval matching strategy, and feed back the user portrait matching vector and scene portrait matching vector with the highest score in the candidate set.

As shown in FIG9 , in one embodiment, the multimodal multitask intelligent interaction method provided by the present invention takes multiple tasks as input of the Transformer multitask model to form a corresponding multi-head model with multiple tasks, fuses information of features of different tasks, establishes a multimodal understanding framework, and outputs recognition results corresponding to multiple tasks, specifically including the following steps:

Step S132, introducing prompt dialog learning in the decoding layer of the Transformer multi-task model to guide the Transformer multi-task model to generate label results corresponding to different tasks.

Step S134, calling the multimodal understanding framework to perform recognition processing on different tasks to generate recognition results corresponding to different tasks, wherein the recognition results have label results corresponding to each task, and the label results are used to determine the interaction strategies corresponding to different tasks.

As shown in FIG. 10 , in one embodiment, the multi-modal multi-task intelligent interaction method provided by the present invention selects a corresponding interaction strategy based on the recognition result, generates corresponding interaction content according to the interaction strategy, and visualizes the interaction content, specifically including the following steps:

Step S142: acquiring the user intention corresponding to the current user based on the recognition result, and feeding back the corresponding first interaction content in response to the user intention.

Step S144, determining whether the first interactive content meets the user needs of the current user by monitoring, and when the first interactive content does not meet the user needs, feeding back the second interactive content to determine the user intention of the current user.

The multimodal multitask intelligent interaction device provided by the present invention is described below. The multimodal multitask intelligent interaction device described below and the multimodal multitask intelligent interaction method described above can be referred to each other.

As shown in FIG. 11 , in one embodiment, a multi-modal multi-task intelligent interaction device includes a knowledge base construction module 1110 , an information identification module 1120 , a task identification module 1130 , and a content interaction module 1140 .

The knowledge base construction module 1110 is used to obtain multimodal data, and perform preprocessing and feature extraction on the multimodal data to construct a multimodal knowledge base.

The information identification module 1120 is used to receive input multimodal information and identify the current user and the usage scenario based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario.

The task identification module 1130 is used to take multiple tasks as the input of the Transformer multi-task model to form a corresponding multi-head model for the multiple tasks, and to fuse the features of different tasks to establish a multimodal understanding framework to output the recognition results corresponding to the multiple tasks.

The content interaction module 1140 is used to select a corresponding interaction strategy based on the recognition result, generate corresponding interaction content according to the interaction strategy, and visualize the interaction content.

Among them, multimodal data includes text data, audio data, video data and image data, preprocessing includes data cleaning and clustering algorithm processing, multimodal information includes text information, audio information, video information and image information actively or passively input by users, multiple tasks include interaction intention recognition tasks, emotion recognition tasks and entity recognition tasks, interaction strategies include task-based interaction, knowledge-based interaction and open domain chat interaction, and the interaction content includes text, pictures and voice.

In this embodiment, in the multi-modal multi-task intelligent interaction device provided by the present invention, the knowledge base construction module 1110 is specifically used for:

The text data is cleaned, useless and duplicate data in the text data is deleted through clustering annotation, and garbled characters and redundant symbols in the text data are removed to obtain the preprocessed text data.

Illegal images in the image data are removed through clustering annotation, and the image data is labeled and annotated, and the image data is adjusted to a uniform size and resolution to obtain preprocessed image data.

In this embodiment, in the multi-modal multi-task intelligent interaction device provided by the present invention, the knowledge base construction module 1110 is further used for:

The video data is segmented into multiple video segments, and multiple video frames are extracted from the video segments to extract features of the video frames.

The audio data is denoised and segmented into multiple audio segments according to set rules to extract features from the multiple audio segments.

In this embodiment, in the multi-modal multi-task intelligent interaction device provided by the present invention, the knowledge base construction module 1110 is further used for:

The preprocessed multimodal data is subjected to feature extraction through a convolutional neural network to extract attribute features and feature vectors in the multimodal data. The attribute features and feature vectors are used to perform similarity analysis and data classification on the multimodal data.

The multimodal data are stored in a multimodal knowledge base including a relational database and a graph database according to the similarity analysis results of attribute features and feature vectors and the data classification results, and indexes between different modal data are established.

In this embodiment, in the multi-modal multi-task intelligent interaction device provided by the present invention, the information identification module 1120 is specifically used for:

Based on the multimodal information input from different users and different scenarios, a user portrait knowledge base and a scene portrait knowledge base are established as candidate sets, and a portrait matching unit is used to generate corresponding input representations according to the multimodal information input by the current user.

The matching score corresponding to the multimodal information input by the current user is calculated through the set retrieval matching strategy, and the user portrait matching vector and scene portrait matching vector with the highest score in the candidate set are fed back.

In this embodiment, in the multi-modal multi-task intelligent interaction device provided by the present invention, the task identification module 1130 is specifically used for:

Prompt dialog learning is introduced in the decoding layer of the Transformer multi-task model to guide the Transformer multi-task model to generate label results corresponding to different tasks.

The multimodal understanding framework is called to perform recognition processing on different tasks to generate recognition results corresponding to different tasks. The recognition results contain label results corresponding to each task, and the label results are used to determine the interaction strategies corresponding to different tasks.

In this embodiment, in the multi-modal multi-task intelligent interaction device provided by the present invention, the content interaction module 1140 is specifically used for:

The user intention corresponding to the current user is obtained based on the recognition result, and the corresponding first interaction content is fed back in response to the user intention.

It is determined by monitoring whether the first interactive content meets the user needs of the current user, and when the first interactive content does not meet the user needs, the second interactive content is fed back to determine the user intention of the current user.

FIG12 illustrates a schematic diagram of the physical structure of an electronic device, which may be a smart terminal, and its internal structure diagram may be shown in FIG12. The electronic device includes a processor, a memory, and a network interface connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a multi-modal multi-task intelligent interaction method is implemented, and the method includes:

Acquire multimodal data, preprocess and extract features from the multimodal data to build a multimodal knowledge base;

Receive input multimodal information, and identify the current user and the usage scenario based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario;

Take multiple tasks as input to the Transformer multi-task model to combine multiple tasks into corresponding multi-head models, fuse the features of different tasks, and establish a multimodal understanding framework to output recognition results corresponding to multiple tasks;

Select the corresponding interaction strategy based on the recognition result, generate the corresponding interaction content according to the interaction strategy and visualize the interaction content;

Among them, multimodal data includes text data, audio data, video data and image data, preprocessing includes data cleaning and clustering algorithm processing, multimodal information includes text information, audio information, video information and image information actively or passively input by users, multiple tasks include interaction intention recognition tasks, emotion recognition tasks and entity recognition tasks, interaction strategies include task-based interaction, knowledge-based interaction and open domain chat interaction, and the interaction content includes text, pictures and voice.

Those skilled in the art will understand that the structure shown in FIG. 12 is merely a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the electronic device to which the solution of the present invention is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

On the other hand, the present invention further provides a computer storage medium storing a computer program, which implements a multi-modal multi-task intelligent interaction method when executed by a processor, the method comprising:

Acquire multimodal data, preprocess and extract features from the multimodal data to build a multimodal knowledge base;

Receive input multimodal information, and identify the current user and the usage scenario based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario;

Take multiple tasks as input to the Transformer multi-task model to combine multiple tasks into corresponding multi-head models, fuse the features of different tasks, and establish a multimodal understanding framework to output recognition results corresponding to multiple tasks;

Select the corresponding interaction strategy based on the recognition result, generate the corresponding interaction content according to the interaction strategy and visualize the interaction content;

Among them, multimodal data includes text data, audio data, video data and image data, preprocessing includes data cleaning and clustering algorithm processing, multimodal information includes text information, audio information, video information and image information actively or passively input by users, multiple tasks include interaction intention recognition tasks, emotion recognition tasks and entity recognition tasks, interaction strategies include task-based interaction, knowledge-based interaction and open domain chat interaction, and the interaction content includes text, pictures and voice.

In another aspect, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and when the processor executes the computer instructions, a multi-modal multi-task intelligent interaction method is implemented, the method comprising:

Acquire multimodal data, preprocess and extract features from the multimodal data to build a multimodal knowledge base;

Receive input multimodal information, and identify the current user and the usage scenario based on the multimodal information to obtain multiple tasks corresponding to the current user and the usage scenario;

Take multiple tasks as input to the Transformer multi-task model to combine multiple tasks into corresponding multi-head models, fuse the features of different tasks, and establish a multimodal understanding framework to output recognition results corresponding to multiple tasks;

Select the corresponding interaction strategy based on the recognition result, generate the corresponding interaction content according to the interaction strategy and visualize the interaction content;

Among them, multimodal data includes text data, audio data, video data and image data, preprocessing includes data cleaning and clustering algorithm processing, multimodal information includes text information, audio information, video information and image information actively or passively input by users, multiple tasks include interaction intention recognition tasks, emotion recognition tasks and entity recognition tasks, interaction strategies include task-based interaction, knowledge-based interaction and open domain chat interaction, and the interaction content includes text, pictures and voice.

Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and/or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

The above-mentioned embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims (10)

1. A multi-modal multitasking intelligent interaction method, the method comprising:

Acquiring multi-modal data, preprocessing the multi-modal data and extracting features to construct a multi-modal knowledge base;

receiving input multi-mode information, and identifying a current user and a use scene based on the multi-mode information so as to acquire a plurality of tasks corresponding to the current user and the use scene;

The tasks are used as input of a transform multitask model, so that the tasks are formed into a corresponding multi-head model, information fusion is carried out on the characteristics of different tasks, a multi-mode understanding framework is built, and recognition results corresponding to the tasks are output;

Selecting a corresponding interaction strategy based on the identification result, generating corresponding interaction content according to the interaction strategy, and visually displaying the interaction content;

The multi-modal data comprises text data, audio data, video data and image data, the preprocessing comprises data cleaning and clustering algorithm processing, the multi-modal information comprises text information, audio information, video information and image information which are actively or passively input by a user, the tasks comprise an interaction intention recognition task, an emotion recognition task and an entity recognition task, the interaction strategy comprises task type interaction, knowledge type interaction and open field chat interaction, and the interaction content comprises words, pictures and voices.

2. The method for intelligent interaction of multiple modes and multiple tasks according to claim 1, wherein the steps of obtaining multiple modes data, preprocessing the multiple modes data, extracting features of the multiple modes data, and constructing a multiple modes knowledge base include:

Carrying out corpus cleaning on the text data, deleting useless data and repeated data in the text data through clustering labeling, and removing messy codes and redundant symbols in the text data to obtain preprocessed text data; and

And removing illegal pictures in the image data through clustering labeling, labeling the image data, and adjusting the image data to be uniform in size and resolution to obtain preprocessed image data.

3. The method for intelligent interaction of multiple modes and multiple tasks according to claim 2, wherein the steps of obtaining multiple modes data, preprocessing the multiple modes data and extracting features to construct a multiple modes knowledge base further comprise:

video segmentation is carried out on the video data, the video data is segmented into a plurality of video segments, and a plurality of video frames are extracted from the video segments so as to carry out feature extraction on the video frames; and

And denoising the audio data, and dividing the audio data into a plurality of sections of audio according to a set rule so as to extract the characteristics of the plurality of sections of audio.

4. The method for intelligent interaction of multiple modes and multiple tasks according to claim 3, wherein said obtaining multiple modes data, preprocessing said multiple modes data and extracting features to build a multiple modes knowledge base, further comprises:

Extracting the characteristics of the preprocessed multi-modal data through a convolutional neural network to extract attribute characteristics and characteristic vectors in the multi-modal data, wherein the attribute characteristics and the characteristic vectors are used for carrying out similarity analysis and data classification on the multi-modal data;

and storing the multi-modal data in the multi-modal knowledge base comprising a relational database and a graph database according to the similarity analysis result and the data classification result of the attribute features and the feature vectors, and establishing indexes among different modal data.

5. The method for intelligent interaction of multiple modes and multiple tasks according to claim 1, wherein the steps of receiving the input multiple modes information, identifying the current user and the usage scenario based on the multiple modes information, and obtaining the multiple tasks corresponding to the current user and the usage scenario include:

establishing a user portrait knowledge base and a scene portrait knowledge base as candidate sets based on the multi-mode information input by different users and different scenes, and generating corresponding input characterization according to the multi-mode information input by the current user through a portrait matching unit;

And calculating a matching score corresponding to the multi-mode information input by the current user through the set search matching strategy, and feeding back a user portrait matching vector and a scene portrait matching vector with the highest scores in the candidate set.

6. The method of claim 5, wherein the inputting the plurality of tasks as a transform multitask model to form the plurality of tasks into a corresponding multi-headed model, and fusing information about features of different tasks, and establishing a multi-modal understanding framework to output recognition results corresponding to the plurality of tasks, includes:

Introducing prompt dialog box learning at a decoding layer of the converter multi-task model to guide the converter multi-task model to generate label results corresponding to different tasks;

And invoking the multi-mode understanding framework to perform recognition processing on different tasks so as to generate recognition results respectively corresponding to the different tasks, wherein the recognition results are provided with the label results corresponding to each task, and the label results are used for determining interaction strategies respectively corresponding to the different tasks.

7. The multi-modal multi-task intelligent interaction method according to any one of claims 1 to 6, wherein selecting a corresponding interaction policy based on the recognition result, generating corresponding interaction content according to the interaction policy, and visually displaying the interaction content, includes:

Acquiring user intention corresponding to a current user based on the identification result, and feeding back corresponding first interactive content in response to the user intention;

and judging whether the first interactive content meets the user requirement of the current user or not through monitoring, and feeding back the second interactive content to determine the user intention of the current user when the first interactive content does not meet the user requirement.

8. A multi-modal multi-tasking intelligent interaction device, the device comprising:

The knowledge base construction module is used for acquiring multi-modal data, preprocessing the multi-modal data and extracting features to construct a multi-modal knowledge base;

the information identification module is used for receiving input multi-mode information and identifying a current user and a use scene based on the multi-mode information so as to acquire a plurality of tasks corresponding to the current user and the use scene;

The task identification module is used for taking the tasks as input of a transform multitask model, forming the tasks into a corresponding multi-head model, fusing the characteristics of different tasks, establishing a multi-mode understanding frame, and outputting identification results corresponding to the tasks;

The content interaction module is used for selecting a corresponding interaction strategy based on the identification result, generating corresponding interaction content according to the interaction strategy and carrying out visual display on the interaction content;

The multi-modal data comprises text data, audio data, video data and image data, the preprocessing comprises data cleaning and clustering algorithm processing, the multi-modal information comprises text information, audio information, video information and image information which are actively or passively input by a user, the tasks comprise an interaction intention recognition task, an emotion recognition task and an entity recognition task, the interaction strategy comprises task type interaction, knowledge type interaction and open field chat interaction, and the interaction content comprises words, pictures and voices.

9. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 7 when the computer program is executed.

10. A computer storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method of any one of claims 1 to 7.