CN114548274A - Multi-modal interaction-based rumor detection method and system - Google Patents

Abstract

The present invention provides a method and system for detecting rumors based on multimodal interaction, including: acquiring several tweets to be detected; extracting sequence enhancement text features of each tweet to be detected; extracting the sequence of each tweet to be detected Enhance visual features; enhance text features based on sequences and enhance visual features based on sequences, use attention mechanism interactively, obtain visual features with text attention and text features with visual attention, and based on visual features with text attention features and text features with visual attention to generate visual features and text features; feature fusion of visual features with text attention, text features with visual attention, visual features and text features, based on the fusion feature to get the result of whether each tweet to be detected is a rumor. It can effectively fuse the multimodal data in the tweets to be detected and generate a joint representation, which improves the accuracy of rumor detection.

Description

A method and system for rumor detection based on multimodal interaction

technical field

The invention belongs to the technical field of natural language processing, and in particular relates to a rumor detection method and system based on multimodal interaction.

Background technique

The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

With the improvement of hardware technology and the advancement of network technology, social media has achieved unprecedented development and has become an indispensable part of people's daily life. Social media is an important source of news and supports traditional media because of its low cost, easy access and rapid dissemination. Information on social media has been used in various fields such as marketing, public event discussion and public opinion analysis, and rumor detection. Users can easily post opinions, get news, and communicate with each other on social media platforms. At the same time, fake news and rumors spread rapidly through social media, misleading the public and even causing bad social impact. In view of the high cost and slow detection speed of manual detection, automatic rumor detection methods are particularly important.

Previous studies focused on using handcrafted features or deep learning-based unimodal features for rumor detection, with the disadvantage of ignoring the complementarity of multimodal data in tweets. Compared to the text content in the tweet, visual elements such as video and images in the tweet can attract the reader's attention more. Therefore, there are also some methods that comprehensively analyze the multimodal content in tweets for rumor detection. However, most of these methods fuse data in heterogeneous modal space, which cannot avoid numerical differences between multi-modal data and are difficult to capture the dependencies between heterogeneous modal data.

SUMMARY OF THE INVENTION

In order to solve the technical problems existing in the above background art, the present invention provides a method and system for rumor detection based on multimodal interaction, which can effectively fuse multimodal data in tweets to be detected and generate a joint representation, thereby improving rumor detection. accuracy.

In order to achieve the above object, the present invention adopts the following technical solutions:

A first aspect of the present invention provides a rumor detection method based on multimodal interaction, which includes:

Get several tweets to be detected that contain textual content and visual content;

Based on the text content of each tweet to be detected, extract the sequence enhanced text features of each tweet to be detected;

Based on the visual content of each tweet to be detected, extract the sequence enhanced visual features of each tweet to be detected;

Based on sequence-enhanced text features and sequence-enhanced visual features, using interactive attention mechanism, obtain visual features with text attention and text features with visual attention, and based on visual features with text attention and with Text features for visual attention, generating visual features and text features;

Fusion of visual features with text attention, text features with visual attention, visual features and text features to obtain the fused features;

Based on the fused features, the result of whether each tweet to be detected is a rumor is obtained.

Further, the specific method for generating visual features and text features is:

Take the text feature with visual attention as the input matrix, take the visual feature with text attention as the target matrix, and obtain the visual feature through Transformer mapping;

Taking the visual features with text attention as the input matrix, and taking the text features with visual attention as the target matrix, the text features are obtained through Transformer mapping.

Further, the specific method of the feature fusion is:

After splicing visual features with text attention and visual features, the first early fusion result is obtained through a fusion recognition block composed of fully connected layers;

After splicing the text features with visual attention and text features, the second early fusion result is obtained through the fusion recognition block composed of fully connected layers;

The first early fusion result and the second early fusion result are fused by assigning weights to obtain the fused features.

Further, the specific method for extracting the sequence-enhanced text feature of each tweet to be detected is:

Perform text preprocessing on the text content of each tweet to be detected, and obtain the primary text feature representation of each tweet to be detected based on the preprocessed text content;

Based on the primary text feature representation of each tweet to be detected, the text features with contextual information of each tweet to be detected are obtained;

The self-attention mechanism is used to enhance the text features with contextual information, and the sequence-enhanced text features are obtained.

Further, the text preprocessing includes: deleting stop words, user handles and punctuation marks in each tweet to be detected.

Further, the specific method for extracting the sequence enhancement visual feature of each tweet to be detected is:

Perform image preprocessing on the visual content of each tweet to be detected, and obtain the regional features of each tweet to be detected based on the preprocessed visual content;

Based on the regional features of each tweet to be detected, the sequence-enhanced visual features are obtained through sequential attention blocks.

Further, the image preprocessing includes: image size adjustment, random horizontal flipping and pixel value normalization.

A second aspect of the present invention provides a rumor detection system based on multimodal interaction, which includes:

a data acquisition module, which is configured to: acquire several tweets to be detected that contain textual content and visual content;

a sequence-enhanced text feature extraction module, which is configured to: extract the sequence-enhanced text feature of each tweet to be detected based on the text content of each tweet to be detected;

a sequence-enhanced visual feature extraction module, which is configured to: extract the sequence-enhanced visual feature of each tweet to be detected based on the visual content of each tweet to be detected;

an interaction module, which is configured to: enhance text features based on sequences and enhance visual features based on sequences, use an attention mechanism interactively, obtain visual features with text attention and text features with visual attention, and based on text features with text Visual features of attention and text features with visual attention, generating visual features and text features;

a feature fusion module, which is configured to: fuse visual features with text attention, text features with visual attention, visual features and text features to obtain fused features;

A detection module, which is configured to: obtain a result of whether each tweet to be detected is a rumor based on the fused features.

A third aspect of the present invention provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the steps in the above-mentioned method for detecting rumors based on multimodal interaction .

A fourth aspect of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor implementing the above-mentioned one when executing the program Steps in a multimodal interaction-based rumor detection method.

Compared with the prior art, the beneficial effects of the present invention are:

The present invention provides a rumor detection method based on multi-modal interaction, which obtains single-modal features with enhanced time series, and performs cross-modal mapping on different modal data, which can effectively reduce the impact of heterogeneity differences; And the interactive attention mechanism is used to capture the dependencies between multimodal data, which improves the accuracy of rumor detection.

Description of drawings

The accompanying drawings forming a part of the present invention are used to provide further understanding of the present invention, and the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, and do not constitute an improper limitation of the present invention.

Fig. 1 is the method flow chart of the first embodiment of the present invention;

2 is a flowchart of text feature extraction according to Embodiment 1 of the present invention;

3 is a comparison diagram of early detection performance on the microblog data set according to Embodiment 1 of the present invention;

FIG. 4 is a comparison diagram of early detection performance on the Twitter data set according to Embodiment 1 of the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and embodiments.

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the invention. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

It should be noted that the terminology used herein is for the purpose of describing specific embodiments only, and is not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural as well, furthermore, it is to be understood that when the terms "comprising" and/or "including" are used in this specification, it indicates that There are features, steps, operations, devices, components and/or combinations thereof.

Example 1

This embodiment provides a rumor detection method based on multimodal interaction, as shown in FIG. 1 , which specifically includes the following steps:

Step 1. Acquire several tweets to be detected that contain textual content and visual content.

The acquired several tweets to be detected can be expressed as Tw={T, V}, where T and V respectively represent the text content and visual content (pictures) in all the tweets to be detected. That is, each tweet to be detected includes multimodal content (textual content and visual content).

Among them, the text content T={T ₁ , T ₂ ,...,T _N } in all the tweets to be detected, T _{i ∈} T represents the text content of the i-th tweet to be detected, and N represents the acquired tweet to be detected. The total number of texts, T _i ={t ₁ ,t ₂ ,...,t _L } represents the character sequence of the i-th tweet to be detected, L represents the length of the sequence; the visual content in all tweets to be detected V={ V ₁ , V ₂ ,...,V _N }, V _i ∈ V represents the visual content of the i-th tweet to be detected.

As an implementation manner, the tweet to be detected may originate from Weibo.

Step 2. Input a number of tweets to be detected into a multimodal rumor detection model (TIMF model) that uses textual content and visual content to detect whether each tweet to be detected is a rumor, that is, the output of the TIMF model is rumor detection Result: Rumor (N) or Not Rumor (NR). Specifically include the following steps:

Step 201: As shown in FIG. 2, based on the text content T _i ={t ₁ ,t ₂ ,...,t _L } of each tweet to be detected, extract the sequence enhanced text feature F of each tweet to be detected _T ={t ₁ ,t ₂ ,...,t _m }, that is, input the text content of all tweets to be detected into the text block, which can be expressed as a function F _T =f _T (T _i ; θ _T ), where F _T represents the text features of the text block output, and θ _T represents the parameters of the text block:

(1) Perform text preprocessing on the text content T _i ={t ₁ ,t ₂ ,...,t _L } of each tweet to be detected, and obtain each tweet to be detected based on the preprocessed text content The primary text feature representation H={h(0),h(1),...,h(L ₁ )}, including:

①Remove stop words, user handles and punctuation marks in each tweet to be detected;

② Call Bert tokenizer to segment the processed text, and obtain the primary text feature representation H={h(0), h(1),..., h(L ₁ )} of each tweet to be detected.

Specifically, the fine-tuned pre-trained Bert model is used to extract text features. The Bert model based on Transformer (Trm) and Attention mechanism is one of the best models in various NLP tasks, and Bert is more because of its predefined length. Suitable for short texts like Weibo. Bert's input consists of three parts: Token Embedding, Segment Embedding and Position Embedding, using E(tok _i ), E(seg _i ), E(pos _i ) to represent the vector representation of the character, the index of the sentence and the position index of the character respectively .

Bert's input BI={I(0),I(1),...,I(L)}, I(i) is the element addition of the i-th character of each tweet to be detected after preprocessing Composition, that is, I(i)=E(tok _i )+E(seg _i )+E(pos _i ), where E(tok _i ), E(seg _i ), E(pos _i ) represent the vector representations of characters, respectively , the index of the sentence and the position index of the character. In Figure 2, cls and sep are the markers added to the beginning and end of the sentence to indicate the beginning and end of the sentence, ti represents the vector representation of the i-th character, and A represents Sentence index. Since the Bert task in the present invention is feature extraction rather than question answering, A is used to indicate that all characters belong to a sentence, and L is used to indicate the number of characters in the sentence. Then use the fine-tuned Bert to extract the primary text feature representation of the text content H={h(0), h(1), ..., h(L ₁ )}, i.e.

H=Bert _ft (BI)

(2) In order to better capture the global information, the primary text feature of each tweet to be detected is represented by H={h(0), h(1),...,h(L ₁ )} input sequence attention Force block (SA module), the sequence attention block consists of Bi-GRU and Self-Attention.

Based on the primary text feature representation H={h(0),h(1),...,h(L ₁ )} of each tweet to be detected, use Bi-GRU to extract the text features of each tweet to be detected to obtain the text feature representation with context information G={g(0), g(1),...,g(L ₁ )}

G=Bi-GRU(H)

The self-attention mechanism (Self-Attention mechanism) can enhance the text features with contextual information, and combine the additive normalization operation (Add&Norm) and the fully connected layer (Full Connected Layer(2)) to obtain the sequence-enhanced text feature F. _T = {t ₁ ,t ₂ ,...,t _m }:

q _i =w _q ×g(i), _ki =w _k ×g( _i ),pi =w _p ×g(i)

att_G={att_g(0),att_g(1),...,att_g(L ₁ )}

F' _T =Add&Norm(G,att_G)

F _T =σ(w _fc gF′ _T +b _fc )

Among them, q _i , k _i , v _i represent the linear transformation matrix (intermediate matrix) in the attention mechanism, w _q , w _k , w _v represent the weight matrix, and g(i) is the text feature representation with context information. The i-th eigenvalue, w _fc represents the weight matrix of the fully connected layer, b _fc represents the bias matrix of the fully connected layer (Full Connected Layer), and σ represents the activation function ReLU.

Step 202: Based on the visual content V _i of each tweet to be detected, extract the sequence enhanced visual feature F _V ={v ₁ ,v ₂ ,..., _vm } of each tweet to be detected, that is, all the tweets to be detected are extracted. The visual content of the tweet is input to the visual block, which can be expressed as a function F _V = f _V (V _i ; θ _V ), where F _V represents the visual features output by the visual module, and θ _V represents the parameters of the visual module:

(1) Perform image preprocessing on the visual content of each tweet to be detected, specifically, image size adjustment, that is, uniformly adjust the image size to 224×224; in order to expand the number of training samples and improve the generalization performance of the model, random The horizontal flip operation augments the image; normalizes pixel values in the range [0-1].

(2) Use the ResNet50 model pre-trained on ImageNet to extract the pre-processed visual content regional features res_F _V ={v′ ₁ , v′ ₂ ,...,v′ _m }, and change the last fully connected layer so that The output sequence length is consistent with the text features.

res_F _V = ResNet _ft (V)

(3) Similar to the text module, use the SA module to obtain sequence-enhanced visual features F _V = {v ₁ , v ₂ , ..., v _m }:

F _V =SA(res_F _V )

Step 203: Input the sequence-enhanced text feature and the sequence-enhanced visual feature into the multimodal fusion block to obtain the result of rumor detection, which can be expressed as a function

Result=f _F ({F _V , F _T }; θ _F )

Among them, Result represents the output of the module (R or NR), and θ _F represents the parameters of the multimodal fusion block.

(1) Cross-modal feature mapping

(101) There are more or less numerical differences and dependencies between the data of different modalities. In order to achieve a better fusion effect, cross-modal feature mapping is required to capture the dependencies between heterogeneous modal data and reduce Small numerical differences, first, for heterogeneous modal data, sequence augmented text features F _T = {t ₁ , t ₂ , ..., t _m } and sequence augmented visual features F _V = {v ₁ , v ₂ , ..., v _m }, using an interactive attention mechanism (Attention(1)) to obtain visual features with textual attention

q _i =tanh(w _q ·t _i +b _q )

_{k i} =tanh(w _k ·vi +b _k )

p _i =tanh( _{w p} ·vi +b _p )

这一步是为了突出图像中的区域与文本中的字符并初步捕捉模态间的依赖关系。In the same way, using the interactive attention mechanism (Attention(2)) can get text features with visual attention

This step is to highlight regions in the image and characters in the text and initially capture the dependencies between modalities.

(102) Using Transformer for cross-modal feature sequence mapping, Transformer is essentially an encoding-decoding framework, and is mainly used to deal with the Sequence to Sequence problem in the NLP field. It consists of six encoding layers and six decoding layers. Use it as a medium for heterogeneous modal feature sequence mapping. The feature matrix after sequence enhancement has been obtained through the above SA-module. Therefore, for the first Transformer (Transformer(1)), the text feature with visual attention is used as input to obtain the generated visual feature gen_F _V =Transformer{src:att_F _T ,tgt:att_F _V }, where src represents the source domain, tgt Represents the target domain. The generated visual feature uses the source visual feature _{att_FT} as the input matrix, and att_F _V as the target matrix, which is obtained by Transformer mapping, thus reducing the heterogeneity difference between the source visual feature and the generated visual feature and further capturing the heterogeneous model. The dependency between state data can get better fusion effect.

Similarly, the generated text feature _{gen_FT} can be obtained by taking the visual feature with text attention as the input of the second Transformer (Transformer(2)). Specifically, the visual features with text attention are used as the input matrix, and the text features with visual attention are used as the target matrix, and the text features are obtained through Transformer mapping.

(2) Fusion

The early fusion consists of three fully connected layers (the first fully connected layer Fc-1, the second fully connected layer Fc-2, and the third fully connected layer Fc-3), and a dropout layer is added between the fully connected layers (the first fully connected layer Fc-1). A Dropout layer Dropout-1 and a second Dropout layer Dropout-2) to prevent overfitting. The source feature and the generated feature are spliced and sent to the fully connected layer to obtain the early fusion result:

After splicing visual features with text attention and visual features, the first early fusion result is obtained through a fusion recognition module composed of fully connected layers, where EF refers to early fusion:

Result-1=EF{att_F _V ,gen_F _V }

After splicing the text features with visual attention and text features, the second early fusion result is obtained through the fusion recognition module composed of fully connected layers:

Result-2=EF{ _{att_FT} , _{gen_FT} }

The late fusion fuses the results from the early fusion by assigning weights, and fuses the first early fusion result and the second early fusion result by assigning weights, where LF refers to the late fusion, and the fused features are obtained, that is, the late fusion result:

Result=LF{Result-1,Result-2}

The loss of the module is defined using cross-entropy, and Y is the event label, and Result(i) is the prediction result of the i-th tweet:

来最小化模块损失函数L_F：By seeking the best parameters

to minimize the module loss function _LF :

The invention can effectively and comprehensively analyze the multimodal content in the text to be detected; it can map the multimodal content to the same data domain to generate an effective joint representation; better detection effect.

The proposed model is validated on the Weibo dataset and Twitter dataset, respectively. The Weibo dataset is a multimodal dataset in which rumor tweets are obtained by crawling all verified fake rumor posts from May 2012 to January 2016. This dataset has objective ground truth labels. The non-rumor tweets included are drawn from tweet data verified by authoritative news organizations. The Twitter dataset was derived from a part of MediaEval’s mission to detect fake content on Twitter, removing tweets that didn’t have any text or images in them.

Table 1. Data set details

As shown in Table 2, the corresponding ablation experiments are performed on the TIMF model in the present invention, which are various modules and simplified variants in TIMF.

Among them, Visual: use only the pre-trained ResNet50 model in the vision module to verify on single-modal visual content; Visual-SA: use the complete vision module including SA-module to verify on single-modal visual content; Texual : Validation on unimodal text content using only the fine-tuned Bert model in the text module; Texual-SA: Validation on unimodal text content using the full text module including the SA-module; EF-1: Validating only on unimodal text content Early recognition model fused with visual source features and visual generative features; EF-2: Early recognition model using only text source features fused with text generative features; TMIF: The full model proposed in this paper. Accuracy: precision; Precision: precision; Recall: recall; F1: F1 indicator.

Table 2. Ablation experiment

As shown in Table 3, the TIMF model in the present invention is compared with the representative algorithms in related work, where K represents knowledge information and S represents social context. Taking into account the differences in task settings, the main network structures of these models are adopted and the necessary adjustments are made for the tasks. Among them, att-RNN is proposed by a multimodal rumor detection model (Multimodal fusion with recurrent neural networks for rumor detection onmicroblogs) proposed in the journal The 25th ACMinternational conference on Multimedia in 2017, and its input is the text in the tweet , visual and social context information; EANN is a rumor detection network (Eann: Event adversarial neural networks formulti-modal fake news detection) proposed in the journal The 24th acm sigkdd international conference on knowledge discovery&datamining in 2018, delete the Temporal discriminator, retaining only two parts: multimodal feature extractor and fake news detector; MAVE is a multimodal variational autoencoder (Mvae: Multimodal variational autoencoder proposed in the journal The world wide web conference in 2019). for fakenews detection), for the task, use its complete structure, including encoder, decoder and detector; MFN is an unsupervised rumor proposed in the journal Pacific-Asia conference on knowledge discovery and data mining in 2018 Detection model (Call attention to rumors: Deep attention based recurrent neural networks for early rumor detection), for tasks, use Text-CNN to capture text features of different granularities, use pre-trained VGG-19 model to capture image features, and finally use attention mechanism and latent subject memory network generation joint representation; MKEMN is a rumor detection method (Multi-modalknowledge-aware event memory network for social media rumor detection) proposed in the journal Proceedings of the 27th ACM International Conference on Multimedia in 2019, using its complete The structure fuses information from three modalities of text, image and knowledge information for rumor detection.

Table 3. Comparative experiments with related algorithms

As shown in FIG. 3 and FIG. 4 , a comparative study on the early detection performance of the TIMF model in the present invention is carried out. Therefore, 20% of the test samples are fixed, and the early detection ability of the model is observed by gradually increasing the training samples.

Embodiment 2

This embodiment provides a rumor detection system based on multimodal interaction, which specifically includes the following modules:

a data acquisition module, which is configured to: acquire several tweets to be detected that contain textual content and visual content;

a sequence-enhanced text feature extraction module, which is configured to: extract the sequence-enhanced text feature of each tweet to be detected based on the text content of each tweet to be detected;

a sequence-enhanced visual feature extraction module, which is configured to: extract the sequence-enhanced visual feature of each tweet to be detected based on the visual content of each tweet to be detected;

an interaction module, which is configured to: enhance text features based on sequences and enhance visual features based on sequences, use an attention mechanism interactively, obtain visual features with text attention and text features with visual attention, and based on text features with text Visual features of attention and text features with visual attention, generating visual features and text features;

a feature fusion module, which is configured to: fuse visual features with text attention, text features with visual attention, visual features and text features to obtain fused features;

A detection module, which is configured to: obtain a result of whether each tweet to be detected is a rumor based on the fused features.

It should be noted here that each module in this embodiment corresponds to each step in Embodiment 1 one by one, and the specific implementation process thereof is the same, which is not repeated here.

Embodiment 3

This embodiment provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the steps in the multimodal interaction-based rumor detection method described in the first embodiment above .

Embodiment 4

This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor, where the processor implements the one described in the first embodiment when the processor executes the program. Steps in a multimodal interaction-based rumor detection method.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the invention may take the form of a hardware embodiment, a software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media having computer-usable program code embodied therein, including but not limited to disk storage, optical storage, and the like.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium. During execution, the processes of the embodiments of the above-mentioned methods may be included. The storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM) or the like.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (10)

1. A rumor detection method based on multi-modal interaction is characterized by comprising the following steps:

acquiring a plurality of tweets to be detected containing text content and visual content;

extracting sequence enhancement text characteristics of each tweed to be detected based on the text content of each tweed to be detected;

extracting the sequence enhancement visual characteristics of each tweed to be detected based on the visual content of each tweed to be detected;

based on the sequence enhanced text features and the sequence enhanced visual features, obtaining visual features with text attention and text features with visual attention by using an interactive use attention mechanism, and generating the visual features and the text features based on the visual features with text attention and the text features with visual attention;

performing feature fusion on the visual features with the text attention, the text features with the visual attention, the visual features and the text features to obtain fused features;

and obtaining the result of whether each tweet to be detected is a rumor or based on the characteristics after fusion.

2. The method for rumor detection based on multi-modal interaction of claim 1, wherein the specific method for generating visual and text features is:

the method comprises the steps of taking text features with visual attention as an input matrix, taking the visual features with the visual attention as a target matrix, and obtaining the visual features through transform mapping;

and (3) taking the visual features with the text attention as an input matrix and the text features with the visual attention as a target matrix, and obtaining the text features through transform mapping.

3. The method for rumor detection based on multi-modal interaction of claim 1, wherein the feature fusion method comprises the following specific steps:

splicing the visual features with text attention with the visual features, and obtaining a first early stage fusion result through a fusion identification block consisting of full connection layers;

after splicing the text features with visual attention with the text features, obtaining a second early fusion result through a fusion recognition block consisting of full connection layers;

and fusing the first early fusion result and the second early fusion result by distributing weights to obtain fused characteristics.

4. The method for rumor detection based on multi-modal interaction of claim 1, wherein the specific method for extracting the sequence enhanced text features of each tweed to be detected is as follows:

performing text preprocessing on the text content of each tweed to be detected, and obtaining primary text characteristic representation of each tweed to be detected based on the preprocessed text content;

obtaining text characteristics with context information of each tweet to be detected based on the primary text characteristic representation of each tweet to be detected;

and enhancing the text features with the context information by using a self-attention mechanism to obtain the text features after sequence enhancement.

5. The method of claim 4, wherein the text preprocessing comprises: and deleting the stop word, the user handle and the punctuation mark in each tweet to be detected.

6. The method for rumor detection based on multi-modal interaction of claim 1, wherein the specific method for extracting the sequence-enhanced visual features of each tweed to be detected is as follows:

performing image preprocessing on the visual content of each tweed to be detected, and obtaining the regional characteristics of each tweed to be detected based on the preprocessed visual content;

and obtaining sequence enhancement visual features through a sequence attention block based on the region features of each tweed to be detected.

7. The method of claim 6, wherein the image pre-processing comprises: image resizing, random horizontal flipping, and pixel value normalization.

8. A system for rumor detection based on multi-modal interactions, comprising:

a data acquisition module configured to: acquiring a plurality of tweets to be detected containing text content and visual content;

a sequence-enhanced text feature extraction module configured to: extracting sequence enhancement text characteristics of each tweed to be detected based on the text content of each tweed to be detected;

a sequence-enhanced visual feature extraction module configured to: extracting the sequence enhancement visual characteristics of each tweed to be detected based on the visual content of each tweed to be detected;

an interaction module configured to: based on the sequence enhanced text features and the sequence enhanced visual features, obtaining visual features with text attention and text features with visual attention by using an interactive use attention mechanism, and generating the visual features and the text features based on the visual features with text attention and the text features with visual attention;

a feature fusion module configured to: performing feature fusion on the visual features with the text attention, the text features with the visual attention, the visual features and the text features to obtain fused features;

a detection module configured to: and obtaining the result of whether each tweed to be detected is a rumor or not based on the fused characteristics.

9. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, performs the steps of the method for detecting rumors based on multi-modal interactions as recited in any one of claims 1-7.

10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor when executing the program performs the steps of the method for detecting rumors based on multi-modal interactions as recited in any one of claims 1-7.

Images