Search Results (707)

This paper explores the fractal properties of token embedding spaces in GPT-2 language models by analyzing the stability of the correlation dimension, a measure of geometric complexity. Token embeddings represent words or subwords as vectors in a high-dimensional space. We hypothesize that the correlation dimension $D_2$ remains consistent across different vocabulary subsets, revealing fundamental structural characteristics of language representation in GPT-2. Our main objective is to quantify and analyze the stability of $D_2$ in these embedding subspaces, addressing the challenges posed by their high dimensionality. We introduce a new theorem formalizing this stability, stating that for any two sufficiently large random subsets $S_1,S_2\subset E$, the difference in their correlation dimensions is less than a small constant $\epsilon$. We validate this theorem using the Grassberger–Procaccia algorithm for estimating $D_2$, coupled with bootstrap sampling for statistical consistency. Our experiments on GPT-2 models of varying sizes demonstrate remarkable stability in $D_2$ across different subsets, with consistent mean values and small standard errors. We further investigate how the model size, embedding dimension, and network depth impact $D_2$. Our findings reveal distinct patterns of $D_2$ progression through the network layers, contributing to a deeper understanding of the geometric properties of language model representations and informing new approaches in natural language processing. Full article

Ensuring the absence of vulnerabilities or flaws in smart contracts before their deployment is crucial for the smooth progress of subsequent work. Existing detection methods heavily rely on expert rules, resulting in low robustness and accuracy. Therefore, we propose EDSCVD, an enhanced deep learning vulnerability detection model based on dual-channel networks. Firstly, the contract fragments are preprocessed by BERT into the required word embeddings. Next, we utilized adversarial training FGM to the word embeddings to generate perturbations, thereby producing symmetric adversarial samples and enhancing the robustness of the model. Then, the dual-channel model combining BiLSTM and CNN is utilized for feature training to obtain more comprehensive and symmetric information on temporal and local contract features.Finally, the combined output features are passed through a classifier to classify and detect contract vulnerabilities. Experimental results show that our EDSCVD exhibits excellent detection performance in the detection of classical reentrancy vulnerabilities, timestamp dependencies, and integer overflow vulnerabilities. Full article

Over the past few decades, the study of leadership theory has expanded across various disciplines, delving into the intricacies of human behavior and defining the roles of individuals within organizations. Its primary objective is to identify leaders who play significant roles in the communication flow. In addition, behavioral theory posits that leaders can be distinguished based on their daily conduct, while social network analysis provides valuable insights into behavioral patterns. Our study investigates five and six types of social networks frequently observed in different organizations. This study is conducted using datasets we collected from an IT company and public datasets collected from a manufacturing company for the thorough evaluation of prediction performance. We leverage PageRank and effective word embedding techniques to obtain novel features. State-of-the-art performance is obtained using various statistical machine learning methods, graph convolutional networks (GCN), automated machine learning (AutoML), and explainable artificial intelligence (XAI). More specifically, our approach can achieve state-of-the-art performance with an accuracy close to $90\%$ for leaders identification with data from projects of different types. This investigation contributes to the establishment of sustainable leadership practices by aiding organizations in retaining their leadership talent. Full article

Radar emitter signal deinterleaving based on pulse description words (PDWs) is a challenging task in the field of electronic warfare because of the parameter sparsity and uncertainty of PDWs. In this paper, a modulation-hypothesis-augmented Transformer model is proposed to identify emitters from a single PDW with an end-to-end manner. Firstly, the pulse features are enriched by the modulation hypothesis mechanism to generate I/Q complex signals from PDWs. Secondly, a multiple-parameter embedding method is proposed to expand the signal discriminative features and to enhance the identification capability of emitters. Moreover, a novel Transformer deep learning model, named PulseFormer and composed of spectral convolution, multi-layer perceptron, and self-attention based basic blocks, is proposed for discriminative feature extraction, emitter identification, and signal deinterleaving. Experimental results on synthesized PDW dataset show that the proposed method performs better on emitter signal deinterleaving in complex environments without relying on the pulse repetition interval (PRI). Compared with other deep learning methods, the PulseFormer performs better in noisy environments. Full article

Spam reviews pose a significant challenge to the integrity of online platforms, misleading consumers and undermining the credibility of genuine feedback. This paper introduces an innovative AI-generated spam review detection framework that leverages Deep Learning algorithms and Natural Language Processing (NLP) techniques to identify and mitigate spam reviews effectively. Our framework utilizes multiple Deep Learning models, including Convolutional Neural Networks (CNNs), Long Short-Term Memory (LSTM) networks, Gated Recurrent Unit (GRU), and Bidirectional LSTM (BiLSTM), to capture intricate patterns in textual data. The system processes and analyzes large volumes of review content to detect deceptive patterns by utilizing advanced NLP and text embedding techniques such as One-Hot Encoding, Word2Vec, and Term Frequency-Inverse Document Frequency (TF-IDF). By combining three embedding techniques with four Deep Learning algorithms, a total of twelve exhaustive experiments were conducted to detect AI-generated spam reviews. The experimental results demonstrate that our approach outperforms the traditional machine learning models, offering a robust solution for ensuring the authenticity of online reviews. Among the models evaluated, those employing Word2Vec embeddings, particularly the BiLSTM_Word2Vec model, exhibited the strongest performance. The BiLSTM model with Word2Vec achieved the highest performance, with an exceptional accuracy of 98.46%, a precision of 0.98, a recall of 0.97, and an F1-score of 0.98, reflecting a near-perfect balance between precision and recall. Its high F2-score (0.9810) and F0.5-score (0.9857) further highlight its effectiveness in accurately detecting AI-generated spam while minimizing false positives, making it the most reliable option for this task. Similarly, the Word2Vec-based LSTM model also performed exceptionally well, with an accuracy of 97.58%, a precision of 0.97, a recall of 0.96, and an F1-score of 0.97. The CNN model with Word2Vec similarly delivered strong results, achieving an accuracy of 97.61%, a precision of 0.97, a recall of 0.96, and an F1-score of 0.97. This study is unique in its focus on detecting spam reviews specifically generated by AI-based tools rather than solely detecting spam reviews or AI-generated text. This research contributes to the field of spam detection by offering a scalable, efficient, and accurate framework that can be integrated into various online platforms, enhancing user trust and the decision-making processes. Full article

The rise of intelligent systems demands good machine translation models that are less data hungry and more efficient, especially for low- and extremely-low-resource languages with few or no data available. By integrating a linguistic feature to enhance the quality of translation, we have developed a generic Neural Machine Translation (NMT) model for Kannada–Tulu language pairs. The NMT model uses Transformer architecture and a state-of-the-art model for translating text from Kannada to Tulu and learns based on the parallel data. Kannada and Tulu are both low-resource Dravidian languages, with Tulu recognised as an extremely-low-resource language. Dravidian languages are morphologically rich and are highly agglutinative in nature and there exist only a few NMT models for Kannada–Tulu language pairs. They exhibit poor translation scores as they fail to capture the linguistic features of the language. The proposed generic approach can benefit other low-resource Indic languages that have smaller parallel corpora for NMT tasks. Evaluation metrics like Bilingual Evaluation Understudy (BLEU), character-level F-score (chrF) and Word Error Rate (WER) are considered to obtain the improved translation scores for the linguistic-feature-embedded NMT model. These results hold promise for further experimentation with other low- and extremely-low-resource language pairs. Full article

The image-recipe cross-modal retrieval task, which retrieves the relevant recipes according to food images and vice versa, is now attracting widespread attention. There are two main challenges for image-recipe cross-modal retrieval task. Firstly, a recipe’s different components (words in a sentence, sentences in an entity, and entities in a recipe) have different weight values. If a recipe’s different components own the same weight, the recipe embeddings cannot pay more attention to the important components. As a result, the important components make less contribution to the retrieval task. Secondly, the food images have obvious properties of locality and only the local food regions matter. There are still difficulties in enhancing the discriminative local region features in the food images. To address these two problems, we propose a novel framework named Dual Cross Attention Encoders for Cross-modal Food Retrieval (DCA-Food). The proposed framework consists of a hierarchical cross attention recipe encoder (HCARE) and a cross attention image encoder (CAIE). HCARE consists of three types of cross attention modules to capture the important words in a sentence, the important sentences in an entity and the important entities in a recipe, respectively. CAIE extracts global and local region features. Then, it calculates cross attention between them to enhance the discriminative local features in the food images. We conduct the ablation studies to validate our design choices. Our proposed approach outperforms the existing approaches by a large margin on the Recipe1M dataset. Specifically, we improve the R@1 performance by +2.7 and +1.9 on the 1k and 10k testing sets, respectively. Full article

Identifying and categorizing proper nouns in text, known as named entity recognition (NER), is crucial for various natural language processing tasks. However, developing effective NER techniques for low-resource languages like Urdu poses challenges due to limited training data, particularly in the nastaliq script. To address this, our study introduces a novel data augmentation method, “contextual word embeddings augmentation” (CWEA), for Urdu, aiming to enrich existing datasets. The extended dataset, comprising 160,132 tokens and 114,912 labeled entities, significantly enhances the coverage of named entities compared to previous datasets. We evaluated several transformer models on this augmented dataset, including BERT-multilingual, RoBERTa-Urdu-small, BERT-base-cased, and BERT-large-cased. Notably, the BERT-multilingual model outperformed others, achieving the highest macro F1 score of 0.982%. This surpassed the macro f1 scores of the RoBERTa-Urdu-small (0.884%), BERT-large-cased (0.916%), and BERT-base-cased (0.908%) models. Additionally, our neural network model achieved a micro F1 score of 96%, while the RNN model achieved 97% and the BiLSTM model achieved a macro F1 score of 96% on augmented data. Our findings underscore the efficacy of data augmentation techniques in enhancing NER performance for low-resource languages like Urdu. Full article

In neural machine translation (NMT), the sophistication of word embeddings plays a pivotal role in the model’s ability to render accurate and contextually relevant translations. However, conventional models with single granularity of word segmentation cannot fully embed complex languages like Chinese, where the granularity of segmentation significantly impacts understanding and translation fidelity. Addressing these challenges, our study introduces the Dynamic Multi-Granularity Translation System (DMGTS), an innovative approach that enhances the Transformer model by incorporating multi-granularity position encoding and multi-granularity self-attention mechanisms. Leveraging a Directed Acyclic Graph (DAG), the DMGTS utilizes four levels of word segmentation for multi-granularity position encoding. Dynamic word embeddings are also introduced to enhance the lexical representation by incorporating multi-granularity features. Multi-granularity self-attention mechanisms are applied to replace the conventional self-attention layers. We evaluate the DMGTS on multiple datasets, where our system demonstrates marked improvements. Notably, it achieves significant enhancements in translation quality, evidenced by increases of 1.16 and 1.55 in Bilingual Evaluation Understudy (BLEU) scores over traditional static embedding methods. These results underscore the efficacy of the DMGTS in refining NMT performance. Full article

Named Entity Recognition (NER) is a crucial component of Natural Language Processing (NLP). When dealing with the high diversity and complexity of the Chinese language, existing Chinese NER models face challenges in addressing word sense ambiguity, capturing long-range dependencies, and maintaining robustness, which hinders the accuracy of entity recognition. To this end, a Chinese NER model based on multi-level representation learning is proposed. The model leverages a pre-trained word-based embedding to capture contextual information. A linear layer adjusts dimensions to fit an Extended Long Short-Term Memory (XLSTM) network, enabling the capture of long-range dependencies and contextual information, and providing deeper representations. An adaptive multi-head attention mechanism is proposed to enhance the ability to capture global dependencies and comprehend deep semantic context. Additionally, GlobalPointer with rotational position encoding integrates global information for entity category prediction. Projected Gradient Descent (PGD) is incorporated, introducing perturbations in the embedding layer of the pre-trained model to enhance stability in noisy environments. The proposed model achieves F1-scores of 96.89%, 74.89%, 72.19%, and 80.96% on the Resume, Weibo, CMeEE, and CLUENER2020 datasets, respectively, demonstrating improvements over baseline and comparison models. Full article

Social media users often express their emotions through text in posts and tweets, and these can be used for sentiment analysis, identifying text as positive or negative. Sentiment analysis is critical for different fields such as politics, tourism, e-commerce, education, and health. However, sentiment analysis approaches that perform well on English text encounter challenges with Arabic text due to its morphological complexity. Effective data preprocessing and machine learning techniques are essential to overcome these challenges and provide insightful sentiment predictions for Arabic text. This paper evaluates a combined CNN-LSTM framework with emoji encoding for Arabic Sentiment Analysis, using the Arabic Sentiment Twitter Corpus (ASTC) dataset. Three experiments were conducted with eight-parameter fusion approaches to evaluate the effect of data preprocessing, namely the effect of emoji encoding on their real and emotional meaning. Emoji meanings were collected from four websites specialized in finding the meaning of emojis in social media. Furthermore, the Keras tuner optimized the CNN-LSTM parameters during the 5-fold cross-validation process. The highest accuracy rate (91.85%) was achieved by keeping non-Arabic words and removing punctuation, using the Snowball stemmer after encoding emojis into Arabic text, and applying Keras embedding. This approach is competitive with other state-of-the-art approaches, showing that emoji encoding enriches text by accurately reflecting emotions, and enabling investigation of the effect of data preprocessing, allowing the hybrid model to achieve comparable results to the study using the same ASTC dataset, thereby improving sentiment analysis accuracy. Full article

This study examines the influence of Tamil (L1) on the processing of English (L2) collocations during reading for Tamil-English bilingual children. Building on existing research in formulaic language, we used an online processing tool to investigate whether cross-linguistic transfer can be extended beyond single lexical items to collocations in bilingual children, a population that is underrepresented in this research area. Fifty-eight children aged 9–10 years from a school in Chennai, India, took part. Using self-paced reading, children’s reading times were measured for both congruent (with equivalent in L2) and incongruent (without equivalent in L2) English collocations embedded in short passages. There were two reading modes (single and chunk), which allowed reading times for the whole collocations and the individual words of the collocations to be examined. Results showed that children read congruent collocations more quickly than incongruent collocations in both modes. For congruent collocations, children read the second word more quickly than the first word, but the reverse was true for incongruent collocations. These results suggest that the L1 (Tamil) is activated during the processing stage of reading English collocations for Tamil-English bilingual children in this context. Full article

TextNetTopics is a novel topic modeling-based topic selection approach that finds highly ranked discriminative topics for training text classification models, where a topic is a set of semantically related words. However, it suffers from several limitations, including the retention of redundant or irrelevant features within topics, a computationally intensive topic-scoring mechanism, and a lack of explicit semantic modeling. In order to address these shortcomings, this paper proposes fasTNT, an enhanced version of TextNetTopics grounded in the Grouping–Scoring–Modeling approach. FasTNT aims to improve the topic selection process by preserving only informative features within topics, reforming LDA topics using fastText word embeddings, and introducing an efficient scoring method that considers topic interactions using Random Forest feature importance. Experimental results on four diverse datasets demonstrate that fasTNT outperforms the original TextNetTopics method in classification performance and feature reduction. Full article

As the number of publications is quickly growing in any area of science, the need to efficiently find relevant information amidst a large number of similarly themed articles becomes very important. Semantic searching through text documents has the potential to overcome the limits of keyword-based searches, especially since the introduction of attention-based transformers, which can capture contextual nuances of meaning in single words, sentences, or whole documents. The deployment of these computational tools has been made simpler and accessible to investigators in every field of research thanks to a growing number of dedicated libraries, but knowledge of how meaning representation strategies work is crucial to making the most out of these instruments. The present work aims at introducing the technical evolution of the meaning representation systems, from vectors to embeddings and transformers tailored to life science investigators with no previous knowledge of natural language processing. Full article

The rise of social media has transformed the landscape of news dissemination, presenting new challenges in combating the spread of fake news. This study addresses the automated detection of misinformation within written content, a task that has prompted extensive research efforts across various methodologies. We evaluate existing benchmarks, introduce a novel hybrid word embedding model, and implement a web framework for text classification. Our approach integrates traditional frequency–inverse document frequency (TF–IDF) methods with sophisticated feature extraction techniques, considering linguistic, psychological, morphological, and grammatical aspects of the text. Through a series of experiments on diverse datasets, applying transfer and incremental learning techniques, we demonstrate the effectiveness of our hybrid model in surpassing benchmarks and outperforming alternative experimental setups. Furthermore, our findings emphasize the importance of dataset alignment and balance in transfer learning, as well as the utility of incremental learning in maintaining high detection performance while reducing runtime. This research offers promising avenues for further advancements in fake news detection methodologies, with implications for future research and development in this critical domain. Full article