A novel method for image captioning using multimodal feature fusion employing mask RNN and LSTM models

Image captioning is a technique that allows us to use computers to interpret the information in photographs and make written text. The use of deep learning to interpret image information and create descriptive text has become a widely researched issue since its establishment. Nevertheless, these strategies do not identify all samples that depict conceptual ideas. In reality, the vast majority of them seem to be irrelevant to the matching tasks. The degree of similarity is determined only by a few relevant semantic occurrences. This duplicate instance can also be thought of as noise, as it obstructs the matching process of a few meaningful instances and adds to the model's computational effort. In the existing scheme, traditional convolutional neural networks (CNN) are presented. For that reason, captioning is not effective due to its structure. Furthermore, present approaches frequently require the deliberate use of additional target recognition algorithms or costly human labeling when extracting information is required. For image captioning, this research presents a multimodal feature fusion-based deep learning model. The coding layer uses mask recurrent neural networks (Faster RCNN), the long short-term memory has been used to decode, and the descriptive text is constructed. In deep learning, the model parameters are optimized through the method of gradient optimization. In the decoding layer, dense attention mechanisms can assist in minimizing non-salient data interruption and preferentially input the appropriate data for the decryption stage. Input images are used to train a model that, when given the opportunity, will provide captions that are very close to accurately describing the images. Various datasets are used to evaluate the model's precision and the fluency, or mastery, of the language it acquires by analyzing picture descriptions. Results from these tests demonstrate that the model consistently provides correct descriptions of input images. This model has been taught to provide captions or words describing an input picture. To measure the effectiveness of the model, the system is given categorization scores. With a batch size of 512 and 100 training epochs, the suggested system shows a 95% increase in performance. The model's capacity to comprehend images and generate text is validated by the experimental data in the domain of generic images. This paper is implemented using Python frameworks and also evaluated using performance metrics such as PSNR, RMSE, SSIM, accuracy, recall, F1-score, and precision.

Enquiries about data availability should be directed to the authors.

The authors have not disclosed any funding.

Authors and Affiliations

1. Department of Computer Science and Engineering, Kongu Engineering College (Autonomous), Perundurai, Erode, Tamil Nadu, India

   Kumaravel Thangavel

2. Department of Computer Science and Engineering, Kongu Engineering College(Autonomous), Perundurai, Erode, Tamil Nadu, India

   Natesan Palanisamy

3. Department of Electronics and Communication Engineering, Kongu Engineering College (Autonomous), Perundurai, Erode, Tamil Nadu, India

   Suresh Muthusamy

4. Department of Electronics and Communication Engineering, Vel Tech Rangarajan Dr.Sagunthala R&D Institute of Science and Technology, Avadi, Chennai, Tamil Nadu, India

   Om Prava Mishra

5. Department of Information Technology, Panimalar Engineering College (Autonomous), Poonamallee, Chennai, Tamil Nadu, India

   Suma Christal Mary Sundararajan

6. Department of Mechanical Engineering, Government Engineering College, Patan, Gujarat, India

   Hitesh Panchal

7. Department of Electrical and Electronics Engineering, PSG College of Technology, Coimbatore, Tamil Nadu, India

   Ashok Kumar Loganathan

8. Department of Electrical and Electronics Engineering, Theni Kammavar Sangam College of Technology, Theni, Tamil Nadu, India

   Ponarun Ramamoorthi

Corresponding author

Correspondence to Suresh Muthusamy.

Competing interests

The authors have not disclosed any competing interests.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions