Image Caption Generation using Natural Language

Processing and Deep Learning

Shreya Sanjay Shingade Dr.S.A. Ubale
Department of Data Science Department of Data Science
Zeal College of Engineering Zeal College of Engineering
and Research and Research
Pune,India Pune,India

Abstract—An image-based web crawler is a web crawler that image correctly and comprehensively, but also to mix and
searches for data using similar photos. On the web, there is a organise the semantics of the image using human language.
large assortment of image assets, with a significant proportion The subtasks of picture captioning, such as identifying
of photographs carrying both named and unidentified captions.
Users must sort through the photos according to their require- semantic aspects like visual objects, object properties, and
ments. A significant proportion of users are unable to recover the sceneries, are difficult enough, but organising words and
necessary images as a result of their unanticipated appropriate phrases to communicate this information adds to the challenge.
inscription on their photographs. The goal of our project is
to create an automated photo caption depending on the image
quality. To begin, a picture’s content should be easily understood, II. R EVIEW OF L ITERATURE
followed by a statement or declaration that is consistent with the
image’s grammatical laws and semantical information. Computer In this paper[1], author propose a novel versatile
vision and natural language processing technologies are required consideration model with a visual sentinel. At each time step,
to merge the two forms of material, which is a difficult task. our model concludes whether to take care of the picture (and
The goal of the paper is to generate mechanical inscriptions provided that this is true, to which districts) or to the visual
by analysing the information of an image. Currently, pictures
must be removed through human involvement, which is nearly sentinel. The model concludes whether to take care of the
impossible in large databases. As a contribution, the picture infor- picture and where all together to extricate significant data for
mation base is sent to a deep neural network. The Convolutional consecutive wordage. Author test his strategy on the COCO
Neural Network encoder creates captions that extract the image’s picture subtitling 2015 test dataset and Flickr30K.
highlights and nuances, while the Recurrent Neural Network
decoder interprets the image’s highlights and articles to produce
a continuous, intelligible description of the image. In this work[2], authors propose a joined base up
Index Terms—Deep Learning, part of speech, image and topdown consideration component that empowers
captioning, multi-task learning thoughtfulness regarding be determined at the degree of
items and other striking image areas. This is the normal
reason for thoughtfulness regarding be thought of. Inside our
I. I NTRODUCTION methodology, the base up system (in light of Faster R-CNN)
Image captioning, which tries to link image with proposes picture districts, each with a related element vector,
language, has become a popular study topic to aid research while the top-down component decides highlight weightings.
in areas such as cross-modal retrieval and the support of Applying this way to deal with picture inscribing, outcomes
visually impaired persons. Not only must an image captioning on the MSCOCO test worker set up another best in class for
model recognise the important objects in a picture, their the assignment, accomplishing CIDEr/SPICE/BLEU-4 scores
qualities, and their relationships, but it must also structure of 117.9, 21.5 and 36.9, individually.
this information into a syntactically and semantically correct
sentence. Recent captioning models, with the advancements In this paper[3], Author present a novel convolutional neural
of Neural Machine Translation, mainly use the encoder organization named SCA-CNN that joins Spatial and Channel
decoder structure to ”translate” a picture into a sentence, with wise Attentions in a CNN. In the undertaking of picture
promising results. Researchers have achieved substantial inscribing, SCA-CNN progressively regulates the sentence
progress in recent years in fields such as image classification, age setting in multi-layer highlight maps, encoding where
feature classification, object detection and recognition, scene (i.e., mindful spatial areas at different layers) and what (i.e.,
recognition, action recognition, and so on. Having a machine mindful channels) the visual consideration is. Authors assess
develop natural language descriptions for an image, on the the proposed SCA-CNN design on three benchmark picture
other hand, remains a complex and challenging issue. This subtitling datasets: Flickr8K, Flickr30K, and MSCOCO. It
challenge unites two very distinct media formats, needing is reliably seen that SCA-CNN fundamentally beats best in
computers to not only interpret the visual information of the class visual consideration based picture inscribing techniques.
 Author propose[8] novel Deep Hierarchical Encoder-Decoder
In this paper[4], authors present Long Short-Term Network (DHEDN) is proposed for picture inscribing, where
Memory with Attributes (LSTM-A) a novel engineering a profound progressive structure is investigated to isolate
that coordinates ascribes into the effective Convolutional the elements of encoder and decoder. This model is able
Neural Networks (CNNs) additionally Recurrent Neural to do productively applying the portrayal limit of profound
Networks (RNNs) picture subtitling system, via preparing organizations to intertwine significant level semantics of
them in a start to finish way. Especially, the learning of vision and language in creating inscriptions. In particular,
characteristics is fortified by coordinating between property visual portrayals in high degrees of deliberation are at the
relationships into Multiple Instance Learning (MIL). To same time considered, and every one of these levels is
consolidate credits into subtitling, Author develop variations related to one LSTM. The base most LSTM is applied as
of designs by taking care of picture portrayals and properties the encoder of printed inputs. The use of the center layer in
into RNNs in various manners to investigate the shared yet encoder-decoder is to upgrade the interpreting capacity of top-
additionally fluffy connection between them. Broad analyses most LSTM. Moreover, contingent upon the presentation of
are led on COCO image subtitling dataset and our system semantic upgrade module of picture highlight and dispersion
shows clear upgrades when contrasted with cutting edge consolidate module of text include, variations of structures
profound models. of our model are built to investigate the effects and shared
collaborations among the visual portrayal, literary portrayals
Author[5] propose Scene Graph Auto-Encoder (SGAE) and the yield of the center LSTM layer. Especially, the system
that consolidates the language inductive inclination into the is preparing under a fortification learning technique to address
encoder decoder image subtitling structure for more human- the presentation predisposition issue between the preparation
like subtitles. Instinctively, we people utilize the inductive and the testing by the arrangement slope enhancement.
inclination to make collocations and logical deduction in talk.
For instance, when we see the connection ”individual on Late works[9] in image subtitling have demonstrated
bicycle”, it is normal to supplant ”on” with ”ride” and surmise very promising crude execution. In any case, we understand
”individual riding bicycle on a street” even the ”street” isn’t that the majority of these encoder-decoded style networks
clear. In this way, misusing such inclination as a language with consideration don’t scale normally to huge jargon
earlier is required to help the regular encoder-decoder models size, making them hard to utilize on implanted framework
more outlandish overfit to the dataset predisposition and with restricted equipment assets. This is on the grounds
spotlight on thinking. that the size of word and yield inserting networks develop
relatively with the size of jargon, antagonistically influencing
In this work[6], Author propose an image subtitling approach the conservativeness of these organizations. To address this
in which a generative intermittent neural organization can impediment, this paper presents a shiny new thought in the
zero in on various pieces of the information image during space of picture inscribing. That is, author tackles the issue
the age of the inscription, by abusing the molding given by of conservativeness of picture inscribing models which is
a saliency forecast model on which parts of the picture are heretofore unexplored. Proposed model, named COMIC,
remarkable and which are logical. Authors show, through accomplishes tantamount outcomes in five basic assessment
broad quantitative and subjective tests for enormous scope measurements with state-of-the-workmanship approaches on
datasets, that our model accomplishes better execution with both of the MS-COCO and InstaPIC1.1M datasets.
deference than subtitling baselines with and without saliency
and to various best in class approaches consolidating saliency In this paper[10], author propose a structure dependent
and subtitling. on scene charts for picture inscribing. Scene charts contain
plentiful organized data since they portray object elements in
In this paper[7], author present ”MLADIC”, a novel pictures as well as present pairwise connections. To use both
Multitask Learning Algorithm for cross-Domain Image visual highlights and semantic information in organized scene
Subtitling. MLADIC is a perform various tasks framework charts, we extricate CNN highlights from the jumping box
that all the while upgrades two coupled targets through a counterbalances of article elements for visual portrayals, and
double learning component: image inscribing and text-to- concentrate semantic relationship highlights from significantly
picture combination, with the expectation that by utilizing the increases (e.g., man riding bicycle) for semantic portrayals.
relationship of the two double undertakings, we can upgrade After acquiring these highlights, we acquaint a various leveled
the picture inscribing execution in the target area. Solidly, the attention based module with learn discriminative highlights
picture inscribing task is prepared with an encoder-decoder for word age at each time step. The test results on benchmark
model (i.e., CNN-LSTM) to create printed depictions of the datasets show the predominance of our strategy contrasted
info pictures. The picture blend task utilizes the contingent and a few cutting edge strategies.
generative ill-disposed organization (CGAN) to integrate
conceivable pictures dependent on text depictions.
 III. P ROPOSED M ETHODOLOGY is the extraction layer of features in which each neuron’s
A. Methodology to solve the task input is directly connected to its previous layer’s local ready
fields and local features are extracted. The spatial relationship
The process of captioning images can be broken up func-
between it and other features will be shown once those local
tionally into two modules, one is an image model that extracts
features are extracted. The other layer is feature map layer;
the characteristics and complexities of our image and the other
Every feature map in this layer is a plane, the weight of the
is a linguistic model, converting features and artefacts that are
neurons in one plane are same. The feature plan”s structure
converted into a natural expression in the image based model.
make use of the function called sigmoid. This function known
Typically using the Convolutional Neural Network algorithm
as activation function of the CNN, which makes the feature
for the image-based model (such as the encoder). And it
map have shift in difference. In the CNN each convolution
depend on a Recurrent Neural Network for the language
layer is come after a computing layer and it’s usage is to
dependent model (viz decoder). Semantic attention has been
find the local average as well as the second extract; this
shown to be effective in improving the performance of image
extraction of two feature is unique structure which decreases
captioning. The core of semantic attention based methods is to
the resolution.
drive the model to attend to semantically important words, or
Step 1: Select the dataset.
attributes. In previous works, the attribute detector and the
Step 2: Perform feature selection using information gain and
captioning network are usually independent, leading to the
ranking
insufficient usage of the semantic information. Also, all the
Step 3: Apply Classification algorithm CNN
detected attributes, no matter whether they are appropriate
Step 4: Calculate each Feature fx value of input layer
for the linguistic context at the current step, are attended to
Step 5: Calculate bias class of each feature
through the whole caption generation process. This may some-
Step 6: The feature map is produced and it goes to forward
times disrupt the captioning model to attend to incorrect visual
pass input layer
concepts. To solve these problems, we introduce two end-to-
Step 7: Calculate the convolution cores in a feature pattern
end trainable modules to closely couple attribute detection
Step 8: Produce sub sample layer and feature value.
with image captioning as well as prompt the effective uses
Step 9: Input deviation of the kth neuron in output layer is
of attributes by predicting appropriate attributes at each time
Back propagated.
step.
Step 10: Finally give the selected feature and classification
B. Architecture results.

IV. C ONCLUSION
In this paper, A proposal of a novel deep neural
network(NDNN) model to improve the image captioning
methods. The NDNN explores the spatio-temporal relationship
in the visual attention and learns the attention transmission
mechanism through a tailored LSTM model, where the matrix-
form memory cell stores and propagates visual attention, and
the output gate is reconstructed to filter the attention values.
Combined with the language model, both of the generated
words and the visual attention areas obtain memory in the
space. The embedding of the NDNN model in three classical
attention-based image captioning frameworks, and adequate
experimental results on the MS COCO and Flicker dataset
demonstrate the superiority of the proposed NDNN.

R EFERENCES
[1] J. Lu, C. Xiong, D. Parikh, and R. Socher, “Knowing when to look:
Adaptive attention via a visual sentinel for image captioning,” in Proc.
IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jul. 2017, pp.
3242–3250.
Fig. 1. Proposed System Architecture [2] P. Anderson et al., “Bottom-up and top-down attention for image caption-
ing and visual question answering,” in Proc. IEEE/CVF Conf. Comput.
C. Algorithms Vis. Pattern Recognit., Jun. 2018, pp. 6077–6086.
[3] L. Chen et al., “SCA-CNN: Spatial and channel-wise attention in con-
1. CNN(Convolution neural network) volutional networks for image captioning,” in Proc. IEEE Conf. Comput.
The structure of CNN algorithm includes two layers.First Vis. Pattern Recognit. (CVPR), Jul. 2017, pp. 5659–5667.
[4] T. Yao, Y. Pan, Y. Li, Z. Qiu, and T. Mei, “Boosting image captioning
with attributes,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Oct. 2017,
pp. 4904–4912.
[5] X. Yang, K. Tang, H. Zhang, and J. Cai, “Auto-encoding scene graphs
for image captioning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern
Recognit. (CVPR), Jun. 2019, pp. 10685–10694.
[6] M. Cornia, L. Baraldi, G. Serra, and R. Cucchiara, “Paying more attention
to saliency: Image captioning with saliency and context attention,” ACM
Transactions on Multimedia Computing, Communications, and Applica-
tions, vol. 14, no. 2, p. 48, 2018.
[7] M. Yang, W. Zhao, W. Xu, Y. Feng, Z. Zhao, X. Chen, and K. Lei, “Mul-
titask learning for cross-domain image captioning,” IEEE Transactions on
Multimedia, vol. 21, no. 4, pp. 1047–1061, 2018.
[8] X. Xiao, L. Wang, K. Ding, S. Xiang, and C. Pan, “Deep hierarchical
encoder-decoder network for image captioning,” IEEE Transactions on
Multimedia, 2019.
[9] J. H. Tan, C. S. Chan, and J. H. Chuah, “Comic: Towards a compact image
captioning model with attention,” IEEE Transactions on Multimedia,
2019.
[10] X. Li and S. Jiang, “Know more say less: Image captioning based on
scene graphs,” IEEE Transactions on Multimedia, 2019.
[11] Z. Zhang, Q. Wu, Y. Wang, and F. Chen, “High-quality image captioning
with fine-grained and semantic-guided visual attention,” IEEE Transac-
tions on Multimedia, vol. 21, no. 7, pp. 1681–1693, 2018.
[12] M. Tanti, A. Gatt, and K. P. Camilleri, “Where to put the image in an
image caption generator,” Natural Language Engineering, vol. 24, no. 3,
pp. 467–489, 2018.