For this use case we leverage a pretrained Resnet-50 as the image feature extractor and LSTM as the decoder for the caption generation. Adding attention for better caption generation.
Both embeddings - Layer embedding which was trained alongside the entire architecture, as well as the pretrained Glove model which was also trained along side the model. Embedding unfreeze point added for better training and convergence.
model.py class has the EncoderCNN class for the image feature extraction, DecoderRNN which leverages LSTM for decoding and caption generation, Additive Attention which attends to one of the 49 spatial patches based on the previous hidden state context and using this as well as the input token embedding decoding step takes place. LSTM is seeded by the initial hidden state by the global average pooling of the output of the Resnet-50. ImageCaptioningModel wraps around all of the above mentioned.
To train the model run python train.py
To predict run python pred.py --image_path "your image path"
Webcam_ui utilizes the webcam to generate caption realtime.
checkpoint has the saved model both normal as well as which utilizes pretrained embedding model.
Training setting with model captioning quality every epoch: Loading GloVe vectors… loaded 400000 tokens Warning: missing image, skipping C:/Users/shubh/Desktop/Workspace/Image caption generator/Flicker8k_Dataset/2258277193_586949ec62.jpg.1.jpg Loaded batch: torch.Size([32, 3, 224, 224]) torch.Size([32, 26])
Epoch 1/25, Loss: 4.6391 Test caption @ epoch 0: a man in a black dog is a .
Epoch 2/25, Loss: 3.7387 Test caption @ epoch 1: a man in a red shirt is jumping on a .
Epoch 3/25, Loss: 3.4482 Test caption @ epoch 2: a man in a red shirt is jumping on a .
Epoch 4/25, Loss: 3.2882 Test caption @ epoch 3: a man in a red shirt is standing on a .
Epoch 5/25, Loss: 3.1602 Test caption @ epoch 4: a man in a red shirt is jumping over a large tree .
Epoch 6/25, Loss: 3.0600 Test caption @ epoch 5: a young girl in a pink shirt is standing on a bench in a park .
Epoch 7/25, Loss: 2.9802 Test caption @ epoch 6: a boy in a red shirt is sitting on a bench .
Epoch 8/25, Loss: 2.9074 Test caption @ epoch 7: a man in a red shirt is standing on a bench in a park .
Epoch 9/25, Loss: 2.8391 Test caption @ epoch 8: a man in a red shirt is standing in a red shirt and a black shirt and a black shirt
Epoch 10/25, Loss: 2.7751 Test caption @ epoch 9: a man in a red shirt is standing in a .
Epoch 11/25, Loss: 2.7231 Test caption @ epoch 10: a young girl in a pink dress is playing with a red ball in the air .
Epoch 12/25, Loss: 2.6785 Test caption @ epoch 11: a woman in a green shirt is standing in front of a crowd of people .
Epoch 13/25, Loss: 2.6345 Test caption @ epoch 12: a woman in a red shirt is standing on a bench .
Epoch 14/25, Loss: 2.5905 Test caption @ epoch 13: a young boy is playing with a tennis ball .
Epoch 15/25, Loss: 2.5640 Test caption @ epoch 14: a woman in a yellow shirt is sitting on a bench .
Epoch 16/25, Loss: 2.5220 Test caption @ epoch 15: a young boy is sitting on a bench with a brown dog .
Epoch 17/25, Loss: 2.4949 Test caption @ epoch 16: a man in a white shirt and a brown dog are standing in the water .
Epoch 18/25, Loss: 2.4524 Test caption @ epoch 17: a woman is standing in a .
Epoch 19/25, Loss: 2.4351 Test caption @ epoch 18: a young boy is playing with a toy in a .
Epoch 20/25, Loss: 2.4106 Test caption @ epoch 19: a man in a yellow shirt is standing on a bench with a brown dog .
Epoch 21/25, Loss: 2.3791 Test caption @ epoch 20: a young boy is standing on a bed with a large stick in his mouth .
Epoch 22/25, Loss: 2.3472 Test caption @ epoch 21: a woman in a white shirt is sitting on a couch .
Epoch 23/25, Loss: 2.3223 Test caption @ epoch 22: a man and a woman are sitting on a .
Epoch 24/25, Loss: 2.2973 Test caption @ epoch 23: a man in a red shirt is standing on a in front of a large brown dog . Epoch 25/25, Loss: 2.2796 Test caption @ epoch 24: two young boys are playing with a .
Saved vocabulary to vocab.json Saved full Vocabulary object to vocab.pkl