TensorMONK

A collection of deep learning architectures (a PyTorch implementation).

Dependencies

• python >= 3.6
• PyTorch > 1.3
• torchvision
• visdom

Training models on 2012 ImageNet recognition task

If you have more nvidia cards & cores available, adjust the batch size (BSZ), number of GPUs (gpus), & number of threads (cpus) accordingly in the ./ImageNet.sh. Next, select an available architecture and update your train & validation folder location (trainDataPath and testDataPath). Finally, run ./ImageNet.sh.

Training CapsuleNet on MNIST

To replicate Hinton's paper on MNIST, run the following:

python Capsule.py -A capsule -B 256 -E 500 --optimizer adam --gpus 2 --cpus 6 --trainDataPath ./data --testDataPath ./data --replicate_paper

Ignore the replicate_paper argument to create a deep architecture (with few residual blocks before primary capsule). You can essentially add any block available in NeuralLayers to create a deeper architecture, which is followed by a primary capsule and secondary capsule. However, do consider two things 1. if you do reconstruction, update the reconstruction network relative to tensor_size, 2. capsule nets do require a good amount of gpu ram.

Generative Adversarial Networks

Progressive Growing of GANs

Trained on CIFAR10 (pggan-cifar10.py) -- requires more training (more gpus)!

References

Activation Functions

• Maxout Networks
• SWISH: A SELF-GATED ACTIVATION FUNCTION
• Mish: Self Regularized Non-Monotonic Activation Function

Classification

• Deep Neural Decision Forests

Generative Models

• Auto-Encoding Variational Bayes
• Progressive Growing of GANs
• Self-Attention Generative Adversarial Networks

Image Recognition Models

• Aggregated Residual Transformations for Deep Neural Networks
• Deep Residual Learning for Image Recognition
• Densely Connected Convolutional Networks
• Dynamic Routing Between Capsules
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions
• ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
• Squeeze-and-Excitation Networks

Image Segmentation Models

• AnatomyNet: Deep Learning for Fast and Fully Automated Whole-volume Segmentation of Head and Neck Anatomy
• ContextNet: Exploring Context and Detail for Semantic Segmentation in Real-time
• U-Net: Convolutional Networks for Biomedical Image Segmentation

Local Features

• Learning Discriminative and Transformation Covariant Local Feature Detectors

Loss Functions

• AN EXPLORATION OF SOFTMAX ALTERNATIVES BELONGING TO THE SPHERICAL LOSS FAMILY
• Analyzing and Improving Representations with the Soft Nearest Neighbor Loss
• ArcFace: Additive Angular Margin Loss for Deep Face Recognition
• CosFace: Large Margin Cosine Loss for Deep Face Recognition
• Rethinking Feature Distribution for Loss Functions in Image Classification
• Tversky loss function for image segmentation using 3D fully convolutional deep networks

Object Detection Models

• SSD: Single Shot MultiBox Detector (MobileNetV2SSD320 and TinySSD320)

Optimizer

• Lookahead Optimizer: k steps forward, 1 step back
• On the Variance of the Adaptive Learning Rate and Beyond

Regularizations

• DropBlock: A regularization method for convolutional networks