ConvNetJS
Deep Learning in the Browser
Christoph Körner
Slides available on bit.ly/1WicamQ 1
About me
• Visual Computing at Vienna University of Technology
• Intern in the Big Data Team at T-Mobile Austria
• Author of Data Visualizations with D3 and AngularJS
• Author of Learning Responsive Data Visualization
• Contributor at n3-line-chart
• Organizer of Vienna Kaggle Meetup
• LinkedIn: at.linkedin.com/in/christophkoerner
• Twitter: @ChrisiKrnr
• Google+: +ChrisiHififm
• Github: github.com/chaosmail
2
Agenda
• Deep Learning
• ConvNetJS
• Caffe models in ConvNetJS
• Whats next
3
Deep Learning (in 5 min)
4
Image Classification
Source: CS231n Github 5
Spatial Localization and Detection
Source: CS231n Lecture Notes 6
General: Regression
Source: CS UBC 7
Example
Source: CS231n Github 8
Neural Nets: Ingredients
• Linear Classification
• Non-linearities (Activation functions)
• Compute Error Loss
• Optimization (Back-propagation)
9
Linear Classifier y = Wx + b
Source: CS231n Github 10
Non-linearities (Activation functions)
• Needed for building universal approximators
Source: deepdish.io 11
Compute the Error (Loss)
Source: Udacity Tensorflow, Interactive Demo 12
Optimization (1)
Source: CS231n Github 13
Optimization (2)
Source: lmjohns3.com 14
Neural Network
Source: CS231n Github 15
Deep Neural Nets: Ingredients
• Convolution of Volumes
• Pooling
• Many layers, many parameters, much fun
16
Convolutions
Source: CS231n Github 17
Pooling
Source: CS231n Github 18
Many Layers, AlexNet
Source: ImageNet Classification with Deep Convolutional Neural Networks 19
And more Layers, GoogLeNet
Source: Going deeper with convolutions, Great Slides 20
And More Layers, VGG vs. ResNet
Source: Deep Residual Learning for Image Recognition 21
ConvNetJS
Source: Github by Andrej Karpathy 22
Convolutional Neural Nets in the Browser
• Network and Layers
○ including Conv, Pool, etc.
• Network Training
○ Including SGD, momentum, etc.
• Visualization of Activations
• JSON import and export
• Cool demos: MNIST Train, CIFAR Train, CIFAR
Source: ConvNetJS 23
ConvNetJS Layers
• Input
• Inner Product:
○ FC, Conv
• Pool (only max pool)
• Dropout
• Normalization:
○ LRN
• Loss:
○ Softmax, Regression, SVM
• Non-linearities:
○ ReLU, Sigmoid, Maxout, Tanh,
24
ConvNetJS Training
• Optimization methods:
○ SGD
○ Adam
○ Adadelta
○ Adagrad
○ Nestrov momentum
25
convnetjs.Net
• Layers stored as Arrays
○ Net.layers
• Forward and Backward pass
○ Net.forward(vol)
○ Net.backward(y)
• Import & Export
○ Net.toJSON
○ Net.fromJSON
26
convnetjs.Layer
• Layer dimensions
○ Layer.out_sx
○ Layer.out_sy
○ Layer.out_depth
• Forward and Backward pass
○ Layer.forward(vol)
○ Layer.backward(y)
• Stores last volumes
○ Layer.in_act
○ Layer.out_act
27
convnetjs.Vol
• Volumes stored as Typed Arrays (column vectors)
○ Vol.get(x, y, d)
○ Vol.set(x, y, d, value)
• Dimensions
○ Vol.sx
○ Vol.sy
○ Vol.depth
• Weights and Gradients
○ Vol.w
○ Vol.dw
28
Typed Arrays
• Float32Array, Float64Array, …
• Typed arrays are fast for storing large blobs
Array Float64Array Float32Array
Create zero filled 0.624 0.266 0.139
Fill iter. / method 0.228 0.228 / 0.817 0.270 / 0.954
Copy array 5.425 0.050 0.024
29
Fast for loops
for (var i = 0, len = A.length; i < len; ++i) {
// Code goes here
}
Source: Stackoverflow 30
Problems of ConvNetJS
• Slow, due to sequential computation
• Array layer structure (no Inception modules)
• No Recurrent Networks
• Dropout
• Limited memory in the browser, JSON
• I don’t want to train models in the browser
31
Using pre-trained Models
• Offline training (many GPUs, long time)
• Export weights
• Perform only forward pass (no BP needed)
32
Finding pre-trained Models
• Popular pre-trained models are available
○ Caffe
○ Model-Zoo
○ FCN Berkley Vision
• Structure in ProtoBuf files *.prototxt
• Weights in binary files *.caffemodel
33
CaffeJS - Caffe Models in the Browser
• Work in progress…
• Parses *.prototxt files to ConvNetJS models
• Transforms *.caffemodel weights into text files
(one file per layer) - #1669 in Keras
• Updates weights in ConvNetJS models
• Graph structure for layers + layerIterator
34
Why CaffeJS
• Teaching & Learning
○ No requirements (but a browser)
○ Understand & analyze Deep Nets
○ Debugging of Deep Nets (FF and BP)
○ Visualize the filters, layers, activations, etc.
○ Feed webcam stream into Deep Nets
35
Problems of CaffeJS
• Forward pass still slow
○ 6s for GoogLeNet
○ Most time spend in early convolutions
• Too much overhead for weights (fc6 and fc7)
• Uses only layers, no blobs ()
• Memory issues (above 200MB) - can we convert
weights into images? 36
Whats next
• Network in a Network (NIN)
• Fully Convolutional Nets (FCN)
• WebCL: Heterogeneous parallel computing
• WebAssembly: Compilation to the web
• Deep Compression: AlexNet on 7MB
• More Layers!
37
Some more useful resources
• Tensorflow Playground: Neural Nets
• CS231n: Lecture, Github, and videos
• CS231n: Caffe Tutorial
• Udacity Tensorflow
• DeepDream
38
Thank you.
39
