Algorithm : Pseudocode of CNN

1. Defining class convolution layer

1.1. Initialize

lr # learning rate , f_in # input channels, f_out # output channels

k # kernel size, p # padding, s # stride

w ← random weights (shape = (f_out, f_in, k, k))

b ← random weights (shape = (f_out, 1))

1.2. Define zero_padding (input):

1.2.1. w, h ← input_width, input_height

1.2.2. w1= (w+2*p), h1= (h+2*p)

1.2.3. x_out = zero array (shape = w1,h1) #creating zero array for output

1.2.4. x_out [p:w+p, p:p+h] ← input

1.2.5. Return x_out

1.3. Define forward (x_in):

1.3.1. c = x_in channels

1.3.2. w = x_in_width + 2*p

1.3.3. h = x_in_height + 2*p

1.3.4. x_p = zero array (shape = (c,w,h))

1.3.5. For i in range (c) :

x_p [i, :, :] = zero_padding (x_in [i, :, :])

1.3.6. w1 = (w-k)/s +1 ; h1 = (h-k)/s +1

1.3.7. x_out = zero array (f_out, w1,h1)

1.3.8. for i in range(f_out):

for j in range(0,w1,s):
 for m in range(0,h1,s):

x_out[i,j,m] = sum(x_p[:, j:j+k, m:m+k]*w[i,:,:,:])+b[i]

1.3.9. return x_out

1.4. Define backward (dy):

1.4.1. c, w, h = x_p.shape

1.4.2. dx = zero array(shape = (c,w,h)) # creating a zero array for gradient at this layer

which is passing to the next layer in the backward propagation

1.4.3. dw = zero array (w.shape) # weight updation factor at this layer

1.4.4. db = zero array (b.shape) # bias updation factor at this layer

1.4.5. f, w1, h1 = dy.shape # input to the layer in the backward propagation

1.4.6. Calculating gradients

For i in range (f):

For j in range (0,w1,stride):

For m in range (0,h1,stride):

dw[i,:,:,:] += (dy[i,j,m] * x_p[:, j:j+k, m:m+k])

dx[:,j:j+k, m:m+k] += (dy[i,j,m] * w[i,:,:,:])

For i in range (f):

2. Defining class max pooling layer

2.1. Initialize

k # kernel size, s # stride

2.2. Define forward (x_in):

2.2.2. w1 = (w-k)/s +1, h1 = (h-k)/s+1

2.2.3. x_out = zero array (shape = (c,w1,h1))

2.2.4. for i in range (c):

for j in range (w1):

for m in range (h1):

x_out[i,j,m] = max(x_in [i, j*s:k+(j*s), m*s:k+(m*s)])

2.2.5. Return x_out

2.3. Define backward (dy):

2.3.1. dx = zero array (x_in.shape)

2.3.2. c,w,h = x_in.shape

2.3.3. for i in range (c) :

for j in range (0, w, k):

for m in range (0, h, k):

img_region = x_in [i, j:j+k, m:m+k]

maxm = max(img_region)

idx, idy = row and column of maximum value

dx[i, j+idx, m+idy] = dy[i, int(j/k), int(m/k)] # placing the dy

values in dx at the positions of maximum values of x_in.

2.3.4. return dx

3. Define class dense layer:

3.1. Initialize

f_in # input filter, f_out # output filter, lr # learning rate

w ← random weights (shape = (f_out, f_in))

b ← random weights (shape = (f_out, 1))

3.2.1. x_out = x_in * w + b

3.2.2. return x_out

3.3. Define backward (dy):

3.3.1. dw = dy * x_in

3.3.2. db = sum (dy, axis=1) # taking sum of dy through axis=1

3.3.3. dx = dy* w

3.3.4. w = w - (lr*dw)

3.3.5. b = b - (lr*db)

3.3.6. return dx

4. Define class flatten layer:

4.1. Define forward (x_in):

4.1.1. c = x_in channels, w = x_in width, h = x_in height

4.1.2. x_out = reshape x_in to (1, c * w * h)

4.1.3. return x_out

4.2. Define backward (dy):

4.2.1. Reshape dy into (c, w, h)

4.2.2. Return dy

5. Define class ReLU :

5.1. Define forward (x_in):

5.1.1. x_out = maximum(0, x_in)

5.1.2. return x_out

5.2. Define backward (dy):

5.2.1. dx = dy

5.2.2. dx = 0, where x_in < 0

6. Define class softmax:

6.1. Define forward (x_in):

6.1.1. x_out = exp(x_in)/sum(exp(x_in))

6.1.2. Return x_out

6.2. Define backward (dy):

6.2.1. dx = x_out - dy

6.2.2. Return dx

7. Define class cross entropy (y, y_hat):

7.1. cost = -(1/m) * sum(y*log(y_hat)) # m = number of samples

7.2. Return cost

8. Define LeNet model:

8.1. Initialize

lr # learning rate

layer = [] # create an empty list

Append Convolution2D(lr = lr, f_in=1, f_out=6, k=5, p=2, s=1) to the list, layer

Append ReLU() to layer

Append Maxpooling2D(k=2, s=2) to layer

Append Convolution2D(lr = lr, f_in=6, f_out=16, k=5, p=0, s=1) to the layer

Append ReLU() to the layer

Append Maxpooling2D(k=2, s=2) to the layer

Append Convolution2D(lr = lr, f_in=16, f_out=120, k=5, p=0, s=1) to the layer

Append ReLU() to the layer

Append Flatten() to the layer

Append denselayer(f_in=120, f_out=84, lr = lr) to the layer

Append denselayer(f_in=84, f_out=10, lr = lr) to the layer

Append softmax() to the layer

lay_num = length of list layer

8.2. Defining training (x_train, y_train, epochs):

8.2.1. cost = [], accuracy = [] # creating empty lists for cost and accuracy

8.2.2. For i in range (epochs):

pred = [] # creating an empty list for predicted values

For j in range (length (x_train)):

x = jth sample of x_train

y = jth sample of y_train

for l in range(lay_num):

out = layer[l].forward(x) # calculating output using forward path

x = out

Appending output to the list pred

loss = cross entropy (y, out) # calculating loss

dy = y

for m in range(lay_num-1,-1,-1):

o = layer[m].backward(dy) # backward propagation

dy = o

Convert list pred to numpy array

Accuracy = sum (pred==y_train)/length(y_train)

Loss = cross entropy (y_train, pred)

Append (Loss) to cost

Append (Accuracy ) to accuracy

8.2.3. return cost, accuracy

8.3. Define prediction (data):

8.3.1. y_p = [] # creating an empty list for prediction

8.3.2. test_size = length of data

8.3.3. for i in range(test_size):

x = ith element of data

for l in range(lay_num):

output = layer[l].forward(x)

x = output

Appending (x) to y_p

8.3.4. return y_p

9. Calling the function:

9.1. lr = 0.01 # defining learning rate

9.2. lenet = LeNet2D(lr) # calling the model

9.3. cost,accuracy = lenet.training(training_data, training_labels, 10) # training

10. Prediction

lenet.prediction(testing_data)