1
Modul Pembelajaran
P152110007- Deep Learning
Optimasi model deep learning
Abstrak Sub-CPMK
Mahasiswa mampu menjelaskan Memahami optimasi model deep learning (CPMK-
mengenai optimasi model deep 4)
learning.
Pendahuluan
Deep learning has been evolving at a very fast pace and its application can be found in
every aspect of human life at the moment. With thousands of models being released every
month and capabilities improving with each one of them it has become essential to
optimize them such that they can be put to practical use. More often than not modelers
find it difficult to productionalize their models owing to hardware limitations or low
throughput rate caused by unoptimized models.
There are various methods one can use to optimize deep learning models, eg:
Quantization, where one performs computations and stores tensors at lower bit widths
than floating point precision. As a result, this allows for a more compact model
representation and the use of high-performance vectorized operations on many hardware
platforms thereby, saving us a lot of inference computation cost and maintaining inference
accuracies. There are other methods as well such as pruning, knowledge distillation, etc.
However, all these methods require changes in network architecture or training
pipeline followed by fine-tuning which might be time-consuming.
Here comes OpenVino, which is an open-source toolkit for optimizing and deploying
deep learning models developed by Intel. It has the potential to improve model
performance and efficiency on various hardware platforms. One can also deploy
OpenVino converted models to cloud instances such as AWS C5 which have dedicated
Intel CPUs and would be highly efficient for running inferences on these optimized
models.
Source: OpenVino docs
In this module, I would walk you through an example to convert the Pytorch CycleGAN
model to OpenVino and perform inference with the same and also observe how it helps in
improving latency. We will first convert the Pytorch model to Onnx and then to OpenVino.
2023 Deep Learning
2 Dr.Hadi Santoso, S.Kom., M.Kom
Biro Bahan Ajar E-learning dan MKCU
http://pbael.mercubuana.ac.id/
For the purpose of this article, we’ll be using CycleGAN’s pre-trained Yosemite summer-
to-winter model and also their inference pipeline.
Conversion Pipeline
Deep Learning Model Optimization, using OpenVino
Pytorch to Onnx conversion pipeline
### Assuming you have cloned the Cycle GAN repo from github:
https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix
### run download_cyclegan_model.sh script to download pretrained
summer2winter_yosemite_pretrained model (instructions on github page)
### Create a jupyter notebook and follow the below code snippets
from models import create_model
from options.test_options import TestOptions
import torch
from polygraphy.backend.onnx import fold_constants
import onnx
opt = TestOptions.parse()
opt.netG = 'resnet_9blocks'
opt.norm = 'instance'
opt.checkpoints_dir = './checkpoints' ### directory storing the pre-
trained model
opt.name = 'summer2winter_yosemite_pretrained'
model = create_model(opt)
model.setup(opt)
### conversion to onnx
dummy_inp = torch.rand(1, 3, 256, 256)
onnx_filepath = './summer2winter_yosemite.onnx'
torch.onnx.export(model, dummy_inp, onnx_filepath, opset_version=10,
input_names = ['inp_img'])
### optional step to do Constant folding which involves pre-computing
expressions that do not depend on runtime information
converted_onnx_model = onnx.load('./summer2winter_yosemite.onnx')
constant_folded_model = fold_constants(onnx_model)
onnx.save(constant_folded_model, './summer2winter_yosemite_folded.onnx')
2023 Deep Learning
3 Dr.Hadi Santoso, S.Kom., M.Kom
Biro Bahan Ajar E-learning dan MKCU
http://pbael.mercubuana.ac.id/
### This will give you converted onnx model, which can itself be used
for on-device deployment
OpenVino installation — docs
python3 -m venv openvino_env
source openvino_env/bin/activate
python -m pip install --upgrade pip
pip install openvino-dev==2022.1.0. ### tested on this version currently
OpenVino test installation
mo -h ### This will return the help message for Model Optimizer if
installation finished successfully
Converting Onnx to OpenVino
### To convert, run the following in your terminal (you can use either
onnx model or the folded onnx model)
### input_model - specify your input onnx file
### data_type - use fp16 to convert model to fp16 precision for better
performance
### output_dir - directory where your openvino files will be saved
mo --input_model summer2winter_yosemite_folded.onnx --data_type fp16 --
output_dir "./summer2winter_yosemite_openvino"
2023 Deep Learning
4 Dr.Hadi Santoso, S.Kom., M.Kom
Biro Bahan Ajar E-learning dan MKCU
http://pbael.mercubuana.ac.id/
Once executed, it will create the below 3 files in your directory
summer2winter_yosemite_folded.bin - this contains the model weights ~23
Mb
summer2winter_yosemite_folded.mapping - ~15 Kb
summer2winter_yosemite_folded.xml - this contrains the model structure
~175 Kb
Now that we have both our Pytorch and OpenVino models, let’s perform inference on the
below sample image and compare their runtimes and qualitative outputs
Input Image: Yosemite Summer (Source)
### Pytorch Model Inference
from util.util import tensor2im
from data.base_dataset import get_transform
from PIL import Image
import numpy as np
input_image_filename = 'Yosemite_summer.jpg'
input_image = Image.open(input_image_filename).convert('RGB')
transformation = get_transform(opt, grayscale = False)
transformed_img = transformation(input_image).unsqueeze(0)
if opt.eval:
2023 Deep Learning
5 Dr.Hadi Santoso, S.Kom., M.Kom
Biro Bahan Ajar E-learning dan MKCU
http://pbael.mercubuana.ac.id/
model.eval()
with torch.no_grad():
output_image = model.netG_A(transformed_img)
output_image = tensor2im(output_image) ### final Pytorch result image
### OpenVino Model Inference
from openvino.inference_engine import IECore, IENetwork
from openvino.runtime import Core
ie_obj = IECore()
openvino_network = ie_obj.read_network(model =
'summer2winter_yosemite_folded.xml', weights =
'summer2winter_yosemite_folded.bin')
executable_network = ie_obj.load_network(network = openvino_network,
device='CPU')
openvino_output = executable_network.infer({'inp_img': transformed_img})
arr_key = list(openvino_output.keys())[0] ### get key for putput image
openvino_output_img =
torch.from_numpy(np.asarray(openvino_output[arr_key]))
openvino_output_image = tensor2im(openvino_output_img) ### final
OpenVino result image
Plotting the resultant images for quality comparison
import matplotlib.pyplot as plt
plt.figure(figsize = (30, 30))
plt.subplot(1, 2, 1)
plt.gca().set_title('Pytorch Output', fontsize = 'xx-large')
plt.imshow(output_image)
plt.subplot(1, 2, 2)
plt.gca().set_title('OpenVino Output', fontsize = 'xx-large')
plt.imshow(openvino_output_image)
2023 Deep Learning
6 Dr.Hadi Santoso, S.Kom., M.Kom
Biro Bahan Ajar E-learning dan MKCU
http://pbael.mercubuana.ac.id/
Ouptut Image comparison for Pytorch and Openvino (FP16)
From the above images, we can see that there are almost no discernible differences
between the two output images. Also, on comparing the inference runtimes for
the Pytorch and OpenVino models for 19 runs it is observed that
the summer2winter_yosemite CycleGAN OpenVino model runs 50% faster as compared to
the Pytorch one, which is a significant gain if we are trying to integrate this model into an
application for a real-world use case.
Conclusion
OpenVino is an open-source toolkit introduced by Intel for deep learning model
optimization. It has introduced a novel intermediate graph representation format known as
IR along with an operation set for representing the deep learning network which is typically
2023 Deep Learning
7 Dr.Hadi Santoso, S.Kom., M.Kom
Biro Bahan Ajar E-learning dan MKCU
http://pbael.mercubuana.ac.id/
represented as a directed graph. It uses an XML file and a binary file representing the
graph or IR. To learn more about IR and Opsets you can refer to their official docs here.
OpenVino has definitely made the path to productionalizing a deep learning model much
more efficient. Not only on various hardware devices, we can also deploy these models on
cloud instances for a seamless application experience.
2023 Deep Learning
8 Dr.Hadi Santoso, S.Kom., M.Kom
Biro Bahan Ajar E-learning dan MKCU
http://pbael.mercubuana.ac.id/
Referensi
https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-
networks-the-eli5-way-3bd2b1164a53
https://www.d2l.ai/chapter_convolutional-modern/index.html
https://www.trivusi.web.id/2022/04/algoritma-cnn.html
2023 Deep Learning
9 Dr.Hadi Santoso, S.Kom., M.Kom
Biro Bahan Ajar E-learning dan MKCU
http://pbael.mercubuana.ac.id/
