Thursday, 05-10-2023 Guide STUDENT
Department of Robotics and Asst Prof Dhanya R Adarsh V S
Automation TOC20RA001
TocH institute of Science and
technology
Seminar
Presentation
Paper title: Design and Development of Robot Arm
System for Classification and Sorting Using
Machine Vision
Authors: Vo Duy Cong, Le Duc Hanh, Le Hoai Phuong, Dang Anh Duy
Page
Table of I Introduction 3
Contents
II Objectives 4
III Methodology 5
IV Block Diagram 7
V Design 11
Vi Result and Discussion 27
Vii Advantages and Disadvantages 31
Viii Reference 32
I. Introduction
• The paper focus on design and implementation of a system consisting of two robot arms
for classifying and sorting objects based on shape and size using machine vision.
• It uses a hierarchical control system consisting of one master and two slaves
• An efficient image processing algorithm is proposed to recognize objects and eliminate
noise
• The system is used in industrial application such as material inspection ,object
recognition, assembly and disassembly etc.
3
II. Objectives
• To develop a robotic vision system to
automatically classify and sort objects based on
their shape and size.
• To develop two robot arms for grasping objects
from conveyor.
• To develop a series of image processing
algorithms including contour finding, border
extraction, centroid algorithm.
• To develop a shape threshold algorithm to
detect and classify objects and find their
position.
4
III. Methodology
1. Robot arm kinematics
• Forward kinematics:
process of obtaining position and velocity of end
effector, given the known joint angles.
• Inverse Kinematics:
Given joint parameters, the position and orientation of
the end effector calculating the
variable joint parameters
2.Image processing
• Median filter:
Median filtering is a popular technique used in image
processing to reduce noise in images. It is a nonlinear
filtering method.
• Otsu’s threshold
Technique used for automatic image thresholding.
5
Methodology
3. Morphology Operation:
• It is used for analyzing and manipulating the structure of objects within an
image.
• used in tasks such as noise removal, object segmentation, image
enhancement, and feature extraction.
• The two most common morphological operations are dilation and erosion.
4. 3D localization
• Process of determining the three-dimensional coordinates of objects or
features within a three-dimensional space based on the information
obtained from one or more 2D images.
5. Trajectory Planning
• Trajectory planning creates reference signals for the robot controller so that
the robot can move in the desired trajectory
6
IV. Block Diagrams & Flow Chart
• The objects in the workspace (camera’s
field of view) are taken by a camera
connected with the master controller.
• The master controller acquires and
process the image of objects to classify
them depending on their size and shape
• The first robot receives the joint values
from the master to pick object and place it
on conveyor.
• The second robot grasp and place the
Basic Components of the system object in right position.
7
Block Diagrams & Flow Chart
Steps of operation
8
Block Diagrams & Flow Chart
9
Block diagram of the image processing
Block Diagrams & Flow Chart • The control circuit of the system consist of
one master circuit and two slave circuits
• The master controller is an embedded
computer Raspberry Pi 4.
• Each slave is an Arduino board and is used
as a simple robot controller.
• The Arduino board creates pulses and
sends them to three A4998 drivers to drive
stepper motors.
• Arduino are used to turn ON and OFF the
relays
• Arduino reads the signal from the proximity
Electronic Circuit block diagram sensor to control the conveyor motor by
outputting a signal to the L298 driver
10
V. Design
Link No. Length
L1 153.3 mm
L2 135 mm
L3 160 mm
L4 45 mm
11
The Robot arm schematic
Design
• The Robot has three degree of freedom corresponding to the rotation angles 𝜃1, 𝜃 2, 𝜃3
By trigonometric calculation the position of the end effector can be written in terms of
the joint angles as follows:
𝑥 = 𝑐𝑜𝑠𝜃1 𝑙2𝑐𝑜𝑠𝜃2 − 𝑙3𝑐𝑜𝑠𝜃3 + 𝑙4 (1)
𝑦 = 𝑠𝑖𝑛𝜃1 𝑙2𝑐𝑜𝑠𝜃2 − 𝑙3𝑐𝑜𝑠𝜃3 + 𝑙4 (2)
𝑧 = 𝑙2𝑠𝑖𝑛𝜃2 − 𝑙3𝑠𝑖𝑛𝜃3 + 𝑙4 (3)
The above equations are called forward kinematic equations of the robot manipulator
that describe the relationship between the end-effector coordinates and joint angles.
12
Design
• To find the joint angles for a given set of end-effector coordinates, we need to solve
the inverse kinematic equations
From the forward kinematic equation we can obtain
After Squaring on both sides in equations (1) and (2) then adding them together:
Consider a and b as following
13
Design
After Squaring both sides of a and b and writing in terms of 𝜃
Rearranging equation for b according to unknown angle 𝜃2
14
Design
The angle φ can be determined by using the atan2 function:
Substituting r and φ into
we get,
Finally, the solution of angle θ2 is:
15
Design
Image Processing
• The RGB image of objects taken by the camera is converted to a gray image and
filters noise by a median filter
• The median filter is used to reduce “salt and pepper” noise and smooth away the
edges. This paper uses 9×9 matrices
• The gray image is thresholded by Otsu’s Thresholding to extract the objects from their
background.
16
Otsu’s threshold
Design
Image Processing
• The morphological operators are applied to fill in small holes and eliminate small objects.
Two basic morphological operators, dilation and erosion
A dilation followed by erosion is performed to fill holes in the objects while keeping the object
sizes
An erosion followed by dilation is applied to separate objects connected by a thin bridge of pixels
and delete small noise objects.
17
morphological operators
Design
Image Processing
Finding the contours of objects in the binary image and extract different features of contours, like
area, perimeter, centroid… These features are used to classify objects and localization.
The shape of an object is recognized by computing the compactness:
where c is the compactness, p is the perimeter and A is the area.
The area is equal to the zeroth-order image moment defined by:
where u and v are the row and column index, I(u,v) = 1 in the case of the binary image. 18
Design
Image Processing
• The centroid of the object in the image is given by the relations:
• The value of compactness and area A are used to classify objects.
• In this paper, objects are divided into four categories: small circle, large circle, small square, large
square.
19
Design
3D localization
• After determining the centroid of the objects in the image We can calculate the 3D coordinates of
the object with the constraint that the height of the object is known in advance
• Extrinsic parameters is used to transform the world points to camera coordinates.
• Intrinsic parameters are used to map the camera coordinates into the image plane
where P = [X Y Z] T is the coordinates of one point in the world frame and
and p = [u v 1]T is the image plane, 𝛼 is a scale factor, K is the intrinsic matrix, R is the rotation
matrix, t is the translation vector
20
Design
3D localization
The Intrinsic and Extrinsic parameters are determined by camera calibration
Equation becomes:
21
Design
3D localization
The equation is simplified into:
Rewriting in terms of unknown X and Y coordinates:
22
Design
3D localization
X and Y coordinates are obtained as
The 3D coordinates of the object’s centroid in the world frame are transformed to the robot frame
using :
where [x,y,z] T and [X, Y, Z] T are the coordinates of the object in the
robot frame and the world frame, respectively, R is the rotation
matrix, t is the translation vector.
These coordinates are converted to the joint angles using the 23
inverse kinematics
Design
Trajectory Planning
• Trajectory planning creates reference signals for the robot controller so that the robot can move in
the desired trajectory.
• Trajectory planning can be done either in the joint space or in the Cartesian space
• For pick and place applications in industrial, joint space trajectories are usually used.
• The planning algorithm generates a function q(t) interpolating the given vectors of joint variables
at each joint.
24
Design
Trajectory Planning The velocity graph consists of three phases
namely constant acceleration, constant
velocity, and constant deceleration.
When the velocity profile is a trapezoid, the
total time is:
The trajectory is formed by a linear segment
connected by two parabolic segments:
Velocity Profile
25
Design
Trajectory Planning
When the velocity profile is a triangle, the
total time is:
The function of angle in the term of time t::
Velocity Profile
26
VI. Result and Discussion
• The parts for the Robot arm and conveyor are fabricated by a 3D printer
• The kinematic dimensions of the robotic arm are as follows: L1 = 153.3 mm, L2 =
135 mm, L3 = 160 mm, L4 = 45 mm, d = 25 mm.
• The joints of the robot arm are driven by 5.17:1 planetary gearbox stepper motors.
• The robot arm’s 3 stepper motors combined together uses a maximum of 30W
only, resulting in significant energy savings.
• The vision system uses a Raspberry Pi Camera Module with a Sony IMX219 8-
megapixel sensor.
Robot arm range
Axis Range Max speed
Joint 1 (base) -90° to +90° 320°/𝑠
Joint 2 (rear arm) -30° to +85° 320°/𝑠
27
Joint 3 (forearm) -50° to +60° 320°/𝑠
Result and Discussion
A chessboard is employed to calibrate the parameters of the camera. The calibration uses the
Camera Calibration Toolbox for Matlab® [25].
The intrinsic and extrinsic parameters after calibration are as follows:
By experiment, the thresholds of c and A to classify objects are as follows:
28
Result and Discussion
Example of successful result Example of successful result with noise objects
29
Result and Discussion
The reasons for the faulty recognition may be due to :
• Poor lighting conditions, resulting in the image is not
thresholded properly
• Some objects being very close to each other that cannot
be separated by using the morphological operators.
Example of unsuccessful result The experiment results show that the estimation method
based on vision has an average error of 3.47 mm.
30
VII. Advantages and Disadvantages
Advantages
• The trajectory planning is done using Joint space so it has less
computation, easier to plan trajectories in real-time, and no problem
with singularities
• The camera module has a resolution of 3280 x 2464 pixels but in this
project only 640 x 480 pixels is used to achieve faster processing
speed
• Minimal process time: during transporting and sorting of the object by
the second robot, the processing on the master is also performed
• To sort one object the system needs a total time of to 2.2 to 2.4s
• Using the trapezoidal profile with the acceleration/deceleration phase
helps the robot move smoothly and reduce vibration.
31
Advantages and Disadvantages
Disadvantages
• The system sort objects only based on size and shape.
• The accuracy and performance of the vision system directly affect the
performance and speed of the sorting process
• There are a few cases of failure due to lighting condition or few objects
being very close to each other
• The estimation method based on vision has an average error of 3.47 mm
32
VIII. Reference
1. Vo Duy Cong, Le Duc Hanh, Le Hoai Phuong, Dang Anh Duy.:
Design and Development of Robot Arm System for Classification and
Sorting Using Machine Vision, FME Transactions (2022) vol 50, 181-
192
2. Sangeetha, G.R., Kumar, N., Hari, P.R., Sasikumar, S.: Implementation
of a Stereo Vision based system for visual feedback control of Robotic
Arm for space manipulations, International Conference on Robotics
and Smart Manufacturing, Procedia Computer Science, Vol. 133, pp.
1066–1073, 2018.
3. Wang Z., Li H., and Zhang X.:Construction waste recycling robot for
nails and screws: Computer vision technology and neural network
approach, Automation in Construction, Vol. 97, No. 8, pp. 220-228,
2019.
4. Mitzias, D.A. and Mertzios, B.G.: A neural multiclassifier system for
object recognition in robotic vision applications, Measurement, Vol. 36,
pp. 315–330, 2004.
32
Thursday, 05-10-2023 Guide STUDENT
Department of Robotics and Asst Prof Dhanya R Adarsh V S
Automation TOC20RA001
TocH institute of Science and
technology
Thank you
