Automatic Sorting System Using Machine vision

Saurin Sheth1#,Rahul Kher2*, Rushabh Shah3#, Parth Dudhat4*, Pratyush Jani5#

Mechatronics , Electronics2*, Mechatronics3#, Mechatronics4*, Mechatronics5# Department, Sardar Patel University
1#

G H Patel College of Engineering & Technology, Vallabh Vidhyanagar, Gujarat, India

1
sheth_saurin@yahoo.co.in 2rahulkher@gcet.ac.in 3rishi_rushabh87@yahoo.co.in 4pjarkk@rediffmail.com 5pratyushjani711@yahoo.co.in

Abstract: Nowadays various automation techniques are being traditionally occupied by human beings whose degree of failure is
adopted & researched on for increase in productivity, for classically high through distraction, illness and circumstance. Human eyes
better accuracy, eliminating the human errors and for safety. are sensitive to electromagnetic wavelength s ranging from 390 to 770
nanometres (nm). Video cameras can be sensitive to a range of
Machine Vision is one such advancement in automatic
wavelengths much wider than this. Some machine-vision systems function
systems. Machine vision performs the tasks that are at infrared (IR ), ultraviolet (UV), or X-ray wavelengths.
equivalent to human vision. It helps to automate the systems
where there are limitations of human vision like detecting 1.3 Origins of Colour in Machine Vision
various shades of colors or determining high precise
dimensions and thus permitting human employees to serve in  Colour in machine vision began in the mid-1980s with the
more appropriate positions. Now, what happens when the commercial availability of colour cameras and imaging boards.
questions turn to "Is this part of correct color?" or "Which  The first application for which colour machine vision was
widely deployed was food processing, specifically the high-
parts are blue and which red?” So in our system, colour based
speed sorting of fruit where the goal was to detect and remove
identification of the parts will be done and then it will be flawed, blemished and/or inappropriately sized fruit as well as
sorted according to different colours. After recognizing the extraneous vegetable matter.[1]
colour of the object, robotic arm will automatically pick &  In industry applications, automotive fuse block inspection used
place it accordingly. If the colour of the work piece is not RGB color imaging to verify the presence of the correct fuses
found in accordance to the required one then it will be was the first use of colour machine vision.
rejected. The complete sorting system operates on image
processing using the MATLAB application & microcontroller 1.4 APPLICATIONS OF THE SYSTEM
which will control different motors in the system. Machine
vision based on color concept has found its wide application in Specifically, manufacturers use colour vision to solve three primary
the pharmaceuticals industry, agriculture industry and vision applications:
1. Color verification -- Verifying that a certain part's colour matches what
assembly of parts especially in automobile industry. Advances the vision system is programmed to find.
in both machine vision technology and related software tools 2. Color sorting -- Sorting parts based on colour.
have enabled manufacturers to apply color machine vision on 3. Color inspection -- Inspecting coloured parts for defects that grayscale
the factory floor in real time. This technology can solve image processing tools can't detect.
problems formerly restricted to manual inspections & sorting.
2. DESIGN OF THE SYSTEM
Keywords- Machine Vision, MATLAB, Microcontroller,
Sorting, Colour Inspection
2.1 Block diagram
1. INTRODUCTION

1.1 Machine Vision

Machine vision is the application of computer vision to industry and
manufacturing.Two important specifications in any vision system are the
sensitivity and the resolution. The better the resolution, the more confined
the field of vision. Sensitivity and resolution are interdependent. All other
factors held constant, increasing the sensitivity reduces the resolution, and
improving the resolution reduces the sensitivity.One of the most common
applications of Machine Vision is the inspection of manufactured goods
such as semiconductor chips, automobiles, food and pharmaceuticals. Just
as human inspectors working on assembly lines visually inspect parts to
judge the quality of workmanship, so machine vision systems use digital
cameras, smart cameras and image processing software to perform similar
inspections. Machine vision systems are programmed to perform narrowly
defined tasks such as counting objects on a conveyor, reading serial
numbers, and searching for surface defects.
 
2.2 Working of the system
1.2 Why these technology
Working of our system is very simple. Initially when product is on the
Manufacturers favour machine vision systems for visual inspections conveyor belt, Sensor will detect the presence of the particular product &
that require high-speed, high-magnification, 24-hour operation, and/or give signal to the microcontroller. Then microcontroller will send this
repeatability of measurements. Frequently these tasks extend roles
signal to the computer by serial interfacing. Image processing software 2. Input port: for proximity sensor
(Matlab) of the system will send the signal to the camera for capturing the 3. Timers: for generating delays for driving stepper motors
image. Once image is captured, the software will process on the captured 4. PWM: for providing delays for the rotation of servo motors
image and will generate signals according to requirement and which in 5. USART: for exchanging data with computer
turn the signals will be send back to microcontroller. Accordingly the 6. Memory (Internal/External): for storing program
microcontroller will control the conveyer belt & robotic arm. Robotic arm
will pick & place the given component according to the color. If color is 3.1.2.1 Selection of microcontroller
not matched with a given requirement, the product will be rejected. This
cycle will be repeated number of times as per requirement. For our project purpose, we have selected AVR based microcontroller
named ‘Atmega32’ of the Atmel Corp. which has few of the following
special features to fulfill our above stated requirements:
3. COMPONENTS OF THE SYSTEM
 Up to 16 MIPS Throughput at 16 MHz
 32K Bytes of In-System Self-programmable Flash program
3.1 Electrical components memory
 1024 Bytes EEPROM
 2K Byte Internal SRAM
3.1.1 Sensor  Two 8-bit Timer/Counters and One 16-bit Timer/Counter
Sensor is a device that measures or detects a physical condition.  Four PWM Channels
An electronic sensor converts this measurement or detection into  8-channel, 10-bit ADC
equivalent analog or digital electrical signal. Here we will be using  Power-on Reset
proximity sensor. A proximity sensor is a sensor able to detect the  External and Internal Interrupt Sources
presence of nearby objects without any physical contact. A proximity  32 Programmable I/O Lines
sensor often emits an electromagnetic or electrostatic field, or a beam of  Operating voltage of 4.5 - 5.5V
electromagnetic radiation (infrared, for instance), and looks for changes in
the field or return signal. The object being sensed is often referred to as
3.1.3 Camera system
the proximity sensor's target. The maximum distance that this sensor can
detect is defined "nominal range". Some sensors have adjustments of the  It will capture the image of object when signal is given to it by
nominal range or means to report a graduated detection distance. computer.
A. Advantages  We are using simple USB web camera. So Interfacing will
Proximity sensors can have a high reliability and long functional life become easy with computers.
because of the absence of mechanical parts and lack of physical contact  Very cheap
between sensor and the sensed object.

B. Specification of sensor
 Input:10 to 30v
 Type: PNP/(NO)
 Range:10cm
Specifications (INTEX-IT-305WC)
3.1.2 Microcontroller  Image resolution: 160x120,176x144,320x240,352x288
,640x480,800x600
 Frame rate:upto30 frames per second
 Camera Controls: Color saturation, brightness, sharpness is
adjustable
 Image Quality:RGB24
 Exposure: Auto or manual
 Focus Range:4cm to infinity

A microcontroller (also microcontroller unit, MCU or µC) is a small 3.2 Mechanical components
computer on a single integrated circuit consisting of a relatively simple
CPU combined with support functions such as a crystal oscillator, 3.2.1 Robot arm
timers, watchdog timer, serial and analog I/O etc. Program memory is also
often included on chip, as well as a typically small amount of RAM. The robot arm is probably the most mathematically complex robot you
Microcontrollers are designed for small or dedicated applications. Thus, in could ever build. But here we have made very simple arm for picking or
placing the object according to its colour.[10]
contrast to the microprocessors used in personal computers and other
high-performance or general purpose applications, simplicity is Design of robot arm
emphasized.[11]
Some of our basic requirement for the project from microcontroller: A. Degree of freedom
B. Workspace of the robot
1. Output port: C. Torque calculations at different joints
4 port pins for each stepper motor with full step sequence D. Selection of motors
2 port pins for each stepper motor with half step sequence
1 port pin each for servo motor
 W3 - Weight of link ( 0.500 kg)
Wm1 – Weight of motor 1 ( .250kg)
A. Degrees of Freedom (DOF) L - Length between gripper and motor1 ( 20 cm)
The degrees of freedom, or DOF, are a very important term to understand. Distance between M1 & M2 is 5cm.
Each degree of freedom is a joint on the arm, a place where it can bend or Torque about motor1
rotate or translate. You can typically identify the number of degrees of = (W1+W2)*L + W3*(L/2)
freedom by the number of actuators on the robot arm. =(0.150+0.350)*20 + (.500*10)
The robotic arm which we are going to use has 3 DOF. =10+5
=15 kg.cm
B. Robot Workspace
Torque about motor2
The robot workspace (sometimes known as reachable space) is all places = (W1+W2)*(L+5) + W3*(L/2 + 5) + Wm1*5
that the gripper can reach. The workspace is dependent on the DOF
= (.150+0.350)*25 + (.500*15) + (.250*5)
angle/translation limitations, the arm link lengths, the angle at which
something must be picked up at, etc. The workspace is highly dependent = 12.5+7.5+1.25
on the robot configuration. = 21.25 kg.cm
The figure shows the free body diagram of robotic arm which we will be
used for our project. D. Motors
 We are using stepper & servo motors in arm to get good
accuracy, precision, repeatability.
 we are using two servo motors (Motor1 & 2) which have
24kg.cm torque.
 For gripper we are using dc stepper motor because it needs less
torque.

3.2.2 Conveyor belt

Conveyor belt is nothing but the mechanical apparatus consisting of a
continuous moving belt that transports materials or packages from one
place to another. Conveyor belts are often driven by variable speed
electric motors or by other moving parts in a complex system. They are
commonly found in factories, grocery stores, warehouses and public
transportation centers.

C. Torque calculations
Torque calculation is a very important factor for designing the robot arm.
As our system needs 20cm of robot arm (according to size of conveyor
belt & block), we can calculate motor torque from that.
Here is the FBD of given robot arm.

Specifications

 Length : 900mm
 Width :135 mm
 Thickness : 5 mm
1. Permanent magnet DC servo motor
1.1. Model : D2415PL20
1.2. SR No : PDD8013
1.3. Torque : 35 kg-cm
1.4. Current Rating: 2.5 Amps
1.5. Voltage : 24 V
1.6. RPM : 80

3.3Software components

3.3.1 Image processing software

Calculation
Image processing software captures the incoming image & processes it
W1 - Weight of block ( 0.150 kg) according to its color. & after that it gives signal to microcontroller.
W2 - Weight of gripper ( 0.350kg,motor weight is included)
We have used RGB color space. RGB color space uses a rectangular  We have made a program in which when require colour
coordinate system with one coordinate axis assigned to each of three color component comes in the camera, signal (Blue Square) will be
components, red, green, and blue. displayed. [fig1].& robot arm will pick & place it according to
.
its colour.
 If required colour is not there then it will be rejected.[fig 2]
 Here we have used GUI (Graphics user interface)for
convenience of user.

4. INTERFACING
Here we are using Mat lab software for image processing.
MATLAB is a high-performance language for technical computing. It Interfacing is a term used in electronics when different electronic
integrates computation, visualization, and programming in an easy-to-use devices are attached. The term is commonly used when attaching memory
environment where problems and solutions are expressed in familiar chips. Interfacing also often refers to the connection of peripheral devices
mathematical notation. [12] to computers. For Micro-controllers it is useful to the extent that they
Typical uses include communicate with other devices, such as sensors, motors, switches,
 Math and computation keypads, displays, memory and even other micro-controllers. Many
 Algorithm development interface methods have been developed over the years to solve the
 Data acquisition complex problem of balancing circuit design criteria such as features,
 Modelling, simulation, and prototyping cost, size, weight, power consumption, reliability, availability,
 Data analysis, exploration, and visualization manufacturability.
 Scientific and engineering graphics In our project we need to do the following interfacing:
 Application development, including graphical user interface A. Computer to Web camera
building B. Computer to MAX232 IC
In Matlab we are concentrating on Image processing toolbox. Image C. MAX232 IC to Microcontroller
Processing Toolbox is a collection of functions that extend the capability D. MICROCONTROLLER to actuators (stepper motor and servo
of the MATLAB numeric computing environment. motors)
The toolbox supports a wide range of image processing operations, E. Microcontroller to sensor
including
 Spatial image transformations A. Computer to Web camera :
 Morphological operations
 Neighbourhood and block operations Requirement:
 Linear filtering and filter design Webcam is used for capturing the image, and this image is needed to be
 Transforms Image analysis and enhancement sanded in to the image processing software for processing, this is achieved
 Image registration by this interfacing.
How it is done:
 Deblurring
As webcam used is a plug and play device, it can be easily connected to
 Region of interest operations
computer, via USB port, in which the image processing needs to be done.

3.3.1.1Technique to Set the Threshold Values B. Computer to MAX232 IC:

For detecting the colour we have to set the threshold values of particular
colour. According to that colour software will give results. Requirement:
Example: Detection of red colour The data between the computer and the microcontroller is exchanged via
The threshold for red component should be the least value of the red serial communication (RS-232). Usually all the digital ICs work on TTL
component found in the region of the red ball. or CMOS voltage levels which cannot be used to communicate over RS-
The threshold for green component should be the maximum value of the 232 protocol. So a voltage or level converter is needed which can convert
green component found in the region of the red ball. TTL to RS232 and RS232 to TTL voltage levels.
The threshold for blue component should be the maximum value of the The most commonly used RS-232 level converter is MAX232. This IC
green component found in the region of the red ball. includes charge pump which can generate RS232 voltage levels (-10V and
+10V) from 5V power supply. It also includes two receiver and two
Algorithm for red colour transmitters and is capable of full-duplex UART/USART communication.
How it is done:
fR= rgb_image(:,:,1); fG=rgb_image(:,:,2); fB=rgb_image(:,:,3); To communicate over UART or USART, we just need three basic signals
I=((fR>=80) & (fG<=70) & (fB<=70)); (Set the threshold values for red which are namely, RXD (receive), TXD (transmit), GND (common
colour) ground). So to interface MAX232 with microcontroller we just need the
basic signals. A simple schematic diagram of connections between a RS-
3.3.1.2Accept/reject decision 232 port of computer and MAX232 is shown below.

C. MAX232 IC to Microcontroller:
Requirement:
As stated above the need for the data transmission between the computer
and microcontroller, by serial communication over UART or USART,
requires some voltage or level converter IC, which is done via MAX232
IC.
How it is done:
The R1OUT and T1IN pins of MAX232 IC are simply connected to RX
and TX pins of microcontroller.
Fig 1 Fig 2
  Ex: In the white color family, shades might include white, off-
white, champagne, pearl and cream. Human operators often
mismatch these colors, while color vision tools can distinguish
more reliably between them.

7. FUTURE SCOPE & CONCLUSION

Here instead of using a webcam vision camera is to be used. The
machine vision camera obviously will have better resolution, zooming
capacity and clarity and moreover an inbuilt circuitry for external
triggering. So it will save the circuitry used for interfacing the sensor with
computer as the sensor output will be directly made available to the
camera so it will also save the time as computer will not be in the
connection at all.
Here only part sorting is done but in future by some modification we can
also include following things:
 Number of parts are sorted
 Number of parts passed
 Checking label on the part
D. MICROCONTROLLER to actuators (stepper
 Check the crack or dent on the part
motor and  Measuring dimension
servo motors)
So the conclusion from the system is we can automate the colour
1. Microcontroller to stepper motor inspection & sorting of the object with accuracy, good
repeatability & high productivity.

8. 3D MODEL OF THE
SYSTEM

Given circuit is nothing but the interfacing between microcontroller to

stepper motor via ULN2003A.

2. Microcontroller to servo motor

REFERENCES

[1]An integral automation of industrial fruit and vegetable sorting by

machine vision By F Pla, JM Sanchiz, JS Sanchez - Proc. 8th IEEE
International Conference on Emerging …, 2001
[2]Prospects of computer vision automated sorting systems in agricultural
E. Microcontroller to sensor process operations By AO Raji, AO Alamutu - … Engineering
International: the CIGR Journal of …, 2005 - journals.sfu.ca
We have connected the sensor to the port1 which will be used as input [3]www.avrfreaks.net
port. So when sensor senses the object, Signal will go to the [4]www.edaboard.com
microcontroller. [5]www.atmel.com
[6]www.seattlerobotics.org
5. ADVANTAGES [7]www.societyofrobots.com
[8]www.lvr.com
 Fast [9]www.mathworks.com
 Accurate [10]Automation, Production System, And Computer-Integrated
 Good repeatability manufacturing. By Mikell P. Groover
 Reduce labor cost [11]The 8051 Microcontroller Architecture, Programming, &
 Less human interference Applications. By Kenneth J. Ayala
[12]Digital Image Processing Using MATLA. By Gonzalez, Woods, &
Eddins