2012 Sixth International Conference on Sensing Technology (ICST)

An Automated Machine Vision Based System for

Fruit Sorting and Grading
Chandra Sekhar Nandi
University Institute of Technology.
The University of Burdwan
Burdwan, India
chandrasekharnandi@gmail.com

Bipan Tudu
IEE Department
Jadavpur University
Kolkata, India
bip_123@rediffmail.com

AbstractThe paper presents a computer vision based system for

automatic grading and sorting of agricultural products like
Mango (Mangifera indica L.) based on maturity level. The
application of machine vision based system, aimed to replace
manual based technique for grading and sorting of fruit. The
manual inspection poses problems in maintaining consistency in
grading and uniformity in sorting. To speed up the process as
well as maintain the consistency, uniformity and accuracy, a
prototype computer vision based automatic mango grading and
sorting system was developed. The automated system collect
video image from the CCD camera placed on the top of a
conveyer belt carrying mangoes, then it process the images in
order to collects several relevant features which are sensitive to
the maturity level of the mango. Finally the parameters of the
individual classes are estimated using Gaussian Mixture Model
for automatic grading and sorting.
Keywordsmachine vision; fruit grading and sorting; video
image; maturity prediction; Gaussian mixture model

I. INTRODUCTION
Automated grading and sorting of agricultural products
are getting special interest because of increased demand in
different quality food with relative affordable prices by the
different group of customers belongs to different living
standards. Thus fruit produced in the garden are sorted
according to quality and maturity level and then transported to
different standard markets at different distances based on the
quality and maturity level. Sorting of fruits according to
maturity level is most important in deciding the market it can
be sent on the basis of transportation delay.
In present common scenario, sorting and grading of fruit
according to maturity level are performed manually before
transportation. This manual sorting by visual inspection is
labour intensive, time consuming and suffers from the problem
of inconsistency and inaccuracy in judgement by different
human. Which creates a demand for low cost exponential
reduction in the price of camera and computational facility
adds an opportunity to apply machine vision based system to
assess this problem.
The manual sorting of fruits replaced by machine vision
with the advantages of high accuracy, precision and processing
speed and more over non-contact detection is an inevitable
trend of the development of automatic sorting and grading
systems [1]. The exploration and development of some
fundamental theories and methods of machine vision for pear
quality detection and sorting operations has been accelerate the
application of new techniques to the estimation of agricultural
products quality [2].

978-1-4673-2248-5/12/$31.00 2012 IEEE

Chiranjib Koley
Electrical Engineering Department
National Institute of Engineering
Durgapur, India
chiranjib@ieee.org

Many color vision systems have been developed for

agricultural grading applications. These applications include
direct color mapping system to evaluate the quality of tomatoes
and dates [3], automated inspection of golden delicious apples
using color computer vision [4].
In recent years, machine vision based systems has been
used in many applications requiring visual inspection. As
examples, a color vision system for peach grading [5],
computer vision based date fruit grading system [6], machine
vision for color inspection of potatoes and apples [7], and
sorting of bell
peppers using machine vision [8]. Some
machine vision systems are also designed specifically for
factory automation tasks such as intelligent system for packing
2-D irregular shapes [9], versatile online visual inspections
[10], [11], automated planning and optimization of lumber
production using machine vision and computer tomography
[12], camera image contrast enhancement for surveillance and
inspection tasks [13], patterned texture material inspection
[14], and vision based closed-loop online process control in
manufacturing applications [15].
With this back ground, the proposed technique applies
machine vision based system to predict the maturity level of
mango from its RGB image frame, collected with the help of a
CCD camera. The materials and method are discussed in
Section II. Details preprocessing of image is discussed in
Section III. Different feature extraction methods are discussed
in Section IV. The theory of GMM is discussed in Section V,
and the result and discussion in Section VI. We summarize our
work and conclude this paper in Section VII.
II. MATERIALS AND METHOD
A. Sample Collection
For the experimental works total 600 number of unsorted
mangoes of four varieties locally termed as Kumrapali
(KU),Sori (SO),Langra (LA) and Himsagar (HI) were
collected from three gardens, located at different places of
West Bengal, India. Collection of mangoes were performed in
three batches with an interval of one week in between batches
and in each batch 200 numbers of mango were collected,
having 50 numbers of each variety i.e. KU, SO, LA and HI.
Steps were taken to ensure randomness in mango collection
process from the gardens in each batch. After collection of
mangoes each mango were tagged with some unique number
generated on the basis of variety, name of the origin garden,
batch number and serial number etc. Three independent human
experts work in the relevant field were selected for manual
prediction of maturity.

195

Each mango was used to pass through a conveyer belt

every day until it rotten and was presented to the experts (after
removing tags) for recording of human expert predicted
maturity level. Then the mangoes were stored in a manner as
used during transportation.
B. Experimental Procedure
A schematic diagram of the proposed automated system is
shown in Fig.1, the camera used in the study was a 10
megapixel CCD camera (Olympus E-520) with maximum
frame rate of 30 frames/sec.

Fig.1. Proposed model of vision based automated fruit grading and sorting
system.

The camera was interfaced with a computer through USB

port. The proposed algorithm was implemented in Lab VIEW
Real Time Environment for automatic sorting. Light
intensity inside the closed image acquisition chamber was
controlled manually and was kept at 120 lux, measured with
the help of lux meter (Instek-GLS-301).
The automated fruit sorting system consists of a motor
driven conveyer belt to carry the fruits serially. The fruit placer
places one fruit at a time on the belt and the belt carries it to the
imaging chamber where the video image of fruit is captured by
the computer through CCD camera. The proposed algorithm
runs into the computer automatically to classify the fruit on the
basis of maturity level and then give a direction to the sorting
unit to place the mango in appropriate bin.
The sorting unit consists of four solenoid valves driven by
respective drive units, which are controlled by the computer.
The time delay in between image capturing of a fruit and the
triggering the solenoid valve is estimated by computer on the
basis of conveyer belt speed.
The color of the conveyer belt was chosen blue for two
reasons. First, blue does not occur naturally in mangoes.
Second, blue is one of the three channels in the RGB color
space, making it easier to separate the background from the
image of mango.
The image capturing chamber is a wooden box and the
ceiling of the chamber is quoted with reflective material to
reduce the shading effect. A CCD camera is mounted in the top
center of the chamber. One fluorescent lamp is mounted at the
top of the chamber. The camera is mounted right side the light
source for the best imaging.
The light intensity inside the imaging chamber is measured
and consequently controlled by a separate light intensity
controller, which keep the light intensity constant irrespective
of power supply voltage and any variation of the filament
characteristics and changes in ambient environment.
However, even with the lamp current being constant, the

light output of the lamp still varies resulting from the lamp
ageing, filament or electrode erosion, gas adsorption or
desorption, and ambient temperature. These effect cause
changes in the RGB values of the images. The light intensity
controller corrects for the lamp output changes, maintaining a
constant short and long-term output from the lamp. The light
output regulating unit is made up of a light sensing head and a
controller. The (silicon based) light sensor. is also mounted
near to the sample fruit inside the chamber monitors part of the
light source output; the controller constantly compares the
recorded signal to the pre-set level and changes the power
supply output to keep the measured signal at the set level i.e.
120 lux.
The still frames are extracted from the video image at the
rate of 30 frames/sec. In our systems the motor speed and
distance between the two consecutive mangoes were taken as
input. If the motor speed and distance between two mangoes
are known then we can find a frame that will be the best still
image of full mango within the imaging chamber. In our
system the speed of the conveyer belt was 2ft/sec, length of the
imaging chamber was 1ft and the distance between two
consecutive mangoes on the conveyer belt was 1ft. So 7200
samples/hour can be sorted by our system. This rate can be
increased by increasing the speed of the conveyer belt and
reducing the distance between two consecutive samples in the
conveyer belt. The size of the still frame was 480X640 pixels.
C. Color Calibration of CCD camera
Color calibration of CCD cameras is essential for color
inspection systems based upon machine vision to provide
accurate and consistent color measurements. Here we have
calibrated the camera using color standard.
An image matrix for four varieties of mango having
different maturity levels are shown in Fig. 2.

Fig. 2. Images of four varieties mango having different maturity level. 1st
row: KU, 2nd row: SO, 3rd row LA and 4th row HI. Images are
taken with an interval of 2 days, shown in (a) raw (b) semimatured
(c) matured (d) over-matured.

III. PRE-PROCESSING OF IMAGES

The performance of the grading system depends on the
quality of the images captured from the video camera, since

196

various measures/features calculated from the images of the

mangoes will be used for grading according to maturity level.
Video signal collected from camera found to be contaminated
with motion artifact and noise, thus proper extraction of images
from video frames and then filtering is essential. On the other
hand mangoes moving through the conveyer belt, can be at any
position along with the background thus in order to reduce
computation, removal of background by detecting the edges of
the image and alignment of the mango images in axial position
is necessary. The present section discuss about all these
preprocessing issues, in brief.
A. Still Frame Extraction from Video Image
At first the video streams acquired by the CCD camera are
separated into sequence of images and then to remove motion
related artifacts block motion compensation was employed
[16]. Motion compensated four frames can be observed from
Fig. 3.

the run-length code of the image is obtained. Then, a 33 pixel

box is adopted to detect the relationship between the objective
pixel and its eight-connected surrounding pixels. All the runs
are categorized into 5 classes, and then each runs serial
number along with the corresponding class label was recorded
in another table named as COD. Then, the algorithm
searches the ABS-table and COD-table sequentially. Based on
the coordinates and class of each run recorded on the table, the
starting pixel of the contour is recognized and the contour is
successfully followed. And the chain code is generated while
following the contour. An image of mango along with the
traced boundary is shown in Fig. 4.
D. Alignment of the Mango Image
After getting the contour of the mango, search to find the
longitudinal axis of the mango is obtained. Let
C (xi , yi ), i = 1,2,....., N ( N is the total number of points in the
contour) represent the obtained contour of a mango, then two
end points x1l , y1l and x2l , y2l are the points for which

l=
Fig.3. Extracted still frames from the video image with an interval of 5
frames, (a) frame no.10, sample entering into the imaging chamber (b) frame
no.15, sample near the middle of the imaging chamber (c) frame no. 20,
sample crossing the middle position of imaging chamber (d) frame no. 25,
sample going out from imaging chamber.

( ) (
(x x )+ (y y ) is
l
i

l
j

l
i

C. Edge Detection and Boundary Tracing

Experimentally it was observed that for all the set of the
RGB mango images the G value is always greater than the B
value for that reason the color of the background was kept blue
with R, G and B value of close to 0, 0 and 255 respectively, as
much as possible. So with the help of simple comparison the
back ground was eliminated, and the image was converted to
binary image (BW). Small patches containing number of pixel
less than 800 were removed. This cutoff number was
determined experimentally, by studying all the images of the
mango.
In order to find the boundary or the contour of the mango,
a graph contour tracking method based on chain-code was
adopted. This algorithm found to work reasonable fast, as the
boundary of the mango is not so complex.
The details of algorithm can be found in [18] here a brief
description is given. The algorithm first detects every run at
each row and records every single runs serial number and the
corresponding start-pixel coordinates and end-pixel coordinates
are stored in a table named as ABS. Through this method,

l
j

maximum,

where

i, j = 1,2,.....N .After getting the coordinates of the two

boundary points along the longitudinal axis the image was
rotated
by
an
angle
determined
by
l
l

y
y

= tan 1 2 1 l
, this rotation will align the
x2 x1l

mango vertically but may not able to place the apex region at
the top, it may be at the bottom position. To fix the problem
another rotation of 180o was made if the apex region is in
bottom position. The detection of the apex or the stalk was
made on the basis of geometrical properties of the varieties of
the mango under test. For all the varieties of the mango the
width of the apex is always higher than the stalk, this properties
was utilized to place the apex region at the top of the image.
The center point of the apex/stalk region is the point lies on the
longitudinal axis at a distance of 0.15 lmax from the two end
points of the longitudinal axis. This relation was determined
experimentally, and found to true for all the varieties of the
mango under test.

B. Filtering of Mango Image

Though the images were taken in controlled environment
under fixed illumination of light of 120 lux with the help of
tungsten filament lamp, but there were some noises in the
picture. To remove these noises a simple median filter found to
provide reasonable good performance but it is computationally
intensive. For that pseudo-median filter [17] was used in the
work, as it is computationally simpler and possesses many of
the properties of the median filter. This filtering process often
helped to obtain smooth continuous boundary of the mango.

(a)

(b)

(c)

Fig. 4. Filtered images of Mango, (a) raw image of a mango, (b) raw mango
image along with the obtained contour, (c) binary image of the same mango
after removing the small patches, also shows the different positions and their
name as used in the work.

IV. EXTRACTION OF FEATURES

In order to predict the maturity level with the help of
computer, some suitable measures collected from the images of
the mangoes need to be investigated, which are most correlated
with the maturity level. This section discuss about various
features, selection of the features are mainly based on the

197

experienced gain by the authors, while discussing this issue

with the experts involves in manual grading process.
A. Average R, G and B value
This represent the average R, G and B value of the entire
mango and was calculated from the following equation:
Ak = R ,G ,B =

1
rc

(I

BW )

i =1 j =1

where, BW is the binary image acting as a mask set the

region outside the contour of the mango to 0, and I k is the
captured RGB image, r and c represent the total number of
rows and columns of the image.
B. Gradient of R, G and B value along the longitudinal axis
Due to the fact that the mango start ripe from the apex
region, so the slope of the R, G and B varies from apex region
to stalk region (shown in Fig.5), and this variation found to be
different under different maturity level of the mango. The slope
(I) of the R, G and B were determined by the following
equation. First by taking a slice image along the longitudinal
axis, the width of the sliced image is the 5% of the width of the
mango at the middle position of the longitudinal axis and
length in between 5% below and above of the two end points of
the longitudinal axis.

S k = R ,G ,B = slope
s k (i, j )

i = 1

where p, is the width of the slice. The slope was determined

by searching the best fit straight line by least mean square
sense.

s = Apex, Equator and Stalk .

D. Derived Features
From these main features other derived features were
calculated, these are as follows:
Differences of average R, G and B value of the entire
mango i.e. ( AR AG ), ( AG AB ) and ( AR AB )
Differences of corresponding average R, G and B value
for
the apex, equator and
stalk region, i.e.
( AapexR AequatorR ), ( AequatorR Astalk R ) and

( AapexR Astalk R )

similarly for G and B also.

The variation of the three measures i.e. average R of entire
mango, average R of apex region and difference between
average R to G value for the three mangoes (two from KU
variety and one from HI variety) w.r.t different maturity level
are shown in Fig. 6 (a), (b) and (c). From the Fig. 6 (a), (b) and
(c), it can be observed that these features are correlated with
the maturity level, on the other hand the nature of variation of
these measures for different variety of mangoes are different.
Prediction of maturity level of mango using Support Vector
Machine based Regression analysis can be found in [19]. The
variation of average R with respect to average G for the four
different maturity levels (i.e. M1, M2, M3 and M4 of KU) is
shown in Fig. 6. (d). From this figure it can be observed that
these two measures are not sufficient to classify the mangoes
into four different classes accurately, due to overlapping of the
features, which is due to variation of color texture of different
samples.
In the present work parameters of the individual classes
are estimated using Gaussian Mixture Model. A brief theory of
GMM is presented in next section, the details and the methods
adopted in the present work for estimation of individual classes
can be found in [20].
V. GAUSSIAN MIXTURE MODEL
A Gaussian mixture density is a weighted sum of mixture
component densities. The Gaussian mixture density can be
described as,

( ) p (x )
M

p x | =

Fig. 5. Variation of R, G and B value along the longitudinal axis.

C. Average R, G and B value of the Apex, Equator and Stalk

region
For collecting the average R, G and B value for these three
regions, slice images along the horizontal axis were extracted
from the RGB image of the mango, the width of the each slice
(along the longitudinal axis) is 0.05 lmax and length is in
between the end points of boundary along the horizontal axis
cutting the center point of each region as shown in Fig. 4 (c).
The average value were calculated according to
Ask =

1
rc

where, M is the no of mixture components and

k , k = 1,......., M , are the mixture weights, subject to k > 0

()

and

= 1, pk x are the component densities and x be a

k =1

d-dimensional feature vector, x d . With d 1 mean vector

k and d d covariance matrix S k , each component density is
a d-variate Gaussian function given by,

Is

k =1

()

pk x N ( k , S k )

i =1 j =1

where Isk , is the sliced RGB image, k = R, G and B and

198

(2 )

d
2

| Sk

1
|2

1
exp x k
2

) S (x )
T

1
k

Fig. 6. Variation with maturity level (a) Average R of entire mango (b) Average R in apex region (c) Difference of average R to G. (d) Variation of average R to
average G with four different maturity level(i.e. M1,M2,M3 and M4 of KU). PDF distribution with (e) average value of total G and B (f) different batches and
different gardens. (g) Box-whiskers plots for most correlated features.

199

GMM

Experts

GMM

M4

Experts

M3
GMM

KU
SO
LA
HI

M2
Experts

M1
VARIETY
(Local
Name)

GMM

The Probability Density Function (PDF) estimated using

only two features i.e. Average values of total G and B values is
shown in Fig. 6 (e). When the same GMM was used to find the
PDF of raw mangoes came from different gardens over three
batches, it was observed that there are strong correlations of the
mangoes in a batch originated from a specific garden, but PDF
distribution of different batches and different gardens are often
found to be different. This fact can be observed from the Fig. 6.
(f), where we can see that the several GMM components has
been formed for the mangoes originated from different gardens
and in different batches.
The summary statistics for some of the most correlated
features is represented by boxwhiskers plots, shown in Fig. 6.
(g). The box corresponds to the inter-quartile range where top
and bottom bound indicate 25th and 75th percentiles of the
samples respectively, the line inside the box represents median,
and the whiskers extend to the minimum and maximum values.

The details of the boxwhiskers plot can be found in [21].

After estimation, the evaluation of the classification
performance was performed on a test data set, in which the
objective was to find the class model which has the maximum
a posteriori probability for a given observation sequence. The
classification accuracy obtained using GMM and the average
classification accuracy by the three experts for the four
varieties of mango is presented in TABLE I.

Experts

VI. RESULT AND DISCUSSION

93.5
92.2
92.1
92.1

92.2
91.7
91.5
91.4

93.7
92.7
92.6
92.2

88.9
88.2
87.6
87.4

93.1
92.6
92.2
91.7

89.4
89.3
88.2
88.5

93.3
91.8
91.5
91.3

91.3
90.5
90.3
90.3

TABLE I. PERFORMANCE ANALYSIS

REFERENCES
From the obtained results as summarized in TABLE I, it can
be observed that the classification performance for the
proposed vision based automatic technique as good as the
manual expert based technique. Since the accuracy is
dependent on the image, which is further affected by the
ambient light intensity, thus controlling the light intensity was
performed. The variance in the probability density function of
the features indicates the variation of color pattern of mangoes
for a particular maturity level. Misclassification may occur
when different maturity level mangoes having similar color
pattern, but it was observed that extraction of multiple features
particularly gradient based features helped to correctly identify
those mangoes, as some raw mangoes having color pattern of
matured mango particularly in the apex region, but those
mangoes shows high gradient value of R along the
longitudinal axis. In some cases, the automatic technique for
extraction of features failed to collect suitable features, when
the surfaces of the mangoes were highly contaminated with
scratches and black color patches.

[1]

[2]

[3]
[4]

[5]
[6]
[7]
[8]
[9]

VII. CONCLUSIONS
The present work is an application of machine vision based
technique for automatic grading and sorting of fruits like
mango according to the maturity level. Different image
processing techniques were evaluated, to extract different
features from the images of mango. The proposed work also
aimed to find the variations of different features with maturity
level of mangoes. It also shows the application of Gaussian
mixture model to estimate the parameters of individual classes
to predict the maturity level. This technique found to be low
cost effective and moreover intelligent. The speed of sorting
system is limited by the conveyer belt speed and the gap
maintained in between two mangoes rather than response time
of the computerized vision based system, which is on the order
of ~50ms.
Test has been conducted only for the four varieties of
mango, but can be extended for other fruits where there are
reasonable changes in skin color texture occur with maturity.
The variations of classification performances with the variation
of other factors like changes in ambient light, camera
resolution, and distance of the camera were not studied. The
study shows that, machine vision based system performance, is
closer to the manual experts, where experts judge the mangoes
maturity level not only by the skin color but also with firmness
and smell.

[10]
[11]
[12]

[13]

[14]
[15]
[16]

[17]
[18]
[19]
[20]

[21]

200

Jarimopas B, and Jaisin N. An experimental Machine Vision system for

sorting sweet Tamarind, Journal of Food Engineering, vol. 89(3), 2008,
pp. 291 297.
Y.Zhao, D. Wang, and D. Qian Machine Vision based image analysis
for the estimation of Pear external quality, Second International
Conference on Intelligent Computation Technology and Automation,
2009, pp 629 632.
D. J. Lee, J. K .Archibald, and Guangming Xiong. Rapid color grading
for fruit quality evaluation using direct color mapping, IEEE Trans.
Autom. Sci. Eng., vol. 8(2), 2011, pp. 292 302.
Z. Varghese, C. T. Morrow, P. H. Heinemann, H. J. Sommer, Y. Tao,
and R. W. Crassweller, Automated inspection of golden delicious
apples using color computer vision, American Society of Agricultural
Engineers, 1991(7002), 16.
B. K. Miller, and M. J. Delwiche, A color vision
system for
Peach grading, Transactions of the ASAE, vol. 32(4), 1989, pp. 484
1490.
A. Janobi, Color line scan system for grading date fruits, ASAE
Annual International Meeting, Orlando, Florida, USA, 12-16 July, 1998.
Y. Tao, P. H. Heinemann, Z. Varghese, C. T. Morrow, and H. J.
Sommer, Machine vision for color inspection of potatoes and apples,
Transactions of the ASAE, vol. 38(5), 1995, pp. 1555-1561.
S. A. Shearer, and F. A. Payne, Color and defect sorting of bell
peppers using machine vision, Transactions of the ASAE, vol. 33(6),
1990, pp. 2045 2050.
A. Bouganis, and M. Shanahan, A vision-based intelligent system for
packing 2-D irregular shapes, IEEE Trans. Autom. Sci. Eng., vol. 4(3),
2007, pp. 382 394.
H. C. Garcia, and J. R. Villalobos, Automated refinement of automated
visual inspection algorithms, IEEE Trans. Autom Sci. Eng., vol. 6(3),
2009, pp. 514 524.
H. C. Garcia, J. R. Villalobos, and G. C. Runger, An automated feature
selection method for visual inspection systems, IEEE Trans. Autom.
Sci. Eng., vol. 3(4), 2006, pp. 394 406.
S. M. Bhandarkar, X. Luo, R. F. Daniels, and E. W. Tollner,
Automated planning and optimization of lumber production using
machine vision and computed tomography, IEEE Trans. Autom. Sci.
Eng., vol. 5(4), 2008, pp. 677695.
N. M. Kwok, Q. P. Ha, D. Liu, and G. Fang, Contrast enhancement and
intensity preservation for gray-level images using multiobjective
particle swarm optimization, IEEE Trans. Autom. Sci. Eng., vol. 6(1),
2009, pp. 145 155.
H. Y. T. Ngan, and G. K. H. Pang, Regularity analysis for patterned
texture inspection, IEEE Trans. Autom. Sci. Eng., vol. 6(1), 2009,
pp.131 144.
Y. Cheng, and M. A. Jafari, Vision-based online process control in
manufacturing applications, IEEE Trans. Autom. Sci. Eng., vol. 5(1),
2008, pp. 140 153.
K. Hilman, H. W. Park, and Y. Kim, Using motion- compensated
frame-rate conversion for the correction of 3 : 2 pulldown artifacts in
video sequences, IEEE Trans. On Circuits and Systems for Video
Technology, vol. 10(6), 2000, pp. 869 877.
A. Rosenfeld and A.C. Kak, Digital Image Processing, New York:
Academic Press, cap 11, 1982
S. D. Kim, J. H. Lee, and J. K. Kim, A new chain-coding algorithm for
binary images using run-length codes [J]. CVGIP, vol. 41, 1988, pp.
114 128.
C. S. Nandi, B. Tudu, and C. Koley Support Vector Machine based
maturity prediction, WASET, International Conference ICCESSE2012, 2012, pp..
S. Biswas, C. Koley, B. Chatterjee, and S. Chakravorty A methodology
for identification and localization of partial discharge sources using
optical sensors, IEEE Trans. on dielectric and electrical insulations,
vol.19(1), 2012, pp.
K. Fukunaga, Introduction to Statistical Pattern Recognition, Academic
Press, 1990.