otics

Rob &

Advances in Robotics
n
Bayati and Fotouhi, Adv Robot Autom 2018, 7:1

Au
Adv ces i

tomation
DOI: 10.4172/2168-9695.1000186
an

ISSN: 2168-9695
& Automation
Research Article Open Access

A Mobile Robotic Platform for Crop Monitoring

Mostafa Bayati and Reza Fotouhi*
Department of Mechanical Engineering, University of Saskatchewan Saskatoon, Canada

Abstract
Development, implementation and performance verification of a field-based high-throughput plant phenotyping
mobile platform for monitoring Canola plants, including both data acquisition/visualization software and measurement
system, was the main contribution of this research. Plant breeders need an efficient tool to monitor a number of plant
traits to achieve a higher yield. Currently, manual measurement to gather the required information is customary.
Manual measurement has many limitations especially to study a large-scale field. In this research, a high-throughput
plant phenotyping platform was developed. A long boom was attached to a farm vehicle to carry different sensors,
cameras and other measurement equipment. A program was developed to read sensors signals and to geo-tag data
using GPS for future retrieval. Three programs were developed for image acquisition via webcam and still cameras
and a central program for data processing and data visualization. The efficiency of different system architecture
including different data transmission networks was examined by conducting several laboratory and field tests
on a Canola nursery in spring-summer of 2017. The performance of the phenotyping platform was validated by
various measures such as conducting manual measurements and comparing the results with the values given
by the platform. The main contributions of this research for plant phenotyping research are automation of image
acquisition, enhancement of the data acquisition cycle to minimize data geo-referencing error, development of a
modular program for data visualization, and faster data collection in a high-throughput fashion (almost 125 times
faster). Works are underway to replace the vehicle with an autonomous ground vehicle (AGV).

Keywords: Mobile platform; Sensors; Automated crop monitoring crop. In particular, it is desirable to find the effects of heat, drought
stress, soil properties, and crop disease, on the growth dynamics, yield,
Introduction disease resistance, and quality of the crop [10]. Imaging is one the most
The heart of plant biology is to understand the link between a common tools used to monitor the response of crops to the stresses
genetic difference and expressed phenotypes. Genotyping now is introduced. However, HTP can be a tedious, time consuming, and
accomplished with low-cost, high-throughput analyses. However, labor intensive job for plant breeders [2]. Measuring a lot of plots in a
phenotyping is labor intensive; it has become a restrictive factor in time-sensitive manner on a large number of genotypes is impossible for
plant biology studies and crop improvement research. Platforms for many traits (transpiration, photosynthesis, heat stress, drought stress).
high-throughput phenotyping, mainly under field settings, are needed HTP via a farm vehicle or automated platform is the way around this
to match the genomic information [1-3]. bottle-neck.

There is growing interest in adapting agricultural machinery and This research presents development and analysis of a prototype of
electronic sensors for field-based high-throughput phenotyping [4]. a mobile field-based phenotyping (FBP) platform; we used an existing
Potential applications include monitoring of the crop response to farm vehicle (a swather). This farm vehicle is narrow enough (about
soil, management variability, and correcting crop management with 8 feet wide) to travel down the pathways between ranges of canola
variable inputs to improve yield and economic efficiency, i.e., precision plots without damaging any plant, while collecting data. A frame was
agriculture [5]. designed to carry the required imaging sensors on both sides of this
vehicle. Furthermore, to measure the height of the plant canopies,
Crop breeding and associated genetic research are based on ultrasound sensors were used. In order to measure the fullness of crop
phenotyping a large number of experimental lines. To date, this during the growing period, vision seems to be an appropriate method.
assessment has been largely labor-intensive [6]. Infrared thermometers were used to capture canopy temperature. Two
RGB cameras were installed on the boom of the tractor and a novel
Recent developments in automation, robotics, accurate
image acquisition program was developed for capturing images of each
environmental control and remote sensing facilities will likely offer
plot autonomously. Additional image processing of these data may be
opportunities for more precise field phenotyping of crop plants
a good indication of the health or stress of crops. The images allow us
through high-throughput plant phenotyping platforms (HTP). To
to analyze the visible differences between different plots. Also, Crop
date, there are no commercial HTP vehicles (manual or autonomous
ground robots) available which can do phenotyping. The research in
this area is still in its infancy. At a recent world symposium on plant *Corresponding author: Reza Fotouhi, Department of Mechanical Engineering,
phenotyping, the most recent development in this area was presented University of Saskatchewan Saskatoon, Canada, Tel: 1-306-966-5453; E-mail:
[7-9]. A French team presented their prototype of Phenomobile- LV reza.fotouhi@usask.ca
which is a robot for field phenotyping with a few sensors installed for Received June 27, 2018; Accepted July 12, 2018; Published July 23, 2018
wheat. An American team presented Enviratron [8]. A Norwegian Citation: Bayati M, Fotouhi R (2018) A Mobile Robotic Platform for Crop Monitoring.
team exploring use of robot and drone for phenotyping [9]. Adv Robot Autom 7: 186. doi: 10.4172/2168-9695.1000186

Phenotyping identifies the most desirable and adaptive plant/ Copyright: © 2018 Bayati M, et al. This is an open-access article distributed under
the terms of the Creative Commons Attribution License, which permits unrestricted
crop architecture by means of monitoring the response of the genetic use, distribution, and reproduction in any medium, provided the original author and
behavior of the crop to the environmental changes imposed on the source are credited.

Adv Robot Autom, an open access journal Volume 7 • Issue 1 • 1000186

ISSN: 2168-9695
Citation: Bayati M, Fotouhi R (2018) A Mobile Robotic Platform for Crop Monitoring. Adv Robot Autom 7: 186. doi: 10.4172/2168-9695.1000186

Page 2 of 7

Circle sensors were used to capture NDVI information [11]. Existing Figure 3 illustrates the developed platform in action, and different
RTK-GPS (real-time kinematic global positioning systems) on the farm components are marked on the mentioned image.
vehicle was exploited to geo-reference all collected data and each plot
in the field. With the utilized GPS system, there will be up to one- inch Electrical and measurement system
accuracy on the location of tractor and crops. The current system takes advantage of four ultrasonic sensors, to
measure the height of the plant canopies (two sensors on each side of the
Materials and Methods boom). In fact, it was found that ultrasonic sensors could be efficiently
The developed HTP platform consists of three different parts: A) used to measure the height of different objects including a plant with
mechanical structure, B) electrical and measurement system and C) an acceptable level of measurement error. Table 1 illustrates the result
software components. In next sections, each part will be introduced of height measurement of four different plants in the laboratory
and discussed in particulars. configuration. As can be seen, the maximum measurement error was
2.5% which seems acceptable for a non-contact height measurement
Mechanical structure system [12].
Overall, the proposed HTP platform has two main mechanical
Two Crop Circle sensors were utilized to measure canopy NDVI
components.
(Normalized Difference Vegetation Index), two infrared thermometers
The carrying vehicle: A high clearance swather was used to carry to capture canopy temperature, and two RGB cameras to capture
a mechanical boom and scanning equipment (Figure 1). The utilized images of canopies (one on each side of the boom). To communicate
farm vehicle can be seen in Figure 2. The utilized farm vehicle was and capture sensors data, a Campbell Scientific data logger was used.
equipped with a highly accurate RTK-GPS and was capable of being Also, a laptop was used to communicate with two webcams to capture
driven autonomously. As can be seen in Figure 1, the HTP platform RGB images. The data acquisition task was assigned to a dedicated unit
can scan two plots at the same time (from the left and right); this (data logger) and image acquisition task was carried out by a laptop
feature significantly improves the data collection speed.
to be able to not only capture the required phenotypic data (which
Mechanical boom: To carry sensors and scanning equipment, two is height, temperature, and NDVI) but also to concurrently capture
different mechanical booms were developed to be attached to the rear RGB images of plant canopies during one trip through the field. This
of swather as can be seen in Figure 2a. feature significantly improved the phenotyping process in terms of the
As can be seen in Figure 2b, the 2nd-generation boom is foldable so required time to conduct a field trial to acquire a high volume of data
it can be transported in highways conveniently in comparison with the in one traverse.
1st-generation design which was a solid and long mechanical frame. In addition, to attach the time and geospatial information tags to
each data point and captured image, a GPS system was required to be
accessed by both utilized data logger and laptop. Longitude, latitude,
and UTC were captured at each sampling point and were tagged to
data and images. As can be seen in Figure 3, GPS data was imported
into both laptop and data logger to geo-tag both captured images and
also collected data.

Figure 1: The traveling path of the platform through the field.

Figure 3: The proposed htp platform in action.

hu (Ultrasonic hm (Manual Diff=hm-hu (cm) % diff

Height in cm) height in cm) Diff

= (hm + hu ) / 2
79.0 79.0 0.0 0.0%
70.0 71.0 1.0 1.4%
63.5 65.0 1.5 2.3%
a) First-generation in action b) Second-generation in reality
50.0 51.0 1.0 2.0%
Figure 2: The developed boom system.
Table 1: Plant height measurement in lab.

Adv Robot Autom, an open access journal Volume 7 • Issue 1 • 1000186

ISSN: 2168-9695
Citation: Bayati M, Fotouhi R (2018) A Mobile Robotic Platform for Crop Monitoring. Adv Robot Autom 7: 186. doi: 10.4172/2168-9695.1000186

Page 3 of 7

Software components stored in the laptop temporary memory (RAM). The user can click on
save to save the captured images. It was found that saving an image is a
The software part of the proposed HTP platform consists of three time- consuming task, so the program was developed so that it doesn’t
programs. The first program was developed for image acquisition to save the images in real-time to avoid delay issues in geo-referencing. By
send trigger commands to cameras to capture pictures when needed, utilizing this technique, the positioning error was significantly reduced.
to read GPS data string for geo- referencing purposes, and to save
captured images by a predefined filename format [13]. The second Data acquisition program; Data acquisition program, which is
program was developed for data acquisition to read sensors values and being executed on datalogger, was developed for reading four differential
GPS data by the data logger and write them into an Excel file. The third analog input channels to capture four signals coming from Ultrasonic
program was developed for data visualization and post- processing. In sensors, two single-ended analog inputs to read IR thermometers
the next three sections, the details of each program are discussed. data, two serial RS232 ports to read NDVI sensors data, and also a
third RS232 port for reading GPS NMEA string for georeferencing;
Image acquisition program: The ability of autonomously all recorded data for convenient retrieval for visualization and further
capturing images of plant canopies during the data collection was one analysis purposes. Figure 5 illustrates the structure of the resulting
of the most important aspects of the proposed HTP platform. In fact, Excel file which stores all collected data in different rows. As can be
automatic image acquisition feature has not been reported anywhere seen, the second column stores the time of data collection, 3rd and 4th
in the related literature on field-based HTP platforms. As can be seen columns store the temperature information, 5th and 6th columns store
in Figure 4, an interactive program was developed in MATLAB GUI to the NDVI information, the 7th column holds the GPS string and the
control the operation of the image acquisition program which includes last four columns store the height information. The length of this Excel
starting image capturing cycle, stopping at the end of data collection file depends on the scale of the field and the traveling speed of HTP
and finally saving images at the end of data collection. As can be seen, platform.
after user started the program by pressing the start button, the GPS
string will be acquired, and then using image acquisition and image It was found that any further processing of data including GPS
processing toolboxes, the program sends control commands to the string should be strictly escaped to avoid delay issues in georeferencing
cameras to capture pictures and this operation is repeated every 0.5 of collected data. In fact, because of the limited processing speed of
seconds (this time can be adjusted by user as can be seen in Figure dataloggers, any further processing should be postponed after data
4). After the entire plots in a field are scanned, the user stops image collection in post-processing and visualization programs, and it is
acquisition process. At this time, captured images (which can be up to adequate to store raw data in real-time (Figure 6).
5000 images, depending on the field size), as well as GPS data, are being
Visualization and data analysis program: Figure 7 illustrates
different sections of the then developed program for data visualization
and further post-processing of the collected data as needed. As can be
seen in this figure, the program consists of the main display area to
show the map of the field and the graphical objects as the representative
of available data and also several control buttons and display area on
the right side of the main window. This program was developed to help
breeders to conveniently extract the required information out of the
raw database. To do so, first, the user needs to import the phenotypic
database into an Excel file, and then the folder which contains all
captured images. Figure 6 illustrates the display area of the developed
visualization program after all data and images are imported into the
program.
In Figure 6, a green rectangle on the background layer represents
the location of a plot in a field with their corresponding numbers on
Figure 4: GUI for image acquisition program.
the lower left side, a blue circle represents a height data point, the red

Figure 5: The structure of excel database to store phenotypic data.

Adv Robot Autom, an open access journal Volume 7 • Issue 1 • 1000186

ISSN: 2168-9695
Citation: Bayati M, Fotouhi R (2018) A Mobile Robotic Platform for Crop Monitoring. Adv Robot Autom 7: 186. doi: 10.4172/2168-9695.1000186

Page 4 of 7

Figure 6: The visualization program after importing collected data and images.

Figure 7: The GUI for the developed data visualization and post-processing.

circles illustrate the temperature data points and finally, the triangles can employ the same approach to extract more information out of
represent an available image within a plot. For example, we can see that the resulting phenotypic databases for more analysis. In next sections,
there are seven available images for plot #3.1 details of each experiment are discussed.
After all data is imported as graphical objects, the user can simply Verifying the validity of the collected data
click on different objects to monitor the captured value for a specific
plot. For example, Figure 8 illustrates how an available height data The first concern about the performance of the proposed HTP
point near the left border of plot #1.2 is selected by the user and the platform was the validity of the collected data. In fact, it had to be
result can be seen in the gray display area on the lower right side of verified that collected data points are associated with their actual plots
the program. As can be seen in this Figure 8, the latitude, longitude in the field. For example, if the visualization program illustrates five
and the captured value for the mentioned data point can be explicitly available images for plot number N, we should somehow make sure this
monitored [14]. is true for actual plot number N in reality as well. To verify this matter,
some visible signs were made and were placed next to about 80 plots
Experimental Results in different locations across the field. Each sign was a red square and
In this section, part of the experimental results obtained after nine a tag showing the corresponding plot number. Figure 9 illustrates an
times data collection using the developed HTP platform at a Canola example for plot number 1.14. As can be seen, we selected an available
breeding facility in Saskatchewan, Canada during June to August 2017, image (a triangle object) on the left border of the mentioned plot in
is discussed. The developed visualization program was used to extract the visualization program. Because we know that there is a physical
information out of raw data. Also, a few strategies were used to verify the sign in that place in the field, the validity of this measurement can be
validity of measurements, including accuracy of height measurements verified only if the plot numbers on the actual field and in the program
as well as the positioning error and data geo-referencing. Breeders are matched.

Adv Robot Autom, an open access journal Volume 7 • Issue 1 • 1000186

ISSN: 2168-9695
Citation: Bayati M, Fotouhi R (2018) A Mobile Robotic Platform for Crop Monitoring. Adv Robot Autom 7: 186. doi: 10.4172/2168-9695.1000186

Page 5 of 7

This examination was continued for all other signs and it was found
that 100% of the signs were matched with their corresponding plots.
Comparison between manual and ultrasonic height
measurements
As can be seen in Figure 10, the height of some random plots in the
field using a scaled bar was manually measured. Because we are looking
for a single value as a representative of the height of different plots, we
captured the height of each plot in three different locations (left, middle
and right sides); the average of these three captured numbers can be
considered a true representative of the height of each plot.
As result persisted, the ultrasonic sensor could capture the height
of different plots with an acceptable level of error for a field-based
Figure 8: Monitoring a height data point within plot # 1.2 measurement system. Figure 11 illustrates a bar graph to compare
manual measurements and ultrasonic measurements for 19 random
plots. As can be seen, the manual height measurement corresponded
well with automatic data collected.
Analyzing the collected data for the whole population
One of the advantages of using a high-throughput plant
phenotyping platform is the fact that the resulting database can be used
in different approaches for different purposes. In fact, breeders have the
flexibility to study both entire population and the individual plots for
closer investigations. In this section, a summary of the results achieved
after studying a population consisting of 252 canola plots by the
proposed plant phenotyping platform is provided. Figure 12 illustrates
F the variation in the average canopy a) NDVI and b) temperature
Figure 9: Utilized visible signs to verify the validity of measurements for plot values of the studied population between June 23-August 18, 2017.
number 1.14. For example, by having a look at Figure 12 it can be comprehended
that the average NDVI and temperature of the entire population (252
plots) are recorded to reach their maximum values on July 7 and June
30, respectively. So we expect that the majority of the plant canopies
should be in their maximum greenness on July 7, and this matter can
be verified by investigating the captured images during July 7 trial.
Likewise, it was found that the average temperature of the entire plots
within the field had a continuous growth between June 23 and July 7,
and after that time we observed a fluctuation in the temperature until
August 4 when another continuous growth could be perceived. In fact,
breeders can study the drought stress by investigating in the resulting
graphs.
Analyzing the growth of individual plots
In the previous section, the variation in the average value of
Figure 10: Manual measurement of canopy height to verify automatic height
different parameters of the entire studied population (252 plots) was
measurement. discussed. In this section, it is shown that how the proposed HTP
platform can be used to analyze the individual plots as well. For

Figure 11: Comparison between manual vs. ultrasonic height measurements.

Adv Robot Autom, an open access journal Volume 7 • Issue 1 • 1000186

ISSN: 2168-9695
Citation: Bayati M, Fotouhi R (2018) A Mobile Robotic Platform for Crop Monitoring. Adv Robot Autom 7: 186. doi: 10.4172/2168-9695.1000186

Page 6 of 7

example, Figure 13a illustrates the growing steps of plot number 1.35 started to increase because the green color was gradually becoming
in three different days. It is notable that all images are extracted from dominant over the yellow color.
the developed visualization.
Conclusions
In fact, the capability of autonomously capturing images of
canopies is an important feature of the developed crop phenotyping A new mobile platform for field-based high-throughput plant
platform as discussed before. The growing stages of the plot number phenotyping was developed. It was shown that the mobile platform can
1.35 can be clearly seen in Figure 13b. As can be seen in Figure 13c, the be reliably utilized to phenotype canola cultivars (wheat or other plants
mentioned plot came into flowering on July 13; breeders can employ with some minor modification) with high repeatability and accuracy.
the same approach to analyze any desired plot. In fact, canopy height, temperature, and NDVI as well as RGB
images of 252 Canola plots in a nursery (in Saskatchewan, Canada)
Furthermore, the variation in different parameters of individual were captured through field experiments between June-August 2017.
plots can be analyzed as well. Figure 14 illustrates two line graphs to Autonomous image acquisition feature was implemented in a field-
illustrate the variation in temperature and NDVI values of plot number based HTP platform for the first time in Canada. The data acquisition
1.35 between June 23 and August 18. As can be perceived from Figure cycle minimized the geo-referencing error. In addition, a modular
14b, the temperature for the mentioned plot is recorded to be ~ 17°C on program with an interactive user interface was developed, using
June 23, and after some ups and downs, it was settled around 22°C on MATLAB for data visualization. This research is a proof of concept that
August 18. On the other hand, NDVI value for the mentioned plot had some of the most popular high-throughput phenotyping features can
a continuous growth after June 23 until the maximum NDVI value was be used automatically to achieve precision phenotypes in crop plants.
captured on July 7, which is ~0.78. After July 7, the majority of plants This technology improves the productivity of farms, and reduce the
came into flowers (with yellow color) so the NDVI value, which is an cost of the crop research in the intermediate and long run.
indication of the greenness level of a canopy, was decreased. A turning
point can be perceived on July 18 because the plots were observed in Acknowledgment
their maximum flower on July 18 (yellow color was dominant to green The authors wish to acknowledge the financial support provided by NSERC
color). After July 18, when flowers began to shrink, the NDVI value (Natural Sciences and Engineering Research Council of Canada) and the industrial

a) Average Canopy NDVI for Different Trials b) Average Canopy temperature for Different Trials
Figure 12: Variation in the entire studied population between June 23 and august 18.

a) Plot # 1.35 on June 23 b) Plot # 1.35 on July 4 c) Plot # 1.35 on July 13

Figure 13: The growing pictures of the plot number 1.35 in three different days.

a) Variation in NDVI values b) Variation in temperature values

Figure 14: Variation of different traits in plot # 1.35.

Adv Robot Autom, an open access journal Volume 7 • Issue 1 • 1000186

ISSN: 2168-9695
Citation: Bayati M, Fotouhi R (2018) A Mobile Robotic Platform for Crop Monitoring. Adv Robot Autom 7: 186. doi: 10.4172/2168-9695.1000186

Page 7 of 7

partner Cargill Canada Ltd. Also, the role of Mr. Tyler Zhang, Dr. Rahim Oraji, field phenotyping: performance and interest of the phenomobile system. 4th Int
Mr. Farzam Ayatizadeh, Robotics Laboratory at the University of Saskatchewan is Plant Phenotyping Symposium, Mexico.
gratefully acknowledged.
8. Whitham S, Howell S, Lawrence-Dill C, Lubberstedt T, Tang L (2016)
References Enviratron-An accelerator for climate change research. 4th Int Plant Phenotyping
Symposium, Mexico.
1. Jitendra K, Aditya P, Shiv K (2015) Phenomics in Crop Plants: Trends, Options
and Limitations, Springer. 9. Lillemo M, Bleken E, Lange G, Grimstad L, Burud I, et al. (2016) Exploring
the use of robot and drone as phenotyping tools in plant breeding, 4th Int Plant
2. White JW, Andrade-Sanchez P, Gore MA, Bronson KF, Coffelt TA, et al. (2012) Phenotyping Symposium, Mexico.
Field-based phenomics for plant genetics research, Field Crops Research 133:
101-112. 10. Araus JL, Cairns JE (2014) Field high-throughput phenotyping: the new crop
breeding frontier. Trends in Plant Science.
3. http://psi.cz/
11. https://hollandscientific.com/portfolio/crop-circle-acs-430/
4. Houle D, Govindaraju DR, Omholt S (2010) Phenomics: the next challenge.
Nature Reviews. Genetics 11: 855-866. 12. Busemeyer L, Mentrup D, Mölle K, Wunder E, Alheit K, et al. (2013) Breed
Vision-A Multi-Sensor Platform for Non-Destructive Field-Based Phenotyping
5. Andrade-Sanchez P, Gore MA, Heun JT, Thorp KR, Carmo-Silva AE, et in Plant Breeding. Sensors 13: 2830-2847.
al. (2014) Development and evaluation of a field-based high throughput
phenotyping platform. Functional Plant Biology 41: 68-79. 13. Araus JL, Cairns JE (2014) Field high-throughput phenotyping: the new crop
breeding frontier. Trends in Plant Science.
6. Montes JM, Melchinger AE, Reif JC (2007) Novel throughput phenotyping
platforms in plant genetic studies. Trends Plant Sciences 12: 433-436. 14. Ruckelshausen A, Biber P, Dorna M, Gremmes H, Klose R, et al. (2009)
BoniRob: an autonomous field robot platform for individual plant phenotyping.
7. de Solan B, Baret F, Thomas S, Moulin O, Meloux G, et al. (2016) Water stress Precision Agriculture 841-847.

Adv Robot Autom, an open access journal Volume 7 • Issue 1 • 1000186

ISSN: 2168-9695