Transfer Learning and Fine Tuning for Deep

Learning-Based Tea Diseases Detection on Small

Datasets
Ade Ramdan, Ana Heryana, Andria Arisal, R. Budiarianto S. Kusumo,
Vicky Zilvan, and Hilman F. Pardede
Research Center for Informatics
Indonesian Institute of Sciences
Bandung, Indonesia
{ader00, anah002, andr001, rbud001, vick001, hilm001}@lipi.go.id

Abstract—It is well-known that large data are required to To mitigate problems of small dataset, several methods have
train deep learning systems. However, collecting large amount been proposed in the past. First is data augmentation [14]. Data
of data are very costly and in some cases impossible. To deal augmentation aims to generate artificial data from existing data
with this, transfer learning could be applied by adopting models
that are trained using large data into the target domain which has to increase their amounts. Traditionally, data augmentation is
small data. In this paper, we adopt models that are pre-trained conducted by applying various image processing such as shift-
on ImageNet data into tea diseases detection data and propose ing, translating, and rotating to the existing data randomly [15].
transfer learning and fine tuning for tea diseases detection. Three However, the generated data would be similar, making the
pre-trained models using deep convolutional neural network systems prone to overfit. To deal with this, other studies
(DCNN) architectures: VGGNet, ResNet, and Xception are used.
To mitigate the dissimilarity of ImageNet and our tasks, we apply combine parts of data to generate new data [16], [17]. In our
fine-tuning on the pre-trained models. Our experiments show previous study, we apply data augmentation by adding noise
applying transfer learning only on our data may not be effective artificially on image data to improve the robustness against
due to the dissimilarity of our task to ImageNet. Applying fine environmental variations [18].
tuning on pre-trained DCNN models is found to be effective. It Currently, there are increasing studies that apply deep
consistently better than that of using transfer learning only or
partial fine-tuning. It is also better than training the model from learning for data augmentation. Auto-encoder, generative ad-
scratch, i.e. without using pre-trained models. versarial networks, and their variants are trained such that they
Index Terms—transfer learning, fine tuning, tea diseases de- produce artificial data that has similar distributions to that of
tection, CNN, deep learning, agriculture the real data [19], [20]. However, the same issues as other deep
learning methods would remain. The generated data would be
I. I NTRODUCTION better when large amount of data are available for training.
Without them, the generated data may not be realistic enough.
The use of machine learning in agriculture are increasing Transfer learning is also a way to deal with limited data
rapidly [1]. It is used for instance in precision agriculture [2], training [21]. Transfer learning aims to use models trained
[3], smart farming [4], species and variety detections [5], [6], one problem/data on different problems. There have been
and diseases detection [7], [8]. Currently, deep learning tech- several approaches for transfer learning [22]. They are instance
nologies [9] have become state-of-the-art technologies in many based, mapping-based, network-based, and adversarial-based
application in agriculture. One of advantages of deep learning methods. In this paper, we only focus on network-based
is its capability to automatically learn useful information of transfer learning. In network-based, the models, often used
raw data [10]. Thus, making it unnecessary to design features as pre-trained models, could be used either as it is or as
manually as it is the case in conventional machine learning initial model for training the models on the target problems.
methods. As the results, it is easier to implement systems with The latter is called fine tuning. Fine tuning is conducted by
good performance without getting too deep knowledge about replacing or appending certain parts of the networks with a
the data. new network. Parts or the whole networks are kept and then
However, deep learning requires huge data for training transferred into target domains. Therefore, there is no need to
to fully benefit from its capability. While there is no exact trained the networks in the target domains from scratch which
formula how to determine the optimal number of data for would reduce the training time as the results. In agriculture,
training deep learning, studies found that having more data their examples are [23], [24]. In these studies, pre-trained
certainly benefit deep learning systems [11]–[13]. Unfortu- AlexNet, GoogleNet, and VGGNet are used. It is found that
nately, acquiring large amount of data are often very costly transfer learning is more effective than training from scratch.
and often impossible. The effectiveness of transfer learning relies on several
factors. First is the similarity of the source models to the target
problems. It is well-known that deep learning architectures
learn different abstraction for each layer [25], [26]. They are
also found to be very good on extracting features of data.
Earlier layers are able to capture related knowledge on data
while latter layers are good to model their relations to the
class labels. Therefore, when the source and target domains
are close, the whole networks could directly be used as feature
extractors and training is only conducted on output layers. But
when the source and target domains are more dissimilar, it is
better to apply fine tuning to more structures of the networks.
It is also found that there is relation between the architectures
of the network and their transferability as reported in [27].
In this paper, we apply transfer learning and fine tuning
for tea diseases detection. We use three pre-trained models:
ResNet, VGGNet, and Xception which are trained using
imageNet data. We replace two last layers of the networks with
two-layers of multi-layer perceptron. Then, we apply various
fine-tuning strategy. First, we use the networks as it is and only
train the networks on the last two layers. Second, we only keep
the several first layers of the networks and re-trained the rest.
Lastly, we only use the pre-trained models as initialization,
and fine-tuned the whole networks on our data.
The remaining of the paper is organised as follows. In
Section II, we explain our proposed method. In Section III,
we describe our data and the experimental setup. We discuss
and analyse our results in Section IV. Finally, our paper is
concluded in Section V.
Fig. 1. The proposed method
II. P ROPOSED M ETHOD
In this paper, three pre-trained models of deep convolutional
neural networks (DCNN) are used. They are VGGNet16 (we
denote it as VGGNet) [28], Resnet50 (we denote it as ResNet)
[29], and Xception [30]. The models are trained with ImageNet
database. For each model, we cut the first layer (input layer)
and the last two layers. Then we append our input layers and
two fully connected (dense) layers unto them. The dense layers
comprise of a dense layer and a layer with output nodes of
4 (it is the same as the number of classes of our data). Our
proposed method is illustrated in Fig. 1.
The details of the appended architectures to the pre-trained
models is shown in Fig. 2. The input size is set to 128×128×3
for VGGNet and Xception while it is set to 200 × 200 × 3 for
ResNet. This is because ResNet could not handle images with
resolution less than 197 × 197 × 3. For VGGNet, we apply
flatten layers that is followed by dense layer with output nodes
of 1024. For ResNet, the dense layer has output nodes of 200 Fig. 2. The architectures of appended networks to the pre-trained models
and it is 256 for Xception.
We apply two approaches to train our models. First is to
freeze all layers except those are appended. So the models instead using random weights in training. We apply two
are used as they are and the DCNN models act as feature schemes of fine-tuning. First is by conducting fine tuning
extraction processes to two layers fully connected networks. partially. We freeze the first ten layers of each model and
We denote this approach as Tl. only update the weights on the rest. We denote this scheme
In the second approach, we apply fine-tuning to the trans- as Tl ft I. Second, we conduct fine tuning on the whole
ferred models. In fine tuning, the models are updated during networks. We denote this as Tl ft II. As a reference, we also
training and the transferred models are used as initial models train all DCNN models are trained from scratch, i.e. without
 TABLE I
C OMPARISON OF ACCURACIES ON TEST SET OF VARIOUS TRAINING
SCHEMES : TRAINING FROM SCRATCH (T S C ), TRANSFER LEARNING (T L ),
TRANSFER LEARNING AND PARTIAL FINE TUNING (T L FT I), AND
TRANFER LEARNING AND FULL FINE TUNING (T L FT II).

DCNN Models T Sc Tl Tl ft I Tl ft II
Fig. 3. Samples of data for each class
ResNet 73.33 19.73 19.73 94.05
VGGNet 84.86 86.04 90.27 91.26
Xception 76.85 56.49 59.19 91.71
transfer learning. We denote this as T Sc.

III. S ETUP
(a) (b)
A. Data
1
We apply transfer learning for tea diseases detection [18]. 6

Accuracy(%)
There are 5,632 leaf images that are collected at Research
Center for Tea and Cinchona, West Java, Indonesia. The data 4 0.8

Loss
comprise of four class labels. The labels consist of three types T Sc

of diseases on tea plants which are commonly found in tea 2 Tl

0.6 Tl ft I
plantation in Indonesia and healthy leaves. The diseases are 0 Tl ft II

blister blight (1,104 images), Empoasca or leaf-hopper attack

0 20 40 60 80 100 0 20 40 60 80 100
(2,055), and looper caterpillars attacks (1,104 images). There
are 1,448 images of healthy class. Samples of data are shown Epochs Epochs
in Fig. 3. (c) (d)

B. Experimental setup 1

Accuracy(%)
In the experiments, the data is divided into three parts: 10 0.8
training, validation, and testing sets with distribution of 70
Loss

0.6
%, 10 %, and 20 % respectively. There are 3,933 for training, 5
589 images for validation, and 1,110 images for testing. 0.4
The images are resized into 128×128×3 and 200×200×3. 0.2
The first size is used for VGGNet and Xception while the latter 0
is used for ResNet since ResNet requires data with size larger 0 20 40 60 80 100 0 20 40 60 80 100
than 197 × 197 × 3. For training, we use Adam as optimizer Epochs Epochs
with initial learning rate is set up at 10−5 . We set the batch
size to 20 and the number of epoch is set at 100. Fig. 4. Progression of (a) Training loss, (b) Training accuracy, (c) validation
Loss, and (d) validation loss of various training schemes: T Sc, Tl, Tl ft I,
and Tl ft II for ResNet
IV. R ESULTS AND D ISCUSSIONS
Table IV compares the accuracy of all evaluated schemes:
training from scratch (T Sc), transfer learning (Tl), transfer recognition, and target problem, i.e. tea diseases detection, are
learning and partial fine tuning (Tl ft I), and transfer learn- quite dissimilar. While it is believed, that DCNN could capture
ing and full fine tuning (Tl ft II). It is clear that transfer various abstraction of data that could discriminate between
learning and fine tuning achieves the best performance over class labels, their physical meaning may not always obvious.
all training schemes. T Sc could only achieve accuracy of This is especially true for complex and nested DCNN struc-
84.86 % for VGGNet. Lower performance is achieved by tures as ResNet and Xception. Meanwhile, stack-structured of
ResNet and Xception. Meanwhile Tl ft II could achieves VGGNet may learn the abstraction in more sequential manner.
the best accuracy of 94.05 % ResNet. Tl ft II consistently VGGNet may learn a more general abstraction than that of
achieves better performance than T Sc for all DCNN models. ResNet and Xception. As the results, VGGNet may be more
Compared to T Sc, Tl ft II achieves 20.72 %, 6.4 %, 14.86 suitable for feature learning.
% absolute improvements and 77.69 %, 42.27 %, and 64.19 Applying fine tuning on the pre-trained models is found
% relative improvements for ResNet, VGGNet, and Xception effective in our task. Tl ft I and Tl ft II are better than
respectively. Tl even though on ResNet and Xception, Tl ft I is worse
We notice that transfer learning is only effective for VG- than T Sc. Since it is believed that later layers of DCNN
GNet. On VGGNet, Tl achieves 1.18 % absolute improvement progressively learn more specific nature of the class-labels,
over T Sc, Tl ft I is 5.41 % better than T Sc. On ResNet fine-tuning these layers may benefit. But, since the source
and Xception, Tl and Tl ft I are largely worse than T Sc. models and target problems is quite dissimilar, it is better to
It is obvious that source model, i.e. imageNet for object fine tuned all layers.
 (a) (b) of ResNet, VGGNet, and Xception respectively for T Sc, Tl,
Tl ft I, and Tl ft II. It is obvious that Tl, Tl ft I, and Tl ft II
1 converge faster than T Sc, indicating the faster training. The

Accuracy(%)
graphs also indicate that using transfer learning only for
1 0.8 ResNet and Xception may not be effective (See red and green
Loss

T Sc lines in Figs. 4 and 6). The ResNet and Xception models may
0.5 0.6 Tl over-fit to ImageNet task, making it trapped to local minimum
Tl ft I
0.4 Tl ft II
when they are directly applied into our data. However, when
0 pre-trained models are only used to initiate training, better
0 20 40 60 80 100 0 20 40 60 80 100 models could be trained on our data, producing models with
Epochs Epochs better performances.
(c) (d)
V. C ONCLUSIONS
In this paper, we apply transfer learning and fine tuning
Accuracy(%)

0.8 on ImageNet pre-trained models of several DCNN models:

1 ResNet, VGGNet, and Xception on tea diseases detection. Due
Loss

to the small size of our data, applying DCNN from scratch

0.6
may not be optimal. Our experiments show that using transfer
0.5
learning only is not effective when applied to models with
0.4
nested structures such as ResNet and Xception. Fine-tuning on
0 20 40 60 80 100 0 20 40 60 80 100 all layers of pre-trained models achieves the best performance
Epochs Epochs over all evaluated training schemes. It is significantly better
than training DCNN from scratch.
Fig. 5. Progression of (a) Training loss, (b) Training accuracy, (c) validation There have been a plethora efforts to deal with limited
Loss, and (d) validation loss of various training schemes: T Sc, Tl, Tl ft I, data. In the future, we plan design better architectures for
and Tl ft II for VGGNet
transfer learning and use more recent and complex models.
Combining various data augmentation and transfer learning
(a) (b) on our problem are also in our future plans.
1 ACKNOWLEDGMENT
1
Accuracy(%)

The experiments on this research are conducted on High

0.8
Loss

Performance Computing (HPC) facility in Research Center

0.5 T Sc
for Informatics, Indonesian Institute of Sciences. All authors
Tl
0.6 Tl ft I contributed equally on the paper.
Tl ft II
0
R EFERENCES
0 20 40 60 80 100 0 20 40 60 80 100
[1] K. G. Liakos, P. Busato, D. Moshou, S. Pearson, and D. Bochtis,
Epochs Epochs “Machine learning in agriculture: A review,” Sensors, vol. 18, no. 8,
(c) (d) p. 2674, 2018.
[2] A. Chlingaryan, S. Sukkarieh, and B. Whelan, “Machine learning
approaches for crop yield prediction and nitrogen status estimation in
6 precision agriculture: A review,” Computers and electronics in agricul-
Accuracy(%)

ture, vol. 151, pp. 61–69, 2018.

0.8 [3] S. Dimitriadis and C. Goumopoulos, “Applying machine learning to
4
Loss

extract new knowledge in precision agriculture applications,” in 2008

Panhellenic Conference on Informatics. IEEE, 2008, pp. 100–104.
2 0.6 [4] R. Varghese and S. Sharma, “Affordable smart farming using iot
and machine learning,” in 2018 Second International Conference on
Intelligent Computing and Control Systems (ICICCS). IEEE, 2018, pp.
0 645–650.
0 20 40 60 80 100 0 20 40 60 80 100 [5] M. Kumar, S. Gupta, X.-Z. Gao, and A. Singh, “Plant species recognition
Epochs Epochs using morphological features and adaptive boosting methodology,” IEEE
Access, vol. 7, pp. 163 912–163 918, 2019.
[6] E. Suryawati, V. Zilvan, R. S. Yuwana, A. Heryana, D. Rohdiana, and
Fig. 6. Progression of (a) Training loss, (b) Training accuracy, (c) validation H. F. Pardede, “Deep convolutional adversarial network-based feature
Loss, and (d) validation loss of various training schemes: T Sc, Tl, Tl ft I, learning for tea clones identifications,” in 2019 3rd International Con-
and Tl ft II for Xception ference on Informatics and Computational Sciences (ICICoS). IEEE,
2019, pp. 1–5.
[7] U. Shruthi, V. Nagaveni, and B. Raghavendra, “A review on machine
learning classification techniques for plant disease detection,” in 2019
Figures 4, 5, and 6 show the progression of training losses, 5th International Conference on Advanced Computing & Communica-
training accuracies, validation losses, and validation accuracies tion Systems (ICACCS). IEEE, 2019, pp. 281–284.
 [8] H. F. Pardede, E. Suryawati, R. Sustika, and V. Zilvan, “Unsupervised [30] F. Chollet, “Xception: Deep learning with depthwise separable convolu-
convolutional autoencoder-based feature learning for automatic detection tions,” in Proceedings of the IEEE conference on computer vision and
of plant diseases,” in 2018 International Conference on Computer, pattern recognition, 2017, pp. 1251–1258.
Control, Informatics and its Applications (IC3INA). IEEE, 2018, pp.
158–162.
[9] A. Kamilaris and F. X. Prenafeta-Boldú, “Deep learning in agriculture: A
survey,” Computers and electronics in agriculture, vol. 147, pp. 70–90,
2018.
[10] Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A
review and new perspectives,” IEEE transactions on pattern analysis
and machine intelligence, vol. 35, no. 8, pp. 1798–1828, 2013.
[11] R. L. Figueroa, Q. Zeng-Treitler, S. Kandula, and L. H. Ngo, “Predicting
sample size required for classification performance,” BMC medical
informatics and decision making, vol. 12, no. 1, p. 8, 2012.
[12] X. Zhu, C. Vondrick, C. C. Fowlkes, and D. Ramanan, “Do we need
more training data?” International Journal of Computer Vision, vol. 119,
no. 1, pp. 76–92, 2016.
[13] P. L. Bartlett and W. Maass, “Vapnik-chervonenkis dimension of neural
nets,” The handbook of brain theory and neural networks, pp. 1188–
1192, 2003.
[14] C. Shorten and T. M. Khoshgoftaar, “A survey on image data augmen-
tation for deep learning,” Journal of Big Data, vol. 6, no. 1, p. 60,
2019.
[15] M. D Bloice, C. Stocker, and A. Holzinger, “Augmentor: An image
augmentation library for machine learning,” JOSS, vol. 2, no. 19, p.
432, 2017.
[16] G. Hu, X. Peng, Y. Yang, T. M. Hospedales, and J. Verbeek, “Franken-
stein: Learning deep face representations using small data,” IEEE
Transactions on Image Processing, vol. 27, no. 1, pp. 293–303, 2017.
[17] E. Okafor, L. Schomaker, and M. A. Wiering, “An analysis of rotation
matrix and colour constancy data augmentation in classifying images of
animals,” Journal of Information and Telecommunication, vol. 2, no. 4,
pp. 465–491, 2018.
[18] R. S. Yuwana, E. Suryawati, V. Zilvan, A. Ramdan, H. F. Pardede,
and F. Fauziah, “Multi-condition training on deep convolutional neural
networks for robust plant diseases detection,” in 2019 International
Conference on Computer, Control, Informatics and its Applications
(IC3INA), Oct 2019, pp. 30–35.
[19] D. Kornish, S. Ezekiel, and M. Cornacchia, “Dcnn augmentation via
synthetic data from variational autoencoders and generative adversarial
networks,” in 2018 IEEE Applied Imagery Pattern Recognition Workshop
(AIPR). IEEE, 2018, pp. 1–6.
[20] C. Han, L. Rundo, R. Araki, Y. Nagano, Y. Furukawa, G. Mauri,
H. Nakayama, and H. Hayashi, “Combining noise-to-image and image-
to-image gans: Brain mr image augmentation for tumor detection,” IEEE
Access, vol. 7, pp. 156 966–156 977, 2019.
[21] S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Trans-
actions on knowledge and data engineering, vol. 22, no. 10, pp. 1345–
1359, 2009.
[22] C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu, “A survey on
deep transfer learning,” in International conference on artificial neural
networks. Springer, 2018, pp. 270–279.
[23] S. P. Mohanty, D. P. Hughes, and M. Salathé, “Using deep learning for
image-based plant disease detection,” Frontiers in plant science, vol. 7,
p. 1419, 2016.
[24] M. M. Ghazi, B. Yanikoglu, and E. Aptoula, “Plant identification using
deep neural networks via optimization of transfer learning parameters,”
Neurocomputing, vol. 235, pp. 228–235, 2017.
[25] R. Kozma, R. Ilin, and H. T. Siegelmann, “Evolution of abstraction
across layers in deep learning neural networks,” Procedia computer
science, vol. 144, pp. 203–213, 2018.
[26] C. Buckner, “Empiricism without magic: Transformational abstraction
in deep convolutional neural networks,” Synthese, vol. 195, no. 12, pp.
5339–5372, 2018.
[27] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, “How transferable are
features in deep neural networks?” in Advances in neural information
processing systems, 2014, pp. 3320–3328.
[28] K. Simonyan and A. Zisserman, “Very deep convolutional networks for
large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
[29] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
recognition,” in Proceedings of the IEEE conference on computer vision
and pattern recognition, 2016, pp. 770–778.