Sensing Fruit Ripeness Using Wireless Signals

Sheng Tan, Linghan Zhang, Jie Yang

Florida State University, Tallahassee, Florida, USA
{tan, lzhang, jie.yang}@cs.fsu.edu

Abstract—This paper presents FruitSense, a novel fruit north America, compare to a 12% loss of fruits and vegetables
ripeness sensing system that leverages wireless signals to enable in distribution and retail process, the loss rate is 28% after
non-destructive and low-cost detection of fruit ripeness. Such a customer purchase. This is due to fruits are sensitive to various
system can reuse existing WiFi devices in homes without the
need for additional sensors. It uses WiFi signals to sense the environmental factors such as temperature, air ventilation,
physiological changes associated with fruit ripening for detecting and illumination condition. Those factors are very difficult
the ripeness of fruit. FruitSense leverages the larger bandwidth to control especially within normal household environments.
at 5GHz (i.e., over 600MHz) to extract the multipath-independent Moreover, ordinary consumers lack the professional knowl-
signal components to characterize the physiological compounds edge and equipment for accurately judging fruit ripeness level
of the fruit. It then measures the similarity between the extracted
features and the ones in ripeness profiles for identifying the by appearance. Monitoring the ripeness level of fruits thus
ripeness level. We evaluate FruitSense in different multipath can benefit customers by avoiding the unpleasantness of tart
environments with two types of fruits (i.e, kiwi and avocado) or rotten fruit and eventually reduce the waste.
under four levels of ripeness. Experimental results show that The challenge in monitoring the ripeness of fruit for
FruitSense can detect the ripeness levels of fruits with an consumers lies in finding solutions that can provide non-
accuracy over 90%.
destructive testing at minimal cost. Existing approaches for
I. I NTRODUCTION inferring the ripeness of fruit mainly depend on penetrome-
Recent advances in wireless technology have greatly ex- ter [11], refractometer [12], or spectrometer [20]. While the
panded the WiFi usage from providing laptop connectivity to penetrometer measures fruit firmness by quantifying the force
connecting mobile and smart devices to the Internet and home required to insert a probe into the fruit [11], refractometer
networks. Such an evolution has resulted in the prevalence of (e.g., Brix [12]) analyzes sugar content of the juice using
WiFi devices, which provides opportunities to extend WiFi’s light refraction. Both methods, however, are destructive and
capabilities beyond communication, particularly in human less acceptable to consumers. In contrast, spectrometer based
sensing. As the wireless signals travel through space, they approach is non-destructive. It splits light signals into a fruit
interact with human body and undergo wave phenomena and then measures the light that is emitted, absorbed or
such as reflection and diffraction. These phenomena lead to scattered by the fruit for ripeness inference [20]. However,
multipath effects, which carry a rich set of information about traditional spectrometer instruments are bulky and expensive
the physical environment including the human location and (i.e., costing several thousands of dollars) and are limited to
activities. Indeed, there has been a growing interest in using controlled laboratory settings [20].
multipath effects to perform human sensing, ranging from More recent work includes using advanced imaging tech-
large scale movements [30], [25], to small scale motions [5], niques and ultrasonic measurement systems to analyze the im-
[27], [19], and location [28], [4], [29]. age features (e.g., color and texture) and measure the ultrasonic
In this work, we further expand the WiFi sensing capabilities attenuation of the fruit for ripeness detection [14], [3], respec-
from human sensing to sensing bio-information of fruits. In tively. However, these methods require specialized equipments,
particular, we seek to sense the degree of ripeness in fruits and are less accessible to ordinary consumers. Recently, due
with WiFi signals. Monitoring the ripeness of fruit provides to advances in materials and fabrication techniques, portable
many benefits for several end-users, ranging from farmers, spectrometers that work together with smartphones have been
to distributors, retailers and consumers. It can help farmers realized. For example, Das et al. [8] propose a smartphone
to determine the optimal harvest time as the quality heavily based spectrometer that can measure UV fluorescence of
depends on when they are harvested. It can also assist fruit Chlorophyll found in apples, whereas the company Consumer
distributors in performing rapid sorting in storage facilities Physics develops the sensor SCiO that can be integrated with
and deciding when to send their stock during post-harvest smartphones to analyze the molecular composition of food [2],
period. Retailers can minimize losses and maintain fruit quality [1]. Although these portable solutions can be adapted by
through effective selling strategies based on accurate catego- consumers, there also are non-negligible costs incurred in
rization of the fruit ripeness. Monitoring the ripeness of fruits purchasing dedicated spectral sensors.
throughout the supply chain thus can reduce waste as well as In this paper, we introduce FruitSense, a novel method
improve the consistency of quality for consumers. for inferring the ripeness of fruit for consumers that is both
In this paper, we focus on inferring the ripeness of fruit for non-destructive and low-cost. FruitSense uses WiFi signals to
consumers. As indicated in previous study from 2011 [10], in sense the physiological changes associated with fruit ripening

978-1-5386-5156-8/18/$31.00 ©2018 IEEE

for ripeness detection. It allows users to reuse off-the-shelf the WiFi NICs in the form of Channel State Information
WiFi devices for ripeness monitoring, and thus can enable (CSI), then goes through the calibration process to correct
large-scale deployment and benefit a large number of users by errors due to hardware limitation of WiFi NICs. We then stitch
leveraging the proliferation of WiFi devices and networks. together the calibrated CSI from each individual channel. Due
In particular, fruit ripening involves a series of physiological to the usable WiFi channels in 5GHz are unequally and non-
changes leading to the development of a soft edible ripe contiguous spaced, the inverse NDFT is used to derive the
fruit. Take avocado as an example, there are changes in fine-grained power delay profile for multipath removal. Next,
the total dry matter and moisture content during maturation we identify and extract the signals that directly went through
and ripening. As moisture content decreases, the dry matter the fruit for ripeness detection. At last, we use the Maximal
increases accordingly. When wireless signals travel through a Overlap Discrete Wavelet Transform (MODWT) to extract
fruit, the changes in the physiological compounds of the fruit signal features over multiple channels at multiple resolutions
during ripening lead to distinct and measurable effects on the and compare the features against known ripeness profiles that
received signals. We thus can infer the ripeness level based on identify the degree of ripeness.
the physiological changes interpreted by the received signals. We evaluate FruitSense in different multipath environments
Accurately discerning the ripeness of fruit is challenging, with two types of fruit: kiwi and avocado. The fruit samples
however, when using a single pair of off-the-shelf WiFi de- are purchased from two venders with different ripeness levels.
vices. First, the measurable changes in the received signals due The volume of each type of fruit is about 75 at each ripeness
to fruit ripening are subtle, and the impact of fruit size on the level. We identify the ripeness as one of the four levels:
received signals might distort such changes. To address these unripen, half ripen, ripen and over ripen. Experimental results
issues, we leverage frequency diversity by probing the fruit show that FruitSense is highly effective in detecting fruit
with WiFi signals at multiple frequency channels. Leveraging ripeness. It achieves an accuracy at over 90% in identifying
frequency diversity not only provides a rich set of information the ripeness level of fruit.
to capture the subtle changes, but also enables us to examine The main contributions are summarized as follows.
the change pattern among multiple channels, instead of a • We show that the WiFi signals can be utilized to capture
single channel, to mitigate the impact of fruit size. the physiological changes of fruit for ripeness detection.
Addtionaly, signal propagation is dominated by multipath The system is non-destructive and low-cost, without the
in typical indoor environments when the line-of-sight (LOS) need for additional sensors.
is blocked by fruit during the ripeness sensing process. While • We leverage frequency diversity to capture the physiolog-
the WiFi based human sensing primarily relies on the signal ical changes of the fruit, and utilize the larger bandwidth
reflections, the reflected signals from surroundings represent at 5GHz to combat multipath for extracting multipath-
interferences to the signal component indicating the ripeness independent signal components for ripeness detection. We
of fruit. Once the indoor environment changes, the changes in further utilize MODWT to extract features over multiple
the received signals mainly reflect the differences in multipath frequencies for identifying the level of fruit ripeness.
propagation instead of the physiological change of the fruit. To • We conduct extensive experiments in different multipath
accurately capture the changes corresponding to fruit ripening, environments with two typical fruits under various condi-
we propose to extract the multipath-independent signal compo- tions. Experimental results show that FruitSense achieves
nents for ripeness sensing by isolating the signal component around 90% ripeness level detection accuracy.
traveling through the fruit directly, from the ones reflected
from the surrounding environment. Specifically, we leverage II. P RELIMINARIES
the larger bandwidth at 5GHz (i.e., over 600MHz) to derive
fine-grained power delay profile for extracting the signals that A. Fruit Ripening
directly went through the fruit from the multipath propagation. There exists two types of fruits according to the regulatory
Moreover, as the usable WiFi channels in 5GHz are un- mechanisms underlying their ripening process: climacteric
equally and non-contiguous spaced, the inverse non-uniform and non-climacteric fruits. Climacteric fruits, such as kiwi,
Discrete Fourier Transform (NDFT) is used to derive the avocado, apple and banana will continue to ripen after the
fine-grained power delay profile. However, there are various fruit has left the plant; however, non-climacteric fruits, such
uncertainties introduced from the inverse NDFT as it relies as grape, orange and pineapple stop the ripening process the
on proximal gradient methods to find solutions to an under minute they leave the plant [24]. In this work, we focus
determined system. To mitigate such uncertainties, we propose on sensing the ripeness of climacteric fruits, as we do not
to use the Maximal Overlap Discrete Wavelet Transform have the access to the non-climacteric fruits that are on the
(MODWT) for multi-resolution feature analysis, which is not plants. Nevertheless, our system can be extended to sensing
sensitive to the uncertainties such as the starting and ending the maturity of non-climacteric fruits during the growth and
points (and possibly outliers) of a series of signal. development process while they are on the plants.
More specifically, our system first probes the fruit with Fruit ripening is a highly coordinated and an irre-
WiFi signals hopping at all available 5GHz channels. The versible phenomenon involving a series of physiological and
sampled channel frequency response, which is exported by organoleptic changes [24]. For example, some fruits, such as
 Path2 Refraction
Path2
Reflection
Path1 Path1

Power
TX RX
Path3
Path3 Path4

Path1
Path4
TX RX
0 τ τ + 1.5 τ + 3 τ + 4.5
Time(ns)

Fig. 1. Multipath propagations in an indoor environment during fruit ripeness Fig. 2. Combining all the channels at 5GHz provides a power delay profile
sensing. with sufficient resolution to differentiate multipath propagations.
00
banana and apple, come in a wide array of colors that change exposed to an electric field. The imaginary
√ part ε is referred
throughout their ripening process, with the brightest colors to as dielectric loss factor, and j = −1. The dielectric loss
often occurring when the fruit is optimally ripened. Such color factor influences both energy attenuation and absorption, and
changes enable people to predict the ripeness level based on is commonly used to describe the ability of the material to
external visual inspection. By contrast, some other fruits, like dissipate electrical energy as heat.
kiwi and avocado, do not exhibit obvious organoleptic changes As fruit ripening involves a series of physiological changes,
during the ripening process. It is hard for ordinary people to fruit at different ripeness levels results in different dielectric
tell the ripeness level based on organoleptic testing. We thus constants and loss factors. The permittivity of the fruit thus
focus on kiwi and avocado and seek to sense the ripeness based could be used as an indicator of its internal quality. Indeed,
on physiological changes, instead of organoleptic changes. there exists prior work on using expensive and dedicated
Existing work uses the physiological changes such as oil dielectric spectroscopy to measure the dielectric properties of
context, dry matter and moisture content to determine the fruit and vegetable for internal quality analysis [17].
ripeness of avocado [20]. Specifically, oil content and dry mat- As we use the off-the-shelf WiFi devices, we exam how
ter increase during development and continue to change during the received signal changes due to the changed physiological
ripening [20]. As oil content increases, the moisture content compounds of the fruit. Specifically, when the WiFi signal
decreases by the same amount, so that the total percentage travels through the fruit, the electric field strength decreases
of oil and moisture content remains constant. Similarly, the with the distance from its surface. To quantify such an effect,
content change within dry matter can be used to track the the attenuation factor α, which depends on the dielectric
ripeness of kiwi fruit [24]. In particular, dry matter of kiwi properties of the fruit, could be leveraged. It is given by:
is mainly comprised by the carbohydrates and starch, which r
2π h 1 0   ε00 2 i1/2
gradually transform to soluble solids such as sugar content α= ε 1+ − 1 (2)
during the ripening. The percentage of carbohydrates and λ0 2 ε0
starch thus could be used as an indicator of fruit ripeness. where λ0 is the free-space wavelength of the WiFi signal [18].
B. WiFi Sensing based Approach Based on Equation 2, we know that the fruits at different
ripeness levels lead to different attenuation factors, which
Figure 1 illustrates a typical experimental setup of sensing
could be measured by analyzing the received signals. There-
fruit ripeness using a pair of WiFi devices in an indoor
fore, instead of using dedicated dielectric spectroscopy, we
environment. The testing fruit is placed in between a pair of
leverage the received signal that directly travels through the
closely spaced transmitter and receiver, thus blocking the line-
fruit for ripeness detection.
of-sight (LOS) signal propagation. We rely on the received
Excepting for the controllable experimental settings such
signal component that directly travels through the fruit to
as the location of the fruit in between the transmitter and
sense its physiological change. The phenomenon where radio
the receiver, the sizes of the fruit could be slightly different,
waves travel through and modulated by the fruit is commonly
which may affect the signal attenuation as well. Note that from
referred as refraction, which describes the signal passes from
Equation 2, we can observe that the WiFi signals at different
one medium to another. In our case, it is the WiFi signal travels
frequencies (wavelengths) result in different attenuation factors
from air to fruit, and then from fruit to air.
as well. We thus can leverage the frequency diversity by
To quantify the effect of the refraction, we could leverage
probing the fruit at multiple channels, and then analyze the
the concept of permittivity, which is a measure of how an
relative changes between multiple frequencies to mitigate the
electric field affects, and is affected by, a dielectric medium
impact of fruit size.
(i.e, the fruit in our case). In particular, the complex relative
permittivity ε∗ of a material to that of free space in frequency C. Practical Issues
domain can be described as following [18]:
While the intuition is simple, there are significant challenges
∗ 0 00 to accurately capture the received signal that directly travels
ε = ε − jε (1)
through the fruit for extracting the multipath-independent
0
The real part ε is referred to as the dielectric constant, which features that characterize the physiological compounds of the
describes the ability of the material to store energy when it is fruit. As shown in Figure 1, besides the signal refraction, there
exists signal reflections. The signals that reflected from the from various samples. To overcome such challenge, we utilize
walls, furniture, and human body will be combined with the Maximal Overlap Discrete Wavelet Transform (MODWT),
signal travels through the fruit at the receiver. This leads to which is shift invariant and is not sensitive to the starting and
the fact that the measured signals at the receiver mainly reflect ending points (and possibly outliers) of the data. By utilizing
the multipath environments. Thus it is highly sensitive to envi- MODWT, we are able to minimize the uncertainty caused by
ronment changes. To make the system robust to the multipath NDFT and extract consistent features from given samples.
propagation, we need to separate the signal component that III. S YSTEM D ESIGN
travels through the fruit from the reflected ones.
Intuitively, this can be done by leveraging the power delay A. System Overview
profile, which gives the power intensity of received signals Our system uses a pair of WiFi devices to sense the phys-
as a function of propagation delay. By performing Inverse iological compounds of fruit for ripeness detection. Figure 3
Fast Fourier Transform (IFFT) of the received signal mea- illustrates the flow of our system. It first probs the fruit with
surements, we are able to extract the first arriving signal, WiFi signals hopping through all usable channels at 5GHz.
which travels through the shortest path among all the paths The system then collects the sampled channel frequency
from the transmitter to the receiver. However, the widely used response in the form of Channel State Information (CSI)
bandwidth of a WiFi channel is either 20MHz or 40MHz, including the information of phase and amplitude. The CSI
which results in a power delay profile with a resolution at measurements are reported by the WiFi NIC at the receiver.
either 50ns or 25ns. Given that the wireless signal travels The CSI measurements are then preprocessed to calibrate
at the speed of light, such resolutions correspond to distance both the phase and amplitude errors. In particular, the raw
resolutions of 15m and 7.5m, respectively. In typical indoor phase contains residual synchronization errors, which are com-
environments, a majority of the reflected signals have path posed of two types of errors: linear errors with respect to the
lengths smaller than 15m or 7.5m. Therefore, the obtained first subcarrier indexes and the constant errors across subcarriers.
arriving signal based on each channel is still a mixture of the Our system calibrates the linear phase errors by searching
signals that travel through the fruit and multipath. Therefore, for an optimum phase compensation value that minimizes
simply performing IFFT on the signal measurements at each the differences in the derived power delay profiles across
WiFi channel provides insufficient time or distance resolution multiple channels. The constant phase errors and amplitude
for extracting the signal traveling through the fruit. errors could be mitigated by averaging the measurements of
As WiFi spans multiple channels at both 2.4GHz and 5GHz, multiple packets that received in the coherence time window.
we propose to probe the fruit at all available channels of Given the calibrated CSI measurements, our system per-
5GHz with larger bandwidth. Combining all the channels at forms multipath removal for extracting the signal that directly
5GHz (i.e., from 5.18GHz to 5.825GHz) brings over 600MHz travels through the fruit. Our system stitches the CSI measure-
bandwidth, which corresponds to a 1.5ns time resolution or ments of all the channels together to enlarge the bandwidth
to a 0.45 meters distance resolution. Such a high resolution for improving the resolution of the power delay profile. The
is sufficient for us to separate the signal traveling through the inverse NDFT technique is used to overcome the problem of
fruit from the reflected ones, as shown in Figure 2. Given the unequal and non-contiguity spacing of 5GHz channels. Our
the signal measurements at each channel, we then stitch these system then identifies and extracts the signal directly travelling
measurements together to derive a fine-grained power delay through the fruit for ripeness detection.
profile for multipath removal. At last, our system identifies the fruit ripeness level by
One problem with that stitching lies in the fact that measuring the similarity between the extracted features and
the usable WiFi channels at 5GHz are unequally and non- the pre-built ripeness profiles. The MODWT is used to extract
contiguous spaced due to FCC regulation. For example, the features over multiple channels and the cross correlation is
channels from 120 to 128 are partially occupied by the used to compare the features against the ripeness profiles in
weather radar usage in the US, and different countries apply the library. The testing fruit is identified as one of the following
their own regulations. Venders usually disable some of the ripeness levels: unripen, half ripen, ripen and over ripen.
5GHz channels in compliance with the regulations of different
countries before shipping the WiFi NICs. We thus cannot B. CFR Sampling and Data Calibration
simply use IFFT, which only works for uniformly-spaced With 802.11n/ac systems, the WiFi NICs track fine-grained
frequency measurements. To overcome this issue, we adopt channel state information, which is a sampled version of
the inverse non-uniform Discrete Fourier Transform (NDFT), the channel response including both phase and amplitude
which is capable of deriving a fine-grained power delay profile information. On the standard 20MHz WiFi channel, it mea-
from non-uniformly spaced channels at 5GHz. sures the amplitude and phase for each of the 56 OFDM
Another challenge lies in the feature extraction process for subcarriers. With wider 40MHz channels, CSI measurements
ripeness level identification. This is because various uncer- are available for 128 subcarriers. In our work, we utilize all
tainties are introduced by the process of solving an under available 20MHz WiFi channels at 5GHz. To ensure that the
determined system with inverse NDFT across different sam- channel hopping goes through all the channels within the
ples. Such uncertainties could result in inconsistent features coherence time, the hopping delay is set as 0.25ms. Note that
 |csi1,1| ‫ ס‬csi1,1 |csi1|

|csi2,1| Amplitude Ripeness Level

Errors
‫ ס‬csi2,1
|csiq| W3 V3
Residual Fruit Ripeness
|csi1,q| ‫ ס‬csi1,q Synchronization ‫ ס‬csi1 W2 W1
Stage Profiles
Errors spectrum

Channel Frequency Similarity

|csi2,q| ‫ ס‬csi2,q CSI Measurement
Multipath Removal
Response Sampling Calibration ‫ ס‬csiq Feature Extraction Matching

Fig. 3. System Overview.

the coherence time in typical indoor environment is around In particular, it can be formulated as an optimization problem
300ms [9]. This allows us to collect multiple packets at each as following:
channel within the coherence time. Q
0 0 0
In particular, we denote csip,q as the CSI complex vector
X
min ||gq (ϕs ) − gq0 (ϕs )||2 , q 6= q , (6)
sampled at the pth packet of the q th channel. Each CSI 0
ϕs
q,q 0 =1
complex vector in frequency domain can be represented as:
0
where gq (ϕs ) denotes the derived power delay profile at
csip,q = [csi1p,q , csi2p,q , · · · , csiK
p,q ], (3) the qth channel and Q is the total number of channels. By
0
where K is the total number of subcarriers reported at each obtaining an optimum value of ϕs , we are able to remove the
channel per packet (i.e., K = 56) and csikp,q is the sampled linear phase error ϕs from the CSI phase measurements.
channel response at the k th subcarrier. Figure 5(a) shows the phases across different channels after
Since the CSI measurements are extracted by sampling calibrating ϕs . As shown in Figure 5(a), after calibration, the
the channel frequency response, the raw CSI measurements phases of the overlapped subcarriers at three different 2.4GHz
incur significant distortions due to the hardware limitations band channels now demonstrate similarity and consistency.
of off-the-shelf WiFi NICs. Thus the reported CSI phase Thus we can pick either channel as reference to stitch them
measurement 6 csikp,q can be further represented as: together. Figure 5(b) shows three non-overlapping channels at
5GHz band after phase calibration.
6 csikp,q = 6 hkp,q + kϕl + ϕc , (4)
C. Multipath Removal
where 6 hkp,q represents the phase rotation due to signal Multipath removal leverages a fine-grained power delay
propagation, ϕl denotes the slope of the linear phase error profile to extract the signal that directly travels through the
and ϕc is the constant phase shift error. fruit for ripeness detection. Since a power delay profile derived
As our system relies on the CSI measurements to derive from one single WiFi channel provides insufficient time delay
fine-grained power delay profile for multipath removal, we resolution, our system stitches all the available channels at
need to correct errors in the CSI measurements. First, the am- 5GHz together to improve the resolution for extracting the
plitude error is discovered to follow Gaussian distribution [15], signal that travels through the shortest path.
it could be mitigated by averaging CSI measurements from Due to the regulations at many countries, the usable chan-
multiple packets that within the coherence time [32]. The nels at 5GHz are unequally and non-contiguous spaced. For
constant phase error ϕc , is caused by the residual central example, the usable channels at 5GHz on the Atheros NICs
frequency offset. We select a reference channel from all the are separated into four segments. The venders disabled some
available channels and mitigate it through correcting the phase channels in compliance with the local regulations. We thus
difference from each channel. The linear phase error ϕl is a cannot simply use IFFT for power delay profile derivation,
frequency dependent error and can be further divided into two as it only works for uniformly-spaced frequency measure-
components: ϕl = ϕd + ϕs . The phase error ϕd is observed to ments. Instead, we adopt inverse Non-uniform Discrete Fourier
follow the Gaussian distribution with zero mean [15]. Thus can Transform (NDFT), which works for non-uniformly spaced
be mitigated by averaging multiple CSI phase measurements channels [28]. For fine-grained power profile derivation, given
collected at each channel [32]. the calibrated CSI measurements from last step as:
The second component ϕs is introduced by the residual
CSI = [CSI
d1 , ..., CSI
dq , ...], (7)
sampling frequency offset [15]. After removing phase errors
ϕc and ϕd and assuming the phase compensation for ϕs is where q denotes the qth channel, we can formulate the inverse
0
ϕs , Equation 4 can be rewritten as: NDFT as following:
0
6 csikp,q = 6 hkp,q + kϕs − kϕs .
d
(5) min ||CSI − Fg||2 , (8)
g
0
By gradually changing the value of ϕs , different power delay where g represents the power delay profile that we search for,
profiles can be derived for each channel. The optimum phase F is Fourier matrix. The goal is to find an optimum solution
compensation ϕs can be found when the differences in derived of g that minimizes the difference between CSI and Fourier
power delay profiles across multiple channels are minimized. Transform of g.
 55 40 50

Amplitude(db)

Amplitude(db)
Power(db)
50
45
Channel 136 20
45 Channel 140
Channel 149 40
Channel 153
40 0
0 100 200 300 400 0 5 10 15 20 25 0 100 200 300 400
Subcarrier Index Time Delay(ns) Subcarrier Index

(a) Before multipath removal (b) Derived power delay profile (c) After multipath removal
Fig. 4. The CSI amplitude for four channels with gap at 5GHz before and after multipath removal.
1 0.5
affected by fruit. Next we need to extract features to identify
Phase(radian)

Phase(radian)
Channel 1 Channel 40
Channel 3 0 Channel 44

0
Channel 5
-0.5
Channel 48 the ripeness level by comparing them to the existing profile.
-1 Intuitively, this can be done by adopting varies techniques from
-1 -1.5
0 20 40 60 80 100 120 0 50 100 150 200 existing works. However, the frequency domain signal from
Subcarrier Index Subcarrier Index
previous step were calculated using NDFT which introduces
(a) 2.4GHz (b) 5GHz
various uncertainty across different signal samples. This is
Fig. 5. Calibrated CSI phases for three overlapping channels at 2.4GHz and
three channels at 5GHz.
caused by the nature of using proximal gradient method
to solve under determined system problem. Since various
Such an optimization problem can be viewed as an under- uncertainty over 600MHz bandwidth signal across different
determined system, which yields several possible solutions. To samples would lead to inconsistent feature extraction. This
pick the best one, we need to add constraints to filter out the might compromise the ripeness level identification process.
less desired ones. To find constraints, we look into the char- To overcome such problem, our system thus adopts Maxi-
acteristic of signal propagation in indoor environment. Based mal Overlap Discrete Wavelet Transform (MODWT) [23] to
on previous observations [28], although multiple paths exist in extract features based on the signals that travel through fruit
typical indoor environment, only a few paths would dominate at multiple frequencies (i.e., spectrum). First, MODWT is
the signal propagation. It is because they travel through shorter shift invariant, which means the decomposition outputs stay
paths and suffer less attenuations when comparing to longer invariant on different starting and ending points. This enables
ones. We thus add one constraint to the inverse NDFT: among us to retrieve consistent features across a large number of
all the solutions of g that satisfy Equation 8, our system favors samples with various uncertainty. Second, the redundancy of
the g with fewer dominating propagation paths. MODWT facilitates detail preservation of the original sample
To solve the Equation 8 with the constraint, we adopt the for multiple level decomposition process.
proximal gradient method that used to solve convex optimiza- In particular, MODWT analyzes the signals in both time and
tion problem [13]. In particular, our system takes CSI as input frequency domains by decomposing signals into successive
and computes the gradient of differentiable term in Equation 8. approximation coefficients along with detailed coefficients.
After obtaining several solutions, our system selects the one The approximation coefficients depict the large scale char-
that with fewer dominating paths. Given the derived fine- acteristic of change pattern, whereas the detailed coefficients
grained power delay profile, we remove the components from capture small scale components that represent the fine details
multipath propagations and only keep the component that goes of the changes. Given a signal X(i), each level of MODWT
through the fruit directly. Then, we covert the trimmed power coefficients are computed based on the following equations:
delay profile back to frequency domain for feature extraction. X
Figure 4 shows the amplitude of the channels cover the Wj,i = l = 0L−1 hj,l Xi−lmodN ,
(9)
same bandwidth before and after multipath removal. As shown
X
Vj,i = l = 0L−1 gj,l Xi−lmodN ,
in Figure 4(a), four channels at 5GHz band cover 100MHz
bandwidth with a 40MHz gap in between as highlighted by a where Vj,i is the approximation coefficient and Wj,i is the
black dot rectangle. After performing inverse NDFT, we are detailed coefficient. Here hj,l and gj,l are the wavelet filter
able to derive a fine-grained power delay profile shown in and scaling filter respectively. j is the level of decomposition
Figure 4(b), which indicated the lack of LoS. Then we remove and l = 1...L is the length of the filter. By applying MODWT
the multipath components from the power delay profile, only to the extracted signal at each channel recursively for 3 levels,
reserve the signal propagation affect by fruit and convert it we obtain both large scale and detailed features.
back to frequency domain. The resulting spectrum is shown To identify the degree of ripeness, we utilize cross correla-
in Figure 4(c). We observe that the amplitude continues to tion to calculate the similarity between the extracted features
decrease with the increasing frequency. It is consistent with and the ones in the profile library. It measures the similarity
Equation 2, and shows that a higher frequency would suffer a based on the relative changes and is independent of translations
larger attenuation when propagating through the fruit. and scaling in the amplitude. This is done repeated over 3
levels of approximation and detailed features and combined
D. Feature Extraction and Ripeness Identification using empirically assigned weights. The ripeness level with
After multipath removal process, we obtained the signal the profile in the library that has the highest similarity with
in frequency domain over 600MHz bandwidth that mainly the testing fruit is then identified as the detected ripeness level.
 8’

3
Fruit Kiwi Avocado
2
Ripeness Vendor 1 Vendor 2 Vendor 1 Vendor 2
7’
Unripen 48 32 45 35 16’
Half Ripen 43 31 43 33
Ripen 44 32 40 34 1
Over Ripen 42 30 44 34 Personnel
WiFi Device
Fruit
Fig. 6. Volumes of fruits in our evaluation. Location
13’

IV. P ERFORMANCE E VALUATION

In this section, we describe our experimental setup and eval- Fig. 7. Illustration of experiment setup.
uate the performance of FruitSense in detecting the ripeness Unripen 0.88 0.06 0.03 0.03 Unripen 0.87 0.06 0.03 0.03

level of Kiwi fruit and Avocado under different environments. Half Ripen 0.03 0.92 0.03 0.02 Half Ripen 0.03 0.93 0.02 0.01

Ripen 0.01 0.04 0.93 0.02 Ripen 0.01 0.04 0.94 0.01

A. Experiment Setup Over Ripen 0.02 0.05 0.06 0.87 Over Ripen 0.02 0.04 0.05 0.89

Unripen Half Ripen Ripen Over Ripen Unripen Half Ripen Ripen Over Ripen
We conduct experiments with two laptops (i.e., Dell LAT-
ITUDE E5540), each connecting with an external antenna. (a) Kiwi (b) Avocado
The distance between two external antennas is about 20cm. Fig. 8. Confusion matrix of ripeness level detection for both kiwi and avocado
The testing fruit is placed in the middle of the two antennas, fruits.
thus blocking the line-of-sight propagation. Both laptops run each ripeness level and each type of fruit, we have around
Ubuntu 12.04 LTS and are equipped with the WiFi NICs of 75 fruits under testing. Figure 6 shows the detailed break
Atheros AR9580 for extracting CSI measurements [32]. The down for each ripeness stage, vender, and type of fruit. To
transmitter and receiver hop through all available 20MHz WiFi build the fruit ripeness profiles, we pre-select 10 fruits at each
channels at 5GHz bands in an 802.11n network. There are ripeness level with various fruit sizes and average the features
total 21 available channels enabled by the Atheros AR9580 extracted from these fruits. All fruits are tested under the room
card. And they fall into four non-contiguous segments. The temperature(23◦ C to 26◦ C).
first segment is from 5.18GHz to 5.32 GHz (i.e., the channels The experiments are conducted in two rooms at three
from 36 to 64), whereas the second segment is from 5.5GHz locations, representing three different multipath environments.
to 5.58 GHz (i.e., the channels from 100 to 116). The third Figure 7 shows the layout of two rooms (i.e., one living room
one and forth one are from 5.66GHz to 5.7 GHz (i.e., the and one bedroom) and three locations. The bedroom has the
channels from 132 to 140) and from 5.745GHz to 5.825GHz size of 7 ft by 8 ft with one bed, one pair of table and chair.
(i.e., the channels from 149 to 165), respectively. For the living room, it is 16 ft by 13 ft with regular living room
Although the entire 5GHz has the bandwidth over 600MHz, furniture setup, such as dining table, book shelf, sofa, and TV.
we can only obtain 420MHz bandwidth from these four The bedroom environment represents a more compact space
segments. We thus need to use inverse NDFT to derive filled with furniture, while the living room setup describes
the power delay profile for over 600MHz bandwidth based a typical home environment with a larger space. To test the
on unequally and non-contiguous spaced 21 channels. The robustness of our system to the environment changes, we
channel hopping delay is set as 0.25ms. As the coherence experiment with people walking around during the ripeness
time in typical indoor environment is about several hundreds detection. In particular, during the data collection, a person
milliseconds [9], we can collect packets across channels within is walking around in the room to create interferences. The
coherence time as well as obtain multiple packets at each walking trajectory is shown in dash curve in Figure 7.
channel within coherence time. For each packet, we extract We use confusion matrix and detection accuracy to evaluate
CSI for 56 subcarriers, which are equally distributed in a our system performance. For confusion matrix, each column
20MHz channel. represents the fruit ripeness level that was classified by our
We experiment with two types of commonly consumed system and each row shows the ground truth of the ripeness
fruits: kiwi fruit and avocado. The specific cultivars of avocado level. Each cell in the matrix corresponds to the fraction of
and kiwi fruit are Hass and Fuzzy respectively. Each type ripeness level in the row that was classified as the ripeness
of fruit samples are purchased from two different vendors. level in the column. The detection accuracy is the percentage
Particularly, the production locations for kiwi fruits are USA of the fruit that is correctly identified by our system.
and New Zealand for vender 1 and vender 2 respectively,
whereas for avocado they are Mexico and USA for vender B. Overall Performance
1 and vender 2, respectively. The purchased fruits have four Figure 8 shows the confusion matrix of fruit ripeness
ripeness levels as commonly adopted by fruit industry: un- detection for both kiwi and avocado. We observe that for both
ripen, half ripen, ripen and over ripen [24], [31]. We use a fruits, our system achieves overall detection accuracy over
spectrometer to log the ground truth of the fruit ripeness. For 90%. In particular, the overall detection accuracy for avocado
 Detection Accuracy
Detection Accuray 1 3 1 5

Improvement(%)
Improvement(%)
0.8 0.8 4
2
0.6 0.6 3
0.4 0.4 2
1
0.2 Kiwi Kiwi 0.2 Kiwi 1 Kiwi
Avocado Avocado Avocado Avocado
0 0 0 0
Unripen Half Ripen Ripen Over Ripen Unripen Half Ripen Ripen Over Ripen Unripen Half Ripen Ripen Over Ripen Unripen Half Ripen Ripen Over Ripen

(a) Overall Accuracy (b) Improvement (a) Overall accuracy (b) Improvement
Fig. 9. Detection accuracy and improvement after separating the small and Fig. 11. Detection accuracy and improvement after separating fruit based on
the large size fruits. producing location.
we further perform a more detailed study on whether the per-
Detection Accuracy

1 1

Detection Accuracy
0.8 0.8
formance changes when using the ripeness profile built at one
0.6 0.6
0.4 0.4 multipath environment to test the fruits at a different multipath
0.2 2nd Environment 0.2 2nd Environment environment. Specifically, we build the ripeness profiles when
3rd Environment 3rd Environment
0 0
Unripen Half Ripen Ripen Over Ripen Unripen Half Ripen Ripen Over Ripen the fruits are at the location 1 in Figure 7. We then use such
(a) Kiwi (b) Avocado profiles to detect the fruits that placed at the location 2 (in the
Fig. 10. Detection accuracy under two multipath environments when using same room), and location 3 (in a different room). Figure 10(a)
the ripeness profiles built under a different multipath environment. and Figure 10(b) present the performance of ripeness detection
is about 91%, whereas it is 90% for kiwi. Moreover, the half for kiwi and avocado, respectively. We observe that our system
ripen and ripen fruit detection, compared to unripen and over does not suffer from performance degradation although the
ripen fruit detection, has a higher detection accuracy for both training and testing environments are under different multipath
kiwi and avocado. Specifically, ripen fruit detection achieves propagation. In particular, the overall detection accuracy still
93% and 94% accuracy for kiwi and avocado respectively. This maintains at around 90% for both kiwi and avocado. These
is due to the fact that the physiological changes at unripen and results further demonstrate the effectiveness of the multipath
over ripen levels are at slower pace, whereas the changes at removal that leverages the larger bandwidth at 5GHz. It shows
half ripen and ripen levels go through faster processes [24]. that our fruit ripeness detection system is location independent
Thus, more physiological changes of fruit at half ripen and and environment independent. Once the profile is built, it can
ripen stages could be captured for ripeness detection. The be applied to different environments without profile updating
above results show that our system could provide high ac- or calibration.
curacy in detecting fruit ripeness by using single pair of WiFi
E. Impact of Fruit Producing Location
devices. The results also show that our system works with
different multipath environments without location specific or We next study the impact of production location of the fruit
environment specific calibration for ripeness profile. Still, the by separating each type of fruits based on their producing
performance could be potentially improved by using additional area. In particular, we build the ripeness profiles based on the
pairs of WiFi devices or with multiple antennas. producing area of each type of fruits. Then the ripeness profiles
are used to test the fruits produced in the same area. The two
C. Impact of Fruit Size producing areas for kiwi is USA and New Zealand, whereas
We next evaluate the performance when the testing fruits they are USA and Mexico for avocado. The results are shown
have similar sizes as that of the fruits used to build ripeness in Figure 11. We find that the detection accuracy has obvious
profiles. In particular, the size of kiwi varies from 55mm to improvement for both kiwi and avocado when compared to
63mm, whereas it is from 69mm to 85mm for avocado. We separating the fruits based on their sizes. Specifically, the
classify each type of fruit into two categories: small fruits overall accuracy is close to 94% for kiwi and at about 93%
and large fruits. Then, we sense the ripeness of small (large) for avocado. The results show that the producing area of fruits
fruits with the profiles built from the small (large) size fruits. plays an important role when building the profile, even for
Figure 9 shows the the overall accuracy for separating small the same type of fruit. This is because different producing
and large size fruits and corresponding improvement with areas result in different percentage of dry matter and moisture
respect to mixed size fruits. We find that our system achieves content, which is also discovered by existing work [21]. It
slightly better performance at each ripeness level detection suggests that a more fine-grained profile based on producing
for both kiwi and avocado. In particular, it achieves an overall areas could improve the ripeness detection accuracy.
accuracy at around 92% for both kiwi and avocado. The overall
V. R ELATED W ORK
improvement is at around 2% when comparing to that of the
mixed fruit case. The results show that our system is not very There have been wide ranges of work utilizing specialized
sensitive to the fruit size, although a more fine-grained fruit or commodity RF devices to perform human sensing. For
profile could slightly improve the accuracy. example, E-Eyes [30] is able to track daily activities and
WiSee [25] uses specialized RF device to sense whole home
D. Detailed Study on Multipath Environments gestures. Moreover, WiKey [5] has the ability to sense typing
Although our overall performance in Section IV-B shows on keyboard, whereas system proposed by Liu et al. [19] is
that our system works for different multipath environments, able to sense heartbeats and breathing. Besides motion and
activity tracking, many RF based localization systems have [2] Scio. https://www.consumerphysics.com/. Accessed: 2017-03-29.
been proposed. For example, Chronos [28] achieves decimeter- [3] S. Adhimantoro and F. L. Gaol. Application of ultrasonic and fuzzy
logic to determine fruit maturity level. IJCAS, 7(1):27–38, 2014.
level localization with a single WiFi AP, and WiTrack [4] [4] F. Adib, Z. Kabelac, and D. Katabi. Multi-person localization via rf
utilizes RF body reflections to enable localization. body reflections. In NSDI, pages 279–292, 2015.
[5] K. Ali, A. X. Liu, W. Wang, and M. Shahzad. Keystroke recognition
In general, the approaches for fruit ripeness detection can using wifi signals. In ACM MobiCom, 2015.
be divided into two categories: destructive and non-destructive [6] P. Barreiro et al. Mealiness assessment in apples and peaches using mri
methods. By measuring the sugar to acid ratio or total content techniques. Magnetic Resonance Imaging, 18(9):1175–1181, 2000.
[7] R. G. Buttery, R. Teranishi, R. A. Flath, and L. C. Ling. Fresh tomato
of sugar or acid with refractometer, Jones and Scott [16] are volatiles. ACS Publications, 1989.
able to track the ripeness level. Similar methods such as gas [8] A. J. Das, A. Wahi, I. Kothari, and R. Raskar. Ultra-portable, wire-
chromatograph [7] and mass spectrometer [26] can also be less smartphone spectrometer for rapid, non-destructive testing of fruit
ripeness. Scientific Reports, 6, 2016.
used to further analyze the chemical compounds of different [9] A. Goldsmith. Wireless communications. Cambridge university, 2005.
fruits to identify different ripeness levels. Such methods how- [10] D. Gunders. Wasted: How america is losing up to 40 percent of its food
ever are destructive and compromise the intactness of sample from farm to fork to landfill. Natural Resources Defense Council, pages
1–26, 2012.
and prevent it from future consumption. [11] F. Harker et al. Penetrometer measurement of apple and kiwifruit
For non-destructive methods, Peirs et al. [22] use NIR firmness: operator and instrument differences. JASHS, 1996.
spectroscopy for post-harvest quality evaluation of apples. [12] R. Harrill. Using a refractometer to test the quality of fruits and
vegetables. P. PUBLISHING, Éd.) Consulté le July, 20:2010, 1998.
Imaging techniques such as X-ray computed tomography [14] [13] K. Hou et al. On the linear convergence of the proximal gradient method
and magnetic resonance imaging [6] have also been shown for trace norm regularization. In NIPS, pages 710–718, 2013.
to be effective in fruit quality tracking. Moreover, ultrasonic [14] J.-A. Jiang et al. An adaptive image segmentation algorithm for x-ray
quarantine inspection of selected fruits. CEA, 60(2):190–200, 2008.
signals have also been utilized to determine fruit matu- [15] V. P. G. Jimenez, M.-G. Garcia, F. G. Serrano, and A. G. Armada.
rity level [3]. The above approaches though provide non- Design and implementation of synchronization and agc for ofdm-based
destructive measuring, usually require expensive laboratory wlan receivers. IEEE Transactions on Consumer Electronics, 2004.
[16] R. Jones and S. Scott. Improvement of tomato flavor by genetically
equipments. Recently, Das et al. [8] use portable spectrometer increasing sugar and acid contents. Euphytica, 32(3):845–855, 1983.
that senses the UV fluorescence of Chlorophyll for ripeness. [17] J. Juansah, I. Budiastra, K. Dahlan, and B. Seminar. Electrical behavior
Commercial devices such as SCiO [2] and Changhong H2 [1] of garut citrus fruits during ripening changes in resistance and capaci-
tance models of internal fruits. IJET-IJENS, 12(4):1–8, 2012.
can also achieve similar functionality using portable optical [18] V. Komarov, S. Wang, and J. Tang. Permittivity and measurements.
sensors. However, there are non-negligible costs incurred in Encyclopedia of RF and microwave engineering, 2005.
purchasing dedicated spectral sensors. [19] J. Liu, Y. Wang, Y. Chen, J. Yang, and X. Chen. Tracking vital signs
during sleep leveraging off-the-shelf wifi. In ACM MobiHoc, 2015.
VI. C ONCLUSION [20] L. S. Magwaza and S. Z. Tesfay. A review of destructive and non-
destructive methods for determining avocado fruit maturity. Food and
This paper presents FruitSense, which is capable of sensing bioprocess technology, 8(10):1995–2011, 2015.
the ripeness of fruit by analyzing wireless signals traveling [21] F. Ozdemir and A. Topuz. Changes in dry matter, oil content and fatty
through the fruit. The proposed system, FruitSense, is non- acids composition of avocado during harvesting time and post-harvesting
ripening period. Food Chemistry, 86(1):79–83, 2004.
destructive and only requires a pair of off-the-shelf WiFi [22] A. Peirs et al. Phpostharvest technology: Comparison of fourier
devices. The insight is that the signals traveling through the transform and dispersive near-infrared reflectance spectroscopy for apple
fruit could capture the physiological changes associated with quality measurements. Biosystems Engineering, 81(3):305–311, 2002.
[23] D. B. Percival and A. T. Walden. Wavelet methods for time series
fruit ripening. Experimental results under different multipath analysis, volume 4. Cambridge university press, 2006.
environments show that FruitSense can detect the ripeness lev- [24] V. Prasanna, T. Prabha, and R. Tharanathan. Fruit ripening phenomena–
els of both kiwi fruit and avocado with an accuracy over 90%. an overview. Critical reviews in food science and nutrition, 2007.
[25] Q. Pu, S. Gupta, S. Gollakota, and S. Patel. Whole-home gesture
FruitSense builds on the growing interest in using wireless recognition using wireless signals. In ACM MobiCom, 2013.
signals for human sensing. The proposed work further expands [26] P. Schreier, F. Drawert, and A. Junker. Identification of volatile
the scope of human sensing to sensing the bio-information of constituents from grapes. JAFC, 24(2):331–336, 1976.
[27] S. Tan and J. Yang. Wifinger: leveraging commodity wifi for fine-grained
fruit crops. We believe the implication of the work could be finger gesture recognition. In ACM MobiHoc, 2016.
extended beyond classifying the fruit ripeness, particularly in [28] D. Vasisht, S. Kumar, and D. Katabi. Decimeter-level localization with
quantifying the physiological compounds of fruit with more a single wifi access point. In USENIX NSDI, 2016.
[29] C. Wang, X. Zheng, Y. J. Chen, and J. Yang. Locating rogue access point
advanced signal process and machine learning techniques. using fine-grained channel information. IEEE Transactions on Mobile
Computing, 16(9):2560–2573, 2017.
VII. ACKNOWLEDGEMENTS [30] Y. Wang et al. E-eyes: device-free location-oriented activity identifica-
This work was partially supported by the National Science tion using fine-grained wifi signatures. In ACM MobiCom, 2014.
[31] R. Wills and J. Golding. Postharvest: an introduction to the physiology
Foundation Grands CNS-1514238, CNS-1816265, and CNS- and handling of fruit and vegetables. UNSW press, 2016.
1505175. [32] Y. Xie, Z. Li, and M. Li. Precise power delay profiling with commodity
wifi. In ACM MobiCom, 2015.
R EFERENCES
[1] Changhong. https://liliputing.com/2017/01/changhong-h2-smartphone-
built-spectrometer.html, note = Accessed: 2017-03-29.