2024 The 8th International Conference on Green Energy and Applications

Online Data Acquisition and Storage for Rooftop

Solar System Utilizing Time Series Databases and
Cloud-based Visualization Platforms
1st Son Tran-Thanh 2nd Duong Nguyen-Dang
Department of Electrical Engineering Department of Electrical Engineering
School of Electrical and Electronic Engineering, HUST School of Electrical and Electronic Engineering, HUST
Hanoi 100000, Vietnam Hanoi 100000, Vietnam
2024 8th International Conference on Green Energy and Applications (ICGEA) | 979-8-3503-4900-9/24/$31.00 ©2024 IEEE | DOI: 10.1109/ICGEA60749.2024.10560782

son.tranthanh@hust.edu.vn duong.nd191478@sis.hust.edu.vn

3rd Anh Nguyen-Hoang 4th Tuyen Nguyen-Duc*

Department of Electrical Engineering Department of Electrical Engineering
School of Electrical and Electronic Engineering, HUST School of Electrical and Electronic Engineering, HUST
Hanoi 100000, Vietnam Hanoi 100000, Vietnam
anh.nh210071@sis.hust.edu.vn tuyen.nguyenduc@hust.edu.vn

Abstract—Renewable energy sources, especially solar PV, are A Supervisory Control and Data Acquisition (SCADA) sys-
emerging as one of the most promising solutions to combat the tem can vary in complexity and cost, from very sophisticated
global warming phenomenon and climate change. However, solar and costly to more affordable and simple. Two examples of
PV systems often require proper supervision due to their depen-
dence on various environmental factors. In practice, supervisory complex SCADA systems are the National Solar Radiation
control and data acquisition (SCADA) systems have been widely Database (NSRDB) [6] and the solar radiation database for
adopted to capture PV system parameters and store them for Chile [7]. These are both large-scale collections of solar
long-term accessibility. This study proposes a comprehensive so- radiation data, and both use satellites in the data acquisition
lution for monitoring rooftop PV systems, including data logging, process, with the exported data acquired every one-hour in-
storage, and backup. The authors also provide a cloud-based
service to access the database and visualize measurements in real- terval. The data sources covered a vast region but were not
time as our main contributions. The conclusions suggest future detailed enough for a small-scale PV project (the NSRDB
improvements and recommend several practical applications had a 4 × 4 km resolution, and the Chile system had a 1-km
utilizing our resources. grid), especially when small and short-lived clouds crossed the
Index Terms—SCADA system, solar power system, time series system. With the NSRDB, errors with each measured point still
database, cloud-based storage, real-time visualization
vary from 5% to 10%. Chile’s system still has cloud detection
errors when the solar zenith angle is higher than 70 degrees.
I. I NTRODUCTION On the other hand, Praijwal mentioned a wireless method for
data acquisition for a specific PV system with an affordable
The annual surge in electricity demand is prompting the price [8]. The setup was tested for 8 hours to obtain data
need for increased power generation. However, relying on every 15-minute interval. However, even though the delay in
traditional fossil fuels like coal, oil, and natural gas to expand the program was made uniform, there was delayed time in
energy production would exacerbate pollution and deplete data received due to the internet speed.
precious natural resources. Consequently, renewable energy Meteorological data, which reflects solar PV system’s poten-
sources have emerged as a prominent solution for providing tial, can be gathered in various ways. Historical data monitored
clean and sustainable power [1], [2]. Solar energy is prominent by third parties is valuable for estimating the performance
among the types of renewable energy, which contributes of PV systems. Open-access data can be advantageous for
many benefits, including inexhaustible, considerable energy investors and researchers with limited resources, enabling them
emitted from the sun [3]. With the expansion of solar energy to make preliminary project assessments without financial
development, meteorological forecasting, measurement, and constraints. However, the uncertainties associated with PV
management of solar energy systems are essential due to systems pose many challenges for power system operators
the significant impact of environmental conditions on the with load dispatch and system control problems. Data collected
performance of PV panels. In addition, identifying errors from the past is meaningful primarily for potential assessment
becomes more straightforward when the system’s technical and planning purposes rather than real-time operation. On-
parameters are regularly monitored [4], [5]. site data monitoring, which gives information of PV systems

979-8-3503-4900-9/24/$31.00 ©2024 IEEE 302

Authorized licensed use limited to: SHIBAURA INSTITUTE OF TECHNOLOGY. Downloaded on February 18,2025 at 10:13:52 UTC from IEEE Xplore. Restrictions apply.
 in real-time, is therefore essential for PV forecasting and collect solar irradiance, air temperature, and air humidity on
optimizing system operations. Furthermore, the collected data the rooftop of the building and pasted adhesive-type sensors to
can be visualized for user observations and saved to the the backplate of PV arrays to record their temperature. These
cloud storage for online sharing purposes. In this study, the sensors use a serial communication protocol called Modbus
authors utilize a time series database (TSDB) and cloud- RTU, which is widely used in industrial communication net-
based visualization platform to record and store data for a works. Hardware requirements for Modbus RTU networks
3-kW rooftop solar system. The primary contributions of our commonly include a master device, slave devices, and cables
work encompass a real-time data observation website and to connect them. The signal transmitted in the Modbus RTU
daily updated high-resolution data sources for solar power network is binary, allowing for the exchange of information
research, all made available with open-access permission. The between devices. Modbus RTU data frame, which is given in
structure of this paper is as follows: Section II explains the Fig. 3 includes:
principles of the on-site setups, including sensors, gateway, • Slave Address: The address of the target slave device.
server, and their interaction. Section III provides visualization • Function Code: The type of operation the master device
and data storage solutions in detail, which even served as an wants to perform on the slave device, such as reading
open-access resource for internet users. Section IV proves the registers or writing on them.
correctness of collected data by comparing the actual power • Data: Specific data or parameters related to the function
output of the PV system with a mathematical model and code.
makes a related discussion. Section V is the conclusion that • CRC Check: Value used for error detection and correc-
summarizes the work and recommends future progress. tion.
• Start and Stop Bits: The beginning and end of each
II. O N - SITE M ONITORING S YSTEM
character within the frame and help with framing and
A. Monitored Solar Power System
synchronization.
The rooftop solar system under examination, depicted in Modbus RTU systems operate based on the master-slave
Fig. 1, is located atop the D9 Building at Hanoi University principle. Accordingly, the master device initiates communi-
of Science and Technology (HUST), Hanoi, Vietnam. The cation by sending a request frame to the slave device with
PV array is constructed by 8 PV panels, each having a peak a specific address, also called slave ID. The slave device
power rating of 480 kW. The configuration of the PV array processes the request by taking required data from its registers
is series-parallel, including two strings connected in parallel, and prepares a response frame transmitted back to the master
with each string consisting of four panels connected in series. device over a wired-connection, commonly twisted pair cables.
The energy generated by the PV array is subsequently fed It is important to note that every sensor within a system must
into the power system via a DC-AC grid-connected inverter, be assigned a unique slave ID, which is initially written to
which makes the solar system operate at its full potential under their respective registers by developers.
various environmental conditions. With the integration of the Our meteorological sensors establish communication with
maximum power point tracking algorithm into the inverter, it the master, or gateway, over the 2-wire RS485 network de-
is popularly confirmed that the output power of the PV array picted in Fig. 4. RS485 is actually a physical layer standard,
is approximately proportional to solar irradiance. However, and it does not define a specific protocol. Communication
intense sunlight can negatively heat the panels, leading to a protocols like Modbus RTU and others are often used on top
slight reduction in output power. of the RS485 physical layer to define the structure of data
packets.
The gateway allows Modbus RTU devices to connect
with Modbus TCP/IP networks over ethernet cables. Dell
PowerEdge R630 Windows 11-based desktop server has the
specification of Intel Xeon CPU E5-2696 with 18 cores
and 36 threads, 64GB ECC-RAM, and two 1TB hard disk
drives running RAID technology. Both desktop server and
gateway possess their respective IP addresses, for instance,
192.168.1.xxx, enabling them to interact and exchange data.
Modbus TCP/IP is better suited for modern ethernet-based
networks focusing on scalability that allows sharing data
globally, while Modbus RTU is often used in local connections
focusing on speed and reliability.
Fig. 1. Solar power system on the roof of D9 Building, HUST. The DC/AC inverter is equipped with internal sensors to
self-monitor its performance. Additionally, the manufacturer
B. Hardware Setups and Communication Methods has included an RS232 port on the inverter for external data
The general configuration of the SCADA system is illus- sharing. In contrast to RS485 networks which allow multiple
trated in Fig. 2. The authors set up meteorological sensors to devices to operate simultaneously, the RS232 standard only

303
Authorized licensed use limited to: SHIBAURA INSTITUTE OF TECHNOLOGY. Downloaded on February 18,2025 at 10:13:52 UTC from IEEE Xplore. Restrictions apply.
 Fig. 2. General configuration of the SCADA system

Start Address Function code Data CRC check Stop

TABLE I
S PECIFICATIONS OF SENSORS
8 bits 8 bits 8n bits 16 bits

Fig. 3. Modbus RTU frame Data Slave ID Function code Register address Unit
Solar irradiance 1 Holding (4x) 0000 W/m2
Air temperature Holding (4x) 0003 0.01 o C
Slave 1 Slave 2 Slave n 2
Air humidity Holding (4x) 0004 0.01 %
PV array temperature 3, 4 Holding (4x) 0000 0.1 o C
Data (B)+

Data (B)+

Data (B)+

…
Data (A)-

Data (A)-

Data (A)-
GND

GND

GND

DC power Input (3x) 0002 0.1 W

AC power Input (3x) 0012 0.1 W

Data (B)+
Grid frequency Input (3x) 0013 0.01 Hz
Data (A)- 5
Grid voltage Input (3x) 0014 0.1 V
GND
Grid current Input (3x) 0015 0.1 A
Energy today Input (3x) 0027 0.1 kW h
Fig. 4. 2-wire RS485 configuration
Inverter temperature Input (3x) 0032 0.1 o C

permits point-to-point connections over short distances and

facilitates data transfer between two devices. As the inverter DC power of the PV system collected from register 3x0002 of
is situated close to the server, they are directly linked using the inverter (register address list is provided in Table. I). After
an RS232 to COM cable with a signal converter module. The successfully giving metrics to corresponding endpoints, there
input registers addressed by the inverter manufacturer can hold is a need to build a database that stores measurements for long-
data and be accessed via the communication port to retrieve term purposes. This study suggests an open-source time series
the necessary information for users, such as DC and AC power, database (TSDB) named Prometheus [10] to pull metrics with
the energy produced, grid-side parameters, etc. [9]. their timestamp in real-time and store them on hard drives.
The communication network operates at a baud rate of 9600. The basic structure of the time series data frame is presented
The gateway and server have IP addresses of 192.168.1.254 in Table. II. The user-defined scrape interval corresponds to
and 192.168.1.100, respectively. The detailed specifications of the sampling frequency within the system. Particularly, if the
sensors are given in Table. I. scrape interval is configured to 5, the Prometheus server will
Short-lived data is pushed to the HTTP metrics end- retrieve metrics every 5 seconds from endpoints, and then store
points and is replaced periodically with new data. A metric them with their timestamps.
query is frequently identified by metric name and a set of The accompanying YAML file with .yml extension needs to
labels. For example, the notation inverter{parameter = be expanded for the Prometheus server to identify targets and
”input power”}, which incorporates the metric inverter and scrape required data from each endpoint. The following script
the label parameter with its value input power, signifies the gives an example of scrape configurations of Prometheus in

304
Authorized licensed use limited to: SHIBAURA INSTITUTE OF TECHNOLOGY. Downloaded on February 18,2025 at 10:13:52 UTC from IEEE Xplore. Restrictions apply.
 TABLE II basic_auth:
E XAMPLE OF TIME SERIES DATA FRAME username: "your_username"
password: "your_password"

Timestamp Value Fig. 5 captures our SCADA interface inspired by the

2023-10-4 8:20:50 108 Grafana Cloud dashboard. With the free plan, Grafana allows
2023-10-4 8:25:55 112 up to three users with administrator roles to have the capability
to download well-formatted data directly from the online
2023-10-4 8:26:00 121
dashboard without needing to access the local server. The
public website of our dashboard allowing guests to view our
system status in real-time is navigated in the Appendix.
YAML file which includes prometheus job to record its states In essence, Prometheus stores metrics locally on the desktop
from localhost : 9090 endpoint, inverter job to pull inverter server. However, local hard drives have limited capacity to
status from localhost : 9000 and roof top job to take the data serve a substantial volume of 24/7 data recorded. Furthermore,
of the meteorological sensors from localhost : 9602. hardware failures can lead to the risk of system disruptions and
global: data loss. Therefore, cloud storage can be a promising solution
scrape_interval: 5s to address these issues. The authors take advantage of Google
Drive, a Google’s cloud-based storage solution, to externally
scrape_configs:
#Scrape metrics from prometheus itself store data and share the SCADA metrics [12]. A sample code
- job_name: "prometheus" below describes the data uploading process to the cloud drive
static_configs: for DC power measurement.
- targets: ["localhost:9090"]
import datetime as dt
#Scrape metrics from endpoint of the inverter from prometheus_api_client import PrometheusConnect,
- job_name: "inverter" MetricRangeDataFrame
static_configs:
- targets: ["localhost:9000"] prom = PrometheusConnect(url ="http://localhost:9090
", disable_ssl=True)
#Scrape metrics from endpoint of the
meteorological sensors ###Save DC power###
- job_name: "rooftop" metric_data = prom.get_metric_range_data(
static_configs: metric_name=’inverter’,
- targets: ["localhost:9602"] label_config={’parameter’:’input_power’},

#Select last 24h data from now

III. O NLINE DATA S TORAGE AND V ISUALIZATION start_time=(dt.datetime.now() - dt.timedelta(
days=1)),
Visualization allows individuals to grasp complex data more end_time=dt.datetime.now(),
quickly and effectively than raw data or numbers. It assists )
users in making sense of large patterns, enabling them to
#Convert data to pandas DataFrame
draw insights and make informed decisions. Prometheus is metric_df = MetricRangeDataFrame(metric_data)
basically a data monitoring and storage toolkit for time se-
ries, therefore it may not be the most suitable choice for #Save data as a CSV file to the cloud drive
metric_df.to_csv(r’Google\Drive\folder\path\pv
data visualization. Prometheus web UI actually provides a output power.csv’)
graph-visualization feature, but it is relatively basic and lacks
extensive customization options. Moreover, it is not feasible ###Other metrics###
#...
to share data externally via the internet with only a local
Prometheus server. Grafana Cloud [11] is an online-based ob- Accordingly, a Python module called prometheus − api −
servability platform that retrieves metrics data from numerous client [13] is used for handling data from the Prometheus
popular data sources, including Prometheus (Fig. 2). Grafana database and converting it into the well-known pandas
Cloud offers real-time data visualization with user-interface DataFrame. Lastly, the program saves data as a CSV file and
customization and the ability to publish dashboards online. The uploads it to the cloud drive. By incorporating additional loops,
below script describes the necessary expansions to the YAML these procedures will be executed automatically ensuring
file, allowing Prometheus to remotely write metrics to Grafana that the data source is updated at regular intervals, thereby
Cloud. The username and password fields are defined by providing users with continuously refreshed data.
users when connecting their local database to Grafana Cloud
IV. DATA O BSERVATION
accounts. Grafana Cloud pulls metrics from Prometheus by
PromeQL queries which are identical to metric notations in A one-week dataset from October 29 to November 3, 2023
the TSDB. scraped from the TSDB is displayed in Fig. 6, Fig. 7, Fig. 8,
and Fig. 9 as an example for meteorological measurements.
remote_write:
- url: "https://prometheus-prod-13-prod-us-east Solar irradiance shown in Fig. 6 reached its peak at approxi-
-0.grafana.net/api/prom/push" mately 800 W/m2 . On the second day, there was a significant

305
Authorized licensed use limited to: SHIBAURA INSTITUTE OF TECHNOLOGY. Downloaded on February 18,2025 at 10:13:52 UTC from IEEE Xplore. Restrictions apply.
 Fig. 5. SCADA monitoring by Grafana Cloud dashboard

decrease in irradiance, which is reasonable because it was a

rainy day on October 29. As observed in Fig. 7 and Fig. 9,
the air temperature and PV panel temperature showed similar
trends as the irradiance. In high-radiation conditions, PV
panels can become significantly hotter than the surrounding
air due to the heat generated by the photoelectric effect. Fig.
8 showed that the humidity is almost inversely related to other
parameters which tend to decrease when solar irradiance and
temperature rise.
Fig. 7. Air temperature during one week

Fig. 6. Solar irradiance during one week

Fig. 8. Air humidity during one week

The authors investigated a mathematical model for estimat-

ing the PV output power based on monitored irradiance and
G
PV panel temperature. Subsequently, we compared the model’s PP V = PPins
V × × [1 + αP (Tpanel − TN M OT )] (1)
GST C
predictions with the actual values obtained from inverter
measurements to validate the data’s correctness. The equation where PP V is the generated power, PPins V is the installed
depicting the correlation between meteorological values and capacity of the system, G is the solar irradiance, GST C is the
the output power is presented below: solar irradiance on the Standard Test Condition (STC), αP is

306
Authorized licensed use limited to: SHIBAURA INSTITUTE OF TECHNOLOGY. Downloaded on February 18,2025 at 10:13:52 UTC from IEEE Xplore. Restrictions apply.
 the future, the authors may suggest some data-driven meth-
ods to enhance the model’s performance which considers
numerous complicated effects on the PV system. Furthermore,
researchers are suggested to handle high-resolution data for
renewable energy system control and real-time forecasting.
A PPENDIX
Readers can click here to take the QR code and the link
to view our public dashboard. If the dashboard becomes
inaccessible, please contact the authors for assistance. Data
Fig. 9. PV panel temperature during one week are also available upon requests.
ACKNOWLEDGEMENT
the temperature coefficient of maximum power, Tpanel is the This research is funded by the Hanoi University of Science
panel temperature and TN M OT = 45 ± 3o C is the temperature and Technology (HUST) under project number T2022-PC-001.
in the Nominal Module Operating Temperature (NMOT). We kindly appreciate Mr. Tran Vu Quoc from Investment
The comparison between the actual and estimated PV power and Technology Service Joint-Stock Company (ITS) for his
during one week is illustrated in Fig. 10. It can be seen valuable assistance.
that the mathematical model offers good approximation. Some
evaluation metrics below are used to evaluate the performance R EFERENCES
of the model: [1] What is the global potential for renewable energy. https://www.science
direct.com/science/article/abs/pii/S1364032111003984.
n
1X [2] Ibrahim Dincer, Renewable energy and sustainable development:
M AE = |yi − ŷi | (2) a crucial review, Renewable and Sustainable Energy Reviews,
n i=1 Volume 4, Issue 2, 2000, Pages 157-175, ISSN 1364-0321,
https://doi.org/10.1016/S1364-0321(99)00011-8.
n [3] Nadarajah Kannan, Divagar Vakeesan, Solar energy for future
1 X yi − ŷi
M AP E = (3) world: - A review, Renewable and Sustainable Energy Re-
n i=1 yi views, Volume 62, 2016, Pages 1092-1105, ISSN 1364-0321,
v https://doi.org/10.1016/j.rser.2016.05.022.
u n [4] Ansari S, Ayob A, Lipu MSH, Saad MHM, Hussain A. A
u1 X Review of Monitoring Technologies for Solar PV Systems Using
RM SE = t (yi − ŷi )2 (4) Data Processing Modules and Transmission Protocols: Progress,
n i=1 Challenges and Prospects. Sustainability. 2021; 13(15):8120.
https://doi.org/10.3390/su13158120.
where yi is the actual value, ŷi is the estimated value and n [5] Siva Ramakrishna Madeti, S.N. Singh, Monitoring system for pho-
is the total number of values. The one-week sample dataset tovoltaic plants: A review, Renewable and Sustainable Energy
Reviews, Volume 67, 2017, Pages 1180-1207, ISSN 1364-0321,
results in M AE = 67.8021W , M AP E = 0.0949, and https://doi.org/10.1016/j.rser.2016.09.088.
RM SE = 85.8652W , which proves the accuracy of the [6] Manajit Sengupta, Yu Xie, Anthony Lopez, Aron Habte, Galen Maclau-
deployed model. rin, James Shelby, The National Solar Radiation Data Base (NSRDB),
Renewable and Sustainable Energy Reviews, Volume 89, 2018, Pages
51-60, ISSN 1364-0321, https://doi.org/10.1016/j.rser.2018.03.003.
[7] Molina, A., Falvey, M. & Rondanelli, R. A solar radiation database
for Chile. Sci Rep 7, 14823 (2017). https://doi.org/10.1038/s41598-017-
13761-x.
[8] P. S and V. Hegde, ”Data Acquisition Systems for monitoring Real
Time Parameters of Rooftop Solar Panels,” 2022 IEEE 2nd Mysore Sub
Section International Conference (MysuruCon), Mysuru, India, 2022,
pp. 1-5, doi: 10.1109/MysuruCon55714.2022.9972631.
[9] Growatt PV Inverter Modbus RS485 RTU Protocol. https://www.amospl
anet.org/wp-content/uploads/2023/07/Growatt-PV-Inverter-Modbus-
RS485-RTU-Protocol-V3-14.pdf. Accessed: Nov 08, 2023.
[10] Prometheus software. https://prometheus.io. Accessed: Nov 08, 2023.
[11] Grafana Labs. https://grafana.com/products/cloud/. Accessed: Nov 08,
2023.
Fig. 10. Actual and estimated PV power during one week [12] Google Drive desktop application. https://drive.google.com . Accessed:
Nov 08, 2023.
[13] Prometheus api client. https://pypi.org/project/prometheus-api-client/.
V. C ONCLUSIONS Accessed: Nov 08, 2023.
This study provides an efficient solution to monitor data
from solar power systems employing a time series database
and cloud drive for data back-up and sharing. In addition, the
authors also offer a free-access website for users to observe
real-time SCADA metrics. A simple theoretical model was
applied to confirm the correctness of our measurements. In

307
Authorized licensed use limited to: SHIBAURA INSTITUTE OF TECHNOLOGY. Downloaded on February 18,2025 at 10:13:52 UTC from IEEE Xplore. Restrictions apply.