FOUNDATION TO DATA SCIENCE

Business Analytics

MODULE II-Unit:3
Time Series Forecasting

Prof. Dr. George Mathew

B.Sc., B.Tech, PGDCA, PGDM, MBA, PhD 1
 Time Series Forecasting
SYLLABUS
3.1 What is Time-Series analysis
3.1.1 Decomposition
3.2 Correlation and Causation
3.3 Autocorrelation
3.4 Forecasting Data
3.5 Autoregressive Modeling, Moving averages and
ARIMA
3.6 Neural networks for Time series data
 Overview of Time Series Forecasting
Time series is a type of data that measures how things change over time. In a time
series data set, the time column does not represent a variable per se: it is actually a
primary structure that you can use to order your data set. This primary temporal
structure makes time series problems more challenging as data scientists need to apply
specific data preprocessing and feature engineering techniques to handle time series
data.
However, it also represents a source of additional knowledge that data scientists can
use to their advantage:
You will learn how to leverage this temporal information to extrapolate insights from
your time series data, like trends and seasonality information, to make your time series
easier to model and to use it for future strategy and planning operations in several
industries. From finance to manufacturing and health care, time series forecasting has
always played a major role in unlocking business insights with respect to time.
Following are some examples of problems that time series forecasting can help you
solve:
➢ What are the expected sales volumes of thousands of food groups in different
grocery stores next quarter?
➢ What are the resale values of vehicles after leasing them out for three years?
➢ What are passenger numbers for each major international airline route and for each
class of passenger?
➢ What is the future electricity load in an energy supply chain infrastructure, so that
suppliers can ensure efficiency and prevent energy waste and theft?.
 What is Time-Series analysis?
Time-Series Analysis (TSA) is the process of extracting a summary and
other statistical information from time-series, most importantly, the analysis
of trend and seasonality. It is often an ad hoc exploration and analysis that
usually involves visualizing distributions, trends, cyclic patterns, and
relationships between features, and between features and
the target(s).
TSA is roughly exploratory data analysis (EDA) that's specific to time-
series data. This comparison can be misleading however since TSA can
include both descriptive and exploratory elements.
Let's see quickly the differences between descriptive and exploratory
analysis:
● Descriptive analysis summarizes characteristics of a dataset
● Exploratory analysis analyzes for patterns, trends, or relationships
between variables
Therefore, TSA is the initial investigation of a dataset with the goal of
discovering patterns, especially trend and seasonality, and obtaining initial
insights, testing hypotheses, and extracting meaningful summary statistics.
 What is Time-Series analysis?
A part of TSA is collecting and reviewing data, examining the distribution of
variables (and variable types), and checking for errors, outliers, and
missing values. Some errors, variable types, and anomalies can be
corrected, therefore EDA is often performed hand in hand with
preprocessing and feature engineering, where columns and fields are
selected and transformed. The whole process from data loading to
machine learning is highly iterative and may involve multiple instances of
TSA at different points
Here are a few crucial steps for working with time-series:
● Importing the dataset
● Data cleaning
● Understanding variables
● Uncovering relationships between variables
● Identifying trend and seasonality
● Preprocessing (including feature engineering)
● Training a machine learning model
 What is Time-Series analysis?
Importing the data can be considered prior to TSA,
and data cleaning, feature engineering, and training
a machine learning model are not strictly part of TSA.
Importing the data includes parsing, for example
extracting dates. The three steps that are central to
TSA are understanding variables, uncovering
relationships between variables, and identifying trend
and seasonality.
The steps belonging to TSA and leading to
preprocessing (feature engineering) and machine
learning are highly iterative, and can be visually
appreciated in the following time-series machine
learning flywheel:
 Components of time series

Ref: Machine Learning for time series Forcasting with Python

 Components of time series
Long-term movement or trend refers to the overall movement of time series
values to increase or decrease during a prolonged time interval. It is common to observe
trends changing direction throughout the course of your time series data set: they may
increase, decrease, or remain stable at different moments. However, overall you will see
one primary trend. Population counts, agricultural production, and items manufactured are
just some examples of when trends may come into play.
There are two different types of short-term movements:
Seasonal variations are periodic temporal fluctuations that show the same variation and
usually recur over a period of less than a year. Seasonality is always of a fixed and known
period. Most of the time, this variation will be present in a time series if the data is
recorded hourly, daily, weekly, quarterly, or monthly. Different social conventions (such as
holidays and festivities), weather seasons, and climatic conditions play an important role in
seasonal variations, like for example the sale of umbrellas and raincoats in the rainy season
and the sale of air conditioners in summer seasons.
Cyclic variations, on the other side, are recurrent patterns that exist when data exhibits
rises and falls that are not of a fixed period. One complete period is a cycle, but a cycle will
not have a specific predetermined length of time, even if the duration of these temporal
fluctuations is usually longer than a year. A classic example of cyclic variation is a business
cycle, which is the downward and upward movement of gross domestic product around its
long-term growth trend: it usually can last several years, but the duration of the current
business cycle is unknown in advance.
 Components of time series

Random or irregular fluctuations are the last element to cause variations in

our time series data. These fluctuations are uncontrollable, unpredictable,
and erratic, such as earthquakes, wars, flood, and any other natural
disasters.
Data scientists often refer to the first three components (long-term movements,
seasonal short-term movements, and cyclic short-term movements) as signals in
time series data because they actually are deterministic indicators that can be
derived from the data itself.
On the other hand, the last component (random or irregular fluctuations) is an arbitrary
variation of the values in your data that you cannot really predict, because each data point
of these random fluctuations is independent of the other signals above, such as long-term
and short-term movements. For this reason, data scientists often refer to it as noise,
because it is triggered by latent variables difficult to observe.
Actual representation of time series components
The time-series machine learning flywheel
 The time-series machine learning flywheel
This flywheel emphasizes the iterative nature of the work. For
example, data cleaning comes often after loading the data, but will
come up again after we've made another discovery about our
variables. In the figure above TSA is highlighted in dark, while steps
that are not strictly part of TSA are grayed out.
Practically, we'll start by loading a dataset. Right after importing the
data, we'd ask questions like what's the size of the dataset (the
number of observations)? How many features or columns do we
have? What are the column types?
We'll typically look at histograms or distribution plots. For assessing
relationships between features and target variables, we'd calculate
correlations and visualize them as a correlation heatmap, where the
correlation strength between variables is mapped to colors.
We'd look for missing values – in a spreadsheet, these would be
empty cells – and we'd clean up and correct these irregularities,
where possible.
 The time-series machine learning flywheel
We are going to be analyzing relationships between variables, and
in TSA, one of its peculiarities is that we need to investigate the
relationship of time with each variable.
We are going to be analyzing relationships between variables, and
in TSA, one of its peculiarities is that we need to investigate the
relationship of time with each variable.
Generally, a useful way of distinguishing different types of
techniques could be between univariate and multivariate analysis,
and between graphical and non graphical techniques. Univariate
analysis means we are looking at a single variable. This means we
could be inspecting values to get the means and the variance, or –
for the graphical side – plotting the distribution. We summarize these
techniques in the Understanding the variables section.
On the other hand, multivariate analysis means we are calculating
correlations between variables, or – for the graphical side – drawing
a scatter plot, for example.
 Identifying trend and seasonality
Trend, seasonality, and cyclic variations are the most important
characteristics of time-series. A trend is the presence of a long-term
increase or decrease in the
sequence. Seasonality is a variation that occurs at specific regular intervals
of less than a year. Seasonality can occur on different time spans such as
daily, weekly, monthly, or
yearly. Finally, cyclic variations are rises and falls that are not of a fixed
frequency.
An important characteristic of time-series is stationarity. This refers to a
property of time-series not to change distribution over time, or in other
words, that the process that produces the time-series doesn't change with
time. Time-Series that don't change over time are called stationary (or
stationary processes). Many models or measures assume stationarity and
might not work properly if the data is not stationary. Therefore, with these
algorithms, the time-series should be decomposed first into the main signal,
and then the seasonal and trend components. In this decomposition, we
would subtract the trend and seasonal components from the original time-
series.
 Decomposition
When we decompose a time series into components, we usually combine
the trend and cycle into a single trend-cycle component (sometimes called
the trend for simplicity).
Thus we think of a time series as comprising three components: a trend-
cycle component, a seasonal component, and a remainder component
(containing anything else in the time series).
Time series components
If we assume an additive decomposition, then we can write:

where 𝑦𝑡 is the data, 𝑆𝑡 is the seasonal component, 𝑇𝑡 is the trend-cycle

component, and 𝑅t is the remainder component, all at period .
Alternatively, a multiplicative decomposition would be written as:
 Decomposition
The additive decomposition is the most appropriate if the
magnitude of the seasonal fluctuations, or the variation around
the trend-cycle, does not vary with the level of the time series.
When the variation in the seasonal pattern, or the variation
around the trend-cycle, appears to be proportional to the level
of the time series, then a multiplicative decomposition is more
appropriate. Multiplicative decompositions are common with
economic time series.
An alternative to using a multiplicative decomposition is to first
transform the data until the variation in the series appears to
be stable over time, then use an additive decomposition. When
a log transformation has been used, this is equivalent to using
a multiplicative decomposition because
Decomposition
 Time Series Data Preparation
Each machine learning algorithm expects data as input that needs to be formatted
in a very specific way, so time series data sets generally require some cleaning
and feature engineering processes before they can generate useful insights. Time
series data sets may have values that are missing or may contains outliers, hence
the need for the data preparation and cleaning phase is essential. Since the time
series data has temporal property, only some of the statistical methodologies are
appropriate for time series data. Good time series data preparation produces
clean and well-curated data, which leads to more practical, accurate predictions.
Specifically, in this session we will discuss the following:
Python for Time Series Data – Python is a very powerful programming language to handle
data, offering an assorted suite of libraries for time series data and excellent support for
time series analysis.
Time Series Exploration and Understanding: Specifically, this section will focus on how to
get started with time series data analysis and how to calculate and review summary
statistics for time series data; then you will learn how to perform data cleaning of missing
periods in a time series; finally, you will see how to perform time series data normalization
and standardization.
Time Series Feature Engineering – Feature engineering is the process of using raw data to
create additional features or variables to augment your data set.
Python for Time Series Data
Python is currently one of the most dominant platforms for time
series data because of the excellent library support. There are a few
Python libraries for time series, as Figure illustrates.
Python for Time Series Data
The primary rationale for adopting Python for time series forecasting is that it is
a general-purpose programming language that you can use both for
experimentation and in production. It is easy to learn and use, primarily because
the language focuses on readability. Python is a dynamic language and very suited
to interactive development and quick prototyping with the power to support the
development of large applications.
Python is also widely used for machine learning and data science because of the
excellent library support, and it has a few libraries for time series, such as NumPy,
pandas, SciPy, scikit-learn, statsmodels, Matplotlib, datetime, Keras, and many
more. Below we will have a closer look at the time series libraries in Python
SciPy: SciPy is a Python-based ecosystem of open-source software for
mathematics, science, and engineering. Some of the core packages are NumPy (a
base n-dimensional array package), SciPy library (a fundamental library for
scientific computing), Matplotlib (a comprehensive library for 2-D plotting),
IPython (an enhanced interactive console), SymPy (a library for symbolic
mathematics), and pandas (a library for data structure and analysis).
Two SciPy libraries that provide a foundation for most others are NumPy and
Matplotlib:
Python for Time Series Data
NumPy is the fundamental package for scientific computing with Python. It
contains, among other things, the following: A powerful n-dimensional array
object Sophisticated (broadcasting) functions Tools for integrating C/C++ and
Fortran code Useful linear algebra, Fourier transform, and random number
Capabilities
Matplotlib: Matplotlib is a Python plotting library that produces publication-
quality figures in a variety of hardcopy formats and interactive environments
across platforms. Matplotlib can be used in Python scripts, the Python and
IPython shells, the Jupyter notebook, web application servers, and four graphical
user interface toolkits.
Matplotlib is useful to generate plots, histograms, power spectra, bar charts, error
charts, scatterplots, and so on with just a few lines of code. The most up-to-date
Matplotlib documentation can be found in the Matplotlib user's guide
Pandas: Pandas is an open-source, BSD-licensed library providing high
performance, easy-to-use data structures, and data analysis tools for the Python
programming language. Python has long been great for data munging and
preparation, but less so for data analysis and modeling. Pandas helps fill this gap,
enabling you to carry out your entire data analysis workflow in Python without
having to switch to a more domain-specific language like R.
Python for Time Series Data
Statsmodels: Statsmodels is a Python module that provides classes and functions
for the estimation of many different statistical models as well as for conducting
statistical tests and statistical data exploration. An extensive list of result statistics
is available for each estimator. The results are tested against existing statistical
packages to ensure that they are correct.
Scikit-learn - Scikit-learn is a simple and efficient tool for data mining and data
analysis. In particular, this library implements a range of machine learning, pre-
processing, cross-validation, and visualization algorithms using a unified interface.
It is built on NumPy, SciPy, and Matplotlib and is released under the open-source
Modified BSD (3- clause) license.
Keras: Keras is a high-level neural networks API, written in Python and capable of
running on top of TensorFlow, CNTK, and Theano. Data scientists usually use
Keras if they need a deep learning library that does the following:
Allows for easy and fast prototyping (through user friendliness,
modularity, and extensibility) Supports both convolutional networks and
recurrent networks, as well as combinations of the two Runs seamlessly on
central processing unit (CPU) and graphics processing unit (GPU)
Common Data Preparation Operations for Time Series
Pandas (pandas.pydata.org) offers a diverse portfolio of capabilities to support
time series data. The following key features are important for time series
forecasting in pandas:
➢ Parsing time series information from various sources and formats
➢ Generating sequences of fixed-frequency dates and time spans
➢ Manipulating and converting date times with time zone information
➢ Resampling or converting a time series to a particular frequency
➢ Performing date and time arithmetic with absolute or relative time increments
Pandas captures four general time related concepts:
➢ Date time: A specific date and time with time zone support, such as month, day,
year, hour, second, microsecond.Time deltas: An absolute time duration
used for manipulating dates.
➢ Time spans: A duration of time defined by a point in time and its associated
frequency.
➢ Date offsets: A relative time duration that respects calendar arithmetic.
Four general time-related concepts supported in pandas:

There are a few additional concepts that are important to explain for those new to Python:
Scalar class: A scalar class is a numerical value alone and defines a vector
space. A quantity described by multiple scalars, such as having both direction and
magnitude, is called a vector.
Array class: An array class is a data structure that holds a fixed number of values of the
same data type.
Data type: A data type is a labeling of different data objects. It represents the type of
values in your data set used to understand how to store and manipulate data.
Primary creation method: A group of functions and operations that are typically written
and used to perform a single or a set of actions on your data. Functions, if properly written,
provide better efficiency, reproducibility, and modularity to your Python code.
Time stamps vs. Periods: The pandas library in Python provides comprehensive
and built-in support for time series data. The examples below and for the rest of
the chapter focuses on energy demand forecasting. In general, demand
forecasting is the practice of predicting future demand (or quantity of specific
products and services) by using historical demand data to make informed business
decisions about future plans, such as planning a product inventory optimization
strategy, allocating marketing investments, and estimating the cost of new
products and services to meet customer demand and needs.
Energy demand forecasting is a type of demand forecasting where the goal is to
predict the future load (or energy demand) on an energy grid. It is a critical
business operation for companies in the energy sector, as operators need to
maintain the fine balance between the energy consumed on a grid and the energy
supplied to it. Typically, grid operators can take short-term decisions to manage
energy supply to the grid and keep the load in balance. An accurate short-term
forecast of energy demand is therefore essential for the operator to make these
decisions with confidence. This scenario details the construction of a machine
learning energy demand forecasting solution.
Let's first look at the concept of time stamped data, which is the most essential
type of time series data and helps you combine values with specific points in time.
This means that each data point of your data set will have a temporal information
that you can leverage.
Time Series Exploration and Understanding
In the following sections, you will learn the first steps to take to explore,
analyze, and understand time series data. We will focus on these topics:
➢ How to get started with time series data analysis
➢ How to calculate and review summary statistics for time series data
➢ How to perform data cleaning of missing periods in time series
➢ How to perform time series data normalization and standardization

How to Get Started with Time Series Data Analysis

As you saw in the first section of this chapter, pandas has proven very effective
as a tool for handling and working with time series data: pandas has some inbuilt
datetime functions that make it simple to work with time series data, because in this
type of data set, time is the most important variable and dimension that data scientists
can leverage to gain useful insights.
Now let's have a look at our ts_data set to see what kind of data we have by using the
head() function. This function returns the first n rows for your time series data set, and it
is useful for quickly getting an overview of the type of data and its structure in your data
set:
ts_data.head(10)
 Exploratory Data Analysis for Time Series
We will break down our discussion of exploratory data analysis for time
series into two distinct sections.
First, we discuss the application of commonly used data methods
on time series. In particular, we discuss how histograms, plotting, and
group-by operations can be applied to time series data.
Second, we highlight fundamentally temporal methods for time series
analysis—that is, methods developed specifically for time series data and
that make sense only in the context of data points that have a temporal
relationship with one another (rather than a cross-sectional one).
You will want to address the same exploratory
questions you would ask about any new data set, such as:
• Are any of the columns strongly correlated with one another?
• What is the overall mean of an interesting variable? What is its variance?
 Exploratory Data Analysis for Time Series
To answer these, you can use familiar techniques such as plotting, taking
summary statistics, applying histograms, and using targeted scatter plots.
You’ll also want to answer explicitly time-oriented questions such as:
What is the range of values you see, and do they vary by time period or
some other logical unit of analysis?
Does the data look consistent and uniformly measured, or does it suggest
changes in either measurement or behavior over time?
To address these, you will want the previously mentioned methods—
histograms, scatter plots, and summary statistics—to take into account the
temporal axis.
To implement this behavior, we will incorporate time into our statistics, as
an axis in our graphs or as a group in group-by operations, such as those
applied to histograms or scatter plots.
Exploratory Data Analysis for Time Series
 Seasonal Data and Cyclical Data
Seasonal time series are time series in which behaviors
recur over a fixed period.
There can be multiple periodicities reflecting different
tempos of seasonality, such as the seasonality of the 24-
hour day versus the 12-month calendar season, both of
which exhibit strong features in most time series relating to
human behavior.
Cyclical time series also exhibit recurring behaviors, but they
have a variable period. A common example is a business
cycle, such as the stock market’s boom and bust cycles,
which have an uncertain duration. Likewise, volcanoes show
cyclic but not seasonal behaviors. We know the approximate
periods of eruption, but these are not precise and vary over
time.
 Application of Time Series Analysis
• Medicine
• Weather
• Economics
• Astronomy
 Time Series Forecasting
Time series forecasting is a technique for the prediction of
events through a sequence of time. It predicts future events by
analyzing the trends of the past, on the assumption that future
trends will hold similar to historical trends. It is used across
many fields of study in various applications.
Time series forecasting occurs when you make scientific
predictions based on historical time stamped data. It involves
building models through historical analysis and using them to
make observations and drive future strategic decision-making.
An important distinction in forecasting is that at the time of the
work, the future outcome is completely unavailable and can
only be estimated through careful analysis and evidence-
based priors.
 Time Series Forecasting
Time series forecasting is the process of analyzing time series data
using statistics and modeling to make predictions and inform strategic
decision-making.
It’s not always an exact prediction, and likelihood of forecasts can
vary wildly—especially when dealing with the commonly fluctuating
variables in time series data as well as factors outside our control.
However, forecasting insight about which outcomes are more likely—or
less likely—to occur than other potential outcomes. Often, the more
comprehensive the data we have, the more accurate the forecasts can be.
While forecasting and “prediction” generally mean the same thing, there is
a notable distinction. In some industries, forecasting might refer to data at a
specific future point in time, while prediction refers to future data in general.
Series forecasting is often used in conjunction with time series analysis.
Time series analysis involves developing models to gain an
understanding of the data to understand the underlying causes. Analysis
can provide the “why” behind the outcomes you are seeing. Forecasting
then takes the next step of what to do with that knowledge and the
predictable extrapolations of what might happen in the future.
 Applications of time series forecasting
Forecasting has a range of applications in various industries.
It has tons of practical applications including:
weather forecasting, climate forecasting, economic
forecasting, healthcare forecasting engineering forecasting,
finance forecasting, retail forecasting, business forecasting,
environmental studies forecasting, social studies forecasting,
and more.
Basically anyone who has consistent historical data
can analyze that data with time series analysis methods and
then model, forecasting, and predict. For some industries, the
entire point of time series analysis is to facilitate forecasting.
Some technologies, such as augmented analytics, can even
automatically select forecasting from among other statistical
algorithms if it offers the most certainty.
When time series forecasting should be used
Naturally, there are limitations when dealing with the unpredictable
and the unknown. Time series forecasting isn’t infallible and isn’t
appropriate or useful for all situations. Because there really is no explicit set
of rules for when you should or should not use forecasting, it is up to
analysts and data teams to know the limitations of analysis and what their
models can support.
Data teams should use time series forecasting when they
understand the business question and have the appropriate data and
forecasting capabilities to answer that question. Good forecasting works
with clean, time stamped data and can identify the genuine trends and
patterns in historical data. Analysts can tell the difference between random
fluctuations or outliers, and can separate genuine insights from seasonal
variations.
Time series analysis shows how data changes over time, and good
forecasting can identify the direction in which the data is changing.
 Time-Series Forecasting
In simple terms data points that are collected sequentially at
a regular interval with association over a time period is
termed time-series data. A time-series data having the
mean and variance as a constant is called a stationary time
series.
Time series tend to have a linear relationship between
lagged variables and this is called an autocorrelation. Hence
a time series historic data can be modeled to forecast
the future data points without involvement of any other
independent variables; these types of models are generally
known as time-series forecasting. To name some key areas
of applications of time series, these include sales
forecasting, economic forecasting, stock market forecasting,
etc.
 Components of Time Series
A time series can be made up of three key components. See Figure .
• Trend – A long-term increase or decrease are termed trends.
• Seasonality An effect of seasonal factors for a fixed or known period.
For example, retail stores sales will be high during weekends and
festival seasons.
• Cycle – These are the longer ups and downs that are not of fixed or
known periods caused by external factors.
 FORECASTING
Machine learning is a subset of artificial intelligence that uses techniques
(such as deep learning) that enable machines to use experience to
improve at tasks. The learning process is based on the following steps:
1. Feed data into an algorithm. (In this step you can provide additional
information to the model, for example, by performing feature extraction.)
2. Use this data to train a model.
3. Test and deploy the model.
4. Consume the deployed model to do an automated predictive task. In
other words, call and use the deployed model to receive the predictions
returned by the model.
Machine learning is a way to achieve artificial intelligence. By using
machine learning and deep learning techniques, data scientists can build
computer systems and applications that do tasks that are commonly
associated with human intelligence. These tasks include time series
forecasting, image recognition, speech recognition, and language
translation
 THE FORECASTING PROCESS
A process is a series of connected activities that
transform one or more inputs into one or more
outputs. The activities in the forecasting
process are:
1. Problem definition
2. Data collection
3. Data analysis
4. Model selection and fitting
5. Model validation
6. Forecasting model deployment
7. Monitoring forecasting model performance
 Time Series Forecasting Template
Our template consists of different steps:
1. Business understanding and performance metrics
definition
2. Data ingestion
3. Data exploration and understanding
4. Data pre-processing and feature engineering
5. Model building and selection
6. Model deployment
7. Forecasting solution acceptance
Time Series Forecasting Template
 GRAPHICAL DISPLAYS
1. Time Series Plots
2. Plotting Smoothed Data
 Time Series (Data Analytics)
A time series is a series of data points arranged
chronologically. Most commonly, the time points are equally
spaced. A few examples are the passenger loads of an airline
recorded each month for the past two years
or the price of an instrument in the share market recorded
each day for the last year. The primary aim of time-series
analysis is to predict the future value of a parameter based on
its past data.
Classification of Variation
Traditionally time-series analysis divides the variation into
three major components, namely, trends, seasonal variations,
and other cyclic changes. The variation that remains is
attributed to “irregular” fluctuations or error term. This
approach is particularly valuable when the variation is
mostly comprised of trends and seasonality.
 Analyzing a Series Containing a Trend
A trend is a change in the mean level that is long-term in
nature.
For example, if you have a series like 2, 4, 6, 8 and someone
asks you for the next value, the obvious answer is 10. You
can justify your answer by fitting a line to the data using the
simple least square estimation or any other
regression method. A trend can also be nonlinear. Figure 6-1
shows an example of a time series with trends.
The simplest type of time series is the familiar “linear trend
plus noise” for which the observation at time t is a random
variable Xt, as follows:
Xt = α + β t +e1
Here, α, β are constants, and εt denotes a random error term
with a mean of 0. The average level at time t is given by mt=
(α + βt). This is sometimes called the trend term.
 Modeling Building and Selection
The modeling phase is where the conversion of the data into a model takes
place. In the core of this process there are advanced algorithms that scan the
historical data (training data), extract patterns, and build a model. That
model can be later used to predict on new data that has not been used to
build the model. Once we have a working reliable model, we can then use it
to score new data that is structured to include the required features. The
scoring process will make use of the persisted model (object from the training
phase) and predict the target variable.
 Modeling Building and Selection
Train data set – The train data set represents the amount of data that data
scientists decide to use to fit machine learning models. Through the train data set,
a machine learning algorithm can be trained to learn from historical data in order
to predict future data points.
Validation data set – The validation data set is the amount of data used to
offer an unbiased evaluation of a model fit on the train data set, while tuning
model hyperparameters. Data scientists usually leverage validation data sets to
fine-tune a machine learning model hyperparameters. Hyperparameters are
additional factors in the model, whose values are used to control, and eventually
improve, the learning process of the model itself.
Data scientists observe the validation set results to update the levels of
hyperparameters and, consequently, improve their models.
Test data set – The test data set is the amount of data used to determine whether
a model is underfitting (the model is performing poorly on the train data set) or
overfitting (the model is performing well on the train data set but not on the test
data set) the train data by looking at the prediction error on the train and test
data sets. The test data set is only used once a model is completely trained and
validated through the train and validation data sets. Data scientists usually
leverage the test data set to evaluate different machine learning models.
Autoregressive Integrated Moving
Average (ARIMA)