TYBBA (CA)

Unit : 6 Spark

Important Questions:

1. Define : Spark

Ans : Apache spark is open source distributed processing system used for big
data workloads. It is cluster computing designed for fast computation.

2. List components of spark

Ans : 1. Apache Spark 2. Spark SQL 3. Spark Streaming 4. MLLib

5. GraphX

3. Which language is not supported by Spark?

Ans : Spark does not have built in support for COBOL

4. Which language is supported by Spark?

Ans : Spark supports languages like Scala, Java, Python and R

5. List features of Spark

Ans : 1. Speed 2. Multiple Language Support 3. Multiple Platform Support

4.Advance Analytics

6. Define : RDD

Ans : RDD is Resilient Distributed Datasets and it is a fundamental data

Structure of Apache Spark. It is an immutable collection of objects

Which computes on the different node of the cluster.

Long Questions

1. How is Apache Spark different from Map Reduce?

Spark Map Reduce

100 times faster than mapreduce Faster than traditional system
Written in scala Written in java
Supports batch/realtime/iterative and Supports only batch processing
 graph
Compact & easy Compact & lengthy
Caching enhance the system Doesn’t support caching of data
performance

2. How does spark work? Explain with the help of its architecture?

Ans :

The Spark follows the master-slave architecture. Its cluster consists of a single master and
multiple slaves.

The Spark architecture depends upon two abstractions:

o Resilient Distributed Dataset (RDD)

o Directed Acyclic Graph (DAG)

Resilient Distributed Datasets (RDD)

The Resilient Distributed Datasets are the group of data items that can be stored in-memory
on worker nodes. Here,

o Resilient: Restore the data on failure.

 o Distributed: Data is distributed among different nodes.
o Dataset: Group of data.

Directed Acyclic Graph (DAG)

Directed Acyclic Graph is a finite direct graph that performs a sequence of computations on
data. Each node is an RDD partition, and the edge is a transformation on top of data. Here,
the graph refers the navigation whereas directed and acyclic refers to how it is done.

Driver Program
The Driver Program is a process that runs the main() function of the application and creates
the SparkContext object. The purpose of SparkContext is to coordinate the spark
applications, running as independent sets of processes on a cluster.

To run on a cluster, the SparkContext connects to a different type of cluster managers and
then perform the following tasks: -

o It acquires executors on nodes in the cluster.

o Then, it sends your application code to the executors. Here, the application code can be
defined by JAR or Python files passed to the SparkContext.
o At last, the SparkContext sends tasks to the executors to run.

Cluster Manager
o The role of the cluster manager is to allocate resources across applications. The Spark is
capable enough of running on a large number of clusters.
o It consists of various types of cluster managers such as Hadoop YARN, Apache Mesos and
Standalone Scheduler.
o Here, the Standalone Scheduler is a standalone spark cluster manager that facilitates to install
Spark on an empty set of machines.

Worker Node
o The worker node is a slave node
o Its role is to run the application code in the cluster.

Executor
o An executor is a process launched for an application on a worker node.
o It runs tasks and keeps data in memory or disk storage across them.
 o It read and write data to the external sources.
o Every application contains its executor.

Task
o A unit of work that will be sent to one executor.

3. Explain the components of Spark.

Ans :

1. Spark Core
2. Spark SQL
3. Spark Streaming
4. MLlib(Machine learning library)
5. GraphX
6. Spark R

Now since we have some understanding of Spark let us dive deeper into Spark and understand the
components Apache Spark consists of. Apache Spark consists of Spark Core Engine, Spark SQL,
Spark Streaming, MLlib, GraphX, and Spark R. You can use Spark Core Engine along with any of the
other five components mentioned above. It is not necessary to use all the Spark components together.
Depending on the use case and application any one or more of these can be used along with Spark
Core.

Let us look at each of these components in detail.

Spark Core: Spark Core is the heart of the Apache Spark framework. Spark Core provides the
execution engine for the Spark platform which is required and used by other components which are
built on top of Spark Core as per the requirement. Spark Core provides the in-built memory
computing and referencing datasets stored in external storage systems. It is Spark’s core responsibility
to perform all the basic I/O functions, scheduling, monitoring, etc. Also, fault recovery and effective
memory management are Spark Core’s other important functions.

Spark Core uses a very special data structure called the RDD. Data sharing in distributed processing
systems like MapReduce need the data in intermediate steps to be stored and then retrieved from
permanent storage like HDFS or S3 which makes it very slow due to the serialization and
deserialization of I/O steps. RDDs overcome this as these data structures are in-memory and fault-
tolerant and can be shared across different tasks within the same Spark process. The RDDs can be any
immutable and partitioned collections and can contain any type of objects; Python, Scala, Java or
some user-defined class objects. RDDs can be created either by Transformations of an existing RDD
or loading from external sources like HDFS or HBase etc. We will look into RDD and its
transformations in-depth in later sections in the tutorial.

Spark SQL: Spark SQL is built on top of Shark which was the first interactive SQL on the Hadoop
system. Shark was built on top of Hive codebase and achieved performance improvement by
swapping out the physical execution engine part of the Hive. But due to the limitations of Hive, Shark
was not able to achieve the performance it was supposed to. So the Shark project was stopped and
Spark SQL was built with the knowledge of Shark on top of Spark Core Engine to leverage the power
of Spark. You can read more about Shark in the following blog by Reynold Xin, one of the Spark
SQL code maintainers.

Spark SQL is named like this because it works with the data in a similar fashion to SQL. In fact it
there is a mention that Spark SQL’s aim is to meet SQL 92 standards. But the gist is that it allows
developers to write declarative code letting the engine use as much of the data and stored structure
(RDDs) as it can to optimize the resultant distributed query behind the scenes. The goal is to allow the
user to not have to worry about the distributed nature as much and focus on the business use case.
Users can perform extract, transform and load functions on data from a variety of sources in different
formats like JSON, Parquet or Hive and then execute ad-hoc queries using Spark SQL.

DataFrame constitutes the main abstraction for Spark SQL. Distributed collection of data ordered into
named columns is known as a DataFrame in Spark. In the earlier versions of Spark SQL, DataFrames
were referred to as SchemaRDDs. DataFrame API in Spark integrates with the Spark procedural code
to render tight integration between procedural and relational processing. DataFrame API evaluates
operations in a lazy manner to provide support for relational optimizations and optimize the overall
data processing workflow. All relational functionalities in Spark can be encapsulated using the
SparkSQL context or HiveContext.

Catalyst, an extensible optimizer is at the core functioning of Spark SQL, which is an optimization
framework embedded in Scala to help developers improve their productivity and performance of the
queries that they write. Using Catalyst, Spark developers can briefly specify complex relational
optimizations and query transformations in a few lines of code by making the best use of Scala’s
powerful programming constructs like pattern matching and runtime metaprogramming. Catalyst
eases the process of adding optimization rules, data sources and data types for machine learning
domains.

Spark Streaming: This Spark library is primarily maintained by Tathagat Das and helped by
MatieZaharia. As the name suggests this library is for Streaming data. This is a very popular Spark
library as it takes Spark’s big data processing power and cranks up the speed. Spark Streaming has the
ability to Stream gigabytes per second. This capability of big and fast data has a lot of potentials.
Spark Streaming is used for analyzing a continuous stream of data. A common example is processing
log data from a website or server.

Spark streaming is not really streaming technically. What it really does is it breaks down the data into
individual chunks that it processes together as small RDDs. So it actually does not process data as
bytes at a time as it comes in, but it processes data every second or two seconds or some fixed interval
of time. So strictly speaking Spark streaming is not real-time but near real-time or micro batching, but
it suffices for a vast majority of applications.

Spark streaming can be configured to talk to a variety of data sources. So we can just listen to a port
that has a bunch of data being thrown at it, or we can connect to data sources like Amazon Kinesis,
Kafka, Flume, etc. There are connectors available to connect Spark to these sources. The good thing
about Spark streaming is it is reliable. It has a concept called “checkpointing” to store state to the disk
periodically and depending on what kind of data sources or receiver we are using, it can pick up data
from the point of failure. It is a very robust mechanism to handle all kinds of failures like disk failure
or node failure etc. Spark Streaming has exactly-once message guarantees and helps recover lost work
without having to write any extra code or adding additional configurations.

Just like how Spark SQL has the concept of Dataframe/Dataset built on top of RDD, Spark streaming
has something called Dstream. This is a collection of RDDs that embodies the entire stream data. The
good thing about Dstream is that we can apply most of the built-in functions on RDDs also on the
DStream like flatMap, map, etc. Also, the Dstream can be broken into individual RDDs and can be
processed one chunk at a time. Spark developers can reuse the same code for stream and batch
processing and can also integrate the streaming data with historical data.
MLlib: Today many companies focus on building customer-centric data products and services which
need machine learning to build predictive insights, recommendations, and personalized results. Data
scientists can solve these problems using popular languages like Python and R, but they spend a lot of
time in building and supporting infrastructure for these languages. Spark has built-in support for
doing machine learning and data science at a massive scale using the clusters. It’s called MLLib
which stands for Machine Learning Library.

MLlib is a low-level machine learning library. It can be called from Java, Scala and Python
programming languages. It is simple to use, scalable and can be easily integrated with other tools and
frameworks. MLlib eases the deployment and development of scalable machine learning pipelines.
Machine learning in itself is a subject and it may not be possible to get into details here. But these are
some of the important features and capabilities Spark MLLib offers:

 Linear regression, logistic regression

 Support Vector Machines
 Naive Bayes classifier
 K-Means clustering
 Decision trees
 Recommendations using Alternating Least Squares
 Basic statistics
 Chi-squared test, Pearsons or Spearman correlation, min, max, mean, variance
 Feature extraction
 Term Frequency/ Inverse Document Frequency useful for search

GraphX: For graphs and graph-parallel processing Apache Spark provides another API called
GraphX. The graph here does not mean charts, lines or bar graphs, but these are graphs in computer
sciences like social networks which consist of vertices where each vertex consists of an individual
user in the social network and there are many users connected to each other by edges. These edges
represent the relationship between the users in the network.

GraphX is useful in giving overall information about the graph network like it can tell how many
triangles appear in the graph and apply the PageRank algorithm to it. It can measure things like
“connectedness”, degree distribution, average path length and other high-level measures of a graph. It
can also join graphs together and transform graphs quickly. It also supports the Pregel API for
traversing a graph. Spark GraphX provides Resilient Distributed Graph (RDG- an abstraction of
Spark RDD’s). RDG’s API is used by data scientists to perform several graph operations through
various computational primitives. Similar to RDDs basic operations like map, filter, property graphs
also consist of basic operators. Those operators take UDFs (user-defined functions) and produce new
graphs. Moreover, these are produced with transformed properties and structure.

Spark R: R programming language is widely used by Data scientists due to its simplicity and ability
to run complex algorithms. But R suffers from a problem that its data processing capacity is limited to
a single node. This makes R not usable when processing a huge amount of data. The problem is
solved by SparkR which is an R package in Apache Spark. SparkR provides data frame
implementation that supports operations like selection, filtering, aggregation, etc. on distributed large
datasets. SparkR also has support for distributed machine learning using Spark MLlib.