Lab Manual Big Data Analytics Lab (LC-CSE-410G) : Department of Computer Science and Engineering
Lab Manual Big Data Analytics Lab (LC-CSE-410G) : Department of Computer Science and Engineering
1
Check list for Lab Manual
S. No. Particulars
2 Course Outcomes
2
Department of Computer Science & Engineering
3
1. Students should be regular and come prepared for the lab practice.
2. In case a student misses a class, it is his/her responsibility to complete that missed
experiment(s).
3. Students should bring the observation book, lab journal and lab manual. Prescribed textbook
and class notes can be kept ready for reference if required.
4. They should implement the given Program individually.
5. While conducting the experiments students should see that their programs would meet the
following criteria:
• Programs should be interactive with appropriate prompt messages, error messages if any,
and descriptive messages for outputs.
• Programs should perform input validation (Data type, range error, etc.) and give
appropriate error messages and suggest corrective actions.
• Comments should be used to give the statement of the problem and every function should
indicate the purpose of the function, inputs and outputs
• Statements within the program should be properly indented
• Use meaningful names for variables and functions.
• Make use of Constants and type definitions wherever needed.
6. Once the experiment(s) get executed, they should show the program and results to the
instructors and copy the same in their observation book.
7. Questions for lab tests and exam need not necessarily be limited to the questions in the
manual, but could involve some variations and / or combinations of the questions.
4
List of Experiments
Sr.
No. Experiment Name
Page No.
Installation of Hadoop Framework, it‘s components and study the
1 HADOOP ecosystem.
2 Write a program to implement word count program using MapReduce
Experiment on Hadoop Map-Reduce / PySpark: -Implementing simple
3 algorithms in Map-Reduce: Matrix multiplication.
Install and configure MongoDB/ Cassandra/ HBase/ Hypertable to execute
4 NoSQL Commands.
Implementing DGIM algorithm using any Programming Language/ Implement
5 Bloom Filter using any programming language
Implement and Perform Streaming Data Analysis using flume for data
6 capture, PYSpark / HIVE for data analysis of twitter data, chat data, weblog
analysis etc.
7 Implement any one Clustering algorithm (K-Means/CURE) using Map-Reduce.
8 Implement Page Rank Algorithm using Map-Reduce.
5
Experiment No. 1
Aim: Installation of Hadoop Framework, it‘s components and study the HADOOP
ecosystem
Hadoop is an open-source framework that allows to store and process big data in a distributed
environment across clusters of computers using simple programming models. It is designed to
scale up from single servers to thousands of machines, each offering localcomputation and
storage.
Hadoop Architecture:
Hadoop Common: Contains Java libraries and utilities needed by other Hadoop modules.
These libraries give file system and OS level abstraction and comprise of the essential Java
files and scripts that are required to start Hadoop.
Hadoop Distributed File System (HDFS): A distributed file-system that provides high-
throughput access to application data on the community machines thus providing very high
aggregate bandwidth across the cluster.
Hadoop MapReduce: This is a YARN- based programming model for parallel processing of
large data sets.
6
Hadoop Ecosystem:
Hadoop has gained its popularity due to its ability of storing, analyzing and accessing large
amount of data, quickly and cost effectively through clusters of commodity hardware. It won‘t
be wrong if we say that Apache Hadoop is actually a collection of several components and not
just a single product.
With Hadoop Ecosystem there are several commercial along with an open source products
which are broadly used to make Hadoop laymen accessible and more usable.
MapReduce
Hadoop MapReduce is a software framework for easily writing applications which process big
amounts of data in-parallel on large clusters of commodity hardware in a reliable, fault- tolerant
manner. In terms of programming, there are two functions which are most common in
MapReduce.
• The Map Task: Master computer or node takes input and convert it into divide it into
smaller parts and distribute it on other worker nodes. All worker nodes solve their own
small problem and give answer to the master node.
7
• The Reduce Task: Master node combines all answers coming from worker node and
forms it in some form of output which is answer of our big distributed problem.
Generally both the input and the output are reserved in a file-system. The framework is
responsible for scheduling tasks, monitoring them and even re-executes the failed tasks.
HDFS is a distributed file-system that provides high throughput access to data. When data is
pushed to HDFS, it automatically splits up into multiple blocks and stores/replicates the data
thus ensuring high availability and fault tolerance.
Note: A file consists of many blocks (large blocks of 64MB and above).
• Name Node: It acts as the master of the system. It maintains the name system
i.e.,directories and files and manages the blocks which are present on the Data Nodes.
• Data Nodes: They are the slaves which are deployed on each machine and provide
theactual storage. They are responsible for serving read and write requests for the
clients.
• Secondary Name Node: It is responsible for performing periodic checkpoints. In
theevent of Name Node failure, you can restart the Name Node using the checkpoint.
Hive
Hive is part of the Hadoop ecosystem and provides an SQL like interface to Hadoop. It is a
data warehouse system for Hadoop that facilitates easy data summarization, ad-hoc queries,
and the analysis of large datasets stored in Hadoop compatible file systems.
HBase is a distributed, column oriented database and uses HDFS for the underlying storage.
As said earlier, HDFS works on write once and read many times pattern, but this isn‘t a case
always. We may require real time read/write random access for huge dataset; this is where
HBase comes into the picture. HBase is built on top of HDFS and distributed on column-
oriented database.
8
Installation Steps –
Step1. Install Java (OpenJDK) - Since hadoop is based on java, make sure you have java jdk
installed on the system. Please check the version of java (It should be 1.7 or above it)
$ java –version
If it returns "The program java can be found in the following packages", If Java isn't been
installed yet, so execute the following command:
$sudo apt-get install default-jdk
1. Open
$sudo geditbashrc file in geditor
~/.bashrc
$source ~/.bashrc
File->new->other->MapReduce project
Step 6: Copy Hadoop packages such as commons-io-2.4.jar commons-lang3-3.4.jar in src file of
MapReduce project
Step 8: Copy Log file log4j.properties from src file of hadoop in src file of MapReduce project
Hadoop is powerful because it is extensible and it is easy to integrate with any component. Its
popularity is due in part to its ability to store, analyze and access large amounts of data, quickly
and cost effectively across clusters of commodity hardware. Apache Hadoop is not actually a
single product but instead a collection of several components. When all these components
are merged, it makes the Hadoop very user friendly.
Experiment No. 2
Aim: Implementing distinct word count problem using Map-Reduce
After this, "aggregation" and "Shuffling and Sorting" done by framework. Then
Reducers task these final pair to produce output.
The second input2 data file (input2.txt : Hello Hadoop Goodbye Hadoop) mapper emits:
<Hello, 1>
<Hadoop, 1>
<Goodbye, 1>
<Hadoop, 1>
WordCount also specifies a combiner. Hence, the output of each map is passed through the
local combiner (which is same as the Reducer as per the job configuration) for local
aggregation, after being sorted on the keys.
The Reducer implementation via the reduce method just sums up the values, which are the
occurence counts for each key (i.e. words in this example).
File1.txt
Output
DWC
Experiment No. 3
Aim: Implementation of Matrix Multiplication using MapReduce.
Definitions:
P is a matrix = MN with element pik in row i and column k, where pik =∑j mijnjk
Mapper function does not have access to the i, j, and k values directly. An extra MapReduce
Job has to be run initially in order to retrieve the values.
For each element mij of M, emit a key-value pair (i, k), (M, j, mij ) for k = 1, 2, . . .number of
columns of N.
For each element njk of N, emit a key-value pair (i, k), (N, j, njk) for i = 1, 2, . . . number of
rows of M.
For each key (i, k), emit the key-value pair (i, k), pik where, Pik = ∑jmij * njk
The product MN is almost a natural join followed by grouping and aggregation.That is, the
natural join of M(I, J, V ) and N(J,K,W), having only attribute J in common, would produce
tuples (i, j, k, v,w) from each tuple(i, j, v) in M and tuple (j, k,w) in N.
This five-component tuple represents the pair of matrix elements (mij ,njk).What we want
instead is the product of these elements, that is, the four-component tuple (i, j, k, v × w), because
that represents the product mijnjk. Once we have this relation as the result of one MapReduce
operation, we can perform grouping and aggregation, with me and K as the grouping attributes
and the sum of V × W as the aggregation. That is, we can implement matrix multiplication as
the cascade of two MapReduce operations, as follows.
The input file contains two matrices M and N. The entire logic is divided into two parts:
• When compared to relational databases, NoSQL databases are more scalable and
provide superior performance, and their data model addresses several issues that the
relational model is not designed to address:
• Large volumes of rapidly changing structured, semi-structured, and unstructured data.
• Agile sprints, quick schema iteration, and frequent code pushes.
• Object-oriented programming that is easy to use and flexible.
• Geographically distributed scale-out architecture instead of expensive, monolithic
architecture.
MongoDB
It is an open-source document database, and leading NoSQL database. MongoDB is written
in c++. It is a cross-platform, document oriented database that provides, high performance,
high availability, and easy scalability. MongoDB works on concept of collection and document.
Output
mongod start/running, process 1611
You can also stop, start, and restart MongoDB using the service command (e.g. service
mongod stop, service mongod start).
Commands
MongoDB use DATABASE_NAME is used to create database. The command will create a
new database, if it doesn't exist otherwise it will return the existing database.
Syntax:
Basic syntax of use DATABASE statement is as follows:
use DATABASE_NAME
Syntax:
db.dropDatabase()
This will delete the selected database. If you have not selected any database, then it will
delete default 'test' database
Syntax:
db.createCollection(name, options)
To query data from MongoDB collection, you need to use MongoDB'sfind() method.
Syntax
>db.COLLECTION_NAME.find().
MongoDB'supdate() and save() methods are used to update document into a collection. The
update() method update values in the existing document while the save() method replaces the
existing document with the document passed in save() method.
MongoDBUpdate() method
Syntax
>db.COLLECTION_NAME.update(SELECTIOIN_CRITERIA, UPDATED_DATA)
Experiment No. : 5
Aim: Write a program to implement bloom filtering
A Bloom filter is a space-efficient probabilistic data structure that is used to test whether an element is
a member of a set. For example, checking availability of username is set membership problem, where
the set is the list of all registered username. The price we pay for efficiency is that it is probabilistic in
nature that means, there might be some False Positive results.
False positive means, it might tell that given username is already taken but actually it’s not.
Interesting Properties of Bloom Filters
• Unlike a standard hash table, a Bloom filter of a fixed size can represent a set with an arbitrarily
large number of elements.
• Adding an element never fails. However, the false positive rate increases steadily as elements are
added until all bits in the filter are set to 1, at which point all queries yield a positive result.
• Bloom filters never generate false negative result, i.e., telling you that a username doesn’t exist
when it actually exists.
• Deleting elements from filter is not possible because, if we delete a single element by clearing
bits at indices generated by k hash functions, it might cause deletion of few other elements.
Example – if we delete “geeks” (in given example below) by clearingbit at 1, 4 and 7, we might
end up deleting “nerd” also Because bit at index 4 becomes 0 and bloom filter claims that “nerd”
is not present.
Working of Bloom Filter
A empty bloom filter is a bit array of m bits, all set to zero, like this –
We need k number of hash functions to calculate the hashes for a given input. When we
want to add an item in the filter, the bits at k indices h1(x), h2(x), … hk(x)are set, where
indices are calculated using hash functions.
Example – Suppose we want to enter “geeks” in the filter, we are using 3 hash
functions and a bit array of length 10, all set to 0 initially. First we’ll calculate the
hashes as following :
h1(“geeks”) % 10 = 1
h2(“geeks”) % 10 = 4
h3(“geeks”) % 10 = 7
Note: These outputs are random for explanation only.
Now we will set the bits at indices 1, 4 and 7 to 1
Now if we want to check “geeks” is present in filter or not. We’ll do the same process but this
time in reverse order. We calculate respective hashes using h1, h2 and h3 and check if all these
indices are set to 1 in the bit array. If all the bits are set thenwe can say that “geeks” is
probably present. If any of the bit at these indices are 0 then “geeks” is definitely not present.
False Positive in Bloom Filters
The question is why we said “probably present”, why this uncertainty. Let’s understand this
with an example. Suppose we want to check whether “cat” is presentor not. We’ll calculate
hashes using h1, h2 and h3
h1(“cat”) % 10 = 1
h2(“cat”) % 10 = 3
h3(“cat”) % 10 = 7
If we check the bit array, bits at these indices are set to 1 but we know that “cat” wasnever
added to the filter. Bit at index 1 and 7 was set when we added “geeks” and bit3 was set we
added “nerd”.
So, because bits at calculated indices are already set by some other item, bloom filter
erroneously claim that “cat” is present and generating a false positive result. Depending on the
application, it could be huge downside or relatively okay.
We can control the probability of getting a false positive by controlling the size of the Bloom
filter. More space means fewer false positives. If we want decrease probability of false
positive result, we have to use more number of hash functionsand larger bit array. This
would add latency in addition of item and checking membership.
Experiment No. 6
Data Streaming is a technique for transferring data so that it can be processed as a steady and
continuous stream. Streaming technologies are becoming increasingly important with the
growth of the Internet.
We can use Spark Streaming to stream real-time data from various sources like Twitter,
Stock Market and Geographical Systems and perform powerful analytics to help businesses.
Spark Streaming is used for processing real-time streaming data. It is a useful addition to the
core Spark API. Spark Streaming enables high-throughput and fault-tolerant streamprocessing
of live data streams.
1. Streaming Context
2. DStream
3. Caching
4. Accumulators, Broadcast Variables and Checkpoints
Streaming Context
Streaming Context consumes a stream of data in Spark. It registers an Input DStream
to produce a Receiver object. It is the main entry point for Spark functionality. Spark
provides a number of default implementations of sources like Twitter, Akka Actor and
ZeroMQ that are accessible from the context.
DStream
Caching
DStreams allow developers to cache/ persist the stream‘s data in memory. This is useful
if the data in the DStream will be computed multiple times. This can be done using the
persist() method on a DStream
Accumulators: Accumulators are variables that are only added through an associative and
commutative operation. They are used to implement counters or sums. Tracking
accumulators in the UI can be useful for understanding the progress of running stages. Spark
natively supports numeric accumulators. We can create named or unnamed accumulators.
Broadcast Variables: Broadcast variables allow the programmer to keep a read-only variable
cached on each machine rather than shipping a copy of it with tasks. They can be used to give
every node a copy of a large input dataset in an efficient manner. Spark also attempts to
distribute broadcast variables using efficient broadcast algorithms to reducecommunication
cost.
Checkpoints: Checkpoints are similar to checkpoints in gaming. They make it run 24/7 and
make it resilient to failures unrelated to the application logic.
Data clustering is the partitioning of a data set or sets of data into similar subsets. During
the process of data clustering a method is often required to determine how similar one
object or groups of objects is to another. This method is usually encompassed by some
kind of distance measure. Data clustering is a common technique used in data analysis and
is used in many fields including statistics, data mining, and image analysis. There aremany
types of clustering algorithms. Hierarchical algorithms build successive clusters using
previously defined clusters. Clustering algorithms can also be partitioned meaning they
determine all clusters at once.
K-Means Algorithm
In this problem, we have considered inputs a set of n 1-dimensional points and desired
clusters of size 3.
Once the k initial centers are chosen, the distance is calculated (Euclidean distance) from
every point in the set to each of the 3 centers & point with the corresponding center is
emitted by the mapper. Reducer collect all of the points of a particular centroid and
calculate a new centroid and emit.
Termination Condition:
When difference between old and new centroid is less than or equal to 0.1
1. Algorithm
1. Initially randomly centroid is selected based on data. (e,g. 3 centroids)
2. The Input file contains initial centroid and data.
3. Mapper is used to first open the file and read the centroids and store in the data
structure(we used ArrayList)
4. Mapper read the data file and emit the nearest centroid with the point to the
reducer.
5. Reducer collect all this data and calculate the new corresponding centroids and
emit.
6. In the job configuration, we are reading both files and checking
if
difference between old and new centroid is less than 0.1 then
convergence is reached
else
repeat step 2 with new centroids.
Experiment No. 8
Aim: Implement Page Rank algorithm using Map Reduce.
PageRank is a link analysis algorithm and it assigns a numerical weighting to each element
of a hyperlinked set of documents, such as the World Wide Web, with the purpose of
measuring its relative importance within the set. The algorithm may be applied to any
collection of entities with reciprocal quotations and references. The numerical weight that
it assigns to any given element E is referred to as the PageRank of E and denoted by PR(E).
Other factors like Author Rank can contribute to the importance of the family. A hyperlink
to a page counts as a vote of support. The PageRank of a page is defined recursively and
depends on the number and PageRank metric of all pages that link to it(―incoming links‖).
A page that is linked to by many pages with high PageRank receives a high rank itself.
Formula
Eqn.1 is the formula to calculate the rank value for each webpage. We will learn this
formula by applying it to the case in Fig.1. There are 11 webpages in Fig.1, which include:
{A, B, C, D, E, F, G1, G2, G3, G4, G5}. Assuming the probability distribution for a web
surfer accessing all these 11 pages in current iteration is {PR(A), PR(B), PR(C),
… PR(G5)}, then the probability for the surfer to access Page B in the next iteration is:
PR(B) = PR(D)/2 + PR(E)/3 + PR(F)/2 + PR(C) + PR(G1)/2 + PR(G2)/2 + PR(G3)/2
In a general case, the PageRank value for any page u can be expressed as:
....Eqn.1
The vertices seen in the right of the formula contain all the webpages that point to target
webpage ‗u‘. The L(v) refers to the out degree of each webpage in the vertices set. The
initial rank values of each webpage, like PR‘(u), can be any double value. After several
iteration calculations, the rank values converge to the stationary distribution regardless of
what their initial values are.
Damping factor
The PageRank theory holds that even an imaginary surfer who is randomly clicking on
links will eventually stop clicking. The probability, at any step, that the person will continue
is a damping factor d. Various studies have tested different damping factors, but it is
generally assumed that the damping factor will be around 0.85. The formula considering
damping factor is shown in Eqn.2. N refers to the total number of uniqueurls.
...... Eqn.2
Example: