Scribd
Upload
9 views5 pages

Big Data Analytics Digital Assignment 1

Uploaded by

pavan karthik
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
9 views5 pages

Big Data Analytics Digital Assignment 1

Uploaded by

pavan karthik
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 5

BIG DATA ANALYTICS

DIGITAL ASSIGNMENT 1
K PAVAN KARTHIK
17MIS1038
Prof: Anita X
Topic: Installing pyspark on Local Machine

• I am working on Linux environment and I have already installed it


on my pc and I am going to show the step process to install it

1. Install Packages Required for Spark


• SCALA
• JDK

Install Java and Scala: sudo apt install default-jdk Scala git –y
2. Download and Set Up Spark on Ubuntu

• Wget is the command for downloading the packages from internet


Command: wget https://downloads.apache.org/spark/spark-3.0.1/spark-3.0.1-bin-hadoop2.7.tgz

1. After Downloading,it will be in the form of” tar.gz” format we need to


extract them for extracting the command is:
tar xvf spark-version

2. We need to move to the other directory for running the spark so the new
directory is “/opt/spark” command is

sudo mv spark-3.0.1-bin-hadoop2.7 /opt/spark

After that we need to append the code in bashrc file


“export SPARK_HOME=/opt/spark
export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbinexport
PYSPARK_PYTHON=/usr/bin/python3”
Above three lines of code needed to added in bashrc file so that it will be working
in command promt when we call
To locate the bashrc file we should have root access and with the text editor we can
access it by the command
Subl bashrc
Installation Completed Now we can run the pyspark with the command pyspark

Result:

Small Exercise:
1. Loaded the csv file and used sql query
2. Loaded the file

a. Convert the file into database and use sql commands

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy