MODULE 1:

INTRODUCTION
Linux & bash shell
I. Linux

2
 What is Linux?
Linux is an operating system (OS) developed
by Linus Torvalds in 199

Based on UNIX - developed in response to Tux

closing legal loophole that made UNIX fre

Many “distributions” - Fedora, RedHat,

CentOS, Debian, Ubunt
Linus Torvalds
Typically free and open source GNU licensin

Command line interface (CLI) and graphical

desktop environments (GDE)
Richard Stallman 3
u

 Why Linux?
Developed by Bell Labs in 1969, and initially free, UNIX
was quickly adopted as de facto scienti c computing O

Powerful CLI enables direct low level access

GDE provides simplicity and usabilit

Free and open source makes code development eas

Linux is everywhere
- 90% of supercomputers run Linux (incl. Blue Waters)
- Android OS is based on a Linux kernel
- Ubuntu distro is the most popular OS in the world

4
y

fi
y

 Can’t I use Windows / Mac OS X?

Maybe

Some software have Mac OS X / Windows / Windows +

Cygwin versions to install on your local machin

Remote login via Mac OS X terminal / [Windows +

Cygwin / Putty] to SSH into EWS Linu

A key learning objectives of this course is to develop

familiarity and competence using Linux - Bon Courage!

5
.

 What distro are we using?

EWS Linux machines run CentOS 7

6
II. bash shell

7
 The command line
CLI and GDE offer alternatives to interact with a machin

Switching to a CLI can be very intimidating for new users

CLI interaction is powerful, concise, and ef cien

CLI scripting enables task automatio

e.g. Download of 1500 daily NASDAQ stock prices

GDE: Point and click le download extremely tedious!
CLI Trivially automated using CLI wget loop

8
:

fi
n

fi
t

 bash shell
“Command line interpreters”
or “shells” convert text inputs
into OS command

Many avors: sh, bash, ksh, csh,

zsh, tcs

The bash shell (“Bourne-again

shell”) is one of the most
popular, and the default on
many Linux distros

9
fl
h

III. bash basics

10
 bash: basics
Pop a bash terminal by clicking on
or navigating Applications è Accessories è Termina

pwd - show path to present working director

ls - list contents of current director

ls -alh - list all contents of cdir in long form with
human readable le size
ls /sw/q - list contents of directory /sw/

cd <path> - change directory into <path

cd .. - change directory up one leve
cd ../.. - change directory up two levels
11


fi
s

>

 bash: basics
touch <file> - make new le < le> or update last
access of existing l
mkdir <dir> - make director

chmod 755 <file> - change le permissions to

r+w+x (user), r+x (group, world)
chmod 644 <file> - change le permissions to
r+w (user), r (group, world)
[N.B. r=4, w=2, x=1

var=ferrari42 - assign ferrari42 to var

echo $var - print $var
12



fi
fi
fi
y

fi
fi
e

 bash: basics
./<execFile> - execute execFile in cdir
<path>/<execFile> - execute execFile in pat

which <cmd> - location of command cm

clear - clear termina

wget -O <file> <url> - download url data into le

e.g wget -O myProf.png http://bit.ly/2jt9NAl

13
.
l

fi
d

 bash: basics
cp <source> <target> - copy le source to target
e.g. cp myFile /apps/doc/

cp -r <source> <target> - copy recursively

(copy source directory and everything in it)
e.g cp -r myDir ./dir1/dir2/

mv <source> <target> - move source to target

(same for les and directories

rm <file> - remove le
rm -r <directory> - remove recursively directory
14
.

fi
fi
fi

)

 bash: safety!
cp / rm / mv - These do exactly what you ask
They do not ask for permission
Furthermore, there is no Trash/Recycling
Once you remove / overwrite a le, it’s gone
Standard “safety” choices: use alias in your .bashrc
alias cp=‘cp -i’
alias rm=‘rm -i’
alias mv=‘mv -i’
setopt noclobber
You don’t have to do this, but you may breathe a little
easier with some safety.

15
fi
.

 bash: basics
whoami - show your login username
who - show everyone currently logged in

cat <file> - show le content

less <file> - show le contents (spacebar ê, b é)

head <file> - show head of le

tail <file> - show tail of le
tail -n <nLines> <file> - show tail nLines of le
tail -f <file> - show tail of le and follow

16
fi
fi
fi
fi
fi
s

fi
 bash: basics
zip <archive> <file1 file2 ...>
- create zip le archive.zip containing le1, le2, ...
unzip <archive>
- unzip zip le archive.zi

tar cvzf <archive.tgz> <file1 file2 ...>

- create gzip compressed tape archive archive.tgz
containing le1, le2, ...
tar xvzf <archive.tgz>
- uncompress end extracted compressed tape
archive archive.tgz
17
fi
fi
fi
fi
p

fi
fi
 bash: basics
top - show active processes
top -o cpu - show active processes ordered by cpu %
top -U <usr> - show active processes owned by us

grep <str> <file> - return lines in le containing

string st

find <path> -name <*str*> -print

- print all les in path containing str in their nam

<cmd> --help - help for cmd

man <cmd> - manual for cmd (spacebar ê, b é
Google is your friend for bash help! 18
fi

r

fi
e

 bash: special symbols

~ - your home director

. - current director

.. - directory one level u

* - wildcard characte

\ - escape succeeding character

e.g mkdir My\ Directory

| - pipe
e.g. cat <file> | grep tungsten 19
.

y

 bash: special symbols

> - redirect standard output and overwrite
>> - redirect standard output and append
e.g. echo “Today was great!” >> myDiary.txt

$var - dereference variable va

“ ” - enclose text string but expand $

‘ ’ - enclose text string but do not expand $
e.g. myVar=“My String With Spaces”
echo “This is $myVar”

`<stuff>` - execute stuff rst

e.g. echo `expr 1 + 1` 20


fi
r

IV. bash utilities

21
 bash: integer arithmetic
expr - integer arithmetic engin

e.g

$ echo `expr 1 + 1`
2

$ var1=`expr 10 \* 2`
$ var2=`expr 21 / 7`
$ echo $var1 $var2 `expr $var1 / $var2`
20 3 6

22
.

 bash: quick calculator

bc -l - arbitrary precision calculator (w/ math lib
$ bc -l
2/3
.66666666666666666666
2^3
8
e(1)
2.71828182845904523536
pi=a(1)*4
pi
3.14159265358979323844
s(pi/6)
.49999999999999999999
c(pi/6)
.86602540378443864676
23

 bash: ssh & scp

SSH CLI remote login is supported by ssh (secure shell)
ssh <user>@<hostname> - login to host
ssh -Y <user>@<hostname> - login to host w/
secure X forwarding (use this to get graphics via SSH!

N.B. For EWS, hostname=remlnx.ews.illinois.edu

SCP CLI le transfers supported by scp (secure copy)

scp <src> <user>@<hostname>:<target>
- upload
scp <user>@<hostname>:<src> <target>
- download

24
fi
)

 bash: ssh & scp

ssh and scp are prepackaged with Linux / Mac OS X and
are accessible directly from the bash termina

On Windows, you need to download a third party ssh

client in order to make a ssh connection with EWS

https://answers.uillinois.edu/
www.putty.org
illinois.engineering/page.php?id=81727
25
l

 bash: sftp
SFTP more sophisticated alternative to scp
(secure file transfer protocol)
sftp <user>@<hostname> - login to host
ls - remote ls
lls - local ls
pwd - remote pwd
lpwd - local pwd
cd - remote cd
lcd - local cd
get <file> - download le
put <file> - upload le
quit - logout
26
fi
fi
 bash: vi/vim
Two built-in CLI text editors: vi/vim & emacs
Seem slow and painful, but invaluable for on-the- y edit

Use whichever you prefer, I use both.

(It is very fashionable to argue over which is better...
vi/vim is fast for text manipulation, uses two mode
emacs is has lots of built-in modules, more “Word”-lik

Two-modes navigation for moving

insertion for editin
Nav mode is the default mode, and can be accessed by
hitting Esc

Ins mode is accessed by hitting i 27

:

g

fl
)

 bash: vi/vim
Nav mod
éêçè - single char / single line movemen
gg - go to top of le
^ - go to beginning of lin
$ - go to end of lin
<n>G - go to line
w - skip forward one wor
b - skip backward one wor

yy or y$ - copy current lin

y<n>ê - copy next n lines
p - paste
28
e

fi
e

 bash: vi/vim
Nav mod
x - delete characte
o - create new line below and enter insert mod

i - enter insert mode to left of current characte

I - enter insert mode at beginning of lin
a - enter insert mode to right of current characte
A - enter insert mode at end of lin

29
e

 bash: vi/vim
Nav mod
dd or d$ - delete current lin
d<n>w - delete next n word
d<n>ê - delete next n line

u - und
Ctrl+r - red

30
e

 bash: vi/vim
Nav mod
/<str><Enter> - search forward for st
?<str><Enter> - search backward for st

n - go to next matc
<N>n - go to Nth matc

31
e

 bash: vi/vim
Nav mod
:w - writes le
:w! - writes le even if read only

:q - quit
:q! - quit and don't question me
(good way to mess things up)

:wq - write quit

:wq! - write quit and don't question me
(very good way to mess things up)

32
e

fi
fi
 bash: vi/vim
Ins mod

Type normally - what you enter appears on screen

éêçè work as in nav mod

Hit Esc to get back to nav mode

33
e

 bash: .bash_pro le & .bashrc

Hidden les start with .

~/.bashrc is executed for every new termina

~/.bash_profile is executed when you login

(~/.bash_profile calls ~/.bashrc

These les are useful to store aliases and modify PATH

N.B. On some systems ~/.bash_profile is replaced

by ~/.profile
34
fi
fi
fi
)

 bash: .bash_pro le & .bashrc

(i) Use vi to add lls as alias for ls -al to .bashr

$ vi ~/.bashrc edit .bashrc

$ G go to end of le
$ o edit line belo
$ alias lls="ls -l" add alia
$ Esc escape to navigate mod
$ :wq write and quit

35
fi
s

fi
c

bash: .bash_pro le & .bashrc

36
fi
 bash: .bash_pro le & .bashrc
(ii) Use vi to add ~/local/bin to your PATH in .bashr

$ vi ~/.bashrc edit .bashrc

$ G go to end of le
$ o edit line belo
$ export PATH=$PATH:~/local/bin add to PAT
$ Esc escape to navigate mod
$ :wq write and quit

37
fi
w

fi
c

bash: .bash_pro le & .bashrc

38
fi
 bash: installing software
Typical anatomy of an installation from source

$ wget <app_url> download

$ tar xvzf <app.tgz> uncompress
$ cd ./app
$ ./configure --prefix=<location>
con gure and specify locatio
$ make compile
$ make install instal

39
fi
l
:

V. bash scripting

40
 What is bash scripting?
A bash script is nothing more than a list of bash
commands in an executable text l

Exactly the same behavior could be achieved by copying

and pasting the script into the bash shel

Extremely powerful way to automate system tasks

e.g le downloads
system backups
job submission
le processing

41
.
fi
fi
fi
e

 Anatomy of a script
A script is nothing more than a text le
- write using vi, emacs, Notepad, or favorite text editor

the “sha-bang” line

comments (start with #)

list of bash commands

42
fi
 Script 1: hello world!
$ touch helloWorld new script le
$ chmod 755 helloWorld making executable
$ vi helloWorld edit line belo
$ i enter insert mode
$ #!/bin/bash <Enter>
$ # this is my hello world script <Enter>
$ echo “Hello World!”
$ Esc escape to navigate mod
$ :wq write and quit

43
fi
w

 Script 2: backup
Passing variables $1, $2, $3, ...

Placing all les in current directory into a

compressed tape archive bkp.tg

Renaming bkp.tgz bkp_<arg>.tgz where

arg is the rst argument in the call to the
executable

44
fi
fi
z

 Script 3: summer
while loo arithmetic comparisons
$# and shift
Initializing sum to

while loop - run loop while the variable

$# is greater than

- $# = number of parameters in exec cal

- shift = kick out $1 and shift rest down

(i.e. $1 ç $2, $2 ç $3, $3 ç $4, ...

- arithmetic comparisons:
-l
-g
-l <
-g >
-e =
-n !=
45
t
e
t
e
q
e
<

>

 Script 4: oracle
if/else statemen
nesting

if loo

nested if loo

- can also use the construct

if [ ] ; the
elif [ ] ; the
elif [] ; the
els

46
f
e

 Script 5: calculator
case conditional
exit
defensive programming

safeguard on usag
- exit terminates scrip

case conditiona
- starts case, ends esac
- ) terminates pattern matc
- ;; terminates each cas
- | is the “or” characte
- * is the wildcard “catch all”
47
l

Script 5: calculator

48
 Script 6: stringer
array
$@
Create an array strArray from parameter
- $@ = all parameters passed to bash cal
- ${ARRAY[@]} = array content

Create empty array leArra

For all strings except “virus” append txt

and store in leArra
- ${#ARRAY[@]} = array siz
- “” terminates $ dereference strin
- str comparisons
equal
! not equa
greater tha
< less than
-n <str not empt
-z <str empt 49
=
>
=

s

>
>
fi
:
y

fi
y
l

y
n

 Script 7: ler
in nite loo read user input
Ctrl + C

In nite loop (Ctrl + C to break

Read user input into st
Test if str is a regular le in the
present working director

- le comparison operators
- le exists (may be directory
- le exists (not directory
- directory exist
- le readabl
- le writabl
- le executable

50
fi
fi
e
f
d
r
w
x
fi
fi
fi
fi
fi
fi
e

fi
fi
r

 Script 8: squarer
iteratin sleep
functions

Declaring a function at top of scrip

As for main function $1,$2,... are passed variable

Setting up the iterative loo

Performing square using our functio

sleep 0.5 = 0.5 s pause between print
incrementing loop variable

51
g

VI. awk

52
 awk
awk is a programming language in its own righ

Developed at Bell Labs in 70’s by Aho, Weinberger, &

Kernigha

Powerful, simple, fast and exible languag

Standard part of most Linux distributions, used primarily

for rapid and ef cient line-by-line text processing

53
n

fi
fl
e

 Why awk?
“Forget awk, I’ll just use vi / emacs / Notepad!
OK, good luck...
- extract the third column of this 50,000 line le
- divide the second eld of each line by the seventh, and
save results in csv format
- extract every 15th line of this le and invert the eld
ordering to run from last to rs

awk can do these things (and many others!) extremely

ef ciently and quickly using “one liner” commands
integrates seamlessly into bash shell cat <file> | awk ...
integrates seamlessly into bash script
great power using only a handful of command
is simply the “right tool” for many text processing jobs 54
fi
fi
fi
fi
t

fi
s

fi
 awk basics
Rudimentary awk, comprehensive beginner’s tutorial at:
http://www.grymoire.com/Unix/Awk.html

Anatomy of an awk progra

awk 'BEGIN { ... Do stuff before starting [optional

{ ... Line-by-line processin
END { ... }' Do stuff after end-of- le [optional
inFile > outFil Read from inFile, write to outFile

Can place within a script, or enter directly into termina

White space doesn’t matte

55
}

fi
g

 awk basics
Alternatively, can pipe input from termina
cat inFile > awk 'BEGIN { ...
{ ...
END { ... }
> outFil

Omit “> outFile” to output directly to termina

Use “>>” instead of “>” to append rather than overwrit

cat inFile > awk 'BEGIN { ...
{ ...
END { ... }
>> outFile
56
}

'

'

 What goes in the { }?

Commands perform line-by-line text processin

Assignment of internal awk variables

Flow control and loops

Pulling in of bash variables from surrounding scrip

Printing to terminal or l

Basic arithmeti

57
c

fi
e

 [bash + awk] script example

Extract the x,y,z coordinates of peptide atoms from pdb
formatted les peptide[1-3].pdb into coords[1-3].txt

Concatenate coords[1-3].txt into coords_concat.txt

Use bash to iterate over le

Use awk to perform text processing

58
fi
fi
s

 [bash + awk] script example

setting up in and out le

setting up iteratio

initializing concat l

awking each le in tur

- printf is formatted print
- formatting like Matla
- $n are eld code
- NR is a special variable
for number of records
= number of line

cat each le into concat l

rm each coordsX.txt le

increment iterator
59
fi
fi
fi
fi
n

fi
fi
n

fi
e

 [bash + awk] script example

Doing this [by hand / in Excel / in Matlab] at any signi cant

scale would be extremely tedious and error prone!
60

fi