Good practices for writting shell

scripts
Good practices for writing shell
scripts | Yoann Bentz <Yoone.eu>
I have seen so many messy shell scripts in my not so long life, and the reasons are always
the same: “I’m the only one to use it”, “It’s meant to be used once”, etc. The problem is that
in practice, those scripts are often reused or dug up months or even years later because
they solved a problem similar to one you are having now. And when you have to understand
a shell script that has been running in production for years and has been written by someone
else, you will have to go through the famous “dusting” part i.e. understanding the code and,
for the most courageous, cleaning it up before thinking of editing anything.

In reality you can almost never expect a developer to provide a test suite for his or her shell
scripts. That is because writing them is often a time sensitive task and adding tests would
lengthen that task quite a bit. It can also be because the scripts themselves interact with too
many components available in production which makes them untestable without having a
whole environment set up for the sole purpose of testing them.

To try and avoid having issues for maintaining or exploiting shell scripts, I am going to point
out a few “good” practices that I try to follow myself. I obviously can’t make everybody agree
on all of them, but I believe that they could ease the work of a lot of developers if they were
followed.

This article was written with the help of Vincent Charpentier who contributed by giving me
some precious advice on shebangs and set .

Shebang and file extension

The first thing I would like to address is the shebang. It is that line starting with #! at the
beginning of the script, and it is used to tell your shell which program it should be executed
by. First of all, not having one can cause really big issues while executing your script,
depending on where and by whom it is executed. You could always feed the script to some
shell using /bin/sh myscript.sh , but what if it uses some specific bash or zsh syntax?

When adding a shebang, you might be tempted to add the absolute path of the shell binary
you want. I will advise against it, in favor of letting env find it for you because it could be
situated somewhere else on some other operating system your script could be executed on.
\
Last but not least, don’t just use the shell with the most features you have installed on your
OS. Keep in mind that while most systems have sh and bash by default, they might not have
zsh, or any other shell you might want to use. Start with sh and change it to bash if you need
some more “advanced” or non-POSIX syntax.

#!/usr/bin/env sh
# ... to start#!/usr/bin/env bash# ... if sh is not enough#!/usr/bin/env
zsh# ... what, isn't bash enough? :)

On an important note, don’t pass arguments to your shell in the shebang. It will result in
an undesired behavior as everything following the first whitespace will be considered as the
first argument. It means that the command is only split in two. Here is some commented
code with an example and the solution to the problem it causes:

# If you want to use "bash -e", you might be tempted to do it like

this:#!/usr/bin/env bash -e# But it will actually be seen as
"/usr/bin/env" "bash -e"# Instead, use set to change bash's
behavior:#!/usr/bin/env bash
set -e

You now need to name your file. I don’t have a fixed rule as to which extension should be
used, as long as there is one and it makes sense. You could use .bash or .zsh depending
on which shell is referenced in the shebang, or simply a generic .sh for all your scripts.
Adding such an extension allows whoever is going to see your scripts to distinguish between
them and ordinary binaries. It will also allow you to search for them easily with a command
such as: find . -name '*.sh' .

Arguments
You can always handle arguments very easily, with a piece of code like this:

#!/usr/bin/env sh
SOME_PARAM="$1"OTHER_PARAM="$2"

I will instead suggest using named arguments, handled by a case/esac. The following
example shows how to do it, and it is pretty straightforward:

#!/usr/bin/env sh
# Default values for blank parametersDEBUG=0IN_FILE=/etc/some-input-
file.conf
OUT_FILE=/var/log/some-output-file.log
 # Option parser, the order doesn't matterwhile [ $# -gt 0 ]; docase"$1" in
-i|--input)
IN_FILE="$2"
shift 2
;;
-o|--output)
OUT_FILE="$2"
shift 2
;;
--debug) # Argument acting as a simple flagDEBUG=1
shift 1
;;
*)
break
;;
esacdone# Some simple argument checks
wrong_arg() {
echo "Error: invalid value for $1" >&2
exit 2
}

[ -f $IN_FILE ] || wrong_arg "input file"

[ -f $OUT_FILE ] || wrong_arg "output file"# The actual script can start
below# ...

I don’t think having a “usage” is necessary with such a shell script, as the option parser
makes the use of the script self-explanatory.

Use functions!
What I am about to say is true for all programming languages, but people tend to forget it
when it comes to writing shell scripts.

Using functions to clarify your code or avoid duplicating too many instructions is always
good. It eases the maintenance and update process, and it helps understand the code
better. Having Donald Knuth’s words in mind, I don’t think you could go wrong by adding a
few functions to your scripts instead of copy pasting code.

Variables and subshells

I like to name my variables using only capital letters, underscores and sometimes digits if
need be. That way it is more difficult to confuse them with functions and commands. The
common good practice differs from that and advises to use only lowercase instead of
uppercase. It prevents collisions with variables that are already declared in your
environment.

This might take you back all the way to C89 but I also like to declare my variables at the
beginning of the script or function, depending on the scope I am in. Loop variables are
excluded from that “rule” because I think it would defeat the purpose of being easier to read
and understand.

Don’t use too many temporary variables if you don’t need to. That brings me to the next
topic: subshells. Use them as much as possible, and combine them with pipes. Do that and
erase all “bad” habits of using too many temporary variables. I will take this opportunity to
provide a quick reminder about subshells: I have often seen the backticks syntax used to
declare subshells, but they need to be saved in temporary variables if those need to be
nested. There is also another syntax, which I use more frequently than backticks:
$(command) . It supports nesting and is, in my opinion, more readable. Avoid using this
syntax with sh , as it was initially not supported by the traditional Bourne shell.

#!/usr/bin/env bash
# Code showing two methods for calculating the total size of all GIF#
files found recursively from the current directory.# Method 1: Without
intelligent use of subshells or pipingTMP_FILE=gifs.tmp
find . -name '*.gif' > $TMP_FILEGIFS_TOTAL_1=0while read l; doSIZE=`stat -
c%s $l`GIFS_TOTAL_1=$(($GIFS_TOTAL_1 + $SIZE))done <$TMP_FILE

rm $TMP_FILE# Method 2: One liner using subshells and

pipingGIFS_TOTAL_2=$(stat -c%s $(find . -name '*.gif') | paste -s -d+ |
bc)

The second example is in my opinion easier to understand because fewer commands are
executed. It is also less likely to cause performance issues because it is opening fewer file
descriptors than the code in the first example and it does not use the disk to store temporary
data.

The set builtin

I would now like to talk about the set builtin. Using it will allow you to alter the shell’s
behavior in order to facilitate debugging or make sure commands are executed the way they
are intended to.
Debugging
Using set -x will activate tracing in a script. That means printing every command that is
executed and every variable that is set to the standard output.

Unofficial Bash strict mode

In order to get a “stricter” bash, you can use a two-line trick that will most likely help the
writing of your scripts. I added information about differences with sh and zsh .

#!/usr/bin/env bash

set -euo pipefail

IFS=$'\n\t'

set -e makes the script exit if a command fails (exit code different from 0 ). It can
help find invisible errors before the script is used in a production environment.
set -u makes the script exit if a referenced variable is not declared. Pretty useful
since an undeclared variable can be mistaken for an empty one.
set -o pipefail affects pipes to once again avoid invisible errors. If one command in
a pipeline fails, its exit code will be returned as the result of the whole pipeline. As you
can imagine, it makes debugging a lot easier when combined with -e . \
N.B. This option is not recognized by the Bourne shell.
IFS stands for Internal Field Separator. Each character it contains will be used as a
delimiter when splitting a string (in a for loop for instance). \
N.B. Default IFS for zsh is: $' \n\t\0' .

The unofficial Bash strict mode is described with more depth and straightforward examples
by Aaron Maxwell on his blog.

N.B. Although set can be used in most cases, setopt is the zsh equivalent. See the zsh
documentation for more information.

Useful tools and references

I am aware that there are a lot of resources about shell scripting out there, but I just wanted
to list a few links I find very useful.

Shell Check: Finds issues in a shell script.

Google’s Styleguide: Guidelines for writing readable shell scripts for Google. Keep in
mind that those are meant for scripts to be executed on Google’s machines.
The POSIX Norm: When you are offered multiple options and are in doubt about which
method or syntax to pick, you can always check with the official POSIX norm, followed
by the Bourne shell and by bash --posix .