Advanced shell scripting with Bash

Presented by Victor Engmark

Huge thanks to Catalyst IT and Toitū Te Whenua Land Information New Zealand for letting me release this with an open license

Made with reveal.js

Prerequisites

Some familiarity with Bash
A terminal running Bash for exercises
GNU tools (find, grep, sort, etc.)
ssh 127.0.0.1 works

Introductions

Who am I?
Who are you?
Got any quick shell horror stories?
What do you hope to get out of this course?

Hints & tips

Please ask questions anytime
Take notes
Try things out
PS1='\$ ' PS2='> '

Motivation, aka why is shell scripting so weird?

Understand
Simplify
Replace

Context is everything

Count commands with quotes in $username’s history.

Syntactic double quotes	`count=""`
Command substitution	`count="$()"`
Word splitting	`count="$(grep --count)"`

…

Context is everything

Count commands with quotes in $username’s history.

Syntactic single quotes	`count="$(grep --count '')"`
Quoted string	`count="$(grep --count '"')"`
Tilde expansion	`count="$(grep --count '"' ~)"`

…

Context is everything

Count commands with double quotes in $username’s Bash history.

…

Syntactic double quotes	`count="$(grep --count '"' ~"")"`☹
Variable expansion	`count="$(grep --count '"' ~"${username}")"`
Double quoted literal	`count="$(grep --count '"' ~"${username}/.bash_history")"`

👍, right?

Context is everything

$ username="$USER"$ count="$(grep --count '"' ~"${username}/.bash_history")"grep: ~victor/.bash_history: No such file or directory$ ls ~victor/.bash_history
/home/victor/.bash_history$ wtf
bash: wtf: command not found

The order of expansions is: […] tilde expansion, […] variable expansion, […]

In other words, the username needs to be a literal.

Context is everything

Solution: eval getent+cut+obscure content

$ username="$USER"
$ user_home="$(getent passwd "$username" | cut --delimiter=':' --fields='6')"
$ count="$(grep --count '"' "${user_home}/.bash_history")"
$ echo "$count"
169

Say no to portability

Even POSIX mode does not lead to portable code:

$ type -a [[
[[ is a shell keyword
$ bash --posix
$ [[ 1 ]]$ echo $?
0

Say no to portability

Readability often suffers — compare

sort -V

with

sort --version-sort

Say no to portability

Bashisms are helpful:

read -a
done <(my_script)
$'\n'
…

Say no to portability

Focus on the applicable shell + version
Avoid premature generalization

Bashing

Speed:

$ wget --quiet https://norvig.com/big.txt$ wc big.txt
 128457 1095695 6488666 big.txt$ time grep foobar big.txtreal	0m0.031s
user	0m0.031s
sys	0m0.001s$ time while read -r -u 9
> do
>    :
> done 9< ./big.txtreal	0m3.532s
user	0m3.113s
sys	0m0.266s

No-op is >100 times slower!

Bashing

Limited data structures:

$ help declare
[Trimmed for brevity]
    Set variable values and attributes.

    Options:
      -f	restrict action or display to function names and definitions

    Options which set attributes:
      -a	to make NAMEs indexed arrays (if supported)
      -A	to make NAMEs associative arrays (if supported)
      -i	to make NAMEs have the `integer' attribute
      -n	make NAME a reference to the variable named by its value

Everything else is a string.

Bashing

No nested data structures:

HTML/XML
JSON
Other formats, other languages

Bashing

No exceptions or try/catch/finally, just exit codes.

Bashing

Very limited functions:

No return value, only exit code
Can’t safely pass by reference
Only string arguments
Always scoped to file, so “nested functions” aren’t

Bashing

Too much state to keep in mind at all times:

local, file and exported variables (including functions)
Local and exec redirects
$PWD
Settings from set and shopt
User, group
umask
Locale
Time zone
Probably more, like SELinux context domain/type

Running away from $HOME

Recommended shebang line:

#!/usr/bin/env bash

Works as long as bash is on the $PATH.

Running away from $HOME

The shebang line is only relevant if you call the script directly:

foo.bash if it’s on the $PATH
./foo.bash
/full/path/to/foo.bash

Otherwise the shebang line is ignored:

. foo.bash uses the interpreter of the parent shell
sh foo.bash uses sh, which may be Bash, Dash or something else

Running away from $HOME

. foo.bash and bash foo.bash also ignore the executable flag:

$ echo 'echo "$SHLVL"' > test.bash$ ./test.bashbash: ./test.bash: Permission denied$ . ./test.bash1$ bash ./test.bash2

Running away from $HOME — 5m exercise

Can you find any other differences between sourcing and running a script?

Solution

$ bash --noprofile --norc
$ cd "$(mktemp --directory)"
$ echo 'declare -p' > test.bash
$ chmod u+x test.bash
$ . test.bash first second > sourced.log
$ ./test.bash first second > run.log
$ git diff sourced.log run.log

Mostly $BASH_ variables.

Your flight has been redirected

Read left-to-right:

$ { echo info; echo error >&2; } > result.txt 2>&1$ cat result.txtinfo
error

$ { echo info; echo error >&2; } 2>&1 > result.txterror$ cat result.txtinfo

Your flight has been redirected

One file per redirect:

$ echo foo > foo.txt
$ echo bar > bar.txt
$ echo > ./*.txt
bash: ./*.txt: ambiguous redirect
$ cat foo.txt bar.txt
foo
bar

Your flight has been redirected

cat is only needed in corner cases like combining stdin and a file:

$ echo foo > foo.txt
$ echo bar | cat foo.txt -
foo
bar

Most of the time, cat FILE | COMMAND can be simplified to COMMAND FILE.

Your flight has been redirected

Redirect the rest of the script:

exec > out.log 2> error.log

Your flight has been redirected

Print and save the output with tee:

$ echo foo | tee output.log
foo
$ cat output.log
foo

--append to append to file.

Your flight has been redirected

Redirect a stream to a command and back again:

$ (echo out; echo foo >&2; echo bar >&2) 2> >(grep bar >&2)out
bar

Your flight has been redirected

Different processes will process input at different rates. This way standard error and standard output can get out of sync:

$ { echo first >&2; echo second; echo third >&2; } | tee out.logfirst
third
second

Your flight has been redirected — 10m exercise

Running du --summarize /* as a non-root user prints a lot of Permission denied messages. Silence these. Additionally:

Don't mix or swap standard output and standard error streams.
Don't change output ordering of either stream.
Extra challenge: print one error message (still on standard error) representative of all of the silenced ones.

Solution

                        du --summarize /* 2> >(grep --invert-match
                            'Permission denied' >&2)

Pipe dreams

Subshell environment for each subsequent command:

$ count=0
$ mount | grep '^tmpfs ' | while read
> do
>     (( ++count ))
> done
$ echo "$count"0

Context is lost.

Pipe dreams

Bring the important command into the current context:

$ count=0
$ while read -r -u 3
> do
>     (( ++count ))
> done 3< <(mount | grep '^tmpfs ')
$ echo "$count"7

Using a file descriptor above 2 avoids input being swallowed by tools such as SSH reading standard input inside the loop.

Pipe dreams

All the exit codes in a pipeline:

$ (exit 2) | true | false$ echo "${PIPESTATUS[@]}"2 0 1

Pipe dreams

Application specific workarounds to get colour output:

grep --color=always […] | less --RAW-CONTROL-CHARS

Colour codes are characters.

Pipe dreams — 10m exercise 1

Given a file with an IP per line (you can just use your own IP repeatedly), print the current time on each host.

Solution

$ while read -r -u 3 ip
> do
>     ssh "$ip" date
> done 3< hosts.txt

Pipe dreams — 5m exercise 2

Each command, including a pipeline, can only have one exit code. How is that determined? Hint:

$ false | false | false
$ false | false | true
$ false | true | false
…

NUL is not your friend

Some tools have a flag to separate or terminate entries with NUL.

You cannot store NUL in a variable.

You cannot put NUL in a literal:

$ printf '%q\n' $'foo\0bar\0baz\0'foo

Bash doesn’t like half the world’s files.

Use More Quotes™!

Single quotes for any literals without single quote:

$ printf '%s\n' '|\|o e$cape'|\|o e$cape

Use More Quotes™!

Double quotes for strings with single quotes, command substitutions or variables:

$ subject='this'
$ printf '%s\n' "Can't $(basename /usr/bin/touch) ${subject}"Can't touch this

Use More Quotes™!

Dollar single quotes for escape sequences:

$ printf '%s\n' $'first\nsecond'first
second

Use More Quotes™!

You can mix quotes even in a single word:

$ "e"c'h'$'o' '"'"'""'

<< HERE-KITTY

<< NAME is almost like a double quoted context:

$ cpu_count=8
$ cat << EOF
> [hardware]
> cpu_count=${cpu_count}
> EOF[hardware]
cpu_count=8

Does a similar job to envsubst.

<< HERE-KITTY

Backslash in << NAME is literal except unescaped before a newline:

$ cat << EOF
> a\
> b
> EOFab

$ cat << EOF
> c\d
> EOFc\d

$ cat << EOF
> e\\
> f
> EOFe\
f

<< HERE-KITTY

Quotes are literal in << NAME:

$ cat << EOF
> '"
> EOF'"

<< HERE-KITTY

<< 'NAME' works like a single quoted context:

$ result=foo
$ cat << 'EOF'
> ${result}
> EOF${result}

<< HERE-KITTY — 10m exercises

What happens if the here document delimiter is indented?
What happens if there is whitespace after the delimiter?
What happens if a variable expands to the delimiter?

<<< 'here string' — bonus level

Here strings are syntactic sugar for echo WORD | my_command:

$ IFS=/ read -a directories -r <<< "$HOME"
$ printf '%q\n' "${directories[@]}"''
home
username

Also moves the command into current context.

Breaking news

Never modify a running script — they are read in chunks:

$ printf '%s\n' 'sleep 1' 'echo foo' > test.bash
$ bash test.bash & sed --in-place 's/foo/bar/' test.bash

The result is either foo or bar, but it is not reproducible!

Escape\ from\ Alcatraz

Avoid word splitting in unquoted strings:

$ printf '%s\n' foo barfoo
bar$ printf '%s\n' foo\ barfoo bar

\n is a printf escaped character, not a shell one.

Escape\ from\ Alcatraz

Literal \n terminators:

$ printf '%s\\n' foo barfoo\nbar\n

No newline at end of output, so it’s followed immediately by $PS1.

Escape\ from\ Alcatraz

Escaping escape sequences:

$ printf %s\\\\n foo barfoo\nbar\n

The number of backslashes always doubles, because each character has to be escaped separately in the next context.

Escape\ from\ Alcatraz

Avoid multiple escape levels.

Quoting
Here documents
Files & envsubst
printf '%q'

Forecast: `$variable`

Default value:

PATH="${PATH-/bin:/usr/bin}"

:- also matches empty value.

Use with set -o nounset.

Forecast: `$variable`

Replacement value:

result="${1+defined}"

:+ only matches non-empty value.

Forecast: `$variable`

The right hand side can be more complex:

$ csv=
$ entry='x'
$ csv="${csv:+"${csv},"}${entry}"
$ echo "$csv"x

$ csv=foo,bar
$ entry='x'
$ csv="${csv:+"${csv},"}${entry}"
$ echo "$csv"foo,bar,x

Forecast: `$variable` — 10m exercise

Add non-empty $path to $PATH cleanly - there should be no leading or trailing colons even if $PATH started out empty.

Solution

$ PATH="${PATH:+"${PATH}:"}${path}"

`$@` is where it’s at

Arguments beyond $9:

$ set -- {a..z}
$ echo ${26}z

Arrays are zero-indexed, but $0 is special — it usually contains the script name.

`$@` is where it’s at

Avoid $*:

$ set -- 'a b' 'c d'
$ for argument in "$*"
> do
>     printf '%s\n' "$argument"
> donea b c d$ for argument in $*
> do
>     printf '%s\n' "$argument"
> donea
b
c
d

All arguments as a single word.

`$@` is where it’s at

Use "$@":

$ for argument
> do
>     printf '%s\n' "$argument"
> donea b
c d

Each argument as a separate word.

Default for loop target, no need for in "$@".

`$@` is where it’s at

Named arguments handling skeleton:

set -o errexit
arguments="$(getopt --options='' --longoptions='case-sensitive,case-insensitive,help,exclude:' --name='script-name' -- "$@")"
eval set -- "$arguments"
unset arguments
while true
do
    case "$1" in
        [continued…]
    esac
done

`$@` is where it’s at

Boolean options (“flags”):

--case-sensitive)
    case_sensitive=1
    shift
    ;;
--case-insensitive)
    unset case_sensitive
    shift
    ;;

`$@` is where it’s at

Usage instructions:

--help)
    echo 'script-name [--case-sensitive|--case-insensitive] [--help] [--exclude=PATTERN ...] [--] FILES'
    exit 0
    ;;

`$@` is where it’s at

Repeating key/value arguments:

--exclude)
    excludes+=("$2")
    shift 2
    ;;

`$@` is where it’s at

End of options separator & unhandled arguments:

--)
    shift
    break
    ;;
*)
    echo "Unhandled option $(printf '%q' "$1"). Please report to …" >&2
    exit 2
    ;;

`$@` is where it’s at — 10m exercise

What does argument handling do in each case?

bash test.bash --case-sensitive . /some/path

bash test.bash --help

bash test.bash --exclude='.git' --exclude='.svn' -- --actual-filename.txt

bash test.bash --blah /some/path

`$@` is where it’s at — solution

What does argument handling do in each case?

bash test.bash --case-sensitive . /some/path
case_sensitive=1
@=('.' '/some/path')

bash test.bash --help
[Prints help message and returns from script with exit code 0]

bash test.bash --exclude='.git' --exclude='.svn' -- --actual-filename.txt
excludes=('.git' '.svn')
@=('--actual-filename.txt')

bash test.bash --blah /some/path
[Prints “script-name: unrecognized option '--blah'” and returns from script with exit code 1 from getopt]

Collect all the things!

Collect non-$IFS characters:

$ matches=($(grep --only-matching . <<< $'some foo\nother foo'))
$ echo "${matches[@]}"s o m e f o o o t h e r f o o

No quotes to enable word splitting.

Collect all the things!

Collect $IFS-separated or -terminated “words”:

$ while IFS=$'\t' read -a cells -r
> do
>     echo Line
>     printf '%s\n' "${cells[@]}"
> done <<< $'column 1\tcolumn 2\nvalue 1\tvalue 2'Line
column 1
column 2
Line
value 1
value 2

Collect all the things!

Append to an array:

$ characters=({a..z})
$ characters+=({0..9})
$ echo "${characters[@]}"a b c d e f g h i j k l m n o p q r s t u v w x y z 0 1 2 3 4 5 6 7 8 9

Collect all the things!

Can be sparse:

$ characters=(a b c)
$ unset 'characters[1]'
$ characters+=([25]=z)
$ echo "${characters[@]}"a c z$ echo "${#characters[@]}"3

Don’t loop from 0 through $(( "${#name[@]}" - 1 )).

Collect all the things! — 10m exercise

Create an array of all the executables on your $PATH.

Solution

$ IFS=: read -a paths -r <<< "$PATH"
$ for path in "${paths[@]}"
> do
>     executables+=("$path"/*)
> done

The value of keys

Associative arrays use:

$ declare -A abbreviations=(['GNU HURD']="GNU's not Unix! HIRD of Unix-replacing daemons" ['sed']='stream editor')
$ declare -p abbreviationsdeclare -A abbreviations=([sed]="stream editor" ["GNU HURD"]="GNU's not Unix! HIRD of Unix-replacing daemons" )$ printf '%s\n' "${!abbreviations[@]}"sed
GNU HURD$ echo "${abbreviations['sed']}"stream editor

Not insertion ordering.

Not numerically indexable.

The value of keys

Must be declared:

$ example=([key]='value')
$ declare -p exampledeclare -a example=([0]="value")

The value of keys — 10m exercise

Print ‘key [length of value]’ for each key:

declare -A nicks=(['Bill Hicks']='William Melvin Hicks' ['Gandhi']='Mohandas Karamchand Gandhi')
[Your code]
Gandhi 26
Bill Hicks 20

Solution

$ for name in "${!nicks[@]}"
> do
>     printf '%s %s\n' "$name" "${#nicks[$name]}"
> done

Conditional surrender

Old style conditionals are easy to break:

$ bash --noprofile --norc -o xtrace
$ [ $foo = 'bar' ]+ '[' = bar ']'bash: [: =: unary operator expected

Only two arguments (ignoring “]”) so Bash assumes a unary operation.

Conditional surrender

Old style conditionals have no way of grouping expressions such as “(A or B) and C”.

Conditional surrender

Use command conditionals and grouping:

{ [[ "$foo" = 'a' ]] || [[ "$foo" = 'b' ]]; } && [[ "$bar" = 'c' ]]

Harder to break
More well known than -a and -o

Conditional surrender

Left associative: foo || bar && baz ≡ { foo || bar; } && baz

$ false || echo failure && echo successfailure
success

Conditional surrender

foo && bar || baz ≡ { foo && bar; } || baz

$ false && echo success || echo failurefailure

Conditional surrender

Break it up:

if some_command
then
    echo success
else
    echo failure
fi

Conditional surrender — 10m exercise

Write a single expression to check whether $x is between 0 and $x_max or $y is between 0 and $y_max (-ge is ≥ and -le is ≤).

Solution

{
    [[ "$x" -ge 0 ]] && [[ "$x" -le "$x_max" ]];
} || {
    [[ "$y" -ge 0 ]] && [[ "$y" -le "$y_max" ]];
}

Function junction

Run in the same shell as the parent script:

$ shell_pid() {
>     echo $$
> }
$ diff <(shell_pid) <(echo $$)$ echo $?
0

Function junction

Use return to return an exit code from the function without exiting the script:

$ escape_key_value_pairs() {
>     if [[ $# -eq 0 ]] || [[ $(( $# % 2 )) -ne 0 ]]
>     then
>         echo "Use: ${FUNCNAME} [KEY VALUE]..." >&2
>         return 1
>     fi
>     printf '%s=%q\n' "$@"
> }$ escape_key_value_pairs PS1 'My prompt
> $ ' PS2Use: escape_key_value_pairs [KEY VALUE]...$ echo $?
1

Functions get their own argument list.

Function junction

Use local to declare function scope variables:

$ directory="$(mktemp --directory)"
$ filename='.bashrc_local'
$ save_current_prompt() {
>     local directory="$HOME"
>     escape_key_value_pairs PS1 "$PS1" PS2 "$PS2" >> "${directory}/${filename}"
> }$ save_current_prompt
$ tail --lines=2 ~/.bashrc_localPS1=\\\$\␠
PS2=\>\␠$ echo "$directory"/tmp/tmp.1ORclnSGb9

Function junction — 5m exercise

This function pollutes the surrounding variable namespace:

reverse() {
    arguments=("$@")
    for index in $(seq $(( $# - 1 )) -1 0)
    do
        printf '%s ' "${arguments[$index]}"
    done
    printf '\n'
}

Change it so that none of the variables it assigns are propagated to the outer scope.

Solution

local arguments index

When is zero equal to one?

Numeric contexts:

$(( index++ )) prints result
index+=1 only if declare -i index first
(( index++ ))
[[ "$index" -eq 0 ]]

When is zero equal to one?

Integers only:

$ [[ 1.1 -eq 1 ]]bash: [[: 1.1: syntax error: invalid arithmetic operator (error token is ".1")

Use tools like bc for more powerful maths.

When is zero equal to one?

0 starts octal:

$ echo $(( 077 ))63

When is zero equal to one?

0x starts hex:

$ echo $(( 0xff ))255

Case insensitive.

When is zero equal to one?

N# starts base N (2-64):

$ echo $(( 64#a ))10

$ echo $(( 64#A ))36

$ echo $(( 64#@ ))62

$ echo $(( 64#_ ))63

Case insensitive if base ≤ 36.

When is zero equal to one?

Comes after variable expansion:

$ msb=BE
$ lsb=EF
$ echo $(( "0x${msb}${lsb}" ))48879

When is zero equal to one?

Beware leading zeros:

$ month=08
$ (( month++ ))bash: let: 08: value too great for base (error token is "08")

Find out in August!

When is zero equal to one?

Type coercion:

$ foo=one
$ [[ "$foo" -eq 0 ]] && echo 'equal'equal

When is zero equal to one? — 10m exercise

Sum an array of hexadecimal number strings without the 0x prefix. For example, (ffff 11) should sum to 65552 (65535 + 17).

Solution

$ declare -i sum=0
$ for number in "${numbers[@]}"
> do
>     sum+="0x${number}"
> done

Chain of command

Build up argument lists or commands using arrays:

$ cat excludes.txt
foo bar
baz$ while read -r -u 9 exclude
> do
>     excludes+=(--regexp "$exclude")
> done 9< excludes.txt$ set -o xtrace$ grep --invert-match "${excludes[@]}" <<< $'foo bar\nfoo baa\n'+ grep --color=auto --invert-match --regexp 'foo bar' --regexp bazfoo baa

Chain of command

Only a single command and its arguments:

No redirects
No pipes

Reading is fun

Read newline-terminated strings:

$ while read -r line
> do
>     echo "$line"
> done < <(printf '%s\n' 'foo' 'bar')foo
bar

A “line” in *nix operating systems.

Reading is fun

Read newline-separated or -terminated strings:

$ while read -r line || [[ -n "$line" ]]
> do
>     echo "$line"
> done < <(printf $'foo\nbar')foo
bar

read populates variable then fails on non-newline character at EOF.

Reading is fun

Read NUL-terminated strings:

$ cd "$(mktemp --directory)"
$ touch 'backslash\separated' $'newline\nseparated' 'space separated'
$ while IFS= read -d '' -r filename
> do
>     printf '%q\n' "$filename"
> done < <(find . -mindepth 1 -exec printf '%s\0' {} +)./backslash\\separated
$'./newline\nseparated'
./space\ separated

Reading is fun

Read $IFS-terminated words:

$ read -r first second rest <<< '   aye   bee   cee   dee   '
$ printf '%q\n' "$first" "$second" "$rest"aye
bee
cee\ \ \ dee

Trims leading, trailing and separating $IFS characters.

End of the line

How many characters does $result contain?

$ result="$(printf '%s' $'foo\n\n')"
$ echo "${#result}"3

$() removes trailing newlines.

End of the line

$() workaround:

$ result="$(printf '%s' $'foo\n\n'; printf x)"
$ result="${result%x}"
$ echo "${#result}"
5

End of the line

<<< (here string) is bad in a different way:

$ wc --bytes <<< $'foo\n\n'
6

Unconditionally adds a newline.

End of the line

Newline-preserving redirects:

$ printf $'foo\n\n' | wc --bytes
5$ printf $'foo\n\n' > result.txt
$ wc --bytes result.txt
5 result.txt$ wc --bytes < <(printf $'foo\n\n')
5

End of the line

echo vs. printf:

$ echo foo | xxd -cols 100000000: 66  f
00000001: 6f  o
00000002: 6f  o
00000003: 0a  .

echo adds newline (0x0a) at end of output.

End of the line

echo vs. printf:

$ printf '%s' foo | xxd -cols 100000000: 66  f
00000001: 6f  o
00000002: 6f  o

printf formats arguments.

End of the line — 15m exercise

Save script arguments to a file, and reuse them in another script.

Hint: The only character which can’t be in a string is NUL (\0 in printf).

Hint: read’s -r flag avoids treating backslashes specially.

Solution

for argument
do
    printf '%s\0' "$argument"
done > arguments.bin

set --

while read -d '' -r argument
do
    set -- "$@" "$argument"
done < arguments.bin

Exit in an orderly fashion

Command “success:”

$ if true
> then
>     echo 'Success'
> fi
Success

Defined as exit code 0.

Exit in an orderly fashion

Completely application specific. For example, zero in arithmetic expressions:

$ count=0$ echo $?
0$ (( count=0 ))$ echo $?
1$ (( count++ ))$ echo $?
1$ (( count++ ))$ echo $?
0$ printf '%s\n' "$count"
2

Exit in an orderly fashion

Some fairly well–documented numbers in /usr/include/sysexits.h/$(nix eval --raw nixpkgs.glibc.dev.outPath)/include/sysexits.h:

$ grep '^#define ' /usr/include/sysexits.h
#define EX_OK		0	/* successful termination */
#define EX__BASE	64	/* base value for error messages */
#define EX_USAGE	64	/* command line usage error */
#define EX_DATAERR	65	/* data format error */
#define EX_NOINPUT	66	/* cannot open input */
#define EX_NOUSER	67	/* addressee unknown */
#define EX_NOHOST	68	/* host name unknown */
#define EX_UNAVAILABLE	69	/* service unavailable */
#define EX_SOFTWARE	70	/* internal software error */
#define EX_OSERR	71	/* system error (e.g., can't fork) */
#define EX_OSFILE	72	/* critical OS file missing */
#define EX_CANTCREAT	73	/* can't create (user) output file */
#define EX_IOERR	74	/* input/output error */
#define EX_TEMPFAIL	75	/* temp failure; user is invited to retry */
#define EX_PROTOCOL	76	/* remote error in protocol */
#define EX_NOPERM	77	/* permission denied */
#define EX_CONFIG	78	/* configuration error */
#define EX__MAX	78	/* maximum listed value */

Exit in an orderly fashion

When a command terminates on a fatal signal whose number is N, Bash uses the value 128+N as the exit status.

$ kill -l INT
2$ sleep 1d
^C
$ echo $?130

Exit in an orderly fashion

Don’t exit "$error_count"!

$ bash
$ exit 256
exit
$ echo $?0

Defensive coding

#!/usr/bin/env bash

set -o errexit

temporary_directory="$(mktemp --directory)"
[…]
mkdir "$temporary_directory"

Caveats galore.

Defensive coding

#!/usr/bin/env bash

set -o errexit -o noclobber

temporary_directory="$(mktemp --directory)"
echo "Start" > "${temporary_directory}/script.log"
[…]
echo "End" > "${temporary_directory}/script.log"

/tmp/[…]/script.log: cannot overwrite existing file

Defensive coding

#!/usr/bin/env bash

set -o errexit -o noclobber -o nounset

temporary_directory="$(mktemp --directory)"
echo "Start" > "${temporary_directry}/script.log"

temporary_directry: unbound variable

Defensive coding

#!/usr/bin/env bash

set -o errexit -o noclobber -o nounset -o pipefail

grep foo "$1" | cut --delimiter=':' --fields=1 | grep --invert-match bar

Easier than $PIPESTATUS.

Defensive coding

Umask:

$ cd "$(mktemp --directory)"
$ umask
0022
$ touch first
$ ls -l first
-rw-r--r-- […] first$ umask 0077
$ touch second
$ ls -l second
-rw------- […] second

Setting traps

Don’t litter!

cleanup() {
    rm --force --recursive "${temporary_directory-}"
}
trap cleanup EXIT
temporary_directory="$(mktemp --directory)"

mktemp --directory is atomic.

mktemp result is only accessible by owner (umask 0077).

Setting traps

Print debugging information on demand:

trap env USR1

kill -USR1 $!

Triggers after the currently running command.

Setting traps

Reload configuration in long-running process:

trap read_configuration HUP

kill -HUP "$server_pid"

Setting traps — 10m exercise

Start with a script which processes standard input until EOF:

#!/usr/bin/env bash
while read -r line
do
    : # Omitted
done

You have no visibility of how far it has processed. Modify it to react to SIGUSR1 by printing the value of $line.

Solution

trap 'echo "$line"' USR1

An open square bracket by any other name

Commands can be defined in several ways, for example:

$ type -a [[ is a shell builtin
[ is /usr/bin/[

Presented in order of decreasing precedence.

Don’t use which to determine what will be run!

An open square bracket by any other name

The precedence order:

alias
keyword
function
builtin
file

An open square bracket by any other name

To find documentation:

help command for builtins
man command for executable files
Try command --help if neither work
No guarantees

Killing me softly

kill -9 "$!"

Now you have to clean up manually.

Killing me softly

Run a command with a timeout:

$ timeout 1s sleep 2s$ echo $?
124

Killing me softly

kill "$!"
timeout="$(date --date='now + 1 minute' +%s)"
while [[ "$(date +%s)" -lt "$timeout" ]]
do
    if kill -0 "$!"
    then
        sleep 0.1
    else
        exit 0 # Win
    fi
done
exit 1 # Fail

SIGTERM is the default
Adjust timeout and interval as necessary
Exit early on success
If you have to kill -9 something is broken

The opposite of WYSIWYG

What you see is not always what you think you see:

$ printf '%s\n' “foo” ‘foo’“foo”
‘foo’

“Typographic” quotes are not syntactic.

Usually caused by WYSIWYG web framework.

The opposite of WYSIWYG

What you see is not always what you think you see:

$ grep —fixed-strings foo <<< foobargrep: foo: No such file or directory

Em dash ≠ double dash.

Usually caused by WYSIWYG web framework.

The opposite of WYSIWYG

Never copy straight to a terminal: git clone /dev/null echo "Hi! I'm a Trojan horse, what do you not need on this machine?" git clone git@gitlab.com:engmark/advanced-shell-scripting-with-bash.git

Even command line editors are vulnerable.

Use a graphical editor if at all possible.

To debug or to debug, that is not a question

ShellCheck can find many common issues:

$ shellcheck --shell=bash - <<< 'while read line; do :; done'In - line 1:
while read line; do :; done
      ^--^ SC2162: read without -r will mangle backslashes.
           ^--^ SC2034: line appears unused. Verify use (or export if used externally).

For more information:
  https://www.shellcheck.net/wiki/SC2034 -- line appears unused. Verify use (...
  https://www.shellcheck.net/wiki/SC2162 -- read without -r will mangle backs...

To debug or to debug, that is not a question

$ bash --noprofile --norc -o xtrace$ find "$(egrep --only-matching '/usr/[^:]+(:|$)' <<< "$PATH" | head --lines=1 | head --bytes=-2)" -mindepth 1++ egrep --only-matching '/usr/[^:]+(:|$)'
++ head --bytes=-2
++ head --lines=1+ find /usr/local/bin -mindepth 1/usr/local/bin/foo

Standard input is not shown.

“+” ($PS4) denotes shell depth ($SHLVL).

Pipelined commands run simultaneously, so ordering is not guaranteed.

set -o xtrace inside scripts.

To debug or to debug, that is not a question

strace prints system calls and signals:

$ strace -e openat cat /dev/nullopenat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/dev/null", O_RDONLY) = 3
+++ exited with 0 +++

man 1 strace

To debug or to debug, that is not a question

lsof lists files currently open by programs:

$ nc example.org 80 &
[3] 9999
$ lsof -p $!COMMAND   PID     USER   FD   TYPE  DEVICE SIZE/OFF    NODE NAME
nc       9999 username  cwd    DIR   254,3     4096 8919615 /home/username
nc       9999 username  rtd    DIR   254,2     4096       2 /
nc       9999 username  txt    REG   254,2    39608 2921200 /usr/bin/netcat
nc       9999 username  mem    REG   254,2    84016 2885772 /usr/lib/libresolv-2.27.so
[…]
nc       9999 username    0u   CHR   136,0      0t0       3 /dev/pts/0
nc       9999 username    1u   CHR   136,0      0t0       3 /dev/pts/0
nc       9999 username    2u   CHR   136,0      0t0       3 /dev/pts/0
nc       9999 username    3u  IPv4 4538734      0t0     TCP machine-name:46748->example.org:http (ESTABLISHED)

man 8 lsof

To debug or to debug, that is not a question

netstat prints networking information:

$ sudo netstat --listening --numeric --program --tcp | sed --quiet '1,2p;/ssh/p'Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1234/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      1234/sshd

man 8 netstat

To debug or to debug, that is not a question

/proc has lots of runtime information:

$ journalctl --catalog --follow --unit=sshd | grep "$USER" &
$ ls -lA /proc/$!/fdtotal 0
lr-x------ 1 username username 64 May  7 14:32 0 -> 'pipe:[5181666]'
lrwx------ 1 username username 64 May  7 14:32 1 -> /dev/pts/3
lrwx------ 1 username username 64 May  7 14:32 2 -> /dev/pts/3

man 5 procfs

To debug or to debug, that is not a question — 10m exercise

Find the maximum number of files your shell (PID $$) can open. Hint: /limits.

Extra exercise: Find the PID of the journalctl command:

$ journalctl --catalog --follow --unit=sshd | grep "$USER" &

Solution

cat /proc/$$/limits
Get pipe inode number from ls -l /proc/$!/fd, then ls -l /proc/*/fd/1 | grep INODE

PhD thesis: looping through files

*.bash in the current directory in current locale’s alphabetical order:

ls *.bash | while read file
do
    something $file
done

Expelled from PhD programme, computer ground to dust and buried using the secret rituals of the church of Stéphane Chazelas.

PhD thesis: looping through files

*.bash in the current directory in current locale’s alphabetical order:

for file in ./*.bash
[…]

👍

PhD thesis: looping through files

*.bash including dotfiles in the current directory in current locale’s alphabetical order:

shopt -s dotglob
for file in ./*.bash
[…]

👍

PhD thesis: looping through files

*.bash except for foo.bash in the current directory in current locale’s alphabetical order:

shopt -s extglob
for file in ./!(foo).bash
[…]

😅

PhD thesis: looping through files

*.bash in and below the current directory in current locale’s alphabetical order:

shopt -s globstar
for file in ./**/*.bash
[…]

😕

PhD thesis: looping through files

Universal ordering:

export LC_COLLATE='C'
for file in ./*
[…]

Or LC_COLLATE='en_NZ.utf8'…

Let me just curl dict://dict.org/d:collate 🧐

PhD thesis: looping through files

Reverse order:

files=(./*)
for (( index = ${#files[@]} - 1; index >= 0; index-- ))
do
    something "${files[$index]}"
done

😟

PhD thesis: looping through files

Non globbable pattern:

while IFS= read -d '' -r -u 9 path
do
    something "$path"
done 9< <(find . \( -type d -regex '^.*/\.git$' -prune -false \) -o -type f -exec printf '%s\0' {} +)

😭

PhD thesis: looping through files — 20m exercise

Combining all of the above 😉

Explain every part of the previous command to someone. For reference:

while IFS= read -d '' -r -u 9 path
do
    something "$path"
done 9< <(find . \( -type d -regex '^.*/\.git$' -prune -false \) -o -type f -exec printf '%s\0' {} +)

Solution

find . finds all files in the current directory and child directories.
$ expression $ overrides find expression precedence.
1. -type d matches directories.
2. -regex '^.*/\.git$' matches filenames (actually directories because of the previous expression) ending with ‘/.git’.
3. -prune stops find from descending into matching directories.
4. -false makes the entire expression false.
-o means "or". Since the previous expression was false we always process the expressions after this.
-type f matches plain files (not directories).
-exec some command {} + runs a command suffixed with as many filenames as possible, if necessary running the command multiple times with different sets of files.
printf '%s\0' prints any subsequent arguments terminated with \0, aka. NUL.
<(some command) creates a named pipe allowing the command output to be treated as a file.
some command 9< causes file descriptor 9 to point to standard input.
while condition command; do inner commands; done runs inner commands as long as condition command returns a zero exit code.
IFS= some command empties the internal field separator during some command, avoiding any trimming of characters when word splitting.
read options path reads a single piece of the input stream into the variable path. Options:
1. -d '' sets the input stream separator to NUL.
2. -r ensures that backslashes are treated literally.
3. -u 9 sets the input stream to file descriptor 9.
some command "$path" uses the now safe value in the path variable.

Welcome to sow’s ear processing!

Automating interactive scripts is the worst
You wouldn’t write Vim or Mutt as a shell script

Conclusion: arguments ≫ menus or prompts.

Welcome to sow’s ear processing!

Filenames are rarely relevant for the processing of their contents
Pipelining commands is several orders of magnitude faster than read loops
Redirect hacks are hard to read and therefore error-prone

Conclusion: support standard input and output before files.

Welcome to sow’s ear processing!

Error handling is rudimentary
Most scripts can do unrecoverable damage with small changes
You only have to debug the code you have run

Conclusion: exit at the first sign of a problem.

Welcome to sow’s ear processing!

Dozens of pieces of context for even simple scripts
Impossible to test scripts exhaustively
Even simple tasks are difficult to do correctly

Conclusion: keep it really simple.

Thank you!

These slides and their sources
Beginning shell scripting with Bash
Greg’s Wiki is opinionated, direct, and the best Bash reading in existence
The Unix & Linux Stack Exchange answers all your questions
The newline Guide to Bash Scripting — my book
Reader beware — there are many sources of bad shell scripting advice online