A script is a file of commands, read word by word

In The terminal without fear you typed commands one at a time, and in Linux from the command line you used them on a store box. A script is a text file with those same commands in order. The first line is the shebang, which tells Linux which program reads the file:

#!/usr/bin/env bash
# hello-store: print who and where we are
store=$(hostname)                 # run a command, keep its output
echo "Running on ${store} as $(whoami)"

Save it as hello-store, make it executable with chmod +x hello-store, and run it with ./hello-store. On a store box it prints Running on store-0142 as techops. Two details matter on Ubuntu. First, sh is a smaller shell called dash, so sh hello-store ignores the shebang and runs the file with dash, and Bash-only features such as [[ ]] fail. Run scripts as ./name or bash name. Second, store=$(hostname) has no spaces around the equals sign. store = x would try to run a program called store.

$(hostname) is a command substitution: Bash runs the command and pastes its output into the line. This is how a script reads the result of one command and decides what to do next.

Now the most important rule in shell scripting. When Bash meets an unquoted variable like $dir, it does not paste the value in as one piece. It first splits the value at every space, tab and newline (word splitting), then treats any * or ? in the pieces as a glob and expands it into file names. Only then does the command run. So:

dir="/srv/till exports"            # a folder name with a space
rm -rf $dir/old                    # rm receives TWO paths: /srv/till and exports/old
rm -rf "$dir/old"                  # rm receives one path: /srv/till exports/old

The first rm deletes the wrong things. The fix is to put double quotes around every variable: "$dir", "$1", "$@". Inside double quotes Bash still replaces the variable, but it keeps the value as one word and does not expand globs. Single quotes stop the replacement completely, so '$dir' is the four characters $dir.

Quotes do not protect you from an empty variable. If dir is empty, rm -rf "$dir"/* becomes rm -rf /*. Write "${dir:?}" in any destructive command: if the variable is unset or empty, Bash prints an error and stops the script before the command runs.

Exit codes: how a script knows something failed

Every command ends with an exit code. Zero means success, and any other number means failure. grep, for example, returns 0 when it finds a match, 1 when it finds none, and 2 when it cannot read the file. Bash keeps the last exit code in $?, and if, && and || all decide on it:

if grep -q "PRINTER OFFLINE" /var/log/store-print/print.log; then
  echo "printer problem seen today"
fi
mkdir -p /srv/reports && cd /srv/reports   # cd only if mkdir worked
systemctl is-active --quiet store-print || echo "print service is down" >&2

By default Bash ignores failures and carries on with the next line. On a store box that is dangerous: if cd /srv/reports fails, the next rm runs in whatever folder the script happened to be in. Three settings, written together at the top of almost every script, change that:

• set -e stops the script when a command fails.
• set -u treats an unset variable as an error, so a typo like $LOG_DRI stops the script instead of becoming an empty string.
• set -o pipefail changes what a pipeline reports. Normally a pipeline's exit code is the last command's. With pipefail, it is the exit code of the last command that failed, so a failure anywhere in the pipe is seen.

set -e is a safety net with holes, and you need to know where they are. It does not fire for a command inside an if or while condition, for any command in an && or || list except the last one, or for a command after !. That is on purpose: those are places where you are already handling the result. Two holes catch people more often. local count=$(cmd) inside a function always succeeds, because the exit code you get is the one from local, not from cmd. Declare first and assign on the next line. And a command that "fails" in a normal way, like grep -c finding no matches, stops the whole script. When no match is a fine answer, say so: count=$(grep -c "PRINTER OFFLINE" print.log || true).

Decisions, loops, functions and arguments

Real jobs need to choose and repeat. Bash's [[ ]] tests a condition and returns 0 or 1, so it plugs straight into if:

if [[ -d "$LOG_DIR" ]]; then echo "folder exists"; fi      # -d folder, -f file
if [[ -z "${ALERT_URL:-}" ]]; then echo "no alert URL"; fi  # -z empty, -n not empty
if (( use > 85 )); then echo "disk too full"; fi           # numbers: use (( )) or -gt

The last line hides a trap. Inside [[ ]], the > sign compares text, letter by letter, so [[ 9 > 10 ]] is true because "9" sorts after "1". For numbers, use (( a > b )) or [[ $a -gt $b ]].

A for loop walks over a list. Loop over a glob directly, never over the output of ls, which breaks on spaces. If nothing matches, Bash leaves the pattern as literal text, so shopt -s nullglob makes an empty match an empty list. A while read loop walks over lines of a file or of a command's output:

shopt -s nullglob
for f in /var/log/store-print/*.log; do
  echo "$(basename "$f"): $(wc -l < "$f") lines"
done

# check a list of store boxes from the TechOps jump host
while IFS= read -r box; do
  ssh -n "$box" df -h /          # -n: do not let ssh swallow the rest of the list
done < stores-north.txt

IFS= read -r reads each line exactly as written, keeping leading spaces and backslashes. The -n on ssh matters: without it, ssh reads from the same input as the loop, eats the remaining lines, and the loop stops after the first store.

A function names a group of commands. Inside it, $1, $2 are its own arguments, and local keeps a variable from leaking into the rest of the script. The script itself gets its arguments the same way: $1 is the first, $# is how many there are, and "$@" is all of them, each kept as one word. A while loop with case and shift reads options like --dry-run:

DRY_RUN=0
while [[ $# -gt 0 ]]; do
  case "$1" in
    --dry-run) DRY_RUN=1; shift ;;                     # shift drops $1, the rest move up
    *) echo "unknown option: $1" >&2; exit 2 ;;
  esac
done

Deleting files safely and running twice safely

Clean-up scripts are where a typo costs the most. The tool for "old files" is find. It walks a folder and checks each file against a list of tests, left to right, and runs an action on the files that pass every test:

find /var/log/store-print -xdev -type f -name '*.log.*' -mtime +14 -print

Read it as: in this folder, stay on this disk (-xdev), only regular files (-type f), only rotated logs whose name has something after .log, like print.log.3.gz (-name '*.log.*', quoted so the shell does not expand it), only files last changed more than 14 days ago (-mtime +14), and print the name. That name pattern is your protection for the live print.log, the file the print service is writing right now.

-mtime has a rule that surprises people: find works out a file's age in whole days and drops the fraction. A file that is 14 days and 20 hours old counts as 14, and 14 is not "more than 14". So -mtime +14 matches files at least 15 full days old, and -mtime 14 (no plus) means "exactly 14", which is rarely what you want.

Three habits make deletes safe. Print first. Run the exact command with -print, read the list, then add -delete at the end. The order matters: find runs left to right, so find /var/log/store-print -delete -name '*.gz' deletes every file in the folder before the name test is ever checked. Build the dry run into the script as a --dry-run flag that prints instead of deleting, so the next person can check it on a new store box too. Keep the printed list in the real run (-print -delete), so the log says what went.

One more fact about deletes on Linux. If a program still has a file open, deleting it removes the name, but the disk space comes back only when the program closes the file. Delete the live print.log and df still shows the disk full until the print service restarts. Rotated logs are closed, which is one more reason the script only touches those.

A daily job should also be idempotent. A retry, a missed night caught up at boot, or a second engineer running it by hand must not do harm. Use mkdir -p (no error if the folder exists), rm -f (no error if the file is gone), "delete files older than 14 days" instead of "delete the oldest 5 files", and a lock so two copies never run at once: flock -n on a lock file exits straight away if another copy holds it.

Logging and scheduling: cron and systemd timers

A job that runs at 2 a.m. has nobody watching it, so it must write down what it did. The simplest rule: print normal progress to standard output with echo, print problems to standard error with echo "…" >&2, and exit with 0 on success and a non-zero code on failure. The scheduler decides where that output goes.

cron is the classic scheduler. A line has five time fields (minute, hour, day of month, month, day of week) and a command. 30 2 * * * means 02:30 every day. Cron runs jobs with a very small environment: on Ubuntu the PATH is only /usr/bin:/bin, there is no terminal, and output that you do not redirect is mailed to the user if a mail system exists, or lost. So in cron, use full paths, send output somewhere on purpose, and remember that % means "new line" in a crontab, so date +%F must be written date +\%F.

A systemd timer is the modern choice on Ubuntu, and it fixes most of cron's gaps. You write two small files: a .service unit that runs the script once, and a .timer unit that says when. Output goes to the journal, so journalctl -u disk-check shows every night's run, the same way you read service logs in Linux from the command line. A failed run marks the unit as failed, which systemctl --failed lists. Persistent=true runs a missed job at boot if the box was off at the planned time, which happens when a store loses power overnight. RandomizedDelaySec= adds a random wait, so 320 store boxes do not all hit the data centre in the same second. And systemd does not start the service again while the last run is still going.

Before a script goes to every store, run ShellCheck on it. It reads the script without running it and points at missing quotes, masked exit codes, globs in the wrong place and dozens of other traps, each with a code like SC2086 that has its own explanation page. Treat its warnings like a review comment from a senior colleague: fix them, or add a comment saying why the line is fine.