Getting in and finding your way

Each store has a store box: a small server running Ubuntu Server 24.04 LTS. It caches prices for the scanners, queues till sales when the internet drops, and runs the receipt print service. It has no screen. You reach it over the store's VPN with SSH, using its hostname, such as store-0142. If cd, ls and pipes are new to you, do The terminal without fear first; this lesson builds on it.

Log in with a key pair, not a password. A password can be guessed, reused or passed around in a chat group. A private key cannot be guessed, and each person's public key is one line in their ~/.ssh/authorized_keys file on the box, so removing one person removes one line.

# On your laptop, once: make a key pair (choose a passphrase when asked)
ssh-keygen -t ed25519 -C "nadia@techops-laptop"
# Copy the public key to the box (asks for your password one last time)
ssh-copy-id nadia@store-0142
# From now on, the key logs you in
ssh nadia@store-0142

Once keys work for everyone who needs access, switch passwords off. Ubuntu's main SSH config, /etc/ssh/sshd_config, reads the files in /etc/ssh/sshd_config.d/ first, and the SSH server uses the first value it finds for each setting. So put your rules in a file whose name sorts early:

# /etc/ssh/sshd_config.d/10-keys-only.conf
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin no

sudo sshd -t                                        # test the config: no output means it is valid
sudo sshd -T | grep -i '^passwordauthentication'    # the value the server will really use
sudo systemctl reload ssh                           # on Ubuntu the unit is called ssh, not sshd

passwordauthentication no

Once you are in, everything has a fixed place. Linux puts every disk under one tree that starts at /, the root. These are the folders you will use every day:

To read files, use three tools. less is a pager: it opens a file of any size instantly, / searches, n jumps to the next match, G goes to the end and q quits. tail shows the end of a file, and tail -f keeps printing new lines as they arrive, which is how you watch a till upload happen live. grep prints only the lines that contain a word.

less /var/log/mangrove/tillqueue.log                          # page through; /word searches, q quits
tail -n 20 /var/log/mangrove/tillqueue.log                    # the last 20 lines
tail -F /var/log/mangrove/tillqueue.log                       # follow new lines, even after the log is rotated; Ctrl+C stops
grep -c "upload failed" /var/log/mangrove/tillqueue.log       # count matching lines
grep -i "timeout" /var/log/mangrove/tillqueue.log | tail -n 5 # last 5 matches, any upper or lower case

Users, groups and permissions

Every process runs as a user, and every file has an owner (a user) and a group. A group is a named set of users, such as techops. The id command tells you who you are:

uid=1001(nadia) gid=1001(nadia) groups=1001(nadia),4(adm),27(sudo),1005(techops)

On Ubuntu, the sudo group may run commands as root, and the adm group may read most logs. Services get their own service accounts: the price cache runs as pricecache and the till queue as tillqueue, so a bug in one cannot overwrite the other's files. You create one with sudo useradd --system --no-create-home --shell /usr/sbin/nologin tillqueue.

ls -l shows the permissions:

-rwxr-x--- 1 root techops 2210 Oct  2 10:15 flush-queue.sh
-rw-r----- 1 root techops  512 Oct  2 10:15 stores.conf
drwxr-x--- 2 root techops 4096 Oct  2 10:15 reports

The first character is the type: - for a file, d for a directory. The next nine are three groups of three: what the owner may do, what members of the group may do, and what everyone else may do. r is read, w is write, x is execute, and - means no. So flush-queue.sh can be read, changed and run by root, read and run by the techops group, and nobody else may touch it.

On a directory the letters mean something different. r lets you list the names inside, x lets you enter it and open files by name, and w (together with x) lets you create, rename and delete files in it. Deleting a file depends on the folder's permissions, not the file's.

Each group of three is often written as one digit: r = 4, w = 2, x = 1, added up. rwx is 7, r-x is 5, rw- is 6, r-- is 4. So rwxr-x--- is 750 and rw-r----- is 640. Common modes: 755 for programs everyone may run, 644 for files everyone may read, 640 for logs, 600 for secrets and SSH files. The SSH server refuses your key if your .ssh folder or authorized_keys file can be written by other users.

sudo chown root:techops /opt/mangrove/tools/flush-queue.sh   # owner root, group techops
sudo chmod 750 /opt/mangrove/tools/flush-queue.sh            # rwx owner, r-x group, nothing for others
chmod 600 ~/.ssh/authorized_keys                             # only you may read or change it
sudo chmod g+r /var/log/mangrove/tillqueue.log               # add read for the group, change nothing else
sudo usermod -aG adm ops-trainee                             # add a user to a group (-a keeps the old groups)

A group change takes effect at the user's next login, so the new member logs out and in again. When Linux decides whether you may open a file, it checks in a fixed order and stops at the first match, as the diagram shows.

Now you can see why chmod 777 is never the fix for "Permission denied". It gives write access to every account on the box: the print service, any program with a bug, and any attacker who breaks into one of them. If a script with mode 777 is run by root, anyone who can edit it can run commands as root. The right fix is to find who needs access (ls -l on the file, id for the user) and give exactly that: change the owner, or add the user to the group.

sudo runs one command as root and writes it to the log, with your name. Use it per command. Avoid staying in a root shell with sudo -i for routine work: one wrong rm in the wrong folder has no safety net.

Processes, signals and load

A process is a running program, and each has a number, its PID. ps takes a snapshot of them; top shows a live view that refreshes every few seconds (press M to sort by memory, P by CPU, q to quit). htop is a friendlier version, if it is installed (sudo apt install htop).

ps -eo pid,user,stat,%cpu,%mem,etime,cmd --sort=-%cpu | head -n 3   # busiest processes first

    PID USER     STAT %CPU %MEM     ELAPSED CMD
   1187 priceca+ Ssl   3.1 10.2 12-03:11:40 /opt/mangrove/pricecache/bin/pricecache
    912 tillque+ Ssl   0.4  2.1 12-03:11:42 /opt/mangrove/tillqueue/bin/tillqueue

Long user names are cut short with a +, so priceca+ is the pricecache account. The STAT column's first letter is the state. R is running or ready to run, S is sleeping (waiting for work, which is normal), D is waiting for the disk or another device and cannot be interrupted, Z is a zombie (finished, but its parent has not collected it). Many D processes mean the disk is the bottleneck.

uptime and the first line of top show the load average: three numbers for the last 1, 5 and 15 minutes. Compare them with the number of CPUs, which nproc prints. On a 4-CPU box, a load of about 4 means the CPUs are roughly fully used; 9 means tasks are queuing. On Linux, the load also counts tasks stuck in state D, so a high load with idle CPUs points at the disk, not the CPU. In top, the wa value (time the CPUs sat waiting for disk) confirms it.

To stop a process you send it a signal. The ones you need:

• TERM (15): "please stop". The default for kill PID. The program can catch it, finish what it is doing and save its data.
• INT (2): what Ctrl+C sends to the program in front of you.
• HUP (1): many server programs reload their config when they get it. This depends on the program, so check its documentation.
• KILL (9): "stop now". The program cannot catch it. The kernel ends it at once, with no clean-up and no saving.

Mangrove's till queue holds the last few sales in memory and writes them to disk when it gets TERM. A kill -9 skips that step. So for anything run as a service, do not use kill at all: use systemctl stop or systemctl restart. systemd sends TERM, waits (90 seconds by default), and sends KILL only if the program is still running. A process in state D often cannot be killed even with -9 until the disk answers.

Services with systemd, logs with journalctl

systemd starts every service on the box, watches it, and restarts it if it dies. Each service is described by a unit file. Packages install theirs in /usr/lib/systemd/system/; your own go in /etc/systemd/system/. This is the till queue's:

# /etc/systemd/system/mangrove-tillqueue.service
# (systemd allows comments only on their own lines, never after a value)
[Unit]
Description=Mangrove till sales queue
# start once the network is up
Wants=network-online.target
After=network-online.target

[Service]
# run as the service account, never as root
User=tillqueue
Group=tillqueue
EnvironmentFile=/etc/mangrove/tillqueue.env
ExecStart=/opt/mangrove/tillqueue/bin/tillqueue
# if it exits with an error, start it again after 5 seconds
Restart=on-failure
RestartSec=5

[Install]
# what "enable" hooks it into: the normal boot
WantedBy=multi-user.target

You drive it with systemctl. start runs a service now; enable makes it start at every boot. They are separate, and forgetting enable is the classic reason a service "disappears" after a reboot. After you edit a unit file, run sudo systemctl daemon-reload so systemd reads the new version. To change a packaged unit, use sudo systemctl edit NAME: it saves your changes in a separate override file that survives package updates, and reloads for you.

systemctl status mangrove-tillqueue               # running? since when? main PID, last log lines
sudo systemctl restart mangrove-tillqueue         # stop (TERM, then KILL if needed), then start
sudo systemctl enable --now mangrove-tillqueue    # start at boot, and start now
systemctl --failed                                # every unit that has failed
systemctl cat mangrove-tillqueue                  # the unit file plus any overrides

Everything a service prints, plus systemd's own notes about starts, stops and crashes, goes to the journal. journalctl reads it, and its filters are what make it useful. On Ubuntu, members of the adm group can read the whole journal; others need sudo.

journalctl -u mangrove-tillqueue --since "07:00"     # one service, from 07:00 today
journalctl -u mangrove-tillqueue --since "2026-10-19 22:00" --until "2026-10-20 06:00"
journalctl -u mangrove-tillqueue -f                  # follow new lines, like tail -f
journalctl -p err --since today                      # only errors, from every service
journalctl -b -1 -n 50                               # last 50 lines of the previous boot
journalctl --disk-usage                              # how much space the journal takes

-b -1 works only if the journal is kept on disk (the folder /var/log/journal exists). Some programs, like the price cache, also write their own log file under /var/log/mangrove/. Check both: the journal tells you when the service started, stopped or crashed, and the app's log tells you what it was doing.

Disk space: df, du and the file that will not go away

A full disk breaks things in strange ways: the till queue cannot save sales, logins fail, services crash with No space left on device. Two commands find the cause. df -h shows each filesystem and how full it is. du adds up the size of folders, so you can walk down to the one that grew.

df -h                                                  # how full is each filesystem?
sudo du -xh --max-depth=1 /var | sort -h | tail -n 5   # the biggest folders in /var
sudo du -h /var/log/* | sort -h | tail -n 5            # then go one level deeper

-h prints sizes like 41G instead of byte counts, -x stays on one filesystem so du does not wander into other disks, and sort -h sorts those human sizes correctly. Work top-down: the biggest folder, then the biggest inside it, until you find the file. If df shows free space but you still get "No space left", run df -i: the disk may have run out of inodes because something made millions of tiny files.

There is one trap everybody meets once. When you rm a file that a running process still has open, the name disappears but the space is not freed until the process closes it. du no longer sees the file, df still counts it, and the two disagree. Restarting the service frees the space. To empty a log a service is still writing, use sudo truncate -s 0 FILE instead of rm.