Linux admin: "I want to..."

Commands for Debian, Ubuntu and the RHEL family. Where they differ, both are shown. Check the host name in your prompt before anything destructive (Linux CLI).

systemd and journalctl

On systemd distributions (most current ones) services are units. After editing a unit file, run sudo systemctl daemon-reload before restarting it (Linux CLI).

A service unit to copy

Save as /etc/systemd/system/sync-agent.service. It restarts the store sync agent if it crashes and waits for the network at boot (Linux CLI).

[Unit]
Description=Mangrove store sync agent
After=network-online.target
Wants=network-online.target

[Service]
User=sync
ExecStart=/opt/sync/bin/sync-agent
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Safer shell scripts

Start every bash script with strict mode, quote every variable, and run ShellCheck on it before it touches 320 stores (Shell scripts).

#!/usr/bin/env bash
set -euo pipefail          # stop on errors, unset variables and failed pipes

store="${1:?usage: rotate-logs.sh STORE_ID}"
dry_run="${DRY_RUN:-1}"    # default to a dry run

for f in /var/log/pos/"$store"/*.log; do
  if [[ "$dry_run" == "1" ]]; then
    echo "would compress $f"
  else
    gzip -- "$f"
  fi
done

Network diagnosis ladder

Work from the bottom up and stop at the first rung that fails. On Windows PowerShell 5.1, curl is an alias for Invoke-WebRequest; type curl.exe to get real curl, which ships with Windows 10 and 11 (Store networks, Networking).

Store network: if you see X, check Y

Mangrove's tills keep selling offline and queue sales for upload, so "the internet is down" is rarely a stop-the-store emergency. Card payments and app orders are the urgent part (Store networks, Offline sync).

Network numbers worth remembering

Accounts and MFA

Every account belongs to one named person, created from HR data and removed the same day they leave. Shared logins on tills become personal PINs or badges, so every sale and refund has a name (Accounts and MFA, Identity).

The device fleet

Mangrove runs about 400 tills, 1,000 handheld scanners and the head-office laptops. Every device is in an asset register and enrolled in a device management tool (MDM) before it leaves the warehouse (Device fleet).

Service desk basics (ITIL words)

Use the same words as the rest of the business and the tickets sort themselves (Service desk).

Priority matrix

Priority comes from impact (how many people or how much money) and urgency (how fast it gets worse). The targets below are an example; agree your own with the business (Service desk).

A ticket the next person can act on

Summary:   Receipt printer offline on till 2, store S-014
Reported:  Ama (store manager), 09:40, by phone
Impact:    1 of 4 tills; customers offered e-receipts; queue normal
Priority:  P3 (medium impact, medium urgency)
Done:      Power cycled printer; checked USB cable; test page fails
Found:     Printer shows "paper jam" light, no paper inside
Next:      Swap printer from the spare stock; ship faulty unit to the DC
Status:    Waiting on store to fit spare (ETA 11:00)

Good first response: confirm what you understood, say what happens next and when you will update them. Close only when the user agrees it is fixed, and turn repeat fixes into a knowledge article so the first line can solve them (Service desk, Toil).

Glossary

Plain definitions of the IT operations terms used in the TechOps course.

Asset register

The list of every device with its serial, owner, location and status. Device fleet

Break-glass account

An emergency admin account for when normal sign-in is broken, kept offline and alerted on. Identity

Change advisory

The review that approves risky changes and their timing and rollback plans. Service desk

CIDR

Writing a network as an address and prefix length, such as 192.168.10.0/24 (254 usable hosts). Store networks

Cron

The Linux scheduler that runs commands at set times from a crontab. Shell scripts

DHCP

The service that hands devices an IP address, gateway and DNS server when they join a network. Store networks

DNS

The system that turns names like api.example.com into IP addresses. Networking

Gateway

The router a device sends traffic to when the destination is outside its own network. Store networks

Joiner, mover, leaver (JML)

The three moments when someone's access must change: they start, change role or leave. Accounts and MFA

Journal

systemd's log store, read with journalctl. Linux CLI

Kiosk mode

Locking a device to one or a few apps, as on handheld scanners and tills. Device fleet

Knowledge article

A short, searchable fix for a known issue, written so the first line can solve it. Service desk

MDM

Mobile device management: a tool that enrols, configures, updates and wipes devices remotely. Device fleet

MFA

Multi-factor authentication: a second proof besides the password. Accounts and MFA

NTP

Network Time Protocol, which keeps device clocks right; wrong clocks break certificates and logs. Store networks

Patch ring

A group that gets updates at a set stage, so problems show up on a few devices first. Device fleet

Problem

In ITIL, the underlying cause of one or more incidents. Service desk

Shift left

Moving fixes to the earliest level that can handle them: self-service, then first line. Service desk

SLA

Service level agreement: the response and fix times promised for each priority. Service desk

systemd unit

A file that tells systemd how to run and restart a service, timer or mount. Linux CLI

VLAN

A separate virtual network on the same switches, such as tills apart from guest Wi-Fi. Store networks

Workaround

A way to restore service before the root cause is fixed. Service desk

Courses on Kolom

• TechOps: Linux, Networks and IT Operations: the Linux command line, shell scripts, store networks, accounts and MFA, the device fleet and the service desk.
• Security Foundations: identity, networks and incidents from the defender's side.
• How the Web Works: DNS, TCP, TLS and what happens on every request.
• Site Reliability Engineering: incidents, postmortems and getting rid of toil.

What to read next

• The Linux Command Line by William Shotts, free at linuxcommand.org. From the first command to shell scripts.
• Windows commands reference at learn.microsoft.com. Every built-in command with its options.
• ShellCheck at shellcheck.net. Paste a script and it explains each bug it finds.
• The Practice of System and Network Administration by Thomas Limoncelli, Christina Hogan and Strata Chalup. How good IT teams run fleets, help desks and changes.
• NIST SP 800-63B at pages.nist.gov. The guidance behind modern password and MFA rules.