DevOps and SRE cheat sheet
CI/CD and reliability on a few screens: a GitHub Actions workflow to copy, Docker, kubectl and Terraform commands, release strategies, SLO and burn-rate numbers, incident roles, a postmortem template, and the Git and Linux commands you reach for on release day and on call.
If you see X, reach for Y
Start with the first fix in each row; most teams jump to the expensive one too early.
| You see | Reach for first | Then, if needed | Lesson |
|---|---|---|---|
| CI takes 25 minutes | Cache dependencies; run lint, unit and build jobs in parallel | Split tests across runners; run only affected packages | Trusted CI, Actions CI |
| "Works on my machine" | A Dockerfile and a lockfile; npm ci in CI | Dev containers for the whole team | Containers |
| Deploys are scary, so they are rare | Smaller changes, deployed more often, with a one-command rollback | Canary releases and feature flags | Safe releases |
| A bad deploy reached all 320 stores | Rings: pilot stores first, then a share, then everyone | Automatic rollback on error-rate or SLO burn | Safe releases |
| Servers configured by hand drift apart | Infrastructure as code, applied only from CI | Drift detection with a scheduled plan | IaC |
| Long-lived cloud keys in CI secrets | OIDC from GitHub Actions to the cloud | Per-environment roles with least privilege | Supply chain, Deploy and secure |
| Pods restart again and again | kubectl logs --previous and describe | Fix probes, memory limits or the crash | Kubernetes |
| Alerts fire all night, nobody acts | Alert on SLO burn and user symptoms, not on every CPU spike | Delete or demote alerts with no runbook action | Burn-rate alerts |
| "Is it down?" takes 20 minutes to answer | A dashboard of the four golden signals per service | Traces with a request id end to end | OpenTelemetry, Observability |
| Incidents are chaotic | Named roles, one channel, regular status updates | Severity levels and paging rules written down | Incident command |
| The same incident keeps coming back | Blameless postmortem with owned actions | Track action completion in reviews | Postmortems |
| The team spends days on manual chores | Measure toil, automate the biggest item first | Cap toil at about half of each engineer's time | Toil |
A GitHub Actions workflow to copy
Test on every pull request, deploy from main through a protected environment. Version tags such as @v4 are shown for readability; in a hardened repo pin each action to a full commit SHA (Actions CI, Trusted CI, Supply chain).
name: ci
on:
pull_request:
push:
branches: [main]
permissions:
contents: read # least privilege for every job by default
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true # a new push cancels the old run
jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 15
services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: postgres
ports: ['5432:5432']
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 20
cache: npm
- run: npm ci
- run: npm run lint
- run: npm test
env:
DATABASE_URL: postgres://postgres:postgres@localhost:5432/postgres
deploy:
needs: test
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
environment: production # required reviewers and secrets live here
permissions:
contents: read
id-token: write # lets the job sign in to the cloud with OIDC
steps:
- uses: actions/checkout@v4
- run: ./scripts/deploy.sh| I want to... | Workflow key | Lesson |
|---|---|---|
| Run by hand from the Actions tab | on: workflow_dispatch | Actions CI |
| Run every night (UTC) | on: schedule: - cron: '0 2 * * *' | Actions CI |
| Test on several Node versions | strategy: matrix: node: [20, 22], then ${{ matrix.node }} | Actions CI |
| Run a job after another | needs: test | Actions CI |
| Skip a step unless on main | if: github.ref == 'refs/heads/main' | Actions CI |
| Use a secret | ${{ secrets.SENTRY_TOKEN }} passed through env:, never echoed | Deploy and secure |
| Pass files between jobs | actions/upload-artifact and actions/download-artifact | Trusted CI |
| Require approval before production | environment: production with required reviewers in repo settings | Safe releases |
| Stop a hung job | timeout-minutes: 15 | Trusted CI |
Docker: "I want to..."
Compose v2 is a plugin, so the command is docker compose with a space; the old docker-compose binary is retired (Containers).
| I want to... | Command | Lesson |
|---|---|---|
| Build and tag an image | docker build -t mangrove/api:1.8.0 . | Containers |
| Run it, mapping a port, removed on exit | docker run --rm -p 3000:3000 -e NODE_ENV=production mangrove/api:1.8.0 | Containers |
| See running containers | docker ps (all, including stopped: -a) | Containers |
| Follow a container's logs | docker logs -f --tail 100 <name> | Containers |
| Open a shell inside it | docker exec -it <name> sh | Containers |
| See what uses disk, then clean up | docker system df, then docker system prune (asks first) | Containers |
| See image layers and sizes | docker history mangrove/api:1.8.0 | Containers |
| Start the local stack in the background | docker compose up -d | Containers |
| Follow one service's logs | docker compose logs -f api | Containers |
| Stop the stack (and wipe its volumes) | docker compose down (add -v to delete volumes) | Containers |
A Dockerfile that follows the rules
Multi-stage build, dependency files copied before source so the install layer is cached, a slim base, and a non-root user. Add a .dockerignore with node_modules, .git and .env (Containers).
FROM node:20-slim AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build && npm prune --omit=dev
FROM node:20-slim
WORKDIR /app
ENV NODE_ENV=production
COPY --from=build /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
USER node
EXPOSE 3000
CMD ["node", "dist/main.js"]kubectl: "I want to..."
Add -n <namespace> to any command, or set it once with kubectl config set-context --current --namespace=api. Check which cluster you are on before you change anything (Kubernetes).
| I want to... | Command | Lesson |
|---|---|---|
| See which cluster I am talking to | kubectl config current-context; switch with config use-context | Kubernetes |
| List pods with their node and IP | kubectl get pods -o wide | Kubernetes |
| Find out why a pod is not running | kubectl describe pod <pod> (read Events at the bottom) | Kubernetes |
| Read logs, or the logs of the last crash | kubectl logs -f deploy/api, kubectl logs <pod> --previous | Kubernetes |
| Recent events, newest last | kubectl get events --sort-by=.metadata.creationTimestamp | Kubernetes |
| Open a shell in a pod | kubectl exec -it <pod> -- sh | Kubernetes |
| Reach a service from my laptop | kubectl port-forward svc/api 8080:80 | Kubernetes |
| Preview, then apply a change | kubectl diff -f k8s/, then kubectl apply -f k8s/ | Kubernetes |
| Watch a rollout, or undo it | kubectl rollout status deploy/api, kubectl rollout undo deploy/api | Safe releases |
| Scale by hand | kubectl scale deploy/api --replicas=6 | Kubernetes |
| See CPU and memory use | kubectl top pods (needs metrics-server) | Kubernetes |
| Check what I am allowed to do | kubectl auth can-i delete pods | Kubernetes |
| Pod shows | Usually means | Check |
|---|---|---|
Pending | No node has room, or a volume cannot be attached | describe pod events; resource requests |
ImagePullBackOff | Wrong image name or tag, or no pull permission | The image reference and the registry secret |
CrashLoopBackOff | The app starts and exits again and again | logs --previous; missing env var or config |
OOMKilled (exit code 137) | The container used more memory than its limit | Memory limit vs real use; a leak |
| Running but not Ready | The readiness probe fails, so it gets no traffic | The probe path and timing; the app's dependencies |
Terraform workflow
Plan in the pull request, apply only from CI, state stored remotely with locking. OpenTofu takes the same commands with tofu in place of terraform (IaC).
| Step | Command | Why |
|---|---|---|
| Download providers, connect to state | terraform init | Once per checkout, and after adding a provider |
| Format and validate | terraform fmt -check, terraform validate | Cheap checks for CI |
| Preview the change and save it | terraform plan -out=tfplan | Reviewers read exactly what will happen |
| Apply the reviewed plan | terraform apply tfplan | Applies only what was reviewed, nothing newer |
| See what state tracks | terraform state list, terraform state show <address> | Investigate without editing state by hand |
| Adopt something built by hand | An import block (Terraform 1.5 and later) or terraform import | Bring drift under code |
| Read outputs | terraform output | Feed values to deploy scripts |
Release strategies
Separate deploying code from releasing a feature. For Mangrove's tills, which must keep selling offline, a new POS build goes to a few pilot stores first and waits a full trading day before the next ring (Safe releases, Shipping).
| Strategy | How | Good for | Rollback |
|---|---|---|---|
| Rolling | Replace instances a few at a time | Stateless APIs, the default in Kubernetes | Roll back the same way; old and new run side by side for a while |
| Blue-green | Bring up the full new copy, switch traffic at once | Fast, clean switch; easy to test the new copy first | Switch back; costs double capacity during the switch |
| Canary | Send 1%, then 10%, then 50% of traffic to the new version while watching metrics | High-traffic services where real users find what tests miss | Send the canary's traffic back; automate on error rate |
| Rings | Pilot group, then a wider group, then everyone | Devices and stores: 5 pilot stores, then 50, then all 320 | Stop the ring; roll back the pilot devices |
| Feature flag | Code ships turned off; turn on by user, store or percentage | Business-timed launches, gradual exposure, kill switches | Turn the flag off, no deploy needed; remove old flags |
| Expand and contract | Add new schema, move code, remove old schema later | Database changes that must not stop the app | The old path still works until the contract step |
SLO and error-budget numbers
An error budget is 100% minus the SLO. Burn rate is how fast you spend it: a burn rate of 1 uses exactly the whole budget over the SLO window (SLOs and budgets, Burn-rate alerts).
| SLO | Budget per 30 days | Per year | Lesson |
|---|---|---|---|
| 99% | 7 h 12 min | about 3.65 days | SLOs and budgets |
| 99.5% | 3 h 36 min | about 1.8 days | SLOs and budgets |
| 99.9% | 43.2 min | about 8 h 46 min | SLOs and budgets |
| 99.95% | 21.6 min | about 4 h 23 min | SLOs and budgets |
| 99.99% | 4.3 min | about 53 min | SLOs and budgets |
| Alert (30-day SLO) | Burn rate | Long and short window | Budget spent when it fires |
|---|---|---|---|
| Page | 14.4 | 1 hour, with a 5-minute check | 2% |
| Page | 6 | 6 hours, with a 30-minute check | 5% |
| Ticket | 1 | 3 days, with a 6-hour check | 10% |
What to watch and measure
- Four golden signals per service: latency (p50, p95, p99), traffic, errors, saturation (Observability).
- RED for request-driven services (rate, errors, duration) and USE for resources (utilisation, saturation, errors) (OpenTelemetry).
- DORA metrics for delivery: deployment frequency, lead time for changes, change failure rate, and time to restore after a failed change (Safe releases).
- Capacity: requests in progress = rate × time per request; plan for Mangrove's festival peak at about 4 times a normal day (Capacity and game days, Estimation).
Incident roles and severities
One person commands; they do not also debug. Roles can be one person each in a small team, but name them out loud (Incident command).
| Role | Does | Does not |
|---|---|---|
| Incident commander | Sets priorities, assigns work, decides on rollback, calls for help | Type commands or chase one theory alone |
| Operations lead | Runs the technical investigation and changes, with responders | Make changes without saying them in the channel |
| Communications lead | Posts status to stakeholders and customers on a fixed schedule | Promise a fix time nobody gave |
| Scribe | Keeps the timeline: what was seen, decided and changed, with times | Rely on memory after the fact |
| Severity | Mangrove example | Response |
|---|---|---|
| SEV1 | App checkout down, or tills in many stores cannot take card payments | Page now, all hands, status updates every 30 minutes |
| SEV2 | Delivery tracking broken; one city's stores sync sales hours late | Page the owning team, updates every hour |
| SEV3 | Back-office report slow; a minor feature fails with a workaround | Ticket, fix in working hours |
Status update template
[SEV1] App checkout failing for about 40% of orders. Update 3, 19:45 UTC.
Impact: customers in all cities see "payment failed"; stores are not affected.
What we know: started 19:05 after release 1.8.0 of the payments API.
What we are doing: rolling back to 1.7.4, about 10 minutes.
Next update: 20:15 UTC or sooner if this changes.
Incident lead: NadiaPostmortem template
Blameless: ask how the system let a reasonable person make this mistake, not who made it. Publish within a week (Postmortems).
# Postmortem: checkout failures after payments API 1.8.0
Date: 2026-10-14 Severity: SEV1 Duration: 19:05 to 19:58 UTC (53 min)
Authors: Nadia, Ken Status: actions in progress
## Summary
Two or three sentences a manager can forward.
## Impact
Orders failed, customers affected, revenue at risk, error budget used.
## Timeline (UTC)
19:05 deploy finished 19:09 burn-rate page 19:21 rollback decided ...
## Root cause and contributing factors
What happened and why the safeguards did not stop it.
## What went well / what was hard
## Actions
| Action | Type (prevent / detect / mitigate) | Owner | Due |Git on release day
The everyday commands are in the Developer cheat sheet. These are the ones for releases and incidents (Branches and releases, Code detective).
| I want to... | Command | Lesson |
|---|---|---|
| Tag a release and push the tag | git tag -a v1.8.0 -m "Release 1.8.0", git push origin v1.8.0 | Branches and releases |
| See what changed between two releases | git log --oneline v1.7.4..v1.8.0, git diff --stat v1.7.4 v1.8.0 | Branches and releases |
| Find which release a build came from | git describe --tags | Branches and releases |
| Undo a merged pull request safely | git revert -m 1 <merge-sha> (a new commit; history stays) | Undo |
| Copy a hotfix onto a release branch | git cherry-pick -x <sha> (-x records where it came from) | Branches and releases |
| Find the commit that broke it | git bisect start, git bisect bad, git bisect good v1.7.4, then git bisect run npm test; finish with git bisect reset | Code detective |
| Clone fast in CI | git clone --depth 1 <url> | Actions CI |
Linux on call
The first commands to run on a sick host. journalctl and systemctl need a systemd distribution; the TechOps cheat sheet has more (Linux CLI).
| I want to know... | Command | Lesson |
|---|---|---|
| How busy the box has been | uptime (load averages for 1, 5, 15 minutes) | Linux CLI |
| Memory and disk | free -h, df -h, du -sh /var/log/* | sort -h | Linux CLI |
| What uses the CPU or memory | top (or htop), ps aux --sort=-%mem | head | Linux CLI |
| Whether a service is up, and its logs | systemctl status api, journalctl -u api --since "15 min ago" -f | Linux CLI |
| What is listening on which port | sudo ss -tlnp | Linux CLI |
| Whether the kernel killed something for memory | sudo dmesg -T | grep -i oom | Linux CLI |
| Status code and time of an endpoint | curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' https://api.example.test/health | Linux CLI |
Glossary
Plain definitions of the DevOps and SRE terms used across the Engineering Roles line.
- Artifact
- The built output of CI, such as a container image, built once and promoted unchanged from staging to production. Trusted CI
- Blue-green deployment
- Running the new version beside the old one and switching all traffic at once. Safe releases
- Burn rate
- How fast the error budget is being spent compared with spending it evenly over the window. Burn-rate alerts
- Canary
- A new version given a small share of real traffic first, watched before it gets more. Safe releases
- CI/CD
- Continuous integration (merge small changes often, tested automatically) and continuous delivery or deployment (every good build can go, or goes, to production). Trusted CI
- Container image
- A packaged app with everything it needs to run, built from a Dockerfile in layers. Containers
- Deployment (Kubernetes)
- The object that keeps a set number of pod copies running and rolls out new versions. Kubernetes
- Drift
- Real infrastructure that no longer matches its code because someone changed it by hand. IaC
- Error budget
- The failure an SLO allows, such as 43 minutes a month at 99.9%, which the team can spend on change. SLOs and budgets
- Feature flag
- A switch that turns a feature on or off at run time without a deploy. Safe releases
- Game day
- A planned exercise that breaks something on purpose to test systems and people. Capacity and game days
- Infrastructure as code (IaC)
- Servers, networks and permissions described in files, reviewed and applied like code. IaC
- Liveness and readiness probes
- Kubernetes checks: liveness restarts a stuck container, readiness keeps traffic away until it is ready. Kubernetes
- MTTR
- Mean time to restore: the average time from an incident starting to service being back. Incident command
- OIDC (in CI)
- CI proves its identity to the cloud with a short-lived token, so no long-lived keys are stored. Supply chain
- OpenTelemetry
- The open standard and libraries for producing traces, metrics and logs. OpenTelemetry
- Pod
- The smallest thing Kubernetes runs: one or more containers sharing a network address. Kubernetes
- Postmortem
- A written, blameless review of an incident with actions that have owners and dates. Postmortems
- Rollback
- Returning to the last good version; the first move when a deploy causes an incident. Safe releases
- Runbook
- Step-by-step instructions for one alert or task, written so anyone on call can follow them. Incident command
- SLI, SLO, SLA
- What you measure, the target you set, and the promise with penalties you sign. SLOs and budgets
- Terraform state
- Terraform's record of which real resources match which code; kept remote, locked and never edited by hand. IaC
- Toil
- Manual, repetitive work that grows with the system and could be automated. Toil
Courses on Kolom
- DevOps: CI/CD, Containers and Infrastructure as Code: trusted pipelines, Docker, Kubernetes, Terraform, safe releases and the supply chain.
- Site Reliability Engineering: SLOs, OpenTelemetry, burn-rate alerts, incident command, postmortems, capacity and toil.
- Running Software in Production: observability, security and shipping basics every engineer needs.
- GitHub for Teams: pull requests, releases, Actions CI and deploying safely.
What to read next
- Site Reliability Engineering and The Site Reliability Workbook from Google, free at sre.google/books. The workbook's alerting chapter is the source of the burn-rate table.
- Accelerate by Nicole Forsgren, Jez Humble and Gene Kim. The research behind the DORA metrics.
- The DevOps Handbook by Gene Kim, Jez Humble, Patrick Debois and John Willis. How flow, feedback and learning fit together.
- GitHub Actions documentation at docs.github.com/actions, including the security hardening guide.
- kubectl reference at kubernetes.io, with the official quick reference of common commands.
- Terraform documentation at developer.hashicorp.com.