1. Courses
  2. Engineering Roles
  3. DevOps: CI/CD, Containers and Infrastructure as Code
  4. Stop 2

35 minIntermediate

Containers from the inside

A container is an ordinary Linux process with walls around it, and an image is a stack of read-only layers. Once you see those two mechanisms, you can write Dockerfiles that build in seconds, ship a fraction of the bytes, and stop cleanly in the middle of a festival rush.

By the end you can

  • Explain what namespaces and cgroups do, and why a container is not a small virtual machine
  • Read an image as layers and order Dockerfile steps so the build cache does its job
  • Write a multi-stage, non-root Dockerfile for the order service and pick a base image with reasons
  • Make a container stop cleanly by getting SIGTERM to your code as PID 1
  • Push images to GHCR from CI and deploy them by immutable digest instead of a moving tag

The everyday picture

Think of a hotel. Every room shares the same building: one foundation, one water supply, one electricity network. But each guest gets a key that opens only their room, a window that shows only their view, and a meter that cuts the power if they use more than their share.

Each room is also furnished from the same catalogue. The bed, the desk and the lamp are standard, and the hotel does not build new ones for every guest. Guests can move things around and leave their own things. When a guest checks out, housekeeping resets the room to the catalogue layout, and anything the guest left behind is gone.

A container works like a hotel room. The building is the Linux , shared by every container on the machine. The key and the window are namespaces, the power meter is a cgroup, the catalogue furniture is the image's read-only layers, and the things a guest leaves behind live in a thin writable layer that disappears when the container is removed. A virtual machine is a separate house with its own foundation.

A container is a process with walls

When you run docker run, nothing boots. The Docker client asks the Docker daemon to start a container, the daemon hands the job to containerd, and containerd uses a small tool called to ask the kernel for two things. Then it starts your program as a normal process.

The first thing is a set of . Each namespace gives the process a private view of one resource. In its own PID namespace, your node process is PID 1 and sees no other processes. In its own network namespace, it has its own interfaces and its own port 3000. In its own mount namespace, it sees the image's files as /. Other namespaces give it its own hostname (UTS) and its own shared-memory area (IPC).

The second thing is a . Namespaces control what the process can see; the cgroup controls how much it can use. docker run --memory=512m --cpus=1 writes those limits into the cgroup. If the order service goes over 512 MB, the kernel's out-of-memory killer ends it, even when the machine has 30 GB free. Docker then reports exit code 137 and OOMKilled: true. Too much CPU does not kill anything: the process is slowed down () instead.

What docker run does docker run order-service API request dockerd, then containerd starts runc runc sets up the process asks the kernel Linux kernel, shared Namespaces own PIDs, network, files, hostname cgroup memory max 512 MB CPU 1 core what it sees its limits node dist/main.js PID 1 in its namespace host: PID 48213
No guest operating system starts. runc asks the shared kernel for namespaces (what the process can see) and a cgroup (how much it can use), then starts node. Inside, node is PID 1; on the host it is one more process with an ordinary PID.

Two practical facts follow. First, containers start fast, usually in well under a second, because there is no kernel to boot. Second, every container on a machine shares that one kernel. A Linux container cannot bring its own kernel, which is why Docker Desktop on a Mac or Windows laptop runs a small Linux virtual machine in the background. It also means isolation is thinner than in a VM. By default, Docker does not turn on the user namespace, so root inside a container is UID 0 on the host kernel, held back by dropped capabilities and security profiles. That is the main reason this lesson runs the order service as an ordinary user. Linux from the command line covers users, processes and signals if you want a refresher.

Quick check

The order service runs in a container on a Linux build server. On the host (outside the container) you run ps aux. What do you see?

Images are stacked layers, and the cache reads them in order

An image is a list of plus a small config file that says which command to run, which user, which environment variables. In a Dockerfile, RUN, COPY and ADD each add a layer with the files they changed. Instructions such as ENV, USER, EXPOSE and CMD only change the config.

Each layer is named by the SHA-256 hash of its content, so it is stored and downloaded once. If twenty Mangrove services use the same base image, a cluster node downloads that base once, and a new order service version only costs the layers that changed. When a container starts, an (overlay2 on most Linux hosts) stacks the layers into one tree and adds a thin writable layer on top. When the app changes a file that came from the image, the kernel first copies the whole file up into the writable layer (). When the container is removed, the writable layer is deleted. Data that must survive belongs in Postgres, object storage or a volume.

Deleting a file in a later layer does not remove it from the image. The later layer only records a marker that hides the file in the merged view. The bytes are still in the earlier layer, and anyone who pulls the image can unpack that layer and read them. That is why RUN apt-get install … && rm -rf /var/lib/apt/lists/* must be one RUN, and why a secret copied in and deleted later is still a leaked secret.

One image, many layers writable layer COPY dist COPY node_modules RUN rm .npmrc COPY .npmrc FROM node:22-slim per container, gone on docker rm read-only, shared, pulled once only hides the file token still inside overlay merges the stack into one tree: top wins
Read the stack from the bottom up. The overlay shows one merged tree, so the deleted .npmrc is invisible inside the container, but its bytes are still in the COPY .npmrc layer that every registry and every node stores.

The same stack drives the . Docker walks the Dockerfile from the top. A step can reuse its old layer only if every step before it was reused and its own inputs are unchanged. For COPY and ADD, the inputs are a checksum of the files being copied. For RUN, the input is the command text: Docker does not look at what the command would download. Once one step misses, every step after it runs again.

So instruction order is a speed decision. If COPY . . comes before RUN npm ci, any change to any file, even a README, invalidates the copy, and npm ci reinstalls every package. If you copy package.json and package-lock.json first, run npm ci, and copy the source after, a normal code change reuses the installed packages and only reruns the compile. The RUN rule has a trap too: RUN apt-get update on its own line is cached forever, so a later apt-get install may use a stale package list. Keep update and install in one RUN.

One line changed in src/ COPY . . first package files first FROM node:22-slim COPY . . RUN npm ci RUN npm run build FROM node:22-slim COPY package*.json RUN npm ci COPY src ./src RUN npm run build npm ci runs again npm ci stays cached cached input changed runs again
The first step whose input changed is where the cache stops. Everything below it reruns, so put the slow, rarely changing steps (installing packages) above the fast, often changing ones (copying source).

Two more tools help. The is everything in the folder you build from, minus what .dockerignore excludes. Without that file, node_modules, .git and your .env go into the context, slow every build, can break the cache on every commit, and can end up in the image. And a BuildKit , RUN --mount=type=cache,target=/root/.npm npm ci, keeps npm's download cache between builds, so even when the install step reruns it downloads less. BuildKit has been Docker's default builder since Docker Engine 23.0.

Quick check

To install a private package, a Dockerfile does COPY .npmrc ./ (it holds a registry token), then RUN npm ci, then RUN rm .npmrc. Is the token in the pushed image?

Small and safe: multi-stage builds, base images and users

Building the order service needs TypeScript, the Nest CLI, test tools and all dev dependencies. Running it needs Node, the compiled dist folder and the production packages. A keeps those apart: the first stage installs everything and compiles, and the final stage starts from a clean base and copies in only the results. Nothing from the build stage reaches the final image unless you copy it. Shipping safely: containers, CI/CD and releases introduced the pattern; here you choose the base image on purpose.

Base imageC libraryShell and package managerUse it for
node:22 (full Debian)glibcYes, plus compilers and gitBuild stages that compile native modules; too big to ship
node:22-bookworm-slimglibcYes (apt)A safe default for build and runtime
node:22-alpinemuslYes (apk)Small images, if every native module supports musl
gcr.io/distroless/nodejs22-debian12glibcNoRuntime only: least to attack, harder to debug

The C library column matters more than the size. Alpine uses instead of glibc. A native module compiled on Debian (a password hashing library, an image resizer, some database drivers) will not load on Alpine, and a few behaviours differ, DNS lookups among them (version dependent). Alpine works well when you build inside an Alpine stage and test there. images go further: they hold Node and the libraries it needs, with no shell and no package manager, so an attacker who gets in has very few tools, and a scanner finds fewer packages to report. The price is debugging: there is no sh to docker exec into. Whatever you pick, keep the build stage and the runtime on the same Node major version and the same C library, so native modules built in one run in the other.

Then take away root. The official Node images include a user called node (UID 1000), and the distroless :nonroot images use UID 65532. Set USER with a number, because Kubernetes can only check a "must not run as root" rule when the user is numeric (Kubernetes for application teams uses this). Leave the copied files owned by root: the app can read its own code but cannot change it. At run time you can go further with --read-only for the root file system and --cap-drop ALL, which a Node API listening on port 3000 does not need.

Finally, one process per container. If the order service, a nightly cron job and nginx share one container under a process supervisor, you cannot restart, scale or read the logs of one without the others, and nobody knows which process the orchestrator is watching. Run each as its own container, and write logs to standard output so the platform collects them.

Quick check

The order service now runs on a distroless image. During an incident, the on-call engineer types docker exec -it orders sh and gets "executable file not found". What is the right move?

Stopping cleanly: PID 1 and signals

Every deploy stops old containers. docker stop sends to the container's PID 1, waits 10 seconds by default, and then sends . Kubernetes does the same with a default grace period of 30 seconds. Your app gets one chance to stop accepting new requests, finish the ones in flight, close its Postgres and Redis connections and exit. If it misses that chance, requests are cut mid-way and the container exits with code 137 (128 + 9, killed by SIGKILL).

Three things make SIGTERM miss your code. The first is the shell form of CMD. CMD node dist/main.js is run as /bin/sh -c "node dist/main.js", so PID 1 may be the shell, and a shell does not pass SIGTERM on to its child (some shells replace themselves with a single command, so do not count on it either way). The exec form, CMD ["node", "dist/main.js"], starts node directly as PID 1. The second is npm start as the command: npm becomes PID 1, adds a process between the signal and your app, and its signal handling has changed between npm versions. Run node directly.

The third is subtle. The kernel treats PID 1 specially: a signal whose default action is "terminate" is ignored by PID 1 unless the process has installed a handler for it. Node installs no SIGTERM handler by default, so node as PID 1 with no handler ignores docker stop and waits to be killed. In NestJS, app.enableShutdownHooks() installs the handlers and runs your shutdown hooks. PID 1 also has a second job: it must collect () exited child processes. If the service starts child processes, for example to render PDF invoices, add a tiny init such as tini as PID 1, or use docker run --init locally; it forwards signals and reaps children.

docker stop, k8s delete SIGTERM PID 1: node PID 1: sh -c stops new requests signal not passed on finishes, closes pool waits 10 s (k8s: 30 s) exits in about 1 s SIGKILL, exit 137 orders in flight finish orders in flight cut
Same image, same stop command. On the left, node is PID 1 and has a SIGTERM handler, so it drains and exits. On the right, a shell is PID 1 and the signal never reaches node, so the grace period runs out and SIGKILL ends everything mid-request.

Health checks have the same two worlds. The Dockerfile HEALTHCHECK instruction makes Docker run a command inside the container every few seconds and mark it healthy or unhealthy. Docker Compose can wait for "healthy" before starting a dependent service, but plain Docker does not restart an unhealthy container, and Kubernetes ignores HEALTHCHECK completely. On Kubernetes you declare readiness, liveness and startup probes in the pod spec instead (ci-03). A HEALTHCHECK that calls curl also fails on a distroless image, which has no curl.

Quick check

The order service image has HEALTHCHECK CMD node dist/health.js. On the Kubernetes cluster, a pod hangs and stops answering, but it is never restarted. Why?

Tags, digests and the registry

A stores images. Mangrove's go to GitHub Container Registry as ghcr.io/mangrove/order-service. Each image has a that lists its layers by hash, and the SHA-256 hash of that manifest is the image's . A tag such as :3f9c2e1 or :latest is a name that points at a digest, and anyone with push rights can move it. A digest cannot move: if one byte changes, the digest changes.

That gives you a simple rule. Tag images for people, deploy them by digest. CI tags each build with the git commit so humans can find it, and the deploy step uses ghcr.io/mangrove/order-service@sha256:…, the exact bytes that passed the tests in staging. The same applies to the base image: node:22-bookworm-slim points at a new digest every time Node or Debian ships a patch. Pinning FROM node:22-bookworm-slim@sha256:… makes builds repeatable, and a bot such as Dependabot or Renovate opens a pull request when a new digest appears, so you still get security fixes, through review. One tag can also point at an index that lists one image per CPU architecture (for example amd64 and arm64), and each machine pulls the one that matches.

Before an image reaches staging, a scanner such as Trivy or Grype compares the packages inside it with public vulnerability lists. Smaller images give shorter, more useful reports. Secure supply chain and delivery metrics turns that into a policy, adds signatures and SBOMs, and pins the GitHub Actions used below to full commit SHAs.

Build it

Build a small, non-root, cache-friendly image for mangrove/order-service, prove it stops cleanly, and push it to GHCR from GitHub Actions with a digest you can deploy.

  1. Keep the build context small

    Create .dockerignore in the repository root. Everything listed here never reaches the builder, so it cannot bust the cache or leak into a layer.

    Output
    # .dockerignore
    node_modules
    dist
    coverage
    .git
    .github
    .env*
    *.log
  2. Write the multi-stage Dockerfile

    The build stage uses Debian slim with glibc; the runtime is distroless on the same Debian 12 base and the same Node major, so native modules carry over. Comments sit on their own lines.

    Dockerfile
    # syntax=docker/dockerfile:1
    # ---- build: all dependencies, compile TypeScript ----
    FROM node:22-bookworm-slim AS build
    WORKDIR /app
    # package files first: this layer changes only when dependencies change
    COPY package.json package-lock.json ./
    # npm's download cache lives in a cache mount, never in a layer
    RUN --mount=type=cache,target=/root/.npm npm ci
    # source last: a code change reruns only the steps below
    COPY tsconfig.json tsconfig.build.json nest-cli.json ./
    COPY src ./src
    # compile, then drop dev dependencies from node_modules
    RUN npm run build && npm prune --omit=dev
    
    # ---- runtime: Node only, no shell, no npm ----
    FROM gcr.io/distroless/nodejs22-debian12:nonroot
    WORKDIR /app
    ENV NODE_ENV=production
    # files stay owned by root: the app can read its code, not change it
    COPY --from=build /app/package.json ./
    COPY --from=build /app/node_modules ./node_modules
    COPY --from=build /app/dist ./dist
    # numeric user, so Kubernetes can verify "not root"
    USER 65532:65532
    EXPOSE 3000
    # the image's entrypoint is node; exec form, so node is PID 1
    CMD ["dist/main.js"]

    In production you would also pin both FROM lines by digest, as described above.

  3. Handle SIGTERM in the app

    Turn on Nest's shutdown hooks and close your connections in a hook. On SIGTERM, Nest stops the HTTP server and calls the hooks.

    TypeScript
    // src/main.ts
    import { NestFactory } from '@nestjs/core';
    import { AppModule } from './app.module';
    
    async function bootstrap() {
      const app = await NestFactory.create(AppModule);
      app.enableShutdownHooks();                  // installs SIGTERM and SIGINT handlers
      await app.listen(process.env.PORT ?? 3000, '0.0.0.0');
    }
    bootstrap();
    
    // src/db/db.service.ts
    import { Inject, Injectable, OnApplicationShutdown } from '@nestjs/common';
    import { Pool } from 'pg';
    
    @Injectable()
    export class DbService implements OnApplicationShutdown {
      constructor(@Inject('PG_POOL') private readonly pool: Pool) {}
    
      async onApplicationShutdown(signal?: string) {
        console.log(`shutting down on ${signal}`); // goes to stdout, so the platform keeps it
        await this.pool.end();                     // close Postgres connections cleanly
      }
    }
  4. Build, inspect and test the stop

    docker history lists each layer with its size and the instruction that made it, so you can see where the bytes are. Then start the container and time a stop.

    Terminal
    docker build -t order-service:dev .
    docker history order-service:dev                 # one line per layer, newest first
    docker image inspect order-service:dev --format '{{.Config.User}}'
    
    docker run -d --name os-test -p 3000:3000 --memory=512m order-service:dev
    time docker stop os-test                         # SIGTERM now, SIGKILL after 10 s
    docker inspect os-test --format '{{.State.ExitCode}}'
    docker rm os-test

    With the old CMD npm run start:prod, the stop looks like this (your timings will differ):

    Output
    65532:65532
    os-test
    
    real    0m10.42s
    user    0m0.02s
    sys     0m0.01s
    137

    About 10 seconds and exit code 137 means the signal never reached your code. With the Dockerfile above, the stop returns in about a second, the log shows "shutting down on SIGTERM", and the exit code is 0 or 143 (128 + 15), depending on how the process ends after cleanup. Either is a clean stop; 137 is the one to fix.

  5. Build and push from GitHub Actions, with a shared cache

    Add this job to the workflow from ci-01. The GITHUB_TOKEN can push to GHCR when the job grants packages: write. The gha cache stores layers between runs, because every runner starts empty.

    YAML
      image:
        needs: test
        if: github.event_name == 'push' && github.ref == 'refs/heads/main'
        runs-on: ubuntu-latest
        permissions:
          contents: read
          packages: write                       # push to ghcr.io with GITHUB_TOKEN
        outputs:
          digest: ${{ steps.build.outputs.digest }}
        steps:
          - uses: actions/checkout@v4
          - uses: docker/setup-buildx-action@v3
          - uses: docker/login-action@v3
            with:
              registry: ghcr.io
              username: ${{ github.actor }}
              password: ${{ secrets.GITHUB_TOKEN }}
          - id: build
            uses: docker/build-push-action@v6
            with:
              context: .
              push: true
              tags: ghcr.io/mangrove/order-service:${{ github.sha }}
              cache-from: type=gha
              cache-to: type=gha,mode=max       # also cache the build stage's layers
          - run: echo "Digest ${{ steps.build.outputs.digest }}" >> "$GITHUB_STEP_SUMMARY"

    Later jobs read needs.image.outputs.digest and deploy ghcr.io/mangrove/order-service@<digest>, never the tag. The action versions are the major versions current when this was written; check for newer ones.

Bring it to your team

Ask these in every pull request that touches a Dockerfile or the image job.

  • "Which base image is this, is it pinned by digest, and what opens a pull request when it gets a security fix?"
  • "If I change one line in src, which steps rerun, locally and in CI?"
  • "What is in the build context? Show me the .dockerignore."
  • "What is PID 1, and what happens in the first second after it gets SIGTERM?"
  • "Which user does the container run as, and can that user change the app's own files?"
  • "Which digest is in staging right now, and will production run exactly that one?"

Red flags

  • FROM node:latest, or a final stage that still holds compilers and dev dependencies
  • COPY . . before npm ci, or no .dockerignore at all
  • Tokens passed with ARG, ENV or a copied file, even one deleted later
  • A shell-form CMD or npm start as the container command
  • The API, a cron job and nginx in one container under a supervisor
  • Deploying the latest tag, or fixing things by hand with docker exec inside a running container

Practice

Predict where the build cache breaks

Write rebuiltSteps(steps, changedFiles, ignored). steps is the Dockerfile as a list of { instruction, reads }, where reads lists what a COPY reads from the build context (empty for other steps). A read pattern is an exact path ("package.json"), a folder ending in a slash ("src/", which matches every file under it) or "." (the whole context). changedFiles lists the paths changed since the last build. ignored (optional, default empty) lists .dockerignore entries in the same exact-or-folder form; ignored files are not in the context, so they never invalidate anything. Return the index of the first step whose cache is invalidated (every later step reruns too), or -1 if the whole build comes from the cache.

JavaScript exerciseRuns in your browser with tests
function rebuiltSteps(steps, changedFiles, ignored = []) {
  // 1. drop changed files that .dockerignore keeps out of the context
  // 2. find the first step that reads one of the remaining files
  return -1;
}
Ctrl + Enter runs
Show a solution
JavaScript
function rebuiltSteps(steps, changedFiles, ignored = []) {
  // an exact path, a folder ending in "/", or "." for everything
  const matches = (file, pattern) =>
    pattern === '.' || file === pattern || (pattern.endsWith('/') && file.startsWith(pattern));

  // ignored files never reach the builder
  const inContext = changedFiles.filter((f) => !ignored.some((p) => p !== '.' && matches(f, p)));

  for (let i = 0; i < steps.length; i++) {
    const reads = steps[i].reads || [];
    if (inContext.some((f) => reads.some((p) => matches(f, p)))) return i;  // first miss
  }
  return -1;                                                                // all cached
}

Note that src-old/legacy.ts does not match src/: the folder pattern includes the slash. In the naive Dockerfile any change hits step 2, which is why npm install reruns on every commit. In the good one, only dependency changes reach the install.

Pass the exercise to finish this task.

Review the old Dockerfile

A teammate opens a pull request for the reporting worker, a small NestJS job that builds the daily sales export and uses a private npm package. Review it. List each problem, why it matters at Mangrove, and the fix.

Dockerfile
FROM node:latest
COPY . .
ARG NPM_TOKEN
RUN echo "//npm.pkg.github.com/:_authToken=${NPM_TOKEN}" > .npmrc
RUN npm install
RUN npm run build
RUN apt-get update
RUN apt-get install -y cron
CMD npm run start:prod && cron -f
  • You flag node:latest as a moving, full-size base and propose a pinned slim or distroless base in a multi-stage build
  • You explain that COPY . . before the install reruns it on every change, and that there is no .dockerignore
  • You say the token stays in the .npmrc layer (and the build argument can show up in history), and propose a secret mount
  • You spot the separate apt-get update line (stale cache) and the missing USER (runs as root)
  • You explain that the shell-form command with npm and cron gives two processes, no clean SIGTERM handling, and suggest separate containers or a scheduled job
Show a model answer

Base. node:latest changes under you and ships compilers and git. Use a multi-stage build: node:22-bookworm-slim pinned by digest to build, distroless or slim to run.

Cache and context. COPY . . first means every commit reinstalls everything. Copy package.json and package-lock.json, run npm ci (not npm install, so the lock file is respected), then copy the source. Add a .dockerignore for node_modules, .git and .env*.

Secret. The token is written into .npmrc in its own layer, so anyone who can pull the image can read it, and the build argument can appear in the image history. Use RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci.

apt and user. A lone RUN apt-get update is cached forever; combine it with the install, or better, drop cron from the image. There is no USER, so it runs as root; add a numeric non-root user.

Command. The shell form runs sh -c as PID 1, and cron -f only starts after the app exits, so cron never runs while the worker is up. Two jobs in one container also cannot be restarted or scaled apart. Run the worker as CMD ["node", "dist/main.js"] and schedule the export with the platform's scheduler (a Kubernetes CronJob, for example) instead of cron inside the container.

Check yourself

The order service Dockerfile already copies the package files before the source. On a laptop, a one-line change rebuilds in 40 seconds. In GitHub Actions the same change takes 6 minutes, and the log shows npm ci running every time. What is the most likely cause?

Staging ran ghcr.io/mangrove/order-service:1.4.0 on Monday and passed every check. Production pulled :1.4.0 on Thursday and fails at startup with a missing module. Nobody changed the code. What happened, and what prevents it?

During a rolling deploy on Kubernetes, every old order service pod takes exactly 30 seconds to stop, ends with exit code 137, and never logs "shutting down". The image uses CMD ["npm", "run", "start:prod"]. What is the best fix?

Flashcards

Open my review

Key takeaways

  • A container is a host process in its own namespaces with cgroup limits on a shared kernel: fast to start, thinner isolation than a VM, so run as a numeric non-root user.
  • Images are read-only layers named by hash. Deleting a file later does not remove it, so keep secrets out with secret mounts and clean up inside the same RUN.
  • The cache stops at the first changed step. Copy package files, run npm ci, then copy source; use .dockerignore and a CI layer cache.
  • Multi-stage builds ship only runtime files. Slim is the safe default, distroless is smallest to attack, and Alpine needs musl-compatible native modules.
  • Exec-form CMD and a SIGTERM handler make stops clean; deploy by digest, not by tag, and leave health checking on Kubernetes to probes.

Sources

Episode transcriptHotel rooms, layers and the stop signal · about 13 minutes · tap any line to play from there
  1. NadiaImran, picture a load test on our staging cluster. Four times normal order traffic, festival level, and in the middle of it we roll out a new version of the order service.
  2. ImranSounds like a normal Tuesday for a deploy pipeline. What went wrong?
  3. NadiaEvery old container took exactly thirty seconds to stop, then died with exit code one three seven. About forty test checkouts failed per rollout. In a real rush, some customers would have tapped retry and paid twice.
  4. ImranAnd the cause was one line in a Dockerfile, I'm guessing, or you wouldn't be telling me this story.
  5. NadiaOne line. To see why, we need to look at containers from the inside. What a container really is, what an image is made of, and what happens when you ask one to stop.
  6. ImranGo on then. What's the everyday picture?
  7. NadiaA hotel. Every room shares one building: one foundation, one water supply, one electricity network. But each guest has a key that opens only their room, a window that shows only their view, and a meter that cuts the power if they use more than their share.
  8. ImranSo the building is the shared part, and the room is my private bit.
  9. NadiaRight. And every room is furnished from the same catalogue. Guests can move things and leave their own stuff, but when they check out, housekeeping resets the room, and whatever they left is gone.
  10. ImranLet me map it. The building is the Linux kernel. The key and the window are what?
  11. NadiaNamespaces. The power meter is a c-group. The catalogue furniture is the image's read-only layers. And the stuff a guest leaves behind lives in a thin writable layer that disappears when the container is removed.
  12. ImranAnd a virtual machine?
  13. NadiaA separate house with its own foundation. That's the big difference. A container has no kernel of its own.
  14. ImranSo what actually happens when I type docker run?
  15. NadiaIf you can see your screen, the first diagram is a column of boxes. If not, picture a chain. The Docker client asks the Docker daemon, the daemon hands the job to containerd, and containerd uses a small tool called run c to ask the kernel for two things.
  16. NadiaFirst, namespaces. Each one gives the process a private view of one resource. In its own process namespace, our node process is process number one and sees nothing else. It gets its own network, its own port three thousand, its own files, its own hostname.
  17. NadiaSecond, a c-group. That controls how much it may use. Say five hundred and twelve megabytes of memory and one CPU core. Then run c starts node as a normal process. Nothing boots.
  18. ImranSo if I log in to the host and list processes, I'd see node right there?
  19. NadiaExactly. With an ordinary host process number, like forty-eight thousand something. Inside its namespace, it thinks it's number one.
  20. ImranWhat happens when it goes over the memory limit? The server has thirty gigabytes free.
  21. NadiaDoesn't matter. The kernel's out of memory killer ends it, and Docker shows exit code one three seven with OOM killed set to true. Too much CPU is different: nothing dies, the process gets throttled and requests get slower.
  22. ImranAnd the security side? If they share a kernel, is a container as safe as a VM?
  23. NadiaIt isolates less. By default Docker doesn't turn on the user namespace, so root inside the container is root on the host kernel, held back by dropped capabilities and security profiles. That's the main reason we run the order service as an ordinary user.
  24. ImranNow the furniture. What's inside an image?
  25. NadiaA list of layers plus a small config. In a Dockerfile, every RUN, COPY and ADD adds a layer with the files it changed. Things like ENV, USER and the start command only change the config.
  26. NadiaEach layer is named by a hash of its content, so it's stored and downloaded once. If twenty services share a base image, a node pulls that base once, and a new order service version costs only the layers that changed.
  27. ImranHere's my guess at a trap. If I copy in a file with a password and delete it in the next step, the image is clean, right?
  28. NadiaThat's the trap, and the answer is no. The second diagram shows it. The delete is only a marker in a later layer that hides the file. The bytes are still in the earlier layer, and anyone who can pull the image can unpack it and read them.
  29. ImranSo the hotel resets the room, but the catalogue still has my diary in it.
  30. NadiaNicely put. For tokens, use a BuildKit secret mount, so the file exists only while one install step runs and never lands in a layer. And clean up package lists inside the same RUN that created them.
  31. ImranLayers also explain the build cache, don't they? Our builds were painfully slow.
  32. NadiaThey do. Docker walks the Dockerfile from the top. A step reuses its old layer only if every step before it was reused and its own inputs didn't change. For a COPY, that's a checksum of the files. For a RUN, it's only the command text. And once one step misses, every step after it runs again.
  33. NadiaThe third diagram has two columns. On the left, the Dockerfile copies the whole folder first. Change one line of source, and the copy misses, so npm c i reinstalls every package. On the right, it copies package dot json and the lock file first, installs, then copies the source.
  34. ImranSo on the right, a code change keeps the install cached and only reruns the compile.
  35. NadiaYes. Put slow steps that rarely change above fast steps that change all the time. And add a dot dockerignore file, so node modules, the git folder and your env file never even reach the builder.
  36. ImranNow base images. Everyone tells me to use Alpine because it's tiny.
  37. NadiaAlpine is small, but it uses a different C library called musl. A native module compiled on Debian won't load on Alpine, and a few behaviours differ, DNS lookups among them. It works if you build and test inside Alpine.
  38. NadiaDebian slim is the safe default. Distroless goes furthest: Node and its libraries, no shell, no package manager. An attacker who gets in finds almost no tools, and the scanner finds far fewer packages to complain about.
  39. ImranOkay. Back to the cold open. Why did the containers take thirty seconds to die?
  40. NadiaWhen a deploy stops a container, it sends the stop signal, SIGTERM, to process number one. Docker waits ten seconds by default, Kubernetes thirty, and then sends the kill signal, which can't be caught. The fourth diagram shows two lanes.
  41. NadiaIn the good lane, node is process number one and has a handler. It stops taking new requests, finishes the ones in flight, closes the database pool and exits in about a second.
  42. NadiaIn the bad lane, a shell or npm is process number one. Our Dockerfile said npm run start prod. npm sat in front of the app, and the app never saw the signal. So it waited the full thirty seconds and got killed mid-request.
  43. ImranAnd that's the one three seven. One hundred twenty-eight plus nine, the kill signal.
  44. NadiaExactly. There's one more subtle part. The kernel treats process number one specially: if it has no handler for the stop signal, the signal is ignored. Node has no handler by default. In NestJS, enable shutdown hooks installs one.
  45. ImranSo the fix is the exec form of the command, node started directly, plus shutdown hooks.
  46. NadiaYes. And if the service starts child processes, say to render PDF invoices, add a tiny init like tini as process number one. It passes signals on and cleans up finished children.
  47. ImranSo what happened with our real order service image?
  48. NadiaThe first Dockerfile had five lines and made an image of about one point one gigabytes. Every commit reinstalled every package, so an image build took about nine minutes. And when I unpacked the layers, I found the whole git history and an env file with the staging database password.
  49. ImranOuch. Rotated, I hope.
  50. NadiaThe same day. After one afternoon of fixes, the image was roughly one hundred and eighty megabytes, and that's approximate, because it depends on versions and how you measure. A source-only change rebuilds in about two minutes.
  51. ImranWalk me through the Build it section without reading code.
  52. NadiaYou write a dot dockerignore file. Then a two-stage Dockerfile: the build stage on Debian slim copies the package files, runs npm c i with a cache mount, copies the source, compiles, and removes dev packages.
  53. NadiaThe runtime stage starts from distroless Node on the same Debian, copies only the compiled code and production packages, sets a numeric non-root user, and starts node in exec form.
  54. NadiaIn the app you turn on shutdown hooks and close the database pool in a hook. Then you time a docker stop. About ten seconds and one three seven means the signal never arrived. About a second means it works.
  55. ImranAnd then CI pushes it to the GitHub container registry?
  56. NadiaWith a job that can write packages, a shared layer cache, a tag with the git commit, and the image digest printed in the summary. Later jobs deploy that digest, never the tag.
  57. ImranWhy does the digest matter so much?
  58. NadiaA tag is a name anyone with push rights can move. A digest is a hash of the content, so it always means the same bytes. Tag images for people, deploy them by digest. Pin your base image by digest too, and let a bot like Dependabot or Renovate open a pull request when a new one appears.
  59. ImranFor my next review, what should I ask when someone changes a Dockerfile?
  60. NadiaThree questions. If I change one line of source, which steps rerun, here and in CI? What is process number one, and what happens in the first second after it gets the stop signal? And which digest is in staging, and will production run exactly that one?
  61. NadiaHere's one to think about. A Dockerfile copies an npm config file with a registry token, installs, and then deletes the file in the next step. Is the token in the image? Take a second.
  62. ImranYes. The copy layer still holds it, and the delete only hides it.
  63. NadiaExactly. Anyone who can pull the image can read it.
  64. ImranLet me sum up. One: a container is a process with namespaces and c-groups on a shared kernel, so run it as a non-root user. Two: images are layers, and the cache stops at the first changed step, so install packages before copying source.
  65. ImranThree: the exec form plus a stop handler, so deploys don't cut orders in flight. And deploy by digest.
  66. NadiaOne nuance on your first point. It's an ordinary process, yes, but the walls are real: you can't see other processes, and the c-group can kill it at its memory limit even when the machine has plenty free.
  67. ImranFair. And next time?
  68. NadiaNext time we put this image on Kubernetes: deployments, services, probes and resource limits, and the classic mistake that turns a slow database into a restart storm.
Finished reading?Mark it complete to fill in its stop on your course line.

Use it at work

Runbook

Symptoms, first checks with commands, fixes and when to escalate, for one alert.

Open in the DevOps & TechOps Lab
Next stopKubernetes for application teams
Was this lesson helpful?
Questions and discussion

Loading the discussion…