Docker and reproducible environments

Why "works on my machine" is not enough

Amina has a test node running on her laptop. Her colleague Brice launches it too: it crashes, because his machine has a different version of a library. Nobody worked badly: the program depended on things nobody had written down.

Three words first.

• A container is a program started as an isolated process: it sees its own files, its own libraries and its own view of the network. Unlike a virtual machine, it shares the Linux kernel of the host (on Windows or macOS, Docker Desktop runs a small Linux virtual machine in between, but the logic is the same).
• An image is the read-only package from which containers are started. It relates to a container like a program file relates to a running program.
• A Dockerfile is the written recipe that builds an image.

Reading a Dockerfile

Example for a small Node.js service (the workshop indexer). Values in angle brackets are for you to fill in.

FROM node:<tag>@sha256:<digest-you-looked-up>
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
USER node
CMD ["node", "index.js"]

Each instruction is a step. Those that change files (COPY, RUN) produce a cached layer: if nothing changed since the last build, Docker reuses it ("CACHED" in the output). As soon as one step changes, every step after it is redone. Note: here COPY . . runs before USER node, so the files in /app belong to root and are only readable by node; if the application must write there, use COPY --chown=node:node . ..

Reproducible means pinned

Three levers, from the most important to the least visible.

1. The base image. FROM ubuntu:latest promises nothing: latest is just the default tag name, the one the publisher moved last. A tag can be moved; a digest (@sha256:…) cannot, because it is the SHA-256 hash of the image manifest: different content gives a different digest, as in the hash-functions workshop. You read it with docker images --digests for an image pulled from a registry (a locally built image has none until it is pushed); otherwise on the registry's page. For a multi-architecture image, the digest you pin is usually that of the index, which points to the per-architecture variants.
2. The dependencies. A lock file (package-lock.json) freezes exact versions; npm ci installs them as they are and fails if the file and package.json disagree. npm install may resolve things differently.
3. Your own version. Tag what you build with an explicit number, for example bac-testnode:lab-1, never only latest.

This course deliberately states no "current" version number: versions change. You learn to pin them and to check them (the project's release page, docker images --digests, npm ls), then to write down what you chose.

Instruction order matters

In the example we copy the two dependency files first, then run npm ci, then copy the rest of the code. If you edit index.js, only the last copy is redone: the slow installation stays cached. Had you written COPY . . before RUN npm ci, every one-line code change would rerun the full installation. Also add a .dockerignore file to keep .git, node_modules and above all .env out of the context sent to the build.

What this does NOT guarantee

• Not identical bytes: a build may embed dates, or an apt-get update that fetches whatever the mirror holds today. "Reproducible" here means: same behaviour from the same pinned inputs.
• Pinned does not mean safe. A digest also freezes a known flaw: you must update on purpose, by changing the digest and retesting.
• A tag that works today may point elsewhere tomorrow.

Recap: base image by digest, locked dependencies, explicit version, well-ordered cache.

Three nodes, one command

To test a protocol, a single node is not enough: you need at least three peers that talk to each other. Running docker run three times with long options invites mistakes. Docker Compose describes the whole set in a YAML file and starts it in one go: docker compose up -d. The -d option gives the terminal back. Here is an excerpt. The bac-testnode image and its health command are fictitious, made up for the exercise.

services:
  node1:
    image: bac-testnode:lab-1
    command: ["--name", "node1", "--listen", "0.0.0.0:30333"]
    ports:
      - "127.0.0.1:9944:9944"
    healthcheck:
      test: ["CMD", "bac-testnode", "health"]
      interval: 5s
      retries: 5
    volumes:
      - node1-data:/data
  node2:
    image: bac-testnode:lab-1
    command: ["--name", "node2", "--bootnode", "node1:30333"]
    depends_on:
      node1:
        condition: service_healthy
    volumes:
      - node2-data:/data
volumes:
  node1-data:
  node2-data:

Service names are addresses

Compose creates a private network for the project. On it, each service is reachable by its name: node1:30333 means port 30333 of the node1 container. A classic trap: writing 127.0.0.1:30333 in node2's configuration. Inside a container, 127.0.0.1 ("localhost") means the container itself, not the machine nor the other containers. The message is then "connection refused", because nobody listens on that port inside node2.

Publishing a port: host:container

The line "127.0.0.1:9944:9944" reads: host address, host port, container port. Without the 127.0.0.1 prefix, the port is opened on all interfaces of the machine. Between containers on the same network no publishing is needed: it is only for reaching a service from the host. With three nodes on one machine, two cannot publish the same host port: you write 9944:9944, 9945:9944, 9946:9944. Same port inside the containers, three different ports on the host.

Started is not ready

A plain depends_on only settles start order: Compose starts node1 before node2, without waiting for node1 to accept connections. To wait, define a healthcheck (a command Docker runs at regular intervals) and write condition: service_healthy. Even then, a robust client retries its connection: a peer can fall over after startup.

Commands to know

What this does NOT guarantee

• A Compose network simulates the topology, not reality: zero latency, no outages, a single machine. The three nodes share the same processor and the same disk.
• The internal network is isolated from the outside by default, but every port you publish opens a door to the host.
• A green healthcheck says the test command succeeds, not that the node is synchronised with the others.

Recap: service names as addresses, host:container to publish, health before order.

The container forgets everything

Your node has synced up to block 500; you type docker compose down then docker compose up -d. Does it resume at block 500 or at zero? It depends on where it writes. A container's filesystem is a disposable writable layer: when the container is removed, it disappears. To keep data, you put it on a volume, managed outside the container.

Two kinds of mount

Three sequences, three results

1. docker compose down then docker compose up -d: the containers are removed and recreated, but the named volume stays. The node resumes where it was (for example block 500).
2. docker compose down -v: the -v option also removes the named volumes declared in the file. The node restarts from zero and must resync everything.
3. docker compose stop then start: the container is not even removed.

Remember the difference between down and down -v: it is like closing a drawer versus throwing it away.

Permissions: back to Linux

A bind mount keeps the host's owner. If your image runs as an unprivileged user (say id 10001) and ./data belongs to root, the node fails with a message like:

node1  | Error: cannot create directory /data/chain: Permission denied (os error 13)

(Illustrative message.) It is the same permission mechanism as in the Linux workshop: fix it with chown on the host folder, never with a chmod 777 that opens everything to everyone.

Chain data and keys: two different things

• On a full node that does not validate, chain data (blocks, state) can be recovered by resyncing from the network. Losing it costs time and bandwidth, not funds.
• A node's or validator's keys cannot be recovered. They do not go in the image. Back them up separately, offline and encrypted.
• Important exception, for a validator. Its protection against double signing often lives in the data directory (for example the priv_validator_state.json file for CometBFT; other clients keep comparable data). Deleting that volume and restarting the validator can wipe this state and cause a double signature, hence a penalty. Never delete that volume on a validator without backing it up, and never restore it to an older version. Locations and details depend on the client: read its documentation.

Mind the opposite risk too: on many proof-of-stake chains, running two instances with the same validator key can be penalised. Restoring a backup next to a still-living node is therefore not harmless.

Watching disk space

A node writes continuously. Illustrative example: 50 MB per hour is 50 × 24 = 1,200 MB, about 1.2 GB per day, so about 36 GB after 30 days. A full disk stops the node, sometimes corrupting a database. Use df -h on the host and docker system df to see what Docker occupies.

What this does NOT guarantee

• A volume is not a backup: it lives on the same disk. If the disk dies, everything goes.
• A volume mounted over an image is not reproducible like the image: its content depends on the node's history.
• Copying a database folder while the node writes can give an inconsistent copy; stop the service first, or use the node's export tool if one exists.

Recap: data on a volume, down keeps, down -v erases, keys outside the image.

A container is not a fortress

Many believe "isolated" means "safe". A container shares the host's kernel: a flaw in the application, plus excessive privileges, and the attacker gets a hold on the machine. Four rules reduce the possible damage.

1. Do not run as root

By default, a container's process runs as root inside the container. In the first lesson's example, USER node replaces it with an unprivileged user. If the application is compromised, the attacker cannot, for example, modify the image's system files. For your own binary, create a dedicated user with a fixed numeric id and give it only the data folder.

2. A minimal base image

Every package in the image is code to keep updated and one more attack surface. A "slim" or minimal base holds fewer packages than a full distribution: fewer flaws to watch, a smaller image to download (valuable on a limited connection). Trade-off: fewer debugging tools inside. You cannot install what you would need in an emergency; better debug from outside.

To slim down further, a multi-stage build compiles in a first, bulky image, then copies only the result into a minimal final image. The build tools do not end up in the delivered image:

FROM node:<tag>@sha256:<digest> AS builder
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:<tag-slim>@sha256:<digest>
WORKDIR /app
COPY --from=builder /app/dist ./dist
USER node
CMD ["node", "dist/index.js"]

(Outline: script and folder names vary by project.)

3. No secret in the Dockerfile

A key written in the Dockerfile becomes part of the image, hence of every copy of the image.

Secrets arrive at run time: a read-only mounted file, an unversioned env_file, or the platform's secrets mechanism. For a build that needs a secret, Docker offers build secrets (--mount=type=secret). If a real secret was ever written into an image or a repository, treat it as compromised: revoke it and replace it. Removing it from history is not enough.

4. Do not open more ports than needed

An RPC admin port published as 9944:9944 is reachable from the whole network of the machine. Write 127.0.0.1:9944:9944 to limit it to the host. A point often missed: by default, Docker manages the host's network rules itself, and a published port can bypass a firewall configured with ufw in the Linux workshop. Check with a test from another machine.

Other habits: never mount the Docker socket (/var/run/docker.sock) into a container without reason, since whoever controls it effectively controls the whole machine; and belonging to the docker group amounts in practice to being an administrator.

An example read in full

FROM ubuntu:latest
ENV NODE_KEY=FAKE-KEY-DO-NOT-USE-0000
COPY . /app
CMD ["/app/bac-testnode", "--rpc-external"]

Defects: base neither pinned nor minimal, a key (fictitious here) in the image, no USER, and the COPY . /app context may carry a .env. The fix combines a dedicated user, injecting the key at run time, and the port limited to 127.0.0.1.

What this does NOT guarantee

• Hardening does not remove application flaws; it limits what a flaw allows.
• A small image is not a flawless image.
• An image scanner finds known flaws, not all of them.

Recap: unprivileged user, minimal base, run-time secrets, restricted ports.