Skip to content

Container Image Runbook

How the stack's third-party images are pinned, where the copies that cannot be pulled any more are kept, and how to get them back onto a host.

Why this exists

MinIO Inc. put the MinIO community edition into maintenance mode in December 2025 and later archived the minio/minio and minio/mc GitHub repositories. In September 2026 it deleted both images from Docker Hub. Checked on 2026-10-06:

Source Result
docker.io/minio/minio, docker.io/minio/mc Hub API object not found (404); registry pull 401 authentication required
quay.io/minio/minio, quay.io/minio/mc the anonymous token grants no pull action; every manifest request is 401, including by digest
github.com/minio/minio, github.com/minio/mc archived, read-only; release tags and commits are still readable

Production was running minio/minio:latest and minio/mc:latest from the API host's image cache. The tag came from MINIO_IMAGE / MINIO_MC_IMAGE in the host .env. A docker image prune -a, a host rebuild or a new host would have left the object store (about 80 GiB of audio) impossible to start as configured.

What runs (pinned by digest)

docker/images.lock is the single list. Every image: in a compose file and every image a CI workflow runs must be repo@sha256:<digest> and must appear there.

Image Version Pullable?
minio/minio@sha256:14cea493…936e RELEASE.2025-09-07T16-13-09Z (commit 07c3a429, go1.24.6) no: archive only
minio/mc@sha256:a7fe349e…1727 RELEASE.2025-08-13T08-35-41Z (commit 7394ce0d, go1.24.6) no: archive only
postgres@sha256:52e6ffd1…16dd0 PostgreSQL 18.3 (production) yes, by digest from Docker Hub; archived as well
postgres@sha256:ddbc320a…7319 PostgreSQL 18.3 (test stack) yes
postgres@sha256:3645570c…3826 17-bookworm (device simulator) yes

These are the exact images production has been running since July 2026, so pinning them changed nothing about behaviour. docker-compose.yml no longer reads MINIO_IMAGE / MINIO_MC_IMAGE. The old lines in the host .env are now inert and can stay.

The application image xylolabs-api is built on the host by scripts/deploy.sh and is the only reference that is not digest-pinned.

Enforcement

  • cargo test -p xylolabs-server --test compose_image_pins fails on any image reference that is tag-only, ${VAR}-overridable or missing from the lock. It also fails on any lock entry that nothing uses, and on any unpullable entry that has no archive checksum.
  • scripts/images/check-images.sh [--archive-dir DIR] [compose files…] proves that each pinned image can be obtained. An image passes if it is in the local store, if docker manifest inspect resolves it from its registry, or if --archive-dir holds its tarball with the locked sha256.
  • scripts/deploy-remote.sh runs check-images.sh on the API host before it builds or recreates anything. A missing image aborts the deploy and leaves the running stack as it was.

Rules for the API host

  • Never docker image rm, docker rmi or docker image prune -a the MinIO images. The docker image prune -f at the end of deploy-remote.sh removes dangling images only. It does not touch these images, which are tagged and in use.
  • The host uses the containerd image store (docker info: driver-type io.containerd.snapshotter.v1). With that store, repo@sha256:<index digest> resolves locally without a pull, which was verified on 2026-10-06. Keep it enabled on any replacement host.

Off-host archive

docker save output, taken on 2026-10-06 from the API host. Every tarball was verified with sha256 on the host, on the workstation and on the NAS.

  • Location: TrueNAS, /mnt/pool/home/hletrd/xylolabs-api/image-rescue-20261006/ on the pool/home dataset (raidz2). The files are read-only (444).
  • Contents: minio-minio.tar, minio-mc.tar, postgres-18.tar, the running application image xylolabs-api-rev-20261002T055703Z_0000000d88.tar (a rollback convenience only, since it can be rebuilt from git), and SHA256SUMS.
  • Architecture: linux/arm64 only. The API host is aarch64, and the other platforms in the multi-arch index were never downloaded.
  • A copy also stays on the API host in /home/ubuntu/image-rescue-20261006/.
File sha256
minio-minio.tar 031922b875ec00cb7b13952f9b3714c042f030a0878da22c5862e4208adf71bd
minio-mc.tar 6349c53d1f8716efc5fc84df752622bf94be90ede79c9519085cd53a5818028d
postgres-18.tar bae6e111b05927adb60771f9a21d0aae47c9d2c25813937cf2dddbf34e581b53
xylolabs-api-rev-20261002T055703Z_0000000d88.tar 64e546344a456c5174e03d18ab67de67c9d99253176689a8f59202c698a0ef55

Restore onto a host

# 1. copy the archive to the host (from a machine that can reach both)
scp -i ~/.ssh/hletrd-nas.pem -r \
  hletrd@172.30.60.100:/mnt/pool/home/hletrd/xylolabs-api/image-rescue-20261006 /tmp/
scp -i ~/.ssh/xylolabs-api.pem -r /tmp/image-rescue-20261006 ubuntu@api.xylolabs.com:/tmp/

# 2. on the host: verify, load, and prove the pinned digests resolve
cd /opt/xylolabs-api/src
bash scripts/images/restore-images.sh /tmp/image-rescue-20261006
bash scripts/images/check-images.sh docker-compose.yml   # every line "ok   local"

restore-images.sh refuses a tarball whose sha256 does not match the lock, skips images that are already present, and fails if a loaded image does not resolve under its pinned digest. That happens on a daemon without the containerd image store; turn the store on rather than repinning.

Source rebuild (last resort)

Use this when the archive is unavailable, or when the host is not arm64:

scripts/images/build-minio-from-source.sh [--platform linux/amd64]

docker/minio/Dockerfile rebuilds the same release tags from the archived GitHub repositories: - MinIO at commit 07c3a429bfed433e49018cb0f78a52145d4bedeb; - mc at commit 7394ce0dd2a80935aded936b09fa12cbb3cb8096.

It uses the same toolchain (golang:1.24.6, pinned by digest) on the same UBI base release (ubi9/ubi-micro:9.6-1754467928, pinned by digest). The env, entrypoint, CMD and bundled mc match the upstream Dockerfile.release at that commit. The script checks that both binaries report the pinned release and commit.

On 2026-10-06 the rebuild was checked on arm64. Both binaries report the pinned release and commit with Runtime: go1.24.6. A rebuilt server read a data directory written by the pinned upstream image, including all object versions and the delete marker.

What differs from upstream: - the image digest (upstream built and signed its binaries in a separate pipeline); - no bundled static curl (nothing uses it: the healthcheck is mc ready local).

That is why compose is never switched to the rebuild silently. To run it, repin docker-compose.yml, the docker/images.lock entry (source stays archive, with a new docker save tarball and sha256 archived as above) and the CI workflow in one reviewed commit. Then deploy.

Changing an image later

  1. Add the new repo@sha256:<index digest> line to docker/images.lock. Use the multi-arch index digest, which docker buildx imagetools inspect reports as Digest:.
  2. If its registry might lose it (any MinIO image, or any third-party publisher), docker save it on a containerd-store daemon. Then archive the tarball next to the others, and record the file name and sha256 in the lock.
  3. Switch the compose reference, run cargo test -p xylolabs-server --test compose_image_pins and scripts/images/check-images.sh, and drop the old lock line.

Long-term

The pinned MinIO release no longer receives security fixes. This is acceptable for now because MinIO is reachable only on 127.0.0.1 and on the compose network, and is never exposed to clients (the backend proxies all S3 traffic). A move to a maintained S3-compatible server is a separate project, because it means migrating about 80 GiB / 8.4 M objects. Candidates include SeaweedFS, Garage, RustFS and the community fork pgsty/minio. pgsty/minio is the closest: it is a fork of the same codebase, but it has a single maintainer and newer releases (RELEASE.2026-08-04), so it would not be the same version.