Skip to content

Deployment Guide

Environment Variables

Copy .env.example to .env and configure.

Required

Variable Description
CORS_ALLOWED_ORIGINS Comma-separated allowed browser origins (production should include https://admin.api.xylolabs.com)
DATABASE_URL PostgreSQL connection string
POSTGRES_USER / POSTGRES_PASSWORD / POSTGRES_DB Docker Compose PostgreSQL bootstrap credentials
JWT_SECRET HMAC-SHA256 signing key (access tokens)
JWT_REFRESH_SECRET HMAC-SHA256 signing key (refresh tokens)
MINIO_ROOT_USER / MINIO_ROOT_PASSWORD MinIO root credentials used by the storage service and bucket bootstrap
S3_ACCESS_KEY / S3_SECRET_KEY MinIO credentials
INITIAL_ADMIN_EMAIL / INITIAL_ADMIN_PASSWORD First admin account (created on first run if no users exist)

Optional

Variable Default Description
BIND_ADDR 0.0.0.0:3000 Listen address inside the container
DATABASE_MAX_CONNECTIONS 20 Connection pool size
JWT_ACCESS_TTL_SECS 900 (15min) Access token lifetime
JWT_REFRESH_TTL_SECS 604800 (7d) Refresh token lifetime
S3_ENDPOINT http://minio:9000 Internal Docker MinIO endpoint
S3_BUCKET -- Storage bucket name
S3_REGION -- S3 region
S3_PATH_STYLE -- Use path-style addressing
TRANSCODE_CONCURRENCY -- Max concurrent transcode jobs
TRANSCODE_DEFAULT_FORMAT -- Default output format
TRANSCODE_DEFAULT_BITRATE -- Default output bitrate
TRANSCODE_STALE_TIMEOUT_SECS 7200 Reap orphaned transcode jobs after this many seconds
GPU_HEALTH_CHECK_INTERVAL_SECS 60 Interval in seconds between GPU server health checks
INFERENCE_WORKER_CONCURRENCY 4 Max concurrent inference jobs processed by the inference worker
UPLOAD_MAX_SIZE 104857600 (100MB) Max upload size in bytes
STATIC_DIR ./frontend/dist Legacy admin dashboard dist path
STATIC_DIR_APP ./frontend-app/dist Operator dashboard (frontend-app) dist path

RBAC

Role Scope Permissions
super_admin Global Full access to all facilities, users, and system config
facility_admin Facility Manages own facility's data and users
user Facility Read-only access to own facility's data

Database Migrations

Migrations run automatically on startup via sqlx::migrate!() in xylolabs-db. If any migration fails, the server process exits and Docker restarts it — which means the container crash-loops until the migration is fixed and redeployed. Migration failures in production are a P0.

Migration files: crates/xylolabs-db/migrations/ (59 files as of 2026-04-24).

Naming and ordering rules

  • Filename format: YYYYMMDDHHMMSS_<snake_case_description>.sql.
  • Version prefixes MUST be strictly monotonic. Never reuse a version number. sqlx's MIGRATOR orders by the numeric prefix; duplicate prefixes produce undefined ordering and will almost always break on the dependent side.
  • Dependency ordering. If migration N references a column, index, or table created by migration M, then N's version MUST be greater than M's.
  • Before committing a new migration, verify monotonicity: bash ls crates/xylolabs-db/migrations | awk -F_ '{print $1}' | sort -c Silent exit means OK; any output means you have a duplicate or out-of-order prefix.

Known incident: facility_id unique index (2026-04-24)

Two migrations were authored under the same prefix 20260418000002:

  • 20260418000002_add_password_changed_at.sql
  • 20260418000002_add_firmware_active_unique_index.sql (referenced firmware_releases.facility_id, which is not created until 20260418000003_add_firmware_facility_scope.sql)

sqlx sorted the unique-index migration before the one that added facility_id, so it failed with column "facility_id" does not exist and the app container crash-looped on deploy. The unique-index migration was renumbered to 20260418000009 so it runs after the facility_scope migration. Use this as the canonical reminder: always check that a migration's dependencies are actually in place by prefix order, not by filename alphabet.

Writing safe migrations

  • Prefer idempotent DDL: CREATE TABLE IF NOT EXISTS, CREATE INDEX IF NOT EXISTS, ADD COLUMN IF NOT EXISTS.
  • Backfill non-null columns in three steps: add nullable → UPDATE → SET NOT NULL.
  • Never edit a migration file that has already been applied in production — write a new, forward-only migration instead. The _sqlx_migrations table records applied versions; changing history triggers a checksum mismatch.

Docker

# Full stack (production)
docker compose up

# Development mode
docker compose -f docker-compose.dev.yml up

# Test infrastructure
docker compose -f docker-compose.test.yml up -d

Production Deployment

Server

  • Host: api.xylolabs.com (AWS EC2)
  • OS: Ubuntu 24.04 LTS
  • Architecture: arm64
  • User: ubuntu
  • SSH Key: ~/.ssh/xylolabs-api.pem
  • SSH: ssh -i ~/.ssh/xylolabs-api.pem ubuntu@api.xylolabs.com

Domains

Domain Purpose Port
api.xylolabs.com API endpoints 3000 (proxied via nginx)
admin.api.xylolabs.com Legacy admin dashboard 3000 (proxied via nginx, serves static + API)
app.xylolabs.com Operator dashboard (frontend-app) 3000 (proxied via nginx, serves static-app + API)
docs.api.xylolabs.com Static documentation bundle 443 (nginx static site)

Infrastructure

  • Reverse proxy: nginx with Let's Encrypt (certbot, webroot flow)
  • App: Docker Compose (app + postgres + minio)
  • Docs: Static bundle generated from docs/ and served by nginx
  • SSL: Auto-renewed via certbot timer

First-time server bootstrap

ssh -i ~/.ssh/xylolabs-api.pem ubuntu@api.xylolabs.com 'bash -s' < scripts/setup-server.sh

The setup script installs Docker, Docker Compose, nginx, and certbot, then creates:

  • /opt/xylolabs-api/.env
  • /opt/xylolabs-api/bootstrap-credentials.txt

Deploy or redeploy

./scripts/deploy.sh

Docker Commands on Server

cd /opt/xylolabs-api
docker compose up -d          # Start services
docker compose logs -f app    # View logs
docker compose restart app    # Restart app
docker compose down           # Stop all

nginx + certificate flow

scripts/deploy.sh now:

  1. Builds the production image locally
  2. Builds the static docs bundle for docs.api.xylolabs.com
  3. Uploads the Docker image plus deployment bundle to the EC2 host
  4. Starts Docker Compose on the server
  5. Applies temporary HTTP-only nginx configs
  6. Obtains Let's Encrypt certificates for api.xylolabs.com, admin.api.xylolabs.com, docs.api.xylolabs.com, and app.xylolabs.com
  7. Switches nginx to the final TLS configs and reloads it