Deployment Guide

Environment Variables

Copy .env.example to .env and configure.

Required

Variable	Description
CORS_ALLOWED_ORIGINS	Comma-separated allowed browser origins (production should include https://admin.api.xylolabs.com)
DATABASE_URL	PostgreSQL connection string
POSTGRES_USER / POSTGRES_PASSWORD / POSTGRES_DB	Docker Compose PostgreSQL bootstrap credentials
JWT_SECRET	HMAC-SHA256 signing key (access tokens)
JWT_REFRESH_SECRET	HMAC-SHA256 signing key (refresh tokens)
MINIO_ROOT_USER / MINIO_ROOT_PASSWORD	MinIO root credentials used by the storage service and bucket bootstrap
S3_ACCESS_KEY / S3_SECRET_KEY	MinIO credentials
INITIAL_ADMIN_EMAIL / INITIAL_ADMIN_PASSWORD	First admin account (created on first run if no users exist)

Optional

Variable	Default	Description
BIND_ADDR	0.0.0.0:3000	Listen address inside the container
DATABASE_MAX_CONNECTIONS	20	Connection pool size
JWT_ACCESS_TTL_SECS	86400 (1d)	Access token lifetime (LOCKED — see Session Length Policy)
JWT_REFRESH_TTL_SECS	31536000 (1y)	Refresh token lifetime (LOCKED — never reduce; logs out every operator)
S3_ENDPOINT	http://minio:9000	Internal Docker MinIO endpoint
S3_BUCKET	--	Storage bucket name
S3_REGION	--	S3 region
S3_PATH_STYLE	--	Use path-style addressing
TRANSCODE_CONCURRENCY	--	Max concurrent transcode jobs
TRANSCODE_DEFAULT_FORMAT	--	Default output format
TRANSCODE_DEFAULT_BITRATE	--	Default output bitrate
TRANSCODE_STALE_TIMEOUT_SECS	7200	Reap orphaned transcode jobs after this many seconds
GPU_HEALTH_CHECK_INTERVAL_SECS	60	Interval in seconds between GPU server health checks
INFERENCE_WORKER_CONCURRENCY	4	Max concurrent inference jobs processed by the inference worker
UPLOAD_MAX_SIZE	104857600 (100MB)	Max upload size in bytes
STATIC_DIR	./frontend/dist	Legacy admin dashboard dist path
STATIC_DIR_APP	./static-app	Operator dashboard (frontend-app) dist path
APP_FRONTEND_HOSTS	app.xylolabs.com	Comma-separated allowed origins for the operator dashboard
ALLOWED_WS_ORIGINS	(empty — allow all)	Comma-separated WebSocket origin allowlist; empty permits all origins
ANOMALY_BROADCAST_CAPACITY	10000	In-memory broadcast ring-buffer capacity for anomaly SSE feed
WEBHOOK_ALLOW_PLAINTEXT_HTTP	false	Allow http:// webhook URLs; enable only for closed-network deployments
INFERENCE_JOB_STALE_TIMEOUT_SECS	600	Reap inference jobs stuck in running after this many seconds
DEVICE_EVENTS_RETENTION_DAYS	30	Retention window for the SP5 device_events feed
API_REQUEST_LOG_RETENTION_DAYS	30	Retention window for api_request_logs (C18-AGG-8)
AUDIT_LOG_RETENTION_DAYS	365	Retention window for audit_log (S8-2). Default 365 because compliance regimes typically require a 1-year audit trail; lower only if your compliance scope permits.
ANOMALY_REPORT_RETENTION_DAYS	90	Retention window for anomaly_reports (S8-1). The table grows at ~26.5k rows/24h; 90 days covers most operational review cadences.
INGEST_SESSIONS_RETENTION_DAYS	90	Retention window for closed ingest_sessions (S8-10). Pruning cascades to metadata_chunks via ON DELETE CASCADE, so one worker bounds both tables.
DEVICE_HEALTH_HISTORY_RETENTION_DAYS	90	Retention window for device_health_history (S8-10). Every health POST writes a row; 90 days covers operational trend review.
LIVE_STREAM_CONNECTIONS_RETENTION_DAYS	90	Retention window for CLOSED live_stream_connections rows (S8-10). Open rows are untouched (partial index).
UWB_SURVEYS_RETENTION_DAYS	90	Retention window for uwb_surveys (C4-DEF-4). Pruning cascades to uwb_survey_edges + uwb_survey_solutions via ON DELETE CASCADE, so one worker bounds all three tables.
ACOUSTIC_DETECTOR_ENABLED	false	Enable the Phase 3 acoustic predictive-maintenance detector worker
FUSION_DETECTOR_ENABLED	false	Enable the Phase 2 multi-sensor fusion anomaly detector worker
FINGERPRINT_WORKER_ENABLED	false	Enable the Phase 7 search-by-sound fingerprint worker
GEMINI_MODEL	gemini-3.8-flash	Model used by the daily report + facility assistant
GEMINI_API_KEY	(empty)	Gemini API key; enables the daily report + facility assistant when set
LLM_ENDPOINT	https://generativelanguage.googleapis.com/v1beta/openai/chat/completions	OpenAI-compatible chat/completions URL all LLM calls POST to (set to OpenRouter to switch providers)
LLM_API_KEY	(falls back to GEMINI_API_KEY)	Bearer key for LLM_ENDPOINT
LLM_MODEL	(falls back to GEMINI_MODEL)	Model id sent to LLM_ENDPOINT (OpenRouter style: google/gemini-3.8-flash)
VOICE_ESCALATION_ENABLED	false	Enable the Phase 12 LLM voice-call escalation (Twilio)
VOICE_ESCALATION_TO_NUMBER	(empty)	Destination number for voice-call escalation
XYLOLABS_ENV	development	Deployment environment (development / test / production); production refuses weak or default secrets
JWT_PREVIOUS_SECRET	(empty)	Previous access-token signing key kept valid during rotation; when set must be ≥32 bytes and differ from JWT_SECRET / JWT_REFRESH_SECRET
TRUSTED_PROXY_IPS	(empty)	Comma-separated IPs/CIDRs of trusted reverse proxies; enables X-Forwarded-For / X-Real-IP client-IP resolution for audit + auth rate limiting
UPLOAD_MAX_SIZE_BYTES	(alias)	Alias for UPLOAD_MAX_SIZE (max upload size in bytes); either name is accepted
VAPID_PUBLIC_KEY / VAPID_PRIVATE_KEY	(empty)	Web-push VAPID keypair; enables browser push notifications when both are set
VAPID_SUBJECT	(built-in default)	VAPID sub contact (mailto: or https:) sent with push requests
SMS_PROVIDER	(empty)	SMS backend: empty (disabled), twilio, or vonage
SMS_FROM_NAME	Xylolabs	Sender name used on outbound SMS
TWILIO_ACCOUNT_SID / TWILIO_AUTH_TOKEN / TWILIO_FROM_NUMBER	(empty)	Twilio SMS + voice-escalation credentials; required when SMS_PROVIDER=twilio or VOICE_ESCALATION_ENABLED=true
VONAGE_API_KEY / VONAGE_API_SECRET / VONAGE_FROM	(empty)	Vonage SMS credentials; required when SMS_PROVIDER=vonage

RBAC

Role	Scope	Permissions
super_admin	Global	Full access to all facilities, users, and system config
facility_admin	Facility	Manages own facility's data and users
user	Facility	Read-only access to own facility's data

Database Migrations

Migrations run automatically on startup via sqlx::migrate!() in xylolabs-db. If any migration fails, the server process exits and Docker restarts it — which means the container crash-loops until the migration is fixed and redeployed. Migration failures in production are a P0.

Migration files: crates/xylolabs-db/migrations/ (140 files; verify with ls crates/xylolabs-db/migrations/*.sql | wc -l).

Naming and ordering rules

• Filename format: YYYYMMDDHHMMSS_<snake_case_description>.sql.
• Version prefixes MUST be strictly monotonic. Never reuse a version number. sqlx's MIGRATOR orders by the numeric prefix; duplicate prefixes produce undefined ordering and will almost always break on the dependent side.
• Dependency ordering. If migration N references a column, index, or table created by migration M, then N's version MUST be greater than M's.
• Before committing a new migration, verify monotonicity: bash ls crates/xylolabs-db/migrations | awk -F_ '{print $1}' | sort -c Silent exit means OK; any output means you have a duplicate or out-of-order prefix.

Known incident: facility_id unique index (2026-04-24)

Two migrations were authored under the same prefix 20260418000002:

• 20260418000002_add_password_changed_at.sql
• 20260418000002_add_firmware_active_unique_index.sql (referenced firmware_releases.facility_id, which is not created until 20260418000003_add_firmware_facility_scope.sql)

sqlx sorted the unique-index migration before the one that added facility_id, so it failed with column "facility_id" does not exist and the app container crash-looped on deploy. The unique-index migration was renumbered to 20260418000009 so it runs after the facility_scope migration. Use this as the canonical reminder: always check that a migration's dependencies are actually in place by prefix order, not by filename alphabet.

Writing safe migrations

• Prefer idempotent DDL: CREATE TABLE IF NOT EXISTS, CREATE INDEX IF NOT EXISTS, ADD COLUMN IF NOT EXISTS.
• Backfill non-null columns in three steps: add nullable → UPDATE → SET NOT NULL.
• Never edit a migration file that has already been applied in production — write a new, forward-only migration instead. The _sqlx_migrations table records applied versions; changing history triggers a checksum mismatch.
• scripts/deploy.sh runs a preflight before building the remote image: it validates local and remote migration filenames/order/duplicates, then compares production _sqlx_migrations versions and SQLx SHA-384 checksums against crates/xylolabs-db/migrations/. If any applied migration file is missing or edited, deploy stops before replacing the running container.

House rule: defensive pre-clean before a new UNIQUE index

A migration that adds a UNIQUE (including partial-unique) index to an EXISTING table must not assume the table already satisfies the constraint — rows written before the index existed can violate it, and CREATE UNIQUE INDEX fails outright on the first violation rather than skipping offending rows. Every such migration MUST include a defensive pre-clean step (an UPDATE/DELETE that resolves or removes the violating rows) immediately before the CREATE UNIQUE INDEX, even when production is believed to already satisfy the constraint.

20260710150000_deployments_active_per_device_unique.sql is the model to follow: it adds idx_deployments_active_per_device (at most one active firmware_deployments row per device) by first UPDATE-ing every non-newest active row per device to cancelled (with an explanatory error_message, keeping the row rather than deleting it), and only then building the index with CREATE UNIQUE INDEX IF NOT EXISTS. On production the index already existed out-of-band, so both the pre-clean and the IF NOT EXISTS no-op there — but a migrations-only database (dev, CI, integration tests) never had the constraint and could have accumulated duplicate active rows, so without the pre-clean the index build would fail on exactly those environments.

Constraint to revisit if hot-table sizes grow: CREATE INDEX CONCURRENTLY cannot run inside sqlx::migrate!() — Postgres refuses CONCURRENTLY inside a transaction block, and sqlx wraps every migration in one. For the tables this house rule applies to today (bounded by retention workers, low row counts), a plain synchronous CREATE UNIQUE INDEX is fast enough not to matter (see "Large-table migrations" below). If a table this rule applies to ever grows into the tens-of-millions-of-rows range, the pre-clean pattern above still holds, but the index build itself will need the same out-of-band CONCURRENTLY treatment (a one-off psql run against production, migration-only for dev/CI) rather than a synchronous in-migration build.

Large-table migrations: CONCURRENTLY / batching (C5-ARCH-14)

sqlx::migrate!() runs synchronously at boot, before the app admits traffic. A migration that builds an index on, or bulk-DELETEs from, a LARGE table (metadata_chunks-class: tens of millions of rows in production) can hold a long lock or simply run for minutes, either crash-looping the container on a deploy health-check timeout or stalling every future deploy. Future migrations touching a table of that class should build indexes CONCURRENTLY out-of-band (outside sqlx::migrate!(), e.g. a one-off psql run against production) or batch the DELETE the way the retention workers do (small batches with an inter-batch pause), rather than a single synchronous statement.

Two cycle-4 migrations built directly against production without batching: 20260708060000_timeline_rollups.sql (creates timeline_rollups + worker_watermarks, plus a non-CONCURRENT index on metadata_chunks) and 20260708061000_ingest_sessions_fk_actions.sql (FK constraint changes plus a single-statement DELETE of orphaned audio_fingerprints rows). Both applied cleanly — the metadata_chunks index build is a narrow exception the boot-before-traffic guarantee allows for a one-time build, and the audio_fingerprints DELETE targeted a small derived table — but neither should be treated as a template for a routinely large or fast-growing table. The cycle-5 rollup covering index (20260708120000_timeline_rollups_device_bucket_idx.sql) is a plain, non-CONCURRENT build for the same underlying reason: timeline_rollups is bounded by design (288 rows/day/stream-name, 90-day retention), so the build is seconds, not the minutes a metadata_chunks-class table would take — see the migration's own sizing note.

Docker

# Full stack (production)
docker compose up

# Development mode
docker compose -f docker-compose.dev.yml up

# Test infrastructure
docker compose -f docker-compose.test.yml up -d

Production Deployment

Server

• Host: api.xylolabs.com (AWS EC2)
• Instance type: t4g.medium (2 vCPU, 4 GiB RAM, arm64)
• Region: ap-northeast-2 (Seoul)
• Public IP: Elastic IP, reassignable — resolve via dig +short api.xylolabs.com (do not hardcode; a previously documented IP went stale on 2026-07-06)
• OS: Ubuntu 24.04 LTS
• Architecture: arm64
• Swap: 4 GiB at /swapfile (vm.swappiness=10)
• User: ubuntu
• SSH Key: ~/.ssh/xylolabs-api.pem
• SSH: ssh -i ~/.ssh/xylolabs-api.pem ubuntu@api.xylolabs.com

Domains

Domain	Purpose	Port
api.xylolabs.com	API endpoints	3000 (proxied via nginx)
admin.api.xylolabs.com	Legacy admin dashboard	3000 (proxied via nginx, serves static + API)
app.xylolabs.com	Operator dashboard (frontend-app)	3000 (proxied via nginx, serves static-app + API)
docs.api.xylolabs.com	Static documentation bundle	443 (nginx static site)

Infrastructure

• Reverse proxy: nginx with Let's Encrypt (certbot, webroot flow)
• App: Docker Compose (app + postgres + minio)
• Docs: Static bundle generated from docs/ and served by nginx
• SSL: Auto-renewed via certbot timer

First-time server bootstrap

ssh -i ~/.ssh/xylolabs-api.pem ubuntu@api.xylolabs.com 'bash -s' < scripts/setup-server.sh

The setup script installs Docker, Docker Compose, nginx, and certbot, then creates:

• /opt/xylolabs-api/.env
• /opt/xylolabs-api/bootstrap-credentials.txt

Deploy or redeploy

./scripts/deploy.sh

Expected downtime. deploy-remote.sh starts services in two phases. Phase 1 (up --no-deps --wait postgres minio) recreates the data services only when their own compose config changed, and is a no-op otherwise. Phase 2 swaps the app against healthy data services. The app stops in under a second on SIGTERM: SSE streams end, WebSockets close, and in-flight HTTP requests get at most 10 s (HTTP_DRAIN_GRACE). An app-only deploy therefore costs the app's stop plus its start (a few seconds). The 90 s stop_grace_period is a backstop, not the expected stop time. Before 2026-10-09 an open dashboard SSE stream held the drain until the SIGKILL at 90 s.

Backups

The postgres service archives WAL continuously, and the host runs nightly dumps and weekly base backups into /var/backups/xylolabs-api/pg. An off-host target pulls them, and MinIO keeps deleted object versions for 30 days. deploy-remote.sh prepares the backup directory before docker compose up. Rollout, restore drills and recovery procedures are in the Backup and Restore Runbook.

Every third-party image in docker-compose.yml is pinned by digest. minio/minio and minio/mc exist only in the API host's image store and in the off-host archive. Never prune them. deploy-remote.sh aborts before building when a pinned image is missing. See the Container Image Runbook.

Docker Commands on Server

cd /opt/xylolabs-api
docker compose up -d          # Start services
docker compose logs -f app    # View logs
docker compose restart app    # Restart app
docker compose down           # Stop all

nginx + certificate flow

scripts/deploy.sh now:

1. Validates local migration filenames, ordering, duplicate versions, and SQLx checksums
2. Builds the static docs bundle for docs.api.xylolabs.com
3. Rsyncs the source tree to the EC2 host ($REMOTE_DIR/src/)
4. Uploads the deployment config bundle (nginx configs, docker-compose.yml) via scp
5. Validates the remote migration set and checks production-applied migration versions/checksums before the Docker build
6. Builds the Docker image on the remote host (docker build -f docker/Dockerfile) — detached (see below)
7. Starts Docker Compose on the server and waits for the local app health endpoint
8. Applies temporary HTTP-only nginx configs
9. Obtains Let's Encrypt certificates for api.xylolabs.com, admin.api.xylolabs.com, docs.api.xylolabs.com, and app.xylolabs.com
10. Switches nginx to the final TLS configs, reloads it, and verifies the public API, admin, docs, and operator-app hosts respond

Detached build + resume (P1023 WP3, C3-DOC-5)

The remote build+swap phase (steps 5-7) runs detached on the server (setsid + nohup under $REMOTE_DIR/deployments/<REV_ID>/, with deploy.log, deploy.pid, and a deploy.exit completion sentinel). The local deploy.sh merely polls the sentinel over short reconnecting SSH sessions (25 s interval, 1 h ceiling). Practical consequences:

• A dropped SSH session / closed laptop / killed local script does NOT cancel the remote build. The ~2-vCPU fat-LTO build takes hours; it keeps running unattended and completes the compose-up + health-check + current repoint on its own.
• Resume instead of re-launching: re-running bash scripts/deploy.sh while a build for a different rev is in flight aborts with a hint; to resume polling an interrupted deploy, re-invoke with the printed rev id: REV_ID='<rev>' bash scripts/deploy.sh (same-rev re-invocation resumes polling the in-flight build rather than starting a duplicate).
• Watch the log directly when needed: ssh ubuntu@api.xylolabs.com tail -f <REMOTE_DIR>/deployments/<REV_ID>/deploy.log.
• Retention keeps the last N rev snapshots and always pins the rev deployments/current points at (plus its image), so stacked failed deploys cannot prune the live rev's rollback material (C3-CRITIC-5).