Testing Guide

Quick Reference

# Server tests
cargo test                                      # All workspace tests
cargo test -p xylolabs-server                   # Server unit + non-ignored tests
./scripts/run-tests.sh                          # Full docker-backed server integration suite

# Protocol crate tests (encoder + decoder roundtrips)
cargo test -p xylolabs-protocol --all-features

# SDK core tests (client, session, ring buffer, config, health)
cargo test -p xylolabs-sdk --all-features

# Audio pipeline tests (decimation, I2S config, full encode pipeline -- runs on host)
cd tests/audio-pipeline && cargo test

# Frontend tests
cd frontend && npx tsc -b --noEmit && npx vite build && pnpm test
cd frontend-app && npx tsc -b --noEmit && npx vite build && pnpm test

cargo test -p xylolabs-server and cargo test -p xylolabs-server --tests --lib do not run the ignored docker-backed integration tests by themselves. Use ./scripts/run-tests.sh (or cargo test -p xylolabs-server --tests -- --ignored) when you need the full server integration lane.

Cross-Process Locking For DB-Gated Suites

Every crates/xylolabs-server/tests/api_*.rs suite goes through TestApp::setup() (crates/xylolabs-server/tests/common/mod.rs), which cleans and reseeds the shared TEST_DATABASE_URL database before each test. An in-process tokio::sync::Mutex already serializes TestApps within one test binary, but that offers no protection when a second process — another suite binary, another CI lane, another agent — targets the same database at the same time; one process's cleanup can interleave with another's fixture setup and produce spurious foreign-key violations that look like real regressions. setup() now also takes a Postgres session-level pg_advisory_lock (a fixed key, held on a dedicated non-pooled connection for the whole test) before cleaning the database, so a second process attempting the same thing simply queues on the lock instead of corrupting shared state. Running one DB-gated suite is unaffected; running two or more concurrently against the same database now serializes safely rather than racing. The acquisition is time-bounded rather than unbounded: if the lock cannot be taken within a generous budget, setup() fails loudly with a diagnostic naming the Postgres backend holding it (looked up from pg_locks) instead of hanging forever, so a stuck or crashed peer surfaces as an actionable error rather than a mysterious stall.

Rust SDK Tests

Protocol Crate

cargo test -p xylolabs-protocol --all-features

Tests XMBP encoder/decoder roundtrips for all 11 value types, boundary conditions, truncation handling, and performance benchmarks (encode/decode must complete in < 100 µs per 2 kHz batch).

SDK Core

cargo test -p xylolabs-sdk --all-features

Tests client initialization, session lifecycle, ring buffer SPSC correctness, config validation, health statistics, codec encode/decode, and cross-language parity with the C SDK.

Audio Pipeline

cd tests/audio-pipeline && cargo test

Runs on host (not cross-compiled). Tests: - FIR decimation (96kHz to 16kHz): coefficient verification, DC passthrough, stereo interleaving, continuity, near-Nyquist attenuation, max-amplitude overflow safety - Full encode pipeline (I2S to decimate to XAP): compression ratio, header format, silence/sine

54 host tests + 7 QEMU tests. CI-gated since cycle 3 (C3-TE-7): the host suite runs in .github/workflows/sdk-tests.yml (rust-tests job) on every push/PR.

Burn-In Tests

# Start test infrastructure + API server
docker compose -f docker-compose.test.yml up -d
cargo run --bin xylolabs-server --release  # with test env vars

# Run burn-in scenarios (requires running API server)
./scripts/run-burnin.sh standard           # 60s quick validation
./scripts/run-burnin.sh stress             # 120s high throughput (4ch @96kHz)
./scripts/run-burnin.sh endurance          # 120s+ sustained load
./scripts/run-burnin.sh multi-device       # 10 concurrent devices

# QEMU ARM emulation (simulates ~150MHz RP2350)
./scripts/run-burnin.sh standard --qemu

# ESP32-S3 Xtensa QEMU emulation
./scripts/run-qemu-esp32.sh

# Cleanup
docker compose -f docker-compose.test.yml down -v

Pass/fail criteria: < 5% batch failure rate, < 3 reconnects. Performance metrics reported per run.

Legacy C SDK Tests

# Run all unit tests natively (requires gcc + libm)
gcc -Wall -Wextra -Werror -O2 -o /tmp/test_xmbp sdk/c/test/test_xmbp.c sdk/c/common/src/xmbp_encoder.c -Isdk/c/common/include -lm && /tmp/test_xmbp
gcc -Wall -Wextra -Werror -O2 -o /tmp/test_xap sdk/c/test/test_xap.c sdk/c/common/src/xap_encoder.c -Isdk/c/common/include -lm && /tmp/test_xap
gcc -Wall -Wextra -Werror -O2 -o /tmp/test_ring_buffer sdk/c/test/test_ring_buffer.c sdk/c/common/src/ring_buffer.c -Isdk/c/common/include && /tmp/test_ring_buffer
gcc -Wall -Wextra -O2 -o /tmp/test_client sdk/c/test/test_client.c sdk/c/common/src/xylolabs_client.c sdk/c/common/src/xmbp_encoder.c sdk/c/common/src/adpcm_encoder.c sdk/c/common/src/xap_encoder.c sdk/c/common/src/ring_buffer.c sdk/c/common/src/http_transport.c sdk/c/common/src/session_manager.c -Isdk/c/common/include -lm && /tmp/test_client

# Run all C tests via Docker (includes ARM cross-compilation syntax check + QEMU)
docker build -t xylolabs-sdk-test -f sdk/c/test/Dockerfile . && docker run --rm xylolabs-sdk-test

# Expected results: XMBP 34/34, XAP 45/45, Ring Buffer 358/358, Client 11/11

# Run E2E test suite (native + ARM cross-compile + QEMU emulation)
./scripts/run-sdk-e2e.sh

# Or manually: build and run E2E Docker image
docker build -f sdk/c/test/Dockerfile.e2e -t xylolabs-sdk-e2e . && docker run --rm xylolabs-sdk-e2e
# E2E covers: 8 native suites, 4 cross-compile targets (M4F/M33/RV64/Xtensa-compat), 16 QEMU-user + 3 QEMU-sys suites

# Run Xtensa (ESP32-S3) tests using Espressif IDF Docker image (~2GB)
./scripts/run-sdk-xtensa.sh

# Or include Xtensa tests in the main E2E suite
RUN_XTENSA=true ./scripts/run-sdk-e2e.sh

E2E Ingest Test

Located in tests/e2e-ingest/. Generates 4-channel sine wave audio (1/2/3/4 kHz), encodes with XAP, packages with XMBP, and uploads to the live API.

cd tests/e2e-ingest
API_KEY=xk_... cargo run --release

Tests the full pipeline: PCM → XAP encode → XMBP frame → HTTP POST with X-Api-Key → server ingest.

Remote E2E Pipeline Test

Located in tests/remote-e2e/. Tests against a live API (e.g. api.xylolabs.com) covering the full XMBP/XAP pipeline with metadata verification.

# Create .env in project root with:
# E2E_API_KEY=xk_...
# E2E_HOST=api.xylolabs.com
# E2E_EMAIL=admin@xylolabs.com
# E2E_PASSWORD=...

cd tests/remote-e2e
cargo run

Test rules (constants): - 4-channel audio: 1 kHz / 2 kHz / 3 kHz / 4 kHz sine waves - 16 kHz sample rate, 10 ms frames, 64 kbps, 500 ms duration - 3 metadata streams: temperature (f64), humidity (f64), pressure (f32) at 10 Hz

Steps: Login → Create ingest session (4 streams) → Send XMBP audio batch (50 XAP frames) → Send XMBP metadata batch (15 samples) → Close session → Query all 4 streams and verify sample counts → Upload XAP file → Verify transcode job created.

XAP Upload & Transcode Tests

Located in crates/xylolabs-server/tests/api_upload_transcode.rs. Tests XAP file upload and transcode job creation (requires Docker test infrastructure).

cargo test --test api_upload_transcode -- --ignored

XMBP Pipeline Tests

Located in crates/xylolabs-server/tests/api_xmbp_pipeline.rs. Full 4-channel XAP + metadata e2e test with ingest session lifecycle.

cargo test --test api_xmbp_pipeline -- --ignored

Atomicity Tripwire Tests

Located in crates/xylolabs-server/tests/api_atomicity_tripwires.rs. Compile-time static-assertion tests that verify critical atomicity invariants are preserved in the source code (e.g. that high-value DB writes are not fire-and-forget). These tests require no Docker and no running database — they pass or fail at compile time by inspecting source text at test startup.

cargo test -p xylolabs-server api_atomicity

XAP Decoder Tests

Unit tests for the server-side XAP decoder (inverse MDCT + dequantization).

cargo test -p xylolabs-transcode --lib xap_decode

23 tests grouped by topic: LC3 transport-header parsing + SDK tag validation + oversized-claim allocation guard (3); decode roundtrips — standard, truncated-tail, embedded-header resync, mid-stream misframe, single-channel extraction, cross-codec liblc3↔lc3-codec parity (6); RMS/dBFS conversion + legacy/garbage none-returns (6); legacy sample-rate inference (2); legacy-frame decode — short header + silence (2); and stream-decoder framing/resync — whole-blob, mid-packet split, drop-before-header, resync-after-corruption (4).

Browser Validation

Headless Playwright script that validates all admin dashboard pages load correctly.

cd tests && npm install playwright && npx playwright install chromium
node browser-validate.mjs

Checks: Login → Dashboard → Uploads → Transcode → Metadata Sessions → Devices → Alerts → Users → API Keys → Settings → Firmware → Facilities, with explicit browser console and pageerror validation.

Backup / Restore Tests

bats tests/backup/lib.bats   # retention selection, WAL naming + pruning (no Docker)
tests/backup/e2e.sh          # production compose services + real backup scripts, end to end (Docker)

The e2e test runs the exact digest-pinned production images. minio/minio and minio/mc cannot be pulled any more, so load them first with scripts/images/restore-images.sh (see RUNBOOK-IMAGES.md). The test refuses to start without them.

Image Pin Checks

cargo test -p xylolabs-server --test compose_image_pins   # every image digest-pinned + in docker/images.lock
scripts/images/check-images.sh docker-compose.yml docker-compose.test.yml \
  test/device-sim/docker-compose.sim.yml                  # every pinned image obtainable (Docker)

The e2e test brings up the production docker-compose.yml postgres, minio and minio-init in an isolated compose project without published ports. It covers WAL archiving, base backup, dump, off-host pull with tamper detection, dump and PITR restore drills (including a timestamp target), MinIO versioning and the mirror, retention, and the freshness check. See RUNBOOK-BACKUP-RESTORE.md.

Frontend Tests

cd frontend && npx tsc -b --noEmit && npx vite build && pnpm test
cd frontend-app && npx tsc -b --noEmit && npx vite build && pnpm test

Frontend unit/component tests (frontend/src: 50 *.test.ts(x) files; frontend-app/src: 32 — enumerate with find frontend/src frontend-app/src -name '*.test.ts*'). Representative suites: - authStore.test.ts -- Auth store state management - client.test.ts -- API client layer - ProtectedRoute.test.tsx -- Route protection component - RoleGate.test.tsx -- Role-based UI gating - Sidebar.test.tsx -- Sidebar navigation component

Test Coverage

Audio pipeline (54 host tests + 7 QEMU tests): - FIR decimation (96kHz to 16kHz): coefficient verification, DC passthrough, stereo interleaving, continuity, near-Nyquist attenuation, max-amplitude overflow safety - Full encode pipeline (I2S to decimate to XAP): compression ratio, header format, silence/sine - QEMU ESP32 firmware: decimation + full pipeline verification

Known Failing Tests

None. The suite is green on main as of 2026-08-03: 715 unit tests and 421 integration tests across 69 suites.

If you hit a failure, it is yours — but first confirm you ran the suite the way CI does. cargo test alone runs almost nothing here. Every integration test is #[ignore], so without -- --ignored you get a screen of test result: ok with 0 passed; N ignored, which looks identical to success:

cargo test -p xylolabs-server --tests --no-fail-fast -- --ignored

--no-fail-fast matters too: cargo stops at the first failing binary, and a run that halts after 15 of 69 suites reports a passing count that means nothing.

What the 2026-08-03 sweep found

The list above used to hold two entries. Running the suite properly surfaced 18, all pre-existing, most invisible for months because nobody executed them. Two were real product defects; the rest were tests that had drifted from the code. Three patterns account for nearly all of them, and each is worth checking before you write a new integration test:

Spawned writes read synchronously (6 tests). log_audit and the API request-log middleware both tokio::spawn their INSERT so the response is never delayed. A test that asserts on those rows immediately after the response races the insert. Poll (60 × 100 ms is the convention in this repo); do not assume the row is there.
Column names that never existed (5 tests). target_id and actor_id on audit_log (really resource_id / user_id), status_code on api_request_logs (really response_status), gpu_servers.name (really hostname). A SQL string only fails when it runs, so an unrun test carries its typos indefinitely.
Assertions outliving the behaviour they pinned (4 tests). The api-logs listing defaulted to API-key-only in 2026-05-07 (5d287ad7) and the tests kept listing JWT traffic; POST /users returns 201 and three suites still expected 200; the raw-mode chunk cap moved 500 → 10 000 and the seed that used to clear it stayed at 505, so that test could no longer fail for the right reason.

The two product defects are worth naming, since both were found by tests rather than by review: voice_clips and device_push_tokens were missing from the facility delete guard (operator-uploaded PCM with no second copy, and the native-app half of push_subscriptions), and malformed JSON bodies answered in text/plain, contradicting the "all errors are JSON" contract in API Reference §18.

Note that running an older commit against the shared test database fails with Migrate(VersionMissing(...)) once a newer migration has been applied to it. To bisect, point TEST_DATABASE_URL at a fresh database rather than resetting the shared one:

docker exec xylolabs-api-postgres-test-1 \
  psql -U xylolabs_test -d postgres -c "CREATE DATABASE bisect OWNER xylolabs_test;"
TEST_DATABASE_URL="postgresql://xylolabs_test:test_secret@localhost:5433/bisect?sslmode=disable" \
  cargo test -p xylolabs-server --test <suite> -- --ignored

Known Test Coverage Gaps

Server (Priority 1) (list re-audited cycle 3, C3-TE-6 — the previous bullets had gone stale): - ~~IngestManager: zero unit tests~~ — now covered (ingest/manager.rs carries 20+ unit tests: sequence gap wrapping, backpressure, flush guard, stale eviction) - ~~WebSocket ingest: zero integration tests~~ — now covered (tests/api_ingest.rs: binary batches, gap detection, wraparound, watchdog stale-session close, dongle-ownership rejection, manager-cache-loss recovery) - ~~JWT middleware tamper/expiry coverage~~ — now covered (C3-TE-1, cycle 3: tests/api_auth.rs test_me_endpoint_rejects_tampered_signature + test_me_endpoint_rejects_expired_token); RBAC middleware unit tests unchanged (middleware/rbac.rs)

SDK (Priority 2 -- legacy C only): - xylolabs_client.c: audio_feed, meta_feed, tick paths untested (only init covered) - session_send_audio: entirely untested - session_ensure_active reconnect: exponential backoff untested - ADPCM codec path: not validated in QEMU CI matrix

Security: - No brute-force rate limiting test for API keys - ~~No JWT signature verification test~~ — covered by the C3-TE-1 tamper test (tests/api_auth.rs)