Architecture

System Overview

See the rendered system diagram: architecture.svg.

At a high level: - firmware application code runs on top of xylolabs-sdk - xylolabs-protocol provides the shared XMBP wire format - xylolabs-hal-* crates implement target-specific platform adapters - xylolabs-server accepts HTTP/WebSocket ingest, enforces auth, and coordinates persistence - PostgreSQL stores metadata/index state while MinIO stores audio/blob payloads

Data Flow

1. MCU captures audio (I2S) and sensor data, feeds them into XylolabsClient.
2. XylolabsClient encodes audio via XAP or ADPCM into the ring buffer, accumulates metadata samples.
3. On each tick(), the client batches metadata into an XMBP packet and sends audio + metadata over HTTP POST (or WebSocket) to the server.
4. Server authenticates via API key, decodes XMBP, buffers samples in IngestManager, and flushes to PostgreSQL (metadata) and MinIO (audio chunks in XMCH format).

Crate Dependency Graph

Dependency flow summary: - xylolabs-server depends on xylolabs-core, xylolabs-db, xylolabs-storage, and xylolabs-transcode - xylolabs-transcode also depends on xylolabs-core and xylolabs-protocol - xylolabs-sdk depends on xylolabs-protocol - xylolabs-hal-* crates sit on top of xylolabs-sdk

Crate Roles

Crate	Role	no_std
xylolabs-core	Domain models, DTOs, error types, chunk format (XMCH), XMBP server-side codec, downsampling	No
xylolabs-db	SQLx repository layer + PostgreSQL migrations	No
xylolabs-storage	S3/MinIO client abstraction	No
xylolabs-transcode	FFmpeg worker, XAP server-side decoder, stale job reaper, transcoding pipeline	No
xylolabs-server	Axum web server, routes, middleware, ingest engine	No
xylolabs-protocol	XMBP wire protocol encoder/decoder (shared with SDK)	Yes
xylolabs-sdk	Embedded SDK core: client, session, codecs, transport	Yes
xylolabs-hal-*	Per-target Platform trait implementations (not in workspace)	Yes

Proprietary Protocols (Summary)

XMBP -- Xylolabs Metadata Batch Protocol

Binary big-endian wire protocol for batched sensor/metadata transmission from MCU to server. Shared between the no_std SDK encoder and the server decoder.

• Magic: 0x584D4250 ("XMBP"), Version: 1
• Supports 11 value types (F64, F32, I64, I32, Bool, String, Bytes, F64Array, F32Array, I32Array, Json)
• Up to 256 streams per batch, microsecond timestamps
• Encoder: zero-copy, zero-alloc into borrowed buffer with sticky overflow
• Decoder: validates all lengths before allocation (DoS-safe)
• Batch sequence wraps at u16::MAX with forward/backward gap detection

Full specification: XMBP-SPECIFICATION.md

XAP -- Xylolabs Audio Protocol

Proprietary MDCT-based spectral audio codec providing ~8:1 to 10:1 compression of 16-bit PCM.

• Sample rates: 8, 16, 24, 32, 48, 96 kHz
• Channels: 1--4 (encoded independently)
• Frame durations: 7.5 ms, 10 ms
• ~10 MIPS per channel (with DSP acceleration), ~8 KB RAM per channel
• Requires FPU; platforms without FPU use IMA-ADPCM instead
• Codec ID in XMBP: 0x03

Server-side decoder: The xylolabs-transcode crate includes an XAP decoder (xap_decode.rs) that performs inverse MDCT + dequantization. When a .xap file is uploaded, the transcode pipeline automatically decodes it to PCM WAV before passing to FFmpeg for transcoding to the target format. Sample rate is inferred from the frame header's frame_samples field.

Full specification: XAP-SPECIFICATION.md

XMCH -- Xylolabs Metadata Chunk Format

Server-side binary format for storing flushed metadata chunks in S3/MinIO.

Layout: - 32-byte header - magic: 0x584D4348 (XMCH) - version: 1 - value type: wire tag - reserved: 2 bytes - sample count: u32 - start timestamp: u64 microseconds - end timestamp: u64 microseconds - data size: u32 - timestamps column: N × u64 big-endian - values column: type-dependent payload

Chunks are optionally zstd-compressed via encode_chunk_compressed() before S3 upload.

Server Ingestion Pipeline

IngestManager

Located at crates/xylolabs-server/src/ingest/manager.rs.

• Maintains per-session state with per-stream StreamBuffers.
• Buffers up to 500,000 samples per stream before flush.
• Flush writes encoded XMCH chunks to S3 and metadata records to PostgreSQL.
• Concurrent flush guard (flushing flag) prevents double-flush. Samples arriving during flush are accumulated separately and merged on completion.
• On flush failure, original samples are prepended back for retry.
• Supports live event broadcasting via tokio::sync::broadcast for real-time WebSocket subscribers.
• Stale session eviction based on last activity timestamp.

Batch Sequence Tracking

The server uses classify_batch_sequence() with forward/backward gap heuristics to distinguish reordering from genuine gaps at u16 wraparound boundaries.

Downsampling

LTTB (Largest Triangle Three Buckets) algorithm at crates/xylolabs-core/src/downsample.rs for query-time visual downsampling of time series data. Preserves visual shape while reducing point count.

Auth Architecture

Dual Auth Model

Path	Method	Auth
/api/v1/ingest/*	API Key	X-Api-Key header, scoped to facility
/api/v1/auth/*	Public	Rate-limited (10 req/min per IP)
All other /api/v1/*	JWT	Bearer token, RBAC-enforced

RBAC Hierarchy

• super_admin -- global access, manages all facilities
• facility_admin -- manages own facility's data and users
• user -- read-only access to own facility's data

Rate Limiting

Auth endpoints use in-memory rate limiting via mini_moka::sync::Cache with 60-second TTL windows. Max 10 attempts per IP per minute. Trusted proxy IPs can be configured for correct X-Forwarded-For extraction.