Skip to content

Inference Ops Console — Module 1: Authoritative Anomaly Thresholds

As-built note (2026-07-15): the mechanism below shipped largely as designed; these drift points are corrected here rather than rewritten into the prose below (kept for historical context): - §7.1/§10 rubric LLM elaboration was never built. services/rubric.rs::generate_rubric is a pure deterministic-template render — no call to alert_llm.rs/http_json.rs exists anywhere in the file. The module's own doc comment marks LLM elaboration a future enhancement; the fleet currently receives resolved numbers in a fixed template, not LLM-authored prose. - §5.3 validation is route-layer, not repo-layer. repo::threshold_override::upsert does a raw INSERT ... ON CONFLICT with no validation; the registry/bounds check lives in routes/inference_ops.rs:238-250, one layer up, called before the repo function. - §5.3 list_for_scopes was never added. The repo only exposes list_all/upsert/delete; the resolver (threshold_resolver.rs) loads all override rows via list_all() and filters in-memory per device/facility/hw_version. - §6 cache invalidation is not a threshold_version counter. The shipped mechanism is a deterministic FNV-1a content-hash snapshot_version (used as the rubric ETag) plus a GenerationGuard-based snapshot_gen/hw_memo_gen pair for race-safe refresh — no incrementing counter exists in code. - §8 DELETE takes query params, not a body. delete_threshold reads scope/scope_id via Query<DeleteQuery>; docs/API.en.md/API.ko.md already document this correctly, so only this internal spec was stale. - §8 audit actions are namespaced. The shipped action strings are inference_ops.set_threshold_override and inference_ops.delete_threshold_override, not the bare set_threshold_override named below. - §10 missing/wrong API-key scope returns 403, not 400. require_api_key_scope (middleware/api_key_auth.rs:169) maps to AppError::Forbidden → HTTP 403; no public doc asserts 400 for this case.

docs/API.en.md/docs/API.ko.md §29 remain the authoritative description of current behavior.

  • Status: Design (approved delivery mechanism: pull-based rubric fetch)
  • Date: 2026-07-12
  • Author: Xylolabs API
  • Supersedes/relates: tactical hotfix fix(inference): suppress overheating verdicts below 60°C operating limit (commit 00000007506)

1. Motivation

On 2026-07-11/12 the anomaly feed paged operators with a "critical overheating, temperature exceeds 46°C" alert. The monitoring hardware's design-spec maximum continuous operating temperature is 60°C, so ~46°C is normal operating warmth — a false positive. Finding and fixing the threshold was the real problem: the number lived inside a served LLM system-prompt on the GPU inference fleet (google/gemma-4-E4B-it), which is not present in any of the four repositories (xylolabs-api, xylolabs-gpu-ops, firmware, nas-ops). It could not be located or edited.

A tactical guard now suppresses sub-60°C temperature-overheating results at crates/xylolabs-server/src/routes/inference_results.rs (MAX_OPERATING_TEMP_C = 60.0). But that is the third independent copy of "what is dangerous," none of them authoritative:

# Where Value(s) Kind
1 GPU fleet gemma system-prompt (off-repo) 46°C, NOx 25000, noise −18 dBFS served LLM rubric
2 crates/xylolabs-server/src/routes/inference_results.rs MAX_OPERATING_TEMP_C = 60.0 Rust const
3 crates/xylolabs-server/src/ingest/manager.rs:1646,1668,1708 warn > 1000.0, critical > 10000.0 Rust literals in the real-time detector

Goal: make the API the single, editable, authoritative source of truth for anomaly thresholds. Every consumer — the inference-result guard, the ingest detector, and the GPU fleet's LLM rubric — derives its numbers from that one place. Structured numbers are authoritative; the LLM only writes prose around them.

2. Goals / non-goals

Goals (this module): - A server-side registry of anomaly threshold keys with typed bounds and defaults. - Hierarchical overrides: global → hardware-version → facility → device. - A pure, tested resolver producing the effective value for any (key, device). - Three consumers read resolved values instead of hardcodes (§6). - A generated, versioned rubric the GPU fleet pulls and feeds to gemma (§7). - An admin surface under a new "Inference Ops" section (§8), audit-logged.

Non-goals (future modules, separate specs): - GPU fleet host health / utilization console. - Inference model registry / deployment management. - A free-form "prompt studio" for editing rubric prose by hand (the prose is generated here, not hand-authored). - Relocating the device-side firmware alarm thresholds (co2_alarm_ppm, voc_alarm_index, …) that already live in config_registry.rs + device-shadow staging. Those keep their existing mechanism; this module is the server/inference-side thresholds only. The two registries may later share a hierarchy resolver, but that is out of scope.

3. What already exists (build on, don't duplicate)

  • Global runtime config — config_manager exposes get_i64(key, default) / get_*, backed by the system_config table, already used for anomaly.sse_freshness_secs etc. (see ingest/manager.rs:1599). This is the global scope's storage.
  • Per-device typed config — config_registry::REGISTRY (KeySpec { key, category, label, kind }, Kind::U32 { min, max } | Bool) plus the device-shadow staging flow (operator PUT → config_version bump → device pulls → clamps to bounds). This is the pattern the threshold registry mirrors, and the device scope reuses its clamping discipline.

The new work is: (a) a threshold registry for inference/server keys, (b) the middle scopes (hardware-version, facility) that neither existing mechanism has, (c) a unified resolver, and (d) rubric generation + a pull endpoint.

4. Architecture overview

                     ┌──────────────────────────────────────────┐
                     │  threshold_registry.rs (static REGISTRY)  │
                     │  key · unit · kind(bounds) · default ·     │
                     │  consumers · rubric slot                   │
                     └──────────────────────────────────────────┘
                                     │ defaults
                                     ▼
 admin edits ─► anomaly_threshold_overrides (scope, scope_id, value)
                                     │
                                     ▼
              ┌────────────────────────────────────────┐
              │ resolve(key, device):                   │
              │  device → facility → hw_version →        │
              │  global → registry default              │
              └────────────────────────────────────────┘
                     │                │                │
        ┌────────────┘        ┌───────┘        ┌───────┘
        ▼                     ▼                ▼
 inference_results     ingest/manager   rubric generator ──► GET /internal/
 guard (60°C)          detector          (numbers + LLM         inference/rubric
                       (1000/10000)       prose, versioned)  ◄── GPU fleet polls

5. Data model

5.1 Threshold registry (code, static)

New crates/xylolabs-server/src/threshold_registry.rs, mirroring config_registry.rs:

pub enum ThresholdKind {
    F64 { min: f64, max: f64 },
    U32 { min: u64, max: u64 },
}

pub struct ThresholdSpec {
    pub key: &'static str,        // e.g. "temperature_max_c"
    pub category: &'static str,   // "thermal" | "air_quality" | "acoustic" | "generic"
    pub label: &'static str,      // "Max operating temperature (°C)"
    pub unit: &'static str,       // "°C", "index", "dBFS", ""
    pub kind: ThresholdKind,      // typed bounds (validation + admin UI)
    pub default: f64,             // == today's hardcoded value (see §9)
    pub rubric_slot: Option<&'static str>, // template variable name, None = not in rubric
}

pub const REGISTRY: &[ThresholdSpec] = &[ /* temperature_max_c=60, nox_raw_index_max=25000,
    noise_dbfs_max=-18, numeric_warn=1000, numeric_critical=10000, … */ ];

The registry is the whitelist. An override for an unknown key is rejected; a missing override falls through to default.

5.2 Override table (migration)

-- migrations/<YYYYMMDDHHMMSS>_create_anomaly_threshold_overrides.sql
-- (version prefix chosen at implementation time, strictly greater than the
--  latest existing migration — see the monotonicity rule note below)
CREATE TABLE anomaly_threshold_overrides (
    id           uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    threshold_key text NOT NULL,          -- must exist in REGISTRY (validated in app)
    scope        text NOT NULL,           -- 'global' | 'hw_version' | 'facility' | 'device'
    scope_id     text,                    -- NULL for global; hw_version string;
                                          -- facility_id/device_id as text
    value        double precision NOT NULL,
    updated_by   uuid,                    -- users.id (NULL for system)
    created_at   timestamptz NOT NULL DEFAULT now(),
    updated_at   timestamptz NOT NULL DEFAULT now()
);

-- One override per (key, scope, scope_id). scope_id NULL (global) needs a
-- partial unique index because NULL is distinct in a normal UNIQUE.
CREATE UNIQUE INDEX uq_threshold_override_scoped
    ON anomaly_threshold_overrides (threshold_key, scope, scope_id)
    WHERE scope_id IS NOT NULL;
CREATE UNIQUE INDEX uq_threshold_override_global
    ON anomaly_threshold_overrides (threshold_key)
    WHERE scope = 'global';

CREATE INDEX ix_threshold_override_lookup
    ON anomaly_threshold_overrides (scope, scope_id);

A row exists only where someone overrode a default — the table is small (bounded by keys × scopes actually touched).

Migration version prefix MUST be strictly monotonic YYYYMMDDHHMMSS and greater than the latest existing migration; verify with ls crates/xylolabs-db/migrations | awk -F_ '{print $1}' | sort -c before commit (per CLAUDE.md / DEPLOYMENT-GUIDE).

5.3 Repo layer

New crates/xylolabs-db/src/repo/threshold_override.rs: - list_all() → all overrides (for the resolver's per-request snapshot / cache). - list_for_scopes(device_id, facility_id, hw_version) → the ≤4 rows relevant to one device (indexed lookup). - upsert(key, scope, scope_id, value, updated_by) — validates key ∈ REGISTRY and value ∈ bounds at the app layer before writing. - delete(key, scope, scope_id) — remove an override (falls back to the next level).

6. Resolution

Pure function, no DB, fully unit-testable:

/// Precedence: device → facility → hw_version → global → registry default.
pub fn resolve(
    spec: &ThresholdSpec,
    overrides: &ResolvedOverrides, // the ≤4 rows fetched for this device
) -> f64 {
    overrides.device
        .or(overrides.facility)
        .or(overrides.hw_version)
        .or(overrides.global)
        .unwrap_or(spec.default)
        .clamp(min_of(spec.kind), max_of(spec.kind))
}

Values are clamped to the registry bounds on read as a defence-in-depth belt (an out-of-range row can never have been written through the validated upsert, but the clamp guarantees a consumer never sees a nonsense number — the same fail-safe discipline as the device-shadow clamp).

Caching. Overrides change rarely (operator action) and are read on hot paths (every ingest batch). The resolver reads from an in-memory snapshot refreshed on a short TTL and invalidated on any upsert/delete (bump a threshold_version counter, mirroring config_version). Cold start / cache miss falls back to a DB read. No per-sample DB round-trip.

7. Consumers (delete the hardcodes)

  1. Inference-result guard — routes/inference_results.rs. should_suppress_operating_limit replaces the MAX_OPERATING_TEMP_C = 60.0 const with resolve("temperature_max_c", device) for the result's device (falling back to facility/global/default when device_id is absent). The fail-open behaviour (unparseable temperature ⇒ not suppressed) is unchanged.
  2. Ingest detector — ingest/manager.rs:1646,1668,1708. val.abs() > 1000.0 → > resolve("numeric_warn", …); val.abs() > 10000.0 → > resolve("numeric_critical", …); the "threshold": 1000.0 value emitted into the report details becomes the resolved warn value. (These generic numeric thresholds may later become per-stream keys; v1 keeps the two global-ish keys to preserve behaviour.)
  3. GPU fleet rubric — §7.1.

7.1 Rubric generation + pull delivery (approved mechanism)

Structured numbers are authoritative; the LLM writes the human-readable prose around them, cached.

  • Generator (services/rubric.rs): for a facility, resolve every rubric_slot key, fill a deterministic template with the numbers, then call the existing LLM helper (services/alert_llm.rs / http_json.rs) to elaborate the surrounding guidance prose. The numbers in the output are the resolved values verbatim — the LLM is instructed to phrase, never to invent or alter, thresholds. Output is cached keyed by (facility_id, threshold_version); a version bump regenerates lazily on next fetch.
  • Endpoint: GET /api/internal/inference/rubric?facility={id} (internal scope, require_api_key_scope(ctx, "internal"), matching the existing /api/internal/inference/results prefix). Returns { rubric_text, config_version, generated_at } with an ETag: <config_version>. Honours If-None-Match → 304 Not Modified so a fleet poll with an unchanged config is a cheap no-op.
  • Fleet side (one-line change in xylolabs-gpu-ops, coordinated but out-of-repo): on boot and every N minutes, fetch with the cached ETag; on 200 swap gemma.system_prompt = rubric_text. On fetch failure keep the last-known rubric (never fall back to an empty/hardcoded prompt).

Why pull: no push infra or fleet-side receiver to secure; the fleet self-heals to the latest config on its own cadence; the endpoint is trivially testable; a missed change simply propagates on the next poll.

8. Admin surface

New "Inference Ops → Thresholds" page (admin frontend/), FacilityAdmin+, audit-logged (log_audit, action set_threshold_override): - A table of threshold key × scope, showing the effective value for the selected facility (and optionally a drilled-in device/hw-version), with the winning scope highlighted so an operator sees why a value is what it is. - Inline edit at any scope level; validation against registry bounds client- and server-side; unit shown from the registry. - "Preview rubric" — render the generated rubric for the facility (a diff against the currently-served version) before it goes live. - Reuses the shared FacilitySelect, IOS_INPUT_STYLE, dark-mode badges, and EN/KO i18n conventions from frontend/AGENTS.md. No emoji; SVG icons only.

REST (admin, JWT): - GET /api/v1/inference-ops/thresholds?facility={id} — registry + effective values + overrides per scope. - PUT /api/v1/inference-ops/thresholds/{key} — body { scope, scope_id, value }, upsert; DELETE same to clear an override. - GET /api/v1/inference-ops/rubric/preview?facility={id} — generated rubric for preview (admin-visible twin of the internal fleet endpoint).

9. Rollout & migration safety

Registry defaults equal today's live values, so merely shipping the resolver and repointing the three consumers is a behavioural no-op:

key default replaces
temperature_max_c 60.0 MAX_OPERATING_TEMP_C const
nox_raw_index_max 25000 gemma rubric literal
noise_dbfs_max -18.0 gemma rubric literal
numeric_warn 1000.0 manager.rs warn literal
numeric_critical 10000.0 manager.rs critical literal

Sequenced, each step independently shippable and reversible: 1. Registry + migration + repo + resolver (no consumer wired) — dead code, safe. 2. Repoint the two Rust consumers to resolve(...) — no-op (defaults == old). 3. Rubric generator + internal pull endpoint + admin preview — additive. 4. Admin thresholds page (read-only first, then editable). 5. Coordinate the one-line fleet fetch change (separate repo/PR).

The tactical 60°C guard from commit 00000007506 becomes the temperature_max_c default rather than a stranded constant — no regression, and the guard's suppression logic is retained (it now reads a resolved value).

10. Error handling

  • Unknown threshold_key on upsert → 400.
  • Out-of-bounds value → 400 (validated against registry kind); reads clamp as a second line of defence.
  • Resolver cache miss / DB error on a hot path → fall back to registry default (never fail an ingest flush or an inference submit because a threshold lookup failed). Log at warn.
  • Rubric LLM elaboration failure → serve the deterministic template (numbers only, no prose) rather than 500, so the fleet always gets valid, correct numbers. Log the elaboration failure.
  • Rubric endpoint: unknown/foreign facility → 404; missing scope → 400.

11. Testing

  • Unit (pure, fast — the flash-shared "copy test binary to /tmp" discipline applies): resolver precedence (device beats facility beats hw beats global beats default), bound-clamp on read, registry-default table matches the documented product spec values (a tripwire so a future edit that drifts a default from the hardware spec fails CI — mirrors session_ttl_locked).
  • Integration (tests/api_inference_ops.rs): PUT a facility override → GET thresholds shows it winning → the ingest detector and inference guard observe the new value → the rubric endpoint's config_version/ETag bumps and the body reflects the new number → If-None-Match returns 304 when unchanged.
  • Rubric fidelity: assert the generated rubric contains the resolved numbers verbatim and that an LLM-elaboration stub failure degrades to the deterministic template (numbers preserved).
  • Auth: rubric endpoint rejects non-internal keys; admin endpoints reject below FacilityAdmin and cross-facility scope_ids.

12. Open questions / future

  • Per-stream numeric thresholds (v1 keeps two global-ish numeric_* keys; per-stream keys are a later registry expansion, no schema change).
  • ~~Whether the hardware-version scope keys off the existing firmware/board version field or a new dimension.~~ Resolved (2026-07-13): the hw_version scope keys off the existing devices.hardware_version column (no new dimension). ThresholdCache::get_for_device(key, device_id, facility_id) looks up that column and applies the hw tier — but only when an hw_version override actually exists for the key (has_hw_scope fast-path), so the hot paths (inference overheating guard, ingest numeric detector) pay no extra DB read in the common case. The admin endpoint accepts hw_version overrides (SuperAdmin-only, cross-facility; scope_id is the hardware-version string).
  • A future module may unify this resolver with the device-side config_registry so firmware alarm thresholds gain the same hierarchy; explicitly deferred here.