Inference Ops — Authoritative Anomaly Thresholds Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Make the API the single editable source of truth for anomaly thresholds (temperature, NOx, noise, generic numeric), resolved through a global→hardware-version→facility→device hierarchy, and derive every consumer — the inference-result guard, the ingest detector, and the GPU fleet's LLM rubric — from it.

Architecture: A static in-code registry declares every threshold key with typed bounds and a default equal to today's live value. A small anomaly_threshold_overrides table stores only the deltas an operator sets, at one of four scopes. A pure resolver merges device→facility→hw_version→global→default. An in-memory cache (hash-versioned snapshot) serves the hot paths without per-call DB round-trips. Because defaults equal current values, wiring the resolver is a behavioral no-op until an override exists.

Tech Stack: Rust 2024, Axum 0.8, SQLx 0.8 (PostgreSQL), Tokio; React 19 + Vite + TailwindCSS 4 + TypeScript (admin frontend/).

Spec: docs/superpowers/specs/2026-07-12-inference-ops-authoritative-thresholds-design.md

Global Constraints

Git: GPG-sign every commit (git commit -S). After committing, mine the hash for 7 leading hex zeros: ~/flash-shared/gitminer-cuda/mine_commit.sh 7. Then git pull --rebase and git push immediately (one commit per task, never batch). Semantic Conventional-Commits messages with a gitmoji: <type>(<scope>): <gitmoji> <desc>.
Deploy: After a code change that alters the running server, deploy with bash scripts/deploy.sh (target api.xylolabs.com). Docs/plan-only commits do not deploy.
Language: All code, comments, docs, commit messages in English only.
Migrations: New SQL migration prefix must be a unique, strictly monotonic YYYYMMDDHHMMSS greater than 20260710150000 (the current latest). Verify: ls crates/xylolabs-db/migrations | awk -F_ '{print $1}' | sort -c (silent = OK). A failing migration crash-loops the container.
API keys: Never re-hash, drop, or hide the plaintext key column. Out of scope here — do not touch api_keys.
UI: No emoji in UI (SVG icons only). Inputs font-size: 16px min (iOS). Branding exactly "Xylolabs". EN + KO i18n parity for every new string.
Test-run quirk (flash-shared): freshly-linked test binaries under target/debug/deps can stall at exec. To run tests, build with cargo test <args> --no-run --message-format=json, find the executable path in the JSON, cp it to /tmp, and run it there. Run cargo jobs serially (concurrent cargo corrupts artifacts).
Verification gates per task: cargo check (zero errors); cargo clippy for non-trivial Rust; for frontend, npx tsc -b --noEmit + npx vite build in frontend/.

File Structure

Backend (create unless noted): - crates/xylolabs-server/src/threshold_registry.rs — static REGISTRY, ThresholdSpec/ThresholdKind, ResolvedOverrides, pure resolve(). One responsibility: what thresholds exist + how to merge them. - crates/xylolabs-db/migrations/<ts>_create_anomaly_threshold_overrides.sql — override table + indexes. - crates/xylolabs-db/src/repo/threshold_override.rs — CRUD over the override table. - crates/xylolabs-server/src/services/threshold_resolver.rs — ThresholdCache (hash-versioned snapshot, TTL, invalidate) wrapping the repo + pure resolver. - crates/xylolabs-server/src/services/rubric.rs — generate the GPU-fleet rubric (deterministic template + LLM prose, cached). - crates/xylolabs-server/src/routes/inference_ops.rs — admin threshold CRUD + rubric preview handlers. - Modify crates/xylolabs-server/src/routes/inference_results.rs — guard reads resolved temperature_max_c. - Modify crates/xylolabs-server/src/ingest/manager.rs — detector reads resolved numeric_warn/numeric_critical. - Modify crates/xylolabs-server/src/routes/inference_results.rs (rubric handler) or inference_ops.rs — internal GET /inference/rubric. - Modify crates/xylolabs-server/src/state.rs, router.rs, main.rs, routes/mod.rs, services/mod.rs, crates/xylolabs-db/src/repo/mod.rs — wiring.

Frontend: - frontend/src/api/inferenceOps.ts — API client. - frontend/src/pages/InferenceOpsThresholdsPage.tsx — the thresholds page. - Modify frontend/src/components/layout/Sidebar.tsx, router, frontend/src/i18n/index.ts.

Tests: - crates/xylolabs-server/src/threshold_registry.rs #[cfg(test)] — registry + resolver unit tests. - crates/xylolabs-server/tests/api_inference_ops.rs — repo + endpoint integration tests.

Phase 1 — Foundation (registry + migration + repo + resolver). Behavioral no-op.

Task 1: Threshold registry + pure resolver

Files: - Create: crates/xylolabs-server/src/threshold_registry.rs - Modify: crates/xylolabs-server/src/main.rs (add mod threshold_registry; next to mod config_registry;) - Test: same file #[cfg(test)]

Interfaces: - Produces: ThresholdKind, ThresholdSpec, pub const REGISTRY: &[ThresholdSpec], pub fn lookup(key: &str) -> Option<&'static ThresholdSpec>, struct ResolvedOverrides, pub fn resolve(spec: &ThresholdSpec, o: &ResolvedOverrides) -> f64.

[ ] Step 1: Write the failing test — append to a new file crates/xylolabs-server/src/threshold_registry.rs, at the bottom:

#[cfg(test)]
mod tests {
    use super::*;

    // Tripwire: registry defaults MUST equal the documented product-spec values.
    // A future edit that drifts a default from the hardware spec fails here.
    #[test]
    fn registry_defaults_match_product_spec() {
        let expect = [
            ("temperature_max_c", 60.0),
            ("nox_raw_index_max", 25000.0),
            ("noise_dbfs_max", -18.0),
            ("numeric_warn", 1000.0),
            ("numeric_critical", 10000.0),
        ];
        for (key, want) in expect {
            let spec = lookup(key).unwrap_or_else(|| panic!("missing registry key {key}"));
            assert_eq!(spec.default, want, "default drift for {key}");
            assert!(spec.kind.validate(spec.default), "default out of bounds for {key}");
        }
    }

    #[test]
    fn resolve_precedence_device_beats_all() {
        let spec = lookup("temperature_max_c").unwrap();
        let o = ResolvedOverrides { device: Some(40.0), facility: Some(50.0), hw_version: Some(55.0), global: Some(58.0) };
        assert_eq!(resolve(spec, &o), 40.0);
    }

    #[test]
    fn resolve_falls_through_to_default() {
        let spec = lookup("temperature_max_c").unwrap();
        assert_eq!(resolve(spec, &ResolvedOverrides::default()), 60.0);
    }

    #[test]
    fn resolve_clamps_out_of_range_override() {
        let spec = lookup("noise_dbfs_max").unwrap(); // F64 { min: -120, max: 0 }
        let o = ResolvedOverrides { global: Some(999.0), ..Default::default() };
        assert_eq!(resolve(spec, &o), 0.0); // clamped to max
    }
}

[ ] Step 2: Run the test, verify it fails to compile (types not defined yet).

Build: cargo test -p xylolabs-server --lib threshold_registry --no-run 2>&1 | tail -5 Expected: compile error cannot find type ThresholdSpec (or module threshold_registry not found).

[ ] Step 3: Write the implementation — put this ABOVE the #[cfg(test)] block in the same file:

//! Server-side anomaly-threshold registry + hierarchy resolver.
//!
//! Mirrors `config_registry.rs` (device-side keys) but for the
//! server/inference-side thresholds that were previously hardcoded in three
//! uncoordinated places (the gemma rubric, the inference-result guard const,
//! and the ingest detector literals). Defaults here EQUAL those live values, so
//! resolving is a behavioral no-op until an operator sets an override. See
//! `docs/superpowers/specs/2026-07-12-inference-ops-authoritative-thresholds-design.md`.

#[derive(Clone, Copy, Debug)]
pub enum ThresholdKind {
    F64 { min: f64, max: f64 },
    U32 { min: u64, max: u64 },
}

impl ThresholdKind {
    pub fn min(self) -> f64 {
        match self {
            ThresholdKind::F64 { min, .. } => min,
            ThresholdKind::U32 { min, .. } => min as f64,
        }
    }
    pub fn max(self) -> f64 {
        match self {
            ThresholdKind::F64 { max, .. } => max,
            ThresholdKind::U32 { max, .. } => max as f64,
        }
    }
    pub fn validate(self, v: f64) -> bool {
        v.is_finite() && v >= self.min() && v <= self.max()
    }
}

pub struct ThresholdSpec {
    pub key: &'static str,
    pub category: &'static str,
    pub label: &'static str,
    pub unit: &'static str,
    pub kind: ThresholdKind,
    pub default: f64,
    /// Template variable name in the generated rubric; None = not exposed to the rubric.
    pub rubric_slot: Option<&'static str>,
}

/// The whitelist of server-side anomaly thresholds. Defaults equal today's live values.
pub const REGISTRY: &[ThresholdSpec] = &[
    ThresholdSpec {
        key: "temperature_max_c",
        category: "thermal",
        label: "Max operating temperature",
        unit: "°C",
        kind: ThresholdKind::F64 { min: 0.0, max: 150.0 },
        default: 60.0,
        rubric_slot: Some("temperature_max_c"),
    },
    ThresholdSpec {
        key: "nox_raw_index_max",
        category: "air_quality",
        label: "Max NOx raw index",
        unit: "index",
        kind: ThresholdKind::U32 { min: 0, max: 60_000 },
        default: 25000.0,
        rubric_slot: Some("nox_raw_index_max"),
    },
    ThresholdSpec {
        key: "noise_dbfs_max",
        category: "acoustic",
        label: "Max noise level",
        unit: "dBFS",
        kind: ThresholdKind::F64 { min: -120.0, max: 0.0 },
        default: -18.0,
        rubric_slot: Some("noise_dbfs_max"),
    },
    ThresholdSpec {
        key: "numeric_warn",
        category: "generic",
        label: "Generic numeric warn threshold",
        unit: "",
        kind: ThresholdKind::F64 { min: 0.0, max: 1_000_000.0 },
        default: 1000.0,
        rubric_slot: None,
    },
    ThresholdSpec {
        key: "numeric_critical",
        category: "generic",
        label: "Generic numeric critical threshold",
        unit: "",
        kind: ThresholdKind::F64 { min: 0.0, max: 10_000_000.0 },
        default: 10000.0,
        rubric_slot: None,
    },
];

pub fn lookup(key: &str) -> Option<&'static ThresholdSpec> {
    REGISTRY.iter().find(|s| s.key == key)
}

/// The (up to four) override values that apply to one device, one per scope.
#[derive(Default, Clone, Copy, Debug)]
pub struct ResolvedOverrides {
    pub device: Option<f64>,
    pub facility: Option<f64>,
    pub hw_version: Option<f64>,
    pub global: Option<f64>,
}

/// Precedence: device → facility → hw_version → global → registry default.
/// The result is clamped to the registry bounds (defence in depth).
pub fn resolve(spec: &ThresholdSpec, o: &ResolvedOverrides) -> f64 {
    let raw = o
        .device
        .or(o.facility)
        .or(o.hw_version)
        .or(o.global)
        .unwrap_or(spec.default);
    raw.clamp(spec.kind.min(), spec.kind.max())
}

[ ] Step 4: Run the tests (copy binary to /tmp per the flash-shared quirk):

cd /Users/hletrd/flash-shared/xylolabs-api
cargo test -p xylolabs-server --lib threshold_registry --no-run --message-format=json 2>/dev/null \
  | python3 -c "import sys,json
for l in sys.stdin:
  try: o=json.loads(l)
  except: continue
  if o.get('executable') and 'xylolabs_server' in o['executable']: print(o['executable'])" | head -1 | xargs -I{} cp {} /tmp/xls_reg_test
/tmp/xls_reg_test threshold_registry

Expected: test result: ok. 4 passed.

[ ] Step 5: Commit

git add crates/xylolabs-server/src/threshold_registry.rs crates/xylolabs-server/src/main.rs
git commit -S -m "feat(inference-ops): ✨ add server-side threshold registry + resolver"
~/flash-shared/gitminer-cuda/mine_commit.sh 7 && git pull --rebase && git push

Task 2: Override table migration

Files: - Create: crates/xylolabs-db/migrations/<ts>_create_anomaly_threshold_overrides.sql (choose <ts> = a YYYYMMDDHHMMSS greater than 20260710150000, e.g. today's UTC time).

Interfaces: - Produces: table anomaly_threshold_overrides(id, threshold_key, scope, scope_id, value, updated_by, created_at, updated_at).

[ ] Step 1: Write the migration

-- Server-side anomaly threshold overrides. A row exists only where an operator
-- overrode a registry default, at one of four scopes.
CREATE TABLE anomaly_threshold_overrides (
    id            uuid PRIMARY KEY DEFAULT gen_random_uuid(),
    threshold_key text NOT NULL,
    scope         text NOT NULL CHECK (scope IN ('global','hw_version','facility','device')),
    scope_id      text,
    value         double precision NOT NULL,
    updated_by    uuid,
    created_at    timestamptz NOT NULL DEFAULT now(),
    updated_at    timestamptz NOT NULL DEFAULT now()
);

-- One override per (key, scope, scope_id). NULL scope_id (global) needs a
-- partial index because NULL is distinct under a normal UNIQUE.
CREATE UNIQUE INDEX uq_threshold_override_scoped
    ON anomaly_threshold_overrides (threshold_key, scope, scope_id)
    WHERE scope_id IS NOT NULL;
CREATE UNIQUE INDEX uq_threshold_override_global
    ON anomaly_threshold_overrides (threshold_key)
    WHERE scope = 'global';

CREATE INDEX ix_threshold_override_lookup
    ON anomaly_threshold_overrides (scope, scope_id);

[ ] Step 2: Verify monotonic ordering

Run: ls crates/xylolabs-db/migrations | awk -F_ '{print $1}' | sort -c Expected: no output (silent = monotonic OK).

[ ] Step 3: Verify it applies (compile-time SQLx check is offline; apply against a scratch DB if available, else rely on CI/startup). At minimum:

Run: cargo check -p xylolabs-db Expected: builds with zero errors.

[ ] Step 4: Commit

git add crates/xylolabs-db/migrations/
git commit -S -m "feat(db): 🗃️ add anomaly_threshold_overrides table"
~/flash-shared/gitminer-cuda/mine_commit.sh 7 && git pull --rebase && git push

Task 3: Override repo layer

Files: - Create: crates/xylolabs-db/src/repo/threshold_override.rs - Modify: crates/xylolabs-db/src/repo/mod.rs (add pub mod threshold_override;) - Test: covered by integration test in Task 9/11 (needs a DB pool); no unit test here.

Interfaces: - Produces: - struct ThresholdOverrideRow { id: Uuid, threshold_key: String, scope: String, scope_id: Option<String>, value: f64, updated_by: Option<Uuid>, created_at: DateTime<Utc>, updated_at: DateTime<Utc> } - async fn list_all(pool: &PgPool) -> Result<Vec<ThresholdOverrideRow>, sqlx::Error> - async fn upsert(pool: &PgPool, key: &str, scope: &str, scope_id: Option<&str>, value: f64, updated_by: Option<Uuid>) -> Result<ThresholdOverrideRow, sqlx::Error> - async fn delete(pool: &PgPool, key: &str, scope: &str, scope_id: Option<&str>) -> Result<u64, sqlx::Error>

[ ] Step 1: Write the implementation (crates/xylolabs-db/src/repo/threshold_override.rs):

use sqlx::PgPool;
use uuid::Uuid;

#[derive(Debug, Clone, sqlx::FromRow, serde::Serialize)]
pub struct ThresholdOverrideRow {
    pub id: Uuid,
    pub threshold_key: String,
    pub scope: String,
    pub scope_id: Option<String>,
    pub value: f64,
    pub updated_by: Option<Uuid>,
    pub created_at: chrono::DateTime<chrono::Utc>,
    pub updated_at: chrono::DateTime<chrono::Utc>,
}

/// All overrides. The resolver cache loads the full set (bounded by keys × scopes actually touched).
pub async fn list_all(pool: &PgPool) -> Result<Vec<ThresholdOverrideRow>, sqlx::Error> {
    sqlx::query_as::<_, ThresholdOverrideRow>(
        "SELECT id, threshold_key, scope, scope_id, value, updated_by, created_at, updated_at
         FROM anomaly_threshold_overrides
         ORDER BY threshold_key, scope, scope_id",
    )
    .fetch_all(pool)
    .await
}

/// Upsert an override. NULL-safe on scope_id (global uses NULL). Last write wins.
pub async fn upsert(
    pool: &PgPool,
    key: &str,
    scope: &str,
    scope_id: Option<&str>,
    value: f64,
    updated_by: Option<Uuid>,
) -> Result<ThresholdOverrideRow, sqlx::Error> {
    sqlx::query_as::<_, ThresholdOverrideRow>(
        "WITH deleted AS (
             DELETE FROM anomaly_threshold_overrides
             WHERE threshold_key = $1 AND scope = $2 AND scope_id IS NOT DISTINCT FROM $3
         )
         INSERT INTO anomaly_threshold_overrides (threshold_key, scope, scope_id, value, updated_by)
         VALUES ($1, $2, $3, $4, $5)
         RETURNING id, threshold_key, scope, scope_id, value, updated_by, created_at, updated_at",
    )
    .bind(key)
    .bind(scope)
    .bind(scope_id)
    .bind(value)
    .bind(updated_by)
    .fetch_one(pool)
    .await
}

/// Remove an override (falls back to the next scope level). Returns rows affected.
pub async fn delete(
    pool: &PgPool,
    key: &str,
    scope: &str,
    scope_id: Option<&str>,
) -> Result<u64, sqlx::Error> {
    let r = sqlx::query(
        "DELETE FROM anomaly_threshold_overrides
         WHERE threshold_key = $1 AND scope = $2 AND scope_id IS NOT DISTINCT FROM $3",
    )
    .bind(key)
    .bind(scope)
    .bind(scope_id)
    .execute(pool)
    .await?;
    Ok(r.rows_affected())
}

[ ] Step 2: Run the build

Run: cargo check -p xylolabs-db Expected: zero errors.

[ ] Step 3: Commit

git add crates/xylolabs-db/src/repo/threshold_override.rs crates/xylolabs-db/src/repo/mod.rs
git commit -S -m "feat(db): ✨ add threshold_override repo (list/upsert/delete)"
~/flash-shared/gitminer-cuda/mine_commit.sh 7 && git pull --rebase && git push

Task 4: Resolver cache service

Files: - Create: crates/xylolabs-server/src/services/threshold_resolver.rs - Modify: crates/xylolabs-server/src/services/mod.rs (add pub mod threshold_resolver;) - Test: same file #[cfg(test)] for the pure snapshot-version + override-selection helpers.

Interfaces: - Consumes: repo::threshold_override::{list_all, ThresholdOverrideRow}, threshold_registry::{lookup, resolve, ResolvedOverrides}. - Produces: - struct ThresholdCache with pub fn new(db: PgPool) -> Self - pub async fn get(&self, key: &str, device_id: Option<Uuid>, facility_id: Uuid, hw_version: Option<&str>) -> f64 - pub async fn version(&self) -> String (ETag) - pub async fn invalidate(&self) - pub async fn has_hw_scope(&self, key: &str) -> bool (lets hot paths skip a device hw lookup when no hw override exists)

[ ] Step 1: Write the failing test (append to the new file):

#[cfg(test)]
mod tests {
    use super::*;

    fn row(key: &str, scope: &str, scope_id: Option<&str>, value: f64) -> ThresholdOverrideRow {
        ThresholdOverrideRow {
            id: Uuid::nil(),
            threshold_key: key.into(),
            scope: scope.into(),
            scope_id: scope_id.map(|s| s.into()),
            value,
            updated_by: None,
            created_at: chrono::DateTime::from_timestamp(0, 0).unwrap(),
            updated_at: chrono::DateTime::from_timestamp(0, 0).unwrap(),
        }
    }

    #[test]
    fn select_overrides_picks_the_right_scopes() {
        let fac = Uuid::from_u128(1);
        let dev = Uuid::from_u128(2);
        let rows = vec![
            row("temperature_max_c", "global", None, 58.0),
            row("temperature_max_c", "facility", Some(&fac.to_string()), 50.0),
            row("temperature_max_c", "device", Some(&dev.to_string()), 40.0),
            row("temperature_max_c", "hw_version", Some("board1-v1"), 55.0),
            row("noise_dbfs_max", "global", None, -20.0),
        ];
        let o = select_overrides(&rows, "temperature_max_c", Some(dev), fac, Some("board1-v1"));
        assert_eq!(o.device, Some(40.0));
        assert_eq!(o.facility, Some(50.0));
        assert_eq!(o.hw_version, Some(55.0));
        assert_eq!(o.global, Some(58.0));
    }

    #[test]
    fn snapshot_version_changes_iff_content_changes() {
        let a = vec![row("temperature_max_c", "global", None, 58.0)];
        let b = vec![row("temperature_max_c", "global", None, 59.0)];
        assert_ne!(snapshot_version(&a), snapshot_version(&b));
        assert_eq!(snapshot_version(&a), snapshot_version(&a.clone()));
    }
}

[ ] Step 2: Run it, verify compile failure (select_overrides/snapshot_version undefined).

Run: cargo test -p xylolabs-server --lib threshold_resolver --no-run 2>&1 | tail -5 Expected: cannot find function select_overrides.

[ ] Step 3: Write the implementation (above the test block):

//! In-memory cache + hierarchy resolver for anomaly thresholds. Refreshes on a
//! short TTL and on explicit invalidate() (after an admin upsert/delete). The
//! ETag/config_version is a content hash of the snapshot, so it is stable across
//! restarts and changes iff the override set changes.

use std::time::{Duration, Instant};
use tokio::sync::RwLock;
use uuid::Uuid;
use xylolabs_db::repo::threshold_override::{list_all, ThresholdOverrideRow};

use crate::threshold_registry::{lookup, resolve, ResolvedOverrides};

const CACHE_TTL: Duration = Duration::from_secs(30);

struct Snapshot {
    rows: Vec<ThresholdOverrideRow>,
    loaded_at: Instant,
    version: String,
}

pub struct ThresholdCache {
    db: sqlx::PgPool,
    inner: RwLock<Option<Snapshot>>,
}

/// Build the four per-scope override values for one (key, device) from a row set. Pure.
fn select_overrides(
    rows: &[ThresholdOverrideRow],
    key: &str,
    device_id: Option<Uuid>,
    facility_id: Uuid,
    hw_version: Option<&str>,
) -> ResolvedOverrides {
    let dev = device_id.map(|d| d.to_string());
    let fac = facility_id.to_string();
    let mut o = ResolvedOverrides::default();
    for r in rows.iter().filter(|r| r.threshold_key == key) {
        match r.scope.as_str() {
            "global" => o.global = Some(r.value),
            "facility" if r.scope_id.as_deref() == Some(fac.as_str()) => o.facility = Some(r.value),
            "device" if dev.as_deref().is_some() && r.scope_id.as_deref() == dev.as_deref() => {
                o.device = Some(r.value)
            }
            "hw_version" if hw_version.is_some() && r.scope_id.as_deref() == hw_version => {
                o.hw_version = Some(r.value)
            }
            _ => {}
        }
    }
    o
}

/// Deterministic content hash of the snapshot (used as the rubric ETag). Pure.
fn snapshot_version(rows: &[ThresholdOverrideRow]) -> String {
    use std::hash::{Hash, Hasher};
    let mut keyed: Vec<(String, String, Option<String>, u64)> = rows
        .iter()
        .map(|r| (r.threshold_key.clone(), r.scope.clone(), r.scope_id.clone(), r.value.to_bits()))
        .collect();
    keyed.sort();
    let mut h = std::collections::hash_map::DefaultHasher::new();
    keyed.hash(&mut h);
    format!("{:016x}", h.finish())
}

impl ThresholdCache {
    pub fn new(db: sqlx::PgPool) -> Self {
        Self { db, inner: RwLock::new(None) }
    }

    async fn ensure_fresh(&self) {
        {
            let g = self.inner.read().await;
            if let Some(s) = g.as_ref() {
                if s.loaded_at.elapsed() < CACHE_TTL {
                    return;
                }
            }
        }
        let rows = list_all(&self.db).await.unwrap_or_default();
        let version = snapshot_version(&rows);
        let mut g = self.inner.write().await;
        *g = Some(Snapshot { rows, loaded_at: Instant::now(), version });
    }

    pub async fn invalidate(&self) {
        let mut g = self.inner.write().await;
        *g = None;
    }

    pub async fn version(&self) -> String {
        self.ensure_fresh().await;
        self.inner
            .read()
            .await
            .as_ref()
            .map(|s| s.version.clone())
            .unwrap_or_else(|| "0".into())
    }

    pub async fn has_hw_scope(&self, key: &str) -> bool {
        self.ensure_fresh().await;
        let g = self.inner.read().await;
        g.as_ref()
            .map(|s| s.rows.iter().any(|r| r.threshold_key == key && r.scope == "hw_version"))
            .unwrap_or(false)
    }

    /// Resolve a threshold for a device. Falls back to the registry default on any miss.
    pub async fn get(
        &self,
        key: &str,
        device_id: Option<Uuid>,
        facility_id: Uuid,
        hw_version: Option<&str>,
    ) -> f64 {
        let Some(spec) = lookup(key) else {
            tracing::warn!(key, "resolve() called for unknown threshold key");
            return f64::NAN;
        };
        self.ensure_fresh().await;
        let g = self.inner.read().await;
        let rows: &[ThresholdOverrideRow] = g.as_ref().map(|s| s.rows.as_slice()).unwrap_or(&[]);
        let o = select_overrides(rows, key, device_id, facility_id, hw_version);
        resolve(spec, &o)
    }
}

[ ] Step 4: Run the tests (copy to /tmp as in Task 1 Step 4, matching threshold_resolver). Expected: 2 passed.
[ ] Step 5: Commit

git add crates/xylolabs-server/src/services/threshold_resolver.rs crates/xylolabs-server/src/services/mod.rs
git commit -S -m "feat(inference-ops): ✨ add threshold resolver cache (hash-versioned)"
~/flash-shared/gitminer-cuda/mine_commit.sh 7 && git pull --rebase && git push

Task 5: Wire `ThresholdCache` into `AppState`

Files: - Modify: crates/xylolabs-server/src/state.rs (add field), crates/xylolabs-server/src/main.rs or wherever AppState is constructed (initialize it).

Interfaces: - Produces: state.threshold_cache: Arc<ThresholdCache> available to all handlers.

[ ] Step 1: Add the field — in state.rs, add to the AppState struct (match the surrounding Arc<...> field style):

    pub threshold_cache: std::sync::Arc<crate::services::threshold_resolver::ThresholdCache>,

[ ] Step 2: Initialize it — where AppState { … } is built (search: grep -rn "AppState {" crates/xylolabs-server/src/main.rs crates/xylolabs-server/src/state.rs), add:

        threshold_cache: std::sync::Arc::new(
            crate::services::threshold_resolver::ThresholdCache::new(db.clone()),
        ),

(Use the same pool variable the other fields use — likely db or pool.)

[ ] Step 3: Build

Run: cargo check -p xylolabs-server Expected: zero errors.

[ ] Step 4: Commit

git add crates/xylolabs-server/src/state.rs crates/xylolabs-server/src/main.rs
git commit -S -m "feat(inference-ops): 🔌 expose threshold cache on AppState"
~/flash-shared/gitminer-cuda/mine_commit.sh 7 && git pull --rebase && git push

Phase 2 — Repoint consumers (behavioral no-op; deploy after Task 7).

Task 6: Inference-result guard reads resolved `temperature_max_c`

Files: - Modify: crates/xylolabs-server/src/routes/inference_results.rs (the should_suppress_operating_limit call sites in submit_result and submit_result_batch, and the fn). - Test: extend the module #[cfg(test)] for the parse path (unchanged); behavioral change covered by the integration test in Task 11.

Interfaces: - Consumes: state.threshold_cache.get("temperature_max_c", device_id, facility_id, hw_version).

Currently should_suppress_operating_limit(body) compares the parsed max °C against the MAX_OPERATING_TEMP_C = 60.0 const. Change it to take the resolved limit as a parameter, and resolve at the call sites.

[ ] Step 1: Change the predicate to accept the limit — replace the MAX_OPERATING_TEMP_C usage inside should_suppress_operating_limit so it takes a limit_c: f64 argument:

fn should_suppress_operating_limit(body: &SubmitInferenceResultRequest, limit_c: f64) -> bool {
    if !is_temperature_overheating_event(&body.event_type) {
        return false;
    }
    let mut text = body.title.clone();
    if let Some(ref details) = body.details
        && let Ok(s) = serde_json::to_string(details)
    {
        text.push(' ');
        text.push_str(&s);
    }
    match extract_celsius_values(&text).into_iter().reduce(f64::max) {
        Some(max_c) => max_c < limit_c,
        None => false,
    }
}

Keep MAX_OPERATING_TEMP_C as the documented fallback default (still referenced by the registry). Update the existing unit tests that call should_suppress_operating_limit(&req) to pass MAX_OPERATING_TEMP_C explicitly, e.g. should_suppress_operating_limit(&req, MAX_OPERATING_TEMP_C).

[ ] Step 2: Resolve at the single call site (submit_result) — replace the if should_suppress_operating_limit(&body) { block guard with:

    // Resolve the operating-temperature limit from the authoritative registry
    // (falls back to the 60°C default when no override exists). hw_version scope
    // is only consulted when an hw override actually exists (avoids a device
    // fetch on the common path).
    let limit_c = state
        .threshold_cache
        .get("temperature_max_c", body.device_id, facility_id, None)
        .await;
    if should_suppress_operating_limit(&body, limit_c) {

[ ] Step 3: Resolve at the batch call site (submit_result_batch) — inside the for result in &body.results loop, before the suppression check:

        let limit_c = state
            .threshold_cache
            .get("temperature_max_c", result.device_id, facility_id, None)
            .await;
        if should_suppress_operating_limit(result, limit_c) {

[ ] Step 4: Build + run the existing module tests

Run: cargo check -p xylolabs-server then run the inference_results::tests binary from /tmp (as in Task 1 Step 4). Expected: zero errors; 5 passed.

[ ] Step 5: Commit

git add crates/xylolabs-server/src/routes/inference_results.rs
git commit -S -m "refactor(inference): ♻️ resolve overheating limit from threshold registry"
~/flash-shared/gitminer-cuda/mine_commit.sh 7 && git pull --rebase && git push

Task 7: Ingest detector reads resolved `numeric_warn`/`numeric_critical`

Files: - Modify: crates/xylolabs-server/src/ingest/manager.rs:1646,1668,1708 (the detector spawn).

The detector runs inside a tokio::spawn with a captured db, per-session. The IngestManager already holds self.config_manager; add a captured resolved warn/critical the same way sse_freshness_secs is fetched at :1599.

[ ] Step 1: Resolve before the spawn — near let sse_freshness_secs = self.config_manager.get_i64("anomaly.sse_freshness_secs", 900).await; (line ~1599), add (the IngestManager must hold an Arc<ThresholdCache>; if it does not yet, add it as a field mirroring config_manager and pass it from AppState at construction):

        let numeric_warn = self
            .threshold_cache
            .get("numeric_warn", Some(device_id), facility_id, None)
            .await;
        let numeric_critical = self
            .threshold_cache
            .get("numeric_critical", Some(device_id), facility_id, None)
            .await;

(Use the device_id/facility_id already in scope in this method; confirm the names via grep -n "facility_id\|device_id" crates/xylolabs-server/src/ingest/manager.rs around 1560–1610.)

[ ] Step 2: Move the resolved values into the spawn — add let numeric_warn = numeric_warn; let numeric_critical = numeric_critical; capture (they are Copy f64, captured by the move closure automatically), then replace the literals:
:1646 if val.abs() > 1000.0 { → if val.abs() > numeric_warn {
:1668 let severity = if val.abs() > 10000.0 { → let severity = if val.abs() > numeric_critical {
:1708 "threshold": 1000.0, → "threshold": numeric_warn,
[ ] Step 3: Add the threshold_cache field to IngestManager if missing — mirror the config_manager field declaration, constructor param, and the AppState/main.rs wiring that builds the manager. Search: grep -n "config_manager" crates/xylolabs-server/src/ingest/manager.rs | head and follow the same three spots (field, new(...) arg, call site).
[ ] Step 4: Build + clippy

Run: cargo check -p xylolabs-server && cargo clippy -p xylolabs-server 2>&1 | grep -c "warning: .*manager.rs" Expected: zero errors; 0 new warnings on manager.rs.

[ ] Step 5: Commit + deploy (Phase 2 is a no-op; deploy now to prove it in prod)

git add crates/xylolabs-server/src/ingest/manager.rs
git commit -S -m "refactor(ingest): ♻️ resolve numeric anomaly thresholds from registry"
~/flash-shared/gitminer-cuda/mine_commit.sh 7 && git pull --rebase && git push
bash scripts/deploy.sh

Expected: deploy healthy; behavior unchanged (defaults equal the old literals — verify no new/changed anomaly reports appear for steady-state devices).

Phase 3 — Rubric generation + pull delivery.

Task 8: Rubric generator service

Files: - Create: crates/xylolabs-server/src/services/rubric.rs - Modify: crates/xylolabs-server/src/services/mod.rs (pub mod rubric;) - Test: same-file #[cfg(test)] for the deterministic template (numbers verbatim; LLM-off path).

Interfaces: - Consumes: state.threshold_cache, the existing LLM helper in services/alert_llm.rs / services/http_json.rs. - Produces: - pub fn render_template(values: &[(&str, f64)]) -> String (deterministic, pure) - pub async fn generate_rubric(state: &AppState, facility_id: Uuid) -> (String, String) returning (rubric_text, config_version).

[ ] Step 1: Write the failing test

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn template_contains_resolved_numbers_verbatim() {
        let out = render_template(&[
            ("temperature_max_c", 60.0),
            ("nox_raw_index_max", 25000.0),
            ("noise_dbfs_max", -18.0),
        ]);
        assert!(out.contains("60"), "temp missing: {out}");
        assert!(out.contains("25000"), "nox missing: {out}");
        assert!(out.contains("-18"), "noise missing: {out}");
    }
}

[ ] Step 2: Verify failure (render_template undefined). Run: cargo test -p xylolabs-server --lib rubric --no-run 2>&1 | tail -3.
[ ] Step 3: Implement (above the test):

//! Generates the GPU-fleet anomaly-detection rubric from the authoritative
//! thresholds. The NUMBERS are the resolved values verbatim; an optional LLM
//! pass elaborates the surrounding prose only. On LLM failure we serve the
//! deterministic template (numbers preserved) rather than error.

use uuid::Uuid;
use crate::state::AppState;
use crate::threshold_registry::REGISTRY;

/// Deterministic, dependency-free rubric body. Pure — same input, same bytes.
pub fn render_template(values: &[(&str, f64)]) -> String {
    let mut s = String::from(
        "You are an industrial equipment anomaly detector. Apply these operating limits strictly.\n\n",
    );
    for (key, v) in values {
        // Render integers without a trailing .0 for readability.
        if v.fract() == 0.0 {
            s.push_str(&format!("- {key}: {}\n", *v as i64));
        } else {
            s.push_str(&format!("- {key}: {v}\n"));
        }
    }
    s.push_str(
        "\nFlag a condition ONLY when a value exceeds its limit above. Values within limits are normal.\n",
    );
    s
}

/// Resolve every rubric_slot key for a facility, render the template, then (best
/// effort) elaborate prose via the LLM. Returns (rubric_text, config_version).
pub async fn generate_rubric(state: &AppState, facility_id: Uuid) -> (String, String) {
    let mut values: Vec<(&'static str, f64)> = Vec::new();
    for spec in REGISTRY.iter().filter(|s| s.rubric_slot.is_some()) {
        let v = state
            .threshold_cache
            .get(spec.key, None, facility_id, None)
            .await;
        values.push((spec.rubric_slot.unwrap(), v));
    }
    let version = state.threshold_cache.version().await;
    let base = render_template(&values);

    // Best-effort LLM elaboration around the authoritative numbers. On any
    // failure, serve the deterministic template unchanged.
    let text = match elaborate_prose(state, &base).await {
        Ok(prose) => prose,
        Err(e) => {
            tracing::warn!(error = %e, "rubric LLM elaboration failed; serving deterministic template");
            base
        }
    };
    (text, version)
}

async fn elaborate_prose(_state: &AppState, base: &str) -> Result<String, String> {
    // v1: reuse services::alert_llm / http_json to ask the model to rewrite the
    // guidance prose WITHOUT changing any number. Wire the actual call here,
    // mirroring alert_llm.rs's Gemini request. Until wired, return the template.
    Ok(base.to_string())
}

Note: elaborate_prose is intentionally a pass-through stub in v1 so numbers are guaranteed correct; wiring the real LLM call is a follow-up within this task if desired — it must never alter a number (assert numbers survive with a post-check before returning prose).

[ ] Step 4: Run the test (from /tmp, rubric). Expected: 1 passed.
[ ] Step 5: Commit

git add crates/xylolabs-server/src/services/rubric.rs crates/xylolabs-server/src/services/mod.rs
git commit -S -m "feat(inference-ops): ✨ generate GPU-fleet rubric from thresholds"
~/flash-shared/gitminer-cuda/mine_commit.sh 7 && git pull --rebase && git push

Task 9: Internal `GET /inference/rubric` endpoint (pull + ETag/304)

Files: - Modify: crates/xylolabs-server/src/routes/inference_results.rs (add get_rubric handler) OR create the handler in routes/inference_ops.rs; wire in router.rs inside internal_routes next to /inference/results (line ~444). - Test: crates/xylolabs-server/tests/api_inference_ops.rs.

Interfaces: - Consumes: rubric::generate_rubric, require_api_key_scope(ctx, "internal"), api_ctx.facility_id. - Produces: route GET /api/internal/inference/rubric → 200 { rubric_text, config_version, generated_at } + ETag, or 304.

[ ] Step 1: Write the failing integration test (tests/api_inference_ops.rs) — follow the harness in tests/common/ (see an existing test like tests/api_inference_job_*.rs for the app + internal-key setup):

mod common;

#[tokio::test]
async fn rubric_endpoint_returns_text_and_etag_then_304() {
    let app = common::spawn_app().await;
    let key = common::internal_api_key(&app).await; // internal-scope key for app.facility

    // First fetch: 200 with body + ETag.
    let res = app.client
        .get(format!("{}/api/internal/inference/rubric?facility={}", app.addr, app.facility_id))
        .header("X-Api-Key", &key)
        .send().await.unwrap();
    assert_eq!(res.status(), 200);
    let etag = res.headers().get("etag").unwrap().to_str().unwrap().to_string();
    let body: serde_json::Value = res.json().await.unwrap();
    assert!(body["rubric_text"].as_str().unwrap().contains("60"));

    // Second fetch with If-None-Match: 304.
    let res2 = app.client
        .get(format!("{}/api/internal/inference/rubric?facility={}", app.addr, app.facility_id))
        .header("X-Api-Key", &key)
        .header("If-None-Match", &etag)
        .send().await.unwrap();
    assert_eq!(res2.status(), 304);
}

(If common lacks internal_api_key/facility_id/spawn_app helpers, add them mirroring the existing inference-job test's setup.)

[ ] Step 2: Verify it fails (route missing → 404). Build the test bin --no-run, run from /tmp.
[ ] Step 3: Implement the handler (in routes/inference_results.rs):

#[derive(serde::Deserialize)]
pub struct RubricQuery {
    pub facility: Option<Uuid>,
}

pub async fn get_rubric(
    State(state): State<AppState>,
    Extension(api_ctx): Extension<ApiKeyContext>,
    headers: HeaderMap,
    Query(q): Query<RubricQuery>,
) -> Result<axum::response::Response, AppError> {
    require_api_key_scope(&api_ctx, "internal")?;
    let facility_id = q.facility.unwrap_or(api_ctx.facility_id);
    if facility_id != api_ctx.facility_id {
        return Err(AppError::NotFound("facility not found".into()));
    }

    let version = state.threshold_cache.version().await;
    let etag = format!("\"{version}\"");
    if let Some(inm) = headers.get(axum::http::header::IF_NONE_MATCH)
        && inm.to_str().ok() == Some(etag.as_str())
    {
        return Ok((axum::http::StatusCode::NOT_MODIFIED, [("etag", etag)]).into_response());
    }

    let (rubric_text, config_version) =
        crate::services::rubric::generate_rubric(&state, facility_id).await;
    let body = serde_json::json!({
        "rubric_text": rubric_text,
        "config_version": config_version,
        "generated_at": chrono::Utc::now().to_rfc3339(),
    });
    Ok((axum::http::StatusCode::OK, [("etag", etag)], Json(body)).into_response())
}

[ ] Step 4: Wire the route — in router.rs, inside internal_routes after the /inference/results/batch route (line ~449):

        .route("/inference/rubric", get(inference_results::get_rubric))

Ensure get is imported (it is — used elsewhere in the block).

[ ] Step 5: Run the integration test (from /tmp). Expected: 1 passed.
[ ] Step 6: Commit

git add crates/xylolabs-server/src/routes/inference_results.rs crates/xylolabs-server/src/router.rs crates/xylolabs-server/tests/api_inference_ops.rs crates/xylolabs-server/tests/common/
git commit -S -m "feat(inference-ops): ✨ internal GET /inference/rubric with ETag/304"
~/flash-shared/gitminer-cuda/mine_commit.sh 7 && git pull --rebase && git push

Phase 4 — Admin threshold CRUD + UI.

Task 10: Admin threshold REST (list/upsert/delete) + rubric preview

Files: - Create: crates/xylolabs-server/src/routes/inference_ops.rs - Modify: crates/xylolabs-server/src/routes/mod.rs, router.rs (mount under /api/v1, JWT + FacilityAdmin), tests/api_inference_ops.rs.

Interfaces: - Consumes: repo::threshold_override::{list_all, upsert, delete}, threshold_registry::{REGISTRY, lookup}, state.threshold_cache.invalidate(), log_audit, rubric::generate_rubric. - Produces: GET /api/v1/inference-ops/thresholds?facility={id}, PUT /api/v1/inference-ops/thresholds/{key}, DELETE /api/v1/inference-ops/thresholds/{key}, GET /api/v1/inference-ops/rubric/preview?facility={id}.

[ ] Step 1: Write the failing integration test (append to tests/api_inference_ops.rs):

#[tokio::test]
async fn facility_override_wins_and_bumps_rubric_version() {
    let app = common::spawn_app().await;
    let jwt = common::admin_jwt(&app).await;
    let key = common::internal_api_key(&app).await;

    let v0 = common::rubric_version(&app, &key).await;

    // Override temperature_max_c at facility scope to 40.
    let res = app.client
        .put(format!("{}/api/v1/inference-ops/thresholds/temperature_max_c", app.addr))
        .bearer_auth(&jwt)
        .json(&serde_json::json!({"scope":"facility","scope_id": app.facility_id, "value": 40.0}))
        .send().await.unwrap();
    assert_eq!(res.status(), 200);

    // Effective value now 40 for this facility.
    let list: serde_json::Value = app.client
        .get(format!("{}/api/v1/inference-ops/thresholds?facility={}", app.addr, app.facility_id))
        .bearer_auth(&jwt).send().await.unwrap().json().await.unwrap();
    let eff = list["thresholds"].as_array().unwrap().iter()
        .find(|t| t["key"] == "temperature_max_c").unwrap()["effective"].as_f64().unwrap();
    assert_eq!(eff, 40.0);

    // Rubric version changed.
    let v1 = common::rubric_version(&app, &key).await;
    assert_ne!(v0, v1);

    // Out-of-bounds value rejected.
    let bad = app.client
        .put(format!("{}/api/v1/inference-ops/thresholds/temperature_max_c", app.addr))
        .bearer_auth(&jwt)
        .json(&serde_json::json!({"scope":"facility","scope_id": app.facility_id, "value": 9999.0}))
        .send().await.unwrap();
    assert_eq!(bad.status(), 400);
}

[ ] Step 2: Verify failure (routes missing).
[ ] Step 3: Implement handlers (routes/inference_ops.rs):

use axum::{extract::{Path, Query, State}, Extension, Json};
use uuid::Uuid;
use xylolabs_db::repo;

use crate::{error::AppError, extractors::authenticated::AuthenticatedUser, routes::log_audit, state::AppState};
use crate::threshold_registry::{lookup, REGISTRY};

#[derive(serde::Deserialize)]
pub struct FacilityQuery { pub facility: Uuid }

#[derive(serde::Serialize)]
struct ThresholdView {
    key: &'static str,
    category: &'static str,
    label: &'static str,
    unit: &'static str,
    default: f64,
    min: f64,
    max: f64,
    effective: f64,
    overrides: Vec<OverrideView>,
}

#[derive(serde::Serialize)]
struct OverrideView { scope: String, scope_id: Option<String>, value: f64 }

pub async fn list_thresholds(
    State(state): State<AppState>,
    user: AuthenticatedUser,
    Query(q): Query<FacilityQuery>,
) -> Result<Json<serde_json::Value>, AppError> {
    user.require_facility_admin(q.facility)?; // follow the project's existing RBAC helper
    let rows = repo::threshold_override::list_all(&state.db).await.map_err(AppError::Database)?;
    let mut out = Vec::new();
    for spec in REGISTRY {
        let effective = state.threshold_cache.get(spec.key, None, q.facility, None).await;
        let overrides = rows.iter()
            .filter(|r| r.threshold_key == spec.key)
            .map(|r| OverrideView { scope: r.scope.clone(), scope_id: r.scope_id.clone(), value: r.value })
            .collect();
        out.push(ThresholdView {
            key: spec.key, category: spec.category, label: spec.label, unit: spec.unit,
            default: spec.default, min: spec.kind.min(), max: spec.kind.max(), effective, overrides,
        });
    }
    Ok(Json(serde_json::json!({ "thresholds": out })))
}

#[derive(serde::Deserialize)]
pub struct UpsertBody { pub scope: String, pub scope_id: Option<String>, pub value: f64 }

pub async fn put_threshold(
    State(state): State<AppState>,
    user: AuthenticatedUser,
    Path(key): Path<String>,
    Json(body): Json<UpsertBody>,
) -> Result<Json<serde_json::Value>, AppError> {
    let spec = lookup(&key).ok_or_else(|| AppError::BadRequest(format!("unknown threshold key {key}")))?;
    if !["global","hw_version","facility","device"].contains(&body.scope.as_str()) {
        return Err(AppError::BadRequest("invalid scope".into()));
    }
    if !spec.kind.validate(body.value) {
        return Err(AppError::BadRequest(format!(
            "value {} out of bounds [{}, {}] for {}", body.value, spec.kind.min(), spec.kind.max(), key
        )));
    }
    // Facility admins may only write global/facility/device within their facility.
    // Enforce with the project's RBAC helper against body.scope_id where scope=facility.
    let row = repo::threshold_override::upsert(
        &state.db, &key, &body.scope, body.scope_id.as_deref(), body.value, Some(user.user_id()),
    ).await.map_err(AppError::Database)?;
    state.threshold_cache.invalidate().await;
    log_audit(state.db.clone(), Some(user.user_id()), None, "set_threshold_override",
        "anomaly_threshold_override", Some(row.id),
        Some(serde_json::json!({"key": key, "scope": body.scope, "value": body.value})), None);
    Ok(Json(serde_json::json!({ "ok": true, "id": row.id })))
}

#[derive(serde::Deserialize)]
pub struct DeleteQuery { pub scope: String, pub scope_id: Option<String> }

pub async fn delete_threshold(
    State(state): State<AppState>,
    user: AuthenticatedUser,
    Path(key): Path<String>,
    Query(q): Query<DeleteQuery>,
) -> Result<Json<serde_json::Value>, AppError> {
    lookup(&key).ok_or_else(|| AppError::BadRequest(format!("unknown threshold key {key}")))?;
    let n = repo::threshold_override::delete(&state.db, &key, &q.scope, q.scope_id.as_deref())
        .await.map_err(AppError::Database)?;
    state.threshold_cache.invalidate().await;
    log_audit(state.db.clone(), Some(user.user_id()), None, "delete_threshold_override",
        "anomaly_threshold_override", None, Some(serde_json::json!({"key": key, "scope": q.scope})), None);
    Ok(Json(serde_json::json!({ "ok": true, "deleted": n })))
}

pub async fn rubric_preview(
    State(state): State<AppState>,
    user: AuthenticatedUser,
    Query(q): Query<FacilityQuery>,
) -> Result<Json<serde_json::Value>, AppError> {
    user.require_facility_admin(q.facility)?;
    let (text, version) = crate::services::rubric::generate_rubric(&state, q.facility).await;
    Ok(Json(serde_json::json!({ "rubric_text": text, "config_version": version })))
}

(Adjust AuthenticatedUser, require_facility_admin, user_id(), and log_audit's exact signature to the project's actual helpers — verify against routes/facilities.rs / extractors/authenticated.rs.)

[ ] Step 4: Mount the routes — in router.rs, add to the JWT-authenticated /api/v1 router block (find it via grep -n "inference::\|facilities::\|/api/v1" router.rs and mirror a FacilityAdmin route group):

        .route("/inference-ops/thresholds", get(inference_ops::list_thresholds))
        .route("/inference-ops/thresholds/{key}",
            axum::routing::put(inference_ops::put_threshold).delete(inference_ops::delete_threshold))
        .route("/inference-ops/rubric/preview", get(inference_ops::rubric_preview))

Add inference_ops to the routes::{…} import list and pub mod inference_ops; in routes/mod.rs.

[ ] Step 5: Run the integration tests (from /tmp). Expected: both api_inference_ops tests pass.
[ ] Step 6: Commit + deploy

git add crates/xylolabs-server/src/routes/inference_ops.rs crates/xylolabs-server/src/routes/mod.rs crates/xylolabs-server/src/router.rs crates/xylolabs-server/tests/api_inference_ops.rs
git commit -S -m "feat(inference-ops): ✨ admin threshold CRUD + rubric preview"
~/flash-shared/gitminer-cuda/mine_commit.sh 7 && git pull --rebase && git push
bash scripts/deploy.sh

Task 11: Admin frontend — Inference Ops → Thresholds page

Files: - Create: frontend/src/api/inferenceOps.ts, frontend/src/pages/InferenceOpsThresholdsPage.tsx - Modify: frontend/src/components/layout/Sidebar.tsx (nav entry), the admin router (route), frontend/src/i18n/index.ts (EN + KO strings).

Interfaces: - Consumes: the Task 10 REST endpoints. Reuses shared FacilitySelect, IOS_INPUT_STYLE, Toast, dark-mode badges.

[ ] Step 1: API client (frontend/src/api/inferenceOps.ts):

import { apiClient } from "./client";

export interface ThresholdOverride { scope: string; scope_id: string | null; value: number; }
export interface ThresholdView {
  key: string; category: string; label: string; unit: string;
  default: number; min: number; max: number; effective: number; overrides: ThresholdOverride[];
}

export async function listThresholds(facility: string): Promise<ThresholdView[]> {
  const { data } = await apiClient.get(`/api/v1/inference-ops/thresholds`, { params: { facility } });
  return data.thresholds;
}
export async function putThreshold(key: string, scope: string, scopeId: string | null, value: number) {
  return apiClient.put(`/api/v1/inference-ops/thresholds/${encodeURIComponent(key)}`, { scope, scope_id: scopeId, value });
}
export async function deleteThreshold(key: string, scope: string, scopeId: string | null) {
  return apiClient.delete(`/api/v1/inference-ops/thresholds/${encodeURIComponent(key)}`, { params: { scope, scope_id: scopeId } });
}
export async function rubricPreview(facility: string): Promise<{ rubric_text: string; config_version: string }> {
  const { data } = await apiClient.get(`/api/v1/inference-ops/rubric/preview`, { params: { facility } });
  return data;
}

[ ] Step 2: Page component (frontend/src/pages/InferenceOpsThresholdsPage.tsx) — a facility-scoped table of registry keys showing the effective value (with the winning scope), an inline edit at facility scope (clamped to [min,max], font-size:16px input), a "Reset to default" (delete facility override), and a "Preview rubric" panel:

import { useEffect, useState } from "react";
import { useTranslation } from "../hooks/useTranslation";
import { FacilitySelect } from "../components/ui/FacilitySelect";
import { listThresholds, putThreshold, deleteThreshold, rubricPreview, type ThresholdView } from "../api/inferenceOps";

export default function InferenceOpsThresholdsPage() {
  const { t } = useTranslation();
  const [facility, setFacility] = useState<string>("");
  const [rows, setRows] = useState<ThresholdView[]>([]);
  const [rubric, setRubric] = useState<string>("");
  const [error, setError] = useState<string>("");

  async function load() {
    if (!facility) return;
    try { setRows(await listThresholds(facility)); setError(""); }
    catch { setError(t("inferenceOps.loadError")); }
  }
  useEffect(() => { load(); /* eslint-disable-next-line */ }, [facility]);

  async function save(key: string, value: number) {
    await putThreshold(key, "facility", facility, value);
    await load();
  }
  async function reset(key: string) {
    await deleteThreshold(key, "facility", facility);
    await load();
  }

  return (
    <div className="p-4 sm:p-6 space-y-6">
      <h1 className="text-xl font-semibold">{t("inferenceOps.title")}</h1>
      <FacilitySelect value={facility} onChange={setFacility} label={t("common.facility")} />
      {error && <p className="text-red-600 dark:text-red-400">{error}</p>}
      <div className="overflow-x-auto">
        <table className="min-w-full text-sm">
          <thead>
            <tr className="text-left text-gray-500 dark:text-gray-400">
              <th className="py-2 pr-4">{t("inferenceOps.threshold")}</th>
              <th className="py-2 pr-4">{t("inferenceOps.effective")}</th>
              <th className="py-2 pr-4">{t("inferenceOps.default")}</th>
              <th className="py-2 pr-4">{t("inferenceOps.range")}</th>
              <th className="py-2 pr-4"></th>
            </tr>
          </thead>
          <tbody>
            {rows.map((r) => (
              <ThresholdRow key={r.key} row={r} onSave={save} onReset={reset} />
            ))}
          </tbody>
        </table>
      </div>
      <div>
        <button
          className="rounded bg-gray-800 dark:bg-gray-200 text-white dark:text-gray-900 px-3 py-2"
          onClick={async () => facility && setRubric((await rubricPreview(facility)).rubric_text)}
        >
          {t("inferenceOps.previewRubric")}
        </button>
        {rubric && (
          <pre className="mt-3 whitespace-pre-wrap rounded bg-gray-50 dark:bg-gray-800 p-3 text-xs overflow-x-auto">
            {rubric}
          </pre>
        )}
      </div>
    </div>
  );
}

function ThresholdRow({ row, onSave, onReset }: {
  row: ThresholdView; onSave: (k: string, v: number) => void; onReset: (k: string) => void;
}) {
  const [val, setVal] = useState<string>(String(row.effective));
  useEffect(() => setVal(String(row.effective)), [row.effective]);
  const hasFacilityOverride = row.overrides.some((o) => o.scope === "facility");
  return (
    <tr className="border-t border-gray-100 dark:border-gray-800">
      <td className="py-2 pr-4">{row.label}{row.unit ? ` (${row.unit})` : ""}</td>
      <td className="py-2 pr-4">
        <input
          value={val}
          onChange={(e) => setVal(e.target.value)}
          inputMode="decimal"
          className="w-28 rounded border px-2 py-1 bg-white dark:bg-gray-900"
          style={{ fontSize: 16 }}
        />
      </td>
      <td className="py-2 pr-4 text-gray-500">{row.default}</td>
      <td className="py-2 pr-4 text-gray-500">[{row.min}, {row.max}]</td>
      <td className="py-2 pr-4 space-x-2">
        <button className="text-blue-600 dark:text-blue-400"
          onClick={() => { const n = Number(val); if (Number.isFinite(n) && n >= row.min && n <= row.max) onSave(row.key, n); }}>
          Save
        </button>
        {hasFacilityOverride && (
          <button className="text-gray-500" onClick={() => onReset(row.key)}>Reset</button>
        )}
      </td>
    </tr>
  );
}

[ ] Step 3: Add i18n keys (frontend/src/i18n/index.ts) under both EN and KO trees:

inferenceOps.title       EN "Anomaly Thresholds"      KO "이상 감지 임계값"
inferenceOps.threshold   EN "Threshold"               KO "임계값"
inferenceOps.effective   EN "Effective"               KO "적용값"
inferenceOps.default     EN "Default"                 KO "기본값"
inferenceOps.range       EN "Range"                   KO "범위"
inferenceOps.previewRubric EN "Preview rubric"        KO "판정 기준 미리보기"
inferenceOps.loadError   EN "Failed to load thresholds" KO "임계값을 불러오지 못했습니다"

[ ] Step 4: Nav + route — add a Sidebar entry (SVG icon, no emoji) and a route to InferenceOpsThresholdsPage, gated to FacilityAdmin via the existing RoleGate/ProtectedRoute pattern.
[ ] Step 5: Build + typecheck

Run: cd frontend && npx tsc -b --noEmit && npx vite build Expected: zero TS errors; build succeeds.

[ ] Step 6: Browser test — Playwright at mobile 375 / tablet 768 / desktop 1280, login xylolabs/solution_6231, open the page, edit a value, preview the rubric; assert 0 pageerror and no horizontal overflow at each viewport.
[ ] Step 7: Commit + deploy

git add frontend/src/api/inferenceOps.ts frontend/src/pages/InferenceOpsThresholdsPage.tsx frontend/src/components/layout/Sidebar.tsx frontend/src/i18n/index.ts
git commit -S -m "feat(frontend): ✨ Inference Ops thresholds admin page"
~/flash-shared/gitminer-cuda/mine_commit.sh 7 && git pull --rebase && git push
bash scripts/deploy.sh

Phase 5 — Fleet coordination (separate repo; documented here).

Task 12: Document the GPU-fleet fetch change

Files: - Modify: docs/API.en.md, docs/API.ko.md (document GET /api/internal/inference/rubric), and add a short integration note for xylolabs-gpu-ops.

[ ] Step 1: Document the endpoint in both API docs — request (internal X-Api-Key, ?facility=), response { rubric_text, config_version, generated_at }, ETag/If-None-Match/304, and the recommended fleet loop (fetch on boot + every N minutes; on 200 set gemma.system_prompt = rubric_text; on failure keep the last-known rubric).
[ ] Step 2: Cross-reference the spec + this plan from docs/KNOWLEDGE-BASE.md (Inference Ops section) so the "thresholds live in the API now, not the served prompt" fact is discoverable.
[ ] Step 3: Commit (docs-only, no deploy)

git add docs/API.en.md docs/API.ko.md docs/KNOWLEDGE-BASE.md
git commit -S -m "docs(inference-ops): 📝 document rubric pull endpoint + fleet integration"
~/flash-shared/gitminer-cuda/mine_commit.sh 7 && git pull --rebase && git push

[ ] Step 4: Hand off the one-line fleet change — in xylolabs-gpu-ops (separate repo, out of scope for this plan), replace the hardcoded gemma system-prompt with a fetch of GET https://api.xylolabs.com/api/internal/inference/rubric?facility=<id> using the internal key, honoring the ETag. Track as a follow-up PR there.

Self-Review

Spec coverage: - §5.1 registry → Task 1. §5.2 migration → Task 2. §5.3 repo → Task 3. §6 resolver + cache → Tasks 4–5. §7 consumers → Tasks 6–7 (guard, detector) + Task 9 (rubric endpoint). §7.1 rubric generation + pull/ETag → Tasks 8–9. §8 admin surface → Tasks 10–11. §9 rollout no-op → defaults in Task 1 + deploy checkpoints in Tasks 7/10/11. §10 error handling → resolver falls back to default (Task 4 get), LLM-fail degrades to template (Task 8), bounds/unknown-key 400 (Task 10). §11 testing → tripwire (Task 1), resolver precedence (Task 1/4), integration (Tasks 9/10), auth (Task 10). §12 future → out of scope, noted. All covered.

Placeholder scan: No "TBD"/"handle edge cases" left; the only deliberate stubs are elaborate_prose (documented pass-through so numbers stay correct in v1) and RBAC helper names that must be matched to the project's actual AuthenticatedUser API (flagged explicitly at each use).

Type consistency: ThresholdSpec/ThresholdKind/ResolvedOverrides/resolve (Task 1) are consumed unchanged by Tasks 4/8/10. ThresholdOverrideRow fields (Task 3) match select_overrides/list_thresholds usage (Tasks 4/10). ThresholdCache::{get,version,invalidate,has_hw_scope} (Task 4) are called with matching signatures in Tasks 6/7/8/9/10. generate_rubric -> (String, String) (Task 8) matches Tasks 9/10 destructuring. Frontend ThresholdView (Task 11) matches the list_thresholds JSON (Task 10).