Skip to content

Inference Admin — LLM Model Registry + Job Queueing from the Admin UI

Date: 2026-07-25 Status: Approved (brainstorming complete)

Problem

The inference integration surface (inference_models registry, inference_jobs SKIP LOCKED queue, GPU dispatch) is reachable only with an internal-scope API key. Operators cannot register the LLM serving presets as first-class models, cannot queue an analysis job for a session from the admin console, and cannot watch job status. The xylolabs-ai-client research doc tracks this as an open follow-up ("register inference_models rows for the LLM presets so jobs can be queued per-model from the admin UI").

Decision Summary

Question Decision
Who registers LLM preset models Admins, manually, via new admin UI CRUD (SuperAdmin). No auto-registration from ai-client, no seed migration.
Queueing UX Both: a dedicated Inference admin page (models + jobs) AND a "request analysis" action on the session detail page.
Automatic queueing Per-model auto_queue flag: when set (and model is_active), every session close in that model's facility enqueues one job automatically.
RBAC Model CRUD + auto_queue toggle = SuperAdmin. Job list/submit/cancel = FacilityAdmin.
Backend approach New JWT route layer routes/inference_admin.rs reusing existing repo functions; internal-scope routes untouched.

1. Schema

One migration (strictly monotonic YYYYMMDDHHMMSS prefix; verify with ls crates/xylolabs-db/migrations | awk -F_ '{print $1}' | sort -c):

ALTER TABLE inference_models ADD COLUMN auto_queue BOOLEAN NOT NULL DEFAULT false;
-- Dedup guard for the auto-queue hook: at most one live (queued/running) job
-- per (session, model). Manual re-submits after completion stay allowed.
CREATE UNIQUE INDEX idx_inference_jobs_session_model_live
    ON inference_jobs (session_id, model_id)
    WHERE status IN ('queued', 'running') AND session_id IS NOT NULL AND model_id IS NOT NULL;
  • InferenceModel model/DTOs gain auto_queue: bool (create/update DTOs take Option<bool>; response includes it).
  • LLM presets are ordinary rows created through the UI: artifact_s3_key holds the HF checkpoint id (e.g. google/gemma-4-12B-it — passes the existing [A-Za-z0-9._/-]+ validation), framework = custom, input_type = audio, free-form config JSONB for runtime hints (e.g. {"profile": "gemma-4-12b", "serving": "vllm"}). No schema special-casing for LLM vs classic models.

2. Backend — JWT admin routes (routes/inference_admin.rs, new)

Mounted under /api/v1/inference-admin/* in the JWT/RBAC section of router.rs (same block as /inference-ops/*). All handlers reuse the existing repo::inference_model / repo::inference_job functions and the validation helpers from routes/inference.rs (extracted or re-exported as needed — no logic duplication).

Models: - GET /models — FacilityAdmin. Facility-scoped list (SuperAdmin may pass facility_id query to view any facility). - POST /models — SuperAdmin. Body = existing CreateInferenceModelRequest + facility_id + optional auto_queue. Audit-logged. - PATCH /models/{id} — SuperAdmin. Existing update fields + auto_queue + is_active. Audit-logged. - DELETE /models/{id} — SuperAdmin. FK-guarded 409 passthrough (existing repo behavior). Audit-logged.

Jobs: - GET /jobs — FacilityAdmin. Facility-scoped, filters: status, model_id, session_id; paginated, newest first. - POST /jobs — FacilityAdmin. Body { model_id, session_id }. The server validates both belong to the caller's facility, builds input_data = {"session_id": "<uuid>"} itself (matching the ai-client /v1/inference contract), and reuses the existing submit validation. Returns 201 + job. Audit-logged. - POST /jobs/{id}/cancel — FacilityAdmin. Reuses existing cancel logic. Audit-logged.

RBAC failure modes follow /inference-ops/* exactly: require_role for the role gate (403), authorize_scope-style facility checks (403 on cross-facility for this admin surface — no tenancy-hiding 404s here).

3. Auto-queue hook (session close)

A shared helper in a new services/auto_inference.rs, called wherever an ingest session transitions to closed (HTTP close, WS close, watchdog/reaper close):

  1. Load the facility's models WHERE is_active AND auto_queue.
  2. For each, insert one inference_jobs row with input_data = {"session_id": …}, priority = -10 (manual jobs, default 0, always win), max_attempts default.
  3. Dedup: rely on the partial unique index; treat unique-violation as already-queued (no error). Belt: skip insert if a live job exists.
  4. Strictly best-effort: any error is tracing::warn! and never blocks or fails the session close path. No new latency-sensitive work inline — the enqueue is a single INSERT per auto model (expected 0-2 models).

Auto-queued jobs are ordinary jobs afterward: same worker, same GPU dispatch, same result path.

4. Admin UI (frontend/)

  • New InferencePage (route + sidebar entry, pattern: FirmwarePage):
  • Models table: name/version, input type, artifact (HF id), active, auto-queue, updated. SuperAdmin sees create/edit/delete + auto toggle; FacilityAdmin sees read-only.
  • Jobs table: created, model, session (link), status chip, attempts, error/result summary, cancel button for queued/running. Poll every ~10 s while the tab is visible.
  • MetadataSessionDetailPage: "분석 요청" button + active-model dropdown; on queue success show a toast linking to the Inference page.
  • New frontend/src/api/inferenceAdmin.ts client; EN + KO i18n keys; mobile-first (≥16px inputs); SVG icons only; no emoji.

5. Testing & Validation

  • Integration tests (docker-gated #[ignore], new api_inference_admin.rs):
  • RBAC: model POST as FacilityAdmin → 403; job POST as plain User → 403.
  • Model CRUD roundtrip incl. auto_queue toggle (SuperAdmin).
  • Job submit → row exists with input_data.session_id, facility checked; cross-facility session/model rejected.
  • Auto-queue: close a session with an active auto model → exactly one job; close again/dup insert → still one live job (index holds).
  • Cancel semantics preserved.
  • cargo check + clippy; tsc -b --noEmit + vite build in both frontends; Playwright at 375×812 / 768×1024 / 1280×800 on the new page; zero pageerror.
  • Docs: new section in docs/API.{en,ko}.md (inference-admin endpoints + auto_queue semantics).
  • Deploy scripts/deploy.sh; post-deploy: subdomains healthy, migration applied, RBAC spot-check with test credentials, queue one job against the live ai-client host and watch it complete.

Out of Scope

  • ai-client self-registration of profiles (rejected in design).
  • Seed migration for presets.
  • Any change to internal-scope routes, the worker, or GPU dispatch.
  • Operator app (frontend-app/) surfaces.