Skip to content

Inference Job Pull Path — Executing Admin-Queued Jobs on Unreachable GPU Hosts

Date: 2026-07-25 Status: Approved (brainstorming complete)

Problem

The inference-admin feature (spec 2026-07-25-inference-admin-design.md, shipped today) lets operators queue analysis jobs, but nothing executes them. Job dispatch is push-based: the API server POSTs to a registered GPU server's http://{ip}:{port}/v1/inference. Production runs on AWS (ap-northeast-2); the xylolabs-ai-client GPU host is a Mac on a private network (172.30.x). Prod has no Tailscale, and the SSRF allowlist rejects private addresses at registration, so that host cannot be registered or reached. Live state: 0 GPU servers registered, so queued jobs sit queued forever (the worker finds no available server and requeues without burning an attempt).

The same client already runs a pull loop successfully — it polls GET /api/internal/sessions for closed sessions and POSTs results. Pull needs no inbound reachability. Giving the job queue a pull path makes the admin queueing feature work with the deployment we actually have.

Decision Summary

Question Decision
Client identity Register as a pull-mode GPU server (no reachable IP required) so the fleet view keeps worker visibility and jobs record gpu_server_id.
Result reporting Dedicated POST /internal/jobs/{id}/complete and /fail — the same mark_completed/mark_failed state machine the push worker uses. Anomaly reports keep going through the existing /internal/inference/results.
ai-client loops Job-pull loop and session-cursor loop coexist, each independently toggled.
Claim mechanics Reuse the existing repo::inference_job::claim_next_queued (SKIP LOCKED → running).

1. Schema

One migration (strictly monotonic prefix; verify with ls crates/xylolabs-db/migrations | awk -F_ '{print $1}' | sort -c):

ALTER TABLE gpu_servers ADD COLUMN mode TEXT NOT NULL DEFAULT 'push'
    CHECK (mode IN ('push', 'pull'));
ALTER TABLE gpu_servers ALTER COLUMN ip_address DROP NOT NULL;
  • push (default, all existing rows): unchanged — the server dispatches to it and probes GET /health.
  • pull: the client claims work itself. ip_address/port carry no meaning and may be omitted.
  • GpuServer model + DTOs gain mode; create DTO takes Option<GpuServerMode> (default push).

2. Backend

2.1 Registration (routes/gpu_servers.rs)

POST /api/internal/gpu-servers accepts mode. For pull: ip_address is optional and the SSRF/reachability validation is skipped (nothing will ever be dispatched to it). For push: unchanged — ip_address required and allowlist-validated. PATCH may not change mode (register a new row instead); attempting it is a 400.

2.2 Health (services/gpu_health_checker.rs)

The probe loop's server query excludes mode = 'pull' — a pull host is unreachable by design and must never be flipped to error for it. Pull servers report liveness through the existing agent-push endpoint POST /gpu-servers/{id}/health, which already drives update_health (utilization, VRAM, last_health_at, degraded-above-90% rule).

2.3 Job pull endpoints (routes/inference.rs, internal scope)

  • POST /api/internal/jobs/claim — body { gpu_server_id }. Validates the server belongs to the key's facility and is mode = 'pull'; calls claim_next_queued(db, facility_id); on a claim, stamps gpu_server_id on the row and returns the job (200). Empty queue → 204 No Content.
  • POST /api/internal/jobs/{id}/complete — body { result: <json ≤1 MB> }. Facility-checked; job must be running. Calls mark_completed.
  • POST /api/internal/jobs/{id}/fail — body { error_message: String ≤2000 }. Facility-checked; job must be running. Calls mark_failed, which keeps the existing requeue-until-max_attempts semantics.

All three are audit-logged like the other internal mutations.

2.4 Push worker (services/inference_worker.rs)

The facility scan skips facilities that have no online, non-degraded, mode = 'push' server. Without this the in-process worker claims pull-destined jobs, fails to find a dispatch target, and requeues them — pure churn that also races pull clients out of their work.

3. ai-client (sibling repo)

  • Settings: AI_CLIENT_JOB_PULL_ENABLED (default false), AI_CLIENT_JOB_POLL_SECS (default 30), AI_CLIENT_JOB_TIMEOUT_SECS (default 480 — must stay below the server's 600 s stale-running reaper). The existing session-cursor loop keeps its own settings and is unaffected.
  • python -m app.cli register gains --mode pull (registers without an IP) and persists gpu_server_id in the worker state file.
  • New JobWorker loop: claim → on 204 sleep → on 200 read payload.session_id → run the existing analyze_session → submit the anomaly report through the existing gated path (AI_CLIENT_SUBMIT_RESULTS etc.) → complete with the verdict JSON as result. Any exception or timeout → fail with the message.
  • Health: the existing push_gpu_health cadence continues, now with the registered pull-server id.

4. Failure semantics

  • Client dies mid-job: the existing stale-running reaper (fail_stale_running, inference_job_stale_timeout_secs = 600 s) requeues or fails the job. The client's own per-job timeout (480 s) fires first in the normal case, so double execution is a narrow window, not the default.
  • At-least-once: a requeued-then-reclaimed job can be analyzed twice. Duplicate anomaly reports are already absorbed by the same-minute idempotency key in repo::anomaly_report. Documented, not engineered around.
  • Tenancy: claim/complete/fail all resolve the facility from the API key; a job or GPU server from another facility is rejected.

5. Testing, docs, rollout

  • API integration tests (docker-gated #[ignore], new api_inference_pull.rs): claim returns a job, marks it running, stamps gpu_server_id; empty queue → 204; complete stores result and sets completed; fail below max_attempts requeues, at the cap fails; cross-facility job → rejected; claim with a push-mode server id → 400; pull-mode registration succeeds without an IP and the health checker's query excludes it.
  • ai-client tests (respx): claim → analyze → complete happy path; analyze raises → fail called with the message; 204 → no work, loop sleeps.
  • Docs: docs/API.{en,ko}.md (new pull-path subsection under the inference section) and the ai-client's docs/PIPELINE.md.
  • Rollout: deploy the API, register the Mac as a pull server, enable AI_CLIENT_JOB_PULL_ENABLED, queue one job from the admin UI, and watch it go queued → running → completed with a verdict stored in job.result.

Out of Scope

  • Making the GPU host inbound-reachable (tunnels, port forwarding, SSRF allowlist changes).
  • Changing the push dispatch path itself, the GPU-server selection heuristic, or the sync proxy.
  • Per-facility auto-queue rate caps (separate, already-validated advisory).
  • Operator app surfaces.