Skip to content

Runbook — Kospo-An wildcard API key retirement

Status: PLANNED, not executed. The trigger is the harbor equipment-replacement cutover (user-stated: the on-site harbor units are being replaced with new hardware soon). Do NOT deactivate the key before the cutover checklist below passes — the key is the ONLY working uplink for the live harbor unit posting as uid100001; killing it early silences that uplink with no compensating data path.

The key

Field Value
api_keys.id f473b259-067a-4662-be7e-d25861239c12
Name Kospo-An
Prefix xk_bec48
Scopes {*} (wildcard — grants every scope incl. ota, live:*)
Facility b666b8c3-bbc2-4675-a3ac-8b6a1e6dad9f (Kospo-An)
Device binding none (unbound)
Created 2026-04-06

Why it must retire

  • {*} scope on an unbound key is a facility-wide skeleton key; any leaked copy can post, listen, and OTA-poll for the whole facility.
  • It enables the legacy identity lane: harbor "Box B" posts XMBP sessions as device_uid=100001 (Xylo-KOSPO #0001 legacy row) through it (~every 10.4 min as of 2026-07-12), invisible to per-device attribution. Its health lane self-reported fw_version "0.1.0" until it went silent on 2026-07-01T04:59Z (see the knowledge-base incident note).
  • The replacement fleet ships with per-device keys + provisioned NVS identities (platform plan 178 hardening) — after cutover nothing legitimate uses the wildcard key.

Preconditions (ALL must hold before deactivation)

  1. Replacement harbor units are on-site, provisioned (per-device NVS identity + per-device key), and CONFIRMED posting: their own dongle_id rows show fresh last_health_at AND fresh ingest sessions over ≥ 48 h.
  2. The old units are physically powered off / recovered for bench analysis (they are the only legitimate posters on the wildcard key; anything still using the key after they power off is unexpected and interesting).
  3. 48-h pre-check on key usage: SELECT count(*), max(created_at) FROM api_request_logs WHERE api_key_id = 'f473b259-…' — expect the last use to be BEFORE the old units powered off. If the key is still being used, STOP and identify the caller first (device_post_attribution view).
  4. Operator sign-off relayed through main (the deactivation is reversible, but every hour deactivated with a live legitimate poster is lost harbor data).

Cutover steps

  1. Snapshot evidence (read-only): last 24 h of api_request_logs rows for the key; uid100001 row state (firmware_version, firmware_version_reported_at, last_seen_at, last_health_at).
  2. Deactivate via the admin UI (API Keys → Kospo-An → delete) or DELETE /api/v1/api-keys/{id} — soft-delete/is_active=false, audited. Do NOT delete the row in SQL; history joins against api_keys.id.
  3. Watch api_request_logs for new 401s from the old callers for 24 h: SELECT created_at, path, ip_address FROM api_request_logs WHERE response_status = 401 AND created_at > now() - interval '24 hours'. A steady 401 stream = something real still depended on the key → evaluate rollback (step 5).
  4. Annotate the legacy rows: uid100001 (and 100002) descriptions get "retired uplink — wildcard key deactivated ; expect no further posts" so their staleness is understood later.
  5. Rollback (if the cutover must abort): re-activate the key row (UPDATE api_keys SET is_active = true … via admin surface, not SQL, if exposed — otherwise the operator re-creates and re-provisions), and record why in the audit log / KB.

Also in scope at the same cutover

  • device_0001 (xk_7707d) and device-0002 (xk_23560) per-device keys: still active, unused since 2026-06-24 / 2026-05-25. Retire together with the wildcard key once the replacement units take over those identities (same preconditions; they are bound keys so blast radius is one device each).
  • The already-inactive legacy keys (kospo-demo ×2, hw-rp2350-…) need no action.

Non-goals

  • No deletion of uid100001 / uid100002 rows or their history — the sessions/health series are evidence for the harbor incident analysis and the fleet-history record.
  • No key rotation for bench/office units here — this runbook is strictly the harbor wildcard lane.

Quarantine sentinel key pairing (plan 185, pinned 2026-07-13)

The firmware hardening cycle replaced the baked default identity/key: dead-NVS boots now send ONE bounded health-shaped beacon per wake as the fleet-wide canonical sentinel xylozero_260501_0000 (the dongle comes solely from the baked BOARD1_DEFAULT_DONGLE_ID, never derived from hardware revision — v2/v3 boards on the universal build emit the SAME identity) with identity_source: "sentinel" plus the device MAC.

Pinned pairing sequence (plan 185 §Cross-repo contract):

  1. Server deploys the sentinel-quarantine handling (this repo's D-C10-SENTINEL-SERVER work).
  2. Seed the single quarantine row xylozero_260501_0000 (admin surface, or accept one beacon via a facility key).
  3. Mint the quarantine key VIA THE AUDITED ADMIN SURFACE, bound to that row, ingest scope ONLY — the binding guard makes it unusable on any real device's routes (bound keys 404 elsewhere).
  4. Deliver the plaintext to the firmware side; it replaces the baked BOARD1_DEFAULT_API_KEY and ships in the next release. The currently baked xk_66c9b… value (firmware commit ee94e06) is a PLACEHOLDER — its only job is to displace the old real default; it 401s harmlessly (single attempt per wake, no retry storm) until the server-minted key replaces it. 401s in api_request_logs are usable evidence the beacon path works pre-registration.

Additional retirement ledger entries (same cutover)

  • xk_c785d… — the OLD baked default key ("board1 xylozero_260501_0003 (cycle-20 reseed)" row). STILL ACTIVE and used daily by every fleet unit running a pre-hardening firmware build: it retires only once the fleet is confirmed off firmware that bakes it (post-OTA / post-replacement), not at the harbor cutover alone.
  • xk_5ec47046… — the legacy device-0003 wildcard (server-side only; referenced in the platform roster .fleet.toml comments). Same preconditions as the Kospo-An key: verify zero recent uses in api_request_logs, then deactivate via the audited admin surface.