Skip to content

OpenAI-Compatible LLM Endpoint (OpenRouter Support) — Design

Date: 2026-07-22 Status: Approved design, pending implementation Scope: xylolabs-api backend only. The sibling Xylolabs-Knowledge-Engine change is specified separately in that repository (docs/superpowers/specs/2026-07-22-openrouter-endpoint-design.md).

Goal

Let operators point the backend's three LLM call sites at any OpenAI-compatible chat/completions endpoint — primarily OpenRouter — as a deploy-time switch, while keeping Gemini (via its OpenAI-compatibility layer) the zero-config default. No runtime failover, no per-feature provider selection.

Background

All three LLM call sites already speak the OpenAI chat/completions wire format against Gemini's compatibility endpoint, authenticated with Authorization: Bearer:

Call site File Uses
Daily report crates/xylolabs-server/src/routes/daily_report.rs config.gemini_model
Facility assistant crates/xylolabs-server/src/routes/facility_assistant.rs assistant.gemini_model registry override, falling back to config.gemini_model
Alert SMS contact selection crates/xylolabs-server/src/services/alert_llm.rs config.gemini_model, reasoning_effort: "low"

Each file duplicates a GEMINI_ENDPOINT const pointing at https://generativelanguage.googleapis.com/v1beta/openai/chat/completions. OpenRouter exposes the same request/response shape at https://openrouter.ai/api/v1/chat/completions, so provider portability reduces to making the endpoint URL, API key, and model configurable.

Configuration

Three new environment variables, all optional:

Env var Default Meaning
LLM_ENDPOINT https://generativelanguage.googleapis.com/v1beta/openai/chat/completions Full URL of the OpenAI-compatible chat/completions endpoint
LLM_API_KEY (falls back to GEMINI_API_KEY) Bearer key sent to LLM_ENDPOINT
LLM_MODEL (falls back to GEMINI_MODEL, default gemini-3.6-flash) Model ID sent in the request body (OpenRouter style: google/gemini-3.6-flash)

Resolution happens once in Config::from_env() (config.rs):

  • New struct field llm_endpoint: String (env LLM_ENDPOINT with the Gemini default above).
  • LLM_API_KEY, when set, wins over GEMINI_API_KEY and is stored in the existing gemini_api_key field; likewise LLM_MODEL over GEMINI_MODEL into gemini_model. Reusing the existing fields keeps the struct, call sites, and ~6 test fixtures unchanged except for the one new field.
  • A code comment on both fields documents that they hold the resolved LLM credential/model, not necessarily a Gemini one; the field names stay for churn-avoidance, not semantics.

Validation (in the existing Config::validate() next to the gemini_model non-empty check): llm_endpoint must be non-empty and start with http:// or https://. Anything else fails startup with a clear message.

Debug output: llm_endpoint printed as-is (it is not a secret); gemini_api_key stays [REDACTED].

Backward compatibility: a deployment that sets none of the new variables behaves byte-for-byte as today (Gemini endpoint, GEMINI_API_KEY, GEMINI_MODEL). The production .env needs no changes until the operator opts into OpenRouter.

Call-site changes

  • Delete the three per-file GEMINI_ENDPOINT consts; POST to config.llm_endpoint (via state.config where AppState is at hand, via the &Config parameter in alert_llm.rs).
  • Request bodies, bearer_auth, timeouts (15 s alert path), response caps (256 KiB success / 8 KiB error), and reasoning_effort: "low" are unchanged — OpenRouter accepts reasoning_effort and ignores or maps it per provider.
  • Log/error strings that say "Gemini" in these paths change to "LLM" only where the edit is a pure string rename; no structural log changes.

Runtime config registry

assistant.gemini_model keeps its key (rows are already seeded in production system_config). Its registry description changes to "Model override for the facility assistant, sent to the configured LLM endpoint". It continues to override only the assistant's model, on whatever endpoint is configured.

Testing

  • Unit tests in config.rs for the resolution rules: LLM_API_KEY wins over GEMINI_API_KEY; LLM_MODEL wins over GEMINI_MODEL; defaults apply when unset; invalid llm_endpoint scheme fails validation. Follow the existing env-var test pattern in config.rs (serialized env mutation).
  • Existing integration-test fixtures gain the llm_endpoint field with the Gemini default.
  • cargo check + cargo clippy clean; full test suite passes.

Documentation

  • .env.example and scripts/setup-server.sh: commented LLM_ENDPOINT / LLM_API_KEY / LLM_MODEL block with an OpenRouter example.
  • docs/DEPLOYMENT-GUIDE.md: three new rows in the env table.
  • docs/API.{en,ko}.md: daily-report and assistant sections note the model resolution order (LLM_MODEL → GEMINI_MODEL; assistant additionally assistant.gemini_model).
  • docs/KNOWLEDGE-BASE.{md,ko.md}: short entry describing the OpenRouter switch procedure.

Rollout

  1. Implement + tests + docs, one feature commit.
  2. Deploy via scripts/deploy.sh; default behavior is unchanged, so the deploy is risk-free for the live Gemini path.
  3. Switching production to OpenRouter later is an .env edit (three variables) + container restart; switching back is deleting them.

Out of scope

  • Runtime failover between providers.
  • Per-feature provider/model selection beyond the existing assistant model override.
  • Streaming responses (no call site streams today).