Gateway & routing

The platform speaks the OpenAI API. A client points at one endpoint, names a model — or asks the platform to choose one — and never learns which pod answered.

Sending a request

# What's available
curl -sS https://llm.priv.cloud.ogenki.io/v1/models \
  -H "Authorization: Bearer $LLM_API_KEY" | jq '.data[].id'

# Name a model explicitly — skips classification entirely
curl -sS https://llm.priv.cloud.ogenki.io/v1/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
        "model": "xplane-qwen-coder",
        "messages": [{"role": "user", "content": "Write a Go worker pool."}]
      }'

# Let the Semantic Router choose from the prompt
curl -sS https://llm.priv.cloud.ogenki.io/v1/chat/completions \
  -H "Authorization: Bearer $LLM_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
        "model": "MoM",
        "messages": [{"role": "user", "content": "Write a Go worker pool."}]
      }'

The two calls reach the same pod. The difference is roughly 250–300 ms of classification on the MoM path — worth it when the client cannot know which model suits the prompt, wasted when it can. See Coding Clients for which client pins which model, and why.

The request path

One OpenAI-compatible request from a laptop to a GPU: the client reaches the Cilium Gateway over Tailscale, the Envoy AI Gateway authenticates it with an API key from AWS Secrets Manager, strips the Authorization header, and routes it through the semantic-router and rate-limit filters onto the vLLM Service backing the requested model

Ingress. External clients arrive over Tailscale at the Cilium Gateway platform-tailscale-general and are forwarded to the Envoy AI Gateway data plane Service. In-cluster clients — OpenWebUI, the nightly Promptfoo eval — address that Service directly. Nothing is reachable from the public internet; see Private access.

Authentication. A SecurityPolicy (apiKeyAuth) targets the Gateway, so every route inherits it rather than each declaring its own. Keys come from AWS Secrets Manager through External Secrets, and the gateway strips the Authorization header before forwarding — vLLM never sees a credential.

Prompt classification (ext_proc, filter index 0). The Semantic Router is wired in as an Envoy ext_proc gRPC filter, inserted ahead of the AI Gateway’s own extproc by an EnvoyPatchPolicy — a raw xDS JSONPatch. The ordering is not optional and an EnvoyExtensionPolicy cannot express it, because that API can only append filters. The router rewrites body.model only when the client sent model: MoM (or the literal auto); an explicit xplane-* name passes through untouched.

Routing. The AI Gateway extproc derives the x-ai-eg-model header from the (possibly rewritten) body and emits gen_ai_* telemetry. An AIGatewayRoute matches that header and forwards through an AIServiceBackend → Backend → the model’s Service on port 8000.

Gateway routing is half-migrated. Only xplane-qwen-coder has its gateway objects composition-owned (spec.gateway.enabled: true). The other three claims still route through a hand-written apps/base/ai/llm/ai-gateway-routes/route.yaml. Adding a model today means remembering to add its route by hand unless the claim opts in.

Semantic routing — `model: MoM`

Sending model: MoM lets the Semantic Router pick from the prompt. Its decision list, highest priority first (infrastructure/base/vllm-semantic-router/helmrelease.yaml):

Priority	Decision	Target
110	code + reasoning	`xplane-qwen-coder` (`use_reasoning: true`)
100	code	`xplane-qwen-coder`
90	reasoning (math / physics)	`xplane-qwen3-8b` (`use_reasoning: true`)
80	multilingual	`xplane-qwen3-8b`
50	general (the default)	`xplane-qwen3-8b`

xplane-llamaguard3-1b and the two LoRA adapters on xplane-qwen-coder appear in no rule above. They hold serving capacity and are reachable only by naming them directly. The router’s own in-pod prompt_guard classifier blocks jailbreak attempts, but that is a filter, not a routing decision — there is no automatic guardrail dispatch, so the always-warm LlamaGuard replica holds a GPU and serves no automatic traffic.

LoRA canaries

xplane-qwen-coder carries two LoRA adapters, each addressable as a model name of its own, and sends 10% of its traffic to one of them:

  gateway:
    enabled: true
    canaries:
      - adapter: xplane-qwen-coder-sql-dpo
        weightPercent: 10

The adapter name is matched verbatim against loraAdapters[].name — it is not derived from the base model’s name, so a typo produces a route to nothing rather than a validation error.

Canaries are mutually exclusive with the Gateway API Inference Extension’s endpoint picker: the EPP is implemented but enabled on zero claims, because the only gateway-enabled claim uses a canary. See the roadmap for what turning it on would take.

Known gaps

No distributed tracing. OTLP export from the AI Gateway extproc is written but not enabled, pending verification against VictoriaTraces.
Route ownership is split between the composition and a hand-written manifest, as above.

The InferenceService claim Autoscaling & GPUs