Roadmap

A now-retired note in the repository, llm-platform-future-paths, originally listed seven upgrade paths for evolving the platform beyond its current shape. None were committed work — they were reference notes for when the open-weights ecosystem, the team’s needs, or the demo scope warranted the next investment. This page carries forward only what is still open, checked against the done spec archive and the pinned composition source as of 2026-08-20.

When picking one of these up, choose the path whose trigger has actually fired, not the most ambitious one. Bigger hardware doesn’t always mean bigger value in a foundation-showcase context.

Shipped — re-introduce InferencePool + EPP

The path that proposed gating a Gateway API Inference Extension InferencePool + Endpoint Picker behind an opt-in composition field has shipped as SPEC-004 (docs/specs/done/2026-Q3/004-per-inferenceservice-inferencepool-endpoint/). Verified directly against the pinned KCL module: spec.gateway.endpointPicker.enabled renders a per-claim InferencePool + EPP HelmRelease and swaps the AIGatewayRoute base rule’s backend to the InferencePool — exactly the mechanism the roadmap entry proposed. All coding tasks in the spec’s plan are complete; only the live-cluster e2e validation tasks remain open, because the field is enabled on zero claims today — it is mutually exclusive with LoRA canaries, and the one gateway-enabled claim (xplane-qwen-coder) uses a canary. Turning it on for a high-traffic model at max ≥ 2 replicas is what remains of this path.

A follow-on spec, docs/specs/done/2026-Q3/011-inferencepool-saturation-keda/, proposes a fourth KEDA trigger reading the InferencePool’s own saturation gauge instead of the three raw vLLM metrics. It is filed under the done archive, but the pinned KCL module renders only the three original triggers — no InferencePool-gauge trigger exists in the composition source — and the spec’s own task and review checklists are almost entirely unchecked. Treat this piece as not shipped, regardless of which directory it lives in.

Still open

1. Bigger coder model on the existing L4 NodePool

Swap Qwen/Qwen2.5-Coder-7B-Instruct for a larger MoE coder (originally proposed: Qwen/Qwen3-Coder-30B-A3B-Instruct at AWQ-4bit) that still fits a single L4’s 24 GiB. The fleet still runs the 7B model today (apps/base/ai/llm/qwen-coder.yaml), so this remains open.

Trigger: the 7B coder hitting tool-call reliability or correctness limits in practice.

2. Frontier coder on L40S in a second region

Run a full-precision 30B-class coder on a single L40S 48GB, which needs an instance family (g6e) not offered in eu-west-3. The platform’s OpenTofu stacks are pinned to eu-west-3 (opentofu/llm-platform/backend.tf) with no second-region stack, so this remains open — and would require a new OpenTofu stack, a new Karpenter NodePool, and cross-region routing from the AI Gateway.

Trigger: an AWQ-4bit quality compromise from path 1 becomes a measurable regression, or the team wants to demo full-context work a single L4 can’t hold.

3. Tensor-parallel `g6.12xlarge` (4× L4)

Run a 30B-class model with tensor-parallel-size: 4 on a single 4-GPU instance for full precision without a region split. The gpu-l4 NodePool explicitly excludes multi-GPU SKUs today (infrastructure/base/karpenter-nodepools-gpu/gpu-l4-nodepool.yaml, by design — a multi-GPU pod would otherwise be able to consume the entire 4-GPU fleet cap on its own), so this remains open and would require lifting that restriction along with revisiting the cap it protects.

Trigger: path 1’s quantized model isn’t enough, and multi-region operational cost (path 2) is the bigger problem.

4. Anthropic↔OpenAI relay for Claude Code

Deploy a translator sidecar exposing Anthropic-style /v1/messages and proxying to the existing OpenAI-compatible AI Gateway, so Claude Code can target the self-hosted fleet. Coding Clients documents this as explicitly not implemented — OpenCode covers the agentic-CLI use case today.

Honest framing, carried forward from the original proposal: this is a UX win wrapped around a quality compromise. Pointing Claude Code at an open-weights model doesn’t give Sonnet/Opus output — it gives that model’s output via Claude Code’s UX. Useful for sovereignty, privacy, or cost relief on bulk tasks; not for raising agentic coding quality.

Trigger: paths 1 or 2 close the open-weights/frontier gap enough that this becomes a competitive daily backend, or an explicit no-telemetry privacy workflow is the use case.

5. Heavier dense models (GLM-4.6, DeepSeek-Coder-V3)

Both require multi-GPU serving (TP=4+ or H100-class hardware) and had known vLLM tool-call parser quirks as of the original proposal. No GPU budget for H100/H200-class SKUs exists in this lab today, so this stays open pending both upstream parser stabilization and a hardware budget decision.

6. Per-tenant FinOps observability

Attribute token spend and cost per consumer by extracting a static x-tenant request header at the gateway and labelling the existing token counters with it. SPEC-006 (docs/specs/done/2026-Q3/006-genai-observability-envoy-gateway/) shipped the gateway’s gen_ai_* token metrics and base-vs-canary attribution — a real prerequisite — but no tenant label or x-tenant header extraction exists anywhere in infrastructure/base/envoy-ai-gateway/ or the LLM dashboards today. This path remains open on top of what SPEC-006 delivered.

What this is not: tenant authentication, quotas, fairness scheduling, or rate limiting — those stay out of scope for this platform’s posture.

Trigger: any real or simulated workload routes through the platform with multiple addressable consumers, including using LoRA adapter names as proxy “tenants” to demo cost attribution without standing up auth.

Coding Clients