Autoscaling & GPUs

Scaling a GPU workload is not scaling a web service. A cold start is 30–90 seconds of model load and CUDA-graph compilation, a replica costs an entire L4, and by the time a lagging signal like request count has moved, the users who caused it have already waited. Everything below follows from that.

Three triggers, OR-combined

One KEDA ScaledObject per model, always rendered, with three Prometheus triggers evaluated against VictoriaMetrics — any one at or above threshold drives a scale-up. Verified against the pinned KCL module (apis/inferenceservice/kcl/main.k in Smana/crossplane-configuration):

All three are leading signals: they fire before the batch saturates, before the cache evicts, before the queue builds. And max() rather than an average is deliberate — it tracks the hottest replica instead of a fleet mean that would hide it behind idle capacity.

Defaults: pollingInterval: 15s, cooldownPeriod: 300s.

Trigger 1’s denominator is the claim’s own model.maxNumSeqs. That field is not only a memory setting: raising it to fit more concurrent requests also raises the bar for a scale-up, because saturation is measured as a fraction of it.

Why minReplicas is never zero

Scale-to-zero on a Prometheus-driven autoscaler is a deadlock, not a trade-off: at zero replicas there is no pod, so there is no vllm:num_requests_running series, so no trigger can fire, so nothing ever scales back up. The metric that would wake the model only exists once the model is already awake.

That deadlock was first worked around with the KEDA HTTP add-on, which put a proxy in the data path and scaled on a lagging request count. It is no longer used. What replaced it is simply minReplicas: 1 plus leading signals — the deadlock disappears once a replica is always present, and the AI Gateway routes directly to each vLLM Service with no scaling proxy in front of the fleet.

The cost is explicit rather than hidden: four models at min=1 hold four GPUs continuously, whether or not anyone sends a request.

The GPU foundation

Karpenter NodePool gpu-l4:

• Single-GPU g6 instances only. The NodePool explicitly excludes multi-GPU SKUs such as g6.12xlarge and g6.48xlarge, so one claim can never consume the whole GPU budget.
• Spot-first, with the reclaim consequence below.
• Bottlerocket NVIDIA AMI. There is no NVIDIA device-plugin DaemonSet — the Bottlerocket NVIDIA variant advertises nvidia.com/gpu through the kubelet natively.
• Capped at nvidia.com/gpu: "4" across the whole pool.
• consolidationPolicy: WhenEmpty, not the default WhenEmptyOrUnderutilized. An always-warm model serving one request a minute looks underutilised to Karpenter, and consolidating its node would trigger exactly the cold start minReplicas: 1 exists to avoid.

Spot reclaim can still force a 30–90s multi-AZ PVC re-attach. That is an accepted trade for a personal platform and would not be for a team tier.

Weights on shared storage

An Amazon S3 Files filesystem — POSIX semantics over S3 — is mounted RWX at /models by both the preload Job and the serving pod, each under its own subPath.

The consequence worth understanding: the serving pod needs no IAM identity at all. Weights arrive over the CSI mount rather than being fetched by the process, so the serving ServiceAccount carries no role, and only the bounded preload Job holds an EKS Pod Identity. That is also why the serving pod’s egress policy can be kube-dns only, while world:443 is granted solely to the short-lived preload Job — the runtime pod cannot reach HuggingFace even in principle.

ADR-0004 records why this replaced an init-container sync, and ADR-0003 why vLLM Production Stack was chosen over KServe + llm-d.