Observability
observability/base/ holds nine component directories. Eight are wired
into Flux; one is not. grafana-oncall is a fully-built HelmRelease pair
(engine + RabbitMQ, external Postgres and Valkey already provisioned) that no
Kustomization anywhere in clusters/ or flux/ ever references — it does
not run on the cluster today. Every other component below does. See
Dashboards & Alerts
for what that means in practice.
| Component | Deployed via | Documented in |
|---|---|---|
victoria-metrics-k8s-stack | observability-victoria-metrics-k8s-stack Kustomization | Metrics |
metrics-server | observability Kustomization | Metrics |
victoria-logs | observability Kustomization | Logs |
kubernetes-event-exporter | observability Kustomization | Logs |
loggen | observability Kustomization | Logs |
victoria-traces | observability-victoria-traces Kustomization | Dashboards & Alerts |
grafana-operator | observability-grafana-operator Kustomization | Dashboards & Alerts |
runlore | observability Kustomization | Dashboards & Alerts |
grafana-oncall | not referenced anywhere | Dashboards & Alerts |
CloudNativePG’s own metrics, logs, backups, and dashboards are covered
separately in PostgreSQL:
that component lives under infrastructure/base/cloudnative-pg*/, not
observability/base/, but it feeds the same VictoriaMetrics/VictoriaLogs
stores and the same Grafana.
Why VictoriaMetrics and VictoriaLogs
Both are PromQL/LogsQL-compatible, single-binary-first alternatives to
Prometheus and Loki — existing dashboards and alerting rules written against
either wire protocol work unchanged, so adopting them cost nothing on that
front. The two products share one operator
(operator.victoriametrics.com/v1beta1 — VMRule, VMServiceScrape,
VMScrapeConfig, VMAlert) and one Grafana, which is what lets a single
VMAlert evaluate both PromQL rules against VictoriaMetrics and LogsQL rules
(type: vlogs) against VictoriaLogs from the same ruleSelector mechanism —
see loggen’s alert
for a concrete type: vlogs example. No ADR in this repository records the
comparison against Prometheus/Loki directly; treat this section as the
current rationale, not a linked decision record.
Single mode today, cluster mode standing by
Both VictoriaMetrics and VictoriaLogs ship two HelmReleases in their
component directories — helmrelease-vmsingle.yaml/helmrelease-vlsingle.yaml
(active) and helmrelease-vmcluster.yaml/helmrelease-vlcluster.yaml
(present on disk, commented out of the Kustomization). The cluster charts
are pre-configured (replication factor 2, HPA 2→10 on vlselect/vlinsert,
zone-aware anti-affinity) but nothing currently runs them — this cluster
runs single-node VictoriaMetrics (retentionPeriod: "1d", explicitly
commented “Minimal retention, for tests only”) and single-node VictoriaLogs.
Scale-out is a matter of un-commenting the four cluster-mode lines per
component’s kustomization.yaml, not a rewrite.