Dashboards & Alerts

Grafana Operator

observability/base/grafana-operator/ installs grafana-operator (5.24.0), but it does not deploy its own Grafana. Its Grafana custom resource (grafana-victoriametrics.yaml) is configured as external, pointing at the Grafana subchart already bundled inside victoria-metrics-k8s-stack:

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana-victoriametrics
  labels:
    dashboards: "grafana"
spec:
  external:
    url: http://victoria-metrics-k8s-stack-grafana
    adminPassword:
      name: victoria-metrics-k8s-stack-grafana-envvars
      key: GF_SECURITY_ADMIN_PASSWORD

grafana-operator’s actual job here is reconciling GrafanaFolder, GrafanaDashboard, and GrafanaDatasource CRs against that one external instance. Every such CR in this repo carries the same two fields: instanceSelector.matchLabels.dashboards: "grafana" (matches the Grafana CR’s own label above) and allowCrossNamespaceImport: true (folders and dashboards are defined across apps, infrastructure, and observability namespaces, all targeting the one Grafana in observability).

Folder registry

A repo-wide grep for kind: GrafanaFolder finds nine CRs. Eight reconcile onto the cluster by default, each defined in the namespace that owns its dashboards; the ninth (llm, apps/base/ai/llm/grafana-folder.yaml) only applies once the opt-in LLM platform’s suspended umbrella Kustomization is resumed (see CLAUDE.md’s Self-Hosted LLM Platform section) and is out of scope for this always-on page:

Dashboard inventory

victoria-logs and victoria-traces also self-ship a dashboard via their own chart’s dashboards.enabled: true, grafanaOperator.enabled: true values, targeting the same logs/traces folders as the manually-authored ones above — two sourcing paths land in the same folder, which is redundant but not a conflict (different dashboard names).

The victoria-metrics-k8s-stack chart’s own defaultDashboards ships into an "observability" folder (not one of the GrafanaFolder CRs above — it’s created implicitly by the chart’s sidecar mechanism, not grafana-operator). Its Kubernetes-views duplicates of the kubernetes folder’s dashboards are explicitly disabled in vm-common-helm-values-configmap.yaml to avoid showing the same dashboard twice.

VictoriaTraces

observability/base/victoria-traces/ runs victoria-traces-single (0.1.11), 3-day retention, wired into Grafana as a Jaeger-protocol datasource:

# observability/base/victoria-traces/grafana-datasource.yaml
datasource:
  type: jaeger
  url: http://victoria-traces-vt-single-server.observability:10428/select/jaeger
  jsonData:
    tracesToLogs:
      datasourceName: VictoriaLogs
      tags: ['trace_id', 'traceId', 'traceID']
    tracesToMetrics:
      datasourceName: VictoriaMetrics
    nodeGraph:
      enabled: true

tracesToLogs/tracesToMetrics are what let a Grafana user pivot from a trace span straight to the matching log lines (via log.trace_id, per the LogsQL rules) or request-rate/latency panels, without re-typing a query by hand. The manifest doesn’t configure an explicit receiver protocol (no OTLP toggles) — VictoriaTraces’ chart defaults apply unmodified, so which ingest protocols are actually enabled isn’t determinable from this repo alone.

Alerting

Two VMRule flavors coexist under the same operator.victoriametrics.com/v1beta1 API: implicit PromQL rules (the default) and explicit type: vlogs rules evaluated as LogsQL — see loggen’s alert for the LogsQL form. PromQL rules live mostly in victoria-metrics-k8s-stack/vmrules/:

• karpenter.yaml — 3 alerts (node-registration failures, nodepool near capacity, cloud-provider errors) for a component this stack doesn’t own but scrapes.
• openbao.yaml — 7 alerts (down, sealed, no active node, lost/at-risk Raft voter, snapshot job failed/stale), extensively commented with the incident history that motivated each one (an expired cert going unnoticed for 9 months; a stale Raft voter silently halving failure tolerance).
• runlore.yaml — 12 alerts covering the agent’s own health (down, no leader, split-brain), pipeline behavior (dropped/stalled/erroring investigations), and cost (token spend, model latency). Each carries a runbook_url pointing at RunLore’s own docs. A byte-identical PrometheusRule twin ships alongside it so either metrics backend works; the file comments call out that the two must never drift independently.

# observability/base/victoria-metrics-k8s-stack/vmrules/runlore.yaml — trimmed
- alert: RunloreAgentDown
  expr: absent(runlore_build_info)
  for: 5m
  labels:
    severity: critical
  annotations:
    runbook_url: "https://github.com/Smana/runlore/blob/main/docs/observability.md#runloreagentdown"

Alertmanager routing

Alertmanager’s routing tree (vm-common-helm-values-configmap.yaml, applied to the victoria-metrics-k8s-stack alertmanager.spec.config) fans every non-blackholed alert to both RunLore and Slack:

route:
  receiver: "slack-monitoring"
  routes:
    - matchers:
        - alertname =~ "InfoInhibitor|Watchdog|KubeCPUOvercommit"
      receiver: "blackhole"
    - receiver: "runlore"
      continue: true   # falls through to the next route instead of stopping
    - receiver: "slack-monitoring"
receivers:
  - name: "runlore"
    webhook_configs:
      - url: "http://runlore.runlore.svc:8080/webhook/alertmanager"
        http_config:
          authorization:
            credentials_file: /etc/vm/secrets/runlore-webhook-token/token

RunLore’s webhook requires a bearer token (its v0.2.0+ fail-closed behavior — alert labels/annotations flow into an LLM prompt, so an unauthenticated trigger path was judged unacceptable) mirrored between its own runlore-webhook ExternalSecret and this stack’s runlore-webhook-token ExternalSecret. The Slack receiver’s actions: list defines five button blocks: Runbook, Query (the alert’s GeneratorURL), Dashboard, and Silence — each sourced from the firing alert’s own annotations — plus a fifth built from the Monzo template’s link_button_text/link_url helpers. Whether that fifth button actually renders on every message or only conditionally (e.g. when link_url is set) can’t be determined from this repo — the Monzo template it calls into isn’t vendored here, only referenced by name.

Grafana OnCall: built but not deployed

The directory is a complete, “external-everything” OnCall install per Grafana’s scalable Helm guide: the oncall chart (1.16.5) for the engine + Celery workers, a companion rabbitmq HelmRelease for the message broker, an SQLInstance claim for a CloudNativePG-backed Postgres database, and a KVStore claim (xplane-oncall, SPEC-012) for Valkey. Both oncall and oncall-rabbitmq HTTPRoutes parent to platform-tailscale-admin, unlike the general-gateway components covered elsewhere on this page.

Despite being undeployed, the active Grafana already carries a live dependency on it: victoria-metrics-k8s-stack’s common values enable the grafana-oncall-app plugin and provision it to call http://oncall-engine:8080/ — a Service that does not exist on this cluster. The plugin will show as installed but non-functional in Grafana until grafana-oncall is wired into a Kustomization.

Neither the oncall nor the rabbitmq HelmRelease defines a CiliumNetworkPolicy in this directory — oncall’s own networkPolicy.enabled value is false, and rabbitmq’s is a plain Kubernetes NetworkPolicy (chart comment: “To be replaced with cilium network policy”), not the constitution-mandated Cilium one. If this component is ever wired in, that gap needs closing first.

CiliumNetworkPolicy coverage

Of the nine component directories under observability/base/, only runlore defines a CiliumNetworkPolicy:

# observability/base/runlore/ciliumnetworkpolicy-ingress.yaml
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: runlore
  ingress:
    - fromEntities: [ingress]
      toPorts:
        - ports: [{port: "8080", protocol: TCP}]

It exists because runlore’s HTTPRoute parents to platform-public (described in Private Access), and this cluster’s shared cilium-envoy DaemonSet runs hostNetwork: true — Gateway-originated traffic lands as Cilium’s reserved ingress entity, which a namespaced Kubernetes NetworkPolicy can never match. grafana-operator, grafana-oncall, kubernetes-event-exporter, loggen, metrics-server, victoria-logs, victoria-metrics-k8s-stack, and victoria-traces have no CiliumNetworkPolicy in these directories, and no cluster-wide default-deny CiliumClusterwideNetworkPolicy covers the observability namespace either — the one such policy in this repo (infrastructure/base/gapi/allow-gateway-l7-proxy.yaml) only allows the Gateway API L7 proxy, it does not deny anything by default.