Writing
Flight Patterns

Part IV: Watching the Flock

At the end of Part III, pushing to main builds images, updates manifests, and FluxCD rolls out new pods. The deployment pipeline works. But "works" is a claim you can only make when you can see what's happening—and for the first few weeks, I couldn't.

When a pod crashlooped, I SSH'd into the VM and ran kubectl describe pod. When the API was slow, I checked the logs. When the logs were gone because the pod had restarted, I shrugged. I told myself I'd add dashboards later, twice, before I actually did.

Choosing a Stack

Managed observability. Datadog, Grafana Cloud, New Relic. Pay per metric series, pay per GB of logs ingested. Easy to set up, hard to predict the bill. For a personal project with twenty pods, the free tiers might cover it until they don't.

Lightweight self-hosted. VictoriaMetrics for metrics, something like Grafana Alloy for logs. Fewer moving parts, less ecosystem integration. You trade the standard kube-prometheus tooling for simplicity.

kube-prometheus-stack. The default answer for Kubernetes monitoring. Prometheus, Grafana, Alertmanager, node exporters, kube-state-metrics—all wired together with ServiceMonitors and recording rules. Well-documented, widely used, designed for exactly this.

I went with the standard stack because I wanted to learn the standard stack. Same reasoning as FluxCD in Part III—this project is a sandbox, and learning the thing everyone uses matters more than optimizing for my specific case. The full setup is four components: Prometheus scrapes metrics, Grafana renders dashboards, Loki aggregates logs, and Promtail ships them there. Total resource requests: roughly 275 millicores of CPU and 1.3Gi of memory. Less than most Java applications at idle.

The ServiceMonitor Trap

If you deploy a service with a /metrics endpoint and a ServiceMonitor, and Prometheus doesn't scrape it, and you spend an hour checking ports and paths and namespaces—it's the label.

The kube-prometheus-stack is configured to only discover ServiceMonitors with a specific label:

prometheus:
prometheusSpec:
  serviceMonitorSelector:
    matchLabels:
      release: monitoring

Your ServiceMonitor needs release: monitoring in its metadata labels:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: api-service
labels:
  release: monitoring  # this line cost me two hours
spec:
selector:
  matchLabels:
    app: api-service
endpoints:
  - port: http
    path: /metrics

It's documented somewhere in the chart's several-thousand-line values.yaml. You will not find it when you need it. I'm putting this near the top of the post because I wish someone had put it near the top of a post for me.

Loki on a Single Node

Loki v2 worked fine for basic log queries. But Grafana's log volume panel—the histogram above your results that shows log frequency over time—returned empty. The logs were there; the aggregation wasn't. Useful when something starts throwing errors at 3x the normal rate and you want to see when it started. Useless when the panel is blank.

Loki v3 rewrote the storage engine around TSDB. The migration:

loki:
schemaConfig:
  configs:
    - from: "2024-01-01"
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
storage:
  type: filesystem
compactor:
  retention_enabled: true
limits_config:
  retention_period: 168h  # 7 days

SingleBinary deployment mode. Filesystem storage on the local SSD. Seven-day retention with compactor cleanup. Distributed Loki—separate read, write, and backend processes—is architectural theater on a single node. SingleBinary is one process, one PVC. Disable the chunk and result caches too; they eat memory for no measurable benefit when the filesystem is local and fast.

After the migration, the log volume panel just worked. The PVC stays around 2-3Gi of its allocated 10Gi.

What You Actually Look At

Grafana is exposed at o11y.crowprose.com through the Cloudflare Tunnel from Part II. A dashboard on the internet needs authentication. I already had Google OAuth configured for the API gateway, so I reused the same credentials—auto_login: true skips the Grafana login page, allowed_emails is a one-item list. Anyone else gets authenticated by Google and promptly rejected by Grafana.

On a single node, most default Prometheus alerts are noise. I don't care about pod scheduling fairness or resource quota warnings. I care about crash loops, disk pressure, and memory pressure:

additionalPrometheusRulesMap:
custom-alerts:
  groups:
    - name: critical
      rules:
        - alert: PodCrashLooping
          expr: rate(kube_pod_container_status_restarts_total[15m]) > 0.2
          for: 5m
        - alert: PVCAlmostFull
          expr: >
            kubelet_volume_stats_used_bytes
            / kubelet_volume_stats_capacity_bytes > 0.85
          for: 5m
    - name: warning
      rules:
        - alert: HighMemoryUsage
          expr: >
            (1 - (node_memory_MemAvailable_bytes
            / node_memory_MemTotal_bytes)) > 0.85
          for: 10m

Alertmanager is disabled. The alerts fire into the void. I check Grafana daily, which is monitoring the way a doctor checks vitals: manually, on a schedule, and with the quiet hope that nothing interesting has happened. Setting up email notifications is on my list, right next to "add proper health checks" and "write integration tests." The list grows faster than it shrinks.

Rough Edges

| Component | CPU Request | Memory Request | Storage | |-----------|------------|---------------|---------| | Prometheus | 100m | 512Mi | 25Gi | | Grafana | 50m | 128Mi | 5Gi | | Loki | 50m | 128Mi | 10Gi | | Promtail | 20m | 64Mi | — | | Node Exporter | ~10m | ~20Mi | — | | Kube State Metrics | ~10m | ~20Mi | — | | Total | ~275m | ~1.3Gi | 40Gi |

Full metrics, log aggregation, dashboards, and alerting rules for 275 millicores and 1.3 gigs of memory. On a managed provider, the equivalent is three SaaS subscriptions billing by ingested data volume. Here, it's some YAML and a fraction of a CPU core.

No alerting pipeline. Monitoring you have to remember to check is monitoring that will eventually fail you. The crash loop alert is useless if nobody sees it fire. I know this. I still haven't set up notifications.

Observability depends on the thing it observes. If the node goes down, Grafana goes with it—along with the ability to see why it went down. On a managed cluster, your monitoring lives elsewhere. Here, everything is on one machine. The VM's btrfs snapshots (Part I) are the closest thing to a black box recorder, and I haven't automated those yet either.

Prometheus retention vs disk. 25Gi for metrics that mostly show flat lines. The default 15-day retention means two weeks of history—enough to spot trends, not enough for seasonal patterns. But you only need deep history when something goes wrong, and when something goes wrong on a personal project, it's usually obvious within hours.

Is this over-engineered for a personal website? Obviously. But there's something satisfying about watching the metrics of a system you built from the VM up. Even when the most exciting event is a brief CPU spike from a blog post redeploy. Boring monitoring is the best kind.