Today I Learned
Raw discoveries, in the moment.
8 things learned
-
kube-proxy is just iptables management
kube-proxy doesn't actually proxy anything in iptables mode — it just writes NAT rules and walks away. The kernel does the real work. Name is a legacy artifact from the userspace proxy days.
-
DROP vs REJECT in iptables
DROP silently discards packets (client sees a timeout). REJECT sends back an ICMP error (client sees immediate connection refused). In production, this is the difference between "the service is slow" and "the service is down" — and they trigger completely different incident responses.
-
Kubernetes rolling updates default to mixed-version traffic
Default RollingUpdate strategy uses maxUnavailable: 25% and maxSurge: 25%. This means old and new pods serve traffic simultaneously with no control over version routing. Setting maxUnavailable: 0 and maxSurge: 1 gives you actual sequential rolling behavior.
-
Kubernetes is a CQRS system
API server is the command bus, etcd is the write model, watch API is the event stream, controllers are projection handlers. Informer caches are read models. The reconciliation loop is just event-driven CQRS with YAML characteristics.
-
LiteLLM is the open-source OpenRouter
BerriAI/litellm is a proxy server that presents a unified OpenAI-compatible API and routes to 100+ LLM providers. Virtual keys, cost tracking, load balancing, rate limiting built in. Could replace OpenRouter for self-hosted setups.
-
The safety marker pattern
From Josh's auroracloner: refuse to delete any resource unless its name contains a specific safety word (e.g. "clone"). Simple string check, but it prevents the 2 AM production deletion. Make the safe path the default path — correct the name if the marker is missing, dont just reject.
-
Kubernetes Secrets are just base64
Not encrypted by default. Stored plaintext in etcd. Visible via kubectl describe pod if mounted as env vars. Fix: enable EncryptionConfiguration with aescbc/secretbox, use External Secrets Operator for real secrets management, and lock down RBAC on Secret read access.
-
Saga choreography pattern
The saga participants are independent service