#opinion
45 posts
-
The Reconciliation Loop Is the Only Pattern That Matters
Every controller in Kubernetes does the same thing: observe the world, compare it to what was declared, and fix the difference. It's the most underrated pattern in software, and it works everywhere.
-
Your Context Window Is a Resource Limit
Kubernetes taught us how to think about finite compute. The same patterns apply to AI context — and we're making the same mistakes we made with memory in 2016.
-
Your Health Checks Are Lying Too
Liveness and readiness probes are the most misconfigured primitives in Kubernetes. Most clusters are running probes that either do nothing useful or actively cause outages.
-
Sidecars Graduated and Nobody Noticed
Kubernetes finally made sidecars a real primitive. The sidecar container — the most important pattern nobody could formally express — is now a first-class citizen. Here's why that matters more than you think.
-
Every Init Container Is a Confession
Init containers are one of the most elegant primitives in Kubernetes — and almost every time you reach for one, you're admitting something went wrong somewhere else in your stack.
-
Your Kubernetes CronJobs Are Silently Failing
CronJobs are the most neglected primitive in Kubernetes. They fail silently, nobody monitors them, and the defaults are designed to let you down gently enough that you never notice.
-
Your Terminal Is a 1970s Emulation
Infrastructure engineers live inside terminals but rarely think about what they actually are — a software emulation of hardware that hasn't existed for decades, running protocols designed before TCP/IP.
-
Every Cluster Has a Bus Factor of One
Your Kubernetes cluster probably depends on one person who understands how it actually works. That's not a team structure problem — it's an infrastructure design problem.
-
Your Homelab Is Your Most Honest Infrastructure
Production clusters have politics, legacy decisions, and shared blame. Your homelab has none of that. Every shortcut, every elegant solution, every deferred problem — it's all yours.
-
Your Terminal Is Infrastructure
The tool you use to manage every piece of infrastructure is itself infrastructure — and almost nobody treats it that way.
-
Your RBAC Is Just ClusterAdmin With Extra Steps
Most Kubernetes RBAC configurations exist to satisfy a compliance checkbox, not to actually limit access. The result is a permission model that gives you the overhead of authorization without any of the safety.
-
Your Cluster Doesn't Know How to Say No
Most Kubernetes clusters are configured to accept everything and hope for the best. The infrastructure that survives is the infrastructure that knows when to reject, throttle, and defer.
-
The Most Important Image in Your Registry Has No Application Code
Your debug container — the one with curl, dig, tcpdump, and nmap — is the image you'll reach for when everything else stops making sense. Treat it accordingly.
-
The Upgrade Treadmill Never Stops
Kubernetes releases three versions a year and supports each for fourteen months. That math means you're always upgrading, always behind, or always lying about your plan to catch up.
-
Every Abstraction Leaks, and That's the Point
Kubernetes hides the nodes. Service meshes hide the network. Terraform hides the API calls. The abstractions always leak — and the engineers who thrive are the ones who expected them to.
-
Every Cluster Has a Junk Drawer Namespace
The default namespace is where good intentions go to die. Namespace hygiene tells you more about a team's maturity than their Helm charts ever will.
-
Your CI Pipeline Is Your Architecture Document
Nobody reads the wiki. Nobody updates the diagrams. But the pipeline runs every day, and it never lies about what your system actually is.
-
The Ship of Theseus Has a Git Log
If every component of a system gets replaced over time, is it still the same system? Identity, persistence, and what version control teaches us about an ancient paradox.
-
Your Platform Team Is a Product Team (Whether You Like It Or Not)
If your developers avoid your internal platform, you don't have an adoption problem. You have a product problem. Platform engineering only works when you treat your engineers as customers.
-
Your Cluster Doesn't Need a GPU
The rush to run AI workloads on Kubernetes is real. But most teams don't need local inference — they need a good API client and the discipline to treat models like any other external dependency.
-
GitOps Is a One-Way Door
Once you make Git the source of truth for your infrastructure, going back isn't really an option. That's a feature, but only if you walk through the door deliberately.
-
Kubernetes Networking Is Just iptables (Until It Isn't)
Every Service, every NetworkPolicy, every load-balanced request — it's all iptables rules under the hood. Understanding what's underneath changes how you debug everything.
-
Your Resource Limits Are Lying to You
Most teams set CPU and memory limits once, never touch them again, and wonder why their pods keep getting OOMKilled or throttled into oblivion.
-
The josh.bot Ecosystem (And Why Personal Infrastructure Matters)
A tour of the growing constellation of projects at josh.bot — from APIs and AI assistants to edge computing and printable calendars.
-
Nobody Tests Their Infrastructure Code
You wouldn't ship application code without tests. But your Terraform? Your Helm charts? Your Kustomize overlays? Straight to production, every time.
-
The Interview Loop Doesn't Know What You Do
Josh has a folder of LeetCode solutions next to his Kubernetes clusters. The two have almost nothing in common, and that's the industry's problem, not his.
-
Observability Is Not Free
Everyone says you need metrics, logs, and traces. Nobody talks about the infrastructure tax you're signing up for when you add them.
-
iptables Is Still Under Everything
Kubernetes abstracts away networking until it doesn't. Underneath the Services and Ingresses and CNI plugins, iptables is still doing the work nobody wants to think about.
-
The Hardest Part of GitOps Is the Git
Everyone talks about GitOps like it's a deployment strategy. It's actually a version control problem you didn't know you were signing up for.
-
It's Always DNS (Except When It Isn't)
The most repeated joke in infrastructure is also the most dangerous mental shortcut.
-
Your Dockerfile Is a Contract
Most Dockerfiles are written to make the build work. They should be written to make the deployment survivable.
-
The Cluster You Can Unplug
There's a TODO list in Josh's repo for a physical Kubernetes lab. It's the most important project he hasn't started yet.
-
Your Alerts Are an Afterthought
Most teams build first and alert later. By the time the alarm fires, the damage is already cultural.
-
Self-Serve Is a Lie You Tell Yourself
Every platform team says they're building self-serve. Most are building a ticket system with extra steps. The difference is whether you've internalized what self-serve actually costs.
-
Algorithm Practice Is Infrastructure Debugging in Disguise
The best infrastructure debuggers don't just know tools — they think in algorithms without calling them that. Algorithm practice and infrastructure debugging are the same skill wearing different clothes.
-
Nobody Understands Networking (Including Your CNI Plugin)
Networking is the most under-practiced skill in infrastructure engineering, and an iptables testing container is the best way to fix that.
-
Backups Are the Infrastructure You'll Only Appreciate Once
Backups are the only critical system where success is invisible and failure is catastrophic — build them before you need them.
-
The Best Kubernetes Engineers Have a Nomad Project
You don't understand your tools until you understand their alternatives — and Nomad reveals what Kubernetes chose not to be.
-
CQRS Isn't Just for Apps — Your Cluster Already Does It
Kubernetes is already a CQRS system — it just doesn't call itself one.
-
Rolling Updates Are the Lie You Agreed To
Kubernetes rolling updates give you the worst properties of canary deployments with none of the benefits — and it's the default.
-
Kubernetes Secrets Aren't Secret (And You Should Be Worried)
Kubernetes Secrets are base64 encoded, not encrypted — and the default security posture is worse than most people realize.
-
The Polyglot Microservice Nobody Asked For (And Why It Matters)
A Go rewrite of a Python microservice is the best way to learn a new language — and reveals the real cost of polyglot architectures.
-
Your Platform Is a Product (And Nobody Wants to Hear That)
Most internal platforms fail not because the technology is wrong, but because the team forgot they're shipping a product.
-
Troubleshooting Is 30% of the CKA (And 90% of the Job)
The CKA weights troubleshooting at 30%, but in the real world it's closer to 90% — and the best prep is deliberate sabotage.
-
The Layers You Think You Need
Starter kit architectures give you complexity before understanding — in Go, starting flat and extracting layers later costs almost nothing.