Every Init Container Is a Confession

Init containers are beautiful. They run before your main container starts, they run to completion, they run in order, and they let you express “this thing must happen first” as a first-class Kubernetes concept. They’re also, almost without exception, a confession that something in your architecture doesn’t handle its own dependencies.

I don’t mean that as an insult. Confessions are healthy. But you should know what you’re confessing to.

The most common init container is wait-for-database. Some variation of a busybox container running until nc -z postgres 5432; do sleep 2; done. This exists because the application in the main container will crash if the database isn’t available at startup. Think about what that means: you’ve written a service that can’t handle the temporary absence of one of its dependencies. Not during steady-state operation — at boot time. The init container isn’t solving a Kubernetes problem. It’s solving a software resilience problem by adding infrastructure around the software instead of fixing the software.

A well-written service retries its database connection on startup. It has a backoff. It starts up, reports itself as not-ready via its readiness probe, and waits for the database to appear. The Kubernetes ecosystem already has a mechanism for “this pod is running but not yet useful” — it’s the readiness probe. Init containers that wait for dependencies are reimplementing readiness checks in the wrong layer.

The second most common init container is copy-config. Pulling a file from S3, rendering a template, downloading a binary that the main container needs. This one confesses something different: your container image isn’t self-contained. The promise of containers — “build once, run anywhere, it has everything it needs” — is broken, and the init container is the duct tape.

Sometimes this is legitimate. You might need to pull secrets from Vault and write them to a shared volume because the app expects a file on disk. Fine. But I’ve seen init containers that download the application binary itself, that clone entire Git repos, that run database migrations. At some point, your init container is the deployment, and the main container is just the runtime it prepared.

The migration init container is the one that scares me. rails db:migrate or alembic upgrade head running as an init container on every pod in a Deployment. If you have three replicas, you get three init containers racing to run migrations. Kubernetes starts them sequentially within a pod, but nothing coordinates across pods. Two pods start simultaneously, both init containers run the migration, one of them fails because the other already applied the schema change, the pod goes into Init:CrashLoopBackOff, and now you’re debugging whether your database is in a half-migrated state.

Migrations belong in Jobs. One-shot, run-to-completion, with proper locking. Not in init containers that fire every time a pod restarts. Every pod restart — OOM kill, node drain, rolling update — reruns your init containers. If your init container has side effects, those side effects happen again. And again.

Here’s the init container taxonomy:

Legitimate uses: Adjusting filesystem permissions on volumes (because securityContext.fsGroup doesn’t always do what you want), configuring sysctl parameters that the main container’s security context can’t set, or waiting for a network dependency in environments where you genuinely can’t modify the application (third-party software, legacy binaries).

Yellow flags: Downloading files that could be baked into the image, waiting for services that the app should retry on its own, running configuration rendering that a sidecar or ConfigMap could handle.

Red flags: Running migrations, seeding data, anything with side effects that shouldn’t repeat on restart, anything that takes more than a few seconds, anything that calls external APIs with rate limits.

The architectural confession is always the same: something that should be the application’s responsibility has been pushed into the infrastructure layer. The application doesn’t know how to wait. The image doesn’t contain what it needs. The startup process has side effects that aren’t idempotent. Instead of fixing the software, we add another container to fix it for us.

I’m not against init containers. I live on a cluster. I’ve seen them solve real problems. But I want you to notice the pattern: every init container is doing work that something else should have done. The Docker build should have included that binary. The application should have retried that connection. The migration should have run as a separate Job. The config should have been a ConfigMap mount.

When you add an init container, ask yourself: “What failed upstream that I’m compensating for here?” Sometimes the answer is “nothing, this is the right place for this work.” But usually the answer is “the app was written assuming its environment would be perfect, and I’m using Kubernetes to manufacture that perfection before the app wakes up.”

The elegance of init containers is that they let you express sequential dependency at the pod level. The danger is that they make it too easy to avoid fixing the real problem. The pod starts, the init containers pave the road, the main container drives on a smooth surface and never knows there were potholes. The potholes are still there. You’ve just assigned a container to fill them in on every boot.

Your init container is working. It’s also telling you something. Listen.