Giving It a Home: K3d, Kustomize & the Three Environments

A container is just a box sitting on your laptop until something decides where it runs, what happens when it crashes, and how it talks to the outside world. That something is Kubernetes, and today's post is about the slightly unglamorous decision I mentioned at the end of yesterday's post: building three separate environments before I had any real reason to need more than one.

First, the practical choice. I didn't reach for a managed cloud cluster to start with, and I didn't use Minikube either. I went with K3d, which runs a lightweight Kubernetes distribution called K3S inside Docker containers on your own machine. The reason I picked it over the more commonly recommended options is that it gets you closer to a real multi-node setup without any of the weight. My local cluster runs with one server node and two agent nodes, all inside Docker, and it comes with a built-in load balancer that I mapped straight to ports 8080 and 8443 on my machine. That meant I could practice deploying to something that actually behaves like a small production cluster — multiple nodes, real scheduling decisions, a real load balancer in front — entirely offline, and tear the whole thing down and rebuild it in under a minute whenever I wanted a clean slate.

Where Day 2 quietly pays off. Remember the /health and /status endpoints I built into the FastAPI backend, before Kubernetes was even in the picture? Kubernetes has a concept called probes, and it uses them to decide two genuinely different things: whether to restart a container, and whether to let it receive traffic. The liveness probe — the one that decides whether to restart the pod — points straight at /health. The readiness probe — the one that decides whether the pod is allowed to receive requests yet — points at /status. I didn't design those two endpoints with Kubernetes in mind at the time; I designed them because "alive" and "actually working" felt like two different questions worth answering separately. By the time I got here, that earlier instinct turned out to be exactly the distinction Kubernetes itself cares about.

It's worth slowing down on what actually happens when each of these fails, because the difference matters more than it sounds. If the liveness probe fails repeatedly, Kubernetes assumes the process is stuck or dead and kills the container, then starts a fresh one in its place — this is the mechanism behind that infamous CrashLoopBackOff status you'll see if something is wrong at a deeper level, because the new container keeps failing the same check and getting restarted again. If the readiness probe fails, nothing gets killed. The pod is simply pulled out of the service's pool of available endpoints — traffic stops being sent to it — until it starts passing the check again, at which point it's quietly added back. One failure mode is "something is broken, try again from scratch." The other is "this one's just not ready yet, give it a moment, don't send it to customers." Conflating the two, which is what happens if you only build one health endpoint and point both probes at it, means you either restart pods that didn't need restarting or keep sending traffic to pods that genuinely needed to be pulled out of rotation.

Where Day 3 quietly pays off, too. Day 3 was about getting the container itself to run as a non-root user. Kubernetes lets you go a layer further and enforce that same intent at the cluster level, regardless of what the image itself does: the deployment explicitly requires the container to run as a non-root user, pins it to a specific non-root user ID, blocks privilege escalation outright, and mounts the root filesystem as read-only so nothing inside the running container can quietly write to itself. None of this replaces what I did in the Dockerfile — it backs it up. If a future image build ever slipped and tried to run as root, the cluster itself would refuse to let that happen, independent of whatever the Dockerfile says.

Three guardrails I added before anything was actually under load. Beyond probes and the security context, the base manifest carries three more pieces that don't get talked about as much, and I think they deserve a mention because none of them was strictly necessary on a single-developer local cluster — I added them anyway, on principle.

The first is a horizontal pod autoscaler, set to scale between 2 and 10 replicas based on CPU and memory utilisation. What I find more interesting than the numbers themselves is the asymmetry in how it scales: scaling up happens fast — within a 60-second window, it can add two pods at a time — but scaling down is deliberately slow, waiting a full five minutes before removing even a single pod, and only one at a time after that. That asymmetry is intentional. If traffic spikes, you want capacity immediately; being slow to react to load is how outages happen. But if you scale down just as aggressively the moment load dips, you end up in a flapping cycle — scale down, traffic ticks back up, scale up again, repeat — which wastes more resources reacting to noise than it would have cost to just stay slightly over-provisioned for a few minutes.

The second is a pod disruption budget, which tells Kubernetes that at least one pod must always stay available, no matter what. This isn't protection against crashes — it's protection against Kubernetes' own routine maintenance. When a node gets drained for an upgrade, or the cluster itself needs maintenance, Kubernetes is otherwise free to take down every pod of a deployment at once if it feels like it. A pod disruption budget puts a floor under that.

The third is a network policy, and this is the one I'd call genuinely forward-thinking for a project at this stage. By default, every pod in a Kubernetes cluster can talk to every other pod, in every namespace, with no restrictions at all — which is convenient and also exactly the kind of default that turns one compromised pod into a clear path to everything else. The network policy I wrote flips that: it only allows incoming traffic on the app's port from the dev namespace itself and from a monitoring namespace, and it only allows outgoing traffic to that same monitoring namespace plus DNS lookups. Everything else, in or out, is denied by default. At the time I wrote this, there was no monitoring namespace actually running yet — that comes later in this series. But the policy already assumed it would exist and already drew the boundary around it, which meant when the observability stack did eventually show up, it had a door already cut for it instead of a wall to get through.

Now, the three environments. On a single local K3d cluster, with nobody depending on this except me, there was no operational reason yet to separate dev, staging, and production. I built that separation anyway, using Kustomize, which lets you define one shared base manifest — the deployment, the service, the ingress, the autoscaler, the disruption budget, the network policy, all of it — and then layer small, environment-specific patches on top instead of maintaining three almost-identical copies of the same set of files.

Each environment patches only what's actually supposed to differ. Dev runs a single replica with the smallest resource budget, pulls its image only from what's already sitting locally on the machine, logs at debug level so I can see everything happening, and gets its own sentinelai-dev namespace with a dev- prefix on every resource so there's no chance of confusing it with anything else. Staging runs two replicas with double the resource budget and logs at a quieter info level — a deliberate middle ground. Production runs three replicas, the largest resource budget of the three, and drops the log level down to warnings only, because a production system logging every debug line is just noise you'll pay to store and never read.

Why bother with this when there's only one cluster and one person using it? Because the moment I actually needed a real staging and production split — which happens a few posts from now, once this moves onto AWS — I wanted the environment boundary to already be a solved problem, not something I'd be retrofitting under pressure. This is the same discipline from Day 2, applied one layer up: design the structure you'll eventually need before you're forced to need it urgently. Three namespaces on a laptop cost me almost nothing. Discovering I needed three namespaces while already mid-migration to a real cloud cluster would have cost a lot more.

The last piece worth mentioning is how traffic actually reaches any of this. A ClusterIP service sits in front of the pods, and a Traefik-routed ingress sits in front of that, accepting traffic on the standard web entry point and forwarding it through to whichever pod is currently marked ready. None of it is exotic — it's the same load balancer K3d already gave me for free, just pointed at the right place.

If Day 2 was about designing contracts before building intelligence, Day 4 is about designing structure and boundaries before there's any pressure forcing you to. Three environments, one shared definition, an autoscaler that's fast to react and slow to retreat, a disruption budget that protects against the cluster's own maintenance, and a network policy that drew a door for a monitoring system that didn't exist yet — all sitting quietly underneath a project that, from the outside, still just looked like one FastAPI service running in a container.

Tomorrow's post is a shorter one — about something that has nothing to do with containers or clusters, and everything to do with whether a project looks like it was built by someone who's worked on a team before.