Cluster Upgrade Runbook for Small Engineering Teams

A cluster upgrade runbook for small engineering teams must fit what the team can do on its own, without a platform department standing by. The safe path is a sequence of minor-version hops, tested on a small node pool, with clear gates and a way back.

Why small teams face a different upgrade problem than the runbook assumes

A two- or three-person team cannot count on a staging fleet that mirrors production, a separate upgrade on-call rotation, or an engineer who has done this before. The same people keeping services running must find time to change the system underneath them. A runbook that assumes otherwise leaves its hardest decisions to the moment something breaks.

For this team, each gate must use evidence it can gather, and each rollback must be something it can execute. When time for upgrades is short, version debt is also harder to clear. For self-managed clusters, the upstream support date is the date, full stop. Kubernetes does not let the team skip minor versions to catch up, so sequencing governs the whole job.

The version skew policy and why it dictates the entire upgrade sequence

Kubernetes version skew policy defines the supported version relationships during a rolling upgrade. It is not optional. With kube-apiserver at version N, controller-manager and scheduler may run N or N-1, while kubelet may run from N down to N-3. Kubectl has a separate allowance of N±1. Upgrade the API server before the other cluster components, because a kubelet newer than its API server is explicitly unsupported.

Minor-version upgrades must be separate hops. A cluster moving from 1.34 to 1.37 must pass through 1.35 and 1.36, with the control plane leading the nodes. The kubelet’s three-minor allowance provides useful slack for small teams. It allows two control-plane hops without two full fleet-wide node rollouts in lockstep, reducing the work packed into an upgrade window without abandoning canary validation.

CircleCI’s outage involved incompatible kubelet and kube-proxy versions that corrupted iptables rules on every sync and required node-by-node restarts to recover. Managed services do not remove that responsibility. EKS, GKE, and AKS hide the control-plane upgrade behind a button or API call, but operators still own skew compliance, node drains, workload disruption budget calculations, and deprecation scans.

Pre-flight checks that catch the failures before they happen in production

Pre-flight is the highest-leverage phase because it catches failures before they reach the upgrade window. Start with removed and deprecated APIs. Use tools such as kubent or Pluto to scan live objects and Helm releases, and scan the source-of-truth manifests in Git too. What is in the live cluster does not show what CI will apply next week. Use kubectl convert to rewrite manifests to current API versions, and remove Secret or ConfigMap references from static pod manifests in staticPodPath before reaching 1.37.

Next, check that every node pool has cgroup v2. Since 1.35, failCgroupV1 defaults to true, so a kubelet on a cgroup v1 node refuses to initialize unless explicitly overridden in KubeletConfiguration. The override buys time for a lasting fix. Replace an old node image before the first hop to avoid discovering the problem during rotation.

Then review target-version compatibility for the CNI plugin, CSI drivers, ingress controller, service mesh, admission webhooks, and operators. Reddit’s 2023 CNI outage followed a master-to-control-plane label rename because add-on configurations still selected the old label. Labels deserve a direct check before every hop. Search CNI, ingress, and monitoring configurations for platform-owned labels, then run kubectl get pdb -A and resolve any zero values under ALLOWED DISRUPTIONS.

Before changing the control plane, snapshot etcd and verify that it can be restored. A backup never restored is still a hypothesis. Downgrades are unsupported, making the snapshot the control plane’s reverse gear, and kubeadm upgrades etcd as part of the sequence. If the restore procedure has never been exercised, rehearse it in a non-production cluster before the first hop.

Before the second hop, also scan for images that write to host-protected paths, init scripts that chown system directories, and Helm charts hardcoding runAsUser to 0 without a capabilities review. These checks look for assumptions that the security changes in 1.36 bring to light. Do not mix discovery with the production rollout. Completed pre-flight checks open the gate to the first hop.

Hop one: upgrading control plane and canary node pool from 1.34 to 1.35

For a kubeadm cluster, begin with kubeadm upgrade plan to see what is available and what will change. Upgrade the kubeadm package first. Run kubeadm upgrade apply v1.35.x on the first control-plane node, then use kubeadm upgrade node on the remaining control-plane nodes. The apply command rewrites static-pod manifests, restarting the API server, controller-manager, and scheduler on their new versions one by one. After each control-plane node, upgrade its kubelet and kubectl packages and restart kubelet.

Verify API server health, controller-manager leader election, and the scheduler before touching workers. Then move only one node pool to the 1.35 kubelet. Cordon each node, drain it while respecting PDBs, upgrade kubelet, and uncordon it. Watch workloads reschedule before draining the next node. Spreading rules govern placement, so confirm the workload is healthy on its new node before proceeding.

The canary must run a representative workload through a normal traffic cycle, not merely pass a smoke test. Define its bake window before starting the rotation. Watch pod start latency, OOM kills, and eviction events against the pre-upgrade baseline, since the effects of cgroup-related changes are visible in those signals. If a pod that was healthy fails to reach Running within that window, stop and find out why. Elevated OOM kills or evictions also block further rollout until the cgroup v2 configuration is understood.

For a team without staging, blue-green node pools provide a practical rollback path. Create a pool at the target kubelet version, cordon the old pool, and drain old nodes one at a time while validating application health. Keep the old pool until validation passes. To reverse the move, uncordon the old pool and drain the new one instead of attempting an unsupported kubelet downgrade. Use faster in-place rolling upgrades only for patch-level bumps.

A kubelet that refuses to initialize is another stop signal. Check its failCgroupV1 setting and node image version. Do not turn a temporary override into the permanent configuration. Delete the old pool only after the replacement has passed the canary gate and application health checks.

Hop two: control plane and node pool rotation from 1.35 to 1.36

For 1.36, follow the same control-plane-first sequence and check health before starting worker rotation. The canary structure stays the same. The risk now shifts toward security assumptions, as User Namespaces, Mutating Admission Policies, and Fine-Grained Kubelet API Authorization reached GA in 1.36. Pay particular attention to the host-path writes, system-directory ownership changes, and root-user settings identified during pre-flight.

Do the security-policy migration apart from the version upgrade. Turn on User Namespaces per workload on the first pass. Mutating Admission Policies can replace some webhook glue, removing a network hop and a failure mode from admission, but migrate those webhooks after the upgrade is stable. Making those changes at the same time makes it harder to judge the canary.

Run a representative workload on one upgraded pool through a normal traffic cycle. Add privileged pod behavior to the existing watch signals. An init container or sidecar that worked on 1.35 and fails on 1.36 points to an unreviewed security assumption.

Once validation passes, the cluster can stay on 1.36. It is a supported release with room ahead of it. Kubernetes 1.37, released August 26, 2026, brings changes that need their own pre-flight work. Stop at 1.36 and stabilize rather than chaining the third hop into the same window.

What changes if the target is 1.37

Before the third hop, fix any static-pod manifest issue that is still open. Kubernetes 1.37 prohibits those manifests from referencing Secrets or ConfigMaps, and the feature gate has been removed. There is no gate left to defer that restriction. The release shipped exactly one tracked deprecation, making the deprecation audit compact enough for an afternoon, but that one restriction can still block the upgrade.

Native HPA scale-to-zero also warrants an add-on audit. For common cases, the pattern of keeping one pod warm to watch a queue becomes optional, as may an event-driven-autoscaling add-on installed partly for that purpose. Check existing KEDA or similar configurations for overlap before the hop. Duplicate scaling logic can produce unexpected behavior. HPA CPU and memory autoscaling also reaches a stable API surface, but there is nothing else to do for that here.

Planning should extend beyond the immediate target. Put the 1.38 code freeze date on the calendar when its release cycle is announced. That creates a concrete window for reviewing breaking changes before the release ships.

Post-upgrade validation: the signals that confirm the cluster is actually healthy

Validation continues after uncordoning the last node ends the rotation. Kubernetes self-healing can hide failures that show up only under real traffic patterns. Each service therefore needs metrics, structured logs, traces, and liveness and readiness probes that show how it is doing. Define thresholds for latency, traffic, error rate, and saturation, then compare them with the pre-upgrade baseline. Watch p99 latency rather than averages, because cgroup and scheduling changes can appear in the tail first.

Use kubectl get nodes to confirm every node is Ready, at the expected version, and no longer cordoned. A node left cordoned is unfinished work. Run kubectl get pdb -A again to confirm the drain sequence has not left a degraded budget. Confirm that the CNI, CSI, and ingress controller are running their expected versions and passing their own health checks.

After the bake window, check pod start latency and OOM kill counts against the baseline. Elevated values still need investigation before declaring stability. Verify etcd health and keep the pre-upgrade snapshot accessible until the cluster has been stable for a full normal traffic cycle. Finally, run reconciliation against the baseline configuration in version control and fix any drift that arose during node rotation.

How the team stays current instead of accumulating upgrade debt

Upgrades every quarter make minor-version changes a task the team can repeat without having to catch up against a deadline. Keep track of the supported version range for every cluster. Upgrade before the current version reaches end of support. Moving from 1.35 to 1.36 to 1.37 over a sustainable rhythm is a different job from clearing all three hops under pressure.

Document centrally controlled settings, including Kubernetes version, networking, and admission policies, separately from service-team settings such as resource requests, autoscaling, and ConfigMaps. Keep the baseline in version control and use reconciliation to detect drift. Exceptions need a justification, an approver, and an expiry date. That record lets the team query documented settings during the next pre-flight.

Automation changes who executes the runbook. For a small team without a dedicated platform engineer, a managed PaaS in its own cloud account can handle cluster upgrades, CVE patching, and node pool rotation as an operational baseline. Direct cluster access preserves control-plane visibility and the ability to verify upgrade state, while supporting the team’s compliance posture. The sequence, gates, and rollback criteria remain the model for judging whether that automation worked.

The lasting lesson of the iximiuz Labs upgrade runbook is to keep the blast radius small, verify state between steps, and rehearse the reverse gear before it is needed.