Work example 1
Kubernetes on AWS: half the nodes, restarts only in an agreed window
Challenge
Several EKS clusters across AWS accounts had grown organically: fixed node groups sized for peak load, on-demand instances everywhere, batch jobs and long-running services on the same nodes. One cluster requested about 60 % of its CPU while using about 10 %. Any automatic node consolidation restarted services at random times.
Approach
Moved the clusters to Karpenter with consolidation of under-used nodes, separated batch jobs into their own node pools protected from eviction by policy, and added spot pools that service owners opt into with a toleration. The first consolidation run produced a node re-creation loop; after root-cause analysis, consolidation and node rotation were limited to one agreed window per day, with alerting rules on node age. Every change went through Terraform/Terragrunt and reviewed merge requests, rolled out cluster by cluster with verification after each.
Result
Node count down by more than half, node cost reduced accordingly. Services restart only inside the window; consolidation-related incidents stopped.