Reyan's Cloudy ThoughtsEKS Auto Mode: What You Need to Know Before Making the Move
Last December at re:Invent 2024, AWS announced a new mode for managing Amazon EKS clusters: Amazon EKS Auto Mode. AWS promises that this mode simplifies cluster operations, improves the performance, availability, and security of applications, and continuously optimizes compute costs.
In this article, we’ll explore how this new management mode compares to other methods for managing nodes. This will help you assess whether it makes sense to adopt this feature to manage your cluster.
Dynamic Compute Scaling with Karpenter and Amazon EKS Auto Mode
Orchestrating containers with Amazon EKS can become challenging when you need to deploy and manage a large number of containers while ensuring scalability and optimizing resources and costs.
You might also face challenges related to isolating workloads between teams and nodes. For example, you may need to dedicate certain instance types to specific workloads.
Karpenter, an open-source tool initially developed by AWS, addresses these challenges by allowing you to customize scaling policies and node assignments for your Amazon EKS workloads.
However, using Karpenter introduces new challenges. You now need to manage the lifecycle of a new component and ensure its maintenance, as it becomes critical for cluster administration.
To deploy Karpenter controllers, we usually recommend isolating them from other workload nodes. One method could be to deploy them on dedicated EC2 instances or even Fargate nodes. This ensures the consistent assignment of a controller to a compute instance.
Amazon EKS with EC2 instances cluster architecture without Auto Mode
Source: AWS re:Invent 2024, Session: “Simplify Kubernetes workloads with Karpenter & Amazon Amazon EKS Auto Mode (KUB312)”
With Amazon EKS Auto Mode, we shift to a model where the customer is no longer responsible for managing the lifecycle of the Karpenter controller (as well as Storage and Networking controllers).
In practice, this means that when starting a cluster with Amazon EKS Auto Mode with only the Control Plane deployed, you only need to initiate a deployment to deploy pods in the cluster.
The nodes are then automatically deployed based on the Node Pools defined beforehand. As a reminder, a Node Pool defines the strategy for assigning pods to a pool of instance types, which can be configured in the settings.
Amazon EKS with EC2 instances cluster architecture with Auto-Mode
Source: AWS re:Invent 2024, Session: “Simplify Kubernetes workloads with Karpenter & Amazon EKS Auto Mode (KUB312)”
Furthermore, when creating an EKS Auto Mode cluster, two types of Node Pools are offered as options: “system” and “general-purpose”.
The “system” Node Pool allows you to provision instances for deploying custom add-ons that you want to isolate from the instances used for your workloads.
The “general-purpose” Node Pool simplifies the getting started experience and allows you to easily deploy pods on general-purpose on-demand instances.
Of course, you can go beyond these ones to create your own Node Pools that meet your specific needs. For example, you can define Node Pools that prioritize spot instances to optimize costs or GPU instances for AI workloads.
To preconfigure the instances created in these cases, you use a Node Class object specific to EKS Auto Mode. This allows you to define security groups, subnet IDs, storage configuration, among other things. However, it’s important to note that you cannot specify an AMI ID.
This is because a key feature of Auto Mode is the automated upgrade management of the cluster, which relies on updating the AMIs used by the data plane nodes.
These nodes run with an AMI enforced by AWS based on Bottlerocket by default (an OS specifically designed by AWS for running containers). Therefore, you cannot use your custom AMIs in this case.
Automated Cluster Upgrades
To delve into the details of the automatic cluster update feature, it’s important to remember that for the Control Plane, you always have the option to trigger the Control Plane upgrade. Then only Amazon EKS checks the managed controller versions and updates only the controllers that are not up to date.
Source: AWS re:Invent 2024, Session: “Simplify Kubernetes workloads with Karpenter & Amazon EKS Auto Mode (KUB312)”
For Data Plane nodes, the upgrade process can be triggered in two scenarios.
- When the Control Plane is upgraded to a new Kubernetes version, the Data Plane nodes are replaced with an AMI that matches the Kubernetes version of the Control Plane
Source: AWS re:Invent 2024, Session: “Simplify Kubernetes workloads with Karpenter & Amazon EKS Auto Mode (KUB312)”
- When a new AMI is released and incorporates the latest OS security patches, the data plane nodes are replaced with the updated AMI.
Source: AWS re:Invent 2024, Session: “Simplify Kubernetes workloads with Karpenter & Amazon EKS Auto Mode (KUB312)”
These processes respect the lifecycle of nodes managed by Karpenter and the disruption budget, which you’ll need to define carefully to avoid unwanted downtime. It’s also important to keep in mind that if nodes are not updated despite the various events affecting Karpenter instances, AWS forces the update on the 21st of each month.
Pricing and Cost Considerations
Regarding Control Plane costs, there’s no change. Amazon EKS Auto Mode follows the same pricing model, which is a fixed price of $0.10/hour, or $72/month for standard support ($0.60/hour for extended support).
However, for the Data Plane, the pricing is quite different. There’s an additional cost applied to EC2 instances, which varies based on the instance type launched. Based on the types of instances I checked, I found an approximate 12% extra fee.
It’s essential to consider this cost factor when evaluating EKS Auto Mode. If cost reduction is a top priority, other options like manually managing autoscaling with Karpenter might be more advantageous. However, if simplifying operations and reducing operational overhead are important factors, the benefits of EKS Auto Mode can justify the additional cost.
In this above example, we see that for three instances of different types, enabling Auto Mode adds an extra fee of $125.62 per month, which makes a total of $1,172.44 for data plane costs.
Source: Amazon EKS Pricing | Managed Kubernetes Service
Key Questions to ask yourself if EKS Auto Mode can be considered
To help you choose the best option for managing your Amazon EKS cluster with nodes auto-scaling capabilities, I write down some questions that can be useful to ask yourself to consider if EKS Auto Mode is a good fit for your use case:
Is simplifying cluster operations and reducing operational overhead a priority?
If managing key components like Karpenter, AWS Load Balancer Controller, or EBS CSI consumes significant time and effort, EKS Auto Mode could be an excellent option as it reduces operational overhead. It should also justify an additional cost for dataplane nodes.
How critical is compliance with internal policies?
If your organization enforces strict compliance requirements such as running instances with custom AMIs or installing specific agents and softwares (e.g., logging, monitoring, or security tools), then EKS Auto Mode might not be the right fit. It only supports AWS provided AMIs, specifically based on Bottlerocket.
Do I need advanced network configuration capabilities?
EKS Auto Mode may not be the right choice if your use case requires for example:
- Using third-party CNIs instead of AWS VPC CNI (Cilium, Calico..)
- Assigning custom CIDR ranges for pods distinct from the VPC’s CIDRs.
- Leveraging advanced features like ENI/IP/Prefix warming or Security Groups per Pod.
Do I want to benefit from automatic cluster, nodes, managed controllers upgrades?
If you want to reduce time spent to upgrade your cluster components, and you don’t require granular control over the upgrade process, then EKS Auto Mode is a strong contender
Based on these questions I made a recap with a decision-making diagram. Based on your needs, it will guide you to identify if EKS Auto Mode is a good choice in comparison with Amazon EKS with EC2 instances managed by Karpenter
Conclusion
Amazon EKS Auto Mode offers a powerful solution for managing Kubernetes clusters with simplified operations, automated scaling, and easy node management. Its ability to handle the lifecycle of key components like Karpenter, networking and storage controllers, as well as provide automatic upgrades for cluster, nodes and managed controllers, makes it an attractive option for teams looking to reduce operational overhead, improve performance, availability, security of applications, and optimize compute costs.
However, this simplicity comes with trade-offs, like limited customisation, reduced granular control and additional costs for data plane nodes. These constraints can deter teams with strict compliance requirements, or advanced customisation needs.