Vlad Ionescu's logo
RSS Feed

Scaling containers on AWS in 2020

Comparing how fast containers scale up under different orchestrators on AWS in 2020

Reading time: about 20 minutes

This all started with a tech curiosity: what's the fastest way to scale containers on AWS? Is ECS faster than EKS? What about Fargate? Is there a difference between Fargate on ECS and Fargate on EKS?

For the fiscally responsible person in me, there is another interesting side to this. Does everybody have to bear the complexities of EKS? Where is the line? Say for a job that must finish in less than 1 hour and that on average uses 20.000 vCPUs and 50.000 GBs, what is the cost-efficient option considering the ramp-up time?

Hand-drawn graph showing EKS scaling from 0 to 3500 containers in about 5 minutes. Fargate on ECS and Fargate on EKS scale up to the same 3500 containers in about an hour. Tuned Fargate scales faster than Fargate, but slower than EKS. There is minimal variance in the results.
The final results

Tl;dr:

Beware, this benchmark is utterly useless for web workloads — the focus here is on background work or batch processing. For a scaling benchmark relevant to web services, you can check out Clare Liguori's awesome demo at re:Invent 2019.


That's it! If you want to check out details about how I tested this, read on.


Warning

This post is outdated!

For the latest information, check out the 2022 version!

Preparation

Before any test can be done, we have to prepare.

Setup

I created a completely separate AWS Account for this — my brand new “Container scaling” account. Any performance tests I plan to do in the future will happen here.

In this account, I submitted requests to increase several limits.

By default, Fargate allows a maximum of 250 tasks to run. The Fargate Spot Concurrent Tasks limit was raised to 10.000.

By default, Fargate allows scaling at 1 task per second (after an initial burst of 10 tasks the first second). After discussions with AWS and validating my workflow, this limit was raised to 3 tasks per second.

By default, EKS does a great job of scaling the Kubernetes Control Plane components (really — I tested this extensively for my customers). As there will be lots of time with 0 work, and as we are not benchmarking EKS scaling, I wanted to take that variable out. AWS is happy to pre-scale clusters depending on the workload, and they did precisely that after some discussions and validation.

By default, EC2 Spot allows for a maximum of 20 Spot Instances. After many talks, the EC2 Spot Instances limit was raised to 250 c5.4xlarge Spot Instances.

Test plan

Numbers

After an initial desired value of 30.000, and changing my mind multiple times, it was finally decided: we will test what is the fastest option to scale to about 3.000 containers!

Multiple reasons went into this:

Regions

We don't want to test just in us-east-1 due to its… size and particularities, so we should also run the tests in eu-west-1, which is the largest European region.

As any European AWS user, I've had us-east-1 issues that only happened during my night-time — actual US day-time. We must run the tests during the US day-time, too, for full relevance.

Measuring

To measure the container creation, we'll use CloudWatch Container Insights. It has the RunningTaskCount metric for Fargate and namespace_number_of_running_pods metric for Kubernetes that give us exactly what we need.

To scale up, we'll just edit Desired Tasks or Replicas to 3.500.

All the tests will start from a base of 1 task/ pod already running.

For EKS testing, the start point will be just 1 node running; cluster-autoscaler will have to add all the relevant nodes.

Container

The container image used was poc-hello-world/namer-service — a straightforward app I plan to use in upcoming workshops and posts.

The container size was decided to be 1 vCPU and 2 GBs of memory. Not too small, but not too large either.

Both tasks and pods can run multiple containers, but for simplicity, we will use just 1 container.

Expectations

Before starting, it is crucial to set expectations. We don't want to be surprised with a 5-digit bill, for example.

Expectations for Fargate on ECS

AWS Fargate on ECS has to respect the default 1 task per second launch limit, and so time to scale from 1 to 3.500 tasks should be around 3.500 seconds, which is about 1 hour. Reasonable.

As I want to focus on realistic results for everybody, we will mostly test Fargate with the default rate limits. AWS does increase the rate limits for relevant workloads, but that is the exception and not the rule.

A scaling test using the default limits would reach 10.000 Tasks in about 2:47 hours, which is indeed not that relevant. The first hour should be more than enough.

For Fargate, the pricing can be kind-of-estimated by multiplying the number of tasks with the hourly cost.

As per AWS Fargate pricing page we get the following values:

Yes, Fargate pricing is the same in Northern Virginia and Ireland. Nice!

Warning

These costs do not include ECR, NAT Gateway, and many others!

To be safe, I'll double the 60$ cost.

Final expectations for a Fargate test: 150$ and about 1 hour.

Expectations for EKS

AWS AutoScaling Group is responsible for adding EC2 instances, and cluster-autoscaler is in charge of deciding the actual number of EC2s needed.

It is a bit unclear how fast AWS would give us an instance. Let's go with 60 seconds.

From my testing a while ago, an EC2 instance took about 21 seconds from actually starting to becoming a Ready node in Kubernetes. Add say 30 seconds for image pulling and starting the container (an overestimation, but let's go wild).

Now, all of this can be modeled mathematically, and we’d get a time estimate. Unfortunately, I am not that smart, so we’ll skip getting a time estimate. It would take me less time to run a test than to do the math.

   

A c5.4xlarge EC2 instance has 16 vCPUs and 32 GBs of memory. Some of it is reserved for Kubernetes components, monitoring, NodeLocal DNS, and so on. Let's say a total of 2 vCPUs and 4 GBs for cluster-level pods on each node.

We are left with 14 vCPUs and 28 GBs, which at our task size is 14 pods per node. At 250 nodes, that is precisely 3.500 pods. Purrfect!

As per AWS EC2 pricing page, we get the following values:

Warning

These costs do not include ECR, NAT Gateway, and many others!

So to be safe, I'll double the 75$ cost.

Final expectations for an EKS test: 200$ per hour and unknown running time.

Expectations for Fargate on EKS

Pricing is the same as Fargate on ECS, so about 150$ per test.

Time is the same as EKS? Is the time closer to Fargate on ECS than to EKS?

I have no idea, and I really look forward to seeing what happens.

Running the tests

I ran a total of about 10 Fargate on EKS and ECS tests — for some of them, I was still figuring out some stuff. I ran a total of 4 EKS tests.

I ran the tests both during the weekend and during the week. I made sure to run the tests during day-time and night-time too.

The data used for the graph on top of the page can be downloaded as a CSV at this link(CSV, less than 1MB file).

Results

Let's recap the results from the top of the page!

Same hand-drawn graph showing EKS scaling from 0 to 3500 containers in about 5 minutes. Fargate on ECS and Fargate on EKS scale up to the same 3500 containers in about an hour. Tuned Fargate scales faster than Fargate, but slower than EKS. There is minimal variance in the results.
The final results

The Terraform code used for all the tests can be found on GitHub at Vlaaaaaaad/blog-scaling-containers-in-aws.

It is not pretty or correct code at all. On the other hand, it is code that works! I firmly believe in the reproducibility of any results, and I believe I have a moral duty to share everything I used to reach the conclusions presented here.

Unfortunately, the costs were not fully tracked. Due to AWS pre-scaling the EKS Control Plane for my clusters, I had to keep them running for the whole duration. I did multiple tests a day in multiple regions, so there was no easy way to get the cost of a single test run. The estimates did help a lot, but they were there to ensure I did not end up spending tens of thousands of dollars.

The total bill for the account was a little under 2.000 $.

Fargate results

Fargate on ECS and Fargate on EKS scaled surprisingly similar.

I expected more variance in the results, but they were almost identical scaling up: an initial burst and then 60 containers per minute. Maybe for a minute, it was 58, and maybe for another minute, it was 62, but variance was minimal.

When running the tests, I did discover a previously-unknown Fargate on ECS limitation: 1 Service can have at most 1.000 Tasks running. To reach the 3.500 running tasks, I had to create 4 separate services. That led to the Fargate on ECS tests starting with 4 containers instead of 1.

Something that I did not thoroughly test was downscaling. I did notice Fargate on EKS scaled down significantly faster. Fargate on ECS scaled down slower, from what I saw similar to the speed of scaling up.

It turns out my costs math was terrible but correct: I totally forgot to account for downscaling! Scaling down takes about the same time as scaling up — I calculated the costs just for the scale up. On the other hand, due to me doubling the costs to account for unexpected things, I was in the right ballpark!

EKS results

EKS results were also very similar between different runs. I tested both at night and during the day, I tested in multiple regions, and the results were almost identical.

EKS being so much faster than Fargate was a bit of a surprise. While Fargate would scale in about 50 minutes, EKS was consistently done in less than 10 minutes.

Since I am using EKS extensively in my work, there were no other surprises.

From a cost perspective, I made the same mistake: I did not account for scaling down.

Disclaimers

First of all, I would like to reiterate that this is utterly useless for web workloads — the focus here is on background work or batch processing. For a relevant web scaling benchmark, you can check out Clare Liguori's awesome demo at re:Invent 2019.

As an aside, Lambda would scale up to 3.500 containers in 1.5 to 7 seconds, depending on the region. It's an entirely different beast, but I thought it's worth mentioning.

Clearly, this was not a very strict or statistically relevant testing process.

I did just a few tests because I saw little variance — I did not see the point in running more.

I only tested in us-east-1 and eu-west-1, which are large regions. Numbers may or may not differ for smaller regions or during weird times — say Black Friday, Cyber Monday, EOY Report time.

The container image I used was the image for poc-hello-world/namer-service, which is small and maybe not that well suited. I did not want to go into the whole “let's optimize image pulling speed”. A study of all images on Dockerhub can be read here.

I only ran pods and tasks with 1 single container. Both pods and tasks can have multiple containers running.

I did not optimize the tests at all. I wanted to showcase the average experience, not a super-custom solution that would not help anybody. The values in here can likely be improved — multiple ECS clusters, multiple ASGs for EKS, and so on.

ECS with EC2 was completely ignored. I cannot say there was a reason, I did just not think of that. Fargate has the cost advantage, the simplicity, and the top-notch AWS integration. EKS has the complexity, all the knobs, and all the buzzwords. Between the two, I did not consider ECS with EC2, and that is on me.

Overall I am happy. After all this, we now have a ballpark figure we can use when designing systems 🙂

Acknowledgments

First of all, thank you so much to Ignacio and Michael for helping me escalate all my Support tickets and for connecting me to the right people in AWS!

Special thanks go out to Mats and Massimo for all their Fargate help and reviews! Your feedback was priceless!

Thanks to everybody else that helped review this, gave feedback, or supported me!

Generating the graph

This part took forever to do, so I decided to add it to the post.

The results are the most exciting thing about this whole post. I desperately wanted to have a pretty graph image showcasing the results.

Since the testing was not statistically correct, I thought a hand-drawn graph would be perfect. It would nicely showcase the data, while at the same time hinting that experiences may vary.

Numbers and Excel could not easily do this. LiveGap Charts had a bunch of hand-drawn options and was outstanding, but not exactly what I wanted.

Matplotlib to the rescue! After a bunch of research and about a day of playing around with it, I got a working script.

It is not pretty or optimized, nor is it correct. But it works!

          
# Install the required font
#  BEWARE: this only works on macOS
#          Linuxbrew does not install fonts
brew install homebrew/cask-fonts/font-humor-sans

# Install Python dependencies
pip3 install matplotlib numpy

# Run and generate the image
python3 draw.py
        


          
# File: draw.py
# The actual image-generation script

import matplotlib.pyplot as plt
from matplotlib.lines import Line2D

import numpy as np

# Load data from CSV
data = np.genfromtxt(
    'graph-results.csv',
    delimiter=',',
    names=True,
)

# Start an XKCD graph
plt.xkcd()

# Make the image pretty
figure = plt.figure()
figure.set_dpi(300)
figure.set_size_inches(8, 7.5)

figure.suptitle(
    'Scaling containers on AWS\n@iamvlaaaaaaad',
    fontsize=16,
)
plt.xlabel('Minutes')
plt.ylabel('Containers')

plt.xticks(np.arange(0, 70, step=10))

# Colors from https://jfly.uni-koeln.de/color/
plt.plot(
    data['EKS'],
    label='EKS with EC2',
    color=(0.8, 0.4, 0.7),
)
plt.plot(
    data['TunedFargate'],
    label='Tuned Fargate',
    color=(0.9, 0.6, 0.0),
)
plt.plot(
    data['FargateOnECS'],
    label='Fargate on ECS',
    color=(0.0, 0.45, 0.70),
)
plt.plot(
    data['FargateOnEKS'],
    label='Fargate on EKS',
    color=(0.35, 0.70, 0.90),
)

# Add a legend to the graph
#  using default labels
plt.legend(
    loc='lower right',
    borderaxespad=1
)

# Export the image
figure.savefig('containers.svg')
figure.savefig('containers.png')