2019 Aiven Kafka benchmark setup

As with the previous test, we really wanted to estimate the true performance you’d expect from using Aiven Kafka. That is, we used standard Aiven plans and configurations, standard client configurations and ran the test over the external network interfaces.

Kafka Business-4 benchmark results

We first tested the performance of our Business-4 plan. That’s a three broker cluster with 1–2 CPU (depending on the cloud) and 4GB RAM per instance. On Amazon Web Services, this plan handled about 135,000 messages per second while the same plan on Google Cloud Platform and Azure handled around 70,000.

Kafka Business-4 performance in MB/second

Kafka Business-4 monthly throughput cost

Those plans are priced $660/month on AWS (us-east-1), $500/month on GCP (us-east1) and $550/month on Azure (eastus2). That’d be just over $10 per MB/s per month for this plan size in AWS, and $15 and $16 per MB/s for GCP and Azure respectively.

Kafka Business-8 benchmark results

Next, we moved on to increasing the size of the brokers. The next test was based on Business-8 plan tier, essentially doubling the resources to 2–4 CPUs and 8 GB RAM per instance. This time, there was a slight increase in AWS to 137,000 messages per second, but larger ones in Azure and GCP to 120,000 and 95,000 respectively.

Kafka Business-8 performance in MB/second

Again, we used these same numbers converted to throughput: that’s 67 MB/s for AWS, 46 MB/s for GCP and 59 MB/s for Azure.

Kafka Business-8 monthly throughput cost

With the Business-8 plans, monthly estimated costs increased across the board at $19 per MB/s for AWS and Azure and around $22 per MB/s for GCP. However, it’s important to note that throughput is only one way to measure value. For example, Business-8 plans come with double the storage of Business-4 plans which allows for longer retention times.

Kafka Premium-6x-8 benchmark results

Our last test doubled the number of brokers. We wanted to verify just how well Kafka scales vertically — as it did quite perfectly in our previous round of testing. Thus, we ran this one with a six broker Premium-6x-8 plan tier, with similarly sized instances as Business-8.

Kafka Premium-6x-8 performance in MB/second

And the same as throughput figures: 132 MB/s on AWS, 116 on Azure and 82 on GCP.

Kafka Premium-6x-8 monthly throughput cost

From the plan pricing, estimated monthly costs are around $19 per MB/s for AWS, $18 for Azure and $23 for GCP.

Wrapping up

Apache Kafka continues to perform just as well as we’ve come to expect and scales nicely with both added resources and increased cluster sizes. It’s performant, scalable and cost effective — a solid centerpiece of the modern data architecture.

Aiven Kafka

Aiven Kafka is one of 8 integrable services offered on the Aiven platform. It’s a fully-managed and hosted Kafka service available in AWS, GCP, Microsoft Azure, DigitalOcean, UpCloud, and Packet.

--

--

--

Your database in the cloud, www.aiven.io

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Dynamics 365 online deployment — MB2–715 study notes

Dissomaster

Dissomaster update

What I coded in 2021

Message of the Day from Abhitech Solutions 🚀 — Issue #6

from Twitter https://twitter.com/build_online

Google Analytics 4: Events Review

From a naive method to a data structure

How to Enhance Phased Rollouts with Feature Flags

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Aiven

Aiven

Your database in the cloud, www.aiven.io

More from Medium

Apache Kafka as cloud-native iPaaS Integration Middleware

Micro services and Conway’s law

Request-Reply pattern using Apache Kafka or How not to loose your data

Developer’s experience with an event-driven solution implementation