Amazon MSK features

Amazon MSK supports deploys native versions of Apache Kafka so applications and tools built for Apache Kafka just work with Amazon MSK out of the box, with no application code changes.

Amazon MSK typically makes newer versions of Apache Kafka available within seven days of public availability. 

You can upgrade Apache Kafka versions on Amazon MSK clusters in just a few clicks, allowing you to decide when to take advantage of features and bug fixes present in new Apache Kafka versions. Amazon MSK automates the deployment of version upgrades on running clusters to maintain client I/O availability for customers following best practices.

With a few clicks in the console, you can create a fully managed Apache Kafka cluster that follows Apache Kafka’s deployment best practices, or you can create your own cluster using your own custom configuration. Once you create your desired configuration, Amazon MSK automatically provisions, configures, and manages the operations of your Apache Kafka cluster and Apache ZooKeeper nodes.

Apache ZooKeeper is required to run Apache Kafka, coordinate cluster tasks, and maintain state for resources interacting with the cluster. Amazon MSK manages the Apache ZooKeeper nodes for you. Each Amazon MSK cluster includes the appropriate number of Apache ZooKeeper nodes for your Apache Kafka cluster at no additional cost.

All clusters are provisioned across multiple availability zones (three AZ is default), are supported by Amazon MSK’s service-level agreement, and are supported by automated systems that detect and respond to issues within cluster infrastructure and Apache Kafka software. If a component fails, Amazon MSK automatically replaces it without downtime to your applications. Amazon MSK manages the availability of your Apache ZooKeeper nodes so you don’t need to start, stop, or directly access the nodes yourself. Amazon MSK also automatically deploys software patches as needed to keep your cluster up-to-date and running smoothly.

Amazon MSK uses multi-AZ replication for high-availability. Data replication is included at no additional cost.

Your Apache Kafka clusters run in an Amazon VPC managed by Amazon MSK. Your clusters are available to your own Amazon VPCs, subnets, and security groups based on the configuration you specify. You have complete control of your network configuration and IP addresses from your VPCs are attached to your Amazon MSK resources through elastic network interfaces (ENIs).

IAM Access Control is a no-cost security option that simplifies cluster authentication and Apache Kafka API authorization using IAM roles or user policies to control access. By using IAM Access Control, you no longer need to build and run one-off access management systems to control client authentication and authorization for Apache Kafka and your clusters are secured using least-privileged permissions by default. You also can use SASL/SCRAM or mutual TLS authentication with Apache Kafka access control lists (ACLs) to control client access. 

Amazon MSK encrypts your data at rest without special configuration or third-party tools. All data can be encrypted at rest using AWS Key Management Service (KMS) Customer Master Key (CMK) by default, or your own CMK. Amazon MSK also encrypts data in-transit via TLS between brokers and between clients and brokers on your cluster.

Amazon MSK lets you get started for less than $2.50 per day. Customers typically pay between $0.05 and $0.07 per GB ingested, all-in, which can be 1/13th the cost of other managed providers. Visit the Amazon MSK Pricing page for current prices and the Amazon MSK Best Practices page to learn how right-size your Amazon MSK clusters.

No other provider offers the breadth and depth of AWS integrations in Amazon MSK. These integrations include:

• AWS IAM for Apache Kafka and service-level API access control
• Amazon Kinesis Data Analytics for running fully managed Apache Flink applications to process streaming data within Apache Kafka
• AWS Glue Schema Registry to centrally control and evolve schemas
• AWS IoT for IoT event streaming into MSK
• AWS DMS for change data capture and analytics 
• AWS VPC for private client connectivity and network isolation
• AWS KMS for at rest encryption
• AWS Certificate Manager Private CAs for mutual TLS client authentication
• AWS Secrets Manager for secure storage and management of SASL/SCRAM secrets
• AWS CloudFormation to deploy Amazon MSK in code
• Amazon CloudWatch for cluster, broker, topic, consumer, and partition-level metrics

You can scale your Amazon MSK clusters by changing the size or family of your Apache Kafka brokers in minutes with no downtime. Changing the size or family of your brokers is a popular way to scale Amazon MSK clusters because it gives you the flexibility to adjust your MSK cluster’s compute capacity for changes in your workloads. This method can be preferred because it does not require partition reassignment which can impact Apache Kafka availability.

Alternatively you can expand the size of your cluster by up to 100s brokers per cluster using the console or command line interface (CLI). Submit a limit increase request if you need more than 15 brokers per cluster or more than 30 brokers per account.

Amazon MSK integrates with Cruise Control, a popular open source tool for Apache Kafka that automatically manages partition assignment on your behalf.

You can seamlessly scale up the amount of storage provisioned per broker to match changes in storage requirements using the AWS management console or AWS CLI or you can create an auto scaling policy to automatically expand your storage to meet your streaming requirements.

Amazon MSK deploys a best practice cluster configuration for Apache Kafka by default, and gives customers the ability to tune more than 30 different cluster configurations while supporting all dynamic and topic-level configurations. For more information, see Custom MSK Configurations in the documentation.

You can visualize and monitor important cluster, broker, topic, consumer, and partition-level metrics using Amazon CloudWatch.

Open Monitoring with Prometheus enables you to monitor Amazon MSK using solutions like Datadog, Lenses, New Relic, Sumo logic, or a Prometheus server, and easily migrate your existing monitoring dashboards to Amazon MSK. For more information, see Open Monitoring with Prometheus in the documentation.