General

Q: What is Amazon MSK?
Amazon Managed Streaming for Apache Kafta (Amazon MSK) is an AWS streaming data service that manages Apache Kafka infrastructure and operations, making it easy for. developers and DevOps managers to run Apache Kafka applications on AWS without the need to become experts in operating Apache Kafka clusters. Amazon MSK is an ideal place to run existing or new Apache Kafka applications in AWS. Amazon MSK operates and maintains Apache Kafka clusters, provides enterprise-grade security features out of the box, and has built-in AWS integrations that accelerate development of streaming data applications. To get started, you can migrate existing Apache Kafka workloads into Amazon MSK, or with a few clicks, you can build new ones from scratch in minutes. There are no data transfer charges for in-cluster traffic, and no commitments or upfront payments required. You only pay for the resources that you use.
 
Q: What is Apache Kafka?
Apache Kafka is an open-source, high performance, fault-tolerant, and scalable platform for building real-time streaming data pipelines and applications. Apache Kafka is a streaming data store that decouples applications producing streaming data (producers) into its data store from applications consuming streaming data (consumers) from its data store. Organizations use Apache Kafka as a data source for applications that continuously analyze and react to streaming data. Learn more about Apache Kafka.
 
Q: What is streaming data?
Streaming data is a continuous stream of small records or events (a record or event is typically a few kilobytes) generated by thousands of machines, devices, websites, and applications. Streaming data includes a wide variety of data such as log files generated by customers using your mobile or web applications, ecommerce purchases, in-game player activity, information from social networks, financial trading floors, geospatial services, security logs, metrics, and telemetry from connected devices or instrumentation in data centers. Streaming data services like Amazon MSK and Amazon Kinesis Data Streams make it easy for you to continuously collect, process, and deliver streaming data. Learn more about streaming data.
 
Q: What are Apache Kafka’s primary capabilities?
Apache Kafka has three key capabilities:
 
  • Apache Kafka stores streaming data in a fault-tolerant way, providing a buffer between producers and consumers. It stores events as a continuous series of records and preserves the order in which the records were produced. 
  • Apache Kafka allows many data producers—e.g. websites, internet of things (IoT) devices, Amazon Elastic Compute Cloud (Amazon EC2) instances—to continuously publish streaming data and categorize it using Apache Kafka topics. Multiple data consumers (e.g. machine learning applications, AWS Lambda functions, or microservices) read from these topics at their own rate, similar to a message queue or enterprise messaging system.
  • Data consumers process data from Apache Kafka topics on a first-in-first-out basis, preserving the order data was produced.
 
Q: What are the key concepts of Apache Kafka?
Apache Kafka stores records in topics. Data producers write records to topics and consumers read records from topics. Each record in Apache Kafka consists of a key, a value, and a timestamp. Apache Kafka partitions topics and replicates these partitions across multiple nodes called brokers. Apache Kafka runs as a cluster on one or more brokers, and brokers can be located in multiple AWS availability zones to create a highly available cluster. Apache Kafka relies on Apache ZooKeeper to coordinate cluster tasks and can maintain state for resources interacting with an Apache Kafka cluster.
 
Q: When should I use Apache Kafka?
Apache Kafka is used to support real-time applications that transform, deliver, and react to streaming data, and for building real-time streaming data pipelines that reliably get data between multiple systems or applications.
 
Q: What does Amazon MSK do?
Amazon MSK makes it easy to get started and run open-source versions of Apache Kafka in AWS with high availability and security while providing integration with AWS services without the operational overhead of running an Apache Kafka cluster. Amazon MSK allows you to use and configure open-source versions of Apache Kafka while the service manages the setup, provisioning, AWS integrations, and on-going maintenance of Apache Kafka clusters.
 
With a few clicks in the console, you can provision an Amazon MSK cluster. From there, Amazon MSK replaces unhealthy brokers, automatically replicates data for high availability, manages Apache ZooKeeper nodes, automatically deploys hardware patches as needed, manages the integrations with AWS services, makes important metrics visible through the console, and supports Apache Kafka version upgrades so you can take advantage of improvements to the open-source version of Apache Kafka.
 
Q: What Apache Kafka versions does Amazon MSK support?
For supported Kafka versions, see the Amazon MSK documentation.
 
Q: Are Apache Kafka APIs compatible with Amazon MSK?
Yes, all data plane and admin APIs are natively supported by Amazon MSK.
 
Q: Is the Apache Kafka AdminClient supported by Amazon MSK?
Yes.
 
Q: Does Amazon MSK support schema registration?
Yes, Apache Kafka clients can use the AWS Glue Schema Registry, a serverless feature of AWS Glue, at no additional charge. Visit the Schema Registry user documentation to get started and to learn more.

Data production and consumption

Q: Can I use Apache Kafka APIs to get data in and out of Apache Kafka?
Yes, Amazon MSK supports the native Apache Kafka producer and consumer APIs. Your application code does not need to change when clients begin to work with clusters within Amazon MSK.
 
Q: Can I use Apache Kafka Connect, Apache Kafka Streams, or any other ecosystem component of Apache Kafka with Amazon MSK?
Yes, you can use any component that leverages the Apache Kafka producer and consumer APIs, and the Apache Kafka Admin Client. Tools that upload .jar files into Apache Kafka clusters are currently not compatible with Amazon MSK, including Confluent Control Center, Confluent Auto Data Balancer, and Uber uReplicator.

Migrating to Amazon MSK

Q: Can I migrate data within my existing Apache Kafka cluster to Amazon MSK?
Yes, you can use third-party tools or open source tools like MirrorMaker that come with open source Apache Kafka to replicate data from clusters into an Amazon MSK cluster. Here is an Amazon MSK migration lab to help you complete a migration.

Version upgrades

Q: Are Apache Kafka version upgrades supported? 
Yes, Amazon MSK supports fully managed in-place Apache Kafka version upgrades. To learn more about upgrading your Apache Kafka version and high availability best practices, see the version upgrades documentation.

Clusters

Q: How do I create my first Amazon MSK cluster?
You can create your first cluster with a few clicks in the AWS management console or using the AWS SDKs. First, in the Amazon MSK console select an AWS region to create an Amazon MSK cluster in. Choose a name for your cluster, the VPC you want to run the cluster with, a data replication strategy for the cluster, and the subnets for each AZ. Next, pick a broker instance type and quantity of brokers per AZ, and click create.

Q: What resources are within a cluster?
Each cluster contains broker instances, provisioned storage, and Apache ZooKeeper nodes.

Q: What types of broker instances can I provision within an Amazon MSK cluster?
You can choose EC2 T3.small or instances within the EC2 M5 instance family.
 
Q: Does Amazon MSK offer Reserved Instance pricing?
No, not at this time.

Q: Do I need to provision and pay for broker boot volumes?
No, each broker you provision includes boot volume storage managed by the Amazon MSK service.

Q: When I create an Apache Kafka cluster, do the underlying resources (e.g. Amazon EC2 instances) show up in my EC2 console?
Some resources, like elastic network interfaces (ENIs), will show up in your Amazon EC2 account. Other Amazon MSK resources will not show up in your EC2 account as these are managed by the Amazon MSK service.

Q: What do I need to provision within an Amazon MSK cluster?
You need to provision broker instances and broker storage with every cluster you create. You do not provision Apache ZooKeeper nodes as these resources are included at no additional charge with each cluster you create.

Q: What is the default broker configuration for a cluster?
Unless otherwise specified, Amazon MSK uses the same defaults specified by the open-source version of Apache Kafka. The default settings are documented here. 
Q: Can I provision brokers such that they are imbalanced across AZs (e.g. 3 in us-east-1a, 2 in us-east-1b, 1 in us-east-1c)?
No, Amazon MSK enforces the best practice of balancing broker quantities across AZs within a cluster.

Q: How does data replication work in Amazon MSK?
Amazon MSK uses Apache Kafka’s leader-follower replication to replicate data between brokers. Amazon MSK makes it easy to deploy clusters with multi-AZ replication and gives you the option to use a custom replication strategy by topic. By default with each of the replication options, leader and follower brokers will be deployed and isolated using the replication strategy specified. For example, if you select a 3 AZ broker replication strategy with 1 broker per AZ cluster, Amazon MSK will create a cluster of three brokers (1 broker in three AZs in a region), and by default (unless you choose to override the topic replication factor) the topic replication factor will also be 3.

Q: Can I change the default broker configurations or upload a cluster configuration to Amazon MSK?
Yes, Amazon MSK allows you to create custom configurations and apply them to new and existing clusters. For more information on custom configurations, see the configuration documentation.

Q: What configuration properties am I able to customize?
The configurations properties that you can customize are documented here.

Q: What is the default configuration of a new topic?
Amazon MSK uses Apache Kafka’s default configuration unless otherwise specified here.

Topics

Q: How do I create topics?
Once your Apache Kafka cluster has been created, you can create topics using the Apache Kafka APIs. All topic and partition level actions and configurations are performed using Apache Kafka APIs. The following command is an example of creating a topic using Apache Kafka APIs:
 
bin/kafka-topics.sh --create —bootstrap-server ConnectionString:9092 --replication-factor 3 --partitions 1 --topic TopicName  

Networking

Q: Does Amazon MSK run in an Amazon VPC?
Yes, Amazon MSK always runs within an Amazon VPC managed by the Amazon MSK service. Amazon MSK resources will be available to your own Amazon VPC, subnet, and security group you select when the cluster is setup. IP addresses from your VPC are attached to your Amazon MSK resources through elastic network interfaces (ENIs), and all network traffic stays within the AWS network and is not accessible to the Internet.
 
Q: Is the connection between my clients and an Amazon MSK cluster always private?
Yes, the only way data can be produced and consumed from an Amazon MSK cluster is over a private connection between your clients in your VPC and the Amazon MSK cluster. Amazon MSK does not support public endpoints.

Q: How will the brokers in my Amazon MSK cluster be made accessible to clients within my VPC?
The brokers in your cluster will be made accessible to clients in your VPC through elastic network interfaces (ENIs) which will appear in your account. The Security Groups on the ENIs will dictate the source and type of ingress and egress traffic allowed on your brokers.

Q: How can I give clients running in different AWS accounts access to my cluster?
You can use VPC peering to give clients running in different AWS accounts access to your cluster.

Connecting to the VPC

Q: How do I connect to my AWS MSK cluster outside of the VPC?

There are several methods to connect to your AWS MSK clusters outside of your VPC.

Encryption

Q: Can I encrypt data in my Amazon MSK cluster?
Yes, Amazon MSK uses Amazon EBS server-side encryption and AWS KMS keys to encrypt storage volumes.

Q: Is data encrypted in-transit between brokers within an Amazon MSK cluster?
Yes, by default new clusters have encryption in-transit enabled via TLS for inter-broker communication. You can opt-out of using encryption in-transit when a cluster is created.

Q: Is data encrypted in-transit between my Apache Kafka clients and the Amazon MSK service?
Yes, by default in-transit encryption is set to TLS only for clusters created from the CLI or AWS Console. Additional configuration is required for clients to communicate with clusters using TLS encryption. You can change the default encryption setting by selecting the TLS/plaintext or plaintext settings. Read More: about  MSK Encryption

Q: Is data encrypted in-transit as it moves between brokers and Apache ZooKeeper nodes in an Amazon MSK cluster?
Yes, Amazon MSK clusters running Apache Kafka version 2.5.1 or greater support TLS in-transit encryption between Kafka brokers and ZooKeeper nodes.
 
Q: Can I update the encryption settings on my cluster?
You can change client-to-broker encryption settings on your clusters from the console or through the update-security API. Please note that broker-to-broker encryption settings cannot be changed on existing clusters. 

Access Management

Q: How do I control cluster authentication and Apache Kafka API authorization?
Amazon MSK offers three options for controlling authentication (AuthN) and authorization (AuthZ). 1) IAM Access Control for both AuthN/Z (recommended), 2) TLS certificate authentication (CA) for AuthN and access control lists for AuthZ, and 3) SASL/SCRAM for AuthN and access control lists for AuthZ. Amazon MSK recommends using IAM Access Control. It’s the easiest to use and because it defaults to least privilege access, the most secure option.

Q: How does authorization work in Amazon MSK?
If you are using IAM Access Control, Amazon MSK uses the policies you write and its own authorizer to authorize actions. If you are using TLS certificate authentication or SASL/SCRAM, Apache Kafka uses access control lists (ACLs) for authorization. To enable ACLs you must enable client authentication using either TLS certificates or SASL/SCRAM.

Q: How can I authenticate and authorize a client at the same time?
If you are using IAM Access Control, Amazon MSK will authenticate and authorize for you without any additional set up. If you are using TLS authentication, you can use the Dname of clients TLS certificates as the principal of the ACL to authorize client requests. If you are using SASL/SCRAM, you can use the username as the principal of the ACL to authorize client requests.

Q: How do I control service API actions?
You can control service API actions using AWS IAM.

Q: Can I enable IAM Access Control for an existing cluster?
No, however a feature that would allow you to update your authentication settings is coming soon.

Q: Can I use IAM Access Control outside of Amazon MSK?
No, IAM Access Control is only available for Amazon MSK clusters.

Q: Can I update authentication settings on my cluster?
You can enable or disable authentication modes for your clusters from the console or through the update-security API. When using the API, the authentication modes that are explicitly declared will be modified accordingly, while those that are omitted will be maintained as is. For example, if your cluster uses mTLS for authentication and you enable IAM Access Control by calling the update-security API, both mTLS and IAM Access Control will be enabled on your cluster.

Q: Can I enable multiple authentication modes on my cluster?
Yes, you can add multiple authentication modes to your cluster, both during creation and updates. The brokers within the cluster have dedicated ports for each authentication mode, and your clients that connect to Kafka through these ports must have the corresponding authentication mode enabled.

Q: Can I disable an authentication mode on my cluster?
Yes, you can disable an authentication mode. To ensure that your clients do not lose connectivity with the brokers, do not disable any existing authentication modes until all the clients have been updated to use other available authentication modes.

Q: Can I track clients using an authentication mode with my cluster?
Yes, you can track the number of open connections by authentication mode using the ClientConnectionCount metric published under AWS/Kafka namespace in Amazon CloudWatch metrics.

Monitoring, metrics, logging, tagging

Q: How do I monitor the performance of my clusters or topics?
You can monitor the performance of your clusters using the Amazon MSK console, Amazon CloudWatch console, or you can access JMX and host metrics using Open Monitoring with Prometheus, an open source monitoring solution.
 
Q: What is the cost for the different CloudWatch monitoring levels?
The cost of monitoring your cluster using Amazon CloudWatch is dependent on the monitoring level and the size of your Apache Kafka cluster. Amazon CloudWatch charges per metric per month and includes a free tier; see Amazon CloudWatch pricing for more information. For details on the number of metrics exposed for each monitoring level, see Amazon MSK monitoring documentation.
 
Q: What monitoring tools are compatible with Open Monitoring with Prometheus?
Tools that are designed to read from Prometheus exporters are compatible with Open Monitoring, like: Datadog, Lenses, New Relic, Sumologic, or a Prometheus server. For details on Open Monitoring, see Amazon MSK Open Monitoring documentation.
 
Q: How do I monitor the health and performance of clients?
You can use any client-side monitoring supported by the Apache Kafka version you are using.
 
Q: Can I tag Amazon MSK resources?
Yes, you can tag Amazon MSK clusters from the AWS CLI or Console.
 
Q: How do I monitor consumer lag?
Topic level consumer lag metrics are available as part of the default set of metrics that Amazon MSK publishes to Amazon CloudWatch for all clusters. No additional setup is required to get these metrics. To get partition level metrics (partition dimension), you can enable enhanced monitoring (PER_PARTITION_PER_TOPIC) on your cluster. Alternatively, you can enable Open Monitoring on your cluster, and use a Prometheus server, to capture partition level metrics from the brokers in your cluster. Consumer lag metrics are available at port 11001, just as other Kafka metrics.
 
Q: How much does it cost to publish the consumer lag metric to Amazon CloudWatch?
Topic level metrics are included in the default set of Amazon MSK metrics, which are free of charge. Partition level metrics are charged as per Amazon CloudWatch pricing.
 
Q: How do I access Apache Kafka broker Logs?
You can enable broker log delivery for new and existing Amazon MSK clusters. You can deliver broker logs to Amazon CloudWatch Logs, Amazon S3, and Kinesis Data Firehose. Kinesis Data Firehose supports Amazon Elasticsearch Service among other destinations. To learn how to enable this feature, see the Amazon MSK Logging Documentation. To learn about pricing refer to CloudWatch Logs and Kinesis Data Firehose pricing pages.
 
Q: What is the logging level for broker logs?
Amazon MSK provides INFO level logs for all brokers within a cluster.
 
Q: How do I access Apache ZooKeeper Logs?
You can request Apache ZooKeeper logs through a support ticket.
 
Q: Can I log the use of Apache Kafka resource APIs, like create topic?
Yes, if you use IAM Access Control, the use of Apache Kafka resource APIs is logged to AWS CloudTrail. 

Apache Kafka Connect

Q: What is Apache Kafka Connect?

Kafka Connect, an open source component of Apache Kafka, is a framework for connecting Apache Kafka with external systems such as databases, key-value stores, search indexes, and file systems. 

Q: Where can I get connectors?

MSK Connect runs any connector that implements the Kafka Connect interfaces. There are many sources of connectors, including from our partners, open source projects like Debezium, and from commercial connector vendors like Confluent and lenses.io.

Q: Can I use MSK Connect to move data between different Apache Kafka clusters?

Yes, you can use connectors that both read from and write to Kafka. The popular Apache Kafka replication tool, MirrorMaker 2, uses Kafka Connect to replicate data between topics on different Apache Kafka clusters. This provides an easy path to synchronize or migrate data between Kafka clusters.

Q: Does MSK Connect support non-MSK Apache Kafka clusters?

MSK Connect works with Amazon MSK clusters, other Apache Kafka and compatible clusters including self-managed clusters in EC2 or non-AWS environments, subject to Amazon VPC connectivity. 

Apache ZooKeeper

Q: What is Apache ZooKeeper?
From https://zookeeper.apache.org/: “Apache ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications,” including Apache Kafka.

Q: Does Amazon MSK use Apache ZooKeeper?
Yes, Amazon MSK uses Apache ZooKeeper and manages Apache ZooKeeper within each cluster as a part of the Amazon MSK service. Apache ZooKeeper nodes are included with each cluster at no additional cost.
 
Q: How do my clients interact with Apache ZooKeeper?
Your clients can interact with Apache ZooKeeper through an Apache ZooKeeper endpoint provided by the service. This endpoint is provided in the AWS management console or using the DescribeCluster API.

Integrations

Q: What AWS services does Amazon MSK integrate with?
Amazon MSK integrates with:

Scaling

Q: How can I scale up storage in my cluster?
You can scale up storage in your cluster using the AWS Management Console or the AWS CLI.
 
Q: How can I automatically expand storage in my cluster?
You can create an auto scaling policy for storage using the AWS Management console or by creating an AWS Application Auto scaling Policy using the AWS CLI or APIs.

Q. Can I scale the number of brokers in an existing cluster?
Yes. You can scale out, or increase, the number of brokers for existing Amazon MSK clusters.
 
Q. Can I scale the broker size in an existing cluster?
Yes. You can choose to scale to a smaller or larger broker type on your Amazon MSK clusters.
 
Q: How do I balance partitions across brokers?
You can use Cruise Control for automatically rebalancing partitions to manage I/O heat. See the  Cruise Control documentation for more information. Alternatively, you can use the Kafka Admin API kafka-reassign-partitions.sh to reassign partitions across brokers.

Pricing and availability

Q: How does Amazon MSK pricing work?
Pricing is per broker-hour and per GB-month of storage provisioned. AWS data transfer rates apply for data transfer in and out of Amazon MSK. For more information, visit our pricing page.

Q: Do I pay for data transfer as a result of data replication?
No, all in-cluster data transfer is included with the service at no additional charge.
 
Q: What AWS regions offer Amazon MSK?
Amazon MSK region availability is documented here. 

Q: How does data transfer pricing work?
You will pay standard AWS data transfer charges for data transferred in and out of an Amazon MSK cluster. You will not be charged for data transfer within the cluster in a region, including data transfer between brokers and data transfer between brokers and Apache ZooKeeper nodes.

Compliance

Q: What compliance programs are in scope for Amazon MSK?
Amazon MSK is compliant or eligible for the following programs:
  • HIPAA eligible
  • PCI
  • ISO
  • SOC 1,2,3

For a complete list of AWS services and compliance programs, please see AWS Services in Scope by Compliance Program.

Service Level Agreement

Q: What does the Amazon MSK SLA guarantee?
Our Amazon MSK SLA guarantees a Monthly Uptime Percentage of at least 99.9% for Amazon MSK.
 
Q: How do I know if I qualify for a SLA Service Credit?
You are eligible for a SLA credit for Amazon MSK under the Amazon MSK SLA if Multi-AZ deployments on Amazon MSK have a Monthly Uptime Percentage of less than 99.9% during any monthly billing cycle.
 
For full details on all of the terms and conditions of the SLA, as well as details on how to submit a claim, please see the Amazon MSK SLA details page.

Get started with Amazon MSK

Calculate your costs
Calculate your costs

Visit the Amazon MSK pricing page.

Review the getting-started guide
Review the getting-started guide

Learn how to set up your Apache Kafka cluster on Amazon MSK in this step-by-step guide.

Run your Apache Kafka cluster
Run your Apache Kafka cluster

Start running your Apache Kafka cluster on Amazon MSK. Log in to the Amazon MSK console.