Analysis: Why You Should Stop Managing Kafka Manually A Guide to Kafka UI and Cruise Control

Managing Kafka Clusters: A Guide to Simplifying Real-Time Data Operations ==================================================================== In the rapidly evolving world of software engineering, Kafka has emerged as a powerful tool for building real-time data pipelines and streaming applications. With over 80% of Fortune 100 companies using Kafka, it's clear that this open-source platform is revolutionizing the way we handle data. However, managing Kafka clusters can be a complex and challenging task, particularly as clusters grow in size and complexity. In this article, we will introduce two tools that can transform Kafka operations from a manual slog into a manageable process: Kafka UI and Cruise Control. Prerequisites ------------- Before diving into the details, it's essential to have the following prerequisites: 1. Docker Desktop installed and running 2. Basic understanding of Kafka concepts such as topics, partitions, brokers, producers, and consumers 3. Familiarity with Docker images, volumes, and networks Setting Up the Unmanaged Kafka Cluster --------------------------------------- To demonstrate the challenges of manual Kafka management, we will first create an unmanaged cluster locally. We'll use Docker to spin up three Kafka brokers running in KRaft mode (the modern, ZooKeeper-free approach).

Setting Up Our Unmanaged Cluster

First, we'll create a Docker Compose file named docker-compose-basic.yml:

 version: '3.8' services: kafka-1: image: confluentinc/cp-kafka:7.6.0 container_name: kafka-1 ports: - "9092:9092" environment: KAFKA_NODE_ID: 1 KAFKA_PROCESS_ROLES: broker, controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:29093,2@kafka-2:29093,3@kafka-3:29093 KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:29093,PLAINTEXT_HOST://0.0.0.0:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:29092,PLAINTEXT_HOST://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk' KAFKA_LOG_DIRS: /var/lib/kafka/data volumes: - kafka-1-data:/var/lib/kafka/data kafka-2: image: confluentinc/cp-kafka:7.6.0 container_name: kafka-2 ports: - "9093:9093" environment: KAFKA_NODE_ID: 2 KAFKA_PROCESS_ROLES: broker, controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:29093,2@kafka-2:29093,3@kafka-3:29093 KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:29093,PLAINTEXT_HOST://0.0.0.0:9093 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:29092,PLAINTEXT_HOST://localhost:9093 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk' KAFKA_LOG_DIRS: /var/lib/kafka/data volumes: - kafka-2-data:/var/lib/kafka/data kafka-3: image: confluentinc/cp-kafka:7.6.0 container_name: kafka-3 ports: - "9094:9094" environment: KAFKA_NODE_ID: 3 KAFKA_PROCESS_ROLES: broker, controller KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-1:29093,2@kafka-2:29093,3@kafka-3:29093 KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:29092,CONTROLLER://0.0.0.0:29093,PLAINTEXT_HOST://0.0.0.0:9094 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-3:29092,PLAINTEXT_HOST://localhost:9094 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2 KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 2 KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1 CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk' KAFKA_LOG_DIRS: /var/lib/kafka/data volumes: - kafka-3-data:/var/lib/kafka/data 

Run the following command in the same directory where you saved the docker-compose-basic.yml file:

 docker compose -f docker-compose-basic.yml up -d 

The command will start the Kafka cluster in the background. Verify that the containers are running with the following command:

 docker ps 

Creating Topics: The Manual Way ------------------------------- Now that we have a cluster running, let's simulate a real-world use case where different teams need Kafka topics for their applications (payments, logs, events, metrics, notifications, etc.). Let's start by creating a topic for logs.

Creating Topics: The Manual Way

To create a topic, we'll use the following command:

 docker exec -it kafka-1 kafka-topics \ --create \ --topic freecodecamp-logs \ --bootstrap-server kafka-1:29092,kafka-2:29092,kafka-3:29092 \ --partitions 12 \ --replication-factor 2 \ --config retention.ms=604800000 \ --config compression.type=snappy 

Repeat the above command to create more topics with different names and partitions. Note that the actual values for partitions and topics are not important for this tutorial.

Viewing Created Topics

To view the topics you have created, use the following command:

 docker exec -it kafka-1 kafka-topics \ --list \ --bootstrap-server kafka-1:29092,kafka-2:29092,kafka-3:29092 

Understanding the Pain of Manual Kafka Management -------------------------------------------------- Managing Kafka clusters manually can be tedious and error-prone, especially as clusters grow in size. The following sections highlight some of the challenges associated with manual Kafka management. ### Partition Imbalance During the topic creation process, we set the number of partitions for each topic. Creating too few partitions can create bottlenecks, while creating too many adds overhead. Finding the optimal number of partitions can be difficult, and manual calculations are prone to errors. ### Leader Election and Partition Rebalancing When adding or removing brokers, manually redistributing partitions can be a complex process. It requires calculating optimal placement and creating complex reassignment plans. One mistake in the reassignment plan can lead to data loss or other issues. ### Consumer Lag Monitoring Monitoring consumer lag (the difference between the offsets that consumers have processed and the offsets that have been committed to Kafka) can be challenging when done manually. It requires tracking which partitions are lagging, which consumer instances own them, and where the lag is growing or shrinking. ### Broker Failures In the event of a broker failure, it's essential to identify under-replicated partitions, trigger leader elections, and create partition reassignment plans manually. These tasks can be time-consuming and error-prone. ### Scalability Managing Kafka clusters manually becomes increasingly difficult as clusters grow in size. The more brokers, topics, and partitions you have, the more complex the manual operations become. Kafka UI: A Visual Solution for Kafka Cluster Management ------------------------------------------------------- Kafka UI is an open-source web interface that provides a visual dashboard for managing Kafka clusters. It simplifies the manual operations of Kafka by replacing cryptic command-line interfaces with a user-friendly interface. ### Key Features 1. **Cluster Overview**: See all brokers, topics, and partitions at a glance. 2. **Topic Management**: Create, configure, and delete topics with a GUI. 3. **Consumer Group Monitoring**: Track lags, offsets, and consumer health in real-time. 4. **Message Browsing**: View actual messages in topics without command-line tools. ### Setting Up Kafka UI To set up Kafka UI, follow these steps: 1. Modify the `docker-compose-basic.yml` file to include the Kafka UI service:

Setting Up Kafka UI

Add the following service to your docker-compose-basic.yml file:

 kafka-ui: image: provectuslabs/kafka-ui:latest container_name: kafka-ui ports: - "8080:8080" environment: DYNAMIC_CONFIG_ENABLED: 'true' KAFKA_CLUSTERS_0_NAME: freecodecamp-cluster KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka-1:29092,kafka-2:29092,kafka-3:29092 depends_on: - kafka-1 - kafka-2 - kafka-3 

2. Start the Kafka cluster and Kafka UI:

Starting the Cluster and Kafka UI

Run the following command to start the Kafka cluster and Kafka UI:

 docker compose -f docker-compose-basic.yml up -d 

3. Access Kafka UI by visiting http://localhost:8080 in your browser. Cruise Control: Automated Self-Healing for Kafka Clusters --------------------------------------------------------- Cruise Control is an open-source automation engine developed by LinkedIn that helps manage Kafka clusters by handling tasks such as cluster balancing, self-healing, and performance optimization. Cruise Control complements Kafka UI by providing intelligent automation and self-healing capabilities. ### Key Features 1. **Metric Monitoring**: Cruise Control collects internal metrics (CPU, disk, network traffic, partition sizes) from Kafka brokers and sends them to a Kafka topic. 2. **Analysis and Proposal Generation**: Based on the collected metrics, Cruise Control analyzes the data and generates proposals for partition movements to optimize cluster performance. 3. **Self-Healing**: Cruise Control can automatically trigger partition movements and leadership changes to maintain optimal cluster performance. ### Setting Up Cruise Control To set up Cruise Control, follow these steps: 1. Modify the `docker-compose-basic.yml` file to include the Cruise Control service:

Setting Up Cruise Control

Add the following service to your docker-compose-basic.yml file:

 cruise-control: image: justramesh2000/cruise-control-kraft:2.5.142 container_name: cruise-control ports: - "9090:9090" volumes: - ./config/cruisecontrol.properties:/opt/cruise-control/config/cruisecontrol.properties - ./config/capacityJBOD.json:/opt/cruise-control/config/capacityJBOD.json:ro - ./config/log4j.properties:/opt/cruise-control/config/log4j.properties:ro 

2. Create the configuration files for Cruise Control:

Creating Cruise Control Configuration Files

Create the following files in a config directory:

• cruisecontrol.properties: Main configuration file
• capacityJBOD.json: Broker capacity information
• log4j.properties: Logging configuration

Fill the files with the appropriate configuration settings as described in the official Cruise Control documentation.

3. Start the Kafka cluster, Kafka UI, and Cruise Control:

Starting the Cluster, Kafka UI, and Cruise Control

Run the following command to start the Kafka cluster, Kafka UI, and Cruise Control:

 docker compose -f docker-compose-basic.yml up -d 

### Using Cruise Control to Balance the Cluster 1. Create a topic and produce messages to create