Here are the steps to install Apache Kafka on Rocky Linux or AlmaLinux 8 server, of course, using command terminal.
Apache Kafka is open-source software that enables the storage and processing of data streams via a distributed streaming platform. In simple words, Apache Kafka is an event streaming platform that acts as a messaging system between the sender and the recipient with high fault tolerance and scalability capabilities because it is based on a distributed architecture that is optimized for the same.
Well, this system was originally developed by LinkedIn as a message queue, however, being a project of Apache Software Foundation, it is open source and a powerful streaming platform with a variety of functions. It offers interfaces to write data to Kafka clusters, to read data, or to import and export data to and from third-party systems. Due to low latency and high throughput, it can process real-time streams easily.
The interfaces also make it possible for users to load data streams from third-party systems or to export them to these systems. This makes Apache Kafka suitable for large amounts of data and applications in the big data environment.
It can be used for a wide range of applications such as tracking website activity in real-time, monitoring distributed applications, aggregating log files from different sources, synchronization of data in distributed systems, training of models in real-time helps in machine learning, and more…
Apache Kafka offers these four main interfaces (APIs – Application Programming Interfaces). Know more about each API at the official documentation page:
- Admin API
- Producer API
- Consumer API
- Streams API
- Connect API
What do you need to follow this tutorial:
- Rocky or AlmaLinux 8 or any other RHEL based server, if possible then clean one.
- A user with sudo access.
Steps to install Apache Kafka on Rocky Linux 8
The given guide is applicable for all the RHEL 8 based Linux systems including CentOS 8 and Oracle Linux 8 to install Kafka if required.
1. Update system
Well, before moving further run the system update command to ensure all the installed packages are up to date. For that run the below command, this will also refresh the repository cache.
sudo dnf update
2. Install Java
Apache Kafka needs Java to run, hence first we need to install that on our local environment and it must be equal or greater than Java 8. Well, we don’t need to add any third repository because the package to get JAVA is already there on the system base repo, hence, let’s use the given command.
For Java 11
sudo dnf install java-11-openjdk
Well, for the latest version such as 16 use the below commands:
sudo dnf install epel-release
sudo dnf install java-latest-openjdk
3. Download the latest Apache Kafka on Rocky Linux 8 or Almalinux
Apache Kafka is available as a tarball file on the official website. Hence, Go to the official website and download the latest version. You can also copy any mirror link and use the
wget command to download using the command line, as we have done here:
sudo dnf install wget nano
Extract the downloaded file
tar -xf kafka_*tgz
To see the file:
Move it to
/usr/local/ just to ensure we won’t delete the Kafka folder accidentally.
sudo mv kafka_2.13-3.0.0/ /usr/local/kafka
4. Create System Service for Zookeeper and Kafka
Although just for the test you can run both Zookeeper and Kafka service script directly, manually, however, for the production server we have to run it in the background. Hence, create systemd units for both the scripts.
Create systems file Zookeeper
As per the official website in the future, Kafka will not need Zookeeper, however, while writing this article we need it. So, create a service file first for Zookeeper.
sudo nano /etc/systemd/system/zookeeper.service
Copy-Paste the below-given lines:
[Unit] Description=Apache Zookeeper server Documentation=http://zookeeper.apache.org Requires=network.target remote-fs.target After=network.target remote-fs.target [Service] Type=simple ExecStart=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-start.sh /usr/local/kafka/config/zookeeper.properties ExecStop=/usr/bin/bash /usr/local/kafka/bin/zookeeper-server-stop.sh Restart=on-abnormal [Install] WantedBy=multi-user.target
Save and close the file by pressing Ctr+O, hit the Enter key, and then exit it using Ctrl+X.
Now, create Kafka systemd File
sudo nano /etc/systemd/system/kafka.service
Paste the following lines. Note – Change the Java_Home, in case you are using some other version. To find it you can use the command –
sudo find /usr/ -name *jdk
[Unit] Description=Apache Kafka Server Documentation=http://kafka.apache.org/documentation.html Requires=zookeeper.service [Service] Type=simple Environment="JAVA_HOME=/usr/lib/jvm/jre-11-openjdk" ExecStart=/usr/bin/bash /usr/local/kafka/bin/kafka-server-start.sh /usr/local/kafka/config/server.properties ExecStop=/usr/bin/bash /usr/local/kafka/bin/kafka-server-stop.sh [Install] WantedBy=multi-user.target
Save the file with Ctrl+O, press Enter key, and then use Ctrl+X to exit the same.
To reflect the above-made changes into the system and use the service files, reload the system daemon once.
sudo systemctl daemon-reload
5. Start Zookeeper and Kafka Server on Rocky Linux
Now, let’s start and enable both server services to make sure they will also get active even after the system reboot.
sudo systemctl start zookeeper sudo systemctl start kafka
sudo systemctl enable zookeeper sudo systemctl enable kafka
Check services status:
sudo systemctl status zookeeper
sudo systemctl status kafka
6. Create Test Topics on Kafka – Rocky or AlmaLinux
Kafka allows us to read, write, store, and process events across the various machines, however, to store these events we need someplace or folder and that called “Topics“. So on your server terminal create at least one topic using the following command, using the same later you can create as many Topics as you want.
Let’s say our first Topic name is – testevent. So to create the same run:
Go to your Kafka directory.
And use the Topics script:
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic testevent
After creating as many as topics you want, to list them all we can use the following command:
bin/kafka-topics.sh --list --bootstrap-server localhost:9092
7. Write some event using Kafka Producer and read with consumer
Kafka offers two APIs- Producer and Consumer, for both it offers a command-line client. The producer is responsible for creating events and the Consumer uses them to display or reads the data generated by the Producer.
Open Two terminal tabs or sessions to understand the event generator and reader setup in real-time.
#On one first terminal:
To test let’s create some events using the Producer script:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic testevent
Type some text that you want to stream and display on the consumer end.
#On the other terminal
Run, the below-given command along with the topic name to check the messages or generated event data in real-time:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic testevent --from-beginning
• How to Install LAMP on Rocky Linux 8 Server
• How to install phpMyAdmin on Rocky Linux 8 with Apache
• Install Apache, MySQL, and PHP on Rocky or AlmaLinux 8
• How To Install InfluxDB on AlmaLinux or Rocky 8