Kafka Free Message Queue Software
Scala & Java based distributed event streaming platform
Apache Kafka is an open source distributed event streaming platform. It is a robust queue broker and built as an internal messaging system developed by Linked-in
Overview
Kafka is an open source messaging system and a robust queue broker. It is distributed event streaming platform and has the ability to handle a high volume of messages. Kafka messages are stored on the disk and it allows you to send messages from one point to another seamlessly. Messages are replicated within the whole Kafka cluster to prevent unwanted operations from happening like any data loss. Kafka messaging platform built to handle real-time event streaming, pipe-lining and replaying of data for fast, scalable operations.
Apache Kafka distributed message queue software is used by thousands of companies for high performance data pipelines and its integration with Apache Storm and Spark. Kafka offers high performance as compared to message and queue brokers like ActiveMQ and RabbitMQ etc. Apache Kafka is an alternative to a variety of enterprise messaging systems. It was built as an internal messaging system developed by Linked-in to handle 1.4 trillion messages in a day. It is a best and suitable platform for the implementation of Queues since it boosts performance by using sequential disk I/O operations. It is also a perfect choice for big data use cases because it can achieve high throughput with limited number of resources, i.e millions of messages per second. Kafka open source software has 19.4K GitHub stars and 10.3k GitHub forks.
System Requirements
Requirements to setup Apache Kafka software include:
- Java 8+
- ZooKeeper
- Ubuntu 20.04 LTS
- Git
Features
Some of the key features of Apache Kafka are listed below:
- Scalability
- High-Volume
- Data Transformations
- Fault Tolerance
- Reliability
- Durability
- Performance
- Zero Downtime
- Extensibility
- Replication
- Open Source
Installation Instructions
Install Apache Kafka on Ubuntu
This guide explains how to setup and Kafka. Below installation steps assume that all the depency packages of Kafka are installed and up to date on your system. Please follow below installation steps. Get kafka by downloading the latest release Kafka and extract it with commands:
tar -xzf kafka_2.13-2.8.0.tgz
cd kafka_2.13-2.8.0
Next, start the kafka ENVIRONMENT. You local system environment must have Java 8+ installed. Execute the following commands in order to start all services in the correct order:
bin/zookeeper-server-start.sh config/zookeeper.properties
Open another terminal session and dtart the Kafka broker service by:
bin/kafka-server-start.sh config/server.properties
when all services have successfully installed, you will have a basic Kafka environment running and ready to access. You need to create a topic before writing your first event. Open another terminal session and run command:
bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092
Now, run the console producer client to write a few separate events into the topic:
bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
Open another console terminal session and run the console consumer client to read the events you just created with:
bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
You can continuously import/export your data into and out of Kafka. Use Ctrl-C to stop the Kafka broker. If you also want to delete any data from your local Kafka environment including any events you have created along the way then run the command:
rm -rf /tmp/kafka-logs /tmp/zookeeper
Congratulations! You have successfully configured Apache Kafka platform on Ubuntu. Enjoy!
FAQs
What is Apache Kafka used for?
Kafka is an open source software which provides a framework for storing, reading and analysing streaming data. Being open source means that it is essentially free to use and has a large network of users and developers who contribute towards updates, new features and offering support for new users.
Is Apache Kafka free?
Apache Kafka is free, and Confluent Cloud is very cheap for small use cases, about $1 a month to produce, store, and consume a GB of data.
Is Apache Kafka open source?
Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. The source code of Ceph application is available at Github.
What language is Kafka written in?
Kafka started as a project in LinkedIn and was later open-sourced to facilitate its adoption. It is written in Scala and Java, and it is part of the open-source Apache Software Foundation.
Why Kafka is so fast?
Compression & Batching of Data: Kafka batches the data into chunks which helps in reducing the network calls and converting most of the random writes to sequential ones. It’s more efficient to compress a batch of data as compared to compressing individual messages.
