How to Install Apache Kafka in Windows 10

Apache Kafka is open source distributed event store and stream-processing platform, which ensures high throughput and high availability of data pipeline. To test Kafka in local environment, we can install in our own desktop directly.

1. Download binary file in official website below

Apache Kafka

Scala 2.13 — kafka_2.13–3.3.1.tgz

2. Decompress the file using Windows PowerShell

tar -xvf {file_name}

And change the name of the decompressed folder for convenience. (optional)

3. Start Zookeeper and Kafka broker

In kafka\bin\windows directory, we can find producer, consumer, zookeeper …etc batch files to be executed. Zookeeper helps maintaining distributed application, especially Kafka brokers and topics.

// terminal 1
.\zookeeper-server-start.bat ..\..\config\zookeeper.properties

// terminal 2
.\kafka-server-start.bat ..\..\config\server.properties

// using --daemon option, the two applications can run in a single terminal

Default port numbers of Zookeeper and Kafka are2181, 9092 respectively. We can check the two applications are executed, using command below.

4. Test: Create topic

Briefly looking overall structure of Kafka, producer applications store data in topic inside Kafka broker server, and consumer applications get the data by subscribing the topic. Among the many types of producer/consumer applications, we can use console producer/consumer to store and read the data stored in topic.

// terminal 3
./kafka-topics.bat --create --bootstrap-server localhost:9092 --topic {topic_name}

It is important to follow kafka topic naming convention because producer/consumer applications are often managed by different application managers. By explicitly showing the contents of topic through its name, developers can access appropriate topics and data stored in expected format.

ex. <message type>.<dataset name>.<data name> / <app type>.<app name>.<dataset name>.<stage of processing> …

cf. test.es.json : data will be used to test elastic search connector, in json format

5. Test: Execute console producer and console consumer

// terminal 3
.\kafka-console-producer.bat --broker-list localhost:9092 --topic {topic_name}

Insert data in json format to the topic. (test.es.json)

// terminal 4
.\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic {topic_name} --from-beginning

Consumer shows data in order (because partition number is 1), from the beginning offset. ( — from-beginning option)

Kafka topics can be connected to various types of source and destination applications using connectors. In the next few posts, I will implement data pipelines using some applications, Kafka Streams, and try to garauntee high availability (partitioning) by building a cluster with multiple brokers in a docker environment.