Kafka: Introduction

 KAFKA

• Kafka can be used to send and receive messages in microservices. 
• It is a middleware and can handle millions of messages. 
• Azure service bus and Rabbit MQ are substitute messaging service to Kafka. 
• Kafka work on Topic- Subscription model of messages not on queues. 
• Kafka was created by Linked in. 
• Kafka has concept of publisher, Broker, Consumer
• Broker - Kafka cluster and kafka ecosystem. 
• Publisher, Consumer - Microservices 
• We also have to up and run zookeeper and its main purpose is
• Maintain list of topics and messages 
• Track status of nodes in distributed system
• Manage Kafka brokers on distributed system. 

=================================================================================

How to work with KAFKA and use it to communicate between .net core microservices. 

Step1. Install Java if not already install on server. check by java -version

Step2. Download Kafka and unzip it. let say folderA

Step3. Download Zookeeper and unzip it. let say Folder B

Step4. Start Zookeeper. Go to bin and double click on exe file. We can start it with cmd also. 

Step5. Start Kafka. with CMD which will act as a broker. 

Step6. Create a Topic

Step7. Create a publisher. it could be console application or a Microservice

Step8. Create a subscriber.  it could be console application or a Microservice

===============================================================================

.net Microservice to create a publisher and a subscriber  

Publisher

Step1. Install nuget pakcage "Confluent.kafka" in api project created on .net core. which we will treat as microservice. 

Step2. In appsettings.json create producer by mentioning bootstrap server.

    value of bootstrap server like below :

BootstrapServers = "localhost:9092"

Step3. Add above producer in Starup.configureservice method. .

Step4. Create an API controller do some coding like below:

                await producer.Produce.Async(topic, message); 

Where message is json serialized object. 

Step5. now if you send something from postman then that thing must come in above api method. 

Post:

http://localhost/api/producer/send?topic=temp-topic

{"id":1,"name":"test"}

Subscriber

1. Create a microservice and install the above nuget package. 

2. in main.cs subscribe to the same topic mentioned above. 

=============================================================================