Apache Kafka is a powerful distributed event streaming platform capable of handling high-throughput, fault-tolerant, and scalable real-time data processing. Initially developed by LinkedIn and later open-sourced under the Apache Foundation, Kafka is widely adopted across industries to manage large-scale data streams in real-time. It serves as a central hub for processing and distributing data, allowing businesses to integrate various systems and applications seamlessly.

In DevOps, Kafka plays a crucial role in streamlining infrastructure management and automation. It is often used to facilitate continuous integration and continuous deployment (CI/CD) pipelines by ensuring smooth communication between microservices, logging events, and handling real-time monitoring. Kafka’s high scalability ensures it can handle vast amounts of log data, metrics, and events across multiple systems, making it ideal for DevOps teams managing complex and dynamic cloud-native architectures.

In the realm of Streaming & Messaging, Kafka stands out as an event-driven platform that supports the reliable and efficient transport of messages between different applications or services. Kafka can process millions of events per second with low latency, making it essential for applications that require real-time data ingestion and processing. It works by storing streams of records in categories called “topics” and allows consumers to read these streams at their own pace, providing flexibility for different use cases. Kafka also enables event-driven architectures, which is increasingly popular in modern microservices, IoT systems, and cloud-native applications.

Kafka’s ability to decouple systems, ensuring that data producers and consumers can operate independently, also makes it highly beneficial in integrating heterogeneous systems. By using Kafka, organizations can centralize data flow across various platforms, ensuring real-time access to critical data across multiple applications and services. Additionally, Kafka’s robust ecosystem, including stream processing frameworks like Kafka Streams and ksqlDB, enables the development of powerful analytics solutions directly on top of the streaming data, further enhancing its utility in Analytics and Monitoring & Diagnostics applications.

To subscribe to this product from Azure Marketplace and initiate an instance using the Azure compute service, follow these steps:

1. Navigate to Azure Marketplace and subscribe to the desired product.
2. Search for “virtual machines” and select “Virtual machines” under Services.
3. Click on “Add” in the Virtual machines page, which will lead you to the Create a virtual machine page.
4. In the Basics tab:

• • Ensure the correct subscription is chosen under Project details.
  • Opt for creating a new resource group by selecting “Create new resource group” and name it as “myResourceGroup.”

5. Under Instance details:

• • Enter “myVM” as the Virtual machine name.
  • Choose “East US” as the Region.
  • Select “Ubuntu 18.04 LTS” as the Image.
  • Leave other settings as default.

6. For Administrator account:

• • Pick “SSH public key.”
  • Provide your user name and paste your public key, ensuring no leading or trailing white spaces.

7. Under Inbound port rules > Public inbound ports:

• • Choose “Allow selected ports.”
  • Select “SSH (22)” and “HTTP (80)” from the drop-down.

8. Keep the remaining settings at their defaults and click on “Review + create” at the bottom of the page.
9. The “Create a virtual machine” page will display the details of the VM you’re about to create. Once ready, click on “Create.”
10. The deployment process will take a few minutes. Once it’s finished, proceed to the next section.

To connect to the virtual machine:

1. Access the overview page of your VM and click on “Connect.”
2. On the “Connect to virtual machine” page:

• • Keep the default options for connecting via IP address over port 22.
  • A connection command for logging in will be displayed. Click the button to copy the command. Here’s an example of what the SSH connection command looks like:
    “`
    ssh [email protected]
    “`

3. Using the same bash shell that you used to generate your SSH key pair, you can either reopen the Cloud Shell by selecting >_ again

or going to https://shell.azure.com/bash.
4. Paste the SSH connection command into the shell to initiate an SSH session.

Usage/Deployment Instructions

Anarion Technologies – Apache Kafka

Note: Search product on Azure marketplace and click on “Get it now”

Click on Continue

Click on Create

Creating a Virtual Machine, enter or select appropriate values for zone, machine type, resource group and so on as per your choice.

After Process of Create Virtual Machine. You have got an Option Go to Resource Group 

Click Go to Resource Group

Copy the Public IP Address

SSH into Terminal and Run these Commands:

$ sudo su
$ sudo apt update
$ cd ../..

Start Zookeeper

Kafka requires Zookeeper to manage its cluster. Start it:

$ cd /usr/local/kafka
$ bin/zookeeper-server-start.sh config/zookeeper.properties

Start Kafka Broker

Open a new terminal and start the Kafka broker:

$ cd /usr/local/kafka
$ bin/kafka-server-start.sh config/server.properties

Create a topic:

$ bin/kafka-topics.sh –create –topic test –bootstrap-server localhost:9092 –partitions 1 –replication-factor 1

List topics:

$ bin/kafka-topics.sh –list –bootstrap-server localhost:9092

Produce messages:

$ bin/kafka-console-producer.sh –topic test –bootstrap-server localhost:9092

Open a new terminal :

$ bin/kafka-console-consumer.sh –topic test –from-beginning –bootstrap-server localhost:9092

After Working on Apache Kafka. Close all your Terminals.

Thanks!!!