Kafka REST-Proxy

Copyright Laso | Adarsha.Lunia

What is REST-Proxy

The Kafka REST Proxy provides a RESTful interface to interact with Kafka clusters, enabling users to produce and consume messages using HTTP-based methods.

Key features of the Kafka REST Proxy include:

1. HTTP Interface: The REST Proxy exposes a RESTful HTTP interface, allowing clients to interact with Kafka using standard HTTP methods such as GET, POST, and DELETE.
2. Producer API: Clients can use the REST Proxy to produce (send) messages to Kafka topics. They can send messages in various formats, including JSON, Avro, or binary.
3. Consumer API: The REST Proxy supports consuming (retrieving) messages from Kafka topics. Consumers can subscribe to topics and retrieve messages in real-time.
4. Schema Registry Integration: For Avro data, the REST Proxy integrates with the Confluent Schema Registry. This allows users to define and evolve schemas for their data, ensuring compatibility between producers and consumers.
5. Security: The REST Proxy supports authentication and authorization mechanisms to secure access to Kafka clusters. It can integrate with various security features provided by Kafka.
6. Compatibility: The Kafka REST Proxy is designed to be compatible with different programming languages and platforms, making it accessible to a wide range of clients.

The Kafka REST Proxy is particularly useful in scenarios where applications or services need to interact with Kafka but are unable to use the native Kafka clients due to language or platform constraints. It provides a straightforward way for web-based or non-JVM-based applications to integrate with Kafka.

Here’s a simple example of using the Kafka REST Proxy to produce a message:

# Produce a message to a Kafka topic using curl
curl -X POST -H "Content-Type: application/vnd.kafka.json.v2+json" \
     --data '{"records":[{"value":{"name":"John","age":30}}]}' \
     http://localhost:8082/topics/my-topic

# Consume messages from a Kafka topic using curl
curl -X GET -H "Accept: application/vnd.kafka.json.v2+json" \
     http://localhost:8082/consumers/my-consumer-group/instances/my-consumer-instance/records