DEV Community

Code Green
Code Green

Posted on • Edited on

2

What is Apache Kafka? explain architecture of Kafka

Apache Kafka

============

Apache Kafka is an open-source distributed event streaming platform used for building real-time data pipelines and streaming applications.

Architecture of Kafka

Kafka has a distributed architecture consisting of the following components:

  • Producer: Applications that produce data and send it to Kafka topics.
  • Broker: Kafka runs as a cluster of one or more servers called brokers, which store and manage the data.
  • Topic: A category or feed name to which messages are published by producers.
  • Partition: Topics are divided into partitions for scalability and parallelism.
  • Consumer: Applications that subscribe to topics and consume the data.
  • ZooKeeper: Coordinates and manages Kafka brokers.

The architecture allows for horizontal scaling, fault tolerance, and high throughput.

Architecture of Kafka

Image of Timescale

🚀 pgai Vectorizer: SQLAlchemy and LiteLLM Make Vector Search Simple

We built pgai Vectorizer to simplify embedding management for AI applications—without needing a separate database or complex infrastructure. Since launch, developers have created over 3,000 vectorizers on Timescale Cloud, with many more self-hosted.

Read more →

Top comments (0)

Billboard image

The Next Generation Developer Platform

Coherence is the first Platform-as-a-Service you can control. Unlike "black-box" platforms that are opinionated about the infra you can deploy, Coherence is powered by CNC, the open-source IaC framework, which offers limitless customization.

Learn more