DEV Community

Cover image for How to Document Kafka APIs: Topics, Schemas and Contracts
Hassann
Hassann

Posted on Originally published at apidog.com

How to Document Kafka APIs: Topics, Schemas and Contracts

To document Kafka APIs, write a contract for every topic. It should name the topic, the event key, the payload schema, the headers, the ordering and delivery guarantees, the consumer groups, the retention period and the owner. Then store that contract next to your HTTP API docs, so producers and consumers read one source of truth instead of reverse-engineering messages from a live cluster.

Try Apidog today

This guide covers each part of that contract, how to write it in AsyncAPI, how to evolve schemas safely, and how to let your coding agent do the work with Apidog CLI.

If you're new to event-driven design, start with how to build event-driven APIs and the guide to webhooks and event-driven architecture.

What is a Kafka API?

A Kafka API is the set of topics, message formats and delivery rules that services use to exchange events through an Apache Kafka cluster. Unlike a REST API, nobody calls an endpoint and waits for a response. Producers write records to topics, consumers read them on their own schedule, and the contract lives in the shape and meaning of those records.

The Kafka documentation puts it plainly: "an event has a key, value, timestamp, and optional metadata headers." Kafka itself doesn't check what's inside the value. A producer can write JSON one day and a different JSON shape the next, and the broker will accept both. That's why Kafka APIs need written contracts more than HTTP APIs do: nothing in the transport stops a breaking change.

Undocumented topics fail in predictable ways: a consumer guesses a field type from one sample and crashes on null, or a new service doesn't know it gets each message at least once and double-charges on retries.

What to document for each Kafka topic

Every topic needs the same core facts. Use this table as your checklist:

Contract field What to write down Example
Topic name Full name and naming convention orders.order-placed.v1
Purpose One sentence on what the event means "An order passed payment and is ready to ship"
Event key Which field is the key, and why customerId, so one customer's orders stay in order
Payload schema Every field, type, required flag and format JSON Schema, Avro or Protobuf
Headers Name, type and meaning of each header traceparent, schema-version
Partitions and ordering Partition count and the ordering promise 12 partitions, ordered per customerId
Delivery semantics At least once, idempotent producer, transactions At least once; consumers must deduplicate on orderId
Consumer groups Known groups and what each one does billing-service, email-notifier
Retention How long records stay readable 7 days
Ownership Owning team, contact and version policy Checkout team, #checkout-events

The ordering and delivery rows matter most: readers can't infer them from a sample message.

Kafka topic contract diagram: a producer writes keyed events to a partitioned topic read by two consumer groups, with the contract fields alongside

How to document Kafka APIs in 6 steps

  1. List every topic and its owner. Pull the topic list from your cluster and mark who produces to each one.
  2. Define the key. Write down which field is the key and what that means for ordering.
  3. Write the payload schema. Describe every field with JSON Schema, Avro or Protobuf.
  4. Describe headers and metadata. Include tracing, schema version and content type headers.
  5. State the guarantees. Document delivery semantics, ordering scope, retention and consumer groups.
  6. Publish it where people look. Put the contract next to your HTTP API docs and version it like code.

Steps 1 and 2: topics and keys

A naming pattern like <domain>.<event>.<version> tells readers who owns the topic and which version they're reading. Then document the key, because it decides the partition. Kafka's docs state that "events with the same event key (e.g., a customer or vehicle ID) are written to the same partition," and that consumers read a partition's events "in exactly the same order as they were written."

So your contract should say "ordered per customerId," not "ordered." Kafka only promises order inside a partition, never across a whole topic.

Step 3: the payload schema

The payload is the part consumers code against, so describe it field by field. Here's a JSON Schema for an order-placed event:

{
  "type": "object",
  "required": ["orderId", "customerId", "totalCents", "currency", "placedAt"],
  "properties": {
    "orderId": { "type": "string", "description": "Unique order ID; use it to deduplicate" },
    "customerId": { "type": "string", "description": "Also the record key" },
    "totalCents": { "type": "integer", "minimum": 0 },
    "currency": { "type": "string", "enum": ["USD", "EUR", "GBP"] },
    "placedAt": { "type": "string", "format": "date-time" },
    "couponCode": { "type": ["string", "null"] }
  }
}
Enter fullscreen mode Exit fullscreen mode

The description "use it to deduplicate" tells a consumer what to do with the field, not just its type. New to the format? Read what is JSON Schema.

Steps 4 and 5: headers and guarantees

List each header (trace context, schema version, content type) and say whether consumers can rely on it. Then write down the delivery guarantee, in Kafka's own terms. The Kafka design docs state that "Kafka guarantees at-least-once delivery by default," which means messages "are never lost but may be redelivered." If your producer uses the idempotent option or transactions, say so. If it doesn't, tell consumers to deduplicate, and name the field to deduplicate on.

Document consumer groups too. Within each subscribing group, a partition "is consumed by exactly one consumer" at a time. That tells readers how far a group can scale out: one active consumer per partition.

Write the contract in AsyncAPI

Tools need a machine-readable format, and for Kafka that's usually AsyncAPI, a specification "used to describe message-driven APIs in a machine-readable format." It plays the role OpenAPI plays for REST.

An AsyncAPI 3.0 document has info, servers, channels, operations and components. A channel maps to a topic through its address, and each operation has an action of send or receive:

asyncapi: 3.0.0
info:
  title: Orders events
  version: 1.0.0
channels:
  orderPlaced:
    address: orders.order-placed.v1
    messages:
      orderPlaced:
        $ref: '#/components/messages/orderPlaced'
operations:
  publishOrderPlaced:
    action: send
    channel:
      $ref: '#/channels/orderPlaced'
components:
  messages:
    orderPlaced:
      payload:
        $ref: './schemas/order-placed.json'
Enter fullscreen mode Exit fullscreen mode

AsyncAPI also has Kafka bindings at the server, channel, operation and message level. They're where Kafka-specific details go, the ones the generic fields can't express. Keep the spec in the producer's repo, so the pull request that changes an event also changes its contract.

Kafka schema evolution: document the compatibility rule

Your schema will change. The contract should say which changes are safe. If you use a schema registry, document its compatibility mode. Confluent's schema evolution docs list these modes, and note that "the Confluent Schema Registry default compatibility type is BACKWARD":

Mode Who can read what
BACKWARD Consumers on the new schema can read data written with the previous schema
BACKWARD_TRANSITIVE Consumers on the new schema can read data from all earlier schemas
FORWARD Consumers on the previous schema can read data written with the new schema
FORWARD_TRANSITIVE Consumers on any earlier schema can read data written with the new schema
FULL Backward and forward compatible with the previous schema
FULL_TRANSITIVE Backward and forward compatible with all earlier schemas
NONE Compatibility checks are off

In practice, under BACKWARD you can add optional fields and remove fields that had defaults. Renaming a required field or changing its type breaks consumers. When you need a breaking change, publish a new topic version (orders.order-placed.v2), document both, and set a retirement date for the old one.

Let your coding agent document Kafka APIs with Apidog CLI

Hand-written contracts drift the moment someone changes a producer. Your coding agent can read the producer code, write the schemas and keep the docs in sync while you review. Apidog CLI gives Claude Code, Codex, Cursor or any terminal agent the commands to do that in an Apidog project.

Install the CLI, log in and add the Apidog skill so your agent knows the commands:

npm install -g apidog-cli@latest
apidog auth login --with-token <YOUR_TOKEN>
apidog skill install
Enter fullscreen mode Exit fullscreen mode

You can also hand the setup to the agent: tell it to read the Apidog CLI installation guide and set itself up.

Then give your agent one prompt:

Read the Kafka producers in ./services/orders. For every topic they write to,
create an Apidog data schema in project <projectId> for the message payload,
then create one Markdown doc per topic with the key, headers, ordering,
delivery guarantee, consumer groups, retention and owner. Validate every
payload before you write it.
Enter fullscreen mode Exit fullscreen mode

The agent checks the payload shape, validates its file, then writes the schema and the contract page:

apidog cli-schema get schema-create
apidog cli-schema validate schema-create --file order-placed.schema.json
apidog schema create --project <projectId> --file order-placed.schema.json
apidog cli-schema get doc-create
apidog doc create --project <projectId> --file order-placed.doc.json
apidog schema list --project <projectId>
Enter fullscreen mode Exit fullscreen mode

apidog schema create stores the payload as a reusable data schema. Your HTTP endpoints can reference the same model, so an Order defined once covers both the REST response and the Kafka event. apidog doc create writes a Markdown document into the project tree, which holds the topic contract from the table above. When a producer changes, rerun the prompt and the agent uses apidog schema update. For a longer walkthrough of agent-driven doc work, see how to document APIs from the CLI and Apidog CLI in Claude Code.

Workflow diagram: a coding agent reads producer code and runs apidog schema create and apidog doc create into one Apidog project

One step stays in the app. Apidog's Custom Protocol API documents Kafka endpoints alongside HTTP ones: choose New Other Protocol APIs, then Other Custom Protocol API, pick Kafka as the protocol, and fill in the URI (the topic), name, status, maintainers, tags and description. These entries share the folder tree, search and Quick Share with your HTTP docs. The CLI has no command for custom protocol endpoints today, so this is the one part you create by hand.

Apidog Free covers up to 4 users, so a small team can keep its REST and Kafka docs in one project at no cost.

FAQ

Is there an OpenAPI for Kafka?

AsyncAPI fills that role. It describes servers, channels (topics), operations and message payloads in YAML or JSON, with Kafka-specific bindings.

Do I need a schema registry to document Kafka APIs?

No, but it helps. Docs state the rules; a registry enforces them at write time. If you run one, document its compatibility mode.

What's the difference between documenting a Kafka topic and a REST endpoint?

A REST doc describes a request and a response. A Kafka doc describes one message that many consumers read later, so it also needs the key, ordering scope, delivery guarantee, retention and consumer groups. None of those appear in a typical REST doc.

Can Apidog test Kafka messages?

Apidog documents Kafka endpoints through Custom Protocol API, but it doesn't send or consume Kafka messages. Use it to hold the contract and the payload schemas, and test the producer and consumer code with your Kafka client library.

How often should I update Kafka documentation?

Every time a producer changes a message. Tie it to code review: the pull request that changes the event updates the contract, or your agent regenerates it with Apidog CLI.

Top comments (0)