DEV Community

Cover image for Study Notes 6.5-6: Kafka Producer, Consumer & Configuration
Pizofreude
Pizofreude

Posted on

Study Notes 6.5-6: Kafka Producer, Consumer & Configuration

#dataengineering #dezoomcamp #kafka #streaming

1. Overview of Kafka Producer & Consumer

  • Objective: Learn how to produce and consume messages in Apache Kafka programmatically using Java.
  • Context:
    • The video demonstrates both producing messages (from a CSV file of ride data) and consuming them via Java.
    • It contrasts Java’s well-supported Kafka libraries with Python’s more limited options, while still noting that Python examples exist.
  • Real-World Relevance: Understanding these concepts is vital for building robust stream processing pipelines and real-time data applications.

2. Setting Up Kafka Topics

  • Topic Creation:

    • Topics are created through the Confluent Cloud (or a similar UI) interface.

    • Configuration Options:

      • Partitions: The example uses 2 partitions.

        Partitions allow Kafka to scale horizontally and provide parallelism in message processing.

      • Retention Policy: Set to one day to manage storage costs and avoid excess data accumulation.

      • Cleanup Policy: The “delete” policy is used, which automatically purges old messages.

  • Why It Matters:

    Proper topic configuration is essential for ensuring high throughput, scalability, and cost management.

3. Implementing a Kafka Producer in Java

a. Core Concepts

  • Producer Role:
    • Reads data from an external source (e.g., a CSV file containing ride data).
    • Converts CSV data into a structured format (e.g., a “Rights” object) and publishes each record to a Kafka topic.
  • Java Advantages:
    • Kafka’s Java client libraries are well maintained and offer robust support for serialization/deserialization, error handling, and configuration management.

b. Key Configuration Properties

  • Bootstrap Servers:
    • Specifies the Kafka cluster’s connection endpoint.

  • Serializers:
    Example configuration snippet:

    • Key Serializer: Typically a String serializer.
    • Value Serializer: Uses a JSON serializer (often provided by Confluent) to convert objects to JSON format. 
    Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "your-bootstrap-server");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringSerializer");
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "io.confluent.kafka.serializers.KafkaJsonSerializer");

c. Producer Workflow

  • Reading Data:
    • Use a CSV reader to ingest ride data.
  • Object Mapping:
    • Convert CSV lines into Java objects (e.g., instances of a “Rights” class).
  • Publishing:
    • Create a Kafka Producer instance and publish each record to the specified topic.
  • Error Handling:
    • Include try-catch blocks and logging to handle issues during message production.

4. Implementing a Kafka Consumer in Java

a. Core Concepts

  • Consumer Role:
    • Subscribes to one or more topics.
    • Reads messages and converts the JSON back into Java objects.
  • Configuration Focus:
    • Differences from the producer mainly revolve around deserialization and consumer-specific settings.

b. Key Configuration Properties

  • Consumer Group ID:
    • A unique identifier for the consumer group that helps manage offset commits.
  • Deserializers:
    • Key Deserializer: Typically a String deserializer.
    • Value Deserializer: Uses a JSON deserializer. Note that the deserializer must be aware of the target Java class (e.g., “Rights”).

  • Auto Offset Reset:

    • Set to “earliest” if there is no initial offset to ensure the consumer starts reading from the beginning of the topic.

    Example configuration snippet:

    Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "your-bootstrap-server");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "your-consumer-group");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "io.confluent.kafka.serializers.KafkaJsonDeserializer");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
// Additional configuration to bind the deserializer to the specific type:
props.put("json.value.type", "com.yourpackage.Rights");

c. Consumer Workflow

  • Subscription:
    • Subscribe to the topic (e.g., “rights”) using the consumer API.
  • Polling:
    • Use a loop to continuously poll for new messages.
  • Processing:
    • Deserialize the JSON messages back into the corresponding Java objects.
  • Commit Offsets:
    • Ensure that offsets are committed so that the consumer knows where to resume on restart.
  • Common Issues:
    • Type mismatch errors if the JSON deserializer is not properly configured with the target type. This was addressed by explicitly setting the JSON value type.

5. Common Pitfalls and Best Practices

a. Serialization/Deserialization Mismatches

  • Symptom:
    • Errors like “LinkedHashMap cannot be cast to…” indicate that the consumer is not correctly configured to deserialize JSON into the expected Java object.
  • Solution:
    • Always set the JSON value type in the consumer configuration.

b. Offset Management

  • Auto Offset Reset:
    • Setting this to “earliest” ensures that when there is no committed offset, the consumer will read from the beginning of the topic.
  • Consumer Group Coordination:
    • Understand the implications of consumer group IDs, especially when scaling out consumption.

c. Environment Differences

  • Development vs. Production:
    • The demo uses Confluent Cloud for ease of setup. In production, consider a more robust configuration for reliability, monitoring, and security.
  • Language Ecosystem:
    • While Java is preferred for its maturity, Python can be used for prototyping; just be aware of its limitations with Kafka.

1. Overview of Kafka Configuration

Objective:

Understand Kafka’s configuration parameters and underlying concepts to design reliable, scalable, and efficient stream-processing architectures.

Key Topics Covered:

  • Kafka cluster architecture
  • Topic and message structure
  • Replication, retention, and partitioning
  • Consumer groups, offsets, and auto offset reset
  • Producer acknowledgment mechanisms

2. Kafka Cluster Fundamentals

a. Kafka Cluster Definition

  • Definition: A Kafka cluster is a collection of machines (nodes) running Kafka that work together. These nodes communicate via Kafka’s own protocols.
  • Past vs. Present: Earlier Kafka used ZooKeeper to store metadata about topics and partitions. Modern Kafka no longer relies on ZooKeeper and instead manages this metadata internally.

b. Node Roles and Communication

  • Nodes: Each node in the cluster can serve as a broker that stores topic data and serves client requests.
  • Leader and Followers: For a given partition, one broker acts as the leader (handling all read/write requests), while others (followers) replicate the data for fault tolerance.

3. Topics and Messages

a. Topic Structure

  • Definition: A topic is a named stream of records (messages) that can be produced to and consumed from.
  • Message Components:
    • Key: Often used for partitioning (e.g., vendor ID in ride data).
    • Value: The actual payload (commonly a JSON string with detailed information).
    • Timestamp: A long value indicating when the event was produced or recorded.

b. Importance of Topics

  • Topics provide the logical separation of data streams and help in organizing and managing data pipelines efficiently.

4. Replication

a. Purpose of Replication

  • Reliability: Replication ensures that if one node fails, data is not lost because copies exist on other nodes.
  • Replication Factor: The number of copies (replicas) of a topic’s partition maintained across the cluster.
    • Example: With a replication factor of 2, if the leader node fails, a follower can be promoted to leader, ensuring continuity.

b. Leader–Follower Mechanism

  • Process:
    • Producers send data to the leader.
    • The leader writes the data to its log and then replicates it to follower nodes.
  • Failover: If the leader becomes unavailable, one of the followers automatically becomes the new leader, minimizing downtime.

5. Retention Policies

a. Retention Definition

  • Retention: Determines how long Kafka stores messages before they are deleted.
  • Configuration: Set in terms of time (e.g., one day, three days) to balance between disk usage and the need for historical data.

b. Practical Considerations

  • High-Volume Use Cases: Shorter retention may be set for high-throughput topics to conserve storage.
  • Audit and Debugging: Longer retention can be useful if historical data is needed for auditing or troubleshooting.

6. Partitioning

a. What Are Partitions?

  • Definition: Partitions divide a topic into multiple segments, allowing Kafka to scale horizontally.
  • Functionality:
    • Each partition is an ordered, immutable sequence of records.
    • Partitions enable parallel processing since each partition can be consumed independently.

b. Impact on Consumption and Scaling

  • Consumer Assignment: Within a consumer group, each partition is consumed by only one consumer at a time.
  • Scaling Out: Increasing the number of partitions allows you to add more consumers to distribute the load.
  • Trade-offs: More partitions may increase overhead on the Kafka brokers, so it’s important to balance parallelism with resource utilization.

7. Consumer Groups and Offsets

a. Consumer Group ID

  • Definition: A unique identifier that groups multiple consumers. Consumers with the same group ID share the work of reading from partitions.
  • Load Balancing: Kafka assigns each partition in a topic to one consumer within a consumer group, ensuring that messages are processed in parallel without duplication.

b. Offsets

  • Definition: Offsets are numerical identifiers assigned to each message within a partition, representing its position.
  • Offset Management:
    • Consumers commit offsets to indicate the last processed message.
    • Kafka stores these in an internal topic (commonly __consumer_offsets), allowing consumers to resume from where they left off after a restart.

c. Auto Offset Reset Policy

  • Latest (Default): New consumer groups start reading only new messages produced after their subscription.
  • Earliest: Configuring auto offset reset to “earliest” allows new consumers to read all available messages from the beginning of the topic.
  • When to Use:
    • Latest: When you only care about real-time processing.
    • Earliest: When you need to process historical data or ensure no messages are missed.

8. Producer Acknowledgments

a. Acknowledgment Options

  • Ack=0 (Fire and Forget):
    • The producer sends the message without waiting for confirmation.
    • Pros: Low latency, high throughput.
    • Cons: Risk of message loss if the leader fails.
  • Ack=1:
    • The producer waits for the leader to acknowledge the message.
    • Trade-off: Faster than waiting for all replicas, but if the leader fails before replication, data loss is possible.
  • Ack=all (or -1):
    • The producer waits for acknowledgments from all in-sync replicas.
    • Pros: Guarantees durability and minimizes the risk of message loss.
    • Cons: Slightly increased latency due to waiting for multiple acknowledgments.

b. Choosing the Right Acknowledgment Level

  • Critical Data (e.g., financial transactions): Use ack=all to ensure reliability.
  • Non-Critical Data (e.g., system metrics): Ack=0 or ack=1 may suffice, prioritizing performance over absolute durability.

9. Advanced Configuration and Best Practices

a. Fine-Tuning Kafka

  • Topic-Level Configurations:
    • Retention period (retention.ms)
    • Log segment size and flush intervals (segment.bytes, flush.ms)
  • Producer and Consumer Settings:
    • Compression types (e.g., gzip, snappy) to reduce network load.
    • Security settings such as SSL/SASL for encryption and authentication.
  • Monitoring: Use tools like Confluent Control Center, Prometheus, or Grafana to monitor broker health, consumer lag, and throughput.

b. Design Considerations for Production

  • Scalability: Balance the number of partitions with the consumer group size to ensure optimal parallelism.
  • Reliability: Configure replication factors and acknowledgment levels based on the criticality of your data.
  • Cost Efficiency: Adjust retention policies and resource allocations (e.g., broker capacity) based on workload patterns.
  • Documentation and Testing: Always refer to the latest Apache Kafka documentation for configuration changes and thoroughly test your settings in a staging environment before production deployment.

c. Additional Resources

  • Apache Kafka Documentation:Kafka Documentation
  • Confluent Resources:Confluent Docs
  • Books & Courses:Kafka: The Definitive Guide by Neha Narkhede, Gwen Shapira, and Todd Palino
  • Community: Engage with communities on StackOverflow, Reddit (r/dataengineering), and Confluent Community Slack.

Top comments (0)