Apache Kafka has become a leading choice for building streaming data platforms, and like many modern data systems, it operates as a distributed system. In this hands-on workshop, participants will dive deep into the core building blocks of Apache Kafka, exploring these components as recurring patterns common across distributed architectures.

By incrementally building a simplified version of Kafka from the ground up, attendees will gain practical insights into how Kafka works, while also learning foundational concepts that apply to other distributed systems. This approach demystifies the complexity of Kafka and provides a transferable understanding that can be applied to other distributed data systems.

Whether you're new to distributed architectures or looking to deepen your understanding, this workshop will provide valuable, actionable insights.

The rough outline of the workshop is as follows:

1. Understand the Need for Distribution

• Understanding the effects of infrastructure saturation.
• A look at the Probability of Failures and its impact on system availability.
• Using Distribution for masking failures.

2. Group Membership and Cluster Metadata Management

• Use Apache Zookeeper Consistent Core for handling group membership
• Understand how to detect failure detection using Lease.

3. Partitioning

• Implement a partitioning strategy using Fixed Partitions.

4. Cluster Coordination with Controller

• Implement a Controller using State Watch.

5. Network IO for Brokers

• Implement a basic network IO to communicate with message brokers.

6. Append Only Storage

• Implement storage using an append only storage similar to Write Ahead Log.

7. Replication with Leader and Followers.

• Implement replication with Leader and Followers.

8. Replication Consistency - HighWatermark.

• Manage replica consistency with High-Watermark.

9. Handling partial failures with Generation Clock.

• Manage partial failure of Controller and Leaders with Generation Clock.

10. Producers and Consumers

• Implement simple Producers and Consumers to interact with our Kafka.

Apache Kafka has become a leading choice for building streaming data platforms, and like many modern data systems, it operates as a distributed system. In this hands-on workshop, participants will dive deep into the core building blocks of Apache Kafka, exploring these components as recurring patterns common across distributed architectures.

By incrementally building a simplified version of Kafka from the ground up, attendees will gain practical insights into how Kafka works, while also learning foundational concepts that apply to other distributed systems. This approach demystifies the complexity of Kafka and provides a transferable understanding that can be applied to other distributed data systems.

Whether you're new to distributed architectures or looking to deepen your understanding, this workshop will provide valuable, actionable insights.

The rough outline of the workshop is as follows:

1. Understand the Need for Distribution

• Understanding the effects of infrastructure saturation.
• A look at the Probability of Failures and its impact on system availability.
• Using Distribution for masking failures.

2. Group Membership and Cluster Metadata Management

• Use Apache Zookeeper Consistent Core for handling group membership
• Understand how to detect failure detection using Lease.

3. Partitioning

• Implement a partitioning strategy using Fixed Partitions.

4. Cluster Coordination with Controller

• Implement a Controller using State Watch.

5. Network IO for Brokers

• Implement a basic network IO to communicate with message brokers.

6. Append Only Storage

• Implement storage using an append only storage similar to Write Ahead Log.

7. Replication with Leader and Followers.

• Implement replication with Leader and Followers.

8. Replication Consistency - HighWatermark.

• Manage replica consistency with High-Watermark.

9. Handling partial failures with Generation Clock.

• Manage partial failure of Controller and Leaders with Generation Clock.

10. Producers and Consumers

• Implement simple Producers and Consumers to interact with our Kafka.