Opening Note We all make mistakes, and fundamentally, the havoc caused by this incident was due to a flaw in the design of rdkafka-ruby. While the disappearance of librdkafka from GitHub was unexpected, this article aims to clarify and explain how rdkafka-ruby should have prevented it and what was poorly designed. By examining this incident, I hope to provide insights into better practices for managing dependencies and ensuring more resilient software builds for the Ruby ecosystem. Incident Summary On July 10, 2024 15:47 UTC, users of the rdkafka gem faced issues when the librdkafka repository ..read more

As an open-source developer, I constantly seek performance gains in the code I maintain. Since I took over rdkafka from AppSignal in November 2023, I promised not only to maintain the gem but to provide a stream of feature improvements and performance enhancements. One key area where performance can often be improved is how synchronization is handled in synchronous operations. This article discusses our significant improvement in rdkafka by replacing sleep with condition variables and mutexes. rdkafka-ruby (rdkafka for short) is a low-level driver used within the Karafka ecosystem to communica ..read more

Introduction I am thrilled to announce the release of Karafka 2.4, a significant milestone in the evolution of my Ruby and Rails multi-threaded efficient Kafka processing framework. This release builds upon the solid foundation laid by its predecessor, Karafka 2.3. It introduces many new features, enhancements, and improvements across the entire Karafka ecosystem, including Karafka, Web UI and WaterDrop. In this article, I will discuss the most significant changes and enhancements introduced in Karafka 2.4. Please note that this is not an exhaustive list of all the changes but rather a curated ..read more

Introduction The journey towards efficient parallelization in library development has often been based on using threads. As Karafka celebrates its eighth anniversary, it's become clear that while threads have served us well for many tasks, there's room to explore further. That's why I've decided to introduce forking capabilities into Karafka, aiming to offer another dimension of parallelization to its users. This move isn't about replacing threads but about providing options to cover a broader spectrum of use cases than before. For those who wonder what Karafka is, Karafka is a Ruby and Rails ..read more

Introduction Leaving Cracow's familiar scenes behind, I headed to Warsaw with anticipation for the Ruby Warsaw Community Conference. The compact yet promising event marked a day dedicated to Ruby enthusiasts like myself. Below, you can find my after-thoughts about this peculiar event. Speaker's Dinner and Before Party The speakers' dinner and pre-party set the tone for the conference, offering a warm, inviting atmosphere. Both venues, a well-chosen restaurant and a lively before-party spot, facilitated great conversations and networking among the attendees. Conference Overview The Ruby Warsaw ..read more

Introduction I'm happy to announce that Karafka 2.3 and its Web UI 0.8 have just been released. For those who wonder what Karafka is, Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework. The Karafka 2.3 release builds upon the foundation set by its predecessor, 2.2, making it a seamless continuation rather than a major rewrite. This means upgrading from version 2.2 to 2.3 can be done without extensive work or significant modifications to the existing codebases. These releases introduce many new features. Please note that I have described only the most significant ch ..read more

While officially End-of-Life (EOL), Ruby 2.7 remains critical in many ongoing projects. Despite its EOL status, a significant user base continues to rely on this version for various reasons, ranging from legacy system compatibility to specific feature dependencies. As a developer of Karafka, an open-source software (OSS), I recognize the importance of supporting Ruby 2.7, giving users more time to upgrade than the EOL time. This commitment is reflected in my integration tests that ensure compatibility with Ruby 2.7. However, a recent update has posed a challenge. On Friday, 15th of December 20 ..read more

Introduction Karafka is a Ruby and Rails framework designed to simplify processing messages consumed from Apache Kafka. One of Karafka's components is the https://karafka.io/docs/Web-UI-Getting-Started/. It provides a convenient way for developers to monitor and manage their Karafka-based applications without using the command line or third-party software. The interface, amongst others, includes: historical metrics, real-time aggregated metrics, real-time information on resource usage, errors details, performance statistics, stale partitions detection (LSO hangs), routing pattern matched topi ..read more

Introduction Managing frontend assets in gems serving Web UI can be tricky. Why? Because while you want assets to expire across versions, you don't want them fetched repeatedly, slowing things down. This challenge popped up when working on the Karafka Web UI, which runs on the Roda framework. In this article, I'll explore how I addressed this, ensuring smooth updates without compromising speed by elevating Roda and its plugins ecosystem. Brief on the Roda web framework. Roda is a versatile web framework for the Ruby programming language. Crafted by Jeremy Evans, Roda is renowned for its excep ..read more

Karafka is a Ruby and Rails framework that simplifies the development of Apache Kafka-based applications. Among its varied features, the Filtering API provides enhanced control over the data flow. The crux of this article is about managing offsets - unique identifiers for messages within Kafka's partitions. Often, there's a need to manage offsets alongside database operations within a transaction, especially when handling potential process crashes and anomalies, minimizing the risk of double processing. For instance, if a SQL operation concludes successfully but the offset commit fails due to ..read more