Maciej Mensfeld - Running with Ruby
288 FOLLOWERS
A blog Mostly on Ruby and Rails - but not only! Also a bit of Karafka, Sinatra, Python and Linux admin stuff.
Maciej Mensfeld - Running with Ruby
2M ago
Introduction
The journey towards efficient parallelization in library development has often been based on using threads. As Karafka celebrates its eighth anniversary, it's become clear that while threads have served us well for many tasks, there's room to explore further. That's why I've decided to introduce forking capabilities into Karafka, aiming to offer another dimension of parallelization to its users. This move isn't about replacing threads but about providing options to cover a broader spectrum of use cases than before.
For those who wonder what Karafka is, Karafka is a Ruby and Rails ..read more
Maciej Mensfeld - Running with Ruby
2M ago
Introduction
Leaving Cracow's familiar scenes behind, I headed to Warsaw with anticipation for the Ruby Warsaw Community Conference. The compact yet promising event marked a day dedicated to Ruby enthusiasts like myself. Below, you can find my after-thoughts about this peculiar event.
Speaker's Dinner and Before Party
The speakers' dinner and pre-party set the tone for the conference, offering a warm, inviting atmosphere. Both venues, a well-chosen restaurant and a lively before-party spot, facilitated great conversations and networking among the attendees.
Conference Overview
The Ruby Warsaw ..read more
Maciej Mensfeld - Running with Ruby
3M ago
Introduction
I'm happy to announce that Karafka 2.3 and its Web UI 0.8 have just been released.
For those who wonder what Karafka is, Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework.
The Karafka 2.3 release builds upon the foundation set by its predecessor, 2.2, making it a seamless continuation rather than a major rewrite. This means upgrading from version 2.2 to 2.3 can be done without extensive work or significant modifications to the existing codebases.
These releases introduce many new features. Please note that I have described only the most significant ch ..read more
Maciej Mensfeld - Running with Ruby
4M ago
While officially End-of-Life (EOL), Ruby 2.7 remains critical in many ongoing projects. Despite its EOL status, a significant user base continues to rely on this version for various reasons, ranging from legacy system compatibility to specific feature dependencies. As a developer of Karafka, an open-source software (OSS), I recognize the importance of supporting Ruby 2.7, giving users more time to upgrade than the EOL time. This commitment is reflected in my integration tests that ensure compatibility with Ruby 2.7.
However, a recent update has posed a challenge. On Friday, 15th of December 20 ..read more
Maciej Mensfeld - Running with Ruby
6M ago
Introduction
Karafka is a Ruby and Rails framework designed to simplify processing messages consumed from Apache Kafka.
One of Karafka's components is the https://karafka.io/docs/Web-UI-Getting-Started/. It provides a convenient way for developers to monitor and manage their Karafka-based applications without using the command line or third-party software.
The interface, amongst others, includes:
historical metrics,
real-time aggregated metrics,
real-time information on resource usage,
errors details,
performance statistics,
stale partitions detection (LSO hangs),
routing pattern matched topi ..read more
Maciej Mensfeld - Running with Ruby
7M ago
Introduction
Managing frontend assets in gems serving Web UI can be tricky. Why? Because while you want assets to expire across versions, you don't want them fetched repeatedly, slowing things down. This challenge popped up when working on the Karafka Web UI, which runs on the Roda framework.
In this article, I'll explore how I addressed this, ensuring smooth updates without compromising speed by elevating Roda and its plugins ecosystem.
Brief on the Roda web framework.
Roda is a versatile web framework for the Ruby programming language. Crafted by Jeremy Evans, Roda is renowned for its excep ..read more
Maciej Mensfeld - Running with Ruby
11M ago
Karafka is a Ruby and Rails framework that simplifies the development of Apache Kafka-based applications. Among its varied features, the Filtering API provides enhanced control over the data flow.
The crux of this article is about managing offsets - unique identifiers for messages within Kafka's partitions. Often, there's a need to manage offsets alongside database operations within a transaction, especially when handling potential process crashes and anomalies, minimizing the risk of double processing.
For instance, if a SQL operation concludes successfully but the offset commit fails due to ..read more
Maciej Mensfeld - Running with Ruby
11M ago
I'm happy to announce that Karafka 2.1 has just been released.
For those who wonder what Karafka is, Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework.
The Karafka 2.1 release builds upon the foundation set by its predecessor, 2.0, making it a seamless continuation rather than a major rewrite. This means that upgrading from version 2.0 to 2.1 can be done without extensive work or significant modifications to existing codebases. With Karafka 2.1, you can expect improved features and enhancements while maintaining the stability and compatibility you have come to rel ..read more
Maciej Mensfeld - Running with Ruby
1y ago
Kafka is a popular distributed streaming platform that is commonly used for building real-time data pipelines and streaming applications. One of the core features of Kafka is its ability to handle high-volume, real-time data streams and reliably process and distribute them to multiple consumers. However, in some cases, it may be necessary to postpone the processing of certain messages for many reasons.
This is where the Karafka's Delayed Topics feature comes in. This is a powerful mechanism that allows you to delay the consumption of messages for a later time, giving you greater control over y ..read more
Maciej Mensfeld - Running with Ruby
1y ago
Kafka topics are a fundamental concept in Apache Kafka. Topics are logical names or labels representing a stream of messages that Kafka clients can produce and consume.
What makes them interesting is the variety of settings that can be applied to them. These settings, amongst others include:
Partition count: The number of partitions that a topic should be split into.
Replication factor: The number of replicas that should be maintained for each partition.
Retention period: The time that messages should be retained in the topic.
Minimum and maximum in-sync replicas: The minimum number of replic ..read more