https://unsplash.com/photos/two-swimming-mallard-ducks-on-still-body-of-water-_Wo1Oq38tVU At willhaben, we sometimes need to analyse data streams from Kafka to ensure data consistency and integrity. Recently, I explored two different approaches to compare the contents of two Kafka streams. In this blog post, I’ll share the key points from these approaches. The Problem We recently switched implementation of how some Kafka events are created. But how can we check if two Kafka streams (legacy and new) contain the same content? A straightforward solution would be to write a custom application ..read more

AI Dev Buddies: What our team learned from using AI Assistants Tools like GitHub Copilot and JetBrains AI Assistant are rapidly gaining traction, promising to revolutionize how we write code. The developer community is buzzing with excitement around AI assistants. The potential to boost productivity and tackle tedious tasks is undeniable. However, it’s crucial to move beyond the hype and understand the true impact of these tools. Nearly a year ago, we at willhaben did an evaluation of Github Copilot and Tabnine. While intrigued by the potential, we found a significant gap between wh ..read more

Multiple Cassandra Keyspaces With Spring Boot Foto von Kevin Ku auf UnsplashIntroduction Here at willhaben, we use AWS Keyspaces to store some of our data. For one of our services we need to aggregate data from multiple keyspaces. When setting this up in practice, while adding the connection to the second keyspace into our codebase, we ran into an issue: Spring Data Cassandra does not support using multiple keyspaces out of the box — at least not in version 3.2.x. Disclaimer: Whether or not multiple keyspaces should be used within a single application is a discussion for a diffe ..read more

Photo by Mike Benna on Unsplash Every day, we need to load fresh data from our Data Lake into our Data Warehouse. Different data sets exist within the Data Lake. To guarantee fresh and correct Dashboards, we need a mechanism to load, transform and quality check the data. In this blog post, we are tackling the load and transform section of the work. Quality checks are accomplished by both Tests and Metrics and will be the content of another post. Here’s a quick overview of our current setup: s3 as a Data Lake Snowflake is used as our Cloud Data Warehouse dbt to model our tr ..read more

Distributed Read Models at willhaben (part 2: Details) Photo by Mathyas Kurmann on Unsplash Strategies and guidelines for successfully sharing entity state using Kafka message queues. Introduction At willhaben our customer services are provided by a distributed system consisting of dozens of back-end components responsible for different DDD domains, and the associated data. These components often need access to data associated with other domains. Part 1 of this series talks about why we choose to make this data available via replication; this part discusses how we do it. Sh ..read more

Distributed Read Models at willhaben (part 1: Motivation) Photo by Bundo Kim on UnsplashIntroduction In a distributed system, how do different components get access to data that is needed by them, but not owned by them? At willhaben we do this by having data-owning components publish that data to Kafka compacted topics, and having components which need a “read only” copy of that data listen to those topics and store the data locally. Martin Fowler calls this Event Driven Architecture with Event-carried State Transfer (quite a mouthful!); we call this a (Distributed) Read Model. This ..read more

Photo by Jeff Loucks on Unsplash Some months ago, here on the willhaben tech blog, a contest among our engineers was announced: ‘Propose a blog post topic, and the topic with the most votes from our readers will win a small prize’. To motivate readers to vote for a blog post, a raffle was also proposed, and some participants received willhaben goodie bags (with a Raspberry Pi inside). So a few lucky winners are likely sitting at home with a Raspberry Pi in their hands, wondering what projects to realise with it and how to impress their colleagues ?. We will cover a more exotic choice ..read more

This blog post describes the implementation of a custom Kafka Connect Single Message Transform (SMT). It offers plenty of links to resources that might help you implement your own custom transformation. The source code is available on Willhaben’s Github. What is Kafka Connect There are already many resources on Kafka Connect, especially the documentation page, blog posts, and tutorials offered by Confluent itself. However, to provide some context, we repeat here the part relevant to understanding the Kafka Connect API that allows us to hook into the data processing and write a custom tran ..read more

Photo by Agence Olloweb on UnsplashThe Crash Last summer, we experienced a new crash after a routine android app release. Even though the numbers showed it was not too widespread, it was still our top crash, and we needed to release a fix. The important part of the stack trace was this: Caused by org.koin.core.error.NoBeanDefFoundException: No definition found for class:'d.a.q0.d.b' & qualifier:'unauthorızed_wıth_actıon'. Check your definitions! Koin, our dependency injection framework, could not provide the class (ErrorResponseHandler) with the specific qualifier. To prov ..read more

The willhaben tech blog you are reading right now tries to give you more insight about how willhaben works, what problems we are facing and how we try to solve them. Normally our software, site reliability and test engineers select the topics they want to write about themselves. But we always wonder what you, the reader, are most curious about. So for Christmas this year, we thought we would start a small contest among our engineers: propose a blog post topic and the topic with the most votes from our readers will win a small prize. look behind the code in our tech blog We received n ..read more