Photo by Mike Benna on Unsplash Every day, we need to load fresh data from our Data Lake into our Data Warehouse. Different data sets exist within the Data Lake. To guarantee fresh and correct Dashboards, we need a mechanism to load, transform and quality check the data. In this blog post, we are tackling the load and transform section of the work. Quality checks are accomplished by both Tests and Metrics and will be the content of another post. Here’s a quick overview of our current setup: s3 as a Data Lake Snowflake is used as our Cloud Data Warehouse dbt to model our tr ..read more

Distributed Read Models at willhaben (part 2: Details) Photo by Mathyas Kurmann on Unsplash Strategies and guidelines for successfully sharing entity state using Kafka message queues. Introduction At willhaben our customer services are provided by a distributed system consisting of dozens of back-end components responsible for different DDD domains, and the associated data. These components often need access to data associated with other domains. Part 1 of this series talks about why we choose to make this data available via replication; this part discusses how we do it. Sh ..read more

Distributed Read Models at willhaben (part 1: Motivation) Photo by Bundo Kim on UnsplashIntroduction In a distributed system, how do different components get access to data that is needed by them, but not owned by them? At willhaben we do this by having data-owning components publish that data to Kafka compacted topics, and having components which need a “read only” copy of that data listen to those topics and store the data locally. Martin Fowler calls this Event Driven Architecture with Event-carried State Transfer (quite a mouthful!); we call this a (Distributed) Read Model. This ..read more

Photo by Jeff Loucks on Unsplash Some months ago, here on the willhaben tech blog, a contest among our engineers was announced: ‘Propose a blog post topic, and the topic with the most votes from our readers will win a small prize’. To motivate readers to vote for a blog post, a raffle was also proposed, and some participants received willhaben goodie bags (with a Raspberry Pi inside). So a few lucky winners are likely sitting at home with a Raspberry Pi in their hands, wondering what projects to realise with it and how to impress their colleagues ?. We will cover a more exotic choice ..read more

This blog post describes the implementation of a custom Kafka Connect Single Message Transform (SMT). It offers plenty of links to resources that might help you implement your own custom transformation. The source code is available on Willhaben’s Github. What is Kafka Connect There are already many resources on Kafka Connect, especially the documentation page, blog posts, and tutorials offered by Confluent itself. However, to provide some context, we repeat here the part relevant to understanding the Kafka Connect API that allows us to hook into the data processing and write a custom tran ..read more

Photo by Agence Olloweb on UnsplashThe Crash Last summer, we experienced a new crash after a routine android app release. Even though the numbers showed it was not too widespread, it was still our top crash, and we needed to release a fix. The important part of the stack trace was this: Caused by org.koin.core.error.NoBeanDefFoundException: No definition found for class:'d.a.q0.d.b' & qualifier:'unauthorızed_wıth_actıon'. Check your definitions! Koin, our dependency injection framework, could not provide the class (ErrorResponseHandler) with the specific qualifier. To prov ..read more

The willhaben tech blog you are reading right now tries to give you more insight about how willhaben works, what problems we are facing and how we try to solve them. Normally our software, site reliability and test engineers select the topics they want to write about themselves. But we always wonder what you, the reader, are most curious about. So for Christmas this year, we thought we would start a small contest among our engineers: propose a blog post topic and the topic with the most votes from our readers will win a small prize. look behind the code in our tech blog We received n ..read more

Maintaining long-term code consistency is a well-known challenge in software development. In this article, we explore how to enforce proper code structure and easily automate it using ArchUnit. Photo by Chris Yang on Unsplash You approach your code. Your plan: to implement a cool feature. But then, as soon as you begin, before you know it, the ground turns slippery, you lose balance, and you fall into mud: The code structure is muddled beyond recognition and it’s nearly impossible to even find things. And worse, even when you find them and try to change them for the better, you ..read more

Photo by Alexander Sinn on UnsplashBackground At willhaben, some 421 million ads have been published since the company was founded in 2006. Around 7 million searches are executed daily on willhaben by about 6 million active monthly users, which makes us the largest internet marketplace in Austria. Until a year ago, our entire reporting and analytics infrastructure was based on a nightly backup-restore of the main production database. This process was augmented with some additional basic click-stream data, aggregated using Apache Spark, and stored in Cassandra. In addition, overall tr ..read more