And just like that, we’re down to the 10th episode of Unapologetically Technical! In this episode, I interview Michael Drogalis, the founder and CEO of ShadowTraffic where we talked about the early Hadoop era and how he saw the need for Kafka in the industry. He shared his journey of starting a new company in his 20s and being acquired by Confluent. We also discussed how software engineers can improve their software creation and projects by understanding more about how products are created and talking to customers. Lastly, we go in-depth into ShadowTraffic, covering how it works and why cr ..read more

I had the pleasure of being one of the speakers at GOTO Amsterdam 2023 where I talked about “Why Most Data Projects Fail & How to Avoid It” and I can’t wait to share this talk with you!   Abstract: Unfortunately, the majority of data projects fail. Yet, they fail for the same reasons. Most management and data teams don’t know the reasons a project succeeds or fails. It just appears to be random, hard work, or luck. To help understand the reasons teams succeed, we will introduce the who, what, when, where, and how of data projects. By answering these questions, teams will understan ..read more

This week on Unapologetically Technical, I had the wonderful pleasure of interviewing Gunnar Morling, the creator of the Billion Row Challenge and Senior Staff Software Engineer at Decodable. In this episode, we talk about why it is so important to stay in a position long enough to gain experience and see the success or failure of decisions. He also shares his experiences at RedHat and working on Debezium. Then, we touch on his work at Decodable where we go in-depth into the Billion Row Challenge. We cover the three critical pieces of good implementations and discuss the hig ..read more

It has been quite a while, but we’re finally back to a new episode this year! In this episode of Unapologetically Technical, I interview Tom Scott, the Founder and CEO of Streambased. Join us as we talk about distributed systems and how he created distributed or what we call the Monte Carlo simulations. We also talk about his work across various companies like how he created and ran a data warehouse at Sky Betting, his work at Cloudera doing Customer Operations Engineering, and how that helped him understand what users were doing with Hadoop, at Confluent, where he worked on the Kafka repl ..read more

The premier of my latest talk covering The State of Data Engineering. I go through the history of the industry to see where we’re heading. This starts with data warehousing and goes into data science. I finish off by showing how data engineering can avoid the same fate as data warehousing and data science. Sorry, we didn’t have a microphone for the questions and I forgot to repeat some of the questions. If you turn on captions, most of the questions are captioned. A big thanks to Aaron Wilkerson for his cameo as the business and to Paige Roberts for filming. Don’t forget to subscribe to my ..read more

What better year to start the Christmas season than to drop a new episode of Unapologetically Technical! In this episode, I interview Stephane Derosiaux from Conduktor. We talk about his time evolving architectures and creating real-time systems at Auchan (grocery) and Adeo/Leroy Merlin (Home Improvement). We discuss the issues of British food and how to find good food in London. We chat about geeking and his views on minimalism. We go in-depth into Conduktor as to how it works and more even so how it works with your Kafka deployment. We also discuss the gateway and the new cases it unlock ..read more

Lately, I’ve been learning how to trade options. Although there’s data and programming involved in options trading, it isn’t as technical as data engineering or software engineering. However, it reflects the current state of learning, whether that’s data engineering or options trading. It gave me a look into learning a skill using videos. Each lesson I learned will directly apply to your learning or skill improvement. As you look at the landscape, there are several different types of learning videos: Hype – These have virtually no substance. In options, these are get-rich-quick videos. Hype ..read more

Another month, another episode! In this episode of Unapologetically Technical, I interview Matteo Merli the co-creator of Apache Pulsar and CTO of StreamNative. We talk about his interest in creating communication protocols and how that morphed into creating Apache Pulsar. He shares why Pulsar was created at Yahoo and how they convinced and managed to create new projects. As a European, we discuss the differences between US and European technology companies and his experiences with the move. Catch us as we go in-depth into Apache Pulsar! Part of my conversation with Matteo is discussing ho ..read more

A Guest Post by Ole Olesen-Bagneux In this blog post I would like to describe a new data team, that I call ‘the data discovery team’. It’s a team that connects naturally into the constellation of the three data teams  Operations team Data engineering team Data Science team as described in Jesse Anderson’s book Data Teams  (2020) Before I explain what the data discovery team should do, it is necessary to add a bit of context on the concept of data discovery itself.  Data discovery is thought of in different ways in data science and in information science respectfully. First o ..read more

Unapologetically Technical is finally back with a new episode! In this episode of Unapologetically Technical, I had the pleasure of interviewing Neil Avery from Liquidlabs.  We discussed his experiences creating grid computing systems at major banks like Royal Bank of Scotland and Deutchebank, as well as his journey to founding a startup called Logscape and working as a consultant at Excellian. We also took a trip down memory lane and discussed the early days of Confluent, including topics like ksqlDB, NIH, and what makes a successful startup in its early stages.  One of the main ..read more