This video covers the latest announcements from StreamNative, Confluent, and WarpStream. We discuss communication protocols, how they’re used, and what they mean for you. We also discuss the various systems using Kafka’s protocol. Finally, we discuss the announcements about writing to Iceberg and DeltaLake directly from the broker and what that means for costs and operational ease.   It’s conference season, and we’re interpreting the latest announcements from the various streaming vendors. This post will consider the recent StreamNative, Confluent, and WarpStream announcements. &nbs ..read more

Unapologetically Technical’s newest episode is now live! In this episode of Unapologetically Technical, I interview AJ Hunyady, the founder and CEO of InfinyOn. We talked about his early experiences with networking systems, such as creating firewalls, email, and web servers, and how those prepared him for data work. We chatted about the various implications of Generative AI and LLMs for the current and future coders of the world. We talked about the difficulty of using batch systems in security and the need for real-time actions. He shared his views on containerization and Kubernetes conso ..read more

In this episode of Unapologetically Technical, I interview Hubert Dulay, the author of Streaming Data Mesh and Developer Advocate at StarTree. We talked about his early experience with web backends like CORBA and SOAP and how those prepared him for data work. He shares his advice for those with web development skills to transition into data and what it’s like for a person leaving a company after a long tenure there. We discuss his time at Cloudera and Confluent and how the data industry has changed over time. We talk about his recent experience with Flink and StarTree. We go in-depth into ..read more

And just like that, we’re down to the 10th episode of Unapologetically Technical! In this episode, I interview Michael Drogalis, the founder and CEO of ShadowTraffic where we talked about the early Hadoop era and how he saw the need for Kafka in the industry. He shared his journey of starting a new company in his 20s and being acquired by Confluent. We also discussed how software engineers can improve their software creation and projects by understanding more about how products are created and talking to customers. Lastly, we go in-depth into ShadowTraffic, covering how it works and why cr ..read more

I had the pleasure of being one of the speakers at GOTO Amsterdam 2023 where I talked about “Why Most Data Projects Fail & How to Avoid It” and I can’t wait to share this talk with you!   Abstract: Unfortunately, the majority of data projects fail. Yet, they fail for the same reasons. Most management and data teams don’t know the reasons a project succeeds or fails. It just appears to be random, hard work, or luck. To help understand the reasons teams succeed, we will introduce the who, what, when, where, and how of data projects. By answering these questions, teams will understan ..read more

This week on Unapologetically Technical, I had the wonderful pleasure of interviewing Gunnar Morling, the creator of the Billion Row Challenge and Senior Staff Software Engineer at Decodable. In this episode, we talk about why it is so important to stay in a position long enough to gain experience and see the success or failure of decisions. He also shares his experiences at RedHat and working on Debezium. Then, we touch on his work at Decodable where we go in-depth into the Billion Row Challenge. We cover the three critical pieces of good implementations and discuss the hig ..read more

It has been quite a while, but we’re finally back to a new episode this year! In this episode of Unapologetically Technical, I interview Tom Scott, the Founder and CEO of Streambased. Join us as we talk about distributed systems and how he created distributed or what we call the Monte Carlo simulations. We also talk about his work across various companies like how he created and ran a data warehouse at Sky Betting, his work at Cloudera doing Customer Operations Engineering, and how that helped him understand what users were doing with Hadoop, at Confluent, where he worked on the Kafka repl ..read more

The premier of my latest talk covering The State of Data Engineering. I go through the history of the industry to see where we’re heading. This starts with data warehousing and goes into data science. I finish off by showing how data engineering can avoid the same fate as data warehousing and data science. Sorry, we didn’t have a microphone for the questions and I forgot to repeat some of the questions. If you turn on captions, most of the questions are captioned. A big thanks to Aaron Wilkerson for his cameo as the business and to Paige Roberts for filming. Don’t forget to subscribe to my ..read more

What better year to start the Christmas season than to drop a new episode of Unapologetically Technical! In this episode, I interview Stephane Derosiaux from Conduktor. We talk about his time evolving architectures and creating real-time systems at Auchan (grocery) and Adeo/Leroy Merlin (Home Improvement). We discuss the issues of British food and how to find good food in London. We chat about geeking and his views on minimalism. We go in-depth into Conduktor as to how it works and more even so how it works with your Kafka deployment. We also discuss the gateway and the new cases it unlock ..read more

Lately, I’ve been learning how to trade options. Although there’s data and programming involved in options trading, it isn’t as technical as data engineering or software engineering. However, it reflects the current state of learning, whether that’s data engineering or options trading. It gave me a look into learning a skill using videos. Each lesson I learned will directly apply to your learning or skill improvement. As you look at the landscape, there are several different types of learning videos: Hype – These have virtually no substance. In options, these are get-rich-quick videos. Hype ..read more