The Two Billion Claps Bug TL;DR A user was able to exploit a race condition in our backend system to manipulate clap counts on posts. Users are supposed to only be able to clap between 0 and 50 times for a given post, but this hack allowed them to go outside those bounds (both above and below). Our fix leverages DynamoDB condition expressions and strongly consistent reads to block updates on data that have been manipulated after a read, but before a write. Additionally, we implemented an eventually consistent clap rectification solution for those posts that were already affected by this&n ..read more

We recently spent a sprint focused on reducing our Snowflake costs. During this sprint, we investigated 22 of our most expensive pipelines (in terms of Snowflake costs), one by one. In total we merged 56 changes, and in this post we’ll be laying out the optimisations that worked best for us. Most of these changes were just common sense and didn’t involve any advanced data engineering techniques. Still, we’re always making mistakes (and that’s okay!) and hopefully this post will help readers avoid a few of the pitfalls we encountered. ⚠️ Medium is now 14 years old. Our team has inherited a ..read more

Learn how we’re using ScyllaDB to build a fast and scalable data layer for lists in our feature store. Photo by Mathew Schwartz on Unsplashtl;dr This is the second installment of our series on the new list feature type in Medium’s feature store, which describes how we implemented the lists data layer using ScyllaDB. See Part 1: Laying the foundations, the previous installment, for an introduction to the feature store and the list feature type. We’ll first introduce ScyllaDB, a database designed for high performance and scalability, and go over the ScyllaDB concepts needed to und ..read more

Explore the foundations of the new list feature type in Medium’s feature store. Photo by Iain Kennedy on Unsplashtl;dr This is the first installment of our series on the new list feature type in Medium’s feature store. Central to Medium’s recommendations system, the feature store handles the high-volume data used by machine learning models. The limitations of the existing relational features prompted the development of the new list feature type, which is designed to handle cross-entity relations in an efficient and scalable way. What is the feature store? The feature store ..read more

The Medium team at our company retreat in Portland, Oregon As Medium turns the page on becoming a profitable company, in engineering we are looking ahead at how that milestone changes the way we think about the work we do and how we want to approach it going forward. This is a note I sent to the engineering team a few months ago. This has started a lot of discussions that have been the basis for a big culture shift we’re trying to make as a company towards long-term stability and success. Team, We spent the last few years improving everything about our engineering culture: our proces ..read more

This story is not about pretty code, it is even less about pretty design. By Ed Uthman — originally posted to Flickr as Apple I Computer, CC BY-SA 2.0, https://commons.wikimedia.org/w/index.php?curid=7180001 The Medium iOS codebase is more than 10 years old, and we still have code dating from 2013 and 2014 that is still in use today. Here you might think why the hell did they never completely rewrite this? This is actually something we wear like a badge of honour in the Medium iOS team and I want to explain why today, by diving into how we work effectively with legacy code, with ..read more

Like in many companies, you probably have some databases lying around that are starting to cost a lot in terms of storage. You are paying more and more every week; the more you wait, the more money will be spent for nothing. It’s time for a cleanup. The basic approach is to write a script that will scan your DB and delete the items that aren’t useful anymore (e.g., really old data). Problem: that can be extremely expensive with Dynamo DB, so expensive that it will take years to pay off… Solution: One way to tackle this is to migrate all the data you need to a new table. From the ..read more

Imagine you are a software engineer, or you really are. You create iOS apps on a daily basis. You use only the latest and finest trends in mobile development, such as SPM, Swift Data, and, of course, SwiftUI. Life couldn’t be better. But reality is cruel. And in real life you probably need to maintain and develop an application that was written several years ago when these technologies didn’t exist at all. Due to your natural curiosity and desire to build top-notch products, you still strive to integrate the best approaches in your app. The Medium iOS team is not an exception. And given t ..read more

Read along with us as we study Building Microservices Medium Engineering started a book club this year! (Really, it’s the backend group, but we’ll extend this to all of engineering in Q3.) Our first book is Building Microservices. We have a 4-month schedule that’s easy enough to follow while still getting everything done. March (already over, but you can catch up!) — Section I: Foundation April-May — Section II: Implementation June — Section III: People Read along with us! Why this book? Medium is built with a microservice architecture. But we have a (relatively) small t ..read more

An example on how we do it in the Medium iOS Application I bet that when you started your first SwiftUI project and when you wanted to paginate some lists, you scratched your head a bit. While there are various techniques to achieve this, I’ll focus on the one I’ve used for years. This time, I’ll not use Ice Cubes as an example but the Medium iOS application, as we’ve often used this technique when rebuilding the app's various features using SwiftUI. An infinite list of stories I recently migrated the search user interface to SwiftUI, and as you can see above, you can scroll th ..read more