Written by Shumpei Goke and Jinshu Niu Why Technical Learning? At Lyft, data scientists tackle challenging technical problems every day. To support and empower our data scientists, Lyft’s Technical Learning Council (TLC) provides diverse and high-quality continuous learning opportunities to hone their technical skills. TLC’s mission is “to equip Data Science team members with the technical knowledge and skills that are applicable to their work and helpful to their career advancement.” Investing in technical learning not only aids data scientists in solving complex problems but also contri ..read more

Summary At Lyft we have built a platform for developing, training and serving Reinforcement Learning models for typical internet industry applications with a focus on Contextual Bandits. Those models have been critical for decision making problems that other techniques such as supervised learning or optimization models struggled with. In this article we describe how we extended our existing machine learning ecosystem to support Reinforcement Learning models, develop models using Off-Policy Evaluation, and the lessons learned along the way. Introduction Reinforcement Learning (RL) has show ..read more

Photo by Frank Olsen UNDER CC BY-SA 3.0 DEEDIntroduction Our payment platform team had the unique challenge to upgrade our Aurora Postgres DB from v10 to v13. This DB was responsible for storing transactions within Lyft and contains ~400 tables (with partitions) and ~30TB of data. Upgrading the database in-place would have resulted in ~30 mins of downtime. Such significant downtime is untenable — it would cause cascading failures across multiple downstream services, requiring a large amount of engineering effort to remediate. Through native replication between the two Postgres major versi ..read more

PUP in action | Photo by philhearing under CC BY 2.0 DEED In this post, we’ll cover how Lyft upgrades Python at scale — 1500+ repos spanning 150+ teams — and the latest iteration of the tools and strategy we’ve built to optimize both the overall time to upgrade and the work required from our engineers. We’ve successfully used (and evolved) this playbook over multiple upgrades, from Python 2 to Python 3.10 and hope you find it useful! Welcome to BLT! Here’s your first assignment. The Backend Language Tooling (BLT) team at Lyft is responsible for the Python and Go experience for our en ..read more

Written by Ritesh Varyani and Jeana Choi at Lyft. Introduction At Lyft, we have used systems like Apache ClickHouse and Apache Druid for near real-time and sub-second analytics. Sub-second query systems allow for near real-time data explorations and low latency, high throughput queries, which are particularly well-suited for handling time-series data. For our customers, this means faster analytics on near real-time data and decision making. This is crucial for use cases like market signaling and forecasting which benefit from, and depend upon, the most up-to-date information. Overall, the ..read more

High-quality data is necessary for the success of every data-driven company. It enables everything from reliable business logic to insightful decision-making and robust machine learning modeling. It is now the norm for tech companies to have a well-developed data platform. This makes it easy for engineers to generate, transform, store, and analyze data at the petabyte scale. As such, we have reached a point where the quantity of data is no longer a boundary. Yet this has come at the cost of quality. In this post we will define data quality at a high-level and explore our motivation to ach ..read more

Background Note: This publication assumes you have basic familiarity with the service mesh pattern (e.g. Istio, Linkerd, Envoy — created at Lyft!) in microservice architectures. In addition, it is recommended you read the 2021 precursor post written by my colleague, Matt Grossman. Lyft runs hundreds of microservices to power the company’s offerings. Our team, the Developer Infrastructure team, aims to build the best tools to enable microservice owners (our “customers”) to reliably and quickly test changes in a local and/or end-to-end environment. Testing in a prod-like environment plays a crit ..read more

Interviewing with a tech company doesn’t have to be overwhelming! Lyft’s team of recruiters supports our talented candidates through every step of the process, from the first getting-to-know-you call, all the way through the offer letter. Our goal is to make your recruiting journey smooth and predictable — and to give you the best opportunity to demonstrate your strengths, learn from our incredible team, and walk away from each conversation with a smile! ? Here’s what to expect, plus a few tips to maximize your success: How we interview First, an intro call with a recruiter: after ..read more

Where’s My Data — A Unique Encounter with Flink Streaming’s Kinesis Connector For years now, Lyft has not only been a proponent of but also a contributor to Apache Flink. Lyft’s pipelines have evolved drastically over the years, yet, time and time again, we run into unique cases that stretch Flink to its breaking points — this is one of those times. Context While Lyft runs many streaming applications, the one specifically in question is a persistence job. Simply put, it streams data from Kinesis, performs some level of serializations and transformations, and writes to S3 every few m ..read more

Our cross-functional development process By: Aastha Bhargava, Jake Hercules, Erik Kamp, Michael Ramdatt, Nathan Van Fleet, Rex Lam, Kieran Gupta Product For years, drivers have been clear about what they wanted: native Lyft support for CarPlay and Android Auto. They’ve made the request across social media platforms, through the app, and in feedback sessions with Lyft researchers. Until recently, Lyft couldn’t provide it due to our app architecture. So, drivers turned to DIY solutions to use infotainment. They used Google Maps, Apple Maps, and Waze, which required constantly switching betw ..read more