Robots are frequently used in the manufacturing industry for numerous use-cases. Amongst many, one case is to eliminate defective products automatically from reaching the finished goods inventory. The same principles of these systems can be adopted to filter out malformed data from datastores. This blog post deep dives into how we rebuilt one of our Cassandra(C*) clusters by removing malformed data using Yelp’s Data Pipeline. Apache Cassandra is a distributed wide-column NoSQL datastore and is used at Yelp for storing both primary and derived data. Many different features on Yelp are powered b ..read more

Manually managing the lifecycle of Kubernetes nodes can become difficult as the cluster scales. Especially if your clusters are multi-tenant and self-managed. You may need to replace nodes for various reasons, such as OS upgrades and security patches. One of the biggest challenges is how to terminate nodes without disturbing tenants. In this post, I’ll describe the problems we encountered administering Yelp’s clusters and the solutions we implemented. Problems At Yelp we use PaaSTA for building, deploying and running services. Initially, PaaSTA just supported stateless services. This meant it ..read more

Abstract   Compared to full-scale ML, multi-armed bandit is a lighter weight solution that can help teams quickly optimize their product features without major commitments. However, bandits need to have a candidate selection step when they have too many items to choose from. Using A/B testing to optimize the candidate selection step causes new bandit bias and convergence selection bias. New bandit bias occurs when we try to compare new bandits with established ones in an experiment; convergence selection bias creeps in when we try to solve the new bandit bias by defining and selecting est ..read more

In this blog post, we introduce Spark-Lineage, an in-house product to track and visualize how data at Yelp is processed, stored, and transferred among our services. What is Spark-Lineage? Spark and Spark-ETL: At Yelp, Spark is considered a first-class citizen, handling batch jobs in all corners, from crunching reviews to identify similar restaurants in the same area, to performing reporting analytics about optimizing local business search. Spark-ETL is our inhouse wrapper around Spark, providing high-level APIs to run Spark batch jobs and abstracting away the complexity of Spark. Spark-ETL is ..read more

Experimentation has become standard practice for companies, and one of the most important aspects is how to evaluate the results to make ship/no-ship decisions. Have you run into experiments where you don’t have enough data for statistically significant results or perhaps the performance of your primary metric seemingly disagrees with that of your secondary metrics? If so, leveraging existing features to perform variance reduction may help with coming to a conclusion. At Yelp, we have found that using features typically used in ML modeling, in particular, can help with measuring treatment effe ..read more

Our community of users will always come first, which is why Yelp takes significant measures to protect sensitive user information. In this spirit, the Database Reliability Engineering team implemented a data sanitization process long ago to prevent any sensitive information from leaving the production environment. The data sanitization process still enables developers to test new features and asynchronous jobs against a complete, real time dataset without complicated data imports. MySQL and other open source project innovations over the last decade have led us on a journey to Vitess, which is ..read more

Yelp’s Kafka infrastructure ingests tens of billions of messages each day to facilitate data driven decisions and power business-critical pipelines and services. We have recently made some improvements to our Kafka deployment architecture by running some of our clusters on PaaSTA, Yelp’s own Platform as a Service. Our Kubernetes (k8s) based deployment leverages a custom Kubernetes operator for Kafka, as well as Cruise Control for lifecycle management. Kafka on PaaSTA on Kubernetes Architectural Motivations and Improvements In the past, all of our Kafka clusters ran on dedicated EC2 instances o ..read more

At Yelp we are always striving to optimize our user experience so we can help guide our customers to success. We aim to streamline the onboarding process for business owners by centralizing customer products into a single page. The Challenge Yelp offers an array of free and paid products that help local businesses connect with consumers. To set up these products on their Yelp page, business owners previously had to navigate through multiple tabs, which negatively impacted product setup rates (roughly 55%) and lowered overall user engagement. Previously, the only way businesses could set up Yel ..read more

Yelp has many teams of mobile developers who collectively maintain two different mobile apps on iOS and Android: Yelp (hereinafter “Consumer App”) and Yelp for Business (hereinafter “Biz App”). We’re always looking for ways to ship features more quickly and consistently on all these platforms! We adopt vendor and open-source libraries when possible, and we develop our own shared libraries when necessary. While many teams were already independently adopting server-driven UI (SDUI) to build their features faster and cheaper, we felt something was missing – a foundation that tied all our librarie ..read more

Yelp’s employee resource group for women in engineering, Awesome Women in Engineering (AWE), recently held its first virtual summit! The summit was designed for women and allies at Yelp to learn, network, and have fun. AWE started in 2013 with a mission to build a strong community for women and allies at Yelp by facilitating professional career-building activities, networking, leadership, and mentorship opportunities. As a resource group, we provide support and organize activities targeted towards professional growth for women engineers, helping them to maximize their potential at Yelp and bey ..read more