The Unofficial Google Data Science Blog
76 FOLLOWERS
This blog is the work of some data scientists at Google who wish to bring out stories of interest to data scientists outside of Google. As editors, we hope to publish articles from across Google written by those individuals most knowledgeable about any given subject matter.
The Unofficial Google Data Science Blog
10M ago
Overview Human-labeled data is ubiquitous in business and science, and platforms for obtaining data from people have become increasingly common. Considering this, it is important for data scientists to be able to assess the quality of the data generated by these systems: human judgements are noisy and are often applied to questions where answers might be subjective or rely on contextual knowledge. This post describes a generic framework for understanding the quality of human-labeled data, based around the concepts of reliability and validity. It then goes on to show how a new framework called ..read more
The Unofficial Google Data Science Blog
2y ago
by AMIR NAJMI & MUKUND SUNDARARAJAN
Data science is about decision making under uncertainty. Some of that uncertainty is the result of statistical inference, i.e., using a finite sample of observations for estimation. But there are other kinds of uncertainty, at least as important, that are not statistical in nature. This blog post introduces the notions of representational uncertainty and interventional uncertainty to paint a fuller picture of what the practicing data scientist is up against.
Data science and uncertainty
Data Science (DS) deals with data-driven decision making u ..read more
The Unofficial Google Data Science Blog
3y ago
by LEE RICHARDSON & TAYLOR POSPISIL
Calibrated models make probabilistic predictions that match real world probabilities. This post explains why calibration matters, and how to achieve it. It discusses practical issues that calibrated predictions solve and presents a flexible framework to calibrate any classifier. Calibration applies in many applications, and hence the practicing data scientist must understand this useful tool.
What is calibration?At Google we make predictions for a large number of binary events such as “will a user click this ad” or “is this email spam”. In addition t ..read more
The Unofficial Google Data Science Blog
3y ago
by TAMAN NARAYAN & SEN ZHAO
A data scientist is often in possession of domain knowledge which she cannot easily apply to the structure of the model. On the one hand, basic statistical models (e.g. linear regression, trees) can be too rigid in their functional forms. On the other hand, sophisticated machine learning models are flexible in their form but not easy to control. This blog post motivates this problem more fully, and discusses monotonic splines and lattices as a solution. While the discussion is about methods and applications, the blog also contains pointers to research papers an ..read more
The Unofficial Google Data Science Blog
3y ago
by ALEXANDER WAKIM
Ramp-up and multi-armed bandits (MAB) are common strategies in online controlled experiments (OCE). These strategies involve changing assignment weights during an experiment. However, if one changes assignment weights when there are time-based confounders, then ignoring this complexity can lead to biased inference in an OCE. In the case of MABs, ignoring this complexity can also lead to poor total reward, making it counterproductive towards its intended purpose. In this post we discuss the problem, a solution, and practical considerations.
Background Online controlle ..read more
The Unofficial Google Data Science Blog
3y ago
by THOMAS OLAVSON
Thomas leads a team at Google called "Operations Data Science" that helps Google scale its infrastructure capacity optimally. ln this post he describes where and how having “humans in the loop” in forecasting makes sense, and reflects on past failures and successes that have led him to this perspective.
Our team does a lot of forecasting. It also owns Google’s internal time series forecasting platform described in an earlier blog post. I am sometimes asked whether there should be any role at all for "humans-in-the-loop” in forecasting. For high stakes, strategic forecasts ..read more
The Unofficial Google Data Science Blog
3y ago
by YI LIU
Importance sampling is used to improve precision in estimating the prevalence of some rare event in a population. In this post, we explain how we use variants of importance sampling to estimate the prevalence of videos that violate community standards on YouTube. We also cover many practical challenges encountered in implementation when the requirement is to produce fresh and regular estimates of prevalence.
BackgroundEvery day, millions of videos are uploaded to YouTube. While most of these videos are safe for everyone to enjoy, some videos violate the community guidelines of You ..read more
The Unofficial Google Data Science Blog
3y ago
by MICHAEL FORTE
Large-scale live experimentation is a big part of online product development. In fact, this blog has published posts on this very topic. With the right experiment methodology, a product can make continuous improvements, as Google and others have done. But what works for established products may not work for a product that is still trying to find its audience. Many of the assumptions on which the "standard" experiment methodology is premised are not valid. This means a small and growing product has to use experimentation differently and very carefully. Indeed, failure to do s ..read more
The Unofficial Google Data Science Blog
3y ago
by BILL RICHOUX
Critical decisions are being made continuously within large software systems. Often such decisions are the responsibility of a separate machine learning (ML) system. But there are instances when having a separate ML system is not ideal. In this blog post we describe one of these instances — Google search deciding when to check if web pages have changed. Through this example, we discuss some of the special considerations impacting a data scientist when designing solutions to improve decision-making deep within software infrastructure.
Data scientists promote principled decisi ..read more
The Unofficial Google Data Science Blog
3y ago
by DANIEL PERCIVAL
Randomized experiments are invaluable in making product decisions, including on mobile apps. But what if users don't immediately uptake the new experimental version? What if their uptake rate is not uniform? We'd like to be able to make decisions without having to wait for the long tail of users to experience the treatment to which they have been assigned. This blog post provides details for how we can make inferences without waiting for complete uptake.
BackgroundAt Google, experimentation is an invaluable tool for making decisions and inference about new products and ..read more