Update on 8-29-2023 Last week, I put up a poll that asks users what they would do if faced with some ethical dilemmas that arise on the business side of data science. I'm presenting the first set of results here. I'm guessing the responses have stabilized but will provide further updates below should the trends shift. Case Study 1 Is it acceptable for BestHotels.com to present fake statistics to influence users? A. Yes (8%) B. No (87%) C. I don't care (5%)   Case Study 2 Has AnyVoice done enough to mitigate such scams? A. Yes (18%) B. No (73%) C. I don't care (9%)   Case Study 3 S ..read more

I've written often about the problems with conducting observational studies. We like to hold up randomized controlled trials (RCTs) as the "gold standard," and if a study is described as an RCT, and especially if its design is pre-registered, we don't ask many questions. This point of view is becoming unsustainable, based on a recent Nature article. The gist of the article is that experts now believe at least one out of every four RCTs contain so many problems that their findings should be ignored, leaving the possibility also that some of these RCTs may be entirely fake. In coming to this co ..read more

There really is not much more to say than to print the first sentence of this Bloomberg artcile (link): Facebook owner Meta Platforms Inc. for years paid a contractor to scrape data from other websites while publicly condemning the practice and suing companies that pulled data from its own social-media platforms. *** Web scraping is a very strange industry that operates in a shadow. At the end of this post, I hope you'll appreciate why the scrapers are hiding. Let's trace how data land on websites. I'm going to use an example of baseball statistics in Baseball-Reference.com. Here is how one s ..read more

In 2016, I posted about ethical dilemmas in the practice of data science, based on a talk I gave at the time. That post continues to be viewed each year so some readers must have found it useful. The post covered several realistic case studies of ethical dilemmas I have seen in industry. Since ethics is subjective, I don't recommend any course of action. These case studies were designed to be useful in courses on data science ethics. It encourages students to debate both sides of the dilemma, and perhaps for the class to vote on a course of action. Today's post is an update to the 2016 post ..read more

Today, I return to the Clalit booster study. I’ve written three prior posts about the study, which was the catalyst for regulatory approval of booster shots of mRNA vaccines for Covid-19 at the turn of 2022. In the first post, I showed how the methodology - common among Covid-19 vaccine studies - contains obvious biases that researchers and their peer reviewers overlooked. (link) The booster study featured a further unexplained wrinkle – some subjects are excluded after their infection status is known (link). In a third post, I developed a sports analogy to ponder what kind of effects such an ..read more

I'm been mulling over how to explain the potential impact of strange data processing procedures used in the observational studies of Covid-19 vaccines. These are by no means standard operating procedures. And I think I've landed on one explanation that is tantalizing. *** Imagine a football (soccer) match that is tied 0-0 and heading into overtime. Overtime is a 30-minute affair, played through, and not sudden death. The overtime period is divded into two 15-minute halves with a short break in the middle. That is the standard operating procedure. *** Now, we are going to change the rulebook ..read more

One of the most frustrating aspects of reading Covid-19 related publications is reading the conclusions. A summary of results is supposed to provide readers with the minimally necessary information to understand the full contents of the paper - but what we encounter is "story time": almost always, these conclusions are stated in the most general sense, ignoring all the caveats, the exclusions, the imperfections of the methodologies and analyses. They are "stories" that go far beyond the limited evidence presented. I've been reading the Clalit study that was a major force behind the approval of ..read more

I'll be writing a number of posts about observational studies of the Pfizer booster, published by the Clalit health insurer in Israel (back in December 2021), which is the primary source supporting the effectiveness of the booster shot (3rd shot). These studies provide fodder forever for classes in observational studies because (a) there are many nooks and crannies in which residual biases hid, and (b) researchers made many assertions (wrong, of course) about having corrected all biases using the simplest, most conventional methods. This first post points out the biggest problem with such stu ..read more

Raphael Lataster has written a response to our JECP paper, which discusses several biases in Covid-19 vaccine observational studies that have not been properly accounted for in many notable studies. He augmented our first example to show another source of bias in later studies. In our example, we demonstrated how the "case counting window bias" could cause vaccine effectiveness estimates to be over optimistic. In typical clinical trials, one starts counting cases (events) from the day the treatment is finished, so in the case of two-shot Covid-19 vaccines, this should be the day after the se ..read more

On the dataviz blog, I recently posted about Wall Street Journal's graphic about U.S. remote workers. See the post here. The original article is securely locked down behind the WSJ paywall, here. I identified a key issue with the dataviz, which is the weak connection between the Question and the Visual, using the Trifecta Checkup framework. In this post, I discuss potential problems in the D(ata) corner: how they developed the message from survey data. Here is the primary insight, as expressed by WSJ: Workers overall spent an average of 5 hours and 25 minutes a day working from home in 2022 ..read more