This is a good article, from causal lens: Enterprise Decision Making Needs More Than Chatbots Channelling Judea Pearl, understanding what makes a difference (causality) requires more than data, it also requires something not in the data to begin with. So much of AI is based on a tools and technology mindset. As Johnny Depp says about ships in Pirates of the Caribbean, a ship is more than sails and rudders, those are things a ship needs. What a ship really is, is freedom. Causal inference is more than methods and theorems, those are things causal inference needs, but what it really is, is a way ..read more

 I've blogged in the past about what I call linear literalism and fundamentalist econometrics. And I've blogged a bit about linear probability models (LPMs). Recently I have had some concerns about people outsourcing their thinking to LLMs and the use of these tools like Dunning-Kruger-as-a-Service (DKaaS) where the critical thinking and actual learning starts and stops with prompt engineering and a response. Out of curiosity I asked ChatGPT about the appropriateness of using linear probability models. Although the overall response was thoughtful about thinking more carefully about causal ..read more

 Scott Cunningham brings up some interesting points about matching and utility maximization in this substack post: https://causalinf.substack.com/p/why-do-economists-so-dislike-conditional  "Because most of the time, when you are fully committed to the notion that people are rational, or at least intentionally pursuing goals and living in the reality of scarcity itself, you actually think they are paying attention to those potential outcomes. Why? Because those potential outcomes represent the gains from the choice you’re making....if you think people make choices because they h ..read more

    We could think of businesses as collections of decisions and processes that move and transform resources. Business value is created by moving resources from lower to higher valued uses. Because of their ability to identify which processes and decisions create the most value,  the value proposition of experimentation and causal inference stand out among all other analytic methods.  How do they do this? In application, experimentation and casual inference represent a way of thinking that carefully considers the business problem and all the ways that our data can fool ..read more

Assessing the balance between treatment and control groups across control variables is an important part of propensity score matching. It's heuristically an attempt to ‘recreate’ a situation similar to a randomized experiment where all subjects are essentially the same except for the treatment (Thoemmes and Kim, 2011). Matching itself should not be viewed so much as an estimation technique, but a pre-processing step to ensure that members assigned to treatment and control groups have similar covariate distributions 'on average' (Ho et al., 2007) This understanding of matching often gets lost ..read more

In previous posts I have discussed the value proposition of business experiments from both mainline and behavioral economic perspectives. This series of posts has been greatly influenced by Jim Manzi's book 'Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society.' Midway through the book Manzi highlights three important things that experiments in business can do: 1) They provide precision around the tactical implementation of strategy 2) They provide feedback on the performance of a strategy which allows for refinements to be driven by evidence 3 ..read more

In my previous post I discussed the value proposition of business experiments from a mainline economic perspective. In this post I want to view this from a behavioral economic perspective. From this point of view business experiments can prove to be invaluable with respect to challenges related to overconfidence and decision making under uncertainty. Heuristic Data Driven Decision Making and Data Story Telling  In a fast paced environment, decisions are often made quickly and often based on gut decisions. Progressive companies have tried as much as possible to leverage big data and anal ..read more

Why should firms leverage randomized business experiments? With recent advancements in computing power and machine learning, why can't they simply base all of their decisions on historical patterns discovered in the data using AI?  Perhaps statisticians and econometricians and others have a simple answer. The kinds of learnings that allow us to separate the impact of business decisions from other factors require understanding causality. This requires something that may not be in the data to begin with. Experiments may be the best (often the golden standard) way of answering causal questio ..read more

If you have taken very many statistics courses you may have gotten the impression that it's mostly a mixed bag of computations and rules for conducting hypothesis tests or making predictions or creating forecasts. While this isn't necessarily wrong, it could leave you with the opinion that statistics is mostly just a box of tools for solving problems. Absolutely statistics provides us with important tools for understanding the world, but to think of statistics as 'just tools' can have some pitfalls (besides the most common pitfall of having a hammer and viewing every problem as a nail) For on ..read more

Many times in a randomized controlled trial (RCT) issues related to non-compliance arise. Subjects assigned to the treatment fail to comply, while in other cases subjects that were supposed to be in the control group actually receive treatment. Other times we may have a new intervention (maybe it is a mobile app or some kind of product, service, or employer or government benefit) that law, contract, or nature implies that it can be accessed by everyone in our population of interest. We know that if we let nature take its course, users, adopters, or engagers are very likely going to be a self s ..read more