Analytics-Toolkit
612 FOLLOWERS
Talking web analytics with a focus on the Google Analytics platform.
Analytics-Toolkit
1M ago
Several charges are commonly thrown at A/B testing while considering it or even after it has become standard practice in a company. They may come from product teams, designers, developers, or management, and can be summed up like this: A good way to address these and to make the business case for experimentation is to […] Read more ..read more
Analytics-Toolkit
2M ago
“Observed power”, “Post hoc Power”, and “Retrospective power” all refer to the statistical power of a statistical significance test to detect a true effect equal to the observed effect. In a broader sense these terms may also describe any power analysis performed after an experiment has completed. Importantly, it is the first, narrower sense that […] Read more ..read more
Analytics-Toolkit
2M ago
The above is a question asked by some practitioners of A/B testing, as well as a number of their clients when examining the outcome of an online controlled experiment. It may be raised regardless if the outcome is statistically significant or not. In both cases the fact the observed effect in an A/B test is […] Read more ..read more
Analytics-Toolkit
2M ago
Observed power, often referred to as “post hoc power” and “retrospective power” is the statistical power of a test to detect a true effect equal to the observed effect size. “Detect” in the context of a statistical hypothesis test means to result in a statistically significant outcome. Some calculators aimed at A/B testing practitioners use […] Read more ..read more
Analytics-Toolkit
1y ago
The Mann Whitney U Test (MWU), also known as the Wilcoxon Rank Sum Test and the Mann-Whitney-Wilcoxon Test, continues to be advertised as the go-to test for analyzing non-normally distributed data. In online experimentation it is often touted as the most suitable for analyses of non-binomial metrics with typically non-normal (skewed) distributions such as average […] Read more ..read more
Analytics-Toolkit
1y ago
Sequential statistics are gathering interest and there are more and more questions posed by CROs looking into the matter. For this article I teamed up with Lucia van den Brink, a distinguished CRO consultant who recently started using Analytics Toolkit and integrated frequentist sequential testing into her client workflow. In this short interview she asks […] Read more ..read more
Analytics-Toolkit
1y ago
A central feature of sequential testing is the idea of stopping “early”, as in “earlier compared to an equivalent fixed-sample size test”. This allows running A/B tests with fewer users and in a shorter amount of time while adhering to the targeted error guarantees. For example, a test may be planned with a maximum duration […] Read more ..read more
Analytics-Toolkit
1y ago
Have you heard how there is a much greater probability than generally expected that a statistically significant test outcome is in fact a false positive? In industry jargon: that a variant has been identified as a “winner” when it is not. In demonstrating the above the terms “False Positive Risk” (FPR), “False Findings Rate” (FFR), […] Read more ..read more
Analytics-Toolkit
1y ago
Google Analytics 4 has been a let down in many aspects based on every discussion I’ve seen and had with professionals of all stripes – marketers, advertising specialists, CROs, GA professionals, online experimentation experts, etc. One of the less discussed issues it brought with it is the default heavyweight GTAG library integration it comes with. […] Read more ..read more
Analytics-Toolkit
2y ago
Analytics Toolkit was conceived in 2012 as a set of tools that automate essential Google Analytics-related tasks and augment the GA functionalities in various ways. This goal was achieved in the years since with the release of over a dozen tools utilizing the Google Analytics API. These were accompanied by dozens of in-depth technical articles […] Read more ..read more