We continue exploring various corner cases for the Lowland multimodality detection. In this post, we consider an example that illustrates the usefulness of THDQE ..read more

When researchers focus on model design, they often worry whether the model is correct or not. I believe that we should accept the fact that all the models are wrong. The world is too complex to be captured by a single model: we are never able to acknowledge all the variables. Therefore, the answer to the question “Is the model correct?” is always “No”. It should not bother us: from the pragmatic perspective, it is irrelevant whether the model is correct or not. If we embrace the model misspecification, we can switch our attention to the question “What is the impact of deviations from the model ..read more

I have just published a preprint of a paper ‘Quantile-Respectful Density Estimation Based on the Harrell-Davis Quantile Estimator’. It is based on a series of my research notes. The paper preprint is available on arXiv: arXiv:2404.03835 [stat.ME]. The paper source code is available on GitHub: AndreyAkinshin/paper-qrdehd. You can cite it as follows: Andrey Akinshin (2024) “Quantile-Respectful Density Estimation Based on the Harrell-Davis Quantile Estimator” arXiv:2404.03835 Abstract: Traditional density and quantile estimators are often inconsistent with each other. Their simultaneous usage ..read more

In A better jittering approach for discretization acknowledgment in density estimation, I discussed the jittering approach that improves Quantile-Respectful Density Estimation for discrete distributions and continuous-discrete mixtures. In this post, I will show a brief example of how such an approach improves the accuracy of the Lowland multimodality detection ..read more

I continue the topic of Quantile-Respectful Density Estimation in the context of Multimodality Detection. In this post, we briefly discuss the handling of the QRDE boundary spikes in order to correctly detect the near-border modes ..read more

In How to build a smooth density estimation for a discrete sample using jittering, I proposed a jittering approach. It turned out that it does not always work well. It is not always capable of preserving the original distribution shape and avoiding gaps. In this post, I would like to propose a better strategy ..read more

Cohen’s d is one of the most popular measures of the effect size. Unfortunately, it was designed for the normal distribution, which may make it a misleading measure in the non-normal case. And the real distributions are never normal. When we discuss deviations from normality, we should treat the illusion of normality not as an atomic mental construction, but rather as a set of independent assumptions, each of which may be violated independently. In this post, I take a look at what kind of issues we may have when the symmetry assumption is heavily violated ..read more

Statistics is one of the most confusing, controversial, and depressing disciplines I know. So many different approaches, so many different opinions, so many arguments, so many person-years of wasted time, and so many flawed peer-reviewed papers. What we want from statistics is an easy-to-use tool that would nudge us toward asking the right questions and then straightforwardly guide us on how to design proper and relevant statistical procedures. What we have is a bunch of vaguely described sets of strange equations, a few arbitrarily chosen magical numbers as thresholds, and no clear understand ..read more

If you are curious if something impacts something else, the answer is probably “yes.” Does that indicator depend on those factors? Yes, it does. If we change this thing, would it affect …? Yes, it would. If a person takes this pill, could it cause a non-exactly-zero change in the body? Yes, the presence of the pill is already a change that can always be detected with the right amount of effort. One may argue that in some cases (assuming the list of specific cases is presented), zero effect does exist. For a moment, let us pretend that it is true. Now, let us imagine a parallel universe, which ..read more

Imagine a city mayor considering a project offering to build parks in several neighborhoods. It can be a good budget investment since it can potentially increase the happiness level of the citizens. However, it is just a hypothesis: if parks do not impact happiness, it is worth considering other city renovation projects. It makes sense to perform a pilot experiment before spending the budget on all the parks. The mayor is thinking about the following plan: pick a random neighborhood, survey the citizens to measure their happiness, build a park, survey the citizens again, compare the survey res ..read more