This post is part of a running series on Bayesian MCMC tutorials. For updates, follow @StableMarkets. Metropolis Review Metropolis-Hastings is an MCMC algorithm for drawing samples from a distribution known up to a constant of proportionality, . Very briefly, the algorithm works by starting with some initial draw then running through the following process for times: Propose a draw from a given proposal distribution . Evaluate posterior up to a proportionality constant under this draw: . Evaluate the posterior up to a proportionality constant under the previous draw:  If , the ..read more

For More posts on Bayesian models, Bayesian Nonparametrics, and causal inference follow me on twitter @stablemarkets. My advisor and his collaborator are teaching a short course on Bayesian Nonparametric Methods for Causal Inference at JSM next week. As part of the short course, I made an interactive tutorial on Dirichlet Processes using R Shiny. All underlying code is hosted on GitHub – you can also run locally from GitHub by running the code below. This local version may be faster if there are too many users on the web version. shiny::runGitHub('jasonroy0/BNP-short-course/', username = 'jas ..read more

Previous posts in this series on MCMC samplers for Bayesian inference (in order of publication): Bayesian Simple Linear Regression with Gibbs Sampling in R Blocked Gibbs Sampling in R for Bayesian Multiple Linear Regression Metropolis-in-Gibbs Sampling and Runtime Analysis with Profviz Speeding up Metropolis-Hastings with Rcpp All code for this (and previous) posts are in my Github repo Recently I’ve been reading about Bayesian additive regression trees (BART). For those interested, the paper is here. I use similar notation when describing the backfitting procedure in this post. I believe th ..read more

Previous posts in this series on MCMC samplers for Bayesian inference (in order of publication): Bayesian Simple Linear Regression with Gibbs Sampling in R Blocked Gibbs Sampling in R for Bayesian Multiple Linear Regression Metropolis-in-Gibbs Sampling and Runtime Analysis with Profviz The code for all of these posts can be found in my BayesianTutorials GitHub Repo. In the most recent post, I profiled a Metropolis-in-Gibbs sampler for estimating the parameters of a Bayesian logistic regression model. The conclusion was that evaluation of the log-posterior was a significant run time bottlenec ..read more

First off, here are the previous posts in my Bayesian sampling series: Bayesian Simple Linear Regression with Gibbs Sampling in R Blocked Gibbs Sampling in R for Bayesian Multiple Linear Regression In the first post, I illustrated Gibbs Sampling – an algorithm for getting draws from a posterior when conditional posteriors are known. In the second post, I showed that if we can vectorize, then drawing a whole “block” per iteration will increase the speed of the sampler. For many models, like logistics models, there are no conjugate priors – so Gibbs is not applicable. And as we saw in the firs ..read more

In a previous post, I derived and coded a Gibbs sampler in R for estimating a simple linear regression. In this post, I will do the same for multivariate linear regression. I will derive the conditional posterior distributions necessary for the blocked Gibbs sampler. I will then code the sampler and test it using simulated data. R code for simulating data and implementing the blocked Gibbs is in by GitHub repo. A Bayesian Model Suppose we have a sample size of subjects. We observe an outcome vector . The Bayesian multivariate regression assumes that this vector is drawn from a multivariate n ..read more

Many introductions to Bayesian analysis use relatively simple didactic examples (e.g. making inference about the probability of success given bernoulli data). While this makes for a good introduction to Bayesian principles, the extension of these principles to regression is not straight-forward. This post will sketch out how these principles extend to simple linear regression. Along the way, I will derive the posterior conditional distributions of the parameters of interest, present R code for implementing a Gibbs sampler, and present the so-called grid point method. I’ve had trouble with R co ..read more

I came across a stackoverflow post the other day touching on first differencing and decided to write a quick review of the topic as well as related random effects and fixed effects methods. In the end we’ll see that random effects, fixed effects, and first differencing are primarily used to handle unobserved heterogeneity within a repeated measures context. One annoying thing about these topics is that there are many synonyms and they vary across (sometimes within) disciplines. Throughout the post I’ll be using a lot of “i.e.”, “aka”, and “/”s to indicate synonymous concepts/idea. Re ..read more

The recent flare-up in discussions on p-values inspired me to conduct a brief simulation study. In particularly, I wanted to illustrate just how p-values vary with different effect and sample sizes. Here are the details of the simulation. I simulated draws of my independent variable : where For each , I define a as where In other words, for each effect size, , the simulation draws and with some error . The following regression model is estimated and the p-value of is observed. The drawing and the regression is done 1,000 times so that for each effect size – sample siz ..read more

Stops in 2014. Red lines indicate high white stop density areas and blue shades indicate high black stop density areas. Notice that high white stop density areas are very different from high black stop density areas. The star in Brooklyn marks the location of officers Liu’s and Ramos’ deaths. The star on Staten Island marks the location of Eric Garner’s death. In my last post, I compiled and cleaned publicly available data on over 4.5 million stops over the past 11 years. I also presented preliminary summary statistics showing that blacks had been consistently stopped ..read more