Here’s another installment in Data Q&A: Answering the real questions with Python. Previous installments are available from the Data Q&A landing page. likert_mean Likert scale analysis¶ Here’s a question from the Reddit statistics forum. I have collected data regarding how individuals feel about a particular program. They reported their feelings on a scale of 1-5, with 1 being Strongly Disagree, 2 being Disagree, 3 being Neutral, 4 being Agree, and 5 being Strongly Agree. I am looking to analyze the data for averages responses, but I see that a basic mean will not do the trick. I am lo ..read more

Here’s another installment in Data Q&A: Answering the real questions with Python. Previous installments are available from the Data Q&A landing page. test_percentile Testing percentiles¶ Here’s a question from the Reddit statistics forum. I have two different samples (about 100 observations per sample) drawn from the same population (or that’s what I hypothesize; the populations may in fact be different). The samples and population are approximately normal in distribution. I want to estimate the 85th percentile value for both samples, and then see if there is a statistically significa ..read more

Here’s another installment in Data Q&A: Answering the real questions with Python. Previous installments are available from the Data Q&A landing page. low_percentile Bootstrapping percentiles¶ Here’s a question from the Reddit statistics forum. I’m trying to figure out how to determine the confidence interval for the .2 percentile temperature for specific set of observed temperatures (all hourly temperatures during January, February, and December since 2000). I have recordings for 53128 of the 53424 possible hourly recordings. How would I go about saying that I am X% sure that the actu ..read more

Here’s another installment in Data Q&A: Answering the real questions with Python. Previous installments are available from the Data Q&A landing page. corr_trend What does “strength” mean?¶ Here’s a question from the Reddit statistics forum. I am currently doing a uni assignment and one of my tasks is analysing the correlation between two variables. When I use the correlation function in Excel, it returns a correlation of -0.0377. When I use the same data to create a scatter plot, the trend line is positive. I need to identify the correlation strength and direction and thereby, I am co ..read more

Here’s another installment in Data Q&A: Answering the real questions with Python. In general, I will try to focus on practical problems, but this one is a little more philosophical. confidence What does a confidence interval mean?¶ Here’s a question from the Reddit statistics forum (with an edit for clarity): Why does a confidence interval not tell you that 90% of the time, [the true value of the population parameter] will be in the interval, or something along those lines? I understand that the interpretation of confidence intervals is that with repeated samples from the population, 90 ..read more

This post is part of a new project with the working title Data Q&A: Answering the real questions with Python. In each installment, I’ll take a question from Reddit’s statistics forum and answer it, using Python code to demonstrate. My answer is in a Jupyter notebook — see the link below to run it in Colab. count_data Is taking the SD of a count variable helpful?¶ Here’s a question from the Reddit statistics forum. A student brought this up to me in class this week and I had no idea how to answer. For some context they are doing in experiment that involves a count variable and then have a ..read more

Think Python has moved into production, on schedule for the official publication date in July — but maybe earlier if things go well. To celebrate, I have posted the next batch of chapters on the new site, up through Chapter 12, which is about Markov text analysis and generation, one of my favorite examples in the book. From there, you can follow links to run the notebooks on Colab. And we have a cover! The new animal is a ringneck parrot, I’ve been told. I will miss the Carolina parakeet that was on the old cover, which was particularly apt because it is an ex-parrot. Nevertheless, I think th ..read more

In previous articles (here, here, and here) I’ve looked at evidence of a gender gap in political alignment (liberal or conservative), party affiliation (Democrat or Republican), and policy preferences. Using data from the GSS, I found that women are more likely to say they are liberal, and more likely to say they are Democrats, by 5-10 percentage points. But in their responses to 15 policy questions that most distinguish conservatives and liberals, men and women give similar answers. In other words, the political gap is mostly in what people say about themselves, not in what they believe abou ..read more

I am happy to announce the third edition of Think Python, which will be published by O’Reilly Media later this year. You can read the online version of the book here. I’ve posted the Preface and the first four chapters — more on the way soon! You can read the Early Release and pre-order from O’Reilly, or pre-order the third edition on Amazon. Here is an excerpt from the Preface that explains… What’s new in the third edition? The biggest changes in this edition were driven by two new technologies — Jupyter notebooks and virtual assistants. Each chapter of this book is a Jupyter notebook, which ..read more

In a previous article, I used data from the General Social Survey (GSS) to see if there is a growing gender gap among young people in political alignment, party affiliation, or political attitudes. So far, the answer is no. Young women are more likely than men to say they are liberal by 5-10 percentage points. But there is little or no evidence that the gap is growing. Young women are more likely to say they are Democrats. In the 1990s, the gap was almost 20 percentage points. Now it is only 5-10 percentage points. So there’s no evidence this gap is growing — if anything, it is shrinking. To ..read more