Introduction It’s been a while since I’ve posted anything on the blog. One of the primary reasons for the hiatus is that I have been using python and pandas but not to do anything very new or different. In order to shake things up and hopefully get back into the blog a bit, I’m going to write about polars. This article assumes you know how to use pandas and are interested in determining if polars can fit into your workflow. I will cover some basic polars concepts that should get you started on your journey. Along the way I will point out some of the things I liked and some of the dif ..read more

Introduction One of the reasons I like using pandas instead of Excel for data analysis is that it is easier to avoid certain types of copy-paste Excel errors. As great as pandas is, there is still plenty of opportunity to make errors with pandas code. This article discusses a subtle issue with pandas groupby code that can lead to big errors if you’re not careful. I’m writing this because I have happened upon this in the past but it still bit me big time just recently. I hope this article can help a few of you avoid this mistake. The Problem To illustrate this problem, we’ll use a si ..read more

Introduction When doing analysis with Jupyter Notebooks, you will frequently find yourself generating ad-hoc Excel reports to distribute to your end-users. After time, you might end up with dozens (or hundreds) of notebooks and it can be challenging to remember which notebook generated which Excel report. I have started using Excel document properties to track which notebooks generate specific Excel files. Now, when a user asks for a refresh of a 6 month old report, I can easily find the notebook file and re-run the analysis. This simple process can save a lot of frustration for your future se ..read more

Introduction Visual Studio Code is one of the most popular text editors with a track record of continual improvements. One area where VS Code has been recently innovating is its Jupyter Notebook support. The early releases of VS Code sought to replicate existing Jupyter Notebook features in VS Code. Recent VS Code releases have continued to develop notebook features that provide an experience that in many cases is better than the traditional Jupyter Notebook experience. I am a big fan of using Jupyter Notebooks for python analysis - even though there are limitations. For the type of adhoc ..read more

Introduction It’s no secret that data cleaning is a large portion of the data analysis process. When using pandas, there are multiple techniques for cleaning text fields to prepare for further analysis. As data sets grow large, it is important to find efficient methods that perform in a reasonable time and are maintainable since the text cleaning process evolves over time. This article will show examples of cleaning text fields in a large data file and illustrates tips for how to efficiently clean unstructured text fields using Python and pandas. The problem For the sake of thi ..read more

Introduction I enjoy hearing from readers that have used concepts from this blog to solve their own problems. It always amazes me when I see examples where only a few lines of python code can solve a real business problem and save organizations a lot of time and money. I am also impressed when people figure out how to do this with no formal training - just with some hard work and willingness to persevere through the learning curve. This example comes from Mark Doll. I’ll turn it over to him to give his background: I have been learning/using Python for about 3 years to help automate b ..read more

Introduction I have talked quite a bit about how pandas is a great alternative to Excel for many tasks. One of Excel’s benefits is that it offers an intuitive and powerful graphical interface for viewing your data. In contrast, pandas + a Jupyter notebook offers a lot of programmatic power but limited abilities to graphically display and manipulate a DataFrame view. There are several tools in the Python ecosystem that are designed to fill this gap. They range in complexity from simple JavaScript libraries to complex, full-featured data analysis engines. The one common denominator is that ..read more

Introduction One of the most basic analysis functions is grouping and aggregating data. In some cases, this level of analysis may be sufficient to answer business questions. In other instances, this activity might be the first step in a more complex data science analysis. In pandas, the groupby function can be combined with one or more aggregation functions to quickly and easily summarize data. This concept is deceptively simple and most new pandas users will understand this concept. However, they might be surprised at how useful complex aggregation functions can be for supporting sophisticate ..read more

Introduction With pandas it is easy to read Excel files and convert the data into a DataFrame. Unfortunately Excel files in the real world are often poorly constructed. In those cases where the data is scattered across the worksheet, you may need to customize the way you read the data. This article will discuss how to use pandas and openpyxl to read these types of Excel files and cleanly convert the data to a DataFrame suitable for further analysis. The Problem The pandas read_excel function does an excellent job of reading Excel worksheets. However, in cases where the data is not a ..read more

Introduction The main purpose of this blog is to show people how to use Python to solve real world problems. Over the years, I have been fortunate enough to hear from readers about how they have used tips and tricks from this site to solve their own problems. In this post, I am extremely delighted to present a real world case study. I hope it will give you some ideas about how you can apply these concepts to your own problems. This example comes from Michael Biermann from Germany. He had the challenging task of trying to gather detailed historical weather data in order to do analysis on t ..read more