PostgreSQL uses a complex system of locks to balance concurrent operations and data consistency, across many transactions. Those intricacies are beyond the scope of this post. Here we want to specifically look at queries that are waiting, whether on locks or for other resources, and learn how to get more insights about why. Balancing concurrency with consistency is an inherent part of the MVCC system that PostgreSQL uses. One of the operational problems that can occur with this system, is that queries get blocked waiting to acquire a lock, and that wait time can be excessive, causing errors. I ..read more

Indexes are, by far, the most feature related to performance provided by every general purpose relational database. Without indexes, there are no such things as efficient search operations, no unique constraints, and no primary keys. Achieving good performance on reasonable amounts of data without indexes is therefore totally impossible. Generating sample data The main question now is: what happens during index creation, and how can we speed up the process? Creating indexes on billions of rows is quite costly because data has to be sorted and turned into an efficient search structure. To demon ..read more

This post will look at the basics of PostgreSQL Savepoints within a Transaction. A transaction is used to form a non-separable unit of work to commit or not, as a unit. Transactions are opened using the BEGIN keyword, then either committed or may be rolled back. Use ROLLBACK without any arguments to do that. Dividing Up a Transaction Within the concept of a transaction, there is a smaller unit that allows for incremental persistence, scoped to the transaction, called “savepoints.” Savepoints create sub-transactions with some similar properties to a transaction. Savepoints Savepoints mark a par ..read more

In a previous blog post, we configured an EC2 instance and installed PostgreSQL on it. After the initial setup, we ran pgbench to measure the initial TPS (transactions per second). Then, we tuned PostgreSQL and reran pgbench to see how much we could improve the TPS. To tune PostgreSQL, we adjusted various parameters based on the system's available resources. In this blog, we will identify which of those parameters contributed the most to the performance improvements. We will use the same instance size on AWS EC2 as before, which is t2.large. Here is the list of parameters that we initially tun ..read more

I'm pleased to announce release 0.7 of pg_statviz, the minimalist extension and utility pair for time series analysis and visualization of PostgreSQL internal statistics. This release comes with a huge new feature: the implementation of resampling with pandas to permit effectively unlimited data analysis. Now you can take thousands, potentially millions of full statistics snapshots and squash all those data points to just 100 plot points (by default, using downsampling with mean values) to enable you to get a long-term view of your system's performance and statistics. The chart above shows wh ..read more

Photo credit: Kenrick Mills From time to time I need to check whether a managed Postgres provider supports auto_explain, and if so, which auto_explain parameters they support. Since it can be time-consuming to set up and test each of these, I wanted to create a reference for myself, and thought it might also be of interest to others. I did my testing over the past week, but plan to keep this up to date. If you notice a mistake, or that a provider changes their support, please do let me know. What is auto_explain? Firstly, in case you’re not familiar, auto_explain is an extension that ships wit ..read more

sparql_fdw sparql_fdw is a foreign data wrapper to access data available over the internet in the SPARQL format from within PostgreSQL based on Multicorn2. The soucre code and documentation is available on GitHub. A merge request to the original repository has been opened. General changes I have tested the extension after recent changes in Multicorn2 against PostgreSQL 15 and 16, which have not been supported by now. The installation examples had to be changed due to changes in Python 3.11, see PEP 668. The problem here is, that Pythons pip3 needs a parameter, –break-system-packages, to instal ..read more

Preface This post was initially intended to introduce my ‘one more Postgres thing' - a built-in fully transparent re-optimisation feature, which I'm really proud of. However, during benchmarking, I discovered that partitioning the table causes performance issues that are hard to tackle. So, let's see the origins of these issues and how PostgreSQL struggles with them. Here, I do quite a simple thing: having the non-trivial benchmark, I just run it over a database with plain tables, do precisely the same thing over the database where all these tables are partitioned by HASH, and watch how it end ..read more

We had a case recently where one of our DBs failed over to a new primary. To this server (old primary, and then new primary) we had connection from some kind of CDC tool (Debezium, I think). The thing is that while there was failover, this tool read (using logical decoding) changes on old primary ..read more

As I’ve been working with Postgres psql cli, I’ve picked up a few good habits from my Crunchy Data co-workers that make my terminal database environment easier to work with. I wanted to share a couple of my favorite things I’ve found that make getting around Postgres better. If you’re just getting started with psql, or haven’t ventured too far out of the defaults, this is the post for you. I’ll walk you through some of the friendliest psql settings and how to create your own preset settings file. Some of the most helpful psql commands Formatting psql output Postgres has an expanded display mod ..read more