Automating Highly Available PostgreSQL Clusters

In March of ‘23, I took over as the Lead Architect of my employer’s Ansible-based automation for creating highly-available PostgreSQL clusters.1 Since then, I’ve been responsible for advancing the product: refactoring the code, adding functionality, rethinking some of its core attributes, etc. I’ve also taken some steps to informally restructure the team that works on things to better divide up responsibilities and make everyone more productive.

In the past ~18 months, we’ve launched the 2.x line of the product, and are currently in the midst of testing the upcoming 2.2.0 release. I thought it might be a good time to start blogging about what the team and I are doing, so I’m going to start posting semi-regularly about our goings on, our tooling, our process, and maybe even the product itself. As such, the posts will vary in topic; some posts will be about Ansible, others about PostgreSQL, some about the rest of our stack, and a few here and there about DevOps-y type things as they relate to our tooling and processes. I will work to tag each post appropriately so that you can follow only those that interest you, but they will all be tagged with ‘cpa’ if you want to follow everything.

As things currently stand, we build highly-available PostgreSQL clusters with the following components:

  • PostgreSQL2 (duh)
  • Patroni3 (for failover/switchover and ‘service uptime’)
  • etcd4 (the DCS that maintains Patroni’s state)
  • HAProxy5 (for transparently routing connections based on read vs write)
  • PgBouncer6 (for connection pooling)
  • pgBackRest7 (for recovery, continuous archiving, self-healing)
  • pgMonitor8 (for monitoring the installed stack)
  • Keepalived9 (for solving SPoF with the components that don’t natively support it)(optional)

And a basic production deployment looks something like this:

That about covers the overview. Next week, we’ll describe the significant differences between the 1.x product line and the new 2.x line. We might even talk a bit of future releases and what is planned for them.

:wq