Welcome back to this blog series on Crunchy Postgres via Automation (CPA). We’re doing something a little different today; we’re going to discuss something that is merged in development but isn’t released yet: package pinning.

On Dec 13, I merged a PR to automatically pin and unpin packages that CPA installs so that we no longer have to worry about accidental upgrades when the system is updated:

hunleyd merged commit 914213e into CrunchyData:development Dec 13, 2024 15:20 EST

This PR teaches the pkgmgr role to versionlock/pin packages as we install them for the packages we care about. Said locks/pins are removed as needed to allow upgrades/removes as well.

And indeed, standing up a CPA cluster with this patch results in a situation like the following on the target nodes:

[keith@cd-dc1-pg1 ~]$ sudo dnf versionlock list
postgresql17-libs-0:17.2-0Crunchy.el9.*
crunchy-backrest-0:2.53.1-0Crunchy.el9.*
policycoreutils-python-utils-0:3.6-2.1.el9.*
postgresql17-contrib-0:17.2-0Crunchy.el9.*
postgresql17-server-0:17.2-0Crunchy.el9.*
python3-psycopg2-0:2.9.9-0Crunchy.el9.*
pgmonitor-pg17-extension-0:2.0.0-1Crunchy.el9.*
python3-patroni-etcd-0:2.1.7-0Crunchy.el9.*
node-exporter-0:1.7.0-6Crunchy.el9.*
pgmonitor-node-exporter-extras-0:5.1.1-1Crunchy.el9.*
pgmonitor-sql-exporter-extras-0:5.1.1-1Crunchy.el9.*
sql-exporter-0:0.15.0-1Crunchy.el9.*
pgbadger-0:12.4-0Crunchy.el9.*

vagrant@ha-192-168-122-16:~$ sudo apt-mark showhold
acl
libpq5
node-exporter
pgbackrest
pgbadger
pgmonitor-node-exporter-extras
pgmonitor-pg16-extension
pgmonitor-sql-exporter-extras
postgresql-16
postgresql-common
python3-patroni-etcd
python3-psycopg2
reprepro
sql-exporter

As you can see, for both RedHat-based and Debian-based systems, we now lock the specific versions of critical packages. This allows the normal system patching to occur without fear of accidentally altering the components we rely on. This solves a long-standing issue with deployed clusters in the field where the Ops team isn’t the same team deploying CPA and isn’t made aware of the necessary package exclusions.

To accomplish this, we’ve added the ansible.builtin.dpkg_selections and community.general.dnf_versionlock modules to our code and we’ve had to enable force_apt_get on our Debian package install task. Finally, we expanded the package install task file to first unpin the package in question, then install/update/remove said package, update the package facts if we made a change, and finally pin the package if we did an install or an upgrade. It’s a little bit slower in execution, both because of the extra steps and because we have to loop over the list of packages, but it’s a performance hit we have to take in the name of correctness and resiliency.

This new feature will be included in CPA 2.3, which will be available in the first half of 2025.