Classic confinement for wal-g

Hi.

WAL-G is a tool to backup databases and archive database write ahead to cloud storage, providing Point in Time Recovery. It is stable for PostgreSQL and with beta support for MySQL, MariaDB, MongoDB, and Redis. My snap only builds for PostgreSQL at the moment. It is the successor to wal-e, which is already snapped. And similarly to wal-e, requires classic confinement. The tool is invoked by PostgreSQL when WAL files need to be shipped off the server, and these are located at an arbitrary location on the filesystem. WAL-G is also used to backup and restore the PostgreSQL database files themselves. By default this is all under /var/lib/postgresql in a standard Debian install, but it is very common for this to be changed to multiple other disk partitions for performance and scaling reasons.

Hi @stub :slight_smile:

It is true that wal-e was granted classic confinement, but that was granted more than 3.5 years ago before lots of snapd improvements and before many of our review processes were in place, so we should not consider wal-e’s classic granting for this request.

You stated: “The tool is invoked by PostgreSQL when WAL files need to be shipped off the server, and these are located at an arbitrary location on the filesystem. WAL-G is also used to backup and restore the PostgreSQL database files themselves. By default this is all under /var/lib/postgresql in a standard Debian install, but it is very common for this to be changed to multiple other disk partitions for performance and scaling reasons.

Typically, access to arbitrary files on the system is not a supported use case for classic, but let’s discuss wal-g’s needs in more detail.

  • the tool is invoked by PostgreSQL when WAL files need to be shipped off the server” - postgresql would be able to launch the snap, and wal-g could plugs ‘network’ to ship files off the server
  • these are located at an arbitrary location on the filesystem” - normally we would recommend that the snap plugs ‘home’ and ‘removable-media’, but in wal-g’s case, I suspect home is uninteresting. ‘removable-media’ allows access to both /media and /mnt though, so that is potentially interesting. Does wal-g need only read access? If so, I suspect the system-backup interface would work very well for you since it gives read access to all of the host filesystem, via /var/lib/snapd/hostfs (with a few exceptions like /dev, /proc and /sys).
  • WAL-G is also used to backup and restore the PostgreSQL database files themselves” - as mentioned, ‘system-backup’ should be able to be used for backup, but you’d need to use something like ‘system-files’ for restore. You mentioned that /var/lib/postgresql is the default location, so you could use system-files with write: [ /var/lib/snapd/hostfs/var/lib/postgresql ]. What are other typical locations (eg, from best practices, upstream documentation, etc) that should also be considered? Do WAL files always have a particular extension?

WAL-G needs both read and write access. For log shipping, on the primary server it reads them, and on standby servers it is writing them. For database backups, read when making backups and write when restoring them.

For PostgreSQL, WAL files do not have file extensions. Neither do the database files. I don’t know about the other databases (which the snap does not currently support).

There are no upstream best practices, as this is platform dependent. /var/lib/postgresql is just the Debian default, and really only suitable for toy databases that fit on the root filesystem. Internally, we tend to use subdirectories under external mounts /srv, /srv2 etc. So the Juju charm for example replaces /var/lib/postgresql/$ver/main with a symlink to /srv/pgdata/main, where the main database files are stored on the Juju provided mount. But locking down these paths would not be useful, even we limited use to Juju charm deployments that I know will follow these conventions, as during manual disaster recovery we will be restoring to wherever we happen to have available disk space. And again, I have no idea about the other databases that WAL-G supports.

Whilst wal-g may be used in various ways such that strict confinement would be difficult, would it be possible for the wal-g snap to establish certain use-case patterns such that strict confinement would be achievable - ie. would it make sense to advise users to bind-mount their databases folders under say /var/snap/wal-g/common so that it has the required access without any additional access required?

A bind mount would be technically possible, but would involve a fairly invasive reconfiguration and downtime of the PostgreSQL database to make it feed the correct paths to WAL-G. The snap distribution would be the worst option (over say a deb package in a PPA, or running ‘go install’).