Snapped daemon running as root cannot create file in directory with odd ownerships/permissions

snapd
1

I am snapping CUPS and everything is working, except creating print queues and doing other administrative CUPS tasks through CUPS’ web interface, which works in classically (.deb packages) installed CUPS.
CUPS needs to create a certificate file to do login/password-authenticated operations with its web interface. The CUPS daemon, running as root creates this file in the /var/run/cups/certs (classic installation) directory with the following permissions/ownerships:

$ ls -ld /var/run/cups/certs/
dr-x--x--x 2 lp lpadmin 60 May 25 21:26 /var/run/cups/certs/
$

The classically installed CUPS daemon has no problem creating a file in this diectory:

$ sudo ls -l /var/run/cups/certs/
total 4
-r--r----- 1 root lpadmin 32 May 25 21:26 0
$

Also one can list this directory and create and remove files there using sudo:

$ sudo ls -l /var/run/cups/certs/
total 4
-r--r----- 1 root lpadmin 32 May 25 21:26 0
$ sudo touch /var/run/cups/certs/1
$ sudo ls -l /var/run/cups/certs/
total 4
-r--r----- 1 root lpadmin 32 May 25 21:26 0
-rw-r--r-- 1 root root     0 May 25 21:33 1
$ sudo rm /var/run/cups/certs/1
$

This does not work for the CUPS daemon in the Snap if the Snap is installed in restricted mode (--dangerous, note that I am working with a Snap which I am locally building, not installing from the Snap Store), but in the unrestricted devmode (--devmode) it works as with classically installed CUPS, though.
To more easily reproduce, I get into a shell equivalent to the one in which the snapped CUPS daemon is running:

$ sudo snap run --shell printing-stack-snap.cupsd
root@till-x1yoga:/home/till/printing/openprinting/printing-stack-snap/x# cd
root@till-x1yoga:~# pwd
/root/snap/printing-stack-snap/x31
root@till-x1yoga:~# env | grep SNAP
SNAP_USER_DATA=/root/snap/printing-stack-snap/x31
SNAP_REVISION=x31
SNAP_ARCH=amd64
SNAP_INSTANCE_KEY=
SNAP_USER_COMMON=/root/snap/printing-stack-snap/common
SNAP=/snap/printing-stack-snap/x31
SNAP_COMMON=/var/snap/printing-stack-snap/common
SNAP_NAME=printing-stack-snap
SNAP_INSTANCE_NAME=printing-stack-snap
SNAP_DATA=/var/snap/printing-stack-snap/x31
SNAP_COOKIE=b7gtjF67Vox-0gfeiF8zNOV7ql4zFKSIxOAoqtF9X7T0KYa5Nl5u
SNAP_REEXEC=
SNAP_CONTEXT=b7gtjF67Vox-0gfeiF8zNOV7ql4zFKSIxOAoqtF9X7T0KYa5Nl5u
SNAP_VERSION=0.1.0
SNAP_LIBRARY_PATH=/var/lib/snapd/lib/gl:/var/lib/snapd/lib/gl32:/var/lib/snapd/void
root@till-x1yoga:~#

This is the directory which contains the certs directory of the snapped CUPS. It is in the $SNAP_DATA file system, where root is allowed to write and create files:

root@till-x1yoga:~# ls -l $SNAP_DATA/var/run
total 12
dr-x--x--x 2 snap_daemon lpadmin 4096 May 25 18:21 certs
-rw-r--r-- 1 root        root       7 May 25 18:21 cups-browsed.pid
-rw-r--r-- 1 root        root       7 May 25 18:21 cupsd.pid
-rw-r--r-- 1 root        root       0 May  6 22:58 stop-cupsd.lock
root@till-x1yoga:~#

You see that the certs directory has the same odd permissions and ownerships as the classic CUPS’ certs directory has: It has no writable bit at all, not even for the owner and the owner is snap_daemon here, the standard drop-privileges user for Snaps. A Snap cannot create a new user, like ‘lp’. The group ownership is lpadmin, the Snap here checks what is available in the system to use as CUPS admin group, trying lpadmin first, then adm and resorts to root otherwise.
The $SNAP_DATA/var/run and most other directories in the $SNAP_DATA file system are owned and writable by root.
Now I try what CUPS wants to do, create a file in the certs directory:

root@till-x1yoga:~# touch $SNAP_DATA/var/run/certs/0
touch: cannot touch '/var/snap/printing-stack-snap/x31/var/run/certs/0': Permission denied
root@till-x1yoga:~#

It fails with “Permission denied” as it happens also to CUPS, so I can reproduce the problem manually and exclude any further bug in CUPS. I also cannot read this directory as root:

root@till-x1yoga:~# ls -l $SNAP_DATA/var/run/certs/
ls: cannot open directory '/var/snap/printing-stack-snap/x31/var/run/certs/': Permission denied
root@till-x1yoga:~#

In the other directories it is no problem to create files:

root@till-x1yoga:~# touch $SNAP_DATA/var/run/0
root@till-x1yoga:~# less $SNAP_DATA/var/run/0
root@till-x1yoga:~# cat $SNAP_DATA/var/run/0
root@till-x1yoga:~# rm $SNAP_DATA/var/run/0
root@till-x1yoga:~#

So I try to change the ownership of the certs directory to root, but I am still not able to create a file:

root@till-x1yoga:~# chown root $SNAP_DATA/var/run/certs
root@till-x1yoga:~# touch $SNAP_DATA/var/run/certs/0
touch: cannot touch '/var/snap/printing-stack-snap/x31/var/run/certs/0': Permission denied
root@till-x1yoga:~#

But I can at least read the directory now:

root@till-x1yoga:~# ls -l $SNAP_DATA/var/run/certs/
total 0
root@till-x1yoga:~# ls -l $SNAP_DATA/var/run/      
total 12
dr-x--x--x 2 root lpadmin 4096 May 25 18:21 certs
-rw-r--r-- 1 root root       7 May 25 18:21 cups-browsed.pid
-rw-r--r-- 1 root root       7 May 25 18:21 cupsd.pid
-rw-r--r-- 1 root root       0 May  6 22:58 stop-cupsd.lock
root@till-x1yoga:~#

So change back ownership but allow only the owner to write:

root@till-x1yoga:~# chown snap_daemon $SNAP_DATA/var/run/certs
root@till-x1yoga:~# ls -l $SNAP_DATA/var/run/
total 12
dr-x--x--x 2 snap_daemon lpadmin 4096 May 25 18:21 certs
-rw-r--r-- 1 root        root       7 May 25 18:21 cups-browsed.pid
-rw-r--r-- 1 root        root       7 May 25 18:21 cupsd.pid
-rw-r--r-- 1 root        root       0 May  6 22:58 stop-cupsd.lock
root@till-x1yoga:~# chmod u+w $SNAP_DATA/var/run/certs
chmod: changing permissions of '/var/snap/printing-stack-snap/x31/var/run/certs': Operation not permitted
root@till-x1yoga:~# chmod u-w $SNAP_DATA/var/run/cups-browsed.pid
root@till-x1yoga:~# chmod u+w $SNAP_DATA/var/run/cups-browsed.pid
root@till-x1yoga:~#

I am not allowed to change the permissions, but allowed to change the permissions of the other files. Seems that here in the Snap root can only change permissions of directories owned by root. So change ownership to root again and then set the owner’s write bit:

root@till-x1yoga:~# chown root $SNAP_DATA/var/run/certs
root@till-x1yoga:~# chmod u+w $SNAP_DATA/var/run/certs
root@till-x1yoga:~# touch $SNAP_DATA/var/run/certs/0
root@till-x1yoga:~# ls -l $SNAP_DATA/var/run/certs
total 4
-rw-r--r-- 1 root root 32 May 25 20:25 0
root@till-x1yoga:~# rm $SNAP_DATA/var/run/certs/0
root@till-x1yoga:~# ls -l $SNAP_DATA/var/run/
total 12
drwx--x--x 2 root lpadmin 4096 May 25 20:25 certs
-rw-r--r-- 1 root root       7 May 25 18:21 cups-browsed.pid
-rw-r--r-- 1 root root       7 May 25 18:21 cupsd.pid
-rw-r--r-- 1 root root       0 May  6 22:58 stop-cupsd.lock
root@till-x1yoga:~#

So with these settings I am actually able to create the file!
The Snap’s CUPS was all the time running while I did these tests, but during the tests I left CUPS idle, not doing any operations on it. Now knowing that root can create a file in my modified certs directory, I want to know whether CUPS can do it, too. I do not restart CUPS here to avoid that it cleans up directory and file permissions and ownerships.
I simply go to the web interface of the Snap’s CUPS (http://localhost:631/ if only the Snap’s CUPS and no classic CUPS is running on the system, http://localhost:10631/ if a classic CUPS is also running) and create a print queue. When I am prompted for user name and password, I log in as a user who is in the lpadmin group. This works now! I get the queue!
And this is due to the fact that now CUPS was able to create the file in certs:

root@till-x1yoga:~# ls -l $SNAP_DATA/var/run/certs
total 4
-r--r----- 1 root root 32 May 25 20:25 0
root@till-x1yoga:~#

You see above that I had deleted my file and also CUPS changes the permissions of the file it creates, but it stays owned by root.
Now I do another test. The certs directory has already owner’s write permission. As I was not able to set the permission when the directory was owned by snap_daemon I switch ownership to snap_daemon now to see whether this combination actually works:

root@till-x1yoga:~# chown snap_daemon $SNAP_DATA/var/run/certs
root@till-x1yoga:~# ls -l $SNAP_DATA/var/run/
total 12
drwx--x--x 2 snap_daemon lpadmin 4096 May 25 20:43 certs
-rw-r--r-- 1 root        root       7 May 25 18:21 cups-browsed.pid
-rw-r--r-- 1 root        root       7 May 25 18:21 cupsd.pid
-rw-r--r-- 1 root        root       0 May  6 22:58 stop-cupsd.lock
root@till-x1yoga:~# touch $SNAP_DATA/var/run/certs/1
touch: cannot touch '/var/snap/printing-stack-snap/x31/var/run/certs/1': Permission denied
root@till-x1yoga:~#

But it does not work.
So owner of the certs directory has to be root and the owner’s write bit has to be set in addition to the other permissions.

Now my first question is, why is root in the classic system allowed to create and write files completely ignoring any ownerships and permissions and why is root in the Snap’s environment not allowed to do so?
It seems that AppArmor is not the reason, as I do not get any ...audit...DENIED... messages in the syslog.

Can I perhaps do some configuration setting in the Snap to allow roo to ignore file permissions and ownerships without dropping the restricted mode?

If not, I will have to patch CUPS to use root ownership and owner’s write permission on its certs directory. Does this compromise the security of my snapped CUPS?

@jdstrand, WDYT? How can I solve this problem?

2

Or will the CUPS Snap have to plug a certain interface for that to work?

3

I think the problem you’re running into is that the snap sandbox limits the capabilities of confined processes. In particular, without CAP_DAC_OVERRIDE you are limited to what the file system permissions allow, even as root.

For example:

root@scruffy:~# echo "Hello" >> foo
root@scruffy:~# chmod a= foo
root@scruffy:~# ls -l foo
---------- 1 root root 6 May 26 14:52 foo
root@scruffy:~# cat foo
Hello
root@scruffy:~# snap run --shell hugo
root@scruffy:/root# cat foo
cat: foo: Permission denied

If you want root to be able to read or write the file, then the file permissions need to allow it.

4

@jamesh, can I run cupsd with the CAP_DAC_OVERRIDE capability in the Snap? By changing cupsd’s file capabilities? By running it with some wrapper? …?

5

The capabilities are restricted by the AppArmor profile your app is running under. It could be expanded via rules introduced by the cups-control slot when that gets updated, but I think it would be better to determine why you need it.

If the desired access control can be expressed through standard file permissions, it would be better to just do that. The questions to ask include:

  1. who should be able to write files to this directory?
  2. who should be able to read files inside the directory?
  3. who should not be able to access the directory at all?
6

Thanks, @jamesh, I have patched CUPS now to set 711 permissions and root ownership of the certs directory. This way CUPS, which is running as root, can create and remove certificate files in this directory. Other processes have only to read the certificate files and for these CUPS creates them with the correct ownerships. Thes other processes only read the certificate files themselves, not the certs directory.
Note that the snapped CUPS only adjusts directory permissions/ownerships in dev mode but when installing in restricted mode afterwards these permissions/ownerships stay conserved, so I only discovered the problem by mixing install runs in dev mode and restricted mode.
Probably CUPS will actually have to run with CAP_DAC_OVERRIDE capability to take care of its directory permissions/ownerships by itself, otherwise they have to get pre-set by run-cupsd. I will also look whether I can patch the permission/ownership-setting function in CUPS to perhaps succeed to obtain desired settings by doing the steps in the correct order (switch ownership to root, set permissions, switch ownership to desired user).

7

Here is the patch which I have applied to CUPS, simply letting CUPS set more suitable permissions/ownerships for the certs directory:

8

Seems that the snapped cupsd is also restricted in which ownerships it can assign to files or directories, probably due to another missing capability.

In our cases it cannot set the group ownership of the certs directory to the CUPS admin group (“lpadmin” or “adm”) as the classically installed CUPS does.

By trying out I have found out that “root” as group ownership also works. It seems that only CUPS itself needs full access to the certs directory and CUPS itself runs as root. For the group ownership it seems only be important to not be “snap_daemon” (“lp” in classically installed CUPS) so that unprivileged sub-processes cannot access.

Here is the commit with this change:

@jamesh, @jdstrand, is this also an AppArmor restriction by blocking out a capability? Which capability in this case? Is there a simple way with snapcraft to enable this capability for cupsd?

9

I suspect it is actually the syscall filter that is blocking this. Using the lpadmin user is a variant of use case ‘1’ of Multiple users and groups in snaps (cc @pedronis). In this case, it sounds like your snap is trying to use an existing lpadmin or adm group on the system. This is currently unsupported.

We can possibly update the slot policy for the snap to do a getgrent() and add policy if one of these groups exists, but that is quite unusual for snaps and likely needs design. AIUI, you have designed your snap to have certain processes drop to snap_daemon and you don’t want them to be able to access these files. You want an additional group (ideally one that is widely used cross-distro) so that non-snap-daemon users that belong in this group can access the files, is that right? If so, @pedronis, this is similar to the microk8s case.

@till.kamppeter - today, sudo is needed to access those files.

10

The classically installed CUPS has defined a privilege-drop user, “lp” and its group “lp”. Filters, backends, and CGI programs are running as this user to drop privileges, while CUPS itself stays running as root. This allows CUPS to open privileged port 631 for sharing printers as IPP printers and also run some backends which only work with root rights, mainly to access printer hardware.

CUPS also has a facility to allow non-root users to do administrative tasks, like creating/modifying/removing queues or deleting someone else’s jobs. For this the user has to be in a system group of CUPS, in classic installation usually “lpadmin”. As the Snap cannot create a group in the system, it checks for existing suitable groups, first “lpadmin”, then “adm” and after that it allows only root to do admin tasks.

CUPS does not need to assign this group to any file, as I have found out by testing that the certs directory does not need it, certs should only be not accessible for the unprivileged users of CUPS and also the general public, see my previous post. CUPS only checks on incoming inquiries whether the client process is owned by a user in this group, to determine whether it accepts administrative tasks or not.

So we do not need to modify anything in the Snap infrastructure to allow CUPS to assign a file/directory to an arbitrary system-provided user/group. As it is now, with my latest commit, it is fine.

11

@jamesh, @jdstrand, thank you very much for all your help.