How to get snapd to work with external mounts?

veritanuda · January 10, 2018, 8:11pm

I posted this query a while back but got no response.

The setup is this. Debian 9 with a LVM logical volume mounted on /var/lib/snapd. This is because the main disk in an NVME one and is of a limited size reserved for home and root.

snapd appears to install correctly save the other I pointed out with apparmor config.

Preparing to unpack .../snapd_2.27.6-2_amd64.deb ...
Unpacking snapd (2.27.6-2) ...
Setting up snapd (2.27.6-2) ...
Created symlink /etc/systemd/system/multi-user.target.wants/snapd.autoimport.service → /lib/systemd/system/snapd.autoimport.service.
Created symlink /etc/systemd/system/timers.target.wants/snapd.refresh.timer → /lib/systemd/system/snapd.refresh.timer.
Created symlink /etc/systemd/system/sockets.target.wants/snapd.socket → /lib/systemd/system/snapd.socket.
Created symlink /etc/systemd/system/multi-user.target.wants/snapd.service → /lib/systemd/system/snapd.service.
Warning from /etc/apparmor.d/usr.lib.snapd.snap-confine.real (/etc/apparmor.d/usr.lib.snapd.snap-confine.real line 396): Unconfined exec qualifier (ux) allows some dangerous environment variables to be passed to the unconfined process; 'man 5 apparmor.d' for details.

snaps also appear to install fine no complaints.

snap install hello-world
2018-01-09T18:28:35Z INFO Waiting for restart...
hello-world 6.3 from 'canonical' installed

But running the snap proves impossible.

hello-world
cannot bind-mount the mount namespace file /proc/25509/ns/mnt -> hello-world.mnt: Permission denied
support process for mount namespace capture exited abnormally

I initially thought it was an apparmor thing but stopping apparmor did not seem to solve the problem.

The lv is mounted with standard perms.

/dev/mapper/vg_ssd-snap on /var/lib/snapd type xfs (rw,relatime,attr2,inode64,noquota)

And the snaps seem to mount fine too.

/var/lib/snapd/snaps/core_3748.snap on /snap/core/3748 type squashfs (ro,nodev,relatime)
tmpfs on /run/snapd/ns type tmpfs (rw,nosuid,noexec,relatime,size=3270688k,mode=755)
/var/lib/snapd/snaps/hello-world_27.snap on /snap/hello-world/27 type squashfs (ro,nodev,relatime)

perms on /proc seems normal to me too.

dr-xr-xr-x 501 root root 0 Jan 8 20:45 proc

So what is happening?

snapd 2.27.6-2 Linux 4.14.0-2-amd64

veritanuda · January 14, 2018, 6:46pm

So been digging a little more and apparmor does seem to be the culprit but it has to be a misconfiguration of /etc/apparmor.d/usr.lib.snapd.snap-confine.real

This is what happens when I run hello from the hello-world snap.

Jan 14 15:27:26 debian-9 kernel: audit: type=1400 audit(1515943646.077:1019): apparmor="DENIED" operation="capable" profile="/usr/lib/snapd/snap-confine" pid=18804 comm="snap-confine" capability=2  capname="dac_read_search"
Jan 14 15:27:26 debian-9 audit[18804]: AVC apparmor="DENIED" operation="capable" profile="/usr/lib/snapd/snap-confine" pid=18804 comm="snap-confine" capability=2  capname="dac_read_search"
Jan 14 15:27:26 debian-9 kernel: audit: type=1400 audit(1515943646.165:1020): apparmor="DENIED" operation="capable" profile="/usr/lib/snapd/snap-confine" pid=18804 comm="snap-confine" capability=2  capname="dac_read_search"
Jan 14 15:27:26 debian-9 audit[18809]: AVC apparmor="DENIED" operation="ptrace" profile="/usr/lib/snapd/snap-confine" pid=18809 comm="snap-confine" requested_mask="trace" denied_mask="trace" peer="/usr/lib/snapd/snap-confine"
Jan 14 15:27:26 debian-9 audit[18809]: AVC apparmor="DENIED" operation="ptrace" profile="/usr/lib/snapd/snap-confine" pid=18809 comm="snap-confine" requested_mask="tracedby" denied_mask="tracedby" peer="/usr/lib/snapd/snap-confine"
Jan 14 15:27:26 debian-9 kernel: audit: type=1400 audit(1515943646.189:1021): apparmor="DENIED" operation="ptrace" profile="/usr/lib/snapd/snap-confine" pid=18809 comm="snap-confine" requested_mask="trace" denied_mask="trace" peer="/usr/lib/snapd/snap-confine"
Jan 14 15:27:26 debian-9 kernel: audit: type=1400 audit(1515943646.189:1022): apparmor="DENIED" operation="ptrace" profile="/usr/lib/snapd/snap-confine" pid=18809 comm="snap-confine" requested_mask="tracedby" denied_mask="tracedby" peer="/usr/lib/snapd/snap-confine"

Something is definitely not right and it boils down to an apparmor config I think.

veritanuda · January 14, 2018, 7:33pm

Ok this is gettting frustrating now. I managed to remove dac_read_search error by adding capability dac_read_search, to the apparmor profile but the trace traceby I can’t fix because from the manpage.

AppArmor also restricts what privileged operations a confined process may execute, even if the process is running as root. A confined process cannot call the following system calls:
           create_module(2) delete_module(2) init_module(2) ioperm(2)
           iopl(2) ptrace(2) reboot(2) setdomainname(2)
           sethostname(2) swapoff(2) swapon(2) sysctl(2)

So how the hell does snapd EVER work normally?

zyga-snapd · January 15, 2018, 12:12pm

Hello.

I think that problems affecting Debian 9 are solved but the package is out of date. The current release is 2.30. (CC @mwhudson )

veritanuda · January 15, 2018, 12:26pm

Hmm really? It would explain a lot I guess. Debian does seem to be lagging behind.

apt-show-versions -a snapd
snapd:amd64 2.27.6-2 install ok installed
snapd:amd64 2.21-2+b1 stretch ftp.uk.debian.org
No stable-updates version
snapd:amd64 2.27.6-2  testing ftp.uk.debian.org
snapd:amd64/testing 2.27.6-2 uptodate
snapd:i386 2.21-2+b1 stretch ftp.uk.debian.org
No stable-updates version
snapd:i386 2.27.6-2  testing ftp.uk.debian.org
snapd:i386 not installed

Is there any way I can fix the apparmor stuff manually until newer packages come out ?

Thanks.

zyga-snapd · January 15, 2018, 1:11pm

The changes are in the apparmor profile and in the regular code. Perhaps it’s possible to just hack the apparmor profile into compliance but I honestly don’t know off the top of my head.

veritanuda · January 15, 2018, 1:54pm

Yeah … trying to build the 2.30 deb myself from source but hitting a bug somewhere. Anyone know why it is complaining about this?

returned exit code 1
debian/rules:120: recipe for target 'override_dh_auto_build' failed
make[1]: *** [override_dh_auto_build] Error 1
make[1]: Leaving directory '/usr/src/snapd-2.30'
debian/rules:85: recipe for target 'build' failed
make: *** [build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
debuild: fatal error at line 1152:

Which seems to point to these lines in debian/rules

override_dh_auto_build:
        # usually done via `go generate` but that is not supported on powerpc
        ./mkversion.sh
        # Build golang bits
        mkdir -p _build/src/$(DH_GOPKG)/cmd/snap/test-data
        cp -a cmd/snap/test-data/*.gpg _build/src/$(DH_GOPKG)/cmd/snap/test-data/
        dh_auto_build -- $(BUILDFLAGS) $(TAGS) $(GCCGOFLAGS)

Can’t for the life of me see what it wrong there…

zyga-snapd · January 15, 2018, 1:59pm

Can you try the 2.29.4 source package from ubuntu bionic please? The 2.30 package is stuck in process due to meltdown/spectre and impact on build farms.

veritanuda · January 15, 2018, 2:00pm

Sure I am building from git tree. I’ll let you know…

Edit: Hmmm fails with the same error. Odd.

Ahh I think that is a misdirection… it is nothing to do with the script it is to do with the build itself. Missing elements in the git repo!

xgettext-go/main.go:16:2: cannot find package "github.com/jessevdk/go-flags" in any of:
	/usr/lib/go-1.9/src/github.com/jessevdk/go-flags (from $GOROOT)
	/usr/src/snapd-2.29.4/_build/src/github.com/jessevdk/go-flags (from $GOPATH)

ogra · January 15, 2018, 2:18pm

just for reference:

and the corresponding commit is:

zyga-snapd · January 16, 2018, 8:22am

Snapd in Debian sid has just (last night) been updated to 2.30-4. I would be curious to see if that package works for everyone and if we can somehow do an update for Debian 9. Man thanks to @mwhudson who did the work!

zyga-snapd · January 16, 2018, 9:09am

I did some quick testing on a fully-up-to-date Debian sid just now and I could run spotify, gimp and a few less frequently used snaps.

veritanuda · January 16, 2018, 10:04am

Well from my side I added sid repo and pulled in the latest snapd, did not work initially because it needed libcap2 as well. So installed that but still had a few odd errors… reloaded apparmor still issues. Purged snapd and removed any dangling files (a README in /snap for some reason) and installed again.The result?

:~$ hello-world Hello World!

WooHoo!!!

I will try it with some other snaps and see if I can break it again

zyga-snapd · January 16, 2018, 10:14am

Woot, thank you! I wonder what the issues were from (the README file should be fixed in the next release, it was fixed in master AFAIK). Keep testing and share your experiences

veritanuda · January 16, 2018, 10:18am

Yup… Installed teleconsole and castersoundboard they both seem to work too! It is a bit of a relief because I was not sure if the mounting of /var/lib/snapd on a different device was causing the problems which would have been headache because like I say I have limited space on my NVME drive. For the record I am doing the same with flatpaks only they never seem to have a problem

Thanks for all the pointers though I have a much better understanding now of snap apparmor and the inherent problems of the namespacing snapd does.

Cheers.