Hi all,
I am snapping up an application which creates containers and the containers it creates need access to the host system, so the application will mount “/” as the snap sees it (which ends up being a combination of the core snap’s “/” as well as some other mounts snapd sets up). The problem with this is that then the container will only see files from the core snap, and anything from inside $SNAP isn’t visible for the application running inside the containers. This largely stems from this particular application being closed source and not configurable, otherwise it could just be made to also mount $SNAP for these containers, but I can’t change how that works unfortunately.
Things I have tried using the layouts feature in snapd:
- I bind mounted the entire rootfs for the snap into a full rootfs from a clean ubuntu container but this fails because snap-confine doesn’t work
- I tried just bind mounting /usr which works for the application in a way, but things like
snap run and hooks don’t work properly and the executables have to be externally run via /var/snap/application-name/.../binary args or /snap/application-name/current/.../binary args
- I tried just symlinking every single file that I need from the snap into the core snap - this failed to finish as apparmor runs out of memory and crashes my whole user session on bionic but I also don’t think this would work because the symlinks won’t resolve properly when inside the container (as the container won’t be able to see outside of it’s rootfs)
Any thoughts on how to proceed with this? In essence what I’m trying to do is make it look to the snap as if files that are in the application’s $SNAP are actually inside the core snap.
ping @zyga-snapd
Thanks,
Ian
Not any kind of expert here!!
One of the dev team over at lxd has written that lxd is on a special kind of snap where there is much more access into the system than is ‘normal’ (whatever that means!!!).
So knowing that it would seem that perhaps the way to do this is to start with snap which installs lxd which then has more access to the ‘system’ than a regular snap and then you build your application. AIUI lxd is fully recursive when on an Ubuntu system and the containers are some form of ubuntu so if you’re comfortable with that - - - -well you have maybe one possible solution. The dev team here may have another.
AIUI, the request is essentially for ‘confinement: classic’ but on an all-snaps (eg, Ubuntu Core) device. This is not supported by design.
There is not enough detail in what the application is (eg, it’s name, URLs, etc) and what it aims to do to suggest a path forward. In general, applications that manage containers don’t need full access to the host filesystem (eg, the lxd and docker snaps do not-- they store files in SNAP_DATA and SNAP_COMMON), but they do need an interface specific to them to allow their very privileged functionality. In fact, providing the access is not going to be terribly interesting since so much of the host filesystem is read-only anyway…
I have tried this and while running lxd containers inside lxd containers works with nesting, launching this kind of container does not work with lxd containers even privileged lxd containers. So unfortunately that solution won’t work here.
Hi @jdstrand,
This is not a request for a classic snap, as you mention this is for a snap that must run on an Ubuntu Core customer device. I am simply looking for help on how to strictly confine this snap. As you mention, docker and lxd can both be made to work within strict confinement, and so my question is how to do a similar thing for a closed source application that launches containers.
Perhaps I didn’t make this clear but I don’t wish to modify the core snap or make the core snap read/write and put files into the snap. What happens is that when this application is running inside a strictly confined snap (currently in devmode) it sees “/” as a read-only mount from the core snap file like this:
/var/lib/snapd/snaps/core_5072.snap on / type squashfs (ro,nodev,relatime)
And so thus when the containers are launched, the application mounts “/” for the containers like so (from a strace):
mount("no_source", "/var/snap/SNAP_NAME/x1/containers/rootfs", "overlay", 0, "lowerdir=/,upperdir=/var/snap/SNAP_NAME/x1/containers/somewhere_else,workdir=/var/snap/SNAP_NAME/x1/containers/somewhere_else2")
Note the "lowerdir=/" present in the options for the mount. This results in the core snap being mounted as “/” for the container child and as such cannot find any of the other dependency files that I have staged into $SNAP. So what I want to do is make it appear as if files from my $SNAP are actually in the rootfs of the core snap for this snap’s application specifically.
Things I have tried:
- Using layouts to bind mount an entire fresh rootfs from an ubuntu container (this fails see my original post)
- Using layouts to bind mount just “/usr” to “$SNAP/usr”. This kind of works, in that if I manually execute /snap/SNAP_NAME/bin/my-exec" the child containers have the necessary dependencies inside their rootfs. However, this breaks a lot of other things such as executing the
my-exec file through snap i.e. /snap/bin/my-exec symlink as well as doing snap run SNAP_NAME.my-exec or even snap run --shell SNAP_NAME.my-exec. All hooks also fail to start. I assume this is because there are files from the core snap’s “/usr” that aren’t present inside my snap (specifically I’m assuming something that /bin/bash needs…)
- Using layouts to symlink every single file from $SNAP into the appropriate location. This fails because there are thousands of such files and apparmor crashes trying to handle this during
snap install ...
- I have also tried using overlayfs to overlay
$SNAP/usr and $SNAP/lib such on top of /usr and /lib, etc. This seems like it works in that from snap run --shell SNAP_NAME I can see the required files in for example /usr/bin/python2.7, but unfortunately the overlayfs mount doesn’t seem to be followed when mounting the child container’s rootfs as the child container doesn’t see that file.
- I tried using overlayfs to just entirely mount
$SNAP on top of “/”, but this doesn’t seem to work and from my research I think that there are still open file descriptors to the old “/” and so the new “/” doesn’t ever show up… I think this usage of overlayfs only would work inside an initrd or initramfs scenario.
Is there anything else that I should try to get this to work?
Thanks,
Ian
Overlayfs is not allowed in any of the interfaces (except greengrass-support) and it sounds like this snap would need its own interface (overlayfs is also currently only usable in a few specific scenarios). Since it seems that you either cannot or do not want to divulge the snap name and fully describe the functionality here in the forum, it is a bit difficult recommending a course of action. Please send me a private email with more details.
