When could we expect this feature?
2.29 is going to be forked today so at earliest you will see parts of this in 2.30, I’ll update this post when something useful lands in edge.
Thanks! Would that mean weeks, or months away?
Snapd is on a monthly release. I’ll do my best to have everything needed in 2.30
1 Like
I’m on a good track to actually land something in 2.29, assuming @mvo agrees and deems it safe.
1 Like
I’d like to give everyone an update on where we are with this feature:
I’m working on two base features:
- simple creation of target / source directories for mounts / bind mounts (PR 4008)
- control of mode and ownership of created directories (PR 3965)
- generalized creation of directories on top of read-only filesystems using overlayfs (Git branch)
The most interesting one is the last one, I will open the PR soon but I’m still working on the undo logic and I want the prerequisite 4008 to land first. Once we have it open we need to examine how viable that is and if poses any security risks. I must say I’m really fond of this approach as it has the potential to simplify everything tremendously and is very elegant on the inside and at runtime.
After this I’d like to explore content interface to update it to the new specification. Using this set of features we should be able to easily create aggregation directories. In parallel we can start exploring using the layout work as all that would be missing is better definition of what is allowed and not allowed and connecting the layout specification with the existing mount backend.
given that not all kernels have overlayfs (specifically custom customer kernels) and we do not enforce it in our kernel plugin config checks either, will you have a fall-back implementation an application snap using layouts falls back to in this case ?
also, what happened to “overlayfs completely breaks LSM (apparmor/selinux)” which initially made us use bind mounts everywhere, was that fixed ? (@jdstrand ? )
I don’t have a fallback yet (I would argue I just barely get the implementation off the ground). The fallback is possible just more ugly and definitely more complex as it involves a very composite undo logic.
As for LSM I think it’s been fixed for SELinux recently, I’ll let @jdstrand comment about Apparmor. I’ll just note that we always use lowerdir=/something/from/squashfs,upperdir=/something/from/tmp. We don’t overlay two writable filesystems in any way. The only thing in the upper dir will be the extra directory node which will be used to create a bind mount to something from regular filesystems (squash or normal writable).
No, overlayfs does not work correctly with AppArmor in all cases. It may work well enough for the described functionality with some later versions of overlayfs but this would have to be extensively tested. I’m very concerned about the approach because while overlayfs was (aiui) introduced in 3.18, it was not designed to be used with LSMs and it underwent a lot of change over the years to get to the point where we are today (which still doesn’t work correctly with all the LSMs). Importantly, the changes required to get it to work with AppArmor are to overlayfs itself, not (just) the AppArmor subsystem. Also, it is unknown at this point how overlayfs will work with stacked LSMs.
I fear the feature will be brittle and a bug factory because the current approach for enablement is pick a kernel, drop the latest apparmor directory in place, run the kernel config checks and tweak here and there. Since overlayfs in 3.18 is different than in 4.14 precisely to address its LSM deficiencies (in part), and because people might try to backport apparmor in the manner described to every 3.18+ kernel, the matrix to verify things work correctly everywhere is large and highly labor-intensive.
The other concern is @ogra’s point that overlayfs is not readily backportable, so snaps that utilize the feature will be unusable on the popular 3.4, 3.10 and 3.14 kernels. It is presumed that overlayfs patches will need to be backported to 3.18+ kernels, but I don’t know how backportable those patches are.
If a fallback solution is going to be required for any kernels, why not simply focus on the fallback that is known to work everywhere?
I’m somewhat surprised in the change of direction. This topic has been discussed several times and the decision was always to not use overlayfs, but all of a sudden that decision seems to have been reversed when nothing in the landscape (backportability, LSMs) has changed appreciably.
If you’re talking about old broken Android kernels, then I would just ignore them. If you’re talking about distro kernels, the only 3.10 kernel I know of in active use is the RHEL kernel, and that one has overlayfs from 4.10 backported into it, as of RHEL 7.4.
I was mainly talking about BSP kernels from IoT board manufacturers and about already existing kernels in vendor stores that used the kernel plugin to create their snap etc…
I actually didn’t even think about all the other possibilities and corner cases that Jamie pointed out here. And indeed there are many other distro kernels we’d have to think about too…
Another small update. While I’m waiting for review for pre-requisites I reached a point where the transparent overlays work correctly with undo and update logic. I’ll focus my day on spread tests that attempt to break this as well as expanding coverage to other distributions.
At this stage I should be able to improve the content interface to do aggregation next week.
This looks super interesting to me. I had another post about using linking to do something similar.
@zyga-snapd I am wondering what the way forward with overlayfs is with the valid points @jdstrand raised above. We already have devices in production which as of today satisfy all requirements for snapd but with using overlayfs by default we’re putting new requirement up which will cost time to implement, test etc. for production devices and might not happen. So layouts will stay an optional feature for these devices and we have to refuse installation of snaps using it.
We can’t ignore any of these Android kernels as they are being used on various boards as a base. We don’t talk about std. distribution kernels here but about those coming with a board support package from a silicon vendor.
This is a valid concern, we need to both support existing devices that may not have a particular feature available as well as support the evolution of the snap platform.
Can you say with certainty that there is a device out there with a kernel that does not support overlayfs? Note that there certainly may be one but I’d like to know if you know about one in particular.
Did i read @jdstrand wrong above or wouldn’t we need the actual recent version backported to whatever kernel version to actually have it work even if it is enabled in the config (which the snapcraft kernel plugin does currently not do nor do the config test scripts require it ) ?
As i understood it only the very latest versions actually work … so you would have to additionaly backport it, regardless of having it “supported” …
Note that our use case of overlayfs is very specific so we may actually get away with it. We only use it to project tmpfs over a squashfs and only to create directories and only so that those directories can be bind mounted over.
That is what I understood too. Only the latest version works properly with AppArmor. So unless we have a device without confinement overlayfs needs to be backported.
We have devices running kernels down to 3.4 today. There was no work to get a newer overlayfs version integrated into any of these kernels so whatever was in the particular upstream release for overlayfs is available.
Thank you for confirming this.
What worries me here is that we silently expand the list of mandatory kernel features with this and not all involved parties (see @jdstrand response above) seem to be aligned on this.