Docker snap: silent failure of bind mounts

So, it turns out that there is a hardcoded list of “allowed” paths in snap, and if you attempt to make a bind mount outside of those allowed paths (e.g. /home/), the docker snap will merrily create the bind mount - but not point it to the actual path you give.

E.g. if you do

sudo mkdir /mybind
sudo docker run -it -v /mybind:/mybind alpine /bin/sh
#root : touch /mybind/myuniquename
#root : exit
ls /mybind
-there is no file-
sudo docker run -it -v /mybind:/mybind alpine /bin/sh
#root: ls /mybind
myuniquename

There will be no file myuniquename in your local storage under /mybind. Docker will run without error, and you will appear to have persistent storage, it will create a file “myuniquename”, and that file will still be there if you run the docker again, but that file is not where it’s supposed to be (ie. /mybind).

In fact, that file is nowhere to be found, making troubleshooting very difficult - i.e. trying to discover find / -name myuniquename does not return any results. Presumably this file exists “inside” the snap in some way that find can’t detect? I was not able to figure out where the file actually ended up.

Very strange/confusing behaviour. Installing docker using apt ofc works perfectly and as expected.

All of which is to say - if the docker snap doesn’t support very basic docker functionality, it should not have been released as the default docker install on e.g. ubuntu, and should be consider a dev/testing kind of thing locked away from regular users who reasonably expect docker to work correctly when installing it using the distros preferred method.

Do you get any denial errors in your journal? Else this is likely not related to the snap confinement…

Try creating a bind mount from $HOME/myuniquename to /uniquename outside of Docker, then passing Docker the path, as is mounted in $HOME.

If you’re having Docker create the bind mount itself, Docker is already in a sandbox by the time it tries to view your root directory, which is now no longer the directory you’re expecting and is a tmpfs at that.

If you create the bind mount outside the sandbox, but place it in a place the sandbox can access normally (so $HOME is valid), I’m thinking you might avoid any issues.

Snap can’t tell the difference between Docker trying to escape its own sandbox and a malicious app in Docker you’re not expecting trying to do the same, so it’s expected that should it try create a bind to somewhere not allowed, it’s having problems, on the other hand, I expect the Docker snap probably doesn’t need this workaround to bind mount to target folders in $HOME itself.

@James-Carroll - yes, this is exactly what’s happening. The probably most confusing part is the magic list of places that are allowed - e.g. /media. I was switching a docker app from a ZFS pool mounted at /media/mymount (on the magic list of allowed paths) to a ZFS pool mounted at /mymount (which ofc is not on the magic list).

I assume making a bind mount outside docker (eg in fstab) as you suggest would work, since then as far as the docker snap can tell it is writing to one of the magic paths.

@ogra - there are no errors in sudo journalctl --no-pager -u snapd, but that’s kind of the point - docker “works” silently and incorrectly, which is much worse than erroring out. It is highly confusing to check docker inspect, see the mount is correctly set up as /mybind:/mybind, and yet see different files inside and outside docker, and on top of that, be unable to locate those files anywhere on disk using standard tools like find. It is extra confusing when some paths work and some don’t, seemingly arbitrarily.

This should not be possible. If the docker snap cannot write to /mybind due to the rules of snaps, all well and good, but it should report something along the lines of The docker snap is attempting to write to /mybind, which is not on the list of paths which snaps can write to. See docs at bla bla for the paths to which snaps can write. and exit with an error code, so the user can understand and correct the issue.

Installing docker from the official apt repo fixed the issue, so it’s definitely either an issue with snap itself, or the interaction of snap and docker, or a bug in the version of docker ubuntu ships as a snap. I would say the first two, most likely - snap not performing the containment correctly (silently writing to some secret location instead of not allowing writes to a disallowed location), and possibly docker not detecting that and complaining.

This is not what I asked for, why would snapd have any errors, it isn’t involved here (like apt wouldn’t print errors for Deb packages at runtime)…

Please check without filtering if there are any denials when you write, confinement is applied by the kernel in snaps so filtering logs by snapd won’t show anything…

And that is exactly what happens if snap issues are involved, the kernel will report them in your logs, else it is something else and not related to snap confinement… You are making assumptions and accusations here but there is no data to back this… You should rather focus on what exactly the issue is instead of wasting energy on blaming snaps in general…

There might be some runtime or build time setting the docker binary has set/unset vs the Deb, there might be permission issues with your bind mount, you might be needing to use a different target for the mount in the snap vs the Deb etc etc ( I don’t use docker so I can’t tell, but I promise you that if you hit any snap boundaries the kernel will let you know about it)

This is not what I asked for, why would snapd have any errors

Well, what you asked for was totally unclear. My mistake for thinking that snapd might have something to do with snap functionality, obviously that was a crazy thing to think lol.

You could have said “grep journalctl for XX” or something, you know, actionable. Looking through potentially thousands of lines of logs for anything that might look like “a denial error” (not even a specific keyword) manually is not really feasible.

Since you still haven’t given any actionable advice for what to look for, I have grepped the journal for all mentions of snap, docker, and the paths in question, and there is nothing that looks like a “denial error”, which is as expected, since the behaviour is to silently perform incorrect behaviour, instead of erroring, which is basically my whole complaint.

Grepping for paths that work or the word “denied” interestingly turned this up:

<<DATE>> <<username>> kernel: audit: type=1400 audit(1730208612.533:139): apparmor="DENIED" operation="open" class="file" profile="snap.docker.docker" name="/myWorkingPath" pid=183085 comm="bash" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000

But again, that is a path that worked correctly, so would not appear to be relevant.

You are making assumptions and accusations here but there is no data to back this…

No, sorry, this is incorrect.

There is very clear data to back pointing to snap:

  • using the docker snap, binding to paths allowed in mount-support.c work correctly (the “magic paths”), paths not allowed there silently work incorrectly as described above
  • using docker from apt, all paths work correctly

That is very obviously a snap issue, barring some really bizarre coincidence.

There might be some runtime or build time setting the docker binary has set/unset vs the Deb

It would be very curious if those settings were the same as the paths allowed/not allowed to snaps. Possible, but would be a strange coincidence. You should rather focus on what is most likely to be the issue, instead of wasting energy on doing trying to exonerate snaps in general, to use your turn of phrase.

there might be permission issues with your bind mount

There are not. I went through extensive testing of this, and again - that would not be resolved by switching from the snap to the deb, and so is irrelevant.

you might be needing to use a different target for the mount in the snap vs the Deb

Well, that would be a snap problem, wouldn’t it lol, and would not address the fact that the snap silently performs incorrect and undesirable behaviour.

I don’t use docker so I can’t tell, but I promise you that if you hit any snap boundaries the kernel will let you know about it

Well, you can try the test I gave above very easily without needing to know much about docker. I gave repro steps, it would be polite to at least give it a shot before getting rude, making vague and unhelpful responses, etc. lol

But again, the problem is that it doesn’t error. It just works incorrectly. I would love to get a descriptive error and have the bind mount simply fail. Besides warning other users about this (afaict) undocumented misbehaviour, that is the whole point of my post - errors need to be treated as errors, and not silently work incorrectly. I don’t think that should be controversial.

Nor should it be controversial that if the snap has the issue and the apt doesn’t, that snap is the culprit until proven otherwise.

Sorry if you think I meant anything above being rude, I clearly didn’t mean it to come across rude…

The thing is that snap is a completely different system and things might work differently, this is not a defect as you claim very loudly. There is no error, so that indicates the snap system itself is not at fault, else you would see denials…

It can very well be that you need to do things differently due to the packaging system involved though (things are in different locations, only certain bits are writable etc etc) and that there is documentation of that fact missing… Blaming the packaging system as a whole for this is unfair though…

I’d start with filing an issue on the GitHub page of the docker snap and see what the maintainer says…

The thing is that snap is a completely different system and things might work differently, this is not a defect as you claim very loudly. There is no error, so that indicates the snap system itself is not at fault, else you would see denials…

The lack of error is the error. If things work differently (which, to the greatest extent possible, they should not), then the system needs to make that abundantly clear to the user. If it doesn’t, that is a defect. If it silently performs incorrect and unwanted behaviour, that is a defect. Maybe it’s snap itself, maybe it’s a gotcha in snap packaging that bit the team making the docker snap, but either way, it’s a snap issue at the core. Again, I’m surprised that’s controversial, but I guess we’ll have to agree to disagree.

I’ll leave the thread there, mostly I just wanted to document this experience so other users facing similar issues have some breadcrumbs to follow.

Linking github issue on docker snap here.

1 Like

As a last resort attempt, I’m curious if this works if you attempt to access your host /content folder, prefixing it with /var/lib/snapd/hostfs/ - in usual circumstances I’d expect this to be blocked by AppArmor, and you won’t even have the backup-files interface available which usually makes this read, not even write.

However, technically you’re not reading or writing it, you’re redefining it entirlely and doing magic with paths, I can’t promise it’ll work or it’ll even run, but, for science perhaps :)?

Hypothetically the Docker snap could potentially detect cases like this and perform the translation automatically. The snap also has its own interface giving it unique privileges just for itself, so perhaps making /var/lib/snapd/hostfs writable by default and trying to rewrite paths to use it, isn’t a bad idea?