Moving /var/lib/lxd

snapd
1

I have moved the contents of /var/lib/lxd to a different (larger) drive, and made /var/lib/lxd a symlink. Doing this causes all snaps to not launch:

$ vectr
cannot perform operation: mount /var/lib/snapd/hostfs/var/lib/lxd /var/lib/lxd -o nosuid,nodev,noexec,rbind,rslave: Permission denied

This was filed as an issue against a particular app; that user “fixed” the problem by moving the contents of /var/lib/lxd back again, but I can’t do that. How am I best to resolve this? It seems like a permissions issue of some kind?

2

Whenever I hit permission problems with symlinks I try bind mounts instead. I don’t know if that will help in this specific case, but maybe worth a try?

1 Like
3

that’s a really odd one, because as far as I can tell snapd doesn’t need to care anything at all about the contents of that folder or it’s location, to the extent that I don’t have the folder on my system:

$ ls -l /var/lib/lxd
ls: cannot access '/var/lib/lxd': No such file or directory
4

Bind mounts might fix it, indeed. (That’s annoying, because “just drop a symlink in place” works forever with no effort, whereas “do a bind mount” requires adding to /etc/fstab.) I’ll giv e that a try… but I feel like snaps should not be broken in this situation anyway :slight_smile:

1 Like
5

I suspect it is an issue with the snap-confine AppArmor profile. Do you have any apparmor denied messages in journalctl?

1 Like
6

Ooh, I do!

Dec 18 15:05:07 dawn audit[1881]: AVC apparmor="DENIED" operation="mount" info="failed srcname match" error=-13 profile="/snap/core/3604/usr/lib/snapd/snap-confine" name="/var/lib/lxd/" pid=1881 comm="snap-confine" srcname="/media/second_drive/lxd/" flags="rw, rbind"
Dec 18 15:05:07 dawn kernel: audit: type=1400 audit(1513609507.233:129): apparmor="DENIED" operation="mount" info="failed srcname match" error=-13 profile="/snap/core/3604/usr/lib/snapd/snap-confine" name="/var/lib/lxd/" pid=1881 comm="snap-confine" srcname="/media/second_drive/lxd/" flags="rw, rbind"

Do I need to poke some config somewhere to tell it about my new directory?

7

If you are running the candidate core snap, you can drop a file into /var/lib/snapd/apparmor/snap-confine with the rules you want. Eg:

mount options=(rw rbind nodev nosuid noexec) /media/second_drive/lxd/ -> /var/lib/lxd/,

Then do:

sudo apparmor_parser -r /etc/apparmor.d/*snap-confine*

(this reloads all of the snap-confine profiles, but you only need to reload the one that corresponds to the revision of the current core snap). You may need to add other rules after adding this.

1 Like
8

Just to confirm: this did fix things. Thank you!

1 Like
9

I just hit this. Thank you for going through the pain already and providing Google results for me :slight_smile:

It seems to me that this has the potential of one of two bugs:

  1. snapd should be able to handle this case without any further intervention.
  2. If it can’t, then it should at least identify what the problem is and tell me what to do.

I don’t really follow what my LXD setup has to do with a snap that doesn’t need it in any way.

Do you want a bug report for this, and if so, for what issue and where?

10

Note that in general we don’t recommend or support /var/lib/lxd being a symlink. It has a tendency to confuse the various LSMs and other kernel security measures in place around the mount namespace.

That’s not to say that symlinks can’t work, just that we’ve seen a lot of subtle bugs that resulted from people using them under LXD, all of which went away after we asked the reporter to switch to bind-mounts :slight_smile:

So our (LXD upstream) position is that we won’t support anyone who’s using a symlinked LXD directory, instead they should switch to bind-mounts and attempt to reproduce their issue again with that setup.

1 Like
11

Is there some documentation on how to migrate a running system to use bind-mounts instead? Adding a symlink was very easy, hence choosing it. Perhaps you could direct me to somewhere explaining how to use a bind-mount for this use case instead, because I don’t know how :slight_smile:

12 
systemctl stop lxd lxd.socket
SOURCE=$(readlink -f /var/lib/lxd)
rm /var/lib/lxd
mount --bind ${SOURCE} /var/lib/lxd
systemctl start lxd

That should take care of replacing your symlink by a proper mount.
To make it persistent, you’ll need a matching /etc/fstab entry, something like:

/path/to/wherever/your/data/is /var/lib/lxd none bind 0 0
2 Likes
13

OK, that seems to have worked, thank you. (One small change; an mkdir /var/lib/lxd is needed after rm /var/lib/lxd, so that there’s something to bind-mount onto.)

14

Should lxd perhaps refuse to start in this scenario? Otherwise what I get as user is no indication at the time that this will be problematic and not a particularly useful error message from snapd when it later fails.

15

If we were to redo LXD, we’d add such a test yes. Right now, there are working setups that unfortunately use symlinks and if we were to make that change, those users would break automatically on upgrade which is unacceptable.

We probably should be logging an error message though so that users will hopefully spot that at some point and fix their setup to something more reliable.

16

While I understand this point entirely, and it’s a good thing that you won’t break users as a matter of policy, I got the impression from what you said above that such users (me included, until yesterday) were already broken, right? Having /var/lib/lxd be a symlink is bad in some ways (otherwise you wouldn’t have said that I shouldn’t do it), so I was already broken, surely? Having it stop on upgrade, with a note saying why (and explaining how I can flip some allow_symlinks_even_though_you_shouldnt back to true in the config) seems better than just being silently mysteriously broken for ever, especially since if I actually break I’ll file a bug and the answer will be “well, you shouldn’t have been doing that in the first place” and then I just look sad :slight_smile:

17

Heh. It comes back to “mcphail’s guide to Linux”: If symlinks break, just use bind mounts; best to not ask questions :wink: