User config inconsistency with snap packages

1

I reported an issue for the lxd snap package wich we have, and according to Stéphane Graber this is a snapd bug:

There seems to be a config inconsistency for updated snap packages:

$ tail -v -n +0 $HOME/snap/lxd/*/.config/lxc/config.yml                                                                                                                         
==> ~/snap/lxd/18402/.config/lxc/config.yml <==
default-remote: local
remotes:
  images:
    addr: https://images.linuxcontainers.org
    protocol: simplestreams
    public: true
  local:
    addr: unix://
    public: false
aliases: {}

==> ~/snap/lxd/18884/.config/lxc/config.yml <==
default-remote: local
remotes:
  images:
    addr: https://images.linuxcontainers.org
    protocol: simplestreams
    public: true
  local:
    addr: unix://
    public: false
  roche:
    addr: https://some-remote.com:8443
    public: true
  roche-dev:
    addr: https://another-remote.com:8443
    auth_type: tls
    project: default
    protocol: lxd
    public: false
aliases: {}

==> ~/snap/lxd/19009/.config/lxc/config.yml <==
default-remote: local
remotes:
  images:
    addr: https://images.linuxcontainers.org
    protocol: simplestreams
    public: true
  local:
    addr: unix://
    public: false
  roche-dev:
    addr: https://some-remote.com:8443
    auth_type: tls
    project: default
    protocol: lxd
    public: false
aliases: {}

$ snap --version                                                                                                                                                                            [130]
snap    2.49
snapd   2.49
series  16
ubuntu  18.04
kernel  5.4.0-59-generic
2

Was roche-dev entry in the last config added after you noticed that the configuration is back to default?

Please provide the output of snap change <id> where <id> is the last auto-refresh that can be found by looking at the output of snap changes lxd.

Just for the record, is your home encrypted?

3

First of all thanks for the super fast response! :tada:

Exactly the remote was added after the auto update.

Here are the changes of snap (added the last 3 as I restarted the service a few times):

$ snap changes lxd
ID   Status  Spawn                     Ready                     Summary
93   Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Auto-refresh snaps "netron", "lxd"
94   Done    yesterday at 14:44 CET    yesterday at 14:45 CET    Running service command
95   Done    yesterday at 16:23 CET    yesterday at 16:23 CET    Running service command

$ snap change 95
Status  Spawn                   Ready                   Summary
Done    yesterday at 16:23 CET  yesterday at 16:23 CET  Run service command "restart" for services ["activate" "daemon"] of snap "lxd"

$ snap change 94
Status  Spawn                   Ready                   Summary
Done    yesterday at 14:44 CET  yesterday at 14:45 CET  Run service command "restart" for services ["activate" "daemon"] of snap "lxd"

$ snap change 93
Status  Spawn                     Ready                     Summary
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Ensure prerequisites for "netron" are available
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Download snap "netron" (158) from channel "latest/stable"
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Fetch and check assertions for snap "netron" (158)
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Mount snap "netron" (158)
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Run pre-refresh hook of "netron" snap if present
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Stop snap "netron" services
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Remove aliases for snap "netron"
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Make current revision for snap "netron" unavailable
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Copy snap "netron" data
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Setup snap "netron" (158) security profiles
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Make snap "netron" (158) available to the system
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Automatically connect eligible plugs and slots of snap "netron"
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Set automatic aliases for snap "netron"
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Setup snap "netron" aliases
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Run post-refresh hook of "netron" snap if present
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Start snap "netron" (158) services
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Clean up "netron" (158) install
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Run configure hook of "netron" snap if present
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Run health check of "netron" snap
Done    2 days ago, at 08:54 CET  2 days ago, at 08:54 CET  Ensure prerequisites for "lxd" are available
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Download snap "lxd" (19766) from channel "latest/stable"
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Fetch and check assertions for snap "lxd" (19766)
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Mount snap "lxd" (19766)
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Run pre-refresh hook of "lxd" snap if present
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Stop snap "lxd" services
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Remove aliases for snap "lxd"
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Make current revision for snap "lxd" unavailable
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Copy snap "lxd" data
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Setup snap "lxd" (19766) security profiles
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Make snap "lxd" (19766) available to the system
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Automatically connect eligible plugs and slots of snap "lxd"
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Set automatic aliases for snap "lxd"
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Setup snap "lxd" aliases
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Run post-refresh hook of "lxd" snap if present
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Start snap "lxd" (19766) services
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Remove data for snap "lxd" (19566)
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Remove snap "lxd" (19566) from the system
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Clean up "lxd" (19766) install
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Run configure hook of "lxd" snap if present
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Run health check of "lxd" snap
Done    2 days ago, at 08:54 CET  2 days ago, at 08:55 CET  Consider re-refresh of "netron", "lxd"

......................................................................
Consider re-refresh of "netron", "lxd"

2021-03-15T08:55:10+01:00 INFO No re-refreshes found.

The refresh time 08:55 matches with when I experienced the issue.

Yes the whole disk is encrypted, is this a problem?

4

Do you know what kind of encryption method is used? Is it encryptfs on $HOME only, or LUKS for the whole system/home partition?

5

It’s LUKS for the whole system except /boot:

nvme0n1                            259:0    0 238.5G  0 disk  
├─nvme0n1p1                        259:1    0   487M  0 part  /boot
├─nvme0n1p2                        259:2    0     1K  0 part  
└─nvme0n1p5                        259:3    0   238G  0 part  
  └─nvme0n1p5_crypt                253:0    0   238G  0 crypt 
    ├─vg--host--system-swap 253:1           0     1G  0 lvm   [SWAP]
    └─vg--host--system-root 253:2           0 213.1G  0 lvm   /
6

The refresh should copy ~/snap/<snap>/<rev> to ~/snap/<snap>/<new-rev> while it’s running, which effectively is a call to cp -av ... This is done for all users, there’s a glob that matches /home/*/snap in the system. The copy will fail loudly, but nothing clearly failed in your case and the task responsible for copying was successful. However, if the new path already exists, nothing will be copied, though it’s unclear why it would exist in your case or whether that actually happened.

Any chance your $HOME is attached over NFS maybe?

Can you try a little experiment? Now that you have modified the config.yaml file, while still logged in, can you run snap refresh --candidate lxd (I see it’s at revision 16808) and check whether the content gets copied correctly?

7

The home is not attached to nfs but it’s /home/users/$USER instead of the default /home/$USER could this be a problem?

Okay this seems to have failed successfully™

$ *sudo snap refresh --candidate lxd*
lxd (candidate) 4.12 from Canonical✓ refreshed
$ lxc remote ls           
To start your first instance, try: lxc launch ubuntu:18.04

+-----------------+------------------------------------------+---------------+-------------+--------+--------+
|      NAME       |                   URL                    |   PROTOCOL    |  AUTH TYPE  | PUBLIC | STATIC |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| images          | https://images.linuxcontainers.org       | simplestreams | none        | YES    | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| local (current) | unix://                                  | lxd           | file access | NO     | YES    |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| ubuntu          | https://cloud-images.ubuntu.com/releases | simplestreams | none        | YES    | YES    |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| ubuntu-daily    | https://cloud-images.ubuntu.com/daily    | simplestreams | none        | YES    | YES    |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+

So my previously added remotes are gone (and the config is different):

tail -v -n +0 $HOME/snap/lxd/*/.config/lxc/config.yml
==> /home/users/$USER/snap/lxd/19766/.config/lxc/config.yml <==
default-remote: local
remotes:
  images:
    addr: https://images.linuxcontainers.org
    protocol: simplestreams
    public: true
  local:
    addr: unix://
    public: false
  roche:
    addr: https://some-remote.com:8443
    public: true
  roche-dev:
    addr: https://another-remote.com:8443
    auth_type: tls
    project: default
    protocol: lxd
    public: false
aliases: {}

==> /home/users/$USER/snap/lxd/19808/.config/lxc/config.yml <==
default-remote: local
remotes:
  images:
    addr: https://images.linuxcontainers.org
    protocol: simplestreams
    public: true
  local:
    addr: unix://
    public: false
aliases: {}

==> /home/users/$USER/snap/lxd/current/.config/lxc/config.yml <==
default-remote: local
remotes:
  images:
    addr: https://images.linuxcontainers.org
    protocol: simplestreams
    public: true
  local:
    addr: unix://
    public: false
aliases: {}

And here is the change output:

$ snap change 97  
Status  Spawn               Ready               Summary
Done    today at 11:58 CET  today at 11:58 CET  Ensure prerequisites for "lxd" are available
Done    today at 11:58 CET  today at 11:58 CET  Download snap "lxd" (19808) from channel "latest/candidate"
Done    today at 11:58 CET  today at 11:58 CET  Fetch and check assertions for snap "lxd" (19808)
Done    today at 11:58 CET  today at 11:58 CET  Mount snap "lxd" (19808)
Done    today at 11:58 CET  today at 11:58 CET  Run pre-refresh hook of "lxd" snap if present
Done    today at 11:58 CET  today at 11:58 CET  Stop snap "lxd" services
Done    today at 11:58 CET  today at 11:58 CET  Remove aliases for snap "lxd"
Done    today at 11:58 CET  today at 11:58 CET  Make current revision for snap "lxd" unavailable
Done    today at 11:58 CET  today at 11:58 CET  Copy snap "lxd" data
Done    today at 11:58 CET  today at 11:58 CET  Setup snap "lxd" (19808) security profiles
Done    today at 11:58 CET  today at 11:58 CET  Make snap "lxd" (19808) available to the system
Done    today at 11:58 CET  today at 11:58 CET  Automatically connect eligible plugs and slots of snap "lxd"
Done    today at 11:58 CET  today at 11:58 CET  Set automatic aliases for snap "lxd"
Done    today at 11:58 CET  today at 11:58 CET  Setup snap "lxd" aliases
Done    today at 11:58 CET  today at 11:58 CET  Run post-refresh hook of "lxd" snap if present
Done    today at 11:58 CET  today at 11:58 CET  Start snap "lxd" (19808) services
Done    today at 11:58 CET  today at 11:58 CET  Remove data for snap "lxd" (19727)
Done    today at 11:58 CET  today at 11:58 CET  Remove snap "lxd" (19727) from the system
Done    today at 11:58 CET  today at 11:58 CET  Clean up "lxd" (19808) install
Done    today at 11:58 CET  today at 11:58 CET  Run configure hook of "lxd" snap if present
Done    today at 11:58 CET  today at 11:58 CET  Run health check of "lxd" snap
Done    today at 11:58 CET  today at 11:58 CET  Consider re-refresh of "lxd"

......................................................................
Consider re-refresh of "lxd"

2021-03-17T11:58:39+01:00 INFO No re-refreshes found.

I also tested this in a nested container, and this seems to have worked:

$ lxc launch ubuntu:20.04 test-nested
Creating test-nested
Starting test-nested                      
$ lxc exec test-nested bash               
root@test-nested:~# lxc remote ls
If this is your first time running LXD on this machine, you should also run: lxd init
To start your first instance, try: lxc launch ubuntu:18.04

+-----------------+------------------------------------------+---------------+-------------+--------+--------+
|      NAME       |                   URL                    |   PROTOCOL    |  AUTH TYPE  | PUBLIC | STATIC |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| images          | https://images.linuxcontainers.org       | simplestreams | none        | YES    | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| local (default) | unix://                                  | lxd           | file access | NO     | YES    |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| ubuntu          | https://cloud-images.ubuntu.com/releases | simplestreams | none        | YES    | YES    |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| ubuntu-daily    | https://cloud-images.ubuntu.com/daily    | simplestreams | none        | YES    | YES    |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
root@test-nested:~# lxc remote add roche-dev https://another-remote.com:8443 --accept-certificate --public
root@test-nested:~# lxc remote ls
+-----------------+-------------------------------------------------------+---------------+-------------+--------+--------+
|      NAME       |                          URL                          |   PROTOCOL    |  AUTH TYPE  | PUBLIC | STATIC |
+-----------------+-------------------------------------------------------+---------------+-------------+--------+--------+
| images          | https://images.linuxcontainers.org                    | simplestreams | none        | YES    | NO     |
+-----------------+-------------------------------------------------------+---------------+-------------+--------+--------+
| local (default) | unix://                                               | lxd           | file access | NO     | YES    |
+-----------------+-------------------------------------------------------+---------------+-------------+--------+--------+
| roche-dev       | https://another-remote.com:8443                       | lxd           | tls         | YES    | NO     |
+-----------------+-------------------------------------------------------+---------------+-------------+--------+--------+
| ubuntu          | https://cloud-images.ubuntu.com/releases              | simplestreams | none        | YES    | YES    |
+-----------------+-------------------------------------------------------+---------------+-------------+--------+--------+
| ubuntu-daily    | https://cloud-images.ubuntu.com/daily                 | simplestreams | none        | YES    | YES    |
+-----------------+-------------------------------------------------------+---------------+-------------+--------+--------+
root@test-nested:~# sudo snap refresh --candidate lxd
lxd (4.0/candidate) 4.0.5 from Canonical✓ refreshed
root@test-nested:~# lxc remote ls
+-----------------+-------------------------------------------------------+---------------+-------------+--------+--------+
|      NAME       |                          URL                          |   PROTOCOL    |  AUTH TYPE  | PUBLIC | STATIC |
+-----------------+-------------------------------------------------------+---------------+-------------+--------+--------+
| images          | https://images.linuxcontainers.org                    | simplestreams | none        | YES    | NO     |
+-----------------+-------------------------------------------------------+---------------+-------------+--------+--------+
| local (default) | unix://                                               | lxd           | file access | NO     | YES    |
+-----------------+-------------------------------------------------------+---------------+-------------+--------+--------+
| roche-dev       | https://another-remote.com:8443                       | lxd           | tls         | YES    | NO     |
+-----------------+-------------------------------------------------------+---------------+-------------+--------+--------+
| ubuntu          | https://cloud-images.ubuntu.com/releases              | simplestreams | none        | YES    | YES    |
+-----------------+-------------------------------------------------------+---------------+-------------+--------+--------+
| ubuntu-daily    | https://cloud-images.ubuntu.com/daily                 | simplestreams | none        | YES    | YES    |
+-----------------+-------------------------------------------------------+---------------+-------------+--------+--------+
8

Unfortunately the default copy mechanism will not work in this case. You can copy the files yourself though.

Supporting homes different from the usual /home/$USER is a bit more involved, you can try to follow the documentation for a way to address that: https://snapcraft.io/docs/home-outside-home

1 Like
9

Oh boy … this is very unexpected behavior, especially becuse it fails silently.
So the actual failure is that the copy fails? And then lxd just creates a default config?

Anyway thanks a lot for the fast help! I will see what I can do.

10

I’m a bit surprised why this config isn’t tracked in ~/snap/common which stays unchanged between snap revisions. Though I’m confident @stgraber knows better what’s best for the use cases supported by lxd. Perhaps there were some incompatible changes in the config format between different versions.

11

I think this would address the issue: https://github.com/lxc/lxd/issues/7322