Creating a netboot Ubuntu image that'll have Snap apps

[YamiYukiSenpai]

Its related to this, but I figure this should be a separate topic, so I deleted it

Snaps lost their "connections"

I have a VMs that I play with from time to time, and Snaps would “lose” their connections, and can’t launch because of that. $ firefox sed: can't read /home/yamiyukisenpai/.config/user-dirs.dirs: Permission denied /snap/firefox/5889/gnome-platform/command-chain/desktop-launch: line 290: /home/yamiyukisenpai/.config/user-dirs.dirs: Permission denied cp: cannot open '/home/yamiyukisenpai/.config/user-dirs.locale' for reading: Permission denied /snap/firefox/5889/gnome-platform/command-chain/deskt…

We’re planning on converting our setups from a classic install into a Netboot setup. Directories, like /home will be on the persistent storage, and I was thinking of making the Snaps and Flatpak directories in the persistent storage as well.

However, based on this

@mborzecki1

FYI the state is split in 2 pieces. Snapd keep its own internal state at /var/lib/snapd/state.json, and then there’s a the state of system, i.e. various artifacts that snapd generated during snap installation such as systemd mount units. Note, that in case of mounts units they reference snap blob files which are at /var/lib/snapd/snaps.

It looks to me that you are preserving only a part of that ‘overall’ state. Specifically the snapd bits, but not the system ones. So you end up in a scenario when systemd mount units from one system refer to files that are no longer present in the shared /var/lib/snapd/ location. On top of this, snapd state file seems to refer to snaps in revisions which there are no mount units for in a particular VM instance, only because the installation or last referesh was done on another system.

With a netboot, its safe to assume the same issue as what I encountered when I put Snaps in a dedicated/shared storage virtual drive will pop up because the systemd services is in the netboot image, and not in the persistent storage.

Is there a way to make the Snaps be placed in persistent storage and won’t need to be “reset” every boot?

Edit: Flatpaks aren’t updated automatically, so maybe having different versions of Flatpak scattered isn’t such a bright idea, unless we setup a server whose main purpose is to host those Flatpaks.

[mborzecki1]

I think I’m bit confused about the approach and I’m not sure what you are trying to achieve, so take my suggestions with a grain of salt.

Netboot systems I’ve dealt with were usually fully ephemeral. No data was preserved locally, or at least not permanently, to prevent the problem of system state going out of sync with what is installed. Note, this isn’t a problem unique to snaps. User data was typically attached through NFS. Those systems usually employed overlayfs in some way, with upper located on some local storage that got wiped during (next) boot.

These days you can also play with immutable systems. Only /usr is pre-populated with data, while the rest of the system is empty and may get populated during boot. See the systemd’s vision of that https://0pointer.net/blog/fitting-everything-together.html

With snapd, the snaps and files generated by the system at runtime are part of system state. You do not share system state if you cannot either ensure full compatibility, which as indicated in other posts is particularly tricky. However, if you follow the fully ephemeral system way, you can auto install a set of snaps by placing them in the seed directory (/var/lib/snapd/seed). It’s essentially what Ubuntu livecds do. In your case however, user data could come from network, rather than stay in tmpfs. I think you could even go as far as attaching the seed from network, so that you do not have to rebuild the image each time you want to change the default set of snaps.

Overlayfs on /, it needs to be set up before inird pivots into the target system. You need to do that in the initrd.

[YamiYukiSenpai]

@mborzecki1

No data was preserved locally, or at least not permanently, to prevent the problem of system state going out of sync with what is installed.

For our Rocky 9 workstations, we do have NFS servers, but we also have an /ondisk partition that’s either using symlink or bind. I’ve also setup a way to bind /ondisk/home to /home back when couple of our sites didn’t have an NFS server setup for their home profiles and would essentially bind at boot if the server isn’t mounted.

It’s also what we’re doing to ensure that the licenses that require to be in the system are stored in the system itself (i.e. Davinci Resolve Studio). I did wonder if I’d be able to store it on the user profile itself, but I didn’t have time to test the theory and essentially created a symlink that points Resolve’s license directory to /ondisk/resolve/license

@mborzecki1

These days you can also play with immutable systems. Only /usr is pre-populated with data, while the rest of the system is empty and may get populated during boot. See the systemd’s vision of that https://0pointer.net/blog/fitting-everything-together.html

I’ll give this a read later. Thanks for this.

@mborzecki1

With snapd, the snaps and files generated by the system at runtime are part of system state. You do not share system state if you cannot either ensure full compatibility, which as indicated in other posts is particularly tricky. However, if you follow the fully ephemeral system way, you can auto install a set of snaps by placing them in the seed directory (/var/lib/snapd/seed). It’s essentially what Ubuntu livecds do. In your case however, user data could come from network, rather than stay in tmpfs. I think you could even go as far as attaching the seed from network, so that you do not have to rebuild the image each time you want to change the default set of snaps.

Is there a documentation for this that I can peruse? In addition, I assume that the snaps that’ll be installed will be whatever install version is in the seed directory? How does autoupdate handle this if the Snap data in the user’s home profile is for the “newer” version than the ones that’ll be installed?

@mborzecki1

I think you could even go as far as attaching the seed from network, so that you do not have to rebuild the image each time you want to change the default set of snaps.

Unless this suggestion would mean the snaps would be autoupdated without needing to rebuild?

[mborzecki1]

@YamiYukiSenpai

Is there a documentation for this that I can peruse? In addition, I assume that the snaps that’ll be installed will be whatever install version is in the seed directory? How does autoupdate handle this if the Snap data in the user’s home profile is for the “newer” version than the ones that’ll be installed?

I looked at the docs and there isn’t anything on the seeding process. It’s not something anyone needs to play with. Try launching 22.04 VM image and look under /var/lib/snapd/seed. Snaps from the seed will be automatically installed during boot, as long as .data.seeded in state.json isn’t set to true.

Though, thinking more about it, you do not seem to be starting from scratch on each boot? I understand /ondisk carries actual system state. In that case, once seeded, the process would not be repeated on another boot.

How about you populate the snapd seed directory, but have a systemd service which goes though the files in seed and calls snap install for each one. That is assuming that you either look at the file or the bundled revision assertion and only do that if the revision is indeed different? Or even not the snapd seed, but some other source you control.