There are some users reported issues install new snap after fresh install[1][2][3][4][5] of Ubuntu Desktop 18.04 or 19.04. It seems to me they are all caused by the same snapd issue.
The seeded snap in a Desktop image will pull the latest snap based on a list of names in live-build config. If a developer builds a new bionic image today without upgrading the pre-seeded snap names[8], the image will end up shipping a new gnome-* snap (eg: gnome-calculator) depends on new gnome-3-28-1804.
Since only gnome-3-26-1604 is pre-seeded[7], snapd never able to solve the prerequisites of the new gnome-calculator. So “Initialize system state” task never gets done. When a user trying to install a new snap, the snapd will complain “too early for operation, device not yet seeded or device model not acknowledged”
When the users have the issue, they need to manually edit /var/lib/snapd/seed/seed.yaml and restart snapd and seeding to fix the dependency issue[5].
There is a similer issue when the pre-seed content snap does not have the right order[9], which blocking snapd to finish the initialization.
Should snapd ignore the broken dependency for seeding by just install the snap without the content snap? At least not blocking a user to solve the issue by installing a required snap.
I’m not understanding those sentences. What is the “seeded snap”, what is the “pull” process referred here?
snapd itself doesn’t pull anything during seeding, it uses what was configured in the image it’s in, if that configuration is inconsistent/broken something will break.
I think this means if you build a new 18.04 iso today it creates a seed that snapd can’t handle because the latest gnome-calculator needs gnome-3-28-1804 which isn’t in the list of seeded snaps for 18.04. That was added in 19.04, but the snaps that require the content have been updated since 18.04 was released.
I think what he’s getting at is that snapd needs to know how to resolve this missing content snap or maybe live-build needs to resolve it during image build.
the latter, snapd assumes that seeds are consistent and there is potentially no network at seed time (snapd could help with the image building if snap prepare-image --classic was used and with some more work on prepare-image --classic itself which has been discussed but is not scheduled).
Currently, the live-build script[1] is downloading snaps one by one for seeding. It seems to me using “snap prepare-image” is a better approach to pull snaps for seeding.
Not sure why the gnome content snaps needs to change it’s naming from gnome-3-26-1604, gnome-3-28-1804, but not using same snap name with different track. If all gnome-* snaps are using same track, we can make sure they have the required content snap when download/install using the same channel/track.
There are some Desktop images with the invalid seed.yaml. There are two approach to fix the issues.
Deploy a script via apt/deb to change the seed.yaml and restart initialization processes of snapd.
snapd ignore the prerequisites of gnome-*, and finish the initialization processes. So the user can use snapd to manually install gnome-3-28-1804 or other snap they need. The users can receive the new snapd via apt upgrade.
When building an ISO, the desktop seed[2] is read to determine which snaps to download and add for pre-seeding. The snap channel branch policy[1] is still effective; requiring a channel branch named “stable/ubuntu-18.04” to build 18.04 images. Once created, the channel branch can be closed again and the snap store will default back to the stable channel. As Ken pointed out, gnome-3-28-1804 was added for the disco release[5].
If a snap like gnome-calculator changes dependencies in a release and a new ISO can’t be built then the seed would need updating. (And the appropriate channel branch created to follow the policy) Alternatively, could a snap express the dependency as a ‘base’? The livecd-rootfs code could be amended to handle that[3] as it is already using the base to determine which core snaps need seeding.
Also, I’ve just looked at the source for ‘snap prepare-image’ as you’ve suggested is as a replacement for how we are doing things in livecd-rootfs. We need to iteratively pre-seed snaps as different hooks run for different image targets (and as derivative images build upon base images with pre-seeded snaps). The ‘prepare-image’ command does not appear to allow this at present[4], if I’m reading that right.
The 18.04 seed didn’t include gnome-3-28-1804, which was added post 18.04. As this is an LTS we needed to update the seed to include that as well as open up a stable/ubuntu-18.04 channel for it. I’ve updated the seed in git but the meta package needs to be updated in the archive still.
I wasn’t aware of this requirement so far, it’s not very natural but could be supported. But is it a case?
that livecd-rootfs literally takes an image (in some cases) and literally massages it into another one?
or that the image definition is layered where layers add snaps in a programmatic way at different points?
In either case does different model assertions get swapped in at different points?
The status quo is that people apparently trust livecd-rootfs to build working imagines (which then are not tested?) but that is not the case, it can create images that fail to seed in practice so something needs to change.
that sounds like they were not tested? as I explain ATM livecd-roots handling of setting up seeeding is too brittle to be trusted without testing.
I don’t see snapd pursuing 2. in a timely fashion. We could consider changing how snapd deals with failure/successes of seeding on classic, but that would still not produce a working system. As I said we cannot assume network/store access during seeding.
Agreed, but if snapd handles that failure/successes of seeding on classic better, the user at least can install the missing snap on their own when there is a network/store access.
Right now, the user can not do anything without fixing the seed.yaml.
this is unlikely to happen any time soon. Images need to be tested. @mvo is working on adding sanity checks to livecd-rootfs but still images need to be tested.
We added the “snap debug validate-seed” command to livecd-rootfs now. We also improved the checks in that command (https://github.com/snapcore/snapd/pull/7113) so once this is available in the image builds then any image with an incomplete seed.yaml will fail early and with a clear error message.
