I have a working system for packaging video games for arcade cabinets in distributed geographical locations. Think, hundreds to thousands of miles apart. These cabinets are not guaranteed to have an always-on internet connection, though most if not all locations have some kind of wifi for their point of sale systems, etc. They are not centrally managed, so any maintenance must be done through phone or email remote support. The cabinets act like vintage 1980s arcade games. The power is switched on and the game starts immediately.
My colleague and I have encountered an unexpected behavior in the snap refresh sub-system. The end result is games never update because they always start up before snapd (we think) which causes unexpected errors when both the refresh procedure and the current snap are running at the same time.
This differs from the pre-download condition, which is working fine. Given a stable internet connection and the game running, snapd will always pre-download the package to /var/lib/snapd/snaps as expected. The problem is snapd will never install this update because the current revision starts automatically with the login session.
My current idea is to add a pre-refresh hook to the snapcraft.yaml file which will check for a pre-downloaded package before refreshing. The goal is for an incomplete update to fail fast so the current revision of the game starts up as fast as possible. This is also why I’m not running snap refresh <app name> every time the cabinet PC boots. Waiting for network is not acceptable for these products.
Is there anything else I could look at? I have some logs of the failures, which I can reproduce on a test system.
Snapd has a refresh app awareness feature which will hold updates of a snap for as long as there is an application running associated with a given snap. Once the application terminates the refresh will be picked up again. Given what you described, there may be no actual opportunity for snapd to complete the refresh.
It is not quite clear to me what you plan to do in a pre-refresh hook. I don’t think the sandbox will allow the hook to poke at any locations under /var/lib/snapd/snap. The hook also runs before a step which unlinks the currently installed snap (i.e. makes it unavailable), where there’s an additional check for any running applications. Unless you somehow force stop the apps from a pre-refresh hook, I do not think it will make any difference.
What if you take a step back and look at the problem differently. Let’s say you keep the current approach of having the game be immediately available after the device has turned on. Now, there is a step where the device will get shut down. What if you launch the updates at this point? Applications which were running will likely already have their new payload available, which should make the whole process quicker. The overall UX isn’t that much far off from what happens on PS5 on shutdown.
Thanks for the advice, I agree that the snap never has an opportunity to refresh because the current revision is always running. My expectations were that the snap itself would be able to detect if a new revision has been pre-downloaded and refresh it before the app starts, but now that I’m typing this out I can think of some reasons how this could be bad. At least it should be an activity left to the user.
Fortunately, I spoke with my colleague and we might be able to just run snap refresh <app name> as part of the startup script, which already exists. Which leads me to my next question…
Can snapd install a new revision that has been pre-downloaded without an active Internet connection?
My results from last night indicate this is not possible but I might be missing something. Logs follow. The auto-refresh was executed without Internet.
lee@phillip:~$ snap changes
ID Status Spawn Ready Summary
156 Done yesterday at 13:50 PDT yesterday at 13:50 PDT Pre-download "perfect-pour", "chromium" for auto-refresh
157 Done yesterday at 14:10 PDT yesterday at 14:10 PDT Pre-download "perfect-pour", "chromium" for auto-refresh
158 Done yesterday at 14:35 PDT yesterday at 14:35 PDT Pre-download "perfect-pour", "chromium" for auto-refresh
159 Done yesterday at 15:00 PDT yesterday at 15:00 PDT Pre-download "perfect-pour", "chromium" for auto-refresh
160 Done yesterday at 15:35 PDT yesterday at 15:35 PDT Pre-download "perfect-pour", "chromium" for auto-refresh
161 Done yesterday at 15:55 PDT yesterday at 15:55 PDT Pre-download "perfect-pour", "chromium" for auto-refresh
162 Done yesterday at 16:20 PDT yesterday at 16:20 PDT Pre-download "perfect-pour", "chromium" for auto-refresh
163 Done yesterday at 16:39 PDT yesterday at 16:39 PDT Auto-refresh snap "chromium"
164 Error yesterday at 16:39 PDT yesterday at 16:40 PDT Auto-refresh snap "perfect-pour"
165 Done yesterday at 16:45 PDT yesterday at 16:45 PDT Pre-download "perfect-pour" for auto-refresh
166 Error yesterday at 17:33 PDT yesterday at 17:36 PDT Auto-refresh snap "perfect-pour"
lee@phillip:~$ snap tasks 166
Status Spawn Ready Summary
Done yesterday at 17:33 PDT yesterday at 17:36 PDT Ensure prerequisites for "perfect-pour" are available
Error yesterday at 17:33 PDT yesterday at 17:36 PDT Download snap "perfect-pour" (22) from channel "latest/edge"
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Fetch and check assertions for snap "perfect-pour" (22)
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Mount snap "perfect-pour" (22)
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Run pre-refresh hook of "perfect-pour" snap if present
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Stop snap "perfect-pour" services
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Remove aliases for snap "perfect-pour"
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Make current revision for snap "perfect-pour" unavailable
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Copy snap "perfect-pour" data
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Setup snap "perfect-pour" (22) security profiles
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Make snap "perfect-pour" (22) available to the system
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Automatically connect eligible plugs and slots of snap "perfect-pour"
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Set automatic aliases for snap "perfect-pour"
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Setup snap "perfect-pour" aliases
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Run post-refresh hook of "perfect-pour" snap if present
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Start snap "perfect-pour" (22) services
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Remove data for snap "perfect-pour" (10)
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Remove snap "perfect-pour" (10) from the system
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Clean up "perfect-pour" (22) install
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Run configure hook of "perfect-pour" snap if present
Hold yesterday at 17:33 PDT yesterday at 17:36 PDT Run health check of "perfect-pour" snap
Done yesterday at 17:33 PDT yesterday at 17:36 PDT Monitoring snap "perfect-pour" to determine whether extra refresh steps are required
......................................................................
Download snap "perfect-pour" (22) from channel "latest/edge"
2024-09-18T17:36:05-07:00 ERROR Get "https://api.snapcraft.io/api/v1/snaps/download/ZC6kuP1Gvo66EyuvYi4yrpgHxLIy9tvu_22.snap": dial tcp: lookup api.snapcraft.io: Temporary failure in name resolution
lee@phillip:~$ ls -lt /var/lib/snapd/snaps/ | grep perfect-pour
-rw------- 1 root root 54259712 Sep 18 16:45 perfect-pour_22.snap
-rw------- 2 root root 54247424 Sep 16 16:10 perfect-pour_21.snap
-rw------- 2 root root 54067200 Sep 14 12:16 perfect-pour_10.snap
Why is snapd trying to download a snap it has pre-downloaded earlier?
This part is interesting. Snapd downloads snaps to the cache under /var/lib/snapd/cache, where data is written to files with the snap’s sha3-384 hash for easier lookup. The files are then hard linked to /var/lib/snapd/snaps. However, in your case, the number of links for perfect-pour_22.snap appears to be 1 instead of 2. I’ll have a look at the snapd code, maybe there’s a bug lurking somewhere.
Can you confirm you did not copy the file yourself into /var/lib/snapd/snaps ?
Looks like an actual issue in snapd. I’ve filed an internal ticket for it and maybe we can get it in time for 2.66 release. As a temporary workaround I think you can copy the file to the side and then use snap install <new-rev-of-snap>.snap.
To confirm, I did not copy that file myself. It was copied there during a pre-download task while the game was running and there was an active internet connection.
Would you have the link to the issue on Launchpad? Me and my collaborator think we’ve discovered a second issue that might be caused or related to this bug.
