Using snap changes does not go far enough to show what action actually broke the system, and snap task NNNN does not show any clear error that could be fixed, and snap abort NNN does not abort broken actions.
What is the proper way to forcefully remove broken packages and to somehow reset the snapd metadata so I get back to a functional state?
I ended up puring and re-installing Snapd to recover. I am posting here some notes for in case it is useful to others.
the snapd state can be browsed via sudo less /var/lib/snapd/state.json | jq | less but it was not useful in this case. The command snap changes shows only a few previous actions and the actions that originally broke my snapd instance were long gone.
To save all app data before purging snapd I first made sure all apps as closed and nothing is running and then I ran cd; mv snap snap-old to have the files saved. This is necessary as snapd removal would delete it (which is very unusual, no other Debian/Ubuntu removal will delete files inside users home directories).
After moving the /home/$USER/snap directory to another name I proceeded with running sudo apt purge snapd that removed everything, including apps and all metadata. Then I simply re-installed with sudo apt install snapd. After this the system was at a blank slate:
$ sudo snap list
No snaps are installed yet. Try 'snap install hello-world'.
The I re-installed all the apps I had previously with sudo snap install app1 app2 app3 etc. One of the apps I had wasn’t available in Snap store anymore, but I had a full system backup and was able to install it with sudo snap install path-to-backup/var/lib/snapd/snaps/whatsapp-for-linux_59.snap --dangerous. Obviously this won’t get any updates and is risky, but right now getting back running was a higher priority than reviewing which of the dozen WhatsApp versions in Snap store is the one to actually use going forward.
You should have taken a look at snap help save, which would be safer than just moving directories around (there might be scripts and whatnot that the app wants to execute on startup to process that data etc)…
By default uninstalling a snap takes a snapshot of all your user data, compresses it and stores it away. Once you reinstall the snap all application data gets restored… If you use --purge for the removal this step will be skipped though…
… or, as I and other asked, maybe posting some guidance about what could have gone wrong and how to fix it would go a long way instead of repeating “use --purge”.
The issue is deep inside the snap command, probably the boltdb that snapduses. And it is ugly.
Well, what could have gone wrong is that there is simply a lot of data (app caches or whatnot) to back up … or that you (or a script you ran) manually tinkered with file permissions in $SNAP_USER_DATA or $SNAP_USER_COMMON … researching this means looking at it case by case … the feature works in 99% of the cases
With the exact same snap? Note that the snap could ship some scripts that changed something in incompatible ways in SNAP_DATA or SNAP_COMMON, download things there that are immutable or whatnot… Again the hang of the command is just fallout and a symptom, you will need to research the cause first…
Wrong, a proper bug research needs to happen to find the root cause…
I just managed to brick snap, no abort nor --purge works. Steps I have done to brick it, knowing they are crazy, they were meant to restore broken snap from earlier steps, only making it worse. The whole process was due to ignorance on how snap works and no information whatsoever on the process to properly restore it, a simple reboot would’ve helped.
opened krusader and unmount all snap squashfs mounts, thought they were stale mounts from updates, they were the actual app mounted in its sandbox
remove manually /var/snap/firefox thinking this was easier way
perform sudo snap remove firefox
the process hangs, killed the process
tried to remove more snaps, both by removing manually and then via sudo snap
now it is in a corrupted state, some snaps work, some not, the ones that do not, cannot be removed, nor reinstalled or repaired.
never ever do this, these mounts are managed through generated systemd units, if you need to unmount something you should use systemctl so any depending pieces will become aware of the unmounting …
Well, you pulled out the carpet underneath snapd with this making it find an unexpected state of the package …
Yeah, that is the expected outcome of the above two actions, like when removing the ignition wires from your car and then wondering why the engine does not start
note that this has not really anything to do with the original post (snap remove and abort not working on a system that has not been tinkered with) and should have been its own new topic …
Fair enough, I replied here because the end result is the same, how to get there is very different probably.
If unmounting snaps is so to avoid it, why is it 3 clicks away in Krusader with no warning about the effect? I already said why I did it, it was due to ignorance, I wish snap would be able to recreate its internal json file and recover, that’ all, I am not justifying what I have done, like I said.
like when removing the ignition wires from your car and then wondering why the engine does not start
I disagree on the car statement though, snap was correctly installed, it should’ve been able to skip steps that fail and recover from anything, that is my personal opinion, not leave ghosts around that can’t be removed nor reinstalled.
Clearly the task is in a never ended state, which seems unrecoverable, this shouldn’t happen.
Well, but you grabbed into its guts and ripped out some pieces …
As I stated above this needs to be researched on a case by case (or snap by snap) basis since the end result is just a symptom but won’t tell you anything about the cause (which in your case was obviously the tinkering with backend bits of the packaging system))
Well, but you grabbed into its guts and ripped out some pieces …
This again doesn’t really mean anything, you could think the same for security bugs, people use them in a way that the author not intended to and tampered with, still a bug to fix, not to complain about.
If you want I can open a new thread with evidence, but if you think this is all by design, then I won’t waste my time. Are you a dev?
Well, ask yourself, if you had a problem with a dpkg package, would your first idea be to hand-edit /var/lib/dpkg/status (probably breaking more stuff for the whole packaging system) or modify files in /var/lib/dpkg/info vs. researching the actual error ?
Please forgive the bluntness, but how can the failure of snap remove (silently, with no error messages) plus the failure of snap abort (with no error messages) be interpreted as a failure of something else?
Even if this were the case, which probably is, snap abort failing silently is a very bad thing.
IMHO, given that my setup is very vanilla and that I just tried to remove firefox (maybe that’s the issue?), your suggestion that I must investigate is of very little help if you do not tell me where and what to investigate.
I am willing to help, I need some help in return.
Or, you can just say “reinstall snap/snapd”, which appears to be what people do anyway, with no certainty that it will properly work, given the apparent intertwining with Ubuntu.
I’m not saying there is no bug with snapd here, but the failure is the result of something, like not being able to perform the mandatory backup of user data when not using --purge …
Since the feature works most of the time for most people it is likely that this is snap specific (a script in the snap could have done something weird to the user data dir, there could be huge amounts of data that simply cause the backup procedure to take extremely long etc etc)
What I am trying to say here is that the actual cause should be researched, if there is zero info from snapd that is definitely also a bug that should be filed alongside but the behavior isnt a standalone thing that will just happen on its own without any underlying reason and that reason should be identified …
