Snapd.service delays startup in Ubuntu 18.04 with 4.15.0-24


#1

Hi there,
I think my sytem is having a problem with snapd services… I don’t know how to diagnose the cause or how to fix this.

I’m running Ubuntu 18.04 LTS and since the last update this is the boot time of my system

➜ ~ systemd-analyze blame
9min 36.123s plymouth-quit-wait.service
8min 9.803s snapd.seeded.service
37.303s snapd.service
6.186s NetworkManager-wait-online.service
4.369s udisks2.service
1.360s dev-sda1.device

as you can see it takes around 10 minutes … but before I updated the system it was working fine with very fast boots.
Any help would be great!


#2

Is it like this on every boot?


#3

I’m experiencing the same issue after the latest upgrade of my Ubuntu 18.04. It takes about 6 minutes (every boot) to get the seeding process done. That’s terrible. I don’t know how long it took before I installed the update, but that was much more faster. Which info should I provide to help the devs to solve it?


#4

Hey sorry to jump in with a “me too” reply, but after an update yesterday I hit the same error. I was able to reproduce the error by doing the following:

  1. fresh install of 18.04
  2. allowing an update through the software updater application
  3. restart.

I thought the boot was freezing altogether but found that if I let it hang for 5+ minutes it would finally make it to the desktop. If it did alt+ctrl+f2 during the splash-screen It would show
“A start job is running for Hold until boot process finishes up (Xmin Xs/no limit)”
and a message saying it was starting snapd followed by
“[FAILED] Failed to start Snappy daemon”

I believe all of these links on AskUbuntu are displaying symptoms of this error and have been asked in the past few days.





#5

Thanks for your problem report! We are currently debugging and this and it would be great if you could provide some additional data.

What is the output of the following commands?

$ apt list snapd
$ snap version
$ systemd-detect-virt
$ uname -r

on the affected system(s) ?

[edit: added uname]


#6

After some digging into the problem it looks like this is a regression in the most recent linux-image-4.15 update. This is tracked in launchpad at https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1779961.

The kernel team is debugging this issue currently and we are looking at mitigation for snapd (i.e. ensure that snpad is not in the path that leads to login).


#7

Because of your snap, I had to lose important data from my computer, why are you laying out the code without thoroughly testing it? I get a hang of your Snappy Daemon and the system does not load


#8

How did you lose your data? From what we understand a regression in the kernel has caused the rate of random data generation to drop. This would not cause any data loss by itself from what we understand.


#9

I could not log in and had to reinstall it


#10

Sorry to hear that this is causing problems for you. We are currently looking into fixing the problem. As a workaround you can press the “shift” key and move the mouse during bootup this will make the login quicker. Regarding the testing - I can assure you that we test snapd very carefully, we run thousands of tests each time we change snapd. However when something external like the kernel changes behaviour we can’t do anything to foresee that. We are working on a fix now that we know what is going on.
[edit: spelling]


#11

There are recovery options on boot, you could select to boot the previous kernel which, to the best of our understanding, boot the system normally. In addition you can reinstall a system without wiping your data which would save you the trouble. In any case we are sorry for the issues you experienced. Snapd cannot block updates of the distribution kernel so this is something that was difficult to avoid.

As a matter of testing we test every single change we make on about 50 virtual machines running a variety of operating systems. Nothing is perfect but we take this very seriously.


#12

Hi! This is my output for the commands.

apt list snapd -a
Listing... Done
snapd/bionic-updates,now 2.32.9+18.04 amd64 [installed]
snapd/bionic 2.32.5+18.04 amd64

snap version
snap    2.33.1
snapd   2.33.1
series  16
ubuntu  18.04
kernel  4.15.0-24-generic

systemd-detect-virt
none

uname -r
4.15.0-24-generic

#13

While this looks like a bug in the kernel it is probably still worth to investigate why we need data from getrandom() at bootup.

Looking at our code it looks like we have two places:

  • catalogUpdate talks to the store at startup and that uses randomness
  • importing “gopkg.in/mgo.v2/bson” which triggers the initialization of var objectIdCounter uint32 = readRandomUint32() which reads 4 bytes of randomness from randr.Reader

The catalogUpdate runs in a goroutine (thanks Samuele for pointing this out) so that is not blocking the READY=1 notification for systemd.

However we also import mgo.v2/bson/ which initializes a variable using (objectIdCounter = readRandomUint32()) which reads rand.Reader. This happens before snapd main() and blocks with the new kernel. Replacing this with a time.Now().Unix() based value makes the problem go away (but I don’t know enough about bson to know if that is correct).

[edit: updated based on feedback from Sameuele]


#14
patrick@patrick-desk:~$ apt list snapd
Listing... Done
snapd/bionic-updates,now 2.32.9+18.04 amd64 [installed]
N: There is 1 additional version. Please use the '-a' switch to see it
patrick@patrick-desk:~$ snap version
snap    2.33.1
snapd   2.33.1
series  16
ubuntu  18.04
kernel  4.15.0-24-generic
patrick@patrick-desk:~$ systemd-detect-virt 
none
patrick@patrick-desk:~$ uname -r
4.15.0-24-generic

#15

This https://github.com/snapcore/snapd/pull/5464 will hopefully fix the issue for most people. Please note that its still not clear if this is not a kernel issue (I would argue it is) but even if it is the above fix should unblock people (will still need review/merge/SRU).


#16

And it looks like the kernel team has reverted the problematic update and is working on a fix now.


#17

Please, tell, how to use this fix. Can’t handle it :frowning:


#18

As a workaround you can press keys/move the mouse at bootup. The real fix is either a kernel upgrade or downgrade. But we will try to push a snapd update out that also will partially fix the issue.


#19

Thanks! Looking forward to seeing superfast ubuntu booting again


#20

A new snapd 2.33.1+18.04ubuntu2 has just hit the archive. Please try to upgrade to this version and that hopefully fixes the slow boot. The usual apt update && apt install snapd should work. It may take a bit until the package is available on all mirrors though.