Snapd.service delays startup in Ubuntu 18.04 with 4.15.0-24

Hi! This is my output for the commands.

apt list snapd -a
Listing... Done
snapd/bionic-updates,now 2.32.9+18.04 amd64 [installed]
snapd/bionic 2.32.5+18.04 amd64

snap version
snap    2.33.1
snapd   2.33.1
series  16
ubuntu  18.04
kernel  4.15.0-24-generic

systemd-detect-virt
none

uname -r
4.15.0-24-generic

While this looks like a bug in the kernel it is probably still worth to investigate why we need data from getrandom() at bootup.

Looking at our code it looks like we have two places:

  • catalogUpdate talks to the store at startup and that uses randomness
  • importing “gopkg.in/mgo.v2/bson” which triggers the initialization of var objectIdCounter uint32 = readRandomUint32() which reads 4 bytes of randomness from randr.Reader

The catalogUpdate runs in a goroutine (thanks Samuele for pointing this out) so that is not blocking the READY=1 notification for systemd.

However we also import mgo.v2/bson/ which initializes a variable using (objectIdCounter = readRandomUint32()) which reads rand.Reader. This happens before snapd main() and blocks with the new kernel. Replacing this with a time.Now().Unix() based value makes the problem go away (but I don’t know enough about bson to know if that is correct).

[edit: updated based on feedback from Sameuele]

1 Like
patrick@patrick-desk:~$ apt list snapd
Listing... Done
snapd/bionic-updates,now 2.32.9+18.04 amd64 [installed]
N: There is 1 additional version. Please use the '-a' switch to see it
patrick@patrick-desk:~$ snap version
snap    2.33.1
snapd   2.33.1
series  16
ubuntu  18.04
kernel  4.15.0-24-generic
patrick@patrick-desk:~$ systemd-detect-virt 
none
patrick@patrick-desk:~$ uname -r
4.15.0-24-generic

This https://github.com/snapcore/snapd/pull/5464 will hopefully fix the issue for most people. Please note that its still not clear if this is not a kernel issue (I would argue it is) but even if it is the above fix should unblock people (will still need review/merge/SRU).

1 Like

And it looks like the kernel team has reverted the problematic update and is working on a fix now.

1 Like

Please, tell, how to use this fix. Can’t handle it :frowning:

As a workaround you can press keys/move the mouse at bootup. The real fix is either a kernel upgrade or downgrade. But we will try to push a snapd update out that also will partially fix the issue.

1 Like

Thanks! Looking forward to seeing superfast ubuntu booting again

A new snapd 2.33.1+18.04ubuntu2 has just hit the archive. Please try to upgrade to this version and that hopefully fixes the slow boot. The usual apt update && apt install snapd should work. It may take a bit until the package is available on all mirrors though.

1 Like

just for completeness, this is

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1780062

1 Like

I’ve got the same issue.
systemd-analyze blame shows 1min 24.186s snapd.service

Thanks mvo for the temporary fix. I just booted to grub, selected an old kernel, and sure enough, fast boot time!
Fast settings below:

jayshomebrew@ubuntu:~$ apt list snapd
Listing... Done
snapd/bionic-updates,now 2.32.9+18.04 amd64 [installed]
N: There is 1 additional version. Please use the '-a' switch to see it
jayshomebrew@ubuntu:~$ 
jayshomebrew@ubuntu:~$ systemd-detect-virt 
none
jayshomebrew@ubuntu:~$ uname -r
4.15.0-23-generic
jayshomebrew@ubuntu:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04 LTS
Release:	18.04
Codename:	bionic
jayshomebrew@ubuntu:~$ snap version
snap    2.33.1
snapd   2.33.1
series  16
ubuntu  18.04
kernel  4.15.0-23-generic

Still seem to be getting the same issue.

tim@TimGJ:~$ sudo systemd-analyze blame
1min 49.280s snapd.seeded.service
     18.778s snapd.service
      7.026s NetworkManager-wait-online.service
      1.212s mariadb.service
       762ms fwupd.service
       747ms dev-sda1.device
       503ms dev-loop0.device
       493ms dev-loop3.device
       490ms dev-loop1.device
       487ms dev-loop2.device
       471ms dev-loop4.device
       468ms dev-loop7.device
       459ms dev-loop5.device
       459ms dev-loop6.device
       442ms dev-loop8.device
       431ms dev-loop9.device
       430ms dev-loop11.device
       430ms dev-loop10.device
       425ms dev-loop12.device
       418ms dev-loop14.device
       417ms dev-loop13.device
       401ms dev-loop15.device
       397ms dev-loop16.device
tim@TimGJ:~$ apt list snapd
Listing... Done
snapd/bionic-updates,now 2.32.9+18.04 amd64 [installed]
N: There is 1 additional version. Please use the '-a' switch to see it
tim@TimGJ:~$ snap version
snap    2.33.1
snapd   2.33.1
series  16
ubuntu  18.04
kernel  4.15.0-24-generic
tim@TimGJ:~$ systemd-detect-virt 
none

My father reported this issue on Friday, and three of my test systems fell ill on Monday & Tuesday. I can confirm 2.33.1+18.04ubuntu2 is not working, nor is the old 2.32.5.

I was only able to solve this problem by either configuring grub to boot revision 23 of the kernel, or by subscribing to the preview channel and upgrading to kernel 4.15.0.26

Hardware seems to play a role. I was able to boot my father’s Ubuntu on my coffeelake system without problem or delay. Coffeelake, skylake and apollo lake are all immune, while braswell and haswell are affected.

I am not sure this is a fault of snapd or something peculiar to revision 24 of the kernel

In my case, Alt+F2 brings me to the login screen.

Try Alt+F2 when it’s stuck on loading.

I reinstalled about a week ago before there was more debugging information available. Then the same thing happened to my wife’s laptop last night. Blocking on reading the random device reminded me of another problem that I had on startup of Ubiquity’s Unifi controller.

Installing haveged:

apt-get install haveged

Made my wife’s laptop boot very quickly. I believe that the problem is the random pool doesn’t have enough entropy at boot and the software reads from from random, not urandom which stirs the pool.

1 Like

Using the shift key works. It didn’t hold the boot and join in X.

➜  ~ apt list snapd   
snapd/bionic-updates,now 2.32.9+18.04 amd64 [installed]
➜  ~ snap version
snap    2.33.1
snapd   2.33.1
series  16
ubuntu  18.04
kernel  4.15.0-24-generic
➜  ~ systemd-detect-virt 
none
➜  ~ uname -r
4.15.0-24-generic

I’m having the same problem as reported here.

apt list snapd -a
snapd/bionic-updates,now 2.34.2+18.04 amd64 [instal·lat]
snapd/bionic 2.32.5+18.04 amd64

snap version
snap 2.35.2
snapd 2.35.2
series 16
ubuntu 18.04
kernel 4.15.0-36-generic

systemd-detect-virt
none

uname -r
4.15.0-36-generic

I experience this issue not always. Normally, when this problem appears, I shut down the computer and start it again. Then the process goes fine. Any idea about? Thanks in advance.

Dear All,
I am having the same issue on Dell XPS-13 running on Ubuntu 18.04:
gerald@gerald:~$ sudo service snapd status
● snapd.service - Snappy daemon
Loaded: loaded (/lib/systemd/system/snapd.service; enabled; vendor preset: en
Active: active (running) since Wed 2020-02-05 08:21:39 GMT; 1min 46s ago
Main PID: 783 (snapd)
Tasks: 20 (limit: 4915)
CGroup: /system.slice/snapd.service
└─783 /usr/lib/snapd/snapd

Feb 05 08:21:38 gerald systemd[1]: Starting Snappy daemon…
Feb 05 08:21:39 gerald snapd[783]: AppArmor status: apparmor is enabled and all
Feb 05 08:21:39 gerald snapd[783]: daemon.go:346: started snapd/2.42.1+18.04 (se
Feb 05 08:21:39 gerald snapd[783]: daemon.go:439: adjusting startup timeout by 1
Feb 05 08:21:39 gerald systemd[1]: Started Snappy daemon.
Feb 05 08:21:39 gerald snapd[783]: stateengine.go:150: state ensure error: Get h
Feb 05 08:21:39 gerald snapd[783]: hotplug.go:199: hotplug device add event igno

gerald@gerald:~$ sudo apt list snapd -a
Listing… Done
snapd/bionic-updates,now 2.42.1+18.04 amd64 [installed,automatic]
snapd/bionic-security 2.37.4+18.04.1 amd64
snapd/bionic 2.32.5+18.04 amd64

gerald@gerald:~$ snap version
snap 2.42.1+18.04
snapd 2.42.1+18.04
series 16
ubuntu 18.04
kernel 5.3.0-28-generic

gerald@gerald:~$ sudo systemd-detect-virt
none

gerald@gerald:~$ uname -r
5.3.0-28-generic

I don’t mean to necrobump, but I just wanted to say that I was having a somewhat related issue that was helped by updating my kernel from 4.14 to 4.19.

See here: Getty and Serial Getty don't seem to be initalizing properly

I’m still seeing symptoms, but boot times have gone from 15 minutes to 5 minutes. Still not acceptable, but getting there.