Extremely slow snap downloads

I’m installing some snaps in a vm, and am getting chronic download speeds. I tried installing firefox which ended with Download snap "firefox" (71) from channel "stable" (unexpected EOF). When I tried a second time, I was getting ~100K/s - ~200K/s or so. My connection is fine, getting 57Mb/s according to fast.com. Is there some debugging I can do to help identify the bottleneck?

I suggest first enabling debugging in snapd

$ sudo systemctl edit snapd.service

[Service]
Environment=SNAPD_DEBUG=1 SNAPD_DEBUG_HTTP=7

$ sudo systemctl restart snapd.service

Then inspecting the logs using
$ journalctl -xeu snapd

Post what you find here if there’s nothing immediately obvious

Frustratingly I can’t reproduce this easily.

sit on the wire to reduce it’s throughput…

2 Likes

Had this again in an OpenSUSE VM (100-200K/s). I nuked the VM and restored it back and re-ran the commands with tracing on. Installing 11 snaps in a row, they all came down at 8MB/s. Really frustrating to not be able to reproduce this on demand.

Again I saw 100-200K/s with one snap download. I wiped (this time Fedora) VM and started again, and again all 11 snaps came down at rock solid 8MB/s. :unamused:

Hm, annoyingly reliable to do if I don’t set the debug options.

Here’s how I’m reproducing this.

  1. Boot up a clean VM (I have done this Ubuntu 16.04, OpenSUSE Tumbleweed and Fedora 27).
  2. Install snapd if it’s not already installed
  3. wget http://popey.com/snaps.sh
  4. chmod +x snaps.sh
  5. ./snaps.sh

This will install and launch a bunch of snaps for testing. When it gets to the 6th one which is node-red, it always drops to 100-200K/s download. If I nuke the VM, enable debug and do the above again, I cannot reproduce it.

Reproduced again on 17.10.

It does also happens to me sometime time… The process start with full speed than keeps on getting slower

@popey so if you set debug options, it works reliably? it’s a heisenbug! Jokes aside, it’ll really be necessary to have a look at which URLs you’re downloading from in order to be able to take action on this, and for that, debugging output would be most useful. The animated gifs are neat but I believe you even without them :slight_smile:

Snap downloads are served from a CDN and the URLs they provide vary according to several factors, which 1) explains why it happens only sometimes, but 2) really needs us to find a common factor in the slow ones so we can find a fix, which will most likely involve bringing this up with the CDN provider.

  • Daniel
1 Like

Right, I’m unable to do this on demand, with debug options enabled. Right now I’m downloading snapcraft from edge amd64, and it’s coming down at 55K/s and is gonna take about 11 minutes to install. What can I do if I can’t reproduce on demand? It’s incredibly frustrating and slows down my daily work dramatically.

I managed to run iftop and can see it’s grabbing from cdce.lon004.internap.com at 52K/s. My IP is 85.119.82.80 in case that’s any use.

I’m also getting abysmal performance downloading the beta core snap from cdce.mia005.internap.com:
Download snap "core" (4327) from channel "beta" oDownload snap "core" (4327) from channel "beta" 5% 52.4kB/s 26.0m

It later got worse to less than 10kbytes/s. Eventually it finished.

Well man - that’s debugging, sometimes you just can’t reproduce on demand, too many out-of-your control variables :slight_smile:

Thanks for people providing info on downloads and internap point-of-presence addresses!

Just to add another data point, I had a very slow download (down to 1.2 KB/s ) which eventually 503d:

- Download snap "node-red" (58) from channel "stable" (received an unexpected http response code (503) when trying to download https://068ed04f23.site.internapcdn.net/download-snap/lwGTfxA0S0e61kiyMAuXQkq6Tvni0dYm_58.snap?t=2018-03-27T18:00:00Z&h=bb8e523969e2db5e3b45ee5ba53001ddde3900e4)

This is the task. We can see it eventually sped up while I was fumbling around debugging options and ctrl-c’ing. lt took about 9min if I’m interpreting this correctly:

root@nsnx:~# snap tasks 18
Status  Spawn                 Ready                 Summary
Done    2018-03-27T14:05:33Z  2018-03-27T14:05:33Z  Ensure prerequisites for "core" are available
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:15Z  Download snap "core" (4327) from channel "beta"
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:17Z  Fetch and check assertions for snap "core" (4327)
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:17Z  Mount snap "core" (4327)
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:17Z  Run pre-refresh hook of "core" snap if present
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:17Z  Stop snap "core" services
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:17Z  Remove aliases for snap "core"
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:17Z  Make current revision for snap "core" unavailable
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:19Z  Copy snap "core" data
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:20Z  Setup snap "core" (4327) security profiles
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:20Z  Make snap "core" (4327) available to the system
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:23Z  Setup snap "core" (4327) security profiles (phase 2)
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:23Z  Automatically connect eligible plugs and slots of snap "core"
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:23Z  Set automatic aliases for snap "core"
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:23Z  Setup snap "core" aliases
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:23Z  Run post-refresh hook of "core" snap if present
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:23Z  Start snap "core" (4327) services
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:23Z  Clean up "core" (4327) install
Done    2018-03-27T14:05:33Z  2018-03-27T14:14:23Z  Run configure hook of "core" snap if present

We have confirmed the slowness issue is on our origin servers and are working on the problem, thank you for your patience.

5 Likes

I was helping some other users based in China to identify the slow download issues.
The download process will access to two servers - 1. api.snapcraft.io 2. CDN server, the mirror of the snap binary.

I use the following script for the end user to report their network routing to these servers, which will help us to see if the CDN is too slow for certain ISPs or area.

sudo apt-get install curl
curl -L https://git.io/vxs3T | bash

I did not find a quick way to see which CDN server will be assigned to the end user, it would be great if there is an API or mechanism to see where will the user download the snap binaries.

1 Like

We don’t offer a way to explicitly select CDN servers. Our primary CDN maintains a global set of PoPs that are selected from based on geolocation. Soon we are adding the ability to automatically redirect to cloud and region-specific CDNs that will help with overall robustness and responsiveness in some scenarios.

China of course brings its own set of challenges. Let’s chat after we have additional CDNs online and do some performance analysis from there.

1 Like

I have less than 100kBytes download speed while installing snaps
ubuntu 17.10 internet connection: 50/25MBit/s
sudo snap install --classic vscode
sudo snap install --classic skype
ctrl+C
journalctl -xeu snapd
log uploaded to gist

From my own experience, this is related to the github access too slow problem (the original post is in Chinese). Some IP addresses are just too slow (said to be rate-limited). While one can manually set the IP of the CDN node and get a performance boost (in my case, it’s 5kB/s v.s. 200KB/s), a robust workaround would be using an unnoticed DNS or setting up your own.

Hello,
I’m suffering the slow download in China. I can’t install any apps except for tic-tac-toe.
Interestingly, during the download, it shows that my speed was around 150,B/s, then the number will decrease. At the same time, my net speed indicator suggests that there’s no download at all.
Does anyone have an idea of what’s going on?
Thanks.