Snap store download slow

[popey]

I noticed recently that installing snaps can be slow, specifically during the downloads step. I wanted to see if this was just me or more widespread so did some testing. I tested installing snaps on remote digital ocean servers in New York and London, and consistently get around 7MB/s / 60Mb/s. The graphs below are taken from the digital ocean dashboard, showing the peaks as I’m downloading snaps.

This is the script I used to test, which is just a big list of snap download foo for each snap:- http://paste.ubuntu.com/25184852/

London:

New York:

These boxes have super fast connections, and can get files from the archive nice and quickly, even when not using the hosting provider cache. But grabbing from the store feels super slow by comparison. Is this a known issue, and is it currently being investigated?

[roadmr]

So you’re getting what looks like about 40 Mb/s in London, and a very inconsistent 40-80 Mb/s from New York, right?

Can you try this command to attempt a core snap download?

curl $(curl -H 'X-Ubuntu-Series: 16' https://api.snapcraft.io/api/v1/snaps/details/core | jq '.download_url' -r)

This will spit out a redirect URL. Can you confirm this is an internap URL?

Next, please actually ask curl to follow the redirect:

curl -L $(curl -H 'X-Ubuntu-Series: 16' https://api.snapcraft.io/api/v1/snaps/details/core | jq '.download_url' -r) >/dev/null

and check the speed it reports for the last download (the actual snap; the others are just redirects and metadata).

If you could also try with libreoffice and/or other largish snaps and do a couple attempts for each and get us that data, that’d be appreciated.

FWIW, my own unscientific tests seem to back up your observation up to a point: I got about 40 MB/s (Bytes, not bits; this is what curl reports) for core downloads, over 10 attempts or so. A larger snap e.g. libreoffice, at 351 MB, got very erratic speeds: one attempt got 350 kB/s, another one got 10-11 MB/s, third attempt got 50 MB/s and fourth got about 200 kB/s again; a few subsequent tries saw speeds as high as 80 MB/s. rocketchat-server, at 150 MB size, got similarly erratic download speeds.

What I’m mainly interested in is whether these all come from internap URLs, because we may need to troubleshoot this on the CDN side. These don’t come directly from the store (unless they do which is why I’m asking you to confirm you’re getting Internap URLs).

• Daniel

[popey]

Thanks @roadmr for the prompt reply and debugging suggestions.

Tried this in NY and London. Both are using internap CDN. Trimmed these (correctly I think) to only show the data you wanted. Hope it’s useful.

NYC

root@ubuntu-2gb-nyc3-01:~# curl $(curl -H 'X-Ubuntu-Series: 16' https://api.snapcraft.io/api/v1/snaps/details/core | jq '.download_url' -r)
<p>You should be redirected automatically to target URL: <a href="https://068ed04f23.site.internapcdn.net/download-snap/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_2462.snap?t=2017-07-31T14:00:00Z&amp;h=2c3d5ddc23f0c64ba2fdeda7b0e6818ffab9935a">https://068ed04f23.site.internapcdn.net/download-snap/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_2462.snap?t=2017-07-31T14:00:00Z&amp;h=2c3d5ddc23f0c64ba2fdeda7b0e6818ffab9935a</a>.  If not click the link.
root@ubuntu-2gb-nyc3-01:~# curl -L $(curl -H 'X-Ubuntu-Series: 16' https://api.snapcraft.io/api/v1/snaps/details/core | jq '.download_url' -r) >/dev/null
100 80.4M  100 80.4M    0     0  17.2M      0  0:00:04  0:00:04 --:--:-- 26.1M
100 80.4M  100 80.4M    0     0  17.2M      0  0:00:04  0:00:04 --:--:-- 26.0M
100 80.4M  100 80.4M    0     0  45.2M      0  0:00:01  0:00:01 --:--:-- 88.5M
100 80.4M  100 80.4M    0     0  44.8M      0  0:00:01  0:00:01 --:--:--  105M
100 80.4M  100 80.4M    0     0  47.2M      0  0:00:01  0:00:01 --:--:--  100M


root@ubuntu-2gb-nyc3-01:~# curl $(curl -H 'X-Ubuntu-Series: 16' https://api.snapcraft.io/api/v1/snaps/details/libreoffice | jq '.download_url' -r)
<p>You should be redirected automatically to target URL: <a href="https://068ed04f23.site.internapcdn.net/download-snap/CpUkI0qPIIBVRsjy49adNq4D6Ra72y4v_21.snap?t=2017-07-31T14:00:00Z&amp;h=d40ba41da4bc07d6b14440805547da2f731fa0bb">https://068ed04f23.site.internapcdn.net/download-snap/CpUkI0qPIIBVRsjy49adNq4D6Ra72y4v_21.snap?t=2017-07-31T14:00:00Z&amp;h=d40ba41da4bc07d6b14440805547da2f731fa0bb</a>.  If not click the link.

root@ubuntu-2gb-nyc3-01:~# curl -L $(curl -H 'X-Ubuntu-Series: 16' https://api.snapcraft.io/api/v1/snaps/details/libreoffice | jq '.download_url' -r) >/dev/null
100  351M  100  351M    0     0  10.6M      0  0:00:32  0:00:32 --:--:-- 6144k
100  351M  100  351M    0     0  8831k      0  0:00:40  0:00:40 --:--:-- 6287k
100  351M  100  351M    0     0  14.6M      0  0:00:23  0:00:23 --:--:-- 6164k
100  351M  100  351M    0     0  83.5M      0  0:00:04  0:00:04 --:--:--  105M
100  351M  100  351M    0     0  79.1M      0  0:00:04  0:00:04 --:--:-- 98.5M

LON

root@ubuntu-2gb-lon1-01:~# curl $(curl -H 'X-Ubuntu-Series: 16' https://api.snapcraft.io/api/v1/snaps/details/core | jq '.download_url' -r)
<p>You should be redirected automatically to target URL: <a href="https://068ed04f23.site.internapcdn.net/download-snap/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_2462.snap?t=2017-07-31T14:00:00Z&amp;h=2c3d5ddc23f0c64ba2fdeda7b0e6818ffab9935a">https://068ed04f23.site.internapcdn.net/download-snap/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_2462.snap?t=2017-07-31T14:00:00Z&amp;h=2c3d5ddc23f0c64ba2fdeda7b0e6818ffab9935a</a>.  If not click the link.
root@ubuntu-2gb-lon1-01:~# curl -L $(curl -H 'X-Ubuntu-Series: 16' https://api.snapcraft.io/api/v1/snaps/details/core | jq '.download_url' -r) >/dev/null
100 80.4M  100 80.4M    0     0  23.2M      0  0:00:03  0:00:03 --:--:-- 18.2M
100 80.4M  100 80.4M    0     0  9664k      0  0:00:08  0:00:08 --:--:--  9.9M
100 80.4M  100 80.4M    0     0  7028k      0  0:00:11  0:00:11 --:--:-- 6126k
100 80.4M  100 80.4M    0     0  25.9M      0  0:00:03  0:00:03 --:--:-- 29.2M
100 80.4M  100 80.4M    0     0  12.1M      0  0:00:06  0:00:06 --:--:-- 13.7M


root@ubuntu-2gb-lon1-01:~# curl $(curl -H 'X-Ubuntu-Series: 16' https://api.snapcraft.io/api/v1/snaps/details/libreoffice | jq '.download_url' -r)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
<p>You should be redirected automatically to target URL: <a href="https://068ed04f23.site.internapcdn.net/download-snap/CpUkI0qPIIBVRsjy49adNq4D6Ra72y4v_21.snap?t=2017-07-31T14:00:00Z&amp;h=d40ba41da4bc07d6b14440805547da2f731fa0bb">https://068ed04f23.site.internapcdn.net/download-snap/CpUkI0qPIIBVRsjy49adNq4D6Ra72y4v_21.snap?t=2017-07-31T14:00:00Z&amp;h=d40ba41da4bc07d6b14440805547da2f731fa0bb</a>.  If not click the link.root@ubuntu-2gb-lon1-01:~# 
root@ubuntu-2gb-lon1-01:~# curl -L $(curl -H 'X-Ubuntu-Series: 16' https://api.snapcraft.io/api/v1/snaps/details/libreoffice | jq '.download_url' -r) >/dev/null
100  351M  100  351M    0     0  11.9M      0  0:00:29  0:00:29 --:--:-- 13.8M
100  351M  100  351M    0     0  4910k      0  0:01:13  0:01:13 --:--:-- 4489k
100  351M  100  351M    0     0  14.1M      0  0:00:24  0:00:24 --:--:-- 9457k
100  351M  100  351M    0     0  47.9M      0  0:00:07  0:00:07 --:--:-- 54.7M
100  351M  100  351M    0     0  37.1M      0  0:00:09  0:00:09 --:--:-- 49.3M

[chipaca]

“files from the archive” is not really comparing the same thing, as the archive is http and inside most clouds, as opposed to https to the public internet.

However…

getting https://dl.google.com/linux/chrome/deb/pool/main/g/google-chrome-unstable/google-chrome-unstable_61.0.3163.25-1_amd64.deb from our testing linode (which is on the west coast afaik) gives me a nice 18.0MB/s the first time, and over 100MB/s thereafter.

Just hitting https://api.snapcraft.io/api/v1/snaps/details/core takes a whole second, every time (0.96s ± 0.13s over 10 iterations).

Getting the snap goes at about 5MB/s the first time, and about 80MB/s thereafter, for both the download_url and anon_download_url.

[noise]

Also comparing apples to oranges. dl.google.com no doubt has a local server near your west-coast linode unit. Hitting api.snapcraft.io incurs latency from the west coast US to London. For the download, 80MB/s once the snap is cached in a local-ish CDN mirror seems pretty good to me.

[bashfulrobot]

Old thread I know, but curious if anyone is noticing any slowdown today. I’m on the west coast (Canada) and I’m getting 250-365 kB/s when installing snaps. Other sites, etc - no issues. Performing additional testing.

EDIT

Looks like it is routing related.

[bashfulrobot]

Actually - I had snappy-dev enabled (PPA), I purged that PPA, and all went back to normal.

Speed and other issues. Will report issues on the package.

EDIT

Reapplied PPA, then ended up movign from:

snap    2.30~rc1+17.10
snapd   2.30~rc1+17.10

to

snap    2.30~rc1+git467.c2f9631~ubuntu17.10.1
snapd   2.30~rc1+git467.c2f9631~ubuntu17.10.1

This is way off topic for this thread - last post, but wanted it in for completion sake so that I did not send anyone down a rabbit hole with my original post.

TLDR: speed issues were due to a bad state on my own machine.

[hazrpg]

I’ve noticed - despite having a 200/10Mbps connection that I get anywhere between 80kB/s to 700kB/s. I’ve been struggling to get freecad installed for the last couple of days because of it. Any news on what’s happening?

This is what I got near the end (after about an hour of downloading):

$ sudo snap install freecad
[sudo] password for hazrpg:
error: cannot perform the following tasks:
- Download snap "freecad" (4) from channel "stable" (received an unexpected http response code (504) when trying to download https://068ed04f23.site.internapcdn.net/download-snap/6gjU50q0lEDaMTtSUxYoVDSmjR5fERpB_4.snap?t=2018-03-28T14:00:00Z&h=92ac243150fc6e765bc55be77827fbc40f244f35)

[roadmr]

Hello!

The store suffered an outage yesterday which resulted in lingering slowness with snap transfers (both uploads and downloads). Please see:

Snap Store partial outage: slow uploads/downloads - round 2

Hello, We’ve received multiple reports of slow snap uploads/downloads and odd failures when trying to scan or release snaps. We’ve confirmed these behaviors stem from a slowness issue is on our origin servers and are working on the problem. Thanks for your patience and apologies for the disruption. Daniel

Snap push to store fails (no valid package)

Hello everyone, I’m currently in the process of pushing my snap to the store (as a private snap). I’ve registered the snap but when I push it I get the following problem: Pushing cavi_0.1.1_amd64.snap [===========================================] 100% Processing...| Error while processing... The store was unable to accept this snap. - __all__: The upload does not appear to be a valid package. If I install the snap locally with…

Extremely slow snap downloads

I was helping some other users based in China to identify the slow download issues. The download process will access to two servers - 1. api.snapcraft.io 2. CDN server, the mirror of the snap binary. I use the following script for the end user to report their network routing to these servers, which will help us to see if the CDN is too slow for certain ISPs or area. sudo apt-get install curl curl -L https://git.io/vxs3T | bash I did not find a quick way to see which CDN server will be ass…