Right, I’m unable to do this on demand, with debug options enabled. Right now I’m downloading snapcraft from edge amd64, and it’s coming down at 55K/s and is gonna take about 11 minutes to install. What can I do if I can’t reproduce on demand? It’s incredibly frustrating and slows down my daily work dramatically.
I managed to run iftop and can see it’s grabbing from cdce.lon004.internap.com at 52K/s. My IP is 85.119.82.80 in case that’s any use.
I’m also getting abysmal performance downloading the beta core snap from cdce.mia005.internap.com:
Download snap "core" (4327) from channel "beta" oDownload snap "core" (4327) from channel "beta" 5% 52.4kB/s 26.0m
It later got worse to less than 10kbytes/s. Eventually it finished.
Well man - that’s debugging, sometimes you just can’t reproduce on demand, too many out-of-your control variables 
Thanks for people providing info on downloads and internap point-of-presence addresses!
Just to add another data point, I had a very slow download (down to 1.2 KB/s ) which eventually 503d:
- Download snap "node-red" (58) from channel "stable" (received an unexpected http response code (503) when trying to download https://068ed04f23.site.internapcdn.net/download-snap/lwGTfxA0S0e61kiyMAuXQkq6Tvni0dYm_58.snap?t=2018-03-27T18:00:00Z&h=bb8e523969e2db5e3b45ee5ba53001ddde3900e4)
This is the task. We can see it eventually sped up while I was fumbling around debugging options and ctrl-c’ing. lt took about 9min if I’m interpreting this correctly:
root@nsnx:~# snap tasks 18
Status Spawn Ready Summary
Done 2018-03-27T14:05:33Z 2018-03-27T14:05:33Z Ensure prerequisites for "core" are available
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:15Z Download snap "core" (4327) from channel "beta"
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:17Z Fetch and check assertions for snap "core" (4327)
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:17Z Mount snap "core" (4327)
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:17Z Run pre-refresh hook of "core" snap if present
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:17Z Stop snap "core" services
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:17Z Remove aliases for snap "core"
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:17Z Make current revision for snap "core" unavailable
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:19Z Copy snap "core" data
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:20Z Setup snap "core" (4327) security profiles
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:20Z Make snap "core" (4327) available to the system
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:23Z Setup snap "core" (4327) security profiles (phase 2)
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:23Z Automatically connect eligible plugs and slots of snap "core"
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:23Z Set automatic aliases for snap "core"
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:23Z Setup snap "core" aliases
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:23Z Run post-refresh hook of "core" snap if present
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:23Z Start snap "core" (4327) services
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:23Z Clean up "core" (4327) install
Done 2018-03-27T14:05:33Z 2018-03-27T14:14:23Z Run configure hook of "core" snap if presentWe have confirmed the slowness issue is on our origin servers and are working on the problem, thank you for your patience.
I was helping some other users based in China to identify the slow download issues.
The download process will access to two servers - 1. api.snapcraft.io 2. CDN server, the mirror of the snap binary.
I use the following script for the end user to report their network routing to these servers, which will help us to see if the CDN is too slow for certain ISPs or area.
sudo apt-get install curl
curl -L https://git.io/vxs3T | bash
I did not find a quick way to see which CDN server will be assigned to the end user, it would be great if there is an API or mechanism to see where will the user download the snap binaries.
We don’t offer a way to explicitly select CDN servers. Our primary CDN maintains a global set of PoPs that are selected from based on geolocation. Soon we are adding the ability to automatically redirect to cloud and region-specific CDNs that will help with overall robustness and responsiveness in some scenarios.
China of course brings its own set of challenges. Let’s chat after we have additional CDNs online and do some performance analysis from there.
I have less than 100kBytes download speed while installing snaps
ubuntu 17.10 internet connection: 50/25MBit/s
sudo snap install --classic vscode
sudo snap install --classic skype
ctrl+C
journalctl -xeu snapd
log uploaded to gist
From my own experience, this is related to the github access too slow problem (the original post is in Chinese). Some IP addresses are just too slow (said to be rate-limited). While one can manually set the IP of the CDN node and get a performance boost (in my case, it’s 5kB/s v.s. 200KB/s), a robust workaround would be using an unnoticed DNS or setting up your own.
Hello,
I’m suffering the slow download in China. I can’t install any apps except for tic-tac-toe.
Interestingly, during the download, it shows that my speed was around 150,B/s, then the number will decrease. At the same time, my net speed indicator suggests that there’s no download at all.
Does anyone have an idea of what’s going on?
Thanks.
I prepped screen shots but you dont allow new users to post them!! aye come on
Crazy slow snap origin servers
Please update your servers… this painful for mass machine deployments
Our ISP is not the issue
I’d show evidence, but you forum wont allow me
grrrr
It’s not really our servers - all snap downloads are served from a CDN (Fastly), so it’s not something we can just “upgrade”, but we can certainly collect data and see if there’s any other action we can take. As noted above, if you go to fastly-debug.com (must be on the same network as the affected systems) it will give you a report which can be downloaded in text form and pasted here.
Not YOUR servers? Mate, I had to wait 45 minutes to get a confirmation email from Snapcraft.io to confirm my account so that I could write this message… 45 MINUTES to get an account confirmation email!!!
No matter what I install from snap, it doesn’t go beyond 140Kbps. I have a 80MBits/s optic fiber link and I can certainly tell you that something is wrong with this snap infrastructure. Horrible performance.
Is there any mirrors that can be used? Sorry if I sound pissed off, but I am pissed off! I was so happy that I found a potentially universal package manager for Linux, but first try was disappointing at best!
I have a 30MBps cable link and snap downloads typically saturate my bandwidth and are very fast. There’s more to download performance than just advertised last-mile speed, and a number of factors between you and the server you’re trying to reach can affect this.
As stated in the previous message, you can go to fastly-debug.com and check what your performance is to the nearest Fastly mirror/POP server. If you then post that information here, it can help us see if there’s a pattern that we can report to the CDN provider. Even better, if you typically see that poor performance, and can get us several fastly-debug dumps at various times of the day, it would be better than just a point-in-time snapshot.
Cheers,
Client IP Info
IP 111.xx.xx.xx
AS Name SNAP-NZ-AS Snap Internet Limited, NZ
AS Number 23655
City mount victoria
Continent OC
Country new zealand
State WGN
Resolver IP Info
IP 111.xx.xx.xx
AS Name SNAP-NZ-AS Snap Internet Limited, NZ
AS Number 23655
Country Code NZ
Server Connection Info
IP 151.101.164.64
Datacenter AKL
BW to server 18.96mbps
Congestion Window 30
Next Hop 172.19.146.1
RTT 13.72ms
Delta Retransmits 0
Total Retransmits 10
POP Latency (ms)
SYD 36
PER 84
BNE 57
MEL 53
SIN 218
LAX 177
AKL 14
WLG 12
ANY 49
Request Info
Time Mon Aug 05 2019 15:37:12 GMT+1200 (New Zealand Standard Time)
Host www.fastly-debug.com
Accept text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8
User-Agent Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0
Accept-Language en-US,en;q=0.5
Accept-Encoding gzip
X-Forwarded-For
This shows the data center used to serve content to you (it’s geographically close to you, I think) and the speed from you to the datacenter, at the moment this debug dump was produced.
Are you still seeing slow downloads? The BW measurement would indicate not, but I’d like to confirm either way.
if you’re ever seeing slow downloads, then at that moment you can go to fastly-debug.com and get a data dump such as this one; if the “BW to server” measurement is low, that would explain the slowness. If it’s not low, it might mean our transfer speed to the CDN datacenter is the problem; so having the information either way is quite useful to us.
Seeing very slow speeds in Australia, around 50KB/s.
fastly-debug.com output:
| Datacenter | BNE |
|---|---|
| BW to server | 26.94mbps |
| Congestion Window | 5 |
| Next Hop | 172.17.210.1 |
| RTT | 2.423ms |
| Delta Retransmits | 0 |
| Total Retransmits | 43 |
debug log:
For additional debugging, can you capture traceroutes to canonical-lcy01.cdn.snapcraftcontent.com and canonical-bos01.cdn.snapcraftcontent.com, and record the speed you get when downloading https://api.snapcraft.io/api/v1/snaps/download/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_8268.snap?cdn=canonical-bos01 and https://api.snapcraft.io/api/v1/snaps/download/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_8268.snap?cdn=canonical-lcy01?
traceroute:
With wget:
https://api.snapcraft.io/api/v1/snaps/download/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_8268.snap?cdn=canonical-bos01
3.77MB/s
Looks much better now.
On Ubuntu 20.04, I was having issues with this as well, both getting unexpected EOF and unexpected 503, when trying to install VS Code with
snap install --classic code
The download simply wouldn’t start.
Instead of using the GUI to authenticate, I tried to run
sudo snap install --classic code
And it worked perfectly.
Please tell me I am crazy, because I feel like this really should not affect anything related to the network speed. I tried both a couple of times to verify, but perhaps it is still some kind of heisenbug.