Extremely slow snap downloads

If it’s the same issue perhaps try applying the suggested solution. :wink:

Can you try disabling powersave? It fixed my problem even with ethernet.

image It is not my first time with this problem. I just restarted downloading and speed returned back to normal. And i saw this problem on my vds with certbot snap. Servers of course don’t use wifi and powersaving can’t harm it?

image And again, same symptoms

Same issue, very unstable download speed. The same snap can take anywhere from 3 to 15 minutes. Network and everything else is exactly the same. The only difference is maybe time of day?

Any official recommendation on how to point to a faster CDN?

1 Like

Same slow download with Ubuntu 20.04 LTS on jetson-tx1 dev board, after enable logging, I’ve got following debug log.

3月 21 15:57:21 py-jetson snapd[56365]: autorefresh.go:281: DEBUG: Next refresh scheduled for 2021-03-21T23:38:55+08:00.
3月 21 15:57:37 py-jetson snapd[56365]: daemon.go:215: polkit error: Authorization requires interaction
3月 21 15:57:37 py-jetson snapd[56365]: daemon.go:317: DEBUG: pid=56500;uid=1000;socket=/run/snapd.socket; POST /v2/snaps/cmake 11.709726ms 401
3月 21 15:57:43 py-jetson snapd[56365]: api.go:383: Installing snap "cmake" revision unset
3月 21 15:57:43 py-jetson snapd[56365]: store_action.go:441: DEBUG: Deltas enabled. Adding header Snap-Accept-Delta-Format: xdelta3
3月 21 15:57:43 py-jetson snapd[56365]: retry.go:49: DEBUG: Retrying https://api.snapcraft.io/v2/snaps/refresh, attempt 1, elapsed time=11.51µs
3月 21 15:57:43 py-jetson snapd[56365]: logger.go:67: DEBUG: > "POST /v2/snaps/refresh HTTP/1.1\r\nHost: api.snapcraft.io\r\nUser-Agent: snapd/2.49 (series 16; classic; devmode) ubuntu/20.04 (arm64) linux/4.9.201-tegra\r\nContent>
3月 21 15:57:44 py-jetson snapd[56365]: logger.go:74: DEBUG: < "HTTP/1.1 200 OK\r\nContent-Length: 3859\r\nContent-Type: application/json\r\nDate: Sun, 21 Mar 2021 07:57:44 GMT\r\nServer: gunicorn/19.7.1\r\nSnap-Store-Version: 34>
3月 21 15:57:44 py-jetson snapd[56365]: retry.go:61: DEBUG: The retry loop for https://api.snapcraft.io/v2/snaps/refresh finished after 1 retries, elapsed time=1.296890804s, status: 200
3月 21 15:57:44 py-jetson snapd[56365]: daemon.go:317: DEBUG: pid=56513;uid=0;socket=/run/snapd.socket; POST /v2/snaps/cmake 1.304347339s 400
3月 21 15:57:51 py-jetson snapd[56365]: api.go:383: Installing snap "cmake" revision unset
3月 21 15:57:51 py-jetson snapd[56365]: store_action.go:441: DEBUG: Deltas enabled. Adding header Snap-Accept-Delta-Format: xdelta3
3月 21 15:57:51 py-jetson snapd[56365]: retry.go:49: DEBUG: Retrying https://api.snapcraft.io/v2/snaps/refresh, attempt 1, elapsed time=12.813µs
3月 21 15:57:51 py-jetson snapd[56365]: logger.go:67: DEBUG: > "POST /v2/snaps/refresh HTTP/1.1\r\nHost: api.snapcraft.io\r\nUser-Agent: snapd/2.49 (series 16; classic; devmode) ubuntu/20.04 (arm64) linux/4.9.201-tegra\r\nContent>
3月 21 15:57:52 py-jetson snapd[56365]: logger.go:74: DEBUG: < "HTTP/1.1 200 OK\r\nContent-Length: 3859\r\nContent-Type: application/json\r\nDate: Sun, 21 Mar 2021 07:57:52 GMT\r\nServer: gunicorn/19.7.1\r\nSnap-Store-Version: 34>
3月 21 15:57:52 py-jetson snapd[56365]: retry.go:61: DEBUG: The retry loop for https://api.snapcraft.io/v2/snaps/refresh finished after 1 retries, elapsed time=1.164744659s, status: 200
3月 21 15:57:52 py-jetson snapd[56365]: daemon.go:317: DEBUG: pid=56584;uid=0;socket=/run/snapd.socket; POST /v2/snaps/cmake 1.205098385s 202
3月 21 15:57:52 py-jetson snapd[56365]: taskrunner.go:439: DEBUG: Running task 247 on Do: Ensure prerequisites for "cmake" are available
3月 21 15:57:52 py-jetson snapd[56365]: taskrunner.go:439: DEBUG: Running task 248 on Do: Download snap "cmake" (835) from channel "stable"
3月 21 15:57:52 py-jetson snapd[56365]: store_download.go:161: DEBUG: Available deltas returned by store: []
3月 21 15:57:52 py-jetson snapd[56365]: store_download.go:197: DEBUG: Starting download of "/var/lib/snapd/snaps/cmake_835.snap.partial".
3月 21 15:57:52 py-jetson snapd[56365]: retry.go:49: DEBUG: Retrying https://api.snapcraft.io/api/v1/snaps/download/lxhU87e4ZIqx4FU6vYgqY0oRWJ4OnxVD_835.snap, attempt 1, elapsed time=24.323µs
3月 21 15:57:52 py-jetson snapd[56365]: logger.go:67: DEBUG: > "GET /api/v1/snaps/download/lxhU87e4ZIqx4FU6vYgqY0oRWJ4OnxVD_835.snap HTTP/1.1\r\nHost: api.snapcraft.io\r\nUser-Agent: snapd/2.49 (series 16; classic; devmode) ubunt>
3月 21 15:57:54 py-jetson snapd[56365]: logger.go:74: DEBUG: < "HTTP/1.1 302 FOUND\r\nContent-Length: 587\r\nContent-Type: text/html; charset=utf-8\r\nDate: Sun, 21 Mar 2021 07:57:54 GMT\r\nLocation: https://canonical-bos01.cdn.s>
3月 21 15:57:54 py-jetson snapd[56365]: logger.go:67: DEBUG: > "GET /download-origin/canonical-lgw01/lxhU87e4ZIqx4FU6vYgqY0oRWJ4OnxVD_835.snap?interactive=1&token=1616324400_6c7f4219e83868be5f7901ea5a0e5bdad793b8f3 HTTP/1.1\r\nHo>
3月 21 15:57:55 py-jetson snapd[56365]: logger.go:74: DEBUG: < "HTTP/1.1 200 OK\r\nContent-Length: 102739968\r\nAccept-Ranges: bytes\r\nCache-Control: max-age=86400\r\nContent-Disposition: attachment; filename=\"lxhU87e4ZIqx4FU6v

Average download speed using snap

sudo snap install cmake --classic

40kbps download speed.

Normal using wget to download the direct link wget https://api.snapcraft.io/api/v1/snaps/download/lxhU87e4ZIqx4FU6vYgqY0oRWJ4OnxVD_835.snap

lxhU87e4ZIqx4FU6vYgqY0oRWJ4OnxVD_835.snap 100%[===================================================================================================================================>] 97.98M 2.76MB/s in 49s

For me the same issues. This is not a 1 time thing.
When 1 boodstrap 3 servers 1 is always fast. other 2 are slow(maybe just accidentally).
This is defently not a internet traffic thing. Cluster have 10GB WAN connections. Also snap is slow, curl is fast on the same machine.

It feels like throttling… so please canonical fix it. it must be on the client or backend site of snap. And if you are throttling let us setup mirrors. We love to host them.

Update: I found https://docs.ubuntu.com/snap-store-proxy/ so i hope this will solve my issues.

Ok snap-store-proxy is a solution, but only limeted to 5 devices :frowning: Also found the issue

canonical-bos01.cdn.snapcraftcontent.com is really slow for me… i don’t know way…

Is there a way to force only to use the other 2 cdns?

canonical-lgw01.cdn.snapcraftcontent.com canonical-lcy01.cdn.snapcraftcontent.com

I’m also getting ridiculously slow speeds over both WiFi and Ethernet (I have 100mbps fibre to my ISP). I’m seeing around 150kb/sec, sometimes dropping as low as 70kb/sec.

traceroutes:

$ traceroute canonical-lcy01.cdn.snapcraftcontent.com
traceroute: Warning: canonical-lcy01.cdn.snapcraftcontent.com has multiple addresses; using 91.189.88.179
traceroute to canonical-lcy01.cdn.snapcraftcontent.com (91.189.88.179), 64 hops max, 52 byte packets
 1  fritz.box (192.168.178.1)  2.212 ms  1.060 ms  0.956 ms
 2  * * *
 3  * * 122.56.113.5 (122.56.113.5)  15.702 ms
 4  122.56.113.4 (122.56.113.4)  15.643 ms  15.979 ms  15.970 ms
 5  ae7-2.akbr7.global-gateway.net.nz (122.56.119.53)  16.498 ms  13.488 ms  15.912 ms
 6  et1-0-0.sgbr3.global-gateway.net.nz (122.56.119.30)  39.610 ms  40.892 ms
    et8-0-0.sgbr3.global-gateway.net.nz (122.56.119.26)  39.667 ms
 7  ae2-10.sgbr4.global-gateway.net.nz (202.50.232.246)  36.587 ms  36.558 ms  40.596 ms
 8  6939.syd.equinix.com (45.127.173.24)  39.801 ms  37.976 ms  39.728 ms
 9  100ge0-36.core1.syd3.he.net (184.104.194.146)  40.526 ms  42.399 ms  38.538 ms
10  100ge0-35.core1.per1.he.net (184.104.194.142)  85.815 ms  86.491 ms  88.384 ms
11  100ge12-1.core1.sin1.he.net (184.104.194.137)  128.871 ms  133.576 ms  129.730 ms
12  100ge11-1.core1.mrs1.he.net (184.105.65.14)  278.939 ms  277.209 ms  278.056 ms
13  100ge5-2.core1.par2.he.net (184.105.81.29)  271.989 ms  276.461 ms  275.429 ms
14  100ge11-1.core1.lon2.he.net (184.105.223.253)  273.361 ms  272.965 ms  387.305 ms
15  100ge0-36.core1.lon7.he.net (184.104.196.6)  272.995 ms  277.918 ms  337.809 ms
16  canonical-group-limited.e0-53.switch1.lon7.he.net (184.104.203.50)  306.490 ms  278.417 ms  276.468 ms
17  bahamutt.canonical.com (91.189.88.179)  278.592 ms  280.050 ms  279.827 ms

and

$ traceroute canonical-bos01.cdn.snapcraftcontent.com
traceroute: Warning: canonical-bos01.cdn.snapcraftcontent.com has multiple addresses; using 91.189.91.43
traceroute to canonical-bos01.cdn.snapcraftcontent.com (91.189.91.43), 64 hops max, 52 byte packets
 1  fritz.box (192.168.178.1)  1.658 ms  1.040 ms  3.093 ms
 2  * * *
 3  * 122.56.113.5 (122.56.113.5)  13.526 ms  14.889 ms
 4  122.56.113.4 (122.56.113.4)  17.489 ms  14.439 ms  16.002 ms
 5  ae2-6.tkbr12.global-gateway.net.nz (122.56.127.17)  16.625 ms  13.609 ms
    ae7-2.akbr7.global-gateway.net.nz (122.56.119.53)  16.184 ms
 6  ae10-10.tkbr12.global-gateway.net.nz (202.50.232.29)  15.503 ms
    xe8-0-9.lebr7.global-gateway.net.nz (122.56.127.22)  138.206 ms
    ae10-10.tkbr12.global-gateway.net.nz (202.50.232.29)  13.596 ms
 7  xe-0-0-6-10.sjbr3.global-gateway.net.nz (203.96.120.194)  139.005 ms
    ae0-10.lebr8.global-gateway.net.nz (202.50.232.42)  143.736 ms
    203.96.120.142 (203.96.120.142)  139.128 ms
 8  ae3-10.lebr7.global-gateway.net.nz (122.56.127.26)  141.992 ms
    ae0-10.lebr8.global-gateway.net.nz (202.50.232.42)  139.603 ms  139.671 ms
 9  lag-14.ear2.losangeles1.level3.net (4.68.37.89)  139.297 ms  138.458 ms *
10  lag-14.ear2.losangeles1.level3.net (4.68.37.89)  171.908 ms
    ae-2-3.bear1.boston1.level3.net (4.69.159.249)  281.012 ms *
11  twdx-level3-100g.boston1.level3.net (4.53.60.66)  209.058 ms
    bbr02-et-0-0-7.bos01.twdx.net (198.160.62.0)  213.606 ms
    twdx-level3-100g.boston1.level3.net (4.53.60.66)  210.319 ms
12  bbr02-et-0-0-29.bos01.twdx.net (198.160.62.2)  215.282 ms
    bbr02-et-0-0-7.bos01.twdx.net (198.160.62.0)  241.898 ms
    bbr02-et-0-0-29.bos01.twdx.net (198.160.62.2)  213.864 ms
13  dcr03-hu-0-8-0-0.bsn04.twdx.net (198.160.62.201)  295.189 ms  209.045 ms
    bbr02-et-0-0-29.bos01.twdx.net (198.160.62.2)  215.818 ms
14  dcr03-hu-0-8-0-0.bsn04.twdx.net (198.160.62.201)  291.438 ms
    swp25.viviani.canonical.com (185.134.181.46)  306.392 ms
    swp25.lebedev.canonical.com (185.134.181.50)  306.805 ms
15  darkbowser.canonical.com (91.189.91.43)  216.258 ms  211.979 ms  212.420 ms

wget speeds for the two requested files are:

$ wget https://api.snapcraft.io/api/v1/snaps/download/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_8268.snap?cdn=canonical-bos01
...
Resolving api.snapcraft.io (api.snapcraft.io)... 91.189.92.38, 91.189.92.20, 91.189.92.19, ...
...
48.2KB/s

and

wget https://api.snapcraft.io/api/v1/snaps/download/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_8268.snap?cdn=canonical-lcy01
...
Resolving api.snapcraft.io (api.snapcraft.io)... 91.189.92.38, 91.189.92.20, 91.189.92.19, ...
...
7.06MB/s

That second one was much faster.

60+ minutes to install a single application like CLion is just crazy. I have three more apps like that to install - that’s 4+ hours of my time sitting around waiting.

EDIT: CLion snap has been downloading for 30+ minutes now and has an ETA of another 92 minutes…

EDIT2: Never mind, I ditched snap and used Jetbrains Toolbox to install my tools - took a total of about 4 minutes to download and install all 4 IDEs. If snap downloads remain several orders of magnitude slower than direct download how does anyone expect it to become a standard package format?

There is defiantly a problem connecting to bos01(from europe for me).
Is there a way to force to use only the 2 other cdns?

1 Like

I’m also seeing consistently slow downloads and it’s been like that for weeks, probably months now. That’s across machines, across router restarts, I think different Ubuntu versions too.

I’m now getting a steady 175kB/s - at this pace downloading Opera takes 15 minutes.

It’s gotten so bad that even though I have premade and working Ansible scripts to bootstrap and update my machines, I started rewriting them to remove all uses of snap - installing two applications with snap takes more time than the rest of the machine setup combined (!).

My ISP is UPC in Warsaw Poland and speedtest.net is reporting 400 Mbps download for me.

For anyone having experienced slow downloads in the past, our team completed a set of improvements on April 8th 2021 (particularly with the Boston endpoint) and if you are still experiencing more than very occasional poor download performance we’d like to hear the details.

Same thing here! Download speed was 408 kb/s when downloading Dbeaver. The most strange thing is the whole internet connection slowing down during the download. I’m from Brazil. Every other service is at normal speed, only snap is extremely slow.

Client IP Info
IP 177.x.xxx.xx
AS Name Brasil Telecom S/A - Filial Distrito Federal, BR
AS Number 8167
City canoas
Continent SA
Country brazil
State RS
Resolver IP Info
IP 172.xxx.xxx.x
AS Name GOOGLE, US
AS Number 15169
Country Code US
Server Connection Info
IP 151.101.192.64
Datacenter GRU
BW to server 7.14mbps
Congestion Window 85
Next Hop 172.24.244.1
RTT 123.312ms
Delta Retransmits 0
Total Retransmits 0
POP Latency (ms)
GRU 43
CWB 257
CGH 41
GIG 46
SCL 197
LIM 184
BOG 193
EZE 227
MIA 185
DFW 205
LAX 222
FRA 249
ANY 38
Request Info
Time Sun May 23 2021 12:58:29 GMT-0300 (Brasilia Standard Time)
Host www.fastly-debug.com
Accept text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9
User-Agent Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36
Accept-Language pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7,la;q=0.6
Accept-Encoding gzip
X-Forwarded-For

Useful discussion, thanks, to all! :handshake:

Hello,

I’m still experiencing a unstable download (update for core20 fail with EOF while using bos01 and a few connection error during the day) and I’m in mainland China.

If I understand correctly, fastly is used as a CDN service, but it seems that fastly doesn’t has any pop in mainland China according to their network map.

So is there any solution for that?

Still getting 100kb/s download speeds sporadically.

There was a fault affecting download speeds from one of our CDN nodes between 4 and 6 hours ago. But everything’s snappy again now.

I’m trying to install apps from Ubuntu Software but it’s not happening. Using the snap watch command I see that the download is extremely slow 10kb/s and then after 5 minutes it fails with the message “connection reset by peer”.

Apparently it was on my end. While my internet was working fine for everything except the snap installs and I tried using different fixes, disabling power saving for my network adapter reinstalling drivers even reinstalling Ubuntu, when I used my phone as a USB tether using 4g the snap installs were working fine. Then I change my phone from 4g to WiFi while still using the USB tether and again the problems with the snap installs were happening. I finally reset my router and behold everything works fine now. Can someone explain to me what is so special about the snap install procedure that my router decided to block the download?

Still getting consistently slow download speeds here too. While using ubuntu-image snap to build an Ubuntu Core image, Fetching snapd is anywhere under 200kB/s and Fetching intel-iotg-kernel is consistently under 50kB/s.

The latter would fail to download multiple times with various error messages:

  • Error: Error preparing image: unexpected EOF
  • Error: Error preparing image: Get "https://canonical-lgw01.cdn.snapcraftcontent.com/download-origin/canonical-lgw01/v7evc8smSqgPwL0KCjpGJf3KuucG2Yap_7.snap?token=redactedToken": net/http: timeout awaiting response headers

Running on Ubuntu Server 22.04 with ethernet connectivity to a 50~90Mbps internet download speed, nothing funky on the server and/or network configuration where usually everything just works fine and (way) faster over the Internet.

Copy/paste of http://www.fastly-debug.com below:

Client IP Info
IP	[redacted]
AS Name	France Telecom - Orange, FR
AS Number	3215
City	theuville-aux-maillots
Continent	EU
Country	france
State	NOR
Resolver IP Info
IP	80.10.201.97
AS Name	France Telecom - Orange, FR
AS Number	3215
Country Code	FR
Server Connection Info
IP	151.101.120.64
Datacenter	CDG
BW to server	30.54mbps
Congestion Window	480
Next Hop	172.22.144.1
RTT	0.926ms
Delta Retransmits	0
Total Retransmits	0
POP Latency (ms)
AMS	43
BMA	79
BRU	44
CDG	31
CPH	57
DUB	68
EDDF	63
FCO	101
HEL	140
HHN	72
LCY	65
LHR	72
LIN	79
LIS	109
LON	64
MAD	96
MAN	80
MRS	76
MUC	88
MXP	108
OSL	113
PMO	130
SOF	123
VIE	103
ANY	69
Request Info
Time	Wed Nov 09 2022 09:48:07 GMT+0100 (heure normale d’Europe centrale)
Host	www.fastly-debug.com
Accept	text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8
User-Agent	Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36
Accept-Language	fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7
Accept-Encoding	gzip
X-Forwarded-For	

Is there anything else I can provide to help diagnose/debug this situation?


Update: cannot go through Fetching intel-iotg-kernel so I’ve just tried an AWS EC2 instance in the eu-west-3c availability zone, and I’m getting similar, very low snap download speeds from there too…