If it’s the same issue perhaps try applying the suggested solution.
Can you try disabling powersave? It fixed my problem even with ethernet.
It is not my first time with this problem. I just restarted downloading and speed returned back to normal. And i saw this problem on my vds with certbot snap. Servers of course don’t use wifi and powersaving can’t harm it?
And again, same symptoms
Same issue, very unstable download speed. The same snap can take anywhere from 3 to 15 minutes. Network and everything else is exactly the same. The only difference is maybe time of day?
Any official recommendation on how to point to a faster CDN?
Same slow download with Ubuntu 20.04 LTS on jetson-tx1 dev board, after enable logging, I’ve got following debug log.
3月 21 15:57:21 py-jetson snapd[56365]: autorefresh.go:281: DEBUG: Next refresh scheduled for 2021-03-21T23:38:55+08:00.
3月 21 15:57:37 py-jetson snapd[56365]: daemon.go:215: polkit error: Authorization requires interaction
3月 21 15:57:37 py-jetson snapd[56365]: daemon.go:317: DEBUG: pid=56500;uid=1000;socket=/run/snapd.socket; POST /v2/snaps/cmake 11.709726ms 401
3月 21 15:57:43 py-jetson snapd[56365]: api.go:383: Installing snap "cmake" revision unset
3月 21 15:57:43 py-jetson snapd[56365]: store_action.go:441: DEBUG: Deltas enabled. Adding header Snap-Accept-Delta-Format: xdelta3
3月 21 15:57:43 py-jetson snapd[56365]: retry.go:49: DEBUG: Retrying https://api.snapcraft.io/v2/snaps/refresh, attempt 1, elapsed time=11.51µs
3月 21 15:57:43 py-jetson snapd[56365]: logger.go:67: DEBUG: > "POST /v2/snaps/refresh HTTP/1.1\r\nHost: api.snapcraft.io\r\nUser-Agent: snapd/2.49 (series 16; classic; devmode) ubuntu/20.04 (arm64) linux/4.9.201-tegra\r\nContent>
3月 21 15:57:44 py-jetson snapd[56365]: logger.go:74: DEBUG: < "HTTP/1.1 200 OK\r\nContent-Length: 3859\r\nContent-Type: application/json\r\nDate: Sun, 21 Mar 2021 07:57:44 GMT\r\nServer: gunicorn/19.7.1\r\nSnap-Store-Version: 34>
3月 21 15:57:44 py-jetson snapd[56365]: retry.go:61: DEBUG: The retry loop for https://api.snapcraft.io/v2/snaps/refresh finished after 1 retries, elapsed time=1.296890804s, status: 200
3月 21 15:57:44 py-jetson snapd[56365]: daemon.go:317: DEBUG: pid=56513;uid=0;socket=/run/snapd.socket; POST /v2/snaps/cmake 1.304347339s 400
3月 21 15:57:51 py-jetson snapd[56365]: api.go:383: Installing snap "cmake" revision unset
3月 21 15:57:51 py-jetson snapd[56365]: store_action.go:441: DEBUG: Deltas enabled. Adding header Snap-Accept-Delta-Format: xdelta3
3月 21 15:57:51 py-jetson snapd[56365]: retry.go:49: DEBUG: Retrying https://api.snapcraft.io/v2/snaps/refresh, attempt 1, elapsed time=12.813µs
3月 21 15:57:51 py-jetson snapd[56365]: logger.go:67: DEBUG: > "POST /v2/snaps/refresh HTTP/1.1\r\nHost: api.snapcraft.io\r\nUser-Agent: snapd/2.49 (series 16; classic; devmode) ubuntu/20.04 (arm64) linux/4.9.201-tegra\r\nContent>
3月 21 15:57:52 py-jetson snapd[56365]: logger.go:74: DEBUG: < "HTTP/1.1 200 OK\r\nContent-Length: 3859\r\nContent-Type: application/json\r\nDate: Sun, 21 Mar 2021 07:57:52 GMT\r\nServer: gunicorn/19.7.1\r\nSnap-Store-Version: 34>
3月 21 15:57:52 py-jetson snapd[56365]: retry.go:61: DEBUG: The retry loop for https://api.snapcraft.io/v2/snaps/refresh finished after 1 retries, elapsed time=1.164744659s, status: 200
3月 21 15:57:52 py-jetson snapd[56365]: daemon.go:317: DEBUG: pid=56584;uid=0;socket=/run/snapd.socket; POST /v2/snaps/cmake 1.205098385s 202
3月 21 15:57:52 py-jetson snapd[56365]: taskrunner.go:439: DEBUG: Running task 247 on Do: Ensure prerequisites for "cmake" are available
3月 21 15:57:52 py-jetson snapd[56365]: taskrunner.go:439: DEBUG: Running task 248 on Do: Download snap "cmake" (835) from channel "stable"
3月 21 15:57:52 py-jetson snapd[56365]: store_download.go:161: DEBUG: Available deltas returned by store: []
3月 21 15:57:52 py-jetson snapd[56365]: store_download.go:197: DEBUG: Starting download of "/var/lib/snapd/snaps/cmake_835.snap.partial".
3月 21 15:57:52 py-jetson snapd[56365]: retry.go:49: DEBUG: Retrying https://api.snapcraft.io/api/v1/snaps/download/lxhU87e4ZIqx4FU6vYgqY0oRWJ4OnxVD_835.snap, attempt 1, elapsed time=24.323µs
3月 21 15:57:52 py-jetson snapd[56365]: logger.go:67: DEBUG: > "GET /api/v1/snaps/download/lxhU87e4ZIqx4FU6vYgqY0oRWJ4OnxVD_835.snap HTTP/1.1\r\nHost: api.snapcraft.io\r\nUser-Agent: snapd/2.49 (series 16; classic; devmode) ubunt>
3月 21 15:57:54 py-jetson snapd[56365]: logger.go:74: DEBUG: < "HTTP/1.1 302 FOUND\r\nContent-Length: 587\r\nContent-Type: text/html; charset=utf-8\r\nDate: Sun, 21 Mar 2021 07:57:54 GMT\r\nLocation: https://canonical-bos01.cdn.s>
3月 21 15:57:54 py-jetson snapd[56365]: logger.go:67: DEBUG: > "GET /download-origin/canonical-lgw01/lxhU87e4ZIqx4FU6vYgqY0oRWJ4OnxVD_835.snap?interactive=1&token=1616324400_6c7f4219e83868be5f7901ea5a0e5bdad793b8f3 HTTP/1.1\r\nHo>
3月 21 15:57:55 py-jetson snapd[56365]: logger.go:74: DEBUG: < "HTTP/1.1 200 OK\r\nContent-Length: 102739968\r\nAccept-Ranges: bytes\r\nCache-Control: max-age=86400\r\nContent-Disposition: attachment; filename=\"lxhU87e4ZIqx4FU6v
Average download speed using snap
sudo snap install cmake --classic
40kbps download speed.
Normal using wget to download the direct link wget https://api.snapcraft.io/api/v1/snaps/download/lxhU87e4ZIqx4FU6vYgqY0oRWJ4OnxVD_835.snap
lxhU87e4ZIqx4FU6vYgqY0oRWJ4OnxVD_835.snap 100%[===================================================================================================================================>] 97.98M 2.76MB/s in 49s
For me the same issues. This is not a 1 time thing.
When 1 boodstrap 3 servers 1 is always fast. other 2 are slow(maybe just accidentally).
This is defently not a internet traffic thing. Cluster have 10GB WAN connections. Also snap is slow, curl is fast on the same machine.
It feels like throttling… so please canonical fix it. it must be on the client or backend site of snap. And if you are throttling let us setup mirrors. We love to host them.
Update: I found https://docs.ubuntu.com/snap-store-proxy/ so i hope this will solve my issues.
Ok snap-store-proxy is a solution, but only limeted to 5 devices
Also found the issue
canonical-bos01.cdn.snapcraftcontent.com is really slow for me… i don’t know way…
Is there a way to force only to use the other 2 cdns?
canonical-lgw01.cdn.snapcraftcontent.com canonical-lcy01.cdn.snapcraftcontent.com
I’m also getting ridiculously slow speeds over both WiFi and Ethernet (I have 100mbps fibre to my ISP). I’m seeing around 150kb/sec, sometimes dropping as low as 70kb/sec.
traceroutes:
$ traceroute canonical-lcy01.cdn.snapcraftcontent.com
traceroute: Warning: canonical-lcy01.cdn.snapcraftcontent.com has multiple addresses; using 91.189.88.179
traceroute to canonical-lcy01.cdn.snapcraftcontent.com (91.189.88.179), 64 hops max, 52 byte packets
1 fritz.box (192.168.178.1) 2.212 ms 1.060 ms 0.956 ms
2 * * *
3 * * 122.56.113.5 (122.56.113.5) 15.702 ms
4 122.56.113.4 (122.56.113.4) 15.643 ms 15.979 ms 15.970 ms
5 ae7-2.akbr7.global-gateway.net.nz (122.56.119.53) 16.498 ms 13.488 ms 15.912 ms
6 et1-0-0.sgbr3.global-gateway.net.nz (122.56.119.30) 39.610 ms 40.892 ms
et8-0-0.sgbr3.global-gateway.net.nz (122.56.119.26) 39.667 ms
7 ae2-10.sgbr4.global-gateway.net.nz (202.50.232.246) 36.587 ms 36.558 ms 40.596 ms
8 6939.syd.equinix.com (45.127.173.24) 39.801 ms 37.976 ms 39.728 ms
9 100ge0-36.core1.syd3.he.net (184.104.194.146) 40.526 ms 42.399 ms 38.538 ms
10 100ge0-35.core1.per1.he.net (184.104.194.142) 85.815 ms 86.491 ms 88.384 ms
11 100ge12-1.core1.sin1.he.net (184.104.194.137) 128.871 ms 133.576 ms 129.730 ms
12 100ge11-1.core1.mrs1.he.net (184.105.65.14) 278.939 ms 277.209 ms 278.056 ms
13 100ge5-2.core1.par2.he.net (184.105.81.29) 271.989 ms 276.461 ms 275.429 ms
14 100ge11-1.core1.lon2.he.net (184.105.223.253) 273.361 ms 272.965 ms 387.305 ms
15 100ge0-36.core1.lon7.he.net (184.104.196.6) 272.995 ms 277.918 ms 337.809 ms
16 canonical-group-limited.e0-53.switch1.lon7.he.net (184.104.203.50) 306.490 ms 278.417 ms 276.468 ms
17 bahamutt.canonical.com (91.189.88.179) 278.592 ms 280.050 ms 279.827 ms
and
$ traceroute canonical-bos01.cdn.snapcraftcontent.com
traceroute: Warning: canonical-bos01.cdn.snapcraftcontent.com has multiple addresses; using 91.189.91.43
traceroute to canonical-bos01.cdn.snapcraftcontent.com (91.189.91.43), 64 hops max, 52 byte packets
1 fritz.box (192.168.178.1) 1.658 ms 1.040 ms 3.093 ms
2 * * *
3 * 122.56.113.5 (122.56.113.5) 13.526 ms 14.889 ms
4 122.56.113.4 (122.56.113.4) 17.489 ms 14.439 ms 16.002 ms
5 ae2-6.tkbr12.global-gateway.net.nz (122.56.127.17) 16.625 ms 13.609 ms
ae7-2.akbr7.global-gateway.net.nz (122.56.119.53) 16.184 ms
6 ae10-10.tkbr12.global-gateway.net.nz (202.50.232.29) 15.503 ms
xe8-0-9.lebr7.global-gateway.net.nz (122.56.127.22) 138.206 ms
ae10-10.tkbr12.global-gateway.net.nz (202.50.232.29) 13.596 ms
7 xe-0-0-6-10.sjbr3.global-gateway.net.nz (203.96.120.194) 139.005 ms
ae0-10.lebr8.global-gateway.net.nz (202.50.232.42) 143.736 ms
203.96.120.142 (203.96.120.142) 139.128 ms
8 ae3-10.lebr7.global-gateway.net.nz (122.56.127.26) 141.992 ms
ae0-10.lebr8.global-gateway.net.nz (202.50.232.42) 139.603 ms 139.671 ms
9 lag-14.ear2.losangeles1.level3.net (4.68.37.89) 139.297 ms 138.458 ms *
10 lag-14.ear2.losangeles1.level3.net (4.68.37.89) 171.908 ms
ae-2-3.bear1.boston1.level3.net (4.69.159.249) 281.012 ms *
11 twdx-level3-100g.boston1.level3.net (4.53.60.66) 209.058 ms
bbr02-et-0-0-7.bos01.twdx.net (198.160.62.0) 213.606 ms
twdx-level3-100g.boston1.level3.net (4.53.60.66) 210.319 ms
12 bbr02-et-0-0-29.bos01.twdx.net (198.160.62.2) 215.282 ms
bbr02-et-0-0-7.bos01.twdx.net (198.160.62.0) 241.898 ms
bbr02-et-0-0-29.bos01.twdx.net (198.160.62.2) 213.864 ms
13 dcr03-hu-0-8-0-0.bsn04.twdx.net (198.160.62.201) 295.189 ms 209.045 ms
bbr02-et-0-0-29.bos01.twdx.net (198.160.62.2) 215.818 ms
14 dcr03-hu-0-8-0-0.bsn04.twdx.net (198.160.62.201) 291.438 ms
swp25.viviani.canonical.com (185.134.181.46) 306.392 ms
swp25.lebedev.canonical.com (185.134.181.50) 306.805 ms
15 darkbowser.canonical.com (91.189.91.43) 216.258 ms 211.979 ms 212.420 ms
wget speeds for the two requested files are:
$ wget https://api.snapcraft.io/api/v1/snaps/download/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_8268.snap?cdn=canonical-bos01
...
Resolving api.snapcraft.io (api.snapcraft.io)... 91.189.92.38, 91.189.92.20, 91.189.92.19, ...
...
48.2KB/s
and
wget https://api.snapcraft.io/api/v1/snaps/download/99T7MUlRhtI3U0QFgl5mXXESAiSwt776_8268.snap?cdn=canonical-lcy01
...
Resolving api.snapcraft.io (api.snapcraft.io)... 91.189.92.38, 91.189.92.20, 91.189.92.19, ...
...
7.06MB/s
That second one was much faster.
60+ minutes to install a single application like CLion is just crazy. I have three more apps like that to install - that’s 4+ hours of my time sitting around waiting.
EDIT: CLion snap has been downloading for 30+ minutes now and has an ETA of another 92 minutes…
EDIT2: Never mind, I ditched snap and used Jetbrains Toolbox to install my tools - took a total of about 4 minutes to download and install all 4 IDEs. If snap downloads remain several orders of magnitude slower than direct download how does anyone expect it to become a standard package format?
There is defiantly a problem connecting to bos01(from europe for me).
Is there a way to force to use only the 2 other cdns?
I’m also seeing consistently slow downloads and it’s been like that for weeks, probably months now. That’s across machines, across router restarts, I think different Ubuntu versions too.
I’m now getting a steady 175kB/s - at this pace downloading Opera takes 15 minutes.
It’s gotten so bad that even though I have premade and working Ansible scripts to bootstrap and update my machines, I started rewriting them to remove all uses of snap - installing two applications with snap takes more time than the rest of the machine setup combined (!).
My ISP is UPC in Warsaw Poland and speedtest.net is reporting 400 Mbps download for me.
For anyone having experienced slow downloads in the past, our team completed a set of improvements on April 8th 2021 (particularly with the Boston endpoint) and if you are still experiencing more than very occasional poor download performance we’d like to hear the details.
Same thing here! Download speed was 408 kb/s when downloading Dbeaver. The most strange thing is the whole internet connection slowing down during the download. I’m from Brazil. Every other service is at normal speed, only snap is extremely slow.
Client IP Info
IP 177.x.xxx.xx
AS Name Brasil Telecom S/A - Filial Distrito Federal, BR
AS Number 8167
City canoas
Continent SA
Country brazil
State RS
Resolver IP Info
IP 172.xxx.xxx.x
AS Name GOOGLE, US
AS Number 15169
Country Code US
Server Connection Info
IP 151.101.192.64
Datacenter GRU
BW to server 7.14mbps
Congestion Window 85
Next Hop 172.24.244.1
RTT 123.312ms
Delta Retransmits 0
Total Retransmits 0
POP Latency (ms)
GRU 43
CWB 257
CGH 41
GIG 46
SCL 197
LIM 184
BOG 193
EZE 227
MIA 185
DFW 205
LAX 222
FRA 249
ANY 38
Request Info
Time Sun May 23 2021 12:58:29 GMT-0300 (Brasilia Standard Time)
Host www.fastly-debug.com
Accept text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9
User-Agent Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36
Accept-Language pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7,la;q=0.6
Accept-Encoding gzip
X-Forwarded-For
Useful discussion, thanks, to all!
Hello,
I’m still experiencing a unstable download (update for core20 fail with EOF while using bos01 and a few connection error during the day) and I’m in mainland China.
If I understand correctly, fastly is used as a CDN service, but it seems that fastly doesn’t has any pop in mainland China according to their network map.
So is there any solution for that?
Still getting 100kb/s download speeds sporadically.
There was a fault affecting download speeds from one of our CDN nodes between 4 and 6 hours ago. But everything’s snappy again now.
I’m trying to install apps from Ubuntu Software but it’s not happening. Using the snap watch command I see that the download is extremely slow 10kb/s and then after 5 minutes it fails with the message “connection reset by peer”.
Apparently it was on my end. While my internet was working fine for everything except the snap installs and I tried using different fixes, disabling power saving for my network adapter reinstalling drivers even reinstalling Ubuntu, when I used my phone as a USB tether using 4g the snap installs were working fine. Then I change my phone from 4g to WiFi while still using the USB tether and again the problems with the snap installs were happening. I finally reset my router and behold everything works fine now. Can someone explain to me what is so special about the snap install procedure that my router decided to block the download?
Still getting consistently slow download speeds here too. While using ubuntu-image snap
to build an Ubuntu Core image, Fetching snapd
is anywhere under 200kB/s and Fetching intel-iotg-kernel
is consistently under 50kB/s.
The latter would fail to download multiple times with various error messages:
Error: Error preparing image: unexpected EOF
Error: Error preparing image: Get "https://canonical-lgw01.cdn.snapcraftcontent.com/download-origin/canonical-lgw01/v7evc8smSqgPwL0KCjpGJf3KuucG2Yap_7.snap?token=redactedToken": net/http: timeout awaiting response headers
Running on Ubuntu Server 22.04 with ethernet connectivity to a 50~90Mbps internet download speed, nothing funky on the server and/or network configuration where usually everything just works fine and (way) faster over the Internet.
Copy/paste of http://www.fastly-debug.com below:
Client IP Info
IP [redacted]
AS Name France Telecom - Orange, FR
AS Number 3215
City theuville-aux-maillots
Continent EU
Country france
State NOR
Resolver IP Info
IP 80.10.201.97
AS Name France Telecom - Orange, FR
AS Number 3215
Country Code FR
Server Connection Info
IP 151.101.120.64
Datacenter CDG
BW to server 30.54mbps
Congestion Window 480
Next Hop 172.22.144.1
RTT 0.926ms
Delta Retransmits 0
Total Retransmits 0
POP Latency (ms)
AMS 43
BMA 79
BRU 44
CDG 31
CPH 57
DUB 68
EDDF 63
FCO 101
HEL 140
HHN 72
LCY 65
LHR 72
LIN 79
LIS 109
LON 64
MAD 96
MAN 80
MRS 76
MUC 88
MXP 108
OSL 113
PMO 130
SOF 123
VIE 103
ANY 69
Request Info
Time Wed Nov 09 2022 09:48:07 GMT+0100 (heure normale d’Europe centrale)
Host www.fastly-debug.com
Accept text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8
User-Agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36
Accept-Language fr-FR,fr;q=0.9,en-US;q=0.8,en;q=0.7
Accept-Encoding gzip
X-Forwarded-For
Is there anything else I can provide to help diagnose/debug this situation?
Update: cannot go through Fetching intel-iotg-kernel
so I’ve just tried an AWS EC2 instance in the eu-west-3c
availability zone, and I’m getting similar, very low snap download speeds from there too…