Snaps download should be resumable

joseph@joseph:~$ snap download blender
Fetching snap "blender"
error: read tcp 10.102.16.91:60812->151.101.54.217:443: read: connection reset by peer

Result was the same…

I’ll try this now… :slight_smile:

joseph@joseph:~$ SNAPD_DEBUG=1 SNAPD_DEBUG_HTTP=7 snap download blender
Fetching snap "blender"
2019/08/21 19:06:13.294698 helpers.go:224: DEBUG: Going to download snap "blender" (unset) from channel "" to "".
2019/08/21 19:06:13.295022 retry.go:41: DEBUG: Retrying https://api.snapcraft.io/v2/snaps/refresh, attempt 1, elapsed time=6.43µs
2019/08/21 19:06:13.295413 logger.go:67: DEBUG: > "POST /v2/snaps/refresh HTTP/1.1\r\nHost: api.snapcraft.io\r\nUser-Agent: snapd/2.37.4-1+b1 (series 16; classic; devmode) snap debian/10 (amd64) linux/4.19.0-5-amd64\r\nContent-Length: 347\r\nAccept: application/json\r\nContent-Type: application/json\r\nSnap-Classic: true\r\nSnap-Device-Architecture: amd64\r\nSnap-Device-Series: 16\r\nAccept-Encoding: gzip\r\n\r\n{\"context\":[],\"actions\":[{\"action\":\"download\",\"instance-key\":\"download-1\",\"name\":\"blender\",\"epoch\":null}],\"fields\":[\"architectures\",\"base\",\"confinement\",\"contact\",\"created-at\",\"description\",\"download\",\"epoch\",\"license\",\"name\",\"prices\",\"private\",\"publisher\",\"revision\",\"snap-id\",\"snap-yaml\",\"summary\",\"title\",\"type\",\"version\",\"media\",\"common-ids\"]}"
2019/08/21 19:06:14.857278 logger.go:74: DEBUG: < "HTTP/1.1 200 OK\r\nContent-Length: 3169\r\nContent-Type: application/json\r\nDate: Wed, 21 Aug 2019 13:36:14 GMT\r\nServer: gunicorn/19.7.1\r\nSnap-Store-Version: 15\r\nX-Request-Id: 6715A552C2700A325C8501BB5D5D48CD151B8B5B\r\nX-Vcs-Revision: 57828e7\r\nX-View-Name: snapdevicegw.webapi_refresh.snap_refresh\r\n\r\n{\"error-list\":[],\"results\":[{\"effective-channel\":\"stable\",\"instance-key\":\"download-1\",\"name\":\"blender\",\"released-at\":\"2019-07-30T14:28:55.511384+00:00\",\"result\":\"download\",\"snap\":{\"architectures\":[\"amd64\"],\"base\":null,\"common-ids\":[],\"confinement\":\"classic\",\"contact\":\"\",\"created-at\":\"2019-07-30T14:22:35.038569+00:00\",\"description\":\"Blender is the free and open source 3D creation suite. It supports the\\nentirety of the 3D pipeline\\u2014modeling, rigging, animation, simulation,\\nrendering, compositing and motion tracking, even video editing and game\\ncreation.\\n\\nBlender is a public project, made by hundreds of people from around the\\nworld; by studios and individual artists, professionals and hobbyists,\\nscientists, students, VFX experts, animators, game artists, modders, and\\nthe list goes on.\",\"download\":{\"deltas\":[],\"sha3-384\":\"9c4bee58188d00f9443aee893dad6d669c44c89431683c880b6ed6195e784f7013c0fc696c63f8b2366814f13ad0c621\",\"size\":123510784,\"url\":\"https://api.snapcraft.io/api/v1/snaps/download/ZWyHJJiPBBfMKpaaKKrpSgAcupWkHWPz_33.snap\"},\"epoch\":{\"read\":[0],\"write\":[0]},\"license\":\"GPL-3.0\",\"media\":[{\"height\":null,\"type\":\"icon\",\"url\":\"https://dashboard.snapcraft.io/site_media/appmedia/2019/07/blender.png\",\"width\":null},{\"height\":null,\"type\":\"screenshot\",\"url\":\"https://dashboard.snapcraft.io/site_media/appmedia/2019/07/blender_screenshot_1.jpg\",\"width\":null},{\"height\":null,\"type\":\"screenshot\",\"url\":\"https://dashboard.snapcraft.io/site_media/appmedia/2019/07/blender_screenshot_2.jpg\",\"width\":null},{\"height\":null,\"type\":\"screenshot\",\"url\":\"https://dashboard.snapcraft.io/site_media/appmedia/2019/07/blender_screenshot_3.jpg\",\"width\":null},{\"height\":null,\"type\":\"screenshot\",\"url\":\"https://dashboard.snapcraft.io/site_media/appmedia/2019/07/blender_screenshot_4.jpg\",\"width\":null}],\"name\":\"blender\",\"prices\":{},\"private\":false,\"publisher\":{\"display-name\":\"Blender Foundation\",\"id\":\"IEzHQPsJJcDJXVbfCdg5mh4cfFs7Xmhq\",\"username\":\"blenderfoundation\",\"validation\":\"verified\"},\"revision\":33,\"snap-id\":\"ZWyHJJiPBBfMKpaaKKrpSgAcupWkHWPz\",\"snap-yaml\":\"name: blender\\nversion: '2.80'\\nsummary: Blender is the free and open source 3D creation suite.\\ndescription: \\\"Blender is the free and open source 3D creation suite. It supports the\\\\n\\\\\\n  entirety of the 3D pipeline\\\\u2014modeling, rigging, animation, simulation,\\\\nrendering,\\\\\\n  \\\\ compositing and motion tracking, and video editing.\\\\n\\\\nBlender is a public project,\\\\\\n  \\\\ made by hundreds of people from around the\\\\nworld; by studios and individual artists,\\\\\\n  \\\\ professionals and hobbyists,\\\\nscientists, students, VFX experts, animators, game\\\\\\n  \\\\ artists, modders, and\\\\nthe list goes on.\\\\n\\\\nThe standard snap channels are used\\\\\\n  \\\\ in the following way:\\\\n\\\\n       stable - Latest stable release.\\\\n    candidate\\\\\\n  \\\\ - Test builds for the upcoming stable release.\\\\n\\\"\\narchitectures:\\n- amd64\\nconfinement: classic\\ngrade: stable\\napps:\\n  blender:\\n    command: command-blender.wrapper\\nlicense: GPL-3.0\\n\",\"summary\":\"Blender is the free and open source 3D creation suite.\",\"title\":\"Blender\",\"type\":\"app\",\"version\":\"2.80\"},\"snap-id\":\"ZWyHJJiPBBfMKpaaKKrpSgAcupWkHWPz\"}]}\n"
2019/08/21 19:06:14.857622 retry.go:53: DEBUG: The retry loop for https://api.snapcraft.io/v2/snaps/refresh finished after 1 retries, elapsed time=1.56260957s, status: 200
2019/08/21 19:06:14.858057 store.go:1352: DEBUG: Starting download of "/home/joseph/blender_33.snap.partial".
2019/08/21 19:06:14.858757 retry.go:41: DEBUG: Retrying https://api.snapcraft.io/api/v1/snaps/download/ZWyHJJiPBBfMKpaaKKrpSgAcupWkHWPz_33.snap, attempt 1, elapsed time=17.679µs
2019/08/21 19:06:14.859114 logger.go:67: DEBUG: > "GET /api/v1/snaps/download/ZWyHJJiPBBfMKpaaKKrpSgAcupWkHWPz_33.snap HTTP/1.1\r\nHost: api.snapcraft.io\r\nUser-Agent: snapd/2.37.4-1+b1 (series 16; classic; devmode) snap debian/10 (amd64) linux/4.19.0-5-amd64\r\nAccept: \r\nX-Ubuntu-Architecture: amd64\r\nX-Ubuntu-Classic: true\r\nX-Ubuntu-Series: 16\r\nX-Ubuntu-Wire-Protocol: 1\r\nAccept-Encoding: gzip\r\n\r\n"
2019/08/21 19:06:15.756700 logger.go:74: DEBUG: < "HTTP/1.1 302 FOUND\r\nContent-Length: 513\r\nContent-Type: text/html; charset=utf-8\r\nDate: Wed, 21 Aug 2019 13:36:15 GMT\r\nLocation: https://fastly.cdn.snapcraft.io/download-origin/fastly/ZWyHJJiPBBfMKpaaKKrpSgAcupWkHWPz_33.snap?token=1566406800_06c56fca6b5f0648018fb312bdf89bb473568c27\r\nServer: gunicorn/19.7.1\r\nSnap-Store-Version: 15\r\nX-Request-Id: 6715A548C6B90A325C9601BB5D5D48CF14FCA6B9\r\nX-Vcs-Revision: 57828e7\r\nX-View-Name: snapdevicegw.webapi_download.snap_download\r\n\r\n"
2019/08/21 19:06:15.757073 logger.go:67: DEBUG: > "GET /download-origin/fastly/ZWyHJJiPBBfMKpaaKKrpSgAcupWkHWPz_33.snap?token=1566406800_06c56fca6b5f0648018fb312bdf89bb473568c27 HTTP/1.1\r\nHost: fastly.cdn.snapcraft.io\r\nUser-Agent: snapd/2.37.4-1+b1 (series 16; classic; devmode) snap debian/10 (amd64) linux/4.19.0-5-amd64\r\nAccept: \r\nReferer: https://api.snapcraft.io/api/v1/snaps/download/ZWyHJJiPBBfMKpaaKKrpSgAcupWkHWPz_33.snap\r\nX-Ubuntu-Architecture: amd64\r\nX-Ubuntu-Classic: true\r\nX-Ubuntu-Series: 16\r\nX-Ubuntu-Wire-Protocol: 1\r\nAccept-Encoding: gzip\r\n\r\n"
2019/08/21 19:06:17.467005 logger.go:74: DEBUG: < "HTTP/2.0 200 OK\r\nContent-Length: 123510784\r\nAccept-Ranges: bytes\r\nAge: 603\r\nCache-Control: max-age=86400\r\nContent-Disposition: attachment; filename=ZWyHJJiPBBfMKpaaKKrpSgAcupWkHWPz_33.snap\r\nContent-Type: application/octet-stream\r\nDate: Wed, 21 Aug 2019 13:36:17 GMT\r\nEtag: f4e2fefe-dc63-4e52-85a5-3daa5eca1289\r\nLast-Modified: Wed, 21 Aug 2019 12:57:57 GMT\r\nServer: TwistedWeb/14.0.2\r\nStrict-Transport-Security: max-age=2592000\r\nVary: Accept-Encoding\r\nVia: 1.1 varnish\r\nVia: 1.1 varnish\r\nX-Bzr-Revision-Number: 7464\r\nX-Cache: HIT, HIT\r\nX-Cache-Hits: 0, 1\r\nX-Request-Id: XV0-1X8AAQEAAHhvAeQAAAAW1\r\nX-Served-By: cache-lcy19235-LCY, cache-sea1034-SEA\r\nX-Timer: S1566394577.335737,VS0,VE1\r\n\r\n"
2019/08/21 19:08:45.706319 retry.go:83: DEBUG: Retrying because of: read tcp 10.102.16.91:60852->151.101.54.217:443: read: connection reset by peer
2019/08/21 19:08:45.706378 retry.go:41: DEBUG: Retrying https://api.snapcraft.io/api/v1/snaps/download/ZWyHJJiPBBfMKpaaKKrpSgAcupWkHWPz_33.snap, attempt 2, elapsed time=2m30.847641533s
2019/08/21 19:08:45.715890 logger.go:67: DEBUG: > "GET /api/v1/snaps/download/ZWyHJJiPBBfMKpaaKKrpSgAcupWkHWPz_33.snap HTTP/1.1\r\nHost: api.snapcraft.io\r\nUser-Agent: snapd/2.37.4-1+b1 (series 16; classic; devmode) snap debian/10 (amd64) linux/4.19.0-5-amd64\r\nAccept: \r\nRange: bytes=819393-\r\nX-Ubuntu-Architecture: amd64\r\nX-Ubuntu-Classic: true\r\nX-Ubuntu-Series: 16\r\nX-Ubuntu-Wire-Protocol: 1\r\n\r\n"
2019/08/21 19:08:46.856673 logger.go:74: DEBUG: < "HTTP/1.1 302 FOUND\r\nContent-Length: 513\r\nContent-Type: text/html; charset=utf-8\r\nDate: Wed, 21 Aug 2019 13:38:46 GMT\r\nLocation: https://fastly.cdn.snapcraft.io/download-origin/fastly/ZWyHJJiPBBfMKpaaKKrpSgAcupWkHWPz_33.snap?token=1566406800_06c56fca6b5f0648018fb312bdf89bb473568c27\r\nServer: gunicorn/19.7.1\r\nSnap-Store-Version: 15\r\nX-Request-Id: 6715A557BB390A325CD101BB5D5D496615288E69\r\nX-Vcs-Revision: 57828e7\r\nX-View-Name: snapdevicegw.webapi_download.snap_download\r\n\r\n"
2019/08/21 19:08:46.857046 logger.go:67: DEBUG: > "GET /download-origin/fastly/ZWyHJJiPBBfMKpaaKKrpSgAcupWkHWPz_33.snap?token=1566406800_06c56fca6b5f0648018fb312bdf89bb473568c27 HTTP/1.1\r\nHost: fastly.cdn.snapcraft.io\r\nUser-Agent: snapd/2.37.4-1+b1 (series 16; classic; devmode) snap debian/10 (amd64) linux/4.19.0-5-amd64\r\nAccept: \r\nRange: bytes=819393-\r\nReferer: https://api.snapcraft.io/api/v1/snaps/download/ZWyHJJiPBBfMKpaaKKrpSgAcupWkHWPz_33.snap\r\nX-Ubuntu-Architecture: amd64\r\nX-Ubuntu-Classic: true\r\nX-Ubuntu-Series: 16\r\nX-Ubuntu-Wire-Protocol: 1\r\n\r\n"
2019/08/21 19:08:52.736703 logger.go:74: DEBUG: < "HTTP/2.0 206 Partial Content\r\nContent-Length: 122691391\r\nAccept-Ranges: bytes\r\nAccept-Ranges: bytes\r\nAge: 0\r\nCache-Control: max-age=86400\r\nContent-Disposition: attachment; filename=ZWyHJJiPBBfMKpaaKKrpSgAcupWkHWPz_33.snap\r\nContent-Range: bytes 819393-123510783/123510784\r\nContent-Type: application/octet-stream\r\nDate: Wed, 21 Aug 2019 13:38:48 GMT\r\nEtag: f4e2fefe-dc63-4e52-85a5-3daa5eca1289\r\nLast-Modified: Wed, 21 Aug 2019 13:11:39 GMT\r\nServer: TwistedWeb/14.0.2\r\nStrict-Transport-Security: max-age=2592000\r\nVary: Accept-Encoding\r\nVia: 1.1 varnish\r\nVia: 1.1 varnish\r\nX-Bzr-Revision-Number: 7464\r\nX-Cache: MISS, MISS\r\nX-Cache-Hits: 0, 0\r\nX-Request-Id: XV1DC38AAQEAAHohEwMAAABE1\r\nX-Served-By: cache-lcy19244-LCY, cache-lax8639-LAX\r\nX-Timer: S1566394728.047012,VS0,VE329\r\n\r\n"
2019/08/21 19:11:01.227418 store.go:1368: DEBUG: download of "https://api.snapcraft.io/api/v1/snaps/download/ZWyHJJiPBBfMKpaaKKrpSgAcupWkHWPz_33.snap" failed: &net.OpError{Op:"read", Net:"tcp", Source:(*net.TCPAddr)(0xc000234ba0), Addr:(*net.TCPAddr)(0xc000234c00), Err:(*os.SyscallError)(0xc00045e080)}
error: read tcp 10.102.16.91:47204->151.101.26.217:443: read: connection reset by peer

I looked at the download file… It disappeared right after download stopped.

I see the bug now, and yes it’s a bug; I’ll fix it. I’d only tested this with errors that also killed the download op… :expressionless:

In the meantime, you could use curl or wget to download that URL (the one that starts https://api.snapcraft.io/api/v1/snaps/download/), rename it blender_33.snap, and run snap download blender (so it checks the hash, and it gets the .assert file). Sorry for the inconvenience.

Do note, however, that there’s something wonky with your network :slight_smile:

1 Like

I feel sorry for the bug… But I’m happy you found it… :smiley:

I’ll do this… :slight_smile:

Yeah I know that… That’s why this resuming is so much important to me… :wink:

Thank you so much for helping me @chipaca

1 Like

Hello @chipaca Problem continues… :open_mouth: I did a little research before ask this and didn’t found any solution.

Today I downloaded blender_33.snap with wget. Then,

  joseph@joseph:~$ snap download blender
Fetching snap "blender"
Fetching assertions for "blender"
Install the snap with:
   snap ack blender_33.assert
   snap install blender_33.snap

then,

joseph@joseph:~$ sudo snap ack blender_33.assert

and then,

joseph@joseph:~$ sudo snap install blender_33.snap
error: This revision of snap "blender_33.snap" was published using classic confinement and thus may
       perform arbitrary system changes outside of the security sandbox that snaps are usually
       confined to, which may put your system at risk.

       If you understand and want to proceed repeat the command including --classic.

so then,

joseph@joseph:~$ sudo snap install blender_33.snap --classic
error: cannot perform the following tasks:
- Download snap "core" (7396) from channel "stable" (read tcp 10.102.132.145:39778->151.101.42.217:443: read: connection reset by peer)

I’m stuck again… :sob:

I’m afraid you’re going to have to repeat the same thing with the ‘core’ snap, i.e. download it via some other means, rename it as snap download expects, and go from there.

Alternatively, if you’re feeling brave, I wrote the fix for this and while it’s not even in master yet you could grab the snap binary built from that branch, and run download using that. Something like

wget -O /tmp/snap.bz2 https://people.canonical.com/~john/snap__amd64__PR7315_download_leave_partial.bz2
bunzip2 /tmp/snap.bz2
chmod +x /tmp/snap
/tmp/snap download core
1 Like

Hello @chipaca

Thank you so much for your help. I finally able to install it in my computer.

joseph@joseph:~$ sudo snap install blender_33.snap --classic
Warning: /snap/bin was not found in your $PATH. If you've not restarted your session since you
         installed snapd, try doing that. Please see https://forum.snapcraft.io/t/9469 for more
         details.

blender 2.80 from Blender Foundation (blenderfoundation✓) installed

It gave me this warning though.

I launched the app. It’s bit slower. Not very smooth. Is that normal? Will it update without any problems?

Thank you so much for your help @chipaca :slight_smile:

Snaps can be quite a bit slower to start the first time (a few seconds of delay, 2-3 seconds in my tests), a little bit slower the following times (less than a second, ~200ms in my tests). Once it’s started it should not be any less smooth than without the snap.
Classic snaps should not suffer these problems, however, so I’m not sure what you’re experiencing.

As to whether it’ll update without problems – I don’t know. Probably not automatically, given no single download succeeds, but … maybe? updates use deltas, so as long as teh delta is small it might be able to get through (an update will retry a number of times before giving up, and that might be enough to progress even in the ~800k chunks your network seems to cut off at).

Ah… thanks @chipaca:slight_smile:

Blender is launching without any problem… But the problem is when working… It’s slower little bit. I’ve worked a little and restart my computer and it’s better than before. Still a little slower, but it’s not slow as before.

Will you guys fix that download problem? I’ve been using this internet connection for years. I didn’t have any problems downloading anything. It’ll be a big help for most of the people out there I think. :slight_smile:

Thank you so much for fixing my problem @chipaca:hugs:

The problem is seemingly specific to your network. Most other installations do not exhibit this issue so we can conclude that it is environmental. How is your network set up? Do you have any proxies? Do you have any firewalls? Are you running any containers or virtual machines which have their own network addresses?

No I don’t @daniel… This is very cheap internet package from my ISP. :blush:

I’ve never had a problem with apt. I update my system and applications without any problem. It keeps downloading. I download things from online web sites. Watch youtube videos. I never had any problem with that too. Only snap install cause this problem. :blush: But I had some help to fix it. I’m happy about it… :slight_smile:

Ah, I’m sorry, I’d understood you to say that you knew your network wasn’t good. If that’s not the case, then, that’s interesting!

We have very rarely seen cases where go’s http2 client and fastly’s http2 server get confused, and fastly sends a reset (which AIUI—but I’m no expert—is http2 for “me no habla gibberish”).

One way to test that would be by asking snapd (or snap download) to not talk http2. This obviously isn’t a good long-term solution, but would you be able to check by doing

GODEBUG=http2client=0 snap download sdlpop

(sdlpop is 29MB which should be enough to trigger the misbehaviour you’re seeing, but feel free to use something bigger if you think it’s needed)

Please let me know how that goes.

I can confirm this in Panamá using Debian 10, downloads are a pain if you only have a “very cheap internet package from my ISP.”

mario@rune:~$ snap version
snap    2.37.4-1+b1
snapd   2.37.4-1+b1
series  16
debian  10
kernel  4.19.0-13-amd64

this is a very old snapd version … debian systems should be capable of using reexec to the snapd of the core snap which should have a more recent version available, try to explicitly run:

snap install core

that should re-exec and snap version should show a higher version

allow me some time to do the downloads, I’ll follow-up asap. thank you for the hint.

core is around 100M, so it took its time, like 2h. now I have this:

mario@rune:~$ snap version
snap    2.48.2
snapd   2.48.2
series  16
debian  10
kernel  4.19.0-13-amd64

I’m with sudo snap install snapcraft and I see that:

  • if I interrupt the download manually, the partial download disappears (I wish it didn’t, or that it would ask me, or that there were a setting affecting this behaviour).
  • if I turn off the wifi for a while then on again, the download does not resume (after more than a minute but less than two I interrupted it manually, so the partial download again disappears).
  • if I link the file being downloaded to an extra different name, manually interrupting the download does not remove my new link, so I can hope to resume. renaming my new link to the partial download and restarting the download seems to append to the file, but the process percentage starts from 0%, which I consider misleading.

its now at 12%, I already created the new link that won’t be removed, let’s see how it goes.

apparently there was a glitch in my internet connection, making the download reset/restart/resume … something. it’s still proceeding in the sense that the .partial file keeps growing, but the percentage restarted at 0%, which is a bit confusing.
also: nethogs reports a RECEIVED value around 15KB/s, but snap’s status line reports around 4000B/s.

nope: this happened and the .partial file was removed

Download snap "snapcraft" (5821) from channel "stable"           5% 13.2kB/s 80.6m
error: cannot perform the following tasks:
- Download snap "snapcraft" (5821) from channel "stable" (unexpected EOF)

just for future reference, once you have manually downloaded a new core, and once you have stopped/unmounted all snap related mountpoints, you must first remove core before you can install it from the local .snap file.

This is not true, you should not have to do this, all you should have to do after downloading the new revision and .assert file with snap download is:

$ snap ack <core_something>.assert
$ snap install <core_something>.snap

as directed in the output from snap download:

$ snap download core
Fetching snap "core"
Fetching assertions for "core"
Install the snap with:
   snap ack core_10584.assert
   snap install core_10584.snap

you are correct, I do not know why I had to do that, and I can’t replicate the situation. my guess is that a slow internet connection causes more trouble than I can measure.