Improvements in snap download

[atomatt]

It would be nice to make snap download consistent with snap install for end users.

Most snap commands make a call to the system’s snapd where the bulk of the processing takes place, but snap download is different - the download is handled entirely by the snap CLI. I believe this is only because access to the user’s file system is required when writing the snap’s binary and assertions.

Unfortunately, processing the download in the CLI means it may behave differently to snap install due to various bits of configuration typically only known to snapd:

• store API URL
• store tenant name
• user’s auth
• device auth

To improve consistency, the above configuration should always be applied for a download, with the user specifying the following on the command line:

• snap name
• channel or revision
• architecture

Therefore, snap/snapd probably needs to do one of the following:

• expose (safely) the snapd configuration to the CLI so it can perform the correct download
• proxy the download via snapd
• something else entirely

Note: proxying is somewhat complicated by the need for a progress meter in the CLI and the download process that involves deltas, retries, resumes, etc.

Would a download be backed by a Change?

[mvo]

One big reason behind snap download (and snap prepare-image) not going through the daemon is that it can be used on systems (like chroots) that cannot run the daemon.

[pedronis]

Ok, this points more to an approach where we optionally get creds to use in snap download if the daemon is around, and seems we need also a snap login mode that doesn’t need snapd

[niemeyer]

I think it’d make sense to have “snap download” depending more on the daemon, as we have a few awkward edge cases that make no sense from a traditional user perspective. Having “snap login” also going out of band would only create more of those problems I think.

Can we have a list of the cases we know about that need no-daemon access to “snap download”, so we can have a more informed decision on what to do?

[pedronis]

Sorry, my thoughts on the out-of-band snap login is more in the context of ubuntu-image, I don’t think it would make sense to ask people to relogin snapd itself with some dev only credentials that might give access to some snaps even possibly with the wrong architecture in order to make ubuntu-image work.

@niemeyer Matt started a different topic about the issues there:

(topic merged below)

[atomatt]

A few existing developer-oriented tools download snaps. The nature of these tools means they’re likely to be very specific about the source of the snap instead of relying on the host machine’s configuration.

For example:

• the snap may be in a store tenant the host machine is not configured to use
• the store may require the download to be authenticated as someone different to the developer’s snap login user
• the download should probably never use the host machine’s device auth

As a quick overview of known tools:

• ubuntu-image. Currently calls out to snap prepare-image, which shares much of the download implementation with snap download, to download core, gadget & kernel snaps referenced by the model assertion, and any extra snaps listed on the command line. The architecture and store tenant are also taken from the model assertion.
• snapcraft. Implements its own download including resume, retries, and caching. However it doesn’t support deltas (not really a problem) or user authentication (but only seems to download core/ubuntu-core anyway).

The current snap download is actually quite well suited to a tooling-style download, although it probably relies on env vars too much and includes an authentication hack (see UBUNTU_STORE_AUTH_DATA_FILENAME). However, snap download is presumably meant for managing the system’s snaps rather than the developer use case.

It may make sense to extract a low-level tool/library to assist with downloads or to defer download responsibility to each tool given that authentication is key and likely to be handled slightly differently, that would mean growing an ubuntu-image login subcommand (internally that could invoke snap login with dedicated flags/options to store an alternative login outside of snapd state).

[sergiusens]

snapcraft is a special snowflake user in this case, we download core to support two scenarios:

• to build a kernel snap
• to unpack in a chroot to build a classic confined snap.

These downloads can go away for each once:

• The initramfs logic is decouple, that is, keeping business logic in the core snap and keeping the initramfs in the kernel snap as hardware enablement only.
• launchpad buildd’s become capable of installing snaps, for which we will then just snap install snapcraft which would bring core in anyways for when classic confinement is set.

I still see value in just keeping all this logic in one place.

[niemeyer]

I have unified these two topics, as this is really the same conversation. The only reason we have the inconsistencies described in the first message is the tooling needs described by @atomatt in the comment above. Once we improve the behavior of snap download as described at the top, these needs above must be accounted for.

[pedronis]

Until these snapd/snap limitations are lifted/fixed the following flow based on snapcraft export-login can be used to download snaps that require authorized access, this is an example with snap download but the same applies to ubuntu-image/snap prepare-image:

$ snap download test-snapd-private
Fetching snap "test-snapd-private"
error: cannot find snap "test-snapd-private": snap not found
$ snapcraft export-login --acls package_access p.login
Enter your Ubuntu One e-mail address and password.
If you do not have an Ubuntu One account, you can create one at https://dashboard.snapcraft.io/openid/login
Email: testdev@canonical.com
Password:

We strongly recommend enabling multi-factor authentication: https://help.ubuntu.com/community/SSO/FAQs/2FA

Login successfully exported to 'p.login'. This file can now be used with
'snapcraft login --with p.login' to log in to this account with no password
and have these capabilities:

snaps:       No restriction
channels:    No restriction
permissions: ['package_access']

This exported login is not encrypted. Do not commit it to version control!
$ UBUNTU_STORE_AUTH_DATA_FILENAME=p.login snap download test-snapd-private
Fetching snap "test-snapd-private"
Fetching assertions for "test-snapd-private"

Notice the requirement to pass --acls package_access to export-login, otherwise its default is for upload access which is different, would not cover downloading.

Be of course careful with the login files as they contain obviously authorisation secrets.

[mbeneto]

H̶i̶ ̶@̶p̶e̶d̶r̶o̶n̶i̶s̶,̶ ̶i̶s̶ ̶t̶h̶i̶s̶ ̶w̶o̶r̶k̶a̶r̶o̶u̶n̶d̶ ̶s̶t̶i̶l̶l̶ ̶w̶o̶r̶k̶i̶n̶g̶?̶ ̶W̶e̶ ̶o̶w̶n̶ ̶a̶ ̶b̶r̶a̶n̶d̶ ̶s̶t̶o̶r̶e̶ ̶a̶n̶d̶ ̶t̶h̶e̶ ̶o̶n̶l̶y̶ ̶w̶a̶y̶ ̶t̶o̶ ̶d̶o̶w̶n̶l̶o̶a̶d̶ ̶s̶n̶a̶p̶s̶ ̶i̶s̶ ̶f̶r̶o̶m̶ ̶d̶e̶v̶i̶c̶e̶s̶ ̶p̶r̶o̶p̶e̶r̶l̶y̶ ̶c̶o̶n̶f̶i̶g̶u̶r̶e̶d̶ ̶u̶s̶i̶n̶g̶ ̶a̶ ̶g̶a̶d̶g̶e̶t̶.̶

For downloading snaps from a brand store using this workaround, it’s also required to pass the UBUNTU_STORE_ID.

UBUNTU_STORE_ID=************* UBUNTU_STORE_AUTH_DATA_FILENAME=p.login snap download test-snapd-private

[pedronis]

While we are actively but slowly working toward having in the default case snap download go through snapd, we don’t plan to disable the workaround. A set of other features is required to fully supplant it.

[abitrolly]

To bulk download of my snap history I used something like this.

export SNAP=yakshaveinc
export FROM=1
export TO=74

snapcraft export-login --snaps=$SNAP --acls=package_access ysi.download

for i in $(seq $FROM $TO); do UBUNTU_STORE_AUTH_DATA_FILENAME=ysi.download snap download $SNAP --target-directory=ysi --revision=$i; done

rm ysi.download