Mechanisms for converging store and snap metadata

[sergiusens]

Background and motivation

Besides the snap binary itself, there are several metadata fields associated to a revision and/or a snap. The metadata from a revision is set and fixed when uploading a new binary, but package metadata is editable the web UI (and initially set on first upload). There should be a simple and clear way to set and update the editable snap package metadata, using snapcraft and through the web UI.

Existing discussions

Snap development sprint ideas/agreements

Development sprint June 26th, 2017

Agreements about how the management of textual information (summary, description, translations, etc) will take place between snapcraft and store: [image]

Relevant bugs

• Please add “contact” field to the API LP: #1666792
• metadata not being updated upon snap push LP: #1681453
• Please allow “contact:” override on a per-snap basis LP: #1709610

Forum thread(s)

Snap metadata updates

We are continuing design of the metadata update improvements, as discussed during the June sprint [1], and also related to previous discussion [2] and bugs [3][4][5]. Given that context, there are a few questions for which it would be nice to get some feedback before getting to work on a concrete plan: After editing some fields in the UI, next uploads will be processed but warn about conflicts/not updating metadata? should some fields (the ones that weren’t changed in the UI) still be updated…

Existing implementation

Metadata fields are initially populated from the snap.yaml when a snap is first uploaded. From here on, some of the fields can be edited via the web UI, while others can’t. Confusingly, if the snap is rebuilt with new metadata in its snap.yaml and pushed, the store metadata is NOT updated.

Proposed solution and implementation

Initial metadata will still be set on first upload with the provided data (needed to keep backwards compatibility with previous snapcraft releases). However, subsequent pushes to the store will also update the metadata, unless there were some change(s) made through the web UI.

In that case, snapcraft will warn the user possibly displaying the conflicting fields information. Developer will have the option to force the update and override the metadata details with his/her values. There will be also the option for the developer to set a specific value for a specific metadata field, individually.

User Experience

Under normal circumstances, the snap will pushed and the metadata updated given the user has never edited the metadata on the dashboard:

$ snapcraft push <.snap>
Revision X created
The metadata has been updated

For the case of a conflict, as a first cut only listing the items with conflicts (text items could display their store values though):

$ snapcraft push <.snap>
Revision X created
The metadata has not been updated due to conflicts in these items:
- description
- icon

Fix the conflicts and run `snapraft push --only-metadata`
or override the store data by running 
`snapcraft push --only-metadata --force-metadata`

At any given time an individual can run,

$ snapcraft push --only-metadata
The metadata has been updated

[mpt]

It’s good to see that it would be possible to edit most of the metadata supplied in snapcraft.yaml outside of snapcraft.yaml as well. That would allow, for example, an editor or translator at a software vendor to supply a description for an upcoming release at the same time (and on the same page) as they upload the screenshots.

I’d like to see a bit more detail on how this interacts with channels. For example, if the store has store-edited metadata for version 1.3 in stable, and I push 2.0-preview.1 to edge with a new description describing the new features in 2.0, what happens? If snapcraft prompted me to override the customized 1.3 metadata, then for however many months before 2.0 was released to stable, I’d be engaging in false advertising — describing features in 1.3 that don’t exist in that version. Maybe one solution is for the conflict handling to kick in only when releasing to stable.

[sergiusens]

@mpt

I’d like to see a bit more detail on how this interacts with channels. For example, if the store has store-edited metadata for version 1.3 in stable, and I push 2.0-preview.1 to edge with a new description describing the new features in 2.0, what happens? If snapcraft prompted me to override the customized 1.3 metadata, then for however many months before 2.0 was released to stable, I’d be engaging in false advertising — describing features in 1.3 that don’t exist in that version. Maybe one solution is for the conflict handling to kick in only when releasing to stable.

Interesting, I had the same thought on metadata being per revision in the sense that some revisions of a snap would advertise something that another wouldn’t (regardless of channels) a description could even say This is an untested CI/CD build, use with caution for things you might intend to release to edge but wouldn’t necessarily want the metadata to affect what is shown for stable.

That said, the question you ask, is already a problem today given that even if on the web, there is only one source of metadata regardless of channels.

Consequentially, this makes me think that maybe the updating of metadata should be left for something to do at the time of release to a stable channel, which could potentially just be handled on the store side completely.

[matiasb]

@sergiusens

Interesting, I had the same thought on metadata being per revision in the sense that some revisions of a snap would advertise something that another wouldn’t (regardless of channels) a description could even say This is an untested CI/CD build, use with caution for things you might intend to release to edge but wouldn’t necessarily want the metadata to affect what is shown for stable.

Note the metadata we are talking about is the package level metadata, i.e. independent from a revision. Right now we don’t have an icon, summary, license or description per revision, but a general/global, per package value. And that’s the metadata we are enabling to set/update with this command. FWIW, the only editable metadata value per revision atm (and only via web) is the changelog.

[mpt]

@matiasb

Right now we don’t have an icon, summary, license or description per revision, but a general/global, per package value.

Maybe, but I don’t see how that’s viable long-term, for the reason I gave. A vendor shouldn’t be prevented from mentioning their app’s cool new features, in the app description, merely because they’re thoughtfully also providing an old version for enterprise/low-end users.

I raised it because it would save time and confusion if conflict handling is designed to work as similarly as practical, now, as it will when per-version descriptions are introduced.

Another issue that might influence conflict handling is localization. Apparently descriptions aren’t localizable yet. But when they are, vendors will want to provide localized screenshots and videos in the store, so it would make sense for them to be able to provide localized descriptions in the same store UI. But then what happens if some languages have had their descriptions changed in the store UI, while other languages have updated descriptions in the snap itself? A choice between overriding all the descriptions on push, and overriding none of them, might be too coarse.

[matiasb]

@mpt

Maybe, but I don’t see how that’s viable long-term, for the reason I gave. A vendor shouldn’t be prevented from mentioning their app’s cool new features, in the app description, merely because they’re thoughtfully also providing an old version for enterprise/low-end users.

I raised it because it would save time and confusion if conflict handling is designed to work as similarly as practical, now, as it will when per-version descriptions are introduced.

I see your point, and I agree we would probably want some metadata per revision. That will require another discussion and extra work to be scheduled. We are adding a session for next week in the rally.

@mpt

Another issue that might influence conflict handling is localization. Apparently descriptions aren’t localizable yet.

FTR, there is (basic) store support to set translated metadata (and also screenshots), although translated details are not yet exposed to clients. There is a discussion going on here, Getting translated snap metadata via snapd, and there is a session scheduled for next week too. It will be great to also have input from the snapcraft team on how translations would be ideally managed from the client side (and discuss how this could affect the task being discussed here).

[facundobatista]

This is the API description for the new metadata endpoints, comments/feedback welcomed!

Text fields

Endpoint: /dev/api/snaps/<snap-id>/metadata

Request:

• POST for normal upload, PUT for the request to force the update (ignoring and clearing conflicts), GET to return the metadata values currently set in the store.
• Required macaroon user authentication (check for owner or collaborator)
• Required content-type: application/json
• Optional data: {“field”: “value”, …} (for POST/PUT only).

Possible responses:

• 200, metadata updated, no issues
• 401, metadata not updated, invalid user authentication
• 404, metadata not updated, snap-id not matching a snap (or user not allowed to update it)
• 409, metadata not updated (may happen on a POST), error response describing conflicting fields

Binary fields

Both ‘icon’ and ‘screenshots’ are binary fields, in the future we may have others.

Binary fields needs to be handled differently than text fields, because uploading these every time would be a waste of bandwidth in clients and servers, also worsening the user experience on every snapcraft push.

Below is a definition of the snapcraft dialog to the server for uploading binary metadata fields, considering the context just described.

Endpoint: /dev/api/snaps/<snap-id>/binary-metadata

Sequence:

1. snapcraft does a GET to the endpoint, sends snap_id, auth info, etc
2. receives a list with (type, hash_sha256) of all metadata binary fields
3. snapcraft checks local binary files, and decides if there is something new/different that needs to be uploaded; if not, dialog stops here
4. snapcraft does a multipart POST (normal) or PUT (force update) to the endpoint; one part is json encoded, “info”, with a list with each item a dict with following info: field type (‘icon’, ‘screenshot’), filename, hash, key (a reference to the rest of the multipart request); the other parts are some binaries that needs to be uploaded. Note that the “info” list must include all the binary files that snapcraft has, but the rest of the parts are only for those files needed to be pushed

Example:

1. Snapcraft has the following files in the just pushed snap:
   1. An icon, ‘icon.png’, hash 7890
   2. A screenshot, ‘sshot1’, hash 1234
   3. A screenshot, ‘sshot2’, hash 2345
   4. A screenshot, ‘sshot3’, hash 3456
2. Snapcraft hits the binary endpoint with a GET request, receiving a list with with following items:
   1. {‘type’: ‘icon’, ‘hash’: 7890}
   2. {‘type’: ‘screenshot’, ‘hash’: 1234}
   3. {‘type’: ‘screenshot’, ‘hash’: 5555}
   4. {‘type’: ‘screenshot’, ‘hash’: 6666}
3. Snapcraft understands that the local files have different information than in the Store, and decides to do a multipart POST, each part having:
   1. “info”: […]
      1. {“type”: icon, “hash”: 7890}
      2. {“type”: screenshot, “hash”: 1234}
      3. {“type”: screenshot, “hash”: 2345, “key”: “1”, “filename”: “ss_a.png”}
      4. {“type”: screenshot, “hash”: 3456, “key”: “2”, “filename”: “ss_b.png”}
   2. “1”: <lot of bytes>
   3. “2”: <lot of bytes>
4. Store receives the request, and as no field was tainted from the web ui, just keeps the icon and screenshot with hash 1234, removes the screenshots with hashes 5555 and 6666, and saves the screenshots with hashes 2345 and 3456.

Possible responses: same as described for the “Text fields” endpoint above

[galgalesh]

@matiasb

I see your point, and I agree we would probably want some metadata per revision. That will require another discussion and extra work to be scheduled. We are adding a session for next week in the rally.

Has that rally resulted in some decision about per-revision metadata?

Currently, publishing a snap to any channel, track and branch will update the store metadata. This is not the functionality I expect. I expect that only publishing a snap to the stable channel updates the metadata. The user installs the stable channel by default so the user should see the metadata of the stable snap by default. I created a bug report for this but was directed towards this forum to discuss this issue.

We are currently automatically building a snap in the edge channel for each PR, in a branch named after that PR. Much to our surprise, publishing a snap to edge/yaru-pr32 updates the snap metadata. This also has security implications as explained in the issue. If you set up an automatic build system to build and publish each snap to a branch in the edge channel, then anyone who can create a PR can change the metadata of the snap.

@mpt also explains how per-snap metadata isn’t a viable long-term solution.

@mpt

A vendor shouldn’t be prevented from mentioning their app’s cool new features, in the app description, merely because they’re thoughtfully also providing an old version for enterprise/low-end users.

So my suggestion is the following:

1. For the short term, only update the metadata when a snap is pushed to the stable channel.
2. For the long term, consider per-revision or more granular metadata.

Also relevant; the license metadata discussion surfaced the same need for per-revision metadata.

[mpt]

@galgalesh

1. For the short term, only update the metadata when a snap is pushed to the stable channel.

Since the previous discussion, the directory now shows all snaps with releases, even if none of the releases is stable.

So when a snap doesn’t have a stable release, that rule wouldn’t work.