Provide a way to get source for a snap in the store

Forgive me if this has been discussed before.

With apt repositories, we have apt source to get source packages. This is often super convenient, because you can download the code of the thing that you are running, look at it, modify it if you want, and then rebuild / redistribute your new version. All that open source good stuff.

It’s a bit more difficult with snaps though, because there’s no way for publishers - even optionally - to provide data that would allow snap on the system to grab the source corresponding to a given version. I’ve only been able to do so by scrabbling around on the interwebs before and trying to unerath the source.

Would the snapd team consider such a feature? Some way for snap publishers to optionally say: here is the source that I built this snap from, and then snap providing a way of getting that onto the system for you.

I don’t know what’s best: the store being able to receive, retain and serve the code (like a Debian/RPM source package), or the snap providing a URL(-like thing, e.g. repo and commit ID/tag) and then snap directly retrieving the code from there. With the latter you risk the source becoming unavailable for any reason I guess.

3 Likes

Snaps can be packaged from multiple sources, not just sourcecode. But there is a manifest for most of them: /snap/<name of snap>/current/snap/manifest.yaml. This doesn’t take you directly to all the source, but it is a starting point.

Note that this was also recently discussed in Official snap for review-tools @noise said they are looking into showing the info from the manifest in the store.

I think the way forward is to expose the manifest in the store and include the snapcraft.yaml source URL in the manifest. I’m not sure why snapcraft doesn’t already do this.

3 Likes

Well, consider that not every application in the store is open source, and sticking the snapcraft.yaml publicly will potentially mean revealing things that app developers don’t want to be revealed. For example the yaml may reference internal code names, source control systems, or paths which would expose private data.

2 Likes

For example the yaml may reference internal code names, source control systems, or paths which would expose private data.

Hm, I would think that when publishers opt-in to generating the manifest, they would not mind also exposing where the source repository is hosted? The entire snapcraft.yaml itself is included in the snap, so why not the URL where it comes from too?

I suspect this is one reason why the manifest/snapcraft.yaml isn’t bundled in the snap by default, but only when building an open source application in launchpad, or when explicitly set to do so via an environment variable at runtime. So yes, if the manifest/snapcraft.yaml is there, I think it might be fine to expose it, given you can see it inside the snap, but not every application will have one.

2 Likes

I also want to note that this already happens for snaps built on Launchpad. Note that Launchpad is also the backend for the snapcraft build service.

As an example, the latest version of the Chromium snap contains this URL in the manifest: https://launchpad.net/~osomon/+snap/chromium-snap-firstrun-notice/+build/1071421

Using this URL as a source of truth, you can be 100% certain which code is used to build the snap. Launchpad isn’t the easiest to navigate but everything should be there.

F.Y.I.

2 Likes

Thanks everyone for the replies.

Neat that a cross reference already exists for some builds!

If it’s easier to do so, then I think doing this only for Launchpad-based builds initially would be a good first step. Ideally we would keep other builders in mind though, to not exclude any future extension. For example if Mozilla were to build the Firefox snap themselves and upload that to the store, they would have a way to let you get their source.

Using Chromium as an example, I’m imagining snap source chromium when this revision is current to essentially give me a checkout of https://git.launchpad.net/~chromium-team/chromium-browser/+git/snap-from-source at 68564b318b3e3ac917932a2c89a6d840c10e6bb2. Assuming all the referenced sources exist - see below - this would give you what you need to get to work.

Now, it’s fair to say that snaps can be built from multiple parts - e.g. in this case the actual upstream source of Chromium itself is downloaded at build time! If that goes away for some reason then you certainly will not be able to hack on the Chromium you’re using. That’s far from ideal and it’s why I wanted to put the idea of the store retaining copies of source on the table (not forgetting that the built-from repo itself could also vanish at any point; aside: is all of this a potential GPL compliance issue for snaps shipped by distros?). But then it becomes more complicated of course: the returned snapcraft.yaml would have to point to the downloaded copies rather than the URLs it did when it was built. And similarly to the above, for non-Launchpad & non-snapcraft builds, I suppose we’d need to just trust the source the developer uploads. Maybe that’s OK though.

Sorry for that long paragraph!

3 Likes

Now, it’s fair to say that snaps can be built from multiple parts - e.g. in this case the actual upstream source of Chromium itself is downloaded at build time! If that goes away for some reason then you certainly will not be able to hack on the Chromium you’re using.

manifest.yaml already goes through great lengths to record the exact source of every part. Moreover it seems to be in a format that can be interpreted by snapcraft itself. I haven’t actually tested this out if you can just feed it into snapcraft though.

The Chromium snap as an example. This is part of the snapcraft.yaml:

 desktop-gtk3:
    source: https://github.com/ubuntu/snapcraft-desktop-helpers.git
    source-subdir: gtk
    stage-packages:
      - libxkbcommon0
      - ttf-ubuntu-font-family
      - shared-mime-info
...

This is part of the manifest.yaml:

 desktop-gtk3:
    source: https://github.com/ubuntu/snapcraft-desktop-helpers.git
    source-commit: 622e2aa7a840b3a7dbb6ea4d432d687d5cc2e8ef
    source-subdir: gtk
    stage-packages:
    - libxkbcommon0=0.8.2-1~ubuntu18.04.1
    - ttf-ubuntu-font-family=1:0.83-2
    - shared-mime-info=1.9-2
...

Notice how the manifest has the same format, it simply adds the versions of all packages and the git commit of repos. It also adds checksums for downloaded binaries etc. (not shown here)

So the sources are not saved, but they are “recorded” in a way that you should be able to reproduce them if the sources for all the parts stay online. Do you think this is sufficient, or do you think the sources should also be saved? I’m not actually sure if the second option is economically feasible given that the Launchpad build service is completely free of charge.

1 Like

Launchpad can actually already optionally build a source tarball after it runs snapcraft pull, for this sort of purpose. It’s configured by the snap.build_source_tarball API property. That should, in principle, alongside manifest.yaml allow the exact source to be obtained and rebuilt, but likely needs some further integration with snapcraft and co. to be properly useful.

6 Likes

Hopefully we will see this implemented soon. Great idea.

2 Likes

I am not expert, but on Ubuntu I did the search sudo updatedb && locate --regex snap/<package name>/.*\\.yaml$ and, in the case of vlc, the result was /snap/vlc/2103/meta/snap.yaml which contained at the start:

name: vlc
version: 3.0.12.1

Of course, this is not enough, because we need the actual source code and this means knowing the specific build. If the source code was adapted by snap maintainers and does not correspond to any build, then it is not less important, because the VLC licence, the GNU license, says

… you have certain responsibilities if you distribute copies of the software, or if you modify it: responsibilities to respect the freedom of others.

For example, if you distribute copies of such a program, whether gratis or for a fee, you must pass on to the recipients the same freedoms that you received. You must make sure that they, too, receive or can get the source code.

The VLC snapcraft.yaml can be found here,

https://code.videolan.org/videolan/vlc/-/blob/b21abf42dfeba30a64b6611530c0baf2f3b9a512/extras/package/snap/snapcraft.yaml#L86

It sets the version with
echo $(git describe HEAD) > $SNAPCRAFT_STAGE/version

As a result, version 3.0.12.1 in the snap store corresponds to the git tag of 3.0.12.1 specifically, and the nightlies even include the commit hash, E.G the latest beta release identifies as 3.0.12.1-145-gf348c4aae8

This is enough information to identify the exact state of the VLC repository at the time it was built.

2 Likes

I could not find the specific commit (mentioned by snap) in the VLC repository and I checked in all the branches in their gitlab (self maintained) repository. I must be looking in the wrong repository. What I did is snap info vlc and this returned installed: 3.0.12.1-145-gf348c4aae8 (2270) 309MB -. Then I cloned the 3.0.0 branch (the other one available was 4.00) and checked directly in the log and also tried to checkout the commit with no success. It would be way simpler if snap directly offered the source code, when it is open source, as they actually should.

Good spot! it looks like there’s a single extra “g” being added to every hash in the version info for some reason, and it’s also in a special repository dedicated to VLC 3 releases, gf348c4aae8 should actually be f348c4aae8 and so can be found here https://code.videolan.org/videolan/vlc-3.0/-/commit/f348c4aae8b5b029fbc642d7f9299e34fa27a567

I appreciate there’s potential to improve the process with some better metadata and I’d be entirely up for making it easier (Using snap to distributing GPL code myself, lawyers are scary!) but the GPL3 doesn’t actually demand that Snapcraft host the code, only that the code is accessible. The GPL2 is less clear here, but my person none lawyery assessment is that it’d apply to both equally or the entire computing ecosystem would have some big problems.

GPL3:
If the place to copy the object code is a network server, the Corresponding Source may be on a different server (operated by you or a third party)

I was just complaining because it did not work, but maybe I should have thought that “g” is not a part of the hexadecimal notation for hash. It works actually perfectly. It just needs to be documented and maybe remind people that g is not in [0-f]. Why an extra “g” anyway. Well, there is an explanation for every thing: https://github.com/pypa/setuptools_scm/issues/152#issuecomment-282271402.

1 Like

Can you report this as an issue to VLC? Since they build the snap package, they are responsible to add the manifest. They can instruct snapcraft to add this manifest by setting the environment variable SNAPCRAFT_BUILD_INFO=1.

2 Likes

Done: https://code.videolan.org/videolan/vlc/-/issues/25612.

1 Like

IANAL for sure, but my worry is being able to guarantee that this is the case if the source is hosted elsewhere. We can refer to (e.g.) upstream’s git repository, but if that goes away and either the snap store is still offering the software, or we’re hosting Ubuntu images with it, that could be a problem I think.

For Ubuntu we do two main things at the minute in this area:

  • Build source ISO images alongside the actual (binary) ones, and host them together with them. That keeps the source around as long as the binaries are. But this only works for .debs at the minute.
  • Keep source packages available as long as their .deb binaries are.

For me thinking about Ubuntu being able to extend both of those to snaps are what I’d like to be able to get to. The snap store I think should care about this for things they host there, one way or another (unless there’s a lawyerly reason it’s not required there).