What's the story of build reproducibility of snaps?

https://reproducible-builds.org/

3 Likes

I’ve been working on this some with snapcraft for myself. You can do some things to make a snap reproducible, but how much work it will be is going to be dependent on your project. I don’t think it was a design goal and this make some sense. I can see where some of the things I’ve done to make my builds more reproducible don’t alighn with the security features snaps afford. It would still be really nice to have and isn’t unachievable depending on your project.

Some hurdles I’ve run into are:

The latest packages are pulled from apt and pip by default.

You can work around this, snapcraft allows you to specify the exact version of the package you desire. Similarly snapcraft pulls down the latest compilers, you need to make sure this is version locked as well.

Unable to add additional or change repo sources.

Using a ROS part, whose repo removes old versions, causes a lot of issues here. You can’t specify the version as it is removed on the next repo update. To get around this requires editing the snapcraft source code, there is a list of sources that can be edited so you can point snapcraft to your own repo or the ROS archive with locked down versions.

I think there are a couple threads about being able to add your own sources here in the forum. Reproducible builds is a good use case for being able to do so. It does not fully align with the auto update policy though. Why force a user to auto update a snap if the developer is using the same outdated packages they were a year ago. Also, if you use local sources you may lose out on the security notifications provided by Canonical for the packages in your snap.

Rosdep uses the latest changes

This issue is ros specific, but it’s possible that ROS will switch a python dependency from pip or apt to the other. Since rosdep in snapcraft uses the latest rosdep you’ll pick up this change.

Pip

I’m not as certain on this one, but IIRC pip doesn’t have the same guarantees apt does. It’s possible for a publisher to push a version again that has different contents overwriting the previous version. For this reason specifying a local pip repo would be helpful.

I could see other packages having similar hurdles, like node pulling in the latest from NPM, but I don’t use them so can’t really speak to them.

After thinking about this further, while I would really like reproducible builds, I’m not sure Snapcraft can/should implement reproducibility. Though solving some of the above could help others implement reproducibility into their own builds. Since any developer could use a custom plugin that breaks reproducibility, I don’t know if it makes sense to advertise snapcraft as a reproducible tool. On the other side of that though, there isn’t anything stopping you from making a custom plugin or source edits to allow your builds to be reproducible since snaps are an open format.

1 Like

I made empty snap that is being automatically rebuilt every time on CI/CD run. Reproducible builds may help avoid new releases if package contents didn’t change.

I decided to see how these empty snaps differ. First I downloaded all my snaps history.

Then I used https://diffoscope.org/ to compare two latest snaps (this forum does not support attachments) and here is the result I get.

image

The only difference is the timestamp that is repeated four times through the file.

✗ unsquashfs -s yakshaveinc_73.snap
Found a valid SQUASHFS 4:0 superblock on yakshaveinc_73.snap.
Creation or last append time Tue May 26 20:27:38 2020
...
✗ unsquashfs -s yakshaveinc_74.snap
Found a valid SQUASHFS 4:0 superblock on yakshaveinc_74.snap.
Creation or last append time Wed May 27 14:25:37 2020
...
1 Like

Squashfs tools 4.4 are now able to do reproducible builds. It looks like it may be used to fix timestamps in existing snaps for comparison.

$ mksquashfs source source.sqsh -mkfs-time $(date +%s -d "Jan 1 2019 19:00")

snapcraft doesn’t understand or handle SOURCE_DATE_EPOCH when building .snap SquashFS filesystem. Opened the bug about it.

1 Like