Flatpak, snap, appimage... where is Gentoo Prefix?

snapd
1

Hi, this is a post about standalone linux apps (the ones the title starts with) and how to do them the gentoo way (personally I use arch).

As a note aside, I do not think the support for these type of apps is first priority. After all, user-space linux is firstly about letting people share each other’s work (through system packages), and only after that - about being a recognized platform that all software vendors should have support for.

Nevertheless, a couple of years ago, when snap and flatpak started to emerge, I was curious about it and have listened to various talks at FOSDEM, GUADEC and so.
I learnt about the 3 technologies present in this title, also about nix/guix (that in principle can serve the same purposes but their usage has been on the server side mostly).

But I realised recently that there can be a simpler way for doing this, that is conceptually close to how a regular distro is organized. After a little of googling I found that the idea has already been implemented in Gentoo Prefix. The idea is you install software in some prefixed path like /gentoo/prefix/ instead of /. The prefix is a hardcoded path and does not ever change. I think they even use the same gentoo packages. You can use the software installed in prefix in any distro - the main requirement is fulfilled.

The prefixed approach has all advantages and drawbacks of a regular distro:

  • does not allow multiple versions of an app

    which is not a problem, i think rolling release works well for apps. The users generally need just the latest version of your app. If the system has moved to a next snapshot in the testing channel you have enough time to fix the app if needed.

  • forces you to use specific versions of libraries (the latest stable, for example)

    While being a restriction, this also means more consistency, easier and smaller updates, etc.

  • no sandboxing

So, my question is - what do you think of Gentoo Prefix as a solution for cross-distro apps and why nobody is talking about it?

2

Your advantages and drawbacks indicate exactly why we don’t use something like this. They are the polar opposite of what we’re aiming for with Snaps:

We want multiple versions to be installable side-by-side.

We explicitly do not want to force a vendor to use specific library versions.

We want sandboxing. That’s a fundamental principle of Snap packaging.

4 Likes
3

Having multiple (and specifically older) versions of apps and libraries on the same system is really useful.

S4A is one of the snaps I created. Although the app is open source and still works fine, it relies on old 32-bit libraries. It is developed as a side project by a group of educators and they don’t have the time to constantly update the app to the latest libraries.

With Snap, I bundle older 32-bit libraries together with that app. + It’s running inside of a sandbox so I have to worry less about the security issues of an app that hasn’t been updated in 4 years. It doesn’t even have internet access!

There are also many Wine apps in the store. Bridge Designer is one of them; an open source Windows app also created by an educator. Depending on the app, certain older versions of Wine work better, so I currently have multiple different wine versions “installed” in multiple different snaps. The sandbox is also really useful here.

2 Likes
4

S4A is pretty cool :slight_smile:

5

I understand it’s polar opposite. I just previously thought that with system packages we can’t have linux apps cross-distro anyway, so we need to build something new. And if so why not add features like sandboxing etc. Now my understanding is more like flatpak/snap think linux is broken and needs fixing. User-space linux, of course

6

This is indeed not the case. Cross-distro apps is only one of the goals of Snap.

  • Decouple OS updates from app updates. This means each app needs to ship their own dependencies and apps are not allowed to use OS libraries.
  • Make OS upgrades and app upgrades more reliable and revertible.
  • Let software developers themselves publish directly to users. Without going through distribution maintainers.

In the original announcement, cross-distro compatibility wasn’t even discussed; that came later.

Flatpak doesn’t try to fix Linux user-space. Flatpak is designed to be an additional tool on top of a distribution. Flatpak is designed for user-facing desktop apps. All other software still needs to be installed using another system like APT, Yum or OSTree. Flatpak is much closer to Gentoo Prefix in that respect. Flatpak’s main goal is to have upstream developers publish a single package to all Linux users, so “cross-distro” is only half of the goal. The “upstream developers” part of the goal requires individual app containers and sandboxing.

Snap can completely replace a traditional package manager like APT but it still needs a system to build the snaps. Snapcraft heavily uses APT to do this, but other approaches exist such as the Nix package manager.

1 Like
7

I know flatpak is not a full replacement for apt (and neither is snap), but don’t think this changes anything. What relates to letting the upstream projects directly release to users, I don’t think that is main bonus either: you can always publish package source (in archlinux this is simple) that can later become an official (binary) package. Also I fail to see how flatpak is close to Gentoo Prefix

8

what makes you think this ?

while flatpak is a desktop-only app delivery mechanism, you can package, servers, cli apps, kernels, bootloaders as snaps.

we even have a fully snap based OS (UbuntuCore) that doesn’t not ship or allow apt/dpkg packages. snap is actually a fully fledged package manager …

1 Like
9

Thank you @ogra. You can package all that - yes. So snap is more like a spec for self-contained binary packages, to enable base snaps and confinement. It is not a package manager because it doesn’t support dependencies between snaps, and it is not easy to reuse build recipes from other snaps. Is is supposed that software vendor has enough expertise to build their app starting from the base snap. Also this: you often see deb packages from ubuntu repositories used in build recipes. That really looks like crutches: that is totally a different ecosystem (though opensource too) with different primary purpose. Also that probably means deb packages are now required to work both for traditional use and in snaps. Btw, does that bring additional troubles for your support engineers? Anyway, snaps are imho for large apps/teams that want and know how to provide all the stack of dependencies

10

This is a weird way to categorize a “package manager” … one of the initial reasons to create a new package manager was that dependencies of deb packages cause the majority of upgrade issues (people with randomly installed library debs that replace system libs, people with broken PPAs enabled etc etc) for our support people, these issues simply can not exist by design with snaps … yet there are extension snaps (shared libs like the gnome-3-26/-28/-34 snaps), there is the content interface so all your snaps can use the same library frameworks, daemon backends, etc … that you only need to install once but can use from all other packages …

I do have close to 100 snap packages in the store most of them use a part from someone elses snapcraft.yaml.
by design snapcraft encourages sharing of part receipes via copy/paste, this is one of its big advantages and strengths … how do you come to such a conclusion ? did you hit an issue where copy/paste of a part from someone else snapcraft.yaml did not work for you ?

if a software vendor knows how to build his own app (which i assume he (or one of his developers) does), this is as much as he needs to know to create a working snapcraft.yaml

it is as easy as picking the right plugin for the build system you are typically using to build your own app and add a name, version and description at the top of the file to get started.

havig packaged software in formats from slackware tarballs over rpm to debs (including plenty of its “special” flavours like ipkg, opkg) in my life, i personally havent seen any packaging system that makes creating a package of your software as easy as snapcraft. and my experience with commercial customers kind of confirms that many professional developers appreciate its ease of use.

yes, you do, because it is convenient … i do have as many snaps that build their libs from upstream sources as i do have simply using binary deb packages from the ubuntu archive. snap and snapcraft are designed to not limit you to either (heck i could even just cp binaries from your gentoo install into a snap package and could make it work if i wanted, there are no restrictions or limits whats inside a snap …)

well …

$ snap info esptool|grep installed
installed:    0.4.13 (x1) 16kB -

snap is just the currently easiest and safest delivery mechanism you can use to get your software (any kind of software) to as many users as possible on as many distros as possible. that they function in new ways does not make them less of of a package format, that dependency resolution is solved in a way cleaner and more elegant way than in other systems doesn’t either …

snaps come with many advantages and features out of the box that you will not find with other package managers … like automated self-tests and rollback, binary delta upgrades, data snapshots, parallel installs of the same app, secure interfaces to talk to all the rest of the system (from hardware over libs towards daemons in other snaps …) and even a REST management API if you need to manage software on a fleet of IoT devices, servers or desktops:

3 Likes
11

Thanks again @ogra. Well, a plugin is a way to build a part, and a part is more or less the analogy of a package. Only it doesn’t have a version, cannot be updated, cannot be marked as conflicting with another part and may use deb packages to build. Also, you do not have a registry of available parts, so you can grep github

12

Sorry, I am probably wrong. I see there is for example source-branch and source-commit. Nix definitions used to be somewhat similar

13

But you can forget to specify the source-commit and spend some time figuring out which sources were used for this single snap that is working

14

Most snaps contain a manifest.yaml file which details which specific versions of sources and packages were used in that build, even if snapcraft.yaml doesn’t contain those details.

Let’s look at Canonical’s Chromium snap as an example.

In snap/snapcraft.yaml, source explains what repository is used to build the package and stage-packages describe what dependencies are included in the snap.

 desktop-gtk3:
    source: https://github.com/ubuntu/snapcraft-desktop-helpers.git
    source-subdir: gtk
    stage-packages:
      - libxkbcommon0
      - ttf-ubuntu-font-family
      - shared-mime-info
...

In snap/manifest.yaml, source-commit is added to narrow down what exact source was used for this build and stage-packages include the exact version of each dependency.

 desktop-gtk3:
    source: https://github.com/ubuntu/snapcraft-desktop-helpers.git
    source-commit: 622e2aa7a840b3a7dbb6ea4d432d687d5cc2e8ef
    source-subdir: gtk
    stage-packages:
    - libxkbcommon0=0.8.2-1~ubuntu18.04.1
    - ttf-ubuntu-font-family=1:0.83-2
    - shared-mime-info=1.9-2
...

Finally, for snaps built on Launchpad and GitHub, snap/manifest.yaml also contains a URL to the build log.

image-info:
  build-request-id: lp-58361925
  build-request-timestamp: '2020-08-05T11:55:18Z'
  build_url: https://launchpad.net/~osomon/+snap/chromium-snap-firstrun-notice/+build/1071421

This manifest is enabled by default on when you build the snap on GitHub or on Launchpad. You can enable it on your local builds by setting an environment variable.

See how to verify the source of a snap package for more information.

1 Like
15

Note that you can use Nix to build snap packages.

1 Like
16

Thank you all very much, snapcraft forum is the best! Given the ambitious goals that snapd is aiming for I think it is very good at it.

However, I got curious about how Gentoo-Prefix approach can be technically implemented and think that the snap team has already solutions for it. From gentoo wiki:

Usage of the Gentoo Prefix consists of using everything from this prefix, before even looking at what the main operating system has installed. It simply changes the search paths such that the prefix comes first. Compilers and linkers installed by Portage in the prefix are configured such that they look into the prefix first as well. Not only that, but also does it make sure that compiled programs will keep on looking in the prefix via rpath directions. Programs compile and run using prefix provided tools and libraries.

So the prefix path is passed to building tools. Still, will it be required to have some wrappers for the apps within the prefix, maybe set some env variables, source some script? Snapd team, I think, you might know

1 Like
17

Are you talking about building a snap using Gentoo Prefix instead of APT, or are you thinking of putting Gentoo Prefix inside of a Snap? If it’s the second one, these are my thoughts on it:

  • Gentoo Prefix requires writable space, I think using $SNAP_USER_COMMON might be best for this.
  • You need a snap that bundles bootstrap-bash.sh and curl. When the snap first launches, it runs bootstrap-bash.sh $SNAP_USER_COMMON/gentoo-prefix.
  • You need a way to run commands in the prefix. I think a run command could set the required environment variables to add all gentoo-prefix apps to the PATH and then execute its arguments. Then a user could run gentoo-prefix.run bash to open a shell in the prefix or something like gentoo-prefix.run emerge -atv nano to manage apps inside the prefix.
  • For GUI applications, you’ll need to use the gnome-3-28 extension. It might be best to create a separate command gentoo-prefix.gui-run which uses desktop-launch to setup all gui-specific stuff.
18

I mean neither of this things, I want to use Gentoo Prefix as it is. I just thought snapd team came across similar problems with snapd

19

I think I’ve made my thoughts more streamlined.
How about the following way to organize a distro: a directory (say it has a unique path on the filesystem, say /prefix) that looks like a root filesystem inside (keeps libraries, headers, etc in predefined places). But you don’t need chroot to call binaries from there, it was compiled with the knowledge of its future location (prefix). Probably you will still need to set some env variables to call applications installed in the prefix, but you want that wrappers to be minimal.

You want the apps installed in the prefix to be portable, i.e. to work regardless of what distro is installed in /. So, /prefix will contain something like a distro inside it, with (mostly gui) apps installed. Still, /prefix is not for system desktop services, systemd, or DE(gnome). Those need to be provided by the main distro.

Now, it looks like there is a lot of duplicated work in /prefix and /. But nothing stops you to make the main distro be aware of /prefix and depend on it. Let’s say both will be rolling release. So, the apps in /prefix should work with any distro but “works best” with X.

Looks like the distro X is better than the typical distro, because the apps you package for it work with any distro. Am I wrong?

20

It’s hard to comment on it because it’s not clear what your goal is. Are there any issues with Snap, Nix, AppImage or Flatpak that you want to solve with this approach? Are there any issues with traditional distro’s that you want to solve? Why a new package manager?