How to handle large optional components in a snap

[ijohnson]

Hi folks,

I am working on a snap that is already somewhat large at about 182M, and this software has the ability to work with some optional machine learning frameworks such as MXNet, Tensorflow, Chainer, etc. all of which are easily at least 100 MB and will only be used by some folks using the snap. Ideally, I’d rather not have to maintain different tracks for each framework, as it’s not very scalable because the snap currently supports 5 different frameworks and will likely support even more as they are supported and released. And a user may want to use different machine learning frameworks simultaneously, which means I would need to have 5! = 120 different tracks for the different combinations which is definitely a bad idea.

Currently, the best way I see to handle this is to have 2 different tracks, the default not containing any machine learning frameworks and the other track say something like ml or something like that contains all of the frameworks.

Also complicating this is the fact that some of these machine learning frameworks aren’t easily relocatable, and so using the content interface presents difficulties. Perhaps layouts helps solve this a bit, but by default many of the files installed would be in different dirs like /usr/bin, /usr/lib, etc. and so there would be a lot of layout rules, and in addition these layout rules would need to “overlay” on top of the files already present in the snap in those directories. Re-compiling all of these libraries by hand to work in relocatable directories is a daunting task that I’d rather not have to maintain… Machine learning frameworks are not famous for being easily compilable…

Lastly having two different snaps is another potential solution, one with the machine learning frameworks and one without it. But this seems redundant and confusing to users, especially since I don’t think the software will work well when installed on a system multiple times and would present (slight) complications if/when a user decides to switch between the versions.

Are there other ways we can handle this problem? I’m specifically looking for input from snapd architects such as @pedronis and others

I note that this seems very similar to the plugin issue with gimp here: GIMP Plugins from third parties

[Lin-Buo-Ren]

Is the content interface met your need?

The content interface

The content interface allows sharing code and data from a slot on a producer snap to one or more plugs on consumer snaps. It’s often used to share common themes and icons from a producer snap, for example, to many snapped applications. Sharing happens at the filesystem level, which means anything that can be expressed as a file can be shared. This includes executables, libraries and data files, but also sockets. Example The Yaru MATE Icons snap is a good producer snap example, letting other ap…

Example: The Hunspell Dictionaries Content Snaps

[ijohnson]

As mentioned in my original post, some of the files that these machine learning libraries expose will be in directories that the original snap also has files in, i.e. the original snap will have:

usr/bin/my-snap-app
usr/lib/my-snap-libs

and the ml frameworks have files like:

usr/bin/ml-launch-thing
usr/lib/ml-lib1
usr/lib/ml-lib2

As such I don’t see how the content interface can be used to solve this without rebuilding all the frameworks to happily live underneath some-new-ml-dir/usr/bin/ml-launch-thing and some-new-ml-dir/usr/lib/ml-lib1, but certainly happy to hear if I’m wrong.

I suppose perhaps I could have individual content interfaces for each individual file in the frameworks, but that doesn’t seem to be scalable when there are many such files and (potentially) many such frameworks.