I think supporting cache makes a great deal of sense. Performance is why it was originally created, no? 
It is perhaps premature, but on a related note - it would be really awesome to see what you have for a timeline chart / profile on an example snap’s startup, even if it’s mostly filled with unknown "?"s for the bulk of it! I am quite curious on the bigger picture.
I recently did some work in snapcraft to sort out library dependencies at snap creation so it can take into account content snaps (in order to warn the user if dependencies are missing, usually stage packages). It pretty much has to effectively do the same thing (compute/cache the dependencies by installing content snaps and look inside them to see what they’re providing to ensure the dependencies are satisfied).
Admittedly, you have got me thinking about this and the root of the problem. I’m sure this has been considered already, but I wonder if the implementation of (content) snaps would be greatly simplified if snap and content snaps were stacked (as in, “merged” in the overlayfs context). From what I’ve seen, most of the layout usage and LD_LIBRARY_PATH extension appears to be due to the unusual filesystem organization that starts with $SNAP for the snap in question, and then again with each content snap’s “target” mapping (e.g. $SNAP/gnome-platform for gnome-platform snap, $SNAP/data-dir/themes for gtk-3-themes, and so on.). I’m sure stacking the snaps as layers would have problems of its own, if it’s even possible, but the filesystem layout would be standard by most accounts and many of the fixups being done wouldn’t be required (for strict snaps)…? I know this is not strictly on topic, but I’ve gone on this tangent because I’m curious (if anyone feels like sharing sometime - I’ll buy the beers!).
Correct me if I am wrong, but I believe the conclusion of the above is that the use of LD_LIBRARY_PATH is expensive compared to ld.so.cache. Excessive usage of RUNPATH will incur a similar penalty. So regardless, I imagine minimizing the usage of LD_LIBRARY_PATH is going to significantly improve performance, even if we don’t eliminate it entirely.
To continue on your analysis, I took a look at snapcraft’s usage of LD_LIBRARY_PATH found in the extensions and binned them:
-
standard / traditional locations (found in /etc/ld.so.conf, /etc/ld.so.conf.d/*.conf) ./extensions/desktop/common/desktop-exports:append_dir LD_LIBRARY_PATH “$SNAP_DESKTOP_RUNTIME/lib/$ARCH” ./extensions/desktop/common/desktop-exports:append_dir LD_LIBRARY_PATH “$SNAP_DESKTOP_RUNTIME/usr/lib/$ARCH” ./extensions/desktop/common/desktop-exports:append_dir LD_LIBRARY_PATH “$SNAP_DESKTOP_RUNTIME/usr/lib” ./extensions/desktop/common/desktop-exports:append_dir LD_LIBRARY_PATH “$SNAP_DESKTOP_RUNTIME/lib”
-
mesa/egl (is this required/used anymore? or perhaps just 16.04?) ./extensions/desktop/common/desktop-exports:append_dir LD_LIBRARY_PATH “$SNAP_DESKTOP_RUNTIME/usr/lib/$ARCH/mesa” ./extensions/desktop/common/desktop-exports:append_dir LD_LIBRARY_PATH “$SNAP_DESKTOP_RUNTIME/usr/lib/$ARCH/mesa-egl” ./extensions/desktop/common/desktop-exports:append_dir LD_LIBRARY_PATH “$LIBGL_DRIVERS_PATH”
NOTES: Is this useful anymore after 16.04 [1]? Can it be limited to just “core”? Should it respect the host’s alternative? [2]
# ls -al /etc/ld.so.conf.d/x86_64-linux-gnu_EGL.conf
lrwxrwxrwx 1 root root 43 Dec 5 03:01 /etc/ld.so.conf.d/x86_64-linux-gnu_EGL.conf -> /etc/alternatives/x86_64-linux-gnu_egl_conf
[1] Bug #1609110 “Do we still need the mesa or mesa-egl directories ...” : Bugs : mesa package : Ubuntu [2] X/EGLDriverPackagingHOWTO - Ubuntu Wiki
-
snapd (not required for traditional apps?) ./extensions/desktop/common/desktop-exports:append_dir LD_LIBRARY_PATH “/var/lib/snapd/lib/gl”
-
libunity (traditional apps relies on RUNPATH set at build-time?) ./extensions/desktop/common/desktop-exports:append_dir LD_LIBRARY_PATH “$SNAP_DESKTOP_RUNTIME/usr/lib/$ARCH/libunity”
-
pulseaudio (traditional apps relies on RUNPATH set at build-time?) ./extensions/desktop/common/desktop-exports:append_dir LD_LIBRARY_PATH “$SNAP_DESKTOP_RUNTIME/usr/lib/$ARCH/pulseaudio”
-
testability (i’m not sure what this is?) ./extensions/desktop/common/desktop-exports:append_dir LD_LIBRARY_PATH “$SNAP/testability” ./extensions/desktop/common/desktop-exports:append_dir LD_LIBRARY_PATH “$SNAP/testability/$ARCH” ./extensions/desktop/common/desktop-exports:append_dir LD_LIBRARY_PATH “$SNAP/testability/$ARCH/mesa”
-
Qt5 (is this even used? I don’t see any matching path in the kde framework snaps?) ./extensions/desktop/kde-neon/launcher-specific:prepend_dir LD_LIBRARY_PATH “$SNAP_DESKTOP_RUNTIME/usr/lib/$ARCH/qt5/libs”
What if each snap can ship its own ld configs (i.e. /etc/ld.so.conf.d/*.conf) if it provides libraries that it wants to use or make available. I suppose it could be annotated as part of the snap.yaml instead, automatically even for the common cases. You can use those to generate the cache, with precedence: core < content < app. I think this approach an improvement over the current approach where snapcraft-preload/extensions make assumptions on the library directories provided by the content snaps (which, even if unlikely, may change over time). It can reduce some coupling.
I think if you are seeing your gains just by computing the cache every time, maybe consider adapting your scripts into snapcraft extensions’ desktop-launch command-chain wrapper to reduce/remove the usage of LD_LIBRARY_PATH and generate the cache after computing the set of search paths (or precomputing/re-assembling the ld config fragments)? Those are the apps that probably need it the most and probably wouldn’t take much effort beyond what you have already done 
. I think you should have all the required context there to eliminate much of LD_LIBRARY_PATH usage and I imagine the rest is a limited set for apps using environment to manipulate it (which should probably be discouraged)…
I want to add that I think it’s great you are taking on the performance work 
I certainly appreciate perceptible improvements to snaps’ performance!
Cheers! -Chris