Yes it is 
And all nessecary bits are in the snap folder, but make (or maybe node-gyp) is trying to fetch the files from /user/lib in root where not all files are accessable (unless in classic)
I’m not a node person so I cannot tell you more but the error message above is a clear indication of your snap trying to use the linker
I realize that is unconvetional, but I still need to compile the files at runtime, I’d have thought using LD_LIBRARY_PATH would have solved it, but regardless of what I do (or try to do), it still tries to get the libmvec_nonshared.a file from /user/lib
LD_LIBRARY_PATH is for the dynamic linker. Why do you think you need to do this at runtime? Dragging the whole linker, compiler, headers and development libraries into this is going to be messy.
The snap in an IoT application running micro services that may change over time. These services are normally just JavaScript’s (npm packages) where we don’t have issues, but in some cases they are bundled with C/C++ files and needs compiling and this is where we run into problems
dynamically building bits of your snap is not a typical use case so you will need to add a lot of manual work …
… to get the environment vars right (i.e. have LD_LIBRARY_PATH point to $SNAP/usr/lib ) you will likely have to either create app entries for each of your binaries used to compile the npm modules (this automatically creates wrappers for them using the right LD_LIBRARY_PATH), or if you do not want to expose them to the outside world (assuming you dont want a <snapname>.gcc ), move all the binaries from <binary> to <binary>.real and create wrappers called <binary> that sets the variables and in the end executes the .real file…
this will be a hell lot of work i suppose…
additionally you will have to make sure that your builds happen in $SNAP_DATA (or any other writable place the snap has) and that your node binaries have the target install dir (which has to be writeable too) added to their search path …
If there’s a way to deploy your app and then have it snapped then do that. Building this at runtime is really more complex and is IMO against the spirit of the snap package where everyone gets the same bits.
Thanks for a great reply, and yes all "dynamic part of the app is written to $SNAP_DATA. My .yaml includes stage- and build packages (python2.7, build-essential) along with node packages (node-gyp@latest). They are all installed, except for necessary env paths.
As the issues seems to relate to gcc/make, is there a special package I should/could add?
Again, many thanks
I can’t use the snap infrastructure as each instance of the snap may have different Micro Services running 
But whatever those are you can compile all of them ahead of time and just enable a subset right? In the end node (or whatever else you are using) will dlopen a .so file that you need to compile ahead of time.
if there are dynamic bits needed snap wont be in your way.
while this is definitely massively complex, you can surely do it and it is not different to i.e. dynamically loading random app plugins for any desktop app …
so i wouldnt say it is “against the spirit” … quite the opposite, it shows that snap is definitely flexible enough to allow it if you put enough work in …
1 Like
@ogra …as the issues seems to relate to gcc/make, is there a special package I should/could add that would also add necessary env variables?
you would have to add all packages your plugins require to build to “stage-packages”. then you would need to add a wrapper for each and every binary these packages include …
then patch the build system (probably even dynamically at runtime if the build scripts/Makefiles come along with the plugin) to:
- use a writable dir of your snap to build in
- install the resulting binaries in a writable place
and then make sure your binaries that make use of them know where to find them (by adjusting the search paths).
Some more findings…
We have been able to reproduce the issue using the simplest of settings and copied the addon hello-world from node.js (https://nodejs.org/docs/latest-v7.x/api/addons.html)
We get the same error:
make: Entering directory '/root/snap/mysnap/x1/demo/build'
SOLINK_MODULE(target) Release/obj.target/addon.node
/snap/mysnap/x1/usr/bin/ld: cannot find /usr/lib/x86_64-linux-gnu/libmvec_nonshared.a
So we search the entire $SNAP structure for libmvec_nonshared.a and found it in $SNAP/usr/lib/x86_64-linux-gnu/libm.so
The content of the file looks like this:
OUTPUT_FORMAT(elf64-x86-64)
GROUP ( /lib/x86_64-linux-gnu/libm.so.6 AS_NEEDED ( /usr/lib/x86_64-linux-gnu/libmvec_nonshared.a /lib/x86_64-linux-gnu/libmvec.so.1 ) )
As the path is not relative the snap (although included in the $SNAP/usr/lib/x86_64-linux-gnu folder), the snap can not access the file (the file is there but the snap is not granted privileges).
Question: Is there a way to either affect this behavior in the snapcraft process (for instance through letting the snap think it’s living in the root), or ultimately give the snap full access to the /usr directory? Also, when you (@ogra) mention “wrapper”, what would that be?
BTW I’ve submitted and issue here.
I renamed the thread to match the problem description more accurately. I still think going this way is harder than building this snap ahead of time.
well, wrappers to set a poper LD_LIBRAY_PATH around anything that might need to access libs … or in the quoted case above a hack to the makefile to prefix any /usr/lib path with $SNAP …
@zyga-snapd Again, I can’t as every device may end up having different services. This is btw a very common scenario for IoT…
Can you please explain why is it difficult to bundle all the binary modules any set of services needs? Unless your snap is a “node as a platform” and you want to deploy on top. In such case I’d say you should use node as a part and still build dedicated snaps that do specific things and are not a generic deployment platform.
As an analogy from distribution packaging: snap that allows you to install ubuntu and “apt install anything” is not a sane snap. Snap that bundles a specific package is sane as it is well-defined and scoped.
CC: @sergiusens any advice?
First of all, we are targeting hundrads of thousands and even millions of devices. Not every device is unique, but many are and will change over time. This should give you an idea of the management tooling required. We have such device management tooling today, which by the way supports many platforms apart from snap.
Node.js devices are all running Services developed using JavaScript which in turn may use npm packages. These are not part of the Snap, and are deployed at runtime from the Device Management tool. In most situations this is not a problem using Snap as all services are deployed to the $SNAP_DATA folder.
-However… some npm packages, such as SerialPort comes with C/C++ files that gets compiled at runtime…and then fails.
The problem is that some lib* files in the $SNAP directory are referencing other files in the /usr directory and don’t have access to them.
For more details: https://github.com/snapcore/snapcraft/issues/1605
Hi @wmmihaa
This could be to your interest
https://git.launchpad.net/~ondrak/ondras-snaps/+git/homebridge-snap/tree/snapcraft.yaml
it’s recipe for npm project, which allows run time npm module installation. Trick is to rebuild glibc from source and tell it it will live in different location. As core snap does not include all needed header files and dev dependencies needed for building, you need to include libc6-dev in your snap anyway. This will just build it from source, rather than using stage package , which allows us to alter prefix. With this trick npm install works just fine. I added also git, as some npm modules are installed from git.
Also remember to set all needed path correctly, you can see that here:
https://git.launchpad.net/~ondrak/ondras-snaps/+git/homebridge-snap/tree/scripts/bin/env-setup
You will notice that on permission side I have used docker interface. This is just to overcome lack of correct interface at the moment. I will try to work with @jdstrand to come up with more appropriate interface to cover this use case.
hope this will unblock your use case