Failing to build on core18

Hi - I’m trying to move Node-RED snap to core18 - and it fails to build at all… can anyone give me a pointer to what else has broken in the yaml definition to fix this ? thanks

error logs here https://build.snapcraft.io/user/dceejay/nodered.snap/667180

the yaml is here

Hi @davecj,

The actual error in your log is this:

Failed to load plugin: properties failed to load for node-red: Additional properties are not allowed ('npm-flags', 'node-engine', 'node-packages' were unexpected)

See https://snapcraft.io/docs/nodejs-plugin for updated documentation of the plugin. The above properties are deprecated, and assuming your project has a package.json file, you should do away with node-packages and npm-flags, and node-engine should be replaced by nodejs-version.

Hope this helps!

so

    1. where does this overall package.json sit ? alongside the .yaml file ?
    1. how do I pass in the --unsafe-perm flag to the install ? as some of the packages call out to node-gyp to compile native libraries - they need that flag to avoid dropping privileges

Hmm - I guess package.json doesn’t live next to the yaml then…

https://build.snapcraft.io/user/dceejay/nodered.snap/667267

The 'snap' directory is meant specifically for snapcraft, but it contains the following non-snapcraft-related paths, which is unsupported and will cause unexpected behavior:
- package.json

Where does it need to live ? It needs to pull in more than just the base node-red package .

move it one level up …

ok… and now another error to pick the bones out of… (this is getting beyond frustrating ! - nothing like a smooth migration )

https://build.snapcraft.io/user/dceejay/nodered.snap/667293

Running repo phase...
Cloning into 'node-red'...
[06/Sep/2019:10:11:14 +0000] "CONNECT github.com:443 HTTP/1.1" 200 132777 "-" "git/2.17.1"
Running pull phase...
Failed to load plugin: properties failed to load for node-red: 'source' is a required property
Build failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 267, in run
    self.pull()
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 233, in pull
    env=env)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 104, in run_build_command
    return self.backend.run(args, env=full_env, **kwargs)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/lxd.py", line 502, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['lxc', 'exec', 'lp-bionic-amd64', '--env', 'LANG=C.UTF-8', '--env', 'SHELL=/bin/sh', '--env', 'SNAPCRAFT_LOCAL_SOURCES=1', '--env', 'SNAPCRAFT_SETUP_CORE=1', '--env', 'SNAPCRAFT_BUILD_INFO=1', '--env', 'SNAPCRAFT_IMAGE_INFO={"build-request-id": "lp-51242789", "build-request-timestamp": "2019-09-06T10:09:36Z", "build_url": "https://launchpad.net/~build.snapcraft.io/+snap/b69a088aa70edd446e36c2072b30e222/+build/667293"}', '--env', 'SNAPCRAFT_BUILD_ENVIRONMENT=host', '--env', 'http_proxy=http://10.10.10.1:8222/', '--env', 'https_proxy=http://10.10.10.1:8222/', '--env', 'GIT_PROXY_COMMAND=/usr/local/bin/snap-git-proxy', '--', '/bin/sh', '-c', 'cd /build/node-red && linux64 snapcraft pull']' returned non-zero exit status 2
Revoking proxy token...
RUN: /usr/share/launchpad-buildd/bin/in-target scan-for-processes --backend=lxd --series=bionic --arch=amd64 SNAPBUILD-667293
Scanning for processes to kill in build SNAPBUILD-667293

well, add a source: . entry … (to point to the local dir that has your package.json)

It’s not clear why you’ve got separate node-red-git and node-red parts in your snapcraft.yaml. Surely you could just move the source: line from node-red-git to node-red and remove the node-red-git part all together (including the after: reference).

and now

Building node-red 
yarn install v1.17.3
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 1.71s.
yarn pack v1.17.3
success Wrote tarball to "node-red-snap-0.20.8.tgz".
Done in 0.11s.
yarn install v1.17.3
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
[06/Sep/2019:10:32:05 +0000] "CONNECT github.com:443 HTTP/1.1" 200 6951 "-" "node-pre-gyp (v0.12.0, yarn/1.17.3 npm/? node/v10.16.3 linux x64)"
[06/Sep/2019:10:32:06 +0000] "CONNECT github-production-release-asset-2e65be.s3.amazonaws.com:443 HTTP/1.1" 200 30249 "https://github.com/kelektiv/node.bcrypt.js/releases/download/v3.0.6/bcrypt_lib-v3.0.6-node-v64-linux-x64-glibc.tar.gz" "node-pre-gyp (v0.12.0, yarn/1.17.3 npm/? node/v10.16.3 linux x64)"
Done in 5.88s.
/bin/sh: 33: npm: not found
/build/node-red/parts/node-red/npm/bin/yarn --proxy http://10.10.10.1:8222/ --https-proxy http://10.10.10.1:8222/ install --offline --prod
/build/node-red/parts/node-red/npm/bin/yarn --proxy http://10.10.10.1:8222/ --https-proxy http://10.10.10.1:8222/ pack --filename node-red-snap-0.20.8.tgz
/build/node-red/parts/node-red/npm/bin/yarn --proxy http://10.10.10.1:8222/ --https-proxy http://10.10.10.1:8222/ install --offline --prod
Failed to run 'override-build': Exit code was 127.
Build failed

err why is it using yarn when the docs say the default is npm ? https://snapcraft.io/docs/nodejs-plugin
(where it would also be useful to mention about source…)

@jamesh - no idea - that was the way it evolved as we had to build from source to get some of the required modules compiled for the different architectures… - again it may not be necessary now but nothing has indicated otherwise. (I don’t live in snap world so it’s a foreign land)

So almost getting there (have forced npm and removed node-red-git as per suggestion) - https://build.snapcraft.io/user/dceejay/nodered.snap/667324
does the actual build - but then complains about npm - which must have been there to do the build…

I think this is where we now needed the --unsafe-perm option . So how do I pass that flag now in this new way of doing things ?

node-pre-gyp WARN Using request for node-pre-gyp https download 
[06/Sep/2019:10:54:27 +0000] "CONNECT github.com:443 HTTP/1.1" 200 6950 "-" "node-pre-gyp (v0.12.0, npm/6.9.0 node/v10.16.3 linux x64)"
[bcrypt] Success: "/build/node-red/parts/node-red/build/package/node_modules/bcrypt/lib/binding/bcrypt_lib.node" is installed via remote
[06/Sep/2019:10:54:28 +0000] "CONNECT github-production-release-asset-2e65be.s3.amazonaws.com:443 HTTP/1.1" 200 30220 "https://github.com/kelektiv/node.bcrypt.js/releases/download/v3.0.6/bcrypt_lib-v3.0.6-node-v64-linux-x64-glibc.tar.gz" "node-pre-gyp (v0.12.0, npm/6.9.0 node/v10.16.3 linux x64)"
npm notice created a lockfile as package-lock.json. You should commit this file.
added 412 packages from 362 contributors and audited 1648 packages in 20.611s
found 0 vulnerabilities

/bin/sh: 33: npm: not found
/build/node-red/parts/node-red/npm/bin/npm install --offline --prod
/build/node-red/parts/node-red/npm/bin/npm pack
/build/node-red/parts/node-red/npm/bin/npm install --offline --prod
Failed to run 'override-build': Exit code was 127.
Build failed

seems you can solve that via an environment setting like in:

Great - OK - must be close now… but still no npm… https://build.snapcraft.io/user/dceejay/nodered.snap/667485
npm is obviously installed with node.js - but how do I get it added as a visible command for the user from inside the snap ? I used to just add it as

npm:
    command: bin/npm
    plugs:
      - home
      - network

but that seemingly no longer works … how do I get it to stick around after the build step ?

And more generally - how do I know it should /bin/npm - ? how do you know what the path may be for any app in general ? it could be in /bin or /usr/bin or /sbin or /usr/local/bin etc… how can you know in advance ?

Any more thoughts on this ? - Where does npm get installed in core18 ?
I really do want to get this working on 18.

I’m looking into this migration process. We have an internal development meeting on Friday at which we’ll work through the issues you’ve uncovered. Thanks @davecj

let me know when I can try again.
Thanks

Just to wrap this thread up, I provided a PR to the upstream project and it’s working now.