Failing to build on core18


#1

Hi - I’m trying to move Node-RED snap to core18 - and it fails to build at all… can anyone give me a pointer to what else has broken in the yaml definition to fix this ? thanks

error logs here https://build.snapcraft.io/user/dceejay/nodered.snap/667180

the yaml is here


#2

Hi @davecj,

The actual error in your log is this:

Failed to load plugin: properties failed to load for node-red: Additional properties are not allowed ('npm-flags', 'node-engine', 'node-packages' were unexpected)

See https://snapcraft.io/docs/nodejs-plugin for updated documentation of the plugin. The above properties are deprecated, and assuming your project has a package.json file, you should do away with node-packages and npm-flags, and node-engine should be replaced by nodejs-version.

Hope this helps!


#3

so

    1. where does this overall package.json sit ? alongside the .yaml file ?
    1. how do I pass in the --unsafe-perm flag to the install ? as some of the packages call out to node-gyp to compile native libraries - they need that flag to avoid dropping privileges

#4

Hmm - I guess package.json doesn’t live next to the yaml then…

https://build.snapcraft.io/user/dceejay/nodered.snap/667267

The 'snap' directory is meant specifically for snapcraft, but it contains the following non-snapcraft-related paths, which is unsupported and will cause unexpected behavior:
- package.json

Where does it need to live ? It needs to pull in more than just the base node-red package .


#5

move it one level up …


#6

ok… and now another error to pick the bones out of… (this is getting beyond frustrating ! - nothing like a smooth migration )

https://build.snapcraft.io/user/dceejay/nodered.snap/667293

Running repo phase...
Cloning into 'node-red'...
[06/Sep/2019:10:11:14 +0000] "CONNECT github.com:443 HTTP/1.1" 200 132777 "-" "git/2.17.1"
Running pull phase...
Failed to load plugin: properties failed to load for node-red: 'source' is a required property
Build failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 267, in run
    self.pull()
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 233, in pull
    env=env)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 104, in run_build_command
    return self.backend.run(args, env=full_env, **kwargs)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/lxd.py", line 502, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['lxc', 'exec', 'lp-bionic-amd64', '--env', 'LANG=C.UTF-8', '--env', 'SHELL=/bin/sh', '--env', 'SNAPCRAFT_LOCAL_SOURCES=1', '--env', 'SNAPCRAFT_SETUP_CORE=1', '--env', 'SNAPCRAFT_BUILD_INFO=1', '--env', 'SNAPCRAFT_IMAGE_INFO={"build-request-id": "lp-51242789", "build-request-timestamp": "2019-09-06T10:09:36Z", "build_url": "https://launchpad.net/~build.snapcraft.io/+snap/b69a088aa70edd446e36c2072b30e222/+build/667293"}', '--env', 'SNAPCRAFT_BUILD_ENVIRONMENT=host', '--env', 'http_proxy=http://10.10.10.1:8222/', '--env', 'https_proxy=http://10.10.10.1:8222/', '--env', 'GIT_PROXY_COMMAND=/usr/local/bin/snap-git-proxy', '--', '/bin/sh', '-c', 'cd /build/node-red && linux64 snapcraft pull']' returned non-zero exit status 2
Revoking proxy token...
RUN: /usr/share/launchpad-buildd/bin/in-target scan-for-processes --backend=lxd --series=bionic --arch=amd64 SNAPBUILD-667293
Scanning for processes to kill in build SNAPBUILD-667293

#7

well, add a source: . entry … (to point to the local dir that has your package.json)


#8

It’s not clear why you’ve got separate node-red-git and node-red parts in your snapcraft.yaml. Surely you could just move the source: line from node-red-git to node-red and remove the node-red-git part all together (including the after: reference).


#9

and now

Building node-red 
yarn install v1.17.3
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 1.71s.
yarn pack v1.17.3
success Wrote tarball to "node-red-snap-0.20.8.tgz".
Done in 0.11s.
yarn install v1.17.3
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
[06/Sep/2019:10:32:05 +0000] "CONNECT github.com:443 HTTP/1.1" 200 6951 "-" "node-pre-gyp (v0.12.0, yarn/1.17.3 npm/? node/v10.16.3 linux x64)"
[06/Sep/2019:10:32:06 +0000] "CONNECT github-production-release-asset-2e65be.s3.amazonaws.com:443 HTTP/1.1" 200 30249 "https://github.com/kelektiv/node.bcrypt.js/releases/download/v3.0.6/bcrypt_lib-v3.0.6-node-v64-linux-x64-glibc.tar.gz" "node-pre-gyp (v0.12.0, yarn/1.17.3 npm/? node/v10.16.3 linux x64)"
Done in 5.88s.
/bin/sh: 33: npm: not found
/build/node-red/parts/node-red/npm/bin/yarn --proxy http://10.10.10.1:8222/ --https-proxy http://10.10.10.1:8222/ install --offline --prod
/build/node-red/parts/node-red/npm/bin/yarn --proxy http://10.10.10.1:8222/ --https-proxy http://10.10.10.1:8222/ pack --filename node-red-snap-0.20.8.tgz
/build/node-red/parts/node-red/npm/bin/yarn --proxy http://10.10.10.1:8222/ --https-proxy http://10.10.10.1:8222/ install --offline --prod
Failed to run 'override-build': Exit code was 127.
Build failed

err why is it using yarn when the docs say the default is npm ? https://snapcraft.io/docs/nodejs-plugin
(where it would also be useful to mention about source…)

@jamesh - no idea - that was the way it evolved as we had to build from source to get some of the required modules compiled for the different architectures… - again it may not be necessary now but nothing has indicated otherwise. (I don’t live in snap world so it’s a foreign land)


#10

So almost getting there (have forced npm and removed node-red-git as per suggestion) - https://build.snapcraft.io/user/dceejay/nodered.snap/667324
does the actual build - but then complains about npm - which must have been there to do the build…

I think this is where we now needed the --unsafe-perm option . So how do I pass that flag now in this new way of doing things ?

node-pre-gyp WARN Using request for node-pre-gyp https download 
[06/Sep/2019:10:54:27 +0000] "CONNECT github.com:443 HTTP/1.1" 200 6950 "-" "node-pre-gyp (v0.12.0, npm/6.9.0 node/v10.16.3 linux x64)"
[bcrypt] Success: "/build/node-red/parts/node-red/build/package/node_modules/bcrypt/lib/binding/bcrypt_lib.node" is installed via remote
[06/Sep/2019:10:54:28 +0000] "CONNECT github-production-release-asset-2e65be.s3.amazonaws.com:443 HTTP/1.1" 200 30220 "https://github.com/kelektiv/node.bcrypt.js/releases/download/v3.0.6/bcrypt_lib-v3.0.6-node-v64-linux-x64-glibc.tar.gz" "node-pre-gyp (v0.12.0, npm/6.9.0 node/v10.16.3 linux x64)"
npm notice created a lockfile as package-lock.json. You should commit this file.
added 412 packages from 362 contributors and audited 1648 packages in 20.611s
found 0 vulnerabilities

/bin/sh: 33: npm: not found
/build/node-red/parts/node-red/npm/bin/npm install --offline --prod
/build/node-red/parts/node-red/npm/bin/npm pack
/build/node-red/parts/node-red/npm/bin/npm install --offline --prod
Failed to run 'override-build': Exit code was 127.
Build failed

#11

seems you can solve that via an environment setting like in:


#12

Great - OK - must be close now… but still no npm… https://build.snapcraft.io/user/dceejay/nodered.snap/667485
npm is obviously installed with node.js - but how do I get it added as a visible command for the user from inside the snap ? I used to just add it as

npm:
    command: bin/npm
    plugs:
      - home
      - network

but that seemingly no longer works … how do I get it to stick around after the build step ?

And more generally - how do I know it should /bin/npm - ? how do you know what the path may be for any app in general ? it could be in /bin or /usr/bin or /sbin or /usr/local/bin etc… how can you know in advance ?


#13

Any more thoughts on this ? - Where does npm get installed in core18 ?
I really do want to get this working on 18.


#14

I’m looking into this migration process. We have an internal development meeting on Friday at which we’ll work through the issues you’ve uncovered. Thanks @davecj


#15

let me know when I can try again.
Thanks