Failing to build on core18

davecj · September 6, 2019, 7:44am

Hi - I’m trying to move Node-RED snap to core18 - and it fails to build at all… can anyone give me a pointer to what else has broken in the yaml definition to fix this ? thanks

error logs here https://build.snapcraft.io/user/dceejay/nodered.snap/667180

the yaml is here

github.com

dceejay/nodered.snap/blob/master/snap/snapcraft.yaml

name: node-red
adopt-info: node-red
summary: Low-code programming for event-driven applications
description: Node-RED is a flow based programming application for wiring together hardware devices, APIs and online services in new and interesting ways. For more information see http://nodered.org
confinement: strict
grade: stable
base: core18

# Limit architectures as ppcel64 doesn't build currently
architectures:
  - build-on: amd64
  - build-on: armhf
  - build-on: arm64
  - build-on: s390x

apps:
  node-red:
    command: bin/startNR
    daemon: simple
    restart-condition: on-failure

This file has been truncated. show original

Saviq · September 6, 2019, 8:09am

Hi @davecj,

The actual error in your log is this:

Failed to load plugin: properties failed to load for node-red: Additional properties are not allowed ('npm-flags', 'node-engine', 'node-packages' were unexpected)

See https://snapcraft.io/docs/nodejs-plugin for updated documentation of the plugin. The above properties are deprecated, and assuming your project has a package.json file, you should do away with node-packages and npm-flags, and node-engine should be replaced by nodejs-version.

Hope this helps!

davecj · September 6, 2019, 9:07am

so

1. where does this overall package.json sit ? alongside the .yaml file ?
1. how do I pass in the --unsafe-perm flag to the install ? as some of the packages call out to node-gyp to compile native libraries - they need that flag to avoid dropping privileges

davecj · September 6, 2019, 9:22am

Hmm - I guess package.json doesn’t live next to the yaml then…

The 'snap' directory is meant specifically for snapcraft, but it contains the following non-snapcraft-related paths, which is unsupported and will cause unexpected behavior:
- package.json

Where does it need to live ? It needs to pull in more than just the base node-red package .

ogra · September 6, 2019, 10:02am

move it one level up …

davecj · September 6, 2019, 10:15am

ok… and now another error to pick the bones out of… (this is getting beyond frustrating ! - nothing like a smooth migration )

Running repo phase...
Cloning into 'node-red'...
[06/Sep/2019:10:11:14 +0000] "CONNECT github.com:443 HTTP/1.1" 200 132777 "-" "git/2.17.1"
Running pull phase...
Failed to load plugin: properties failed to load for node-red: 'source' is a required property
Build failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 267, in run
    self.pull()
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 233, in pull
    env=env)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/build_snap.py", line 104, in run_build_command
    return self.backend.run(args, env=full_env, **kwargs)
  File "/usr/lib/python2.7/dist-packages/lpbuildd/target/lxd.py", line 502, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['lxc', 'exec', 'lp-bionic-amd64', '--env', 'LANG=C.UTF-8', '--env', 'SHELL=/bin/sh', '--env', 'SNAPCRAFT_LOCAL_SOURCES=1', '--env', 'SNAPCRAFT_SETUP_CORE=1', '--env', 'SNAPCRAFT_BUILD_INFO=1', '--env', 'SNAPCRAFT_IMAGE_INFO={"build-request-id": "lp-51242789", "build-request-timestamp": "2019-09-06T10:09:36Z", "build_url": "https://launchpad.net/~build.snapcraft.io/+snap/b69a088aa70edd446e36c2072b30e222/+build/667293"}', '--env', 'SNAPCRAFT_BUILD_ENVIRONMENT=host', '--env', 'http_proxy=http://10.10.10.1:8222/', '--env', 'https_proxy=http://10.10.10.1:8222/', '--env', 'GIT_PROXY_COMMAND=/usr/local/bin/snap-git-proxy', '--', '/bin/sh', '-c', 'cd /build/node-red && linux64 snapcraft pull']' returned non-zero exit status 2
Revoking proxy token...
RUN: /usr/share/launchpad-buildd/bin/in-target scan-for-processes --backend=lxd --series=bionic --arch=amd64 SNAPBUILD-667293
Scanning for processes to kill in build SNAPBUILD-667293

ogra · September 6, 2019, 10:21am

well, add a source: . entry … (to point to the local dir that has your package.json)

jamesh · September 6, 2019, 10:31am

It’s not clear why you’ve got separate node-red-git and node-red parts in your snapcraft.yaml. Surely you could just move the source: line from node-red-git to node-red and remove the node-red-git part all together (including the after: reference).

davecj · September 6, 2019, 10:38am

and now

Building node-red 
yarn install v1.17.3
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 1.71s.
yarn pack v1.17.3
success Wrote tarball to "node-red-snap-0.20.8.tgz".
Done in 0.11s.
yarn install v1.17.3
[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
[06/Sep/2019:10:32:05 +0000] "CONNECT github.com:443 HTTP/1.1" 200 6951 "-" "node-pre-gyp (v0.12.0, yarn/1.17.3 npm/? node/v10.16.3 linux x64)"
[06/Sep/2019:10:32:06 +0000] "CONNECT github-production-release-asset-2e65be.s3.amazonaws.com:443 HTTP/1.1" 200 30249 "https://github.com/kelektiv/node.bcrypt.js/releases/download/v3.0.6/bcrypt_lib-v3.0.6-node-v64-linux-x64-glibc.tar.gz" "node-pre-gyp (v0.12.0, yarn/1.17.3 npm/? node/v10.16.3 linux x64)"
Done in 5.88s.
/bin/sh: 33: npm: not found
/build/node-red/parts/node-red/npm/bin/yarn --proxy http://10.10.10.1:8222/ --https-proxy http://10.10.10.1:8222/ install --offline --prod
/build/node-red/parts/node-red/npm/bin/yarn --proxy http://10.10.10.1:8222/ --https-proxy http://10.10.10.1:8222/ pack --filename node-red-snap-0.20.8.tgz
/build/node-red/parts/node-red/npm/bin/yarn --proxy http://10.10.10.1:8222/ --https-proxy http://10.10.10.1:8222/ install --offline --prod
Failed to run 'override-build': Exit code was 127.
Build failed

err why is it using yarn when the docs say the default is npm ? https://snapcraft.io/docs/nodejs-plugin
(where it would also be useful to mention about source…)

@jamesh - no idea - that was the way it evolved as we had to build from source to get some of the required modules compiled for the different architectures… - again it may not be necessary now but nothing has indicated otherwise. (I don’t live in snap world so it’s a foreign land)

davecj · September 6, 2019, 11:25am

So almost getting there (have forced npm and removed node-red-git as per suggestion) - https://build.snapcraft.io/user/dceejay/nodered.snap/667324
does the actual build - but then complains about npm - which must have been there to do the build…

I think this is where we now needed the --unsafe-perm option . So how do I pass that flag now in this new way of doing things ?

node-pre-gyp WARN Using request for node-pre-gyp https download 
[06/Sep/2019:10:54:27 +0000] "CONNECT github.com:443 HTTP/1.1" 200 6950 "-" "node-pre-gyp (v0.12.0, npm/6.9.0 node/v10.16.3 linux x64)"
[bcrypt] Success: "/build/node-red/parts/node-red/build/package/node_modules/bcrypt/lib/binding/bcrypt_lib.node" is installed via remote
[06/Sep/2019:10:54:28 +0000] "CONNECT github-production-release-asset-2e65be.s3.amazonaws.com:443 HTTP/1.1" 200 30220 "https://github.com/kelektiv/node.bcrypt.js/releases/download/v3.0.6/bcrypt_lib-v3.0.6-node-v64-linux-x64-glibc.tar.gz" "node-pre-gyp (v0.12.0, npm/6.9.0 node/v10.16.3 linux x64)"
npm notice created a lockfile as package-lock.json. You should commit this file.
added 412 packages from 362 contributors and audited 1648 packages in 20.611s
found 0 vulnerabilities

/bin/sh: 33: npm: not found
/build/node-red/parts/node-red/npm/bin/npm install --offline --prod
/build/node-red/parts/node-red/npm/bin/npm pack
/build/node-red/parts/node-red/npm/bin/npm install --offline --prod
Failed to run 'override-build': Exit code was 127.
Build failed

ogra · September 6, 2019, 11:49am

seems you can solve that via an environment setting like in:

github.com

jehon/kiosk/blob/cb88b9aa1dc502ef1d8d92089551bf2a7254b92e/snap/snapcraft.yaml#L11


name: kiosk
base: core
version: '1'
summary: A kiosk for your home
description: |
  A kiosk displaying all your local/samba pictures on a screen
grade: stable
confinement: strict
# https://git.launchpad.net/~ondrak/ondras-snaps/+git/homebridge-snap/tree/snap/snapcraft.yaml
environment:
    npm_config_unsafe_perm: "true"
    NODE_ENV: "production"
apps:
  kiosk:
    command: npm start
    plugs: 
      - network-bind
      - browser-sandbox
      - opengl

davecj · September 6, 2019, 2:11pm

Great - OK - must be close now… but still no npm… https://build.snapcraft.io/user/dceejay/nodered.snap/667485
npm is obviously installed with node.js - but how do I get it added as a visible command for the user from inside the snap ? I used to just add it as

npm:
    command: bin/npm
    plugs:
      - home
      - network

but that seemingly no longer works … how do I get it to stick around after the build step ?

And more generally - how do I know it should /bin/npm - ? how do you know what the path may be for any app in general ? it could be in /bin or /usr/bin or /sbin or /usr/local/bin etc… how can you know in advance ?

davecj · September 9, 2019, 10:16pm

Any more thoughts on this ? - Where does npm get installed in core18 ?
I really do want to get this working on 18.

popey · September 11, 2019, 12:16pm

I’m looking into this migration process. We have an internal development meeting on Friday at which we’ll work through the issues you’ve uncovered. Thanks @davecj

davecj · September 12, 2019, 3:59pm

let me know when I can try again.
Thanks

popey · September 27, 2019, 11:25am

Just to wrap this thread up, I provided a PR to the upstream project and it’s working now.