Npm install fails with sqlite3@5.0.0 in dependencies

I am attempting to package an application which has sqlite3@5.0.0 as a dependency. When running snapcraft I always get a failure during the install phase.

+ npm install -g --prefix /root/parts/cli-worm/install cli-worm-0.2.0.tgz
npm WARN deprecated circular-json@0.5.9: CircularJSON is in maintenance only, flatted is its successor.
npm WARN deprecated request@2.88.2: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated har-validator@5.1.5: this library is no longer supported
/root/parts/cli-worm/install/bin/cli-worm -> /root/parts/cli-worm/install/lib/node_modules/cli-worm/index.js

> sqlite3@5.0.0 install /root/parts/cli-worm/install/lib/node_modules/cli-worm/node_modules/sqlite3
> node-pre-gyp install --fallback-to-build

sh: 1: node-pre-gyp: Permission denied
npm ERR! code ELIFECYCLE
npm ERR! syscall spawn
npm ERR! file sh
npm ERR! errno ENOENT
npm ERR! sqlite3@5.0.0 install: `node-pre-gyp install --fallback-to-build`
npm ERR! spawn ENOENT
npm ERR! 
npm ERR! Failed at the sqlite3@5.0.0 install script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-11-28T20_51_35_266Z-debug.log
Failed to build 'cli-worm'.

I do not have any issues building this package outside of snapcraft, but I have found some blog posts where others have seen this same error. The suggested work around for folks who are seeing this is often to:

npm config set user 0
npm config set unsafe-perm true

https://github.com/mapbox/node-sqlite3/issues/972#issuecomment-531482332

Indeed if I run with the --debug flag, then once I get the failure I can use the cli to set the node paths up, run the npm config commands above and I am able to build the project inside the lxde container.

What I don’t understand is how, within the context of the npm plugin to make it such that the build is ran with these config values set. I can override the build step, but the plugin doesn’t download node until during the build stage; so, at that point there is no npm to configure. I have tried overriding the build and doing

env npm_config_user=0
env npm_config_unsafe_perm=true
snapcraftctl build

That also has no effect.

Here is my current yaml

name:  cli-worm
base: core20
version: '0.2.0'
summary:  an epub reader for the command line
description: |
  A simple command line ncurses style interface for reading epubs

grade: stable
confinement: strict
apps:
  cli-worm:
    command: env LC_ALL=C.UTF-8 cli-worm # set LC_ALL env value when running to properly display utf8 characters
    plugs:
      - home
      - removable-media

parts:
  cli-worm:
    source: .
    plugin: npm
    npm-node-version: 14.15.1
    override-build: |
      echo 'build start'
      pwd
      echo 'npm_config_unsafe_perm=true' >> ~/.npmrc
      echo 'npm_config_user=0' >> ~/.npmrc
      env npm_config_user=0
      env npm_config_unsafe_perm=true
      snapcraftctl build
    stage-packages:
      - w3m # used to render xhtml to cli safe text
      - gcc # trying to make sqllite compile correction on npm install
      - g++ # trying make sqllite compile correction on npm install
      - libsqlite3-0

and package.json

{
  "name": "cli-worm",
  "version": "0.2.0",
  "description": "an epub reader for the command line",
  "main": "index.js",
  "bin": {
    "cli-worm": "./index.js"
  },
  "repository": "chriswininger/cli-worm",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [
    "epub",
    "e-reader",
    "ebooks",
    "cli",
    "worm",
    "cli-worm"
  ],
  "author": "Chris Wininger",
  "license": "MIT",
  "dependencies": {
    "async": "^3.1.1",
    "blessed": "^0.1.81",
    "dotenv": "^5.0.0",
    "log4js": "^3.0.6",
    "mkdirp": "^1.0.4",
    "sqlite": "^4.0.17",
    "sqlite3": "5.0.0",
    "tmp": "0.0.33",
    "yauzl": "2.10.0",
    "xml-parser": "^1.2.1"
  }
}

What are my options here? Is there a way to control what flags the npm plugin uses when executing npm install? Why do I need these to to run npm install within the context of the snapcraft build process in the first place?

Any help would be much appreciated.

I should also say, I’m on snapcraft, version 4.4.2

My host OS is Pop 20.10. I have also tried from within a VM running Ubuntu 20.04

And this comes from the .npm debug logs:

3760 info lifecycle sqlite3@5.0.0~install: sqlite3@5.0.0
3761 verbose lifecycle sqlite3@5.0.0~install: unsafe-perm in lifecycle false
3762 verbose lifecycle sqlite3@5.0.0~install: PATH: /root/parts/cli-worm/install/lib/node_modules/npm/node_modules/npm-lifecycle/node-gyp-bin:/root/parts/cli-worm/install/lib/node_modules/cli-worm/node_modules/sqlite3/node_modules/.bin:/root/parts/cli-worm/install/lib/node_modules/cli-worm/node_modules/.bin:/root/parts/cli-worm/install/lib/node_modules/.bin:/root/parts/cli-worm/install/bin:/root/parts/cli-worm/install/usr/bin:/snap/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
3763 verbose lifecycle sqlite3@5.0.0~install: CWD: /root/parts/cli-worm/install/lib/node_modules/cli-worm/node_modules/sqlite3
3764 silly lifecycle sqlite3@5.0.0~install: Args: [ '-c', 'node-pre-gyp install --fallback-to-build' ]
3765 info lifecycle sqlite3@5.0.0~install: Failed to exec install script
3766 timing action:install Completed in 58ms
3767 verbose unlock done using /root/.npm/_locks/staging-4cc85e191ebc6079.lock for /root/parts/cli-worm/install/lib/node_modules/.staging
3768 timing stage:rollbackFailedOptional Completed in 167ms
3769 timing stage:runTopLevelLifecycles Completed in 8390ms
3770 verbose stack Error: sqlite3@5.0.0 install: `node-pre-gyp install --fallback-to-build`
3770 verbose stack spawn ENOENT
3770 verbose stack     at ChildProcess.<anonymous> (/root/parts/cli-worm/install/lib/node_modules/npm/node_modules/npm-lifecycle/lib/spawn.js:48:18)
3770 verbose stack     at ChildProcess.emit (events.js:315:20)
3770 verbose stack     at maybeClose (internal/child_process.js:1048:16)
3770 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)
3771 verbose pkgid sqlite3@5.0.0
3772 verbose cwd /root/parts/cli-worm/build
3773 verbose Linux 5.8.0-7630-generic
3774 verbose argv "/root/parts/cli-worm/install/bin/node" "/root/parts/cli-worm/install/bin/npm" "install" "-g" "--prefix" "/root/parts/cli-worm/install" "cli-worm-0.2.0.tgz"
3775 verbose node v14.15.1
3776 verbose npm  v6.14.8
3777 error code ELIFECYCLE
3778 error syscall spawn
3779 error file sh
3780 error errno ENOENT
3781 error sqlite3@5.0.0 install: `node-pre-gyp install --fallback-to-build`
3781 error spawn ENOENT
3782 error Failed at the sqlite3@5.0.0 install script.
3782 error This is probably not a problem with npm. There is likely additional logging output above.
3783 verbose exit [ 1, true ]
snapcraft-cli-worm # 

In my latest experiment I tried adding node-pre-gyp to my stage-packages. This did change the error message slightly to:

> sqlite3@5.0.0 install /root/parts/cli-worm/install/lib/node_modules/cli-worm/node_modules/sqlite3
> node-pre-gyp install --fallback-to-build

fs.js:114
    throw err;
    ^

Error: EACCES: permission denied, open '/root/parts/cli-worm/install/lib/node_modules/cli-worm/node_modules/sqlite3/package.json'
    at Object.openSync (fs.js:443:3)
    at Object.readFileSync (fs.js:343:35)
    at Run.parseOpts [as parseArgv] (/usr/lib/nodejs/node-pre-gyp/lib/node-pre-gyp.js:136:36)
    at Object.<anonymous> (/usr/lib/nodejs/node-pre-gyp/bin/node-pre-gyp:24:6)
    at Module._compile (internal/modules/cjs/loader.js:778:30)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)

I’m not sure if that’s a good or a bad sign :shrug:

I figured a kind of work around; though I don’t like it. It feels like it’s bypassing a lot of what the plugin is supposed to do and I suspect it may be brittle, but here it is:

name:  cli-worm
base: core20
version: '0.2.0'
summary:  an epub reader for the command line
description: |
  A simple command line ncurses style interface for reading epubs

grade: stable
confinement: strict
apps:
  cli-worm:
    command: bin/cli-worm
    plugs:
      - home
      - removable-media

parts:
  cli-worm:
    source: .
    plugin: npm
    npm-node-version: 14.8.0
    override-build: |
      echo 'build start'
      echo "https://nodejs.org/dist/v14.8.0/node-v14.8.0-linux-$ARCH.tar.gz"
      curl -s https://nodejs.org/dist/v14.8.0/node-v14.8.0-linux-x64.tar.gz | tar xzf - -C "$SNAPCRAFT_PART_INSTALL" --strip-components=1
            npm config set user 0
            npm config set unsafe-perm true
            npm install
            npm install -g --prefix "${SNAPCRAFT_PART_INSTALL}" $(npm pack . | tail -1)
      echo 'build end'
    stage-packages:
      - w3m # used to render xhtml to cli safe text
      - node-pre-gyp

The main difference is that I’m no longer invoking snapcraftctl build at all. I’m instead dowloading and invoking the correct version of node myself, allowing me to use node config ahead of running npm install.

Ideally I’d like to not have to subvert the entire build stage in order to run

 npm config set user 0
 npm config set unsafe-perm true

or event better I’d like a way to build the snap such that I don’t have to alter the npm config at all. I’m guessing this has something to do with what user the snapcraft scripts are using to invoke the build, but there is a lot a I don’t fully understand about this process.

From what I remember, the unsafe-perm permission is already set in the plugin, but it only covers the exact portion that snapcraft itself handles (I.E, either when not using override-build, or the specific snapcraftctl build command and not anything around it).

Anyway, my guess/recommendations would be to add the following as build-environment parameters

parts:
  cli-worm:
    build-environment:
      - npm_config_unsafe_perm: 'true'
      - SUDO_UID: '0'
      - SUDO_GID: '0'
      - SUDO_USER: 'root'
      - npm_config_prefer_offline: 'true'

It’s been a while so I can’t remember 100%, but there was 3 major problems for me using NPM

  1. The unsafe perm flag, to allow NPM to run as root

  2. Some modules drop into a subshell to execute other commands (E.G, git). When they drop into the subshell, they try running as a lower priveledged user, they end up failing since they can’t modify the build. This is what SUDO_* variables fix, and instruct any subshells/forks/etc into running as root too, and I assume is what’s causing your actual problem above

3778 error syscall spawn
3779 error file sh
  1. NPM can get picky and demand packages to be cached offline, and won’t actually grab them if they don’t already exist, so prefer_config_offline nudges them into prefering locally cached modules but not straight up failing if they don’t have them (Since Snapcraft environments are cleaned regularly so failing like that doesn’t make sense).

OK, interesting. I bet it does have something to do with the subshell running as a lower user. I’ll try setting some of those flags and see if it helps.

Does anyone have any luck with this?

I’m still not able to produce a snap with this plugin.

Here is a post I just wrote on my details on this: Error building with npm plugin and lxd

Hope for some help.