Snapctl not found in base: core18 snaps' hooks


#1

Hi all,
I’m having difficulty getting some of my snaps to work with base: core18 that use snapctl in the hooks. For example, this simple snap:

name: snapctl-core18
version: '0.1'
summary: Test snapctl on core18
description: Test snapctl on core18
grade: stable
confinement: strict
passthrough:
  base: core18

apps:
  service:
    command: echo hello-world
    daemon: oneshot

parts:
  scripts:
    plugin: nil

and the install hook:

#!/bin/bash -e

snapctl stop --disable snapctl-core18.service

It fails to install with:

error: cannot perform the following tasks:
- Run install hook of "snapctl-core18" snap if present (run hook "install": 
-----
/snap/snapctl-core18/x1/meta/hooks/install: line 3: snapctl: command not found
-----)

Here’s snapd version:

$ snap version
snap    2.35
snapd   2.35+git398.g0d192285b
series  16
ubuntu  18.04
kernel  4.15.0-33-generic
$ snap info core18 | grep installed:
installed:   18  (358) 45MB base

ping @mvo

Thanks,
Ian


#2

I should also point out that I also tried using snapd edge:

$ snap version
snap    2.35.1+git944.b5355ba~ubuntu16.04.1
snapd   2.35.1+git944.b5355ba~ubuntu16.04.1
series  16
ubuntu  18.04
kernel  4.15.0-33-generic

with the same results


#3

The problem is that right now /usr/bin/snapctl in core18 is a symlink to /usr/lib/snapd/snapctl, and that’s missing.


#4

What revision/version of core18 are you using? Could you please try to install the version from beta? Can you please run the following to see if that works:

$ sudo snap install --beta core18
$ sudo snap install test-snapd-tools-core18
$ snap run --shell test-snapd-tools-core18.echo
$ snapctl -h
$ /usr/lib/snapd/snapctl -h

Its a bit of an indirect test because it does not directly run the hook just checks if the binaries are there.

Fwiw I also added a spread test for this here: https://github.com/snapcore/snapd/pull/5811 (which indicates there are issues on suse and amazon linux, I have not debugged those yet).

[edit: updated after actually having time to properly look into this :)]


#5

I tried installing core18 beta: (note that I was already using edge in the original post):

$ sudo snap refresh core18 --beta
core18 (beta) 18 from Canonical✓ refreshed
$ snap info core18
name:      core18
summary:   Runtime environment based on Ubuntu 18.04
publisher: Canonical✓
license:   unset
description: |
  The base snap based on the Ubuntu 18.04 release.
type:         base
snap-id:      CSO04Jhav2yK0uz97cr0ipQRyqg0qQL6
tracking:     beta
refresh-date: today at 08:32 CDT
channels:                   
  stable:    0.1 (19)  24MB -
  candidate: ↑              
  beta:      18  (351) 45MB -
  edge:      18  (358) 45MB -
installed:   18  (351) 45MB base

I still cannot install my original test snap with the install hook using core18 beta, it errors with the same result as in my original post.

If I install the test snap you mention and run snap run --shell ..., I see this:

$ sudo snap install test-snapd-tools-core18
test-snapd-tools-core18 1.0 from Canonical✓ installed
$ snap run --shell test-snapd-tools-core18.echo
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

$ snapctl -h
Usage:
  snapd [OPTIONS] <command>

Help Options:
  -h, --help  Show this help message

Available commands:
  get      The get command prints configuration and interface connection settings.
  restart  Restart services
  set      Changes configuration options
  start    Start services
  stop     Stop services
$ /usr/lib/snapd/snapctl -h
Usage:
  snapd [OPTIONS] <command>

Help Options:
  -h, --help  Show this help message

Available commands:
  get      The get command prints configuration and interface connection settings.
  restart  Restart services
  set      Changes configuration options
  start    Start services
  stop     Stop services

I’m pretty sure the problem is not that snapctl doesn’t exist for the snap mount namespace, just that for some reason it doesn’t exist while running the hooks, i.e. the install hook. I also see this problem with the config hook though.


#6

Thanks, I will dig deeper into this then.


#7

Just a random guess, but this commented out code I was looking at for entirely unrelated reasons seems like it might be highly related: https://github.com/snapcore/snapd/blob/master/cmd/snap-confine/mount-support.c#L416
:man_shrugging:


#8

Thanks for this pointer. I was caught in some different issues today so didn’t dig deeper yet. My plan is to use your example for a spread test to ensure we get this right. What puzzles me is that snap run --shell shows you the snapctl, hooks should see a similar environment. But the spread test (and some quality time with the code) will explain things :slight_smile:


#9

Hey @ijohnson - I used your example to create a spread test that is very close to the one above in https://github.com/snapcore/snapd/pull/5825 - while doing so I found issues in our packaging for fedora/suse/arch that prevented this to work. However on ubuntu this is working fine for me (with snapd from master).


#10

So I still wasn’t able to use snapctl, but remembered that I was hacking on a local version of snapd and had disabled re-exec (I’m on Ubuntu 18.04, so it’s enabled by default) with systemctl edit snapd.service and added SNAPD_REEXEC=0, which then led to the rootfs for the snap being taken from the snapd deb package, which doesn’t appear to contain snapctl. I think that may be a bug. When I re-enabled re-exec however I still wasn’t able to use snapctl. I had to do the following steps:

  1. Re-enabled re-exec
  2. Restart snapd.service
  3. Discarding the mount namespace (using sudo /usr/lib/snapd/snap-discard-ns SNAP_NAME) saved for the failed install of the snap (as this namespace saves mounting the deb, which doesn’t have snapctl)
  4. Now installing the snap succeeds.

This is the version of snapd I have installed via the deb on bionic:

$ apt show snapd 2>/dev/null | grep Version
Version: 2.34.2+18.04

I think there are 2 bugs here:

  1. The deb package for snapd doesn’t include snapctl on Ubuntu 18.04 at least (or at least a new version with snapctl hasn’t been released yet)
  2. When undoing the attempt to install a snap, the snap’s namespace should be discarded as well. I can’t think of a reason why we would want to keep the mount namespace for a snap that failed to be installed.

I can file a LP bug for the later issue if desired.

Thanks for looking into this!