Travis-worker service isn't getting started


#1

Hi - @stgraber/@Wimpress

I have installed travis-worker and lxd snaps on power arch(ppc64le) and below are my env details.

ubuntu@travisci-lxd-worker2:~$ snap --version
snap    2.39.3
snapd   2.39.3
series  16
ubuntu  18.04
kernel  4.15.0-45-generic
ubuntu@travisci-lxd-worker2:~$ lxd --version
3.0.3
ubuntu@travisci-lxd-worker2:~$ travis-worker --version
travis-worker v=v6.2.0-68-gc937f26 rev=c937f263ff0f0c4260577d4792585958084c63d3 d=2019-06-27T16:16:04+0000 go=go1.12.6
ubuntu@travisci-lxd-worker2:~$ lxc --version
3.0.3

ubuntu@travisci-lxd-worker2:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"

ubuntu@travisci-lxd-worker2:~$ arch
ppc64le

Getting issue in starting the service

ubuntu@travisci-lxd-worker2:~$ journalctl -xu snap.travis-worker.daemon.service | tail -f
Jul 08 06:19:24 travisci-lxd-worker2 travis-worker.daemon[14300]: time="2019-07-08T06:19:24Z" level=error msg="couldn't create backend provider" err="Get http://unix.socket/1.0: dial unix /var/snap/lxd/common/lxd/unix.socket: connect: permission denied" pid=14300 self=cli


ubuntu@travisci-lxd-worker2:~$ id
uid=1000(ubuntu) gid=108(lxd) groups=108(lxd),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),113(lpadmin),114(sambashare),1000(ubuntu)

ubuntu@travisci-lxd-worker2:~$ ls -l /var/snap/lxd/common/lxd/unix.socket
srw-rw---- 1 ubuntu lxd 0 Jul  8 04:34 /var/snap/lxd/common/lxd/unix.socket

Checking lxd service

ubuntu@travisci-lxd-worker2:~$ systemctl status lxd
â lxd.service - LXD - main daemon
   Loaded: loaded (/lib/systemd/system/lxd.service; indirect; vendor preset: enabled)
   Active: active (running) since Mon 2019-07-08 06:20:38 UTC; 1h 31min ago
     Docs: man:lxd(1)
  Process: 14426 ExecStartPost=/usr/bin/lxd waitready --timeout=600 (code=exited, status=0/SUCCESS)
  Process: 14409 ExecStartPre=/usr/lib/powerpc64le-linux-gnu/lxc/lxc-apparmor-load (code=exited, status=0/SUCCESS)
 Main PID: 14425 (lxd)
    Tasks: 57
   CGroup: /system.slice/lxd.service
           ââ14425 /usr/lib/lxd/lxd --group lxd --logfile=/var/log/lxd/lxd.log

Jul 08 06:20:37 travisci-lxd-worker2 systemd[1]: Starting LXD - main daemon...
Jul 08 06:20:37 travisci-lxd-worker2 lxd[14425]: t=2019-07-08T06:20:37+0000 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored."
Jul 08 06:20:38 travisci-lxd-worker2 systemd[1]: Started LXD - main daemon.

and lxd.socket service

ubuntu@travisci-lxd-worker2:~$ systemctl status lxd.socket
â lxd.socket - LXD - unix socket
   Loaded: loaded (/lib/systemd/system/lxd.socket; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2019-07-08 06:20:31 UTC; 1h 32min ago
     Docs: man:lxd(1)
   Listen: /var/lib/lxd/unix.socket (Stream)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/lxd.socket

Jul 08 06:20:31 travisci-lxd-worker2 systemd[1]: Starting LXD - unix socket.
Jul 08 06:20:31 travisci-lxd-worker2 systemd[1]: Listening on LXD - unix socket.

ubuntu@travisci-lxd-worker2:~$ ls -l  /var/lib/lxd/unix.socket
srw-rw---- 1 ubuntu lxd 0 Jul  8 06:20 /var/lib/lxd/unix.socket

I have done everything possible as mentioned here - https://github.com/lxc/lxd/issues/1635 however didn’t help resolve the error

Please let me of any pointers to resolve this issue ( : dial unix /var/snap/lxd/common/lxd/unix.socket: connect: permission denied)


#2

In order to have the travis-worker talk to LXD you must connect the lxd interface for it.

snap connect travis-worker:lxd lxd:lxd should do


#3

Note that for such things, you can always look at snap interfaces for a list of things that can be connected but currently aren’t (look at the bottom).


#4

@stgraber - Firstly thanks for formatting my comments above …
I did try out but it didn’t help , still getting same error

ubuntu@travisci-lxd-worker2:~$ sudo snap connect travis-worker:lxd lxd:lxd
ubuntu@travisci-lxd-worker2:~$ snap interfaces
Slot                       Plug
:account-control           -
:accounts-service          -
:adb-support               -
:alsa                      -
:autopilot-introspection   -
:avahi-control             -
:avahi-observe             -
:block-devices             -
:bluetooth-control         -
:bluez                     -
:broadcom-asic-control     -
:browser-support           -
:calendar-service          -
:camera                    -
:can-bus                   -
:cifs-mount                -
:classic-support           -
:contacts-service          -
:core-support              -
:cpu-control               -
:cups-control              -
:daemon-notify             -
:dcdbas-control            -
:desktop                   -
:desktop-legacy            -
:device-buttons            -
:display-control           -
:docker-support            -
:dvb                       -
:firewall-control          -
:framebuffer               -
:fuse-support              -
:gpg-keys                  -
:gpg-public-keys           -
:gpio-memory-control       -
:greengrass-support        -
:gsettings                 -
:hardware-observe          -
:hardware-random-control   -
:hardware-random-observe   -
:home                      -
:hostname-control          -
:intel-mei                 -
:io-ports-control          -
:joystick                  -
:juju-client-observe       -
:kernel-module-control     -
:kernel-module-observe     -
:kubernetes-support        -
:kvm                       -
:libvirt                   -
:locale-control            -
:log-observe               -
:lxd-support               lxd
:modem-manager             -
:mount-observe             -
:multipass-support         -
:netlink-audit             -
:netlink-connector         -
:network                   lxd,travis-worker
:network-bind              lxd
:network-control           -
:network-manager           -
:network-manager-observe   -
:network-observe           -
:network-setup-control     -
:network-setup-observe     -
:ofono                     -
:opengl                    -
:openvswitch               -
:openvswitch-support       -
:optical-drive             -
:password-manager-service  -
:personal-files            -
:physical-memory-control   -
:physical-memory-observe   -
:ppp                       -
:process-control           -
:pulseaudio                -
:raw-usb                   -
:removable-media           -
:screen-inhibit-control    -
:screencast-legacy         -
:shutdown                  -
:snapd-control             -
:ssh-keys                  -
:ssh-public-keys           -
:system-files              -
:system-observe            lxd
:system-trace              -
:time-control              -
:timeserver-control        -
:timezone-control          -
:tpm                       -
:u2f-devices               -
:udisks2                   -
:uhid                      -
:unity7                    -
:upower-observe            -
:wayland                   -
:x11                       -
lxd:lxd                    travis-worker
'snap interfaces' is deprecated; use 'snap connections'.

Even snap connections reflecting …

   ubuntu@travisci-lxd-worker2:~$ snap connections`
    Interface       Plug                   Slot             Notes`
    lxd             travis-worker:lxd      lxd:lxd          manual`
    lxd-support     lxd:lxd-support        :lxd-support     -`
    network         lxd:network            :network         -`
    network         travis-worker:network  :network         -`
    network-bind    lxd:network-bind       :network-bind    -`
    system-observe  lxd:system-observe     :system-observe  -`

However it didnt help , still getting “permission denied” error

    ubuntu@travisci-lxd-worker2:/etc/systemd/system$ journalctl -xu snap.travis-worker.daemon.service | tail -f
    Jul 08 13:58:45 travisci-lxd-worker2 travis-worker.daemon[27054]: time="2019-07-08T13:58:45Z" level=error msg="couldn't create backend provider" err="Get http://unix.socket/1.0: dial unix /var/snap/lxd/common/lxd/unix.socket: connect: permission denied" pid=27054 self=cli
    Jul 08 14:13:24 travisci-lxd-worker2 systemd[1]: Started Service for snap application travis-worker.daemon.
    -- Subject: Unit snap.travis-worker.daemon.service has finished start-up
    -- Defined-By: systemd
    -- Support: http://www.ubuntu.com/support
    --
    -- Unit snap.travis-worker.daemon.service has finished starting up.
    --
    -- The start-up result is RESULT.
    Jul 08 14:13:24 travisci-lxd-worker2 travis-worker.daemon[27270]: `time="2019-07-08T14:13:24Z" level=error msg="couldn't create backend provider" err="Get http://unix.socket/1.0: dial unix /var/snap/lxd/common/lxd/unix.socket: connect: permission denied" pid=27270 self=cli`

Any further pointers…if any ? Am i missing something here…


#5

any pointers to share @stgraber please ?


#6

Did you restart the travis-worker snap following the connection?


#7

@stgraber - did restart travis-worker snap too …however not much to avail.

currently running “lxd” services on my vm

root@travisci-lxd-worker2:/etc/systemd/system# systemctl status lxd.service
â lxd.service - LXD - main daemon
   Loaded: loaded (/lib/systemd/system/lxd.service; indirect; vendor preset: enabled)
   Active: active (running) since Wed 2019-07-10 13:10:28 UTC; 22h ago
     Docs: man:lxd(1)
 Main PID: 32686 (lxd)
    Tasks: 66
   CGroup: /system.slice/lxd.service
           ââ  488 dnsmasq --strict-order --bind-interfaces --pid-file=/var/lib/lxd/networks/lxdbr0/dnsmasq.pid --except-interface=lo --interface=lxdbr0 --quiet-dhcp --quiet-dhc
           ââ32686 /usr/lib/lxd/lxd --group lxd --logfile=/var/log/lxd/lxd.log

Jul 11 11:35:58 travisci-lxd-worker2 dnsmasq[488]: using nameserver 127.0.0.53#53
Jul 11 11:40:25 travisci-lxd-worker2 dnsmasq[488]: reading /etc/resolv.conf
Jul 11 11:40:25 travisci-lxd-worker2 dnsmasq[488]: using local addresses only for domain lxd
Jul 11 11:40:25 travisci-lxd-worker2 dnsmasq[488]: using nameserver 127.0.0.53#53
Jul 11 11:44:50 travisci-lxd-worker2 dnsmasq[488]: reading /etc/resolv.conf
Jul 11 11:44:50 travisci-lxd-worker2 dnsmasq[488]: using local addresses only for domain lxd
Jul 11 11:44:50 travisci-lxd-worker2 dnsmasq[488]: using nameserver 127.0.0.53#53
Jul 11 11:49:22 travisci-lxd-worker2 dnsmasq[488]: reading /etc/resolv.conf
Jul 11 11:49:22 travisci-lxd-worker2 dnsmasq[488]: using local addresses only for domain lxd
Jul 11 11:49:22 travisci-lxd-worker2 dnsmasq[488]: using nameserver 127.0.0.53#53

root@travisci-lxd-worker2:/etc/systemd/system# systemctl status lxd.socket
â lxd.socket - LXD - unix socket
   Loaded: loaded (/lib/systemd/system/lxd.socket; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2019-07-10 13:10:22 UTC; 22h ago
     Docs: man:lxd(1)
   Listen: /var/lib/lxd/unix.socket (Stream)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/lxd.socket

Jul 10 13:10:22 travisci-lxd-worker2 systemd[1]: Starting LXD - unix socket.
Jul 10 13:10:22 travisci-lxd-worker2 systemd[1]: Listening on LXD - unix socket.
root@travisci-lxd-worker2:/etc/systemd/system#

snaps installed on my vm …

root@travisci-lxd-worker2:/etc/systemd/system# snap list
Name           Version               Rev    Tracking  Publisher   Notes
core           16-2.39.3             7273   stable    canonical*  core
core18         20190704              1057   stable    canonical*  base
lxd            git-0ac28b2           11162  edge      canonical*  -
travis-worker  v6.2.0+git68.c937f26  43     edge      travis-ci*  -

Snaps services running

root@travisci-lxd-worker2:/etc/systemd/system# snap services
Service               Startup  Current   Notes
lxd.activate          enabled  inactive  -
lxd.daemon            enabled  inactive  socket-activated
travis-worker.daemon  enabled  inactive  -

snaps started

root@travisci-lxd-worker2:/etc/systemd/system# snap start travis-worker.daemon
Started.
root@travisci-lxd-worker2:/etc/systemd/system# snap start lxd.daemon
Started.
root@travisci-lxd-worker2:/etc/systemd/system# snap start lxd.activate
Started.

However travis-worker service is still unable to start as seen below

root@travisci-lxd-worker2:/etc/systemd/system# journalctl -xu snap.travis-worker.daemon.service | tail -f
Jul 11 06:47:22 travisci-lxd-worker2 travis-worker.daemon[2502]: time="2019-07-11T06:47:22Z" level=error msg="couldn't create backend provider" err="Get http://unix.socket/1.0: dial unix /var/snap/lxd/common/lxd/unix.socket: connect: permission denied" pid=2502 self=cli
Jul 11 11:56:34 travisci-lxd-worker2 systemd[1]: Started Service for snap application travis-worker.daemon.
-- Subject: Unit snap.travis-worker.daemon.service has finished start-up
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit snap.travis-worker.daemon.service has finished starting up.
--
-- The start-up result is RESULT.
Jul 11 11:56:35 travisci-lxd-worker2 travis-worker.daemon[3502]: time="2019-07-11T11:56:35Z" level=error msg="couldn't create backend provider" err="Get http://unix.socket/1.0: dial unix /var/snap/lxd/common/lxd/unix.socket: connect: permission denied" pid=3502 self=cli

I expected by installing “travis-worker” using snap , i would have got a running service , however getting this error- level=error msg=“couldn’t create backend provider” err=“Get http://unix.socket/1.0: dial unix /var/snap/lxd/common/lxd/unix.socket: connect: permission denied” pid=3502 self=cli

@stgraber- Any pointers to resolve above would be appreciated :slight_smile:


#8

Works perfectly fine here.

root@c1:~# snap install lxd --edge
2019-07-11T13:03:08Z INFO Waiting for restart...
lxd (edge) git-0ac28b2 from Canonical✓ installed
root@c1:~# lxd.migrate -yes
=> Connecting to source server
=> Connecting to destination server
=> Running sanity checks
The source server is empty, no migration needed.

The migration is now complete and your containers should be back online.

All done. You may need to close your current shell and open a new one to have the "lxc" command work.
To migrate your existing client configuration, move ~/.config/lxc to ~/snap/lxd/current/.config/lxc

root@c1:~# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: 
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=lxdbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like LXD to be available over the network? (yes/no) [default=no]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes] 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 

root@c1:~# snap install travis-worker --edge
travis-worker (edge) v6.2.0+git68.c937f26 from Travis CI✓ installed
root@c1:~# snap connect travis-worker:lxd lxd:lxd
root@c1:~# vim /var/snap/travis-worker/common/worker.env
root@c1:~# snap restart travis-worker
Restarted.

root@c1:~# journalctl -u snap.travis-worker.daemon.service -f
-- Logs begin at Thu 2019-07-11 13:02:41 UTC. --
Jul 11 13:23:09 c1 travis-worker.daemon[13180]: time="2019-07-11T13:23:09Z" level=info msg="setting up heartbeat" pid=13180 self=cli
Jul 11 13:23:09 c1 travis-worker.daemon[13180]: time="2019-07-11T13:23:09Z" level=info msg="starting signal handler loop" pid=13180 self=cli
Jul 11 13:23:09 c1 travis-worker.daemon[13180]: time="2019-07-11T13:23:09Z" level=info msg="starting processor" pid=13180 processor=0bd678ed-2f4a-4905-a9dd-a07794b13418@13180.c1 self=processor
Jul 11 13:23:09 c1 travis-worker.daemon[13180]: time="2019-07-11T13:23:09Z" level=info msg="starting processor" pid=13180 processor=84d729b5-4ac7-4e4e-aea9-2ebf21575b55@13180.c1 self=processor
Jul 11 13:23:09 c1 travis-worker.daemon[13180]: time="2019-07-11T13:23:09Z" level=info msg="starting processor" pid=13180 processor=d01db90f-79d5-480c-aa08-c635a9ce832c@13180.c1 self=processor
Jul 11 13:23:09 c1 travis-worker.daemon[13180]: time="2019-07-11T13:23:09Z" level=info msg="starting processor" pid=13180 processor=bac42bfe-9cc7-4858-a8d6-be3ff5051d48@13180.c1 self=processor
Jul 11 13:23:09 c1 travis-worker.daemon[13180]: time="2019-07-11T13:23:09Z" level=info msg="starting processor" pid=13180 processor=98ecbf95-8ecb-43da-822c-bcca2e02b55b@13180.c1 self=processor
Jul 11 13:23:09 c1 travis-worker.daemon[13180]: time="2019-07-11T13:23:09Z" level=info msg="starting processor" pid=13180 processor=dd5ca026-fbf2-4bd5-a293-1cc809f11db0@13180.c1 self=processor
Jul 11 13:23:09 c1 travis-worker.daemon[13180]: time="2019-07-11T13:23:09Z" level=info msg="starting processor" pid=13180 processor=b36cf041-9801-4e10-9d47-621e71550f60@13180.c1 self=processor
Jul 11 13:23:09 c1 travis-worker.daemon[13180]: time="2019-07-11T13:23:09Z" level=info msg="starting processor" pid=13180 processor=3d6971a2-43fc-4f91-838d-dd7dd53cd4c5@13180.c1 self=processor

#9

hey @stgraber - strangely above sequence of installations steps worked for me …Closing this ticket.