Easy-openvpn snap broken with latest core update

snapd
1

auto upgrade of core-16 to 16-2.49.1 breaks snap easy-openvpn.client

sudo easy-openvpn.connect-server test.ovpn
Options error: In [CMD-LINE]:1: Error opening configuration file: test.ovpn
Use --help for more information.

snap revert core

back to 16-2.49 and all is well.

This has stranded multiple field devices that had auto open vpn connections that were working perfectly and now they have lost connectivity

they are installed in field 2500 miles away 25ft up light poles. I need a update auto installed, and for a restart to happen, because the /etc/systemd/system/easy-openvpn-client.service failed and isnā€™t going to restart on its own

otherwise, iā€™ll have to send a service truck out to remove and reinstall new devices.

what a disaster. makes me wonder why im using snap.

2

Hi,

Are you able to provide any logs for these devices or explain further how the easy-openvpn snap became broken with the 2.49.1 update?

3

Yes itā€™s very repeatable - can revert to 2.49 and easy openvpn client works, then refresh/install core 2.49.1 and it breaks.
It looks like an app armor issue.
Iā€™ll send logs tomorrow - itā€™s getting late here now.

David Howard

4

you answered that yourself a few lines above:

:wink:

also note that /etc/systemd/system/easy-openvpn-client.service is definitely not coming from a snap, if you have such a file on disk then something is seriously mixed up ā€¦

5

i wonder if we could have cross-snap checks and rollback for such use cases in the future ā€¦

(i know this might be complex to implement, but it would be awesome if an app snap that detects it is broken after an update of the base snap could auto-trigger a rollback of the offending update)

6

Thanks David for your problem report and sorry for the trouble this is causing you.

We take regressions like this extremely seriously. I did try to reproduce this today by

  • installing core 2.49 on 20.04 (classic)
  • installing the easy-openvpn snap from stable
  • creating an nonsense config file in /root/test.ovpn
  • ran sudo easy-openvpn.connect-server test.ovpn
    I got the expected ā€œOptions error: ā€¦ā€ (and not the error opening you describe above)

I then upgraded core 2.49.1 and:

  • ran sudo easy-openvpn.connect-server test.ovpn
    I also got the expected ā€œOptions error: ā€¦ā€ and not the error you describe above. So it seems I can not reproduce this with the information I have.

Can you please help us reproducing this issue by providing details about what OS is running (Ubuntu or Ubuntu Core, what version) and where exactly test.ovpn is stored? Also logs like the dmesg|tail output after the command fails. Any further information that could help us understand this issue would also be appreciated.

7

This feature has indeed been requested before but is complicated to implement as you suggest.

9

easy-openvpn snap failure after snap core auto refresh

when core auto refreshed from 16-2.49 to 16-2.49.1, the snap easy-openvpn stopped working

ubuntu@bbbdh1:~$ snap list

Name Version Rev Tracking Publisher Notes

core 16-2.49 10861 latest/candidate canonicalāœ“ core

easy-openvpn 2.3.10-4 8 latest/stable canonicalāœ“ -

Specifically, I have the easy-openvpn client configured to auto start on power-up or

reboot to establish a client VPN connection to a cloud instance / hub so that secure

communications to/from the edge device are possible. Without the VPN there is no way

to remote connect to the field device as it is using a public APN (NATā€™d) cellular

connection.

ubuntu@bbbdh1:~$ systemctl status easy-openvpn-client.service

ā— easy-openvpn-client.service - Easy OpenVPN Autostart

Loaded: loaded (/etc/systemd/system/easy-openvpn-client.service; enabled; vendor preset: enabled)

Active: active (running) since Wed 2021-03-31 01:58:12 UTC; 13h ago

Main PID: 12207 (openvpn)

Tasks: 0 (limit: 1027)

CGroup: /system.slice/easy-openvpn-client.service

ā€£ 12207 openvpn --config /home/ubuntu/bbbdh1.ovpn

Mar 31 01:58:12 bbbdh1 systemd[1]: Started Easy OpenVPN Autostart.

(from the server side)

ubuntu@ip-172-31-17-204:~/openvpn$ sudo easy-openvpn.status

OpenVPN CLIENT LIST

Updated,Wed Mar 31 15:09:22 2021

Common Name,Real Address,Bytes Received,Bytes Sent,Connected Since

bbb004,172.58.239.21:39162,4101491,4084929,Wed Mar 24 13:46:49 2021

bbb001,172.58.236.152:37421,1047817,1083703,Mon Mar 29 18:55:06 2021

bbbdh1,104.220.84.132:33797,3944,3878,Wed Mar 31 15:08:34 2021

ROUTING TABLE

Virtual Address,Common Name,Real Address,Last Ref

192.168.255.6,bbbdh1,104.220.84.132:33797,Wed Mar 31 15:08:35 2021

192.168.254.4,bbb004,172.58.239.21:39162,Wed Mar 31 14:46:36 2021

192.168.254.1,bbb001,172.58.236.152:37421,Wed Mar 31 15:05:51 2021

GLOBAL STATS

Max bcast/mcast queue length,0

END

10

so maybe the difference is that 16-2.49 allows sudo access to the home folder (/home/ubuntu), while 16-2.49.1 does not.

ubuntu@bbbdh1:~$ snap connections

Interface Plug Slot Notes

firewall-control easy-openvpn:firewall-control :firewall-control -

home easy-openvpn:home :home -

network easy-openvpn:network :network -

network-bind easy-openvpn:network-bind :network-bind -

network-control easy-openvpn:network-control :network-control -

This would break easy-openvpn in my case, because the .ovpn file is in (/home/ubuntu) and thatā€™s where the ā€˜autostart_easy-openvpn_client.shā€™ is located, containing:

#!/bin/bash

echo ā€œ/snap/bin/easy-openvpn.connect-server /home/ubuntu/bbb???.ovpnā€ | bash

David Howard

11

I think a user written test that is called after a refresh/update might be the answer ā€” pushes the complexity on the end developer, but enables them to run a test after any refresh occurs and decide what acrtion they want to take, e.g. revert, notify, etc.

DH

12

Thanks David,

I did double check the diff of snapd 2.49 to 2.49.1 and we did change the way we generate apparmor profiles when non-Ubuntu kernels are used [1].

From apparmors point-of-view the /home/ubuntu directory belongs to the user ā€œubuntuā€ and because sudo will switch the user to ā€œrootā€ the apparmor profile does not allow the user ā€œrootā€ to read the data of the user ā€œubuntuā€.

So this change might explain the issue, to be sure we need some more details (see below):

Could you please provide the output of:

$ ls /sys/kernel/security/apparmor/features/
$ uname -r -v

And also what base OS is used, e.g. Ubuntu or UbuntuCore, some other distro? And if a custom kernel is used that maybe does not have all the apparmor patches?

Thanks,
Michael

[1] https://github.com/snapcore/snapd/pull/9936

13

Michael ā€“

Thanks for quick response!

This is a beagle board black SBC running ubuntu linux distro;
ubuntu@bbbdh1:~$ ls /sys/kernel/security/apparmor/features/
capability caps domain file mount namespaces network_v8 policy ptrace query rlimit signal

ubuntu@bbbdh1:~$ uname -r -v
4.19.94-ti-r36 #1bionic SMP PREEMPT Mon Mar 2 15:55:39 UTC 2020

ubuntu@bbbdh1:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic

ubuntu@bbbdh1:~$ hostnamectl
Static hostname: bbbdh1
Icon name: computer
Machine ID: f70b02da51ed17911969467c5e6b145a
Boot ID: f179c2083926462099c246f763652bb9
Operating System: Ubuntu 18.04.5 LTS
Kernel: Linux 4.19.94-ti-r36
Architecture: arm

David Howard

14

orgra
thats just the systemd service that calls the snapped easy-openvpn.client command; itā€™s used to auto connect the vpn on powerup/reset

15

well, that means you are not really using the snap but just consume some binaries from it and hack your own service and scripts around it without any proper integration ā€¦ which calls for breakage ā€¦

the easy-openvpn snap has integrated configuration management and ships with its own service file as well. you are pretty much completely levering out the snap integration with what you are doing which means it is not a surprise that it breaks at some point ā€¦

16

Im not using the server service. Im using the client only. I didnā€™t see a client service integration.

I assume you re referring to ā€˜/etc/systemd/system/snap.easy-openvpn.easy-openvpn.serviceā€™
How would that start a client?
How do you recommend I auto launch an openvpn client ā€” easy-openvpn.connect-server test.ovpn?

Enlighten me please

(BTW it had been working perfectly fine for many months)

17

simply by calling snap start .... after you ran the integrated configuration wizard (sudo easy-openvpn.setup)ā€¦

$ snap info --verbose easy-openvpn|grep -A1 ^services
services:
  easy-openvpn:      simple, enabled, inactive

snaps are self contained and ship all bits and pieces they need inside you should never just hack around that ā€¦

18

Thanks Iā€™ll try it. I didnā€™t see that in the docs; all i saw was setup for a server

sudo easy-openvpn.setup
ā€¦
nopeā€¦ it looks like its for setting up a server, not starting a clientā€¦
it creates a pki, and starts a server.

so it would appear that all the bits to auto start a client to server connection are not in the snap in this case

btw i use the server integration (snap start easy-openvpn) on my server already, but never found the equivalent capability for starting a client ā€” which is why i resorted to using systemd. I wasnā€™t hacking around the snaps capability.

for now i have updated my systemd file as below, and moved the ovpn file to /root;

[Unit]
Description=easy-openvpn.connect-server autostart

[Service]
ExecStart=/snap/bin/easy-openvpn.connect-server /root/client.ovpn
Restart=always
RestartSec=30

[Install]
WantedBy=multi-user.target

19

well, looking at the documentation of the snap

it is not even intended to be used as a client ā€¦ if you really want to use an easy openvpn client you should either ask the maintainer to include some properly integrated client support (by simply filing an issue above) or quickly create a snap yourself ā€¦ ā€œusing systemdā€ and invoking some random shell scripts to call binaries from inside the smap is a hack ā€¦

ā€¦ something iā€™d probably use at home for my self tinkered stuff that i can quickly fix, but definitely nothing iā€™d ever use in a commercial production setup ā€¦

20

Itā€™s a POC field deployment not production - so i used what was readily availableā€¦ https://snapcraft.io/easy-openvpn is the Canonical official snap.
Server side is fineā€¦ client needs to be updated. Iā€™ll do that and create a new snap later - for now its working again. Itā€™s just unfortunate that an auto update of core changed the permissions in such a way as to break something that had been workingā€¦ thankfully, thereā€™s only 4 that are deployed on streetlight poles so it wonā€™t be a huge expense to replace them with updated images - just a hassle because I have to get it done from 2500 miles away. Iā€™m curious what people using ubuntu core (native) are using for a vpn client on field deployed devices, as I intend to switch to core for pilot deployments.

21

Iā€™m happy to give a hand with any snap you willl package :slight_smile: regarding the client side, here is a discussion about extending the network-manager snap:

there seems to also be

https://github.com/openfortivpn/snap

that looks well maintained ā€¦

1 Like