Hi!
We’re still in the manual review process so no snap store URL available yet, but the username/package name: husarnet/husarnet. Dashboard link, snapcraft.yaml
First we’d like to request a manual review (I believe these are done asynchronously, but there’s so much going on with manual acceptance recently, I could have lost track) as this is a time-sensitive issue (we’d like to showcase it at Embedded World conference next week)
Secondly, we’d like to request a couple of auto-connections (and a hint, whether we got those right in the first place)
network-control - this one is straightforward - our product is a VPN, so we need some usual low level networking bits - namely TUN/TAP interface creation and control, IP address and route management - I believe this is the exact plugin for that (random note: props for including /dev/tun in it )
system-files for rw access to /etc/hosts (and /etc/hosts.tmp) - this one is slightly less obvious. We’re using IPv6 addresses for our network and those addresses are generated from public keys used for encryption, thus human-wise are almost random. That’s why we have a mechanism for naming each of the hosts on the network (and sharing those names with other hosts). We’ve tried a couple of methods for exposing those names to operating systems (we even had a custom DNS proxy…) but almost all of them required much more setup from the user and were more fragile than good, old /etc/hosts entries. As this is a heavily used file on most of the systems, we’re trying to be respectful and careful - we’re not touching any lines that do not have our suffix, we’re preparing new version of the file on the side (hence /etc/hosts.tmp) and overwriting file with a file instead of rewriting it from out code, risking it being invalid while we do that. As per the plugin choice - I’m aware that this plugin is heavily discouraged (for a right reason), but I’ve not found any that would fit this case better. If that’s not the case - please do let me know.
Overall please treat this request as a “v1” version. I’ve already seen many possible integrations that can be made between Snap and Husarnet, but, I believe, they will come with time and experience
I’d definitively support the request for auto-connection, however write access to /etc/hosts looks potentially dangerous to me and I cant encourage it.
The approach to prepare a new version of the file on the side (hence /etc/hosts.tmp) and overwriting the original one may cause races if all those operations are not done atomically. Even if that would not be case, /etc/hosts file is part of a read-only file system (base snap) and cannot be directly modified.
For me the best approach to handle this scenario would be to use a libnss plugin (maybe libnss-db or libnss-ldap can be a good fit even though I never used it).
thank you for the feedback, please let me clarify our intents and testing done so far. Let’s start with nity-gritty details and then go to the more high-level stuff.
Our approach with preparing a new version on the side assumes that /etc/hosts file is rarely changed but read very often. This is why we prioritize never allowing anyone to read a half-baked content instead of us reading a potentially invalid file. (Hence we first try to prepare new content on the side and then mv it. Writing directly is a fallback option in our codebase.) You can find related code here and here if you feel like reading it. We have never had any problems with this approach so far, but I can see a point in your feedback and we can change it to a less complicated flow (read full file in a single syscall, modify the content, write full file in a single syscall).
As per /etc/hosts being a part of a base snap and thus being read-only - I have no idea how snap works under the hood, but system-files seems to be overriding it and after a manual connection it allows me to write to the file with no problem. I can see correct content on the host even if I manually clear it to the most vanilla content possible and then restart the service. Can you tell me more about “cannot be directly modified”? Is it more of an architecture choice, good habit or something entirely else?
Now to the more high-level part. Let me first draw some more picture of our product - we’re providing users with a VPN service that spans across as many platforms (think bare metal, containers, embedded stuff like ESP-IDF,… not architectures) as possible. Each of those platforms has totally different opinions (and technical solutions) around host names, DNS and so on. Because we want to keep our solutions battle tested we’re trying to minimize the number of bits dedicated to each of the platforms and keep the most users using a single mechanism. This is why, even though I really like your suggestion with libnss, that wouldn’t be the solution for our case - we have a solid portion of users using platforms without libc (i.e. Husarnet built into containers based on Alpine by our users). We most certainly can add libnss as one of the available solutions (i.e. dedicated for the platforms like snap) but I really doubt it’d be given as much love as it would deserve.
We also explored some other ways of exposing DNS names to the users and apps - like a custom DNS server, DNS proxy, systemd-resolved integration but all of them seemed… surprisingly more fragile, error prone and shared the same concern of not being used enough to be well maintained.
As per /etc/hosts being a part of a base snap and thus being read-only … Is it more of an architecture choice, good habit or something entirely else?
Some directories are mounted in the snap from the base snap (like some binaries and configuration files) to create a reproducible environment, what makes them intrinsically immutable (as base snap is a read-only file system). However, it seems that /etc is mounted from the host system rather than from the base snap and only few files (/etc/alternatives, /etc/nsswitch.conf, …) are mounted from the base. Thus, I was wrong and directly editing /etc/hosts is technically possible.
Our approach with preparing a new version on the side assumes that /etc/hosts file is rarely changed but read very often. This is why we prioritize never allowing anyone to read a half-baked content instead of us reading a potentially invalid file.
That’s certainly better than directly editing the file, but it still It may drive other applications to an inconsistent state in case of a race I guess.
Each of those platforms has totally different opinions (and technical solutions) around host names, DNS and so on. … we’re trying to minimize the number of bits dedicated to each of the platforms
I fully understand your position.
All in all, editing /etc/hosts from the snap is technically possible and taking this approach makes sense for the product strategy (minimizing platform specific bits). On the other hand, directly editing /etc/hosts will alter the host system, potentially leading to inconsistencies, what doesn’t seem to completely fit with the snap philosophy to me, so it is hard to say
I think you could do a few things that are technically sound and would let you solve this.
You may continue to use the “atomic” rename of /etc/hosts.$RANDOM to /etc/hosts coupled with a fdirsync on /etc is as good as it gets on “typical” file systems and lets you replace the file in a way that concurrent racing readers will never see partial content. It doesn’t let you synchronise with other applications attempting the same trick (other writers)
You might explore using leases fcntl with F_SETLEASE (see man fcntl) which would at least notify you when other applications are attempting to read or (in your case) write to the file. Lease won’t let you handle the rename case, though.
You might explore using dnotify/inotify/fanotify which would let you see when others are messing up hosts file. We can make the appropriate interfaces grant the right permissions if this is something that other applications have not yet explored.
The conceptual problem is that we have long standing issue with /etc and, at some point, will have to re-visit that with a synthetic /etc that is provided to the snap application. When that happens we will surely associate it with a base change (our current policy of making breaking changes associated with new bases so that they don’t happen retroactively) and we have a number of interfaces that we could use to create the illusion that the application is accessing real host /etc. This problem is not something that you should be concerned about today. I’m merely mentioning because I want to follow up with a discussion of what is going on for snaps, at runtime in terms of /etc.
When snap applications are started, most of the file system of the host is stashed away and a new root file system is created as a combination of the base snap, the application snap, API file systems (/proc, /dev and /sys, with appropriate changes) and portions of the original host file system where we expect applications to write (/var/snap, /home, /root) - but also, critically, /etc. Any access controls that happen on top of that are handled with Linux security module (apparmor on distributions that are supported), with a small influence from seccomp and bpf depending on the type of access used. In this sense we can commonly “open up” access to specific files in /etc.
As a special exception, on ubuntu-core systems, the /etc directory is largely a read-only image, with only special places being either bind-mounted to writable locations (with consequences) or being symbolic links pointing to writable places (with other consequences). Depending on the type of access, this may break your scheme of updating /etc/hosts. Bind mounts make the mount point (file in this case) impossible to remove as well as impossible to be renamed or be renamed onto. This is exactly how /etc/hosts is handled on core systems for at least a few generations now. In addition the fact that /etc is really a read-only image (as can be confirmed by using stat -f /etc) means that you cannot create temporary files to be used for renaming (neither with the classic random name trick nor with the more modern O_TMPFILE flag for open). This limits your choice as now you really must rewrite the file without replacing it.
With all this context I would lean you towards a hybrid approach that includes, at the very least, a way to rewrite the file with ftruncate and write which would work everywhere. If you have resources you should explore using leases as a way to make that better. Leases allow you to write the file or drop the lease (and write the file later). See the manual page of fcntl for a discussion on how to use leases correctly.
Woah, this info is exactly what I needed to upgrade this mechanism to to feel technically-confident about it. Thank you @zyga a million. Thanks for the background info and a peek into future plans too - those really help with understanding the reasons and estimating the value of solutions for the future.
In this case, plan for us is now:
add fdirsync call to the code
add fcntl with F_SETLEASE to the code
rework the file rename/write functions to be a little more verbose about their hybrid approach (rename if possible, rewrite if not)
test on Ubuntu Core
As lease breakers from the F_SETLEASE may be blocked by the OS during the lock, I’ll probably won’t go the dnotify/inotify/fanotify route, but that may change after I spend some time with the code.
I’ll let myself to leave this thread open and come back here after I implemented said changes (think a couple of weeks with a current workload on other things).
)