Hi!
We’re still in the manual review process so no snap store URL available yet, but the username/package name: husarnet/husarnet. Dashboard link, snapcraft.yaml
First we’d like to request a manual review (I believe these are done asynchronously, but there’s so much going on with manual acceptance recently, I could have lost track) as this is a time-sensitive issue (we’d like to showcase it at Embedded World conference next week)
Secondly, we’d like to request a couple of auto-connections (and a hint, whether we got those right in the first place)
network-control - this one is straightforward - our product is a VPN, so we need some usual low level networking bits - namely TUN/TAP interface creation and control, IP address and route management - I believe this is the exact plugin for that (random note: props for including /dev/tun in it )
system-files for rw access to /etc/hosts (and /etc/hosts.tmp) - this one is slightly less obvious. We’re using IPv6 addresses for our network and those addresses are generated from public keys used for encryption, thus human-wise are almost random. That’s why we have a mechanism for naming each of the hosts on the network (and sharing those names with other hosts). We’ve tried a couple of methods for exposing those names to operating systems (we even had a custom DNS proxy…) but almost all of them required much more setup from the user and were more fragile than good, old /etc/hosts entries. As this is a heavily used file on most of the systems, we’re trying to be respectful and careful - we’re not touching any lines that do not have our suffix, we’re preparing new version of the file on the side (hence /etc/hosts.tmp) and overwriting file with a file instead of rewriting it from out code, risking it being invalid while we do that. As per the plugin choice - I’m aware that this plugin is heavily discouraged (for a right reason), but I’ve not found any that would fit this case better. If that’s not the case - please do let me know.
Overall please treat this request as a “v1” version. I’ve already seen many possible integrations that can be made between Snap and Husarnet, but, I believe, they will come with time and experience
I’d definitively support the request for auto-connection, however write access to /etc/hosts looks potentially dangerous to me and I cant encourage it.
The approach to prepare a new version of the file on the side (hence /etc/hosts.tmp) and overwriting the original one may cause races if all those operations are not done atomically. Even if that would not be case, /etc/hosts file is part of a read-only file system (base snap) and cannot be directly modified.
For me the best approach to handle this scenario would be to use a libnss plugin (maybe libnss-db or libnss-ldap can be a good fit even though I never used it).
thank you for the feedback, please let me clarify our intents and testing done so far. Let’s start with nity-gritty details and then go to the more high-level stuff.
Our approach with preparing a new version on the side assumes that /etc/hosts file is rarely changed but read very often. This is why we prioritize never allowing anyone to read a half-baked content instead of us reading a potentially invalid file. (Hence we first try to prepare new content on the side and then mv it. Writing directly is a fallback option in our codebase.) You can find related code here and here if you feel like reading it. We have never had any problems with this approach so far, but I can see a point in your feedback and we can change it to a less complicated flow (read full file in a single syscall, modify the content, write full file in a single syscall).
As per /etc/hosts being a part of a base snap and thus being read-only - I have no idea how snap works under the hood, but system-files seems to be overriding it and after a manual connection it allows me to write to the file with no problem. I can see correct content on the host even if I manually clear it to the most vanilla content possible and then restart the service. Can you tell me more about “cannot be directly modified”? Is it more of an architecture choice, good habit or something entirely else?
Now to the more high-level part. Let me first draw some more picture of our product - we’re providing users with a VPN service that spans across as many platforms (think bare metal, containers, embedded stuff like ESP-IDF,… not architectures) as possible. Each of those platforms has totally different opinions (and technical solutions) around host names, DNS and so on. Because we want to keep our solutions battle tested we’re trying to minimize the number of bits dedicated to each of the platforms and keep the most users using a single mechanism. This is why, even though I really like your suggestion with libnss, that wouldn’t be the solution for our case - we have a solid portion of users using platforms without libc (i.e. Husarnet built into containers based on Alpine by our users). We most certainly can add libnss as one of the available solutions (i.e. dedicated for the platforms like snap) but I really doubt it’d be given as much love as it would deserve.
We also explored some other ways of exposing DNS names to the users and apps - like a custom DNS server, DNS proxy, systemd-resolved integration but all of them seemed… surprisingly more fragile, error prone and shared the same concern of not being used enough to be well maintained.
As per /etc/hosts being a part of a base snap and thus being read-only … Is it more of an architecture choice, good habit or something entirely else?
Some directories are mounted in the snap from the base snap (like some binaries and configuration files) to create a reproducible environment, what makes them intrinsically immutable (as base snap is a read-only file system). However, it seems that /etc is mounted from the host system rather than from the base snap and only few files (/etc/alternatives, /etc/nsswitch.conf, …) are mounted from the base. Thus, I was wrong and directly editing /etc/hosts is technically possible.
Our approach with preparing a new version on the side assumes that /etc/hosts file is rarely changed but read very often. This is why we prioritize never allowing anyone to read a half-baked content instead of us reading a potentially invalid file.
That’s certainly better than directly editing the file, but it still It may drive other applications to an inconsistent state in case of a race I guess.
Each of those platforms has totally different opinions (and technical solutions) around host names, DNS and so on. … we’re trying to minimize the number of bits dedicated to each of the platforms
I fully understand your position.
All in all, editing /etc/hosts from the snap is technically possible and taking this approach makes sense for the product strategy (minimizing platform specific bits). On the other hand, directly editing /etc/hosts will alter the host system, potentially leading to inconsistencies, what doesn’t seem to completely fit with the snap philosophy to me, so it is hard to say
I think you could do a few things that are technically sound and would let you solve this.
You may continue to use the “atomic” rename of /etc/hosts.$RANDOM to /etc/hosts coupled with a fdirsync on /etc is as good as it gets on “typical” file systems and lets you replace the file in a way that concurrent racing readers will never see partial content. It doesn’t let you synchronise with other applications attempting the same trick (other writers)
You might explore using leases fcntl with F_SETLEASE (see man fcntl) which would at least notify you when other applications are attempting to read or (in your case) write to the file. Lease won’t let you handle the rename case, though.
You might explore using dnotify/inotify/fanotify which would let you see when others are messing up hosts file. We can make the appropriate interfaces grant the right permissions if this is something that other applications have not yet explored.
The conceptual problem is that we have long standing issue with /etc and, at some point, will have to re-visit that with a synthetic /etc that is provided to the snap application. When that happens we will surely associate it with a base change (our current policy of making breaking changes associated with new bases so that they don’t happen retroactively) and we have a number of interfaces that we could use to create the illusion that the application is accessing real host /etc. This problem is not something that you should be concerned about today. I’m merely mentioning because I want to follow up with a discussion of what is going on for snaps, at runtime in terms of /etc.
When snap applications are started, most of the file system of the host is stashed away and a new root file system is created as a combination of the base snap, the application snap, API file systems (/proc, /dev and /sys, with appropriate changes) and portions of the original host file system where we expect applications to write (/var/snap, /home, /root) - but also, critically, /etc. Any access controls that happen on top of that are handled with Linux security module (apparmor on distributions that are supported), with a small influence from seccomp and bpf depending on the type of access used. In this sense we can commonly “open up” access to specific files in /etc.
As a special exception, on ubuntu-core systems, the /etc directory is largely a read-only image, with only special places being either bind-mounted to writable locations (with consequences) or being symbolic links pointing to writable places (with other consequences). Depending on the type of access, this may break your scheme of updating /etc/hosts. Bind mounts make the mount point (file in this case) impossible to remove as well as impossible to be renamed or be renamed onto. This is exactly how /etc/hosts is handled on core systems for at least a few generations now. In addition the fact that /etc is really a read-only image (as can be confirmed by using stat -f /etc) means that you cannot create temporary files to be used for renaming (neither with the classic random name trick nor with the more modern O_TMPFILE flag for open). This limits your choice as now you really must rewrite the file without replacing it.
With all this context I would lean you towards a hybrid approach that includes, at the very least, a way to rewrite the file with ftruncate and write which would work everywhere. If you have resources you should explore using leases as a way to make that better. Leases allow you to write the file or drop the lease (and write the file later). See the manual page of fcntl for a discussion on how to use leases correctly.
Woah, this info is exactly what I needed to upgrade this mechanism to to feel technically-confident about it. Thank you @zyga a million. Thanks for the background info and a peek into future plans too - those really help with understanding the reasons and estimating the value of solutions for the future.
In this case, plan for us is now:
add fdirsync call to the code
add fcntl with F_SETLEASE to the code
rework the file rename/write functions to be a little more verbose about their hybrid approach (rename if possible, rewrite if not)
test on Ubuntu Core
As lease breakers from the F_SETLEASE may be blocked by the OS during the lock, I’ll probably won’t go the dnotify/inotify/fanotify route, but that may change after I spend some time with the code.
I’ll let myself to leave this thread open and come back here after I implemented said changes (think a couple of weeks with a current workload on other things).
Hi,
sorry for the long delay - we’ve squeezed an unexpected refactor of a large portion of code that took surprisingly large amount of time.
We’ve managed to implement most of the said changes in: core/husarnet/ports/linux/filesystem.cpp in our main repository (can’t post a link to Github for some reason)
TLDR: we’re now using renameat2 with RENAME_EXCHANGE flag as a first try (as it’s atomic replace done by the OS) and fcntl with F_SETLEASE(F_WRLCK) as a backup. Also we’ve changed the API so we accept a transformation function (old content → new content), instead of providing plain read/write methods for the rest of the code, which makes it much faster and concise.
We’re still testing those changes but, I believe, we now should be good to go with going forward with a snap.
So as a reviewer I am a bit wary of granting write access to /etc/hosts as this comes from the host file-system and thus any changes to this are propagated outside of the husarnet snap. However, given the entire purpose of this snap is to allow to create a peer-to-peer VPN with easily accessible hostnames for the peers AND that these hostnames are expected to be used by things outside of the snap I also am not sure of a better way to implement this feature. I also note that the upstream documentation GitHub - husarnet/husarnet: Husarnet is a Peer-to-Peer VPN to connect your laptops, servers and microcontrollers over the Internet with zero configuration. clearly mentions the use of /etc/hosts as well.
So +1 for me for use of and auto-connect of a system-files instance named etc-hosts for write permission to /etc/hosts for husarnet. Also +1 for auto-connect of network-control as this is clearly needed for a VPN application.
One quick question - is the snap still able to operate if this is not connected (ie if the user disconnects the interface)? ie. does it fail gracefully and still setup the VPN tunnels etc but just without the nice hosts entries?
Well, that’s a tricky part
Technically it will work without the system-files interface if network-control one is granted (and network-control is strictly required for obvious reasons). The tricky part is how will it work.
Based on snapd/interfaces/builtin/network_control.go at master · snapcore/snapd · GitHubnetwork-control interface is already granting permissions to /etc/hosts, but is not granting permissions for /etc/hosts.tmp, so we’re limited in ways update the files. We can’t use the metod that’s preparing new content in hosts.tmp file and then atomically swapping it… but we’re able to use the fnctl and locks method (or use the fallback for writing files directly if it fails for some reason).
As I think about it… maybe it’d be worth adding that /etc/hosts.tmp path to network-control interface and commenting it with a list of recommended ways of handling updates of those files? In that case special system-files instance wouldn’t be needed and more people would stumble upon a list od good ways of handling such changes I think.
Out of curiosity, is there a reason why both files (hosts and hosts.tmp) needs to be in the same directory to perform the renameat2? I didn’t find anything in https://man7.org/linux/man-pages/man2/rename.2.html
If there is no such a reason, maybe you could just create the hosts.tmp file in the snap temporal directory (or in $SNAP_COMMON if any mechanism to revert changes is used). That will remove the need for the special system files request.
Otherwise, I’ll be fine granting write access to the requested /etc/hosts.tmp. In this situation it may be worth considering to add this path to the network-control and add some documentation as you proposed. If this use case becomes more popular at some, I will probably be more inclined to remove write access to /etc/hosts from network-control and invoke snapd to perform the file update/substitution, ensuring it is done in the preferred way on all scenarios
oldpath and newpath are not on the same mounted
filesystem. (Linux permits a filesystem to be mounted at
multiple points, but rename() does not work across
different mount points, even if the same filesystem is
mounted on both.)
We’ve came across situations (mostly in containers, but also some complex ZFS setups, or even embedded ones) where /etc was mounted separately - so IMHO a safer rule of thumb for such replacement files is “same directory”. I can add a list of directories to try, but I honestly don’t know which places would make sense in this case as most of the temporary locations would be a separate mount to something like tmpfs.
+2 votes for, 0 votes against, granting auto-connect of interfaces network-control and system-files (write: /etc/hosts.tmp) to snap husarnet. I will begin publisher vetting.
)
Technically it will work without the