Standard for bootstrapping network (on Raspberry Pi and similar devices)

[vpetersson]

I have spent a fair amount of time recently thinking about bootstrapping network configuration in IoT devices (in particular headless devices that do not allow system access).

Traditionally, most devices fall into one of the two:

• DHCP (or a static hard coded IP), and then a web interface for configuring advanced network (think switches, routers and similar)
• Ad-hoc WiFi which uses a captive portal for configuring the network (think Chromecast etc)

The former works well for wired devices where you have a tech savvy user and where you only bring up one device at the time. The latter on the other hand works great for consumer devices where you usually only configure one or two.

The premise of Core is to be the foundation to build on for IoT products. Networking is of course a fundamental part of modern IoT. Moreover, it is highly desirable in many situations to not let the customer access the actual device, but rather have it fully locked down and just receive updates. That wouldn’t necessarily rule out either of the two above, but neither of them are very scale-able if you’re deploying say 100+ devices.

A few years ago we added support in Screenly for reading a file called network.ini on the boot partition of the Raspberry Pi. We also created a tool (wizard.screenly.io) for generating these files. The idea was simply that you can generate this once and then just copy it in every time you need a new file.

Raspbian recently adopted a similar model, but using wpa_supplicant.conf (and NOOBS does something similar IIRC). If the file exist on the boot partition, it will copy it into the system during the boot process. The problem with this approach is that it only takes into account wireless configuration, while our solution also added support for things like DNS and NTP (as well as Ethernet configuration).

We are right now in the process of adopting this into our Core migration, but that got me thinking. This is a very common situation, and it would make sense to have this in the public domain instead of having everyone reinventing the wheel.

What I’m thinking is that we would create a rather universal configuration file that can define things like:

• Ethernet configuration (static, DHCP)
  • We’d also need support for 802.1x etc at some point, so that’d need to be considered
• Wireless configuration [1]
  • In addition to regular WPA/WPA2 etc, we’d also need to support WPA2 Enterprise
• Hostname and domain config
• NTP servers
• DNS servers
• Arbitrary application configs (such as API keys)

This would then be combined with some lookup logic during the boot:

• Priority 0: USB mass storage drive
• Priority 1: boot partition

If neither of the files exist, then silently skip it and use DHCP on Ethernet. The benefit of both being able to read it from USB and boot partition is that it covers both the use case of a box product (i.e. manufactured in large quantities) and home users who flash out themselves. Moreover, there is an interesting discussion going on over in Etcher’s Github repo about being able to make this part of the SD flashing workflow.

Perhaps out of the scope of this, but in a perfect world, we’d also add a final step that would kick in if the system boots up and there is no active connection, we’d spin up a hotspot with a captive portal for network configuration with simple options (i.e. no WPA2 enterprise etc) [2].

To date there is no such standard (to my knowledge). It would make a lot of sense to create such standard and we’re happy to put our work in the public domain. If the config file can be standardized, it would be rather straight forward to do clients that can consume the file both in Raspbian and Core.

File format

The two obvious candidates for formats would be YAML and JSON. Both have pros and cons. The primary benefit with YAML would be that it is more human readable and that it’s fairly intuitive to embed something like a snippet in it (say if we want to embed wpa_supplicant.conf for instance).

To wpa_supplicant or to not wpa_supplicant

There’s already an API exposed for Network Manager through Core. For simple network configurations, that’s fine, but if you were to do something more complicated, such as this example, that would be rather challenging.

As such, there is an argument to be made that it would be possible to pass on full wpa_supplicant files as part of the standard. This could be embedded in the configuration file. On Raspbian that would be easy to simply write to disk, but on Core unfortunately that is not possible to my knowledge as Core 16 doesn’t use wpa_supplicant.

Footnotes

[1] This is really where wpa_supplicant.conf excels. It’s widely used and many users simply copy their files across platforms. Perhaps we would need support for consuming wpa_supplicant.conf files too to cover all user cases, but that will be challenging on Core since it doesn’t support it (afaik).
[2] This gets a bit tricky since we need to have some kind of logic determining if the device is just temporarily offline (e.g network outage) or if it actually does require network reconfiguration.

[kyleN]

@vpetersson

as Core 16 doesn’t use wpa_supplicant.

I think Core does include wpa_supplicant. Here’s a recent build manifest of Core deb pkgs: https://launchpadlibrarian.net/288335774/core_16.04.1_amd64.manifest

[vpetersson]

Good catch, Kyle. Do you know if it is possible to feed it a config directly?

[kyleN]

No I don’t know that yet.

[kyleN]

@vpetersson

To date there is no such standard (to my knowledge)

Perhaps it would be useful to extend the netplan file semantics to include any additional bits? It is already the common config format for various net config backends (networkd, network-manager).

[awe]

Ubuntu used to support static configuration of networking devices via /etc/network/interfaces file and ifupdown. Both have been deprecated in recent releases in favor of netplan, which is the default mechanism used on Ubuntu Server and Ubuntu Core to configure networking.

That said, in the case of Core, as systemd’s networkd only supports basic networking (ie. no modem support), many Ubuntu Core images include the network-manager and modem-manager snaps. On these images, control of networking is ceded entirely to NetworkManager, and netplan isn’t used for any network configuration.

It is possible to use netplan to load configuration into NetworkManager using netplan’s NetworkManager renderer, however to date, we haven’t used this type of setup much, if at all. One potential issue is that the NM renderer generates NM connection files in /run, which means they’re temporary, and thus are re-generated on every boot. Connections created via nmcli or the direct via the DBus interface are persisted in one of the writable areas of the NM snap. In theory, NM should be able to handle reading system connections from multiple directories, but I’ve never run NM configured in that way. Another potential for trouble would be that changes made to a netplan generated NM connection might not persist across reboot. So, while this may be do-able, it will take some investigation and potential changes to the network-manager snap and/or netplan.

Regarding the eduroam configuration, this sounds like a bug in NM that could be fixed.

The core snap does include wpa_supplicant, however there also is a wpa-supplicant snap which contains a fix for WoWLAN from S5. If/when we land this patch upstream, the wpa-supplicant snap may be deprecated.

Regarding the auto startup of an AP that can be used to configure a network connection if none is detected after boot, this can be quite tricky, as not all WiFi drivers support AP mode, and some drivers require new firmware to be loaded for AP mode. I also agree that trying to determine if the device is temporarily offline vs. requires network reconfiguration is easier said than done…

[vpetersson]

Thanks, @awe

Ubuntu used to support static configuration of networking devices via /etc/network/interfaces file and ifupdown. Both have been deprecated in recent releases in favor of netplan1, which is the default mechanism used on Ubuntu Server and Ubuntu Core to configure networking.

I’m all for using Netplan if that will do the job. In fact, that is what we are doing right now. My issue with Netplan is that it is falling short as far as support goes. It doesn’t support any advanced network configuration for WiFi (such as WPA2 Enterprise etc). That’s why we’re looking into this. We really just want to take the path of least resistance while still meeting our requirements.

It is possible to use netplan to load configuration into NetworkManager using netplan’s NetworkManager renderer,

Could you elaborate on this? If it would be possible to pass on NM configurations directly in Netplan (similar to what I described in the config), that would do the job. The only thing I can find is this stanza:

network:
  version: 2
  renderer: NetworkManager

As far as I interpret the description “This will make NetworkManager manage all devices”, it will merely disable Netplan and there is no way to pass in additional arguments, but perhaps I misinterpret it.

Right now however, the only way to accomplish this is to have some kind of abstraction (such as described here) and then have a snap that would parse it and pass on commands to nmcli.

Regarding the eduroam configuration, this sounds like a bug in NM that could be fixed

Oh I didn’t read the post itself in details. I merely trying to illustrate a more ‘advanced’ (but not all that rare) configuration that would not be possible in Netplan.

Regarding the auto startup of an AP that can be used to configure a network connection if none is detected after boot, this can be quite tricky, as not all WiFi drivers support AP mode, and some drivers require new firmware to be loaded for AP mode.

This is true in general, but in our particular case that wouldn’t be an issue as we run on the Raspberry Pi.

[kyleN]

related bug/feature request: https://bugs.launchpad.net/netplan/+bug/1739578

[kyleN]

@vpetersson

As far as I interpret the description “This will make NetworkManager manage all devices”, it will merely disable Netplan and there is no way to pass in additional arguments, but perhaps I misinterpret it.

I think the meaning is that the netplan configuration is passed to Network Manager, not that netplan is disabled.

[ogra]

Sorry for chiming in so later here … you can indeed just create a snap with a daemon script in it that writes a non-network-manager netplan yaml to /etc/netplan by using the “network-setup-control” interface in your snap …

note that this interface will not allow “netplan generate” and “netplan apply” which means your script snap should also use “shutdown” interface and issue a reboot after putting the config in place (a reboot will call netplan generate/apply during boot)

wpa-supplicant is shipped by default in the core snap and is directly used by netplan if you ship a proper config like:

network:
  version: 2
  wifis:
    wlan0:
      access-points:
        MYESSID: {password: MYPASSWORD}
      addresses: []
      dhcp4: true

[vpetersson]

@ogra

wpa-supplicant is shipped by default in the core snap and is directly used by netplan if you ship a proper config like:

That’s super interesting, @ogra. How would you apply a more complex wpa_supplicant.conf file, such as this:

ctrl_interface=/var/run/wpa_supplicant
ap_scan=2
fast_reauth=1
network={
        scan_ssid=1
        ssid="ssid"
        key_mgmt=WPA-EAP
        eap=PEAP
        identity="identity"
        ca_cert="/etc/cert/pca3-g5ss.cer"
        phase1="peaplabel=auto peapver=0 "
        phase2="auth=MSCHAPV2"
        password="passwd"
}

(from here)

[ogra]

@vpetersson

How would you apply a more complex wpa_supplicant.conf file

well, if the documentation at https://wiki.ubuntu.com/Netplan/Design is not sufficient, this would probably be a question for “cyphermox” on the #snappy IRC channel (he does not seem to have an account here in the forum), i’m not sure how complex you can go with the plain wpa-supplicant setup with todays netplan, i have only used it yet with standard stuff …

[vpetersson]

Ah, no I mean using the wpa_supplicant.conf support you mentioned. I’m aware that this is currently not supported in native Netplan. Maybe I misunderstood you.

When you said wpa-supplicant is shipped by default in the core snap and is directly used by netplan if you ship a proper config like, you mean that it is only using wpa-supplicant behind the scenes, not that you can provide a custom file?

[ogra]

well, netplan is the preferred config tool for all networky stuff on core … in that way it should be able to configure everything you need even without NM installed (especially on low end embedded headless devices) but i dont think complex wpa-supllicant setups had much focus yet… the above example definitely works though, for more complex bits we might need additions in netplan (which is why i pointed to cyphermox, he is the upstream for it)

[awe]

@vpetersson Sorry the delayed response, I was off for a few weeks over the holidays.

@kyleN That’s correct. The fragment instructs netplan to use the NetworkManager renderer, but then no additional configuration is specified, so netplan doesn’t generate anything further. On systems where we use this fragment to cede control to NM, we create system connection files within the network-manager snap under $SNAP_DATA.

If you were instead to place the following in your netplan.yaml file, then on boot, ‘netplan generate’ would create a NM system connection under /var/run which then should be picked up by NetworkManager. Note, as I previously mentioned, this hasn’t been well-tested with the network-manager snap, so your mileage may vary.

network:
  version: 2
  renderer: NetworkManager
  wifis:
    wlan0:
      access-points:
        MYESSID: {password: MYPASSWORD}
      addresses: []
      dhcp4: true

That said, as you’ve pointed out, netplan only supports very simple WiFi configurations, so it would need to be extended to support WPA Enterprise connections. This could be done for one or both renderers.

Regarding the wpa-supplicant that ships with core, it might be possible to create a custom configuration file, as this is how netplan itself works (it writes a wpa-supplicant conf file to /var/run/netplan), however I’m not sure if a snap can write to this directory. Another approach that should work would be to dynamically modify wpa-supplicant’s configuration using its DBus interface. This is how NetworkManager controls wpa-supplicant.

Also, I’ve reached out to cyphermox directly about the bug referenced above.

[vpetersson]

Hey guys,

Just an update from my end. We have added Netplan support to wizard.screenly.io now (or rather, it’s in this PR). That should let anyone easily create a Netplan file. We also plan to open source our snap that allows you to provide the system with a Natplan file using a USB stick.

Again, this only solves basic network configuration, and we’re really keen to extend that with more complex configurations as we have had a lot of requests for it.

[awe]

We’ve discussed with the netplan maintainer, and he said that he would add WPA Enterprise to the roadmap. He also mentioned that code contributions are always welcome, so if you’re feeling adventurous, the code can be found here:

Git : Code : netplan

The code is written in C, and is pretty straightforward. Adding WPA Enterprise support involves adding YAML fields for the wireless attributes used to describe a WPA Enterprise connection, and logic to map, and then write these fields to the corresponding wpa_supplicant or NetworkManager configuration files.

For more information see:

Netplan - Ubuntu Wiki

[ogra]

@awe

it writes a wpa-supplicant conf file to /var/run/netplan), however I’m not sure if a snap can write to this directory.

at least not via the current network-setup-control interface (only /etc/netplan is writable for the netplan part):

canonical/snapd/blob/master/interfaces/builtin/network_setup_control.go

[vpetersson]

@awe

We’ve discussed with the netplan maintainer, and he said that he would add WPA Enterprise to the roadmap.

That’s awesome. Any ideas about the time frame?

@awe

He also mentioned that code contributions are always welcome, so if you’re feeling adventurous, the code can

Unfortunately I don’t have anyone I can pull and assign to that right now.

[awe]

I was talking about the networkd renderer, and I actually got the path wrong, there’s no leading /var, the wpa-supplicant conf file gets written to the dir /run/netplan by the function write_wpa_conf():

networkd.c\src - netplan - [no description]