How to preconfigure custom image?

device
1

Following the discussion in Gadget providing serial-port slot, I’m building an image with a custom gadget. So far so good, but I have a few questions about what is and is not possible in terms of pre-configuration of the image. This is all with a view to allowing hassle-free (read: no console-conf) provisioning of devices.

Namely, is it possible to:

  • Have the device skip interactive configuration on first boot. Specifically
    • Use a pre-defined default network configuration (e.g. straightforward DHCP on Ethernet port). It looks like there’s a plan to enable this via netplan, but that’s currently on hold (per Standard for bootstrapping network (on Raspberry Pi and similar devices))
    • Have a particular SSO user signed into the device on first boot. For instance, in order to allow access to private snaps
    • Possibly related to (but not dependent on) the above, create a user account with a given authorized SSH public key that allows console access.
  • Set a given environment variable to a certain value. I’m not sure whether creating a /system-data/etc/environment file within the image is a great idea. Or whether setting an environment variable in the gadget snap will have any effect whatsoever.
  • Have a particular plug in a pre-installed snap automatically connected to an interface provided by the gadget. From what I understand, the “proper” way to achieve this is to request auto-connection of the snap via this forum, and to also release the gadget to the store and have it approved (based on https://kyrofa.com/posts/ros-production-create-an-ubuntu-core-image-with-our-snap-preinstalled-5-5). Is there maybe a more straightforward way?

From Cloud-init with netplan it sounds like using cloud-init may be a sensible way to achieve some of this, but I have not looked further into that. Would that be considered best practice? Could hooks also potentially play a role here, such as the prepare-device gadget hook?

Apologies for the multitude of questions, and big thanks in advance for any pointers!

3 Likes
2

For a pre-defined network config you could have a snap shipping a simple script that writes out netplan config and has the network-setup-control interface connected … or as you already noticed you could use cloud-init here.

To create a SSO user (who also has ssh access) there are system-user assertions. the system will attempt to auto import them on boot if you put them on a usb device that you attach:

https://docs.ubuntu.com/core/en/reference/assertions/system-user

for /etc/environment there are a few core snap configs (proxy and the like) … if you want to go beyond this you will have to apply some hackery. i.e. you could ship an initrd script snippet in your gadget to modify the file befored boot using:

(here is an example where i add a splash screen to a gadget: add splash support · ogra1/pi3-gadget@8fa1658 · GitHub, look particulary at the install scriptlet in snapcraft.yaml and the changes to uboot.env.in)

there is:

3

Thanks a lot for these pointers, very helpful! I’ll try them out and report on results. Just a couple of thoughts at this point:

Indeed, this looks like it should do the trick! Any idea whether I can add the system-user assertion to the image prior to flashing, in a way that will get it to be picked up on boot (rather than needing for it to be on a USB stick at boot)? My use case here is that I’d like to send someone an image that they can flash to an SD card themselves - and have that result in an operational device, without them taking additional configuration steps.

Very cool! In this context, any suggestions as to what the best way is to change (or add to) the default text that’s displayed on boot? (e.g. if I want to display some more device info)

4

there is /etc/update-motd.d/ … you can put a 01-vendor (or name it as you like) file in that dir to show additional info…

5

Thanks again! Based on the above info I managed to tick off pretty much all my main requirements - primarily by editing files in the tree inside the image that was produced by ubuntu-image. I imagine some of this is not-quite-best-practice, but I’ll share my approach here with the intent of maybe helping others and getting some feedback on what can be improved.

Thanks to @kyleN for providing some scripts that facilitate several of these steps, at GitHub - knitzsche/core-build-scripts

Network configuration

Simply placing a netplan file inside system-data/etc/netplan/ did the trick. It gets picked up on first boot, and the device for example successfully connects to WiFi.

By the way, I did not actually see what further benefit I would derive from using cloud-init, and did not end up using it here.

Creating a system user for SSH authentication

As suggested by @ogra, creating a system user assertion and then applying it by copying it to a USB stick that’s inserted during first boot (or at any time, really) allowed to me to get more or less what I need.

I created the assertion by following the steps in https://docs.ubuntu.com/core/en/guides/manage-devices/. The make-system-user script used for that example sets up a user with password authentication instead of SSH key authentication, but that’s fine for now. Putting the model together manually (rather than using the helper script) clearly allows you to also supply a public key.

Question 1: Is it possible to incorporate a system user assertion into an image directly, rather than installing it from a USB stick?

Question 2: It’s not totally clear to me how the system-user assertion is actually used for SSO authentication, in a way that for example allows the downloading of private snaps. The created system user doesn’t appear to have access to private snaps published by the authority/brand that signed the assertion. But that’s not currently a high priority for us, so am happy to let it be for now.

Setting environment variables

Creating a system-data/etc/environment file in the image got the job done just fine.

Auto-connection of interfaces

This was a relatively tough one; while there is concept for how this would be done, it’s not clear to me what the current stage of implementation is (Interface connection from gadget in firstboot). In the end I did something pretty ugly, which was to create a systemd service file that stipulates the execution of commands along the lines of /usr/bin/snap connect [<snap>:<plug>] [<snap>:<slot>], and activate it (i.e. create symlink to it under system-data/etc/systemd/system/multi-user.target.wants on the image).

If anyone has any suggestions as to how to do this in a less ugly way, I’d love to hear them.

Bonus: display a message on boot screen

That doesn’t quite seem to do it. On my system I just get /etc/issue followed by a plain login prompt:

My plain boot screen

IMG_0428
IMG_04283264×2448 1.24 MB

Of course MOTD is displayed once I log in, but what I’d like to do is have the output of a particular bash script be shown on the bootup screen already. Something more like what is shown on boot once console-conf is complete

Boot screen after SSO authentication

IMG_0427
IMG_0427816×612 564 KB

…but with content that I define. Is this achievable? Am I missing something with respect to MOTD?

1 Like
6

sorry for the late reply (i’m travelling a lot atm) … did you consider taking a look at the source of subiquity (the source name of “console-conf”) i think the latter bit is shipped from there …

7

This script was recommended to me for core image setup.

I was not able to get the auto import to work as the script does. The user did not exist on boot even though I had a user assertion in the same location. But with the user-assertion already existing on the device is was possible to run a snap ack on the user assertion then snap create-user --known --force-managed --sudoer in a system service that runs on first boot creating the user.

8

WOW! This is definitely the most wrong thing you can do … injecting random stuff into the writable partition will eventually bite you back (how would you update this in case you discover any bug in the injected files). If you actually use something like this you should at least make sure to ship the payload data in a snap and simply have a script that copies the snap content in place instead (so a snap update and a subsequent run of the copy script could save your butt when there is any issue with the scripts or files you ship)

1 Like
9

That is how > 90% of anything like this is handled. But it doesn’t solve every solution. I’m only using things like this for first time config which is what this question is about. I would recommend the same copying payload data from a snap in most cases. The only thing I place in the writable partition are things that are first run, one time configs that is they needed updated I would generate a new core image.

One of the reasons I’m doing this is to disable console-conf. The initial network config offered by netplan isn’t capable of the configuration required and in a manufacturing environment removing any manual process, read manual as error prone, is desirable.

Disabling console-conf requires me to add users in another method though.

10

Yeah, i understand that this is an interim solution (and it is a shame that we still do not have a proper answer to these issues). for console-conf i filed:

Regarding the users, i understand that for development you actually need a way to log in to your robot, but do you plan to actually keep this in production systems too ?

11

The user is deleted at the end of the manufacturing process. Access is needed for calibration of physical properties of the system during manufacturing (cameras, sensors, etc.), but the system is then completely headless afterwards.

12

Thank you both for all the additional input on this!

I know this wasn’t in direct response to what I said, but it’s still a stark reminder that most of what I did to get the functionality I was after is “the most wrong” way to go about it. I understand that setting up the system fully via snaps is the right way forward. Yet, at this point in time that’s still not feasible for a few of the above requirements - such as the auto-connection of interfaces. With respect to that example, I understand that the proper way to do it in the long run is to (a) get the gadget approved in the store and (b) get the interface auto-connections on the snap approved also - but it would be nice if there was a more direct way for custom images.

Thanks for the tip. Will try to go down this route whenever possible.

13

Maybe I missed something when reading through the above, but is there a solution to this?

I would like to build a custom image that will have:

  1. My own OpenVPN setup (with a custom certificate just for this image)
  2. My own custom ~/.ssh/known_hosts

Is there any way to achieve this?

14

I’m not yet aware of any further developments (i.e. a better way of achieving this than what I mentioned above). Specifically to your questions:

You could install an OpenVPN snap of course (possibly your own pre-configured version), though I’m not sure what the best way to do device-specific certificate provisioning would be. If you find out I’d be interested.

In order to put that in the user’s home directory, you’d first need to create said user after booting the device, via the “system-user assertion on USB stick” mentioned above. I imagine it’s possible to include some scripts to automatically do that as part of the assertion, but am not sure.

Another solution could be to pre-populate the central /etc/ssh/known_hosts file rather than the one in the user’s home directory. And in case you actually mean authorized_keys rather than known_hosts, apparently you can store those in a central location also: https://serverfault.com/questions/313465/is-a-central-location-for-authorized-keys-a-good-idea

15

It took me a long time to figure out how to accomplish what @svet was trying to do without directly mounting and editing the image.

Cloud-init was definitely the way to go, as you can create the user, configure the network and disable console-conf (by creating “/var/lib/console-conf/complete”) without having to create your own gadget.

The docs were a bit inconsistent on how to do this, and many of my google searches brought me back here. So I’ve created this project as an example, in hope it helps someone else:

In short - adding cloud.conf to your own gadget doesn’t work. Instead you need to pass your cloud.conf file to ubuntu-image ($ sudo ubuntu-image -O image --cloud-init cloud.conf your_model.model)

You can use cloud-init to disable console-conf by using the bootcmd option (runcmd does not work, as it happens too late).

My whole cloud.conf file is here (the first line is not a comment, cloud-init expects it to actually read “#cloud-config”):

#cloud-config
users:
  - name: craig
    gecos: Craig Ulliott
    homedir: /home/craig
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: users, admin
    lock_passwd: true
    shell: /bin/bash
    ssh_authorized_keys:
      - your public key here
bootcmd:  
  - mkdir /var/lib/console-conf
  - touch /var/lib/console-conf/complete
3 Likes
16

note that creating a user is fully supported via system-user assertions OOTB, you just plug in a USB key to your device and the system will automagically import the assertion and create a user …

that said, the network setup is a bit trickier and there cloud-init is probably the easiest way to go if you do not maintain your own gadget… if you do though, you can create a snap with the appropriate interfaces connected by the gadget to allow auto-importing of a netplan.yaml from USB key as well … like in:

there I use the udisks2 snap from my config snap to import a file called netplan.yaml and overwrite the default config … this is helpful for appliance images where you have no user at all and want to still be able to provision a fleet of IoT gateways or smart displays by going around and plugging in a USB key for the initial confg.

17

Hi @svet have you had any luck with Ubuntu Core 18? I am also planning on releasing an Ubuntu Core headless solution to several hundreds gateways.

Unfortunately due to company policies I cannot post any snaps publicly, and we have collectively decided to not have a privately hosted snap store ($15k/year base cost!!). So as work arounds I too have been injecting systemd services to auto connect snaps to slots (gadget snap solution requires snap IDs which require snaps to be placed in a store)

I have found that on Ubuntu Core 18, only when injecting these systemd services, that the system doesn’t fully boot and the console-conf is never started. I see that a service named snapd.core-fixup.service is started during boot and says that the system permissions/ownership have been modified (even though I changed the ownership to root when mounting the file system).

For the most part I agree with @ogra, on doing such customizations, however I too also struggle with this same type of issues on headless devices that will be executing ubuntu core in the thousands.

1 Like
18

…

…well, did you do some math how much man-hours it will cost you to develop and maintain the infrastucture for managing thousands of gateways securely ?

The brand store is far more than just a store, it enables you to provision and de-provision devices as needed, add/remove users or to integrate a management snap with it to enable/disable connections dynamically, install/remove snaps, control updates etc …

Assuming your self built solution would/should provide similar features (plus hosting costs, ongoing maintenance, bugfixes, regular security audits etc), I imagine you probably get away with a similar price in the end … just that you burn your work hours in building/maintaining your own store instead of being able to focus on development of your management tools.

i’m surely biased here given this store pays part of my salary :wink:

…but if i’d want to be a gateway company and not a hosting company i’d definitely consider twice if i would use an already widely used solution that i can hold someone liable for if something goes wrong, one that gets regular security reviews and that was actually designed for what i want to do with my gateways in the end.

There are a few rigado webinars around that show the store in action and what abilities it offers to allow a company to be able to simply focus on their product instead of the infrastructure. Rigado manages already thousands of devices with it, perhaps thats worth a look :slight_smile:

19

The base cost is tied to a max. of 1000 devices ATM I guess. Especially for small startups which are going to have way less devices in the field and are potentially able to maintain devices in the field by sending service technicians to customers (which customers have to pay as well) the cost/benefit is open for discussion / hard to argue in front of management.

Especially from a technical perspective a private store is super cool :wink:

20

Never thought configuring an image could be that complicated