Adding files to writable-paths in a UC20 custom image

I want to add some extra files to a custom image in one of the writable-paths at image build time, like /etc/sysctl.d

After a bit of googling around, it seems that is normally done by creating the image, then mounting the image partition labelled writable, and dropping files under the in the writable-path locations under /system-data/.

Hopefully someone can correct me if I got that wrong at all :slight_smile:

However, it looks like for UC20 the process is different, as the image file created by ubuntu-image does not contain the partition labelled writable. It seems like that partition is only created on first boot.

Again, please correct me if I have not got that correct.

So the question is, how do I create drop in files to writable-paths when create a UC20 image ? I guess I could use cloud-init to write files, but sticking entire files in cloud-init as blobs seems a bit messy. It feels like there must be a better way! :slight_smile:

Any links to examples would be awesome. Thanks!

Cheers, Just

1 Like

While this is correct in that this is somewhat “standard practice”, as you have noticed this is not supported on UC20 because the “writable” partition is now created dynamically during install mode rather than at image-build time.

It really depends on what files you want to add, it is difficult for us to support allowing arbitrary files to be added to the writable partition at image build time as this can change behavior of the system in arbitrary ways that make it difficult to maintain over the course of an operating system release as we add new features, etc. What we have done for some of these kinds of customizations is to expose them as system settings that you can set using the defaults keyword in the gadget.yaml of your gadget snap. See for example log persistence, disabling SSH, enabling kernel debug log level, etc. in the docs: https://snapcraft.io/docs/system-options

IIRC there was a long standng open plan to support setting sysctl values via snap set system ... (not sure where this stands though)

Thanks for the info.

Specifically regarding sysctl, there have been many times in the past where tuning has been essential for us to overcome various issues. Therefor being able to set arbitrary sysctl params is important to us, and if that was exposed via snapd that would be very useful for us. If the feature @ogra mentioned is being tracked somewhere, please share a link.

Another specific example. If we want to add a management user that is fully configured as part of the image, not via console-conf and not via a user assertion on a USB flash drive, how do we do that if we can’t add a user assertion to the image at create time ? Is cloud-init the only way with UC20 ? Any links to example would be useful.

On a more general note, I understand the reasoning for not wanting to support adding arbitrary files to the image. But on the other hand, as creators of an appliance image we seem to have lost quite a bit of flexibility. It would be nice to have some kind of middle ground where it’s clear that such tinkering is not supported, but people that understand what they are doing can accept the risks and proceed. Just some thoughts…

Cheers,
Just

If you have a list of which ones would be useful, it is easier for us to implement specific support for specific features than providing an arbitrary interface to sysctl due to the above problem about making arbitrary changes and not being able to maintain that over a long period of time.

You can also create users with a snap using snapd-control using the snapd REST API, see POST to /v2/users at https://snapcraft.io/docs/snapd-api#heading--users.

The issue with such middle grounds is that their usage always widens over time, and it is much harder to “close” features out and drop support for them over the lifetime of an OS than it is to start with a small scope and open up specific things that we know we can support as folks come to us with real world use-cases.

I get that. We could probably come up with a list as we go.

But, rightly or wrongly, what can happens is, we have some kind of production incident relating to some kind of ceiling that wasn’t found in testing [ for a random example, something related to high TCP connection churn, and we need to alter net.ipv4.tcp_fin_timeout ]. In that situation, waiting for support to be added is not ideal.
Arguably, that also means testing is not rigorous enough, and I may not disagree - but stuff happens and we may need to respond in that way.

It may be that we have to just accept this as a constraint of leveraging UC, and that may be fine.

From a “new customer” point of view, it has taken me by surprise, as all my previous research indicated it was possible and I didn’t realise it was changing with UC20. I’m not trying to argue this is wrong, but giving some feedback from my point of view as a “new guy” and new customer.

Thanks! That’s useful.

At the end of the day, we are just more used to being masters of our own destiny in every aspect of an OS, but clearly that is a mindset that will need to change to some extent as we consume UC. There are a lot of strong features with UC that we want to leverage, to get those we are going to have to do some compromising of course.

Cheers,
Just

The other thing to realize is that while some system configuration in /sys needs to be managed from a system standpoint by using snapd configuration, some other things like what you have mentioned can actually be managed/altered by your own snaps on the system using appropriate interfaces, and so for devices it may be better for your snaps to configure this themselves without exposing specific, snapd system-wide configuration for it. For example this particular setting can be set if you plug network-control. In general we actually prefer to be able to grant snaps the ability to configure things through interfaces rather than through snap set configuration because the security policy is easier to manage than the system configuration, however some configuration is too powerful or too arbitrary that we need it to be mediated by snapd via snap set system foo=bar, etc.

Aha! Assuming you are talking about net.ipv4.tcp_fin_timeout still - from just reading https://core.docs.ubuntu.com/en/reference/interfaces/#interfaces I still had no idea that it granted some /sys access, or even more specifically net.ipv4.tcp_fin_timeout

Are the interfaces documented in more detail anywhere [ other than source code ] ?

Cheers,
Just

Well the interfaces are meant to be named to reflect their semantic usage, so for your example setting the net.ipv4 sysctl setting is generically “controlling some networking aspect” and thus fits in there. Note that that specific path is not even mentioned in the source code, it is instead a more generic path that is allowed as part of the interface:

https://github.com/snapcore/snapd/blob/877e78c3d55f60eab510f97f2214acc985cd6fab/interfaces/builtin/network_control.go#L90

so just grepping for your specific path is not useful in the codebase usually (though sometimes it is).

Instead what’s more useful is to usually just try to set all the settings or do all the things you want to do to configure the device from a snap that is confined with devmode, and then look at the denial logs and you can use Install snappy-debug on Linux | Snap Store to try and find the right interface, or you can come here to the forum with the denial messages and ask for help finding what interface would allow the denials you see (though when your snap is in devmode, they are not actually denied, they are instead allowed but “audited” so you see that it would have been denied if it was in strict confinement).

See also the “Interfaces” section on Debugging building snaps | Snapcraft documentation, but that basically is just a nicer explanation of what I tried to say above :smile:

Hopefully that helps clear things up a bit.

1 Like

It does - thanks for taking the time :slight_smile:

Cheers, Just