Cloud-init on Core20 seems to break and not execute my gadget cloud-init config

Hello there, from this serial console log, it seems that cloud-init does start executing when I add my own cloud.conf file in my gadget, but it does not execute my cloud-init config, but seems to break before it. See below:

[   40.108341] cloud-init[894]: Cloud-init v. 23.1.2-0ubuntu0~20.04.1 running 'init' at Wed, 17 May 2023 10:01:01 +0000. Up 36.70 seconds.
[   40.132282] cloud-init[894]: ci-info: +++++++++++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++++++++++++
[   40.156845] cloud-init[894]: ci-info: +--------+-------+------------------------------+---------------+--------+-------------------+
[   40.180249] cloud-init[894]: ci-info: | Device |   Up  |           Address            |      Mask     | Scope  |     Hw-Address    |
[   40.200283] cloud-init[894]: ci-info: +--------+-------+------------------------------+---------------+--------+-------------------+
[   40.224597] cloud-init[894]: ci-info: |  can0  | False |              .               |       .       |   .    |         .         |
[   40.248566] cloud-init[894]: ci-info: |  eth0  |  True |         192.168.8.26         | 255.255.255.0 | global | f8:dc:7a:a0:c3:96 |
[   40.272666] cloud-init[894]: ci-info: |  eth0  |  True | fe80::fadc:7aff:fea0:c396/64 |       .       |  link  | f8:dc:7a:a0:c3:96 |
[   40.296615] cloud-init[894]: ci-info: |  eth1  | False |              .               |       .       |   .    | f8:dc:7a:a0:c3:97 |
[   40.320303] cloud-init[894]: ci-info: |   lo   |  True |          127.0.0.1           |   255.0.0.0   |  host  |         .         |
[   40.344328] cloud-init[894]: ci-info: |   lo   |  True |           ::1/128            |       .       |  host  |         .         |
[   40.368251] cloud-init[894]: ci-info: +--------+-------+------------------------------+---------------+--------+-------------------+
[   40.392317] cloud-init[894]: ci-info: ++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++
[   40.412257] cloud-init[894]: ci-info: +-------+-------------+-------------+-----------------+-----------+-------+
[   40.436325] cloud-init[894]: ci-info: | Route | Destination |   Gateway   |     Genmask     | Interface | Flags |
[   40.456262] cloud-init[894]: ci-info: +-------+-------------+-------------+-----------------+-----------+-------+
[   40.476580] cloud-init[894]: ci-info: |   0   |   0.0.0.0   | 192.168.8.1 |     0.0.0.0     |    eth0   |   UG  |
[   40.496318] cloud-init[894]: ci-info: |   1   | 192.168.8.0 |   0.0.0.0   |  255.255.255.0  |    eth0   |   U   |
[   40.516287] cloud-init[894]: ci-info: |   2   | 192.168.8.1 |   0.0.0.0   | 255.255.255.255 |    eth0   |   UH  |
[   40.536606] cloud-init[894]: ci-info: +-------+-------------+-------------+-----------------+-----------+-------+
[   40.556584] cloud-init[894]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[   40.572741] cloud-init[894]: ci-info: +-------+-------------+---------+-----------+-------+
[   40.592279] cloud-init[894]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[   40.612255] cloud-init[894]: ci-info: +-------+-------------+---------+-----------+-------+
[   40.636308] cloud-init[894]: ci-info: |   2   |    local    |    ::   |    eth0   |   U   |
[   40.652331] cloud-init[894]: ci-info: |   3   |  fe80::/64  |    ::   |    eth0   |   U   |
[   40.672287] cloud-init[894]: ci-info: |   4   |  multicast  |    ::   |    eth0   |   U   |
[   40.692275] cloud-init[894]: ci-info: +-------+-------------+---------+-----------+-------+
[   40.708290] cloud-init[894]: 2023-05-17 10:01:05,330 - activators.py[WARNING]: Running ['netplan', 'apply'] resulted in stderr output: Warning: The unit file, source configuration file or drop-ins of netplan-ovs-cleanup.service changed on disk. Run 'systemctl daemon-reload' to reload units.
[   40.744360] cloud-init[894]: Failed to connect system bus: No such file or directory
[   40.764605] cloud-init[894]: Falling back to a hard restart of systemd-networkd.service
[   40.780312] cloud-init[894]: 2023-05-17 10:01:05,514 - schema.py[WARNING]: Deprecated cloud-config provided:
[   40.800308] cloud-init[894]: growpart.mode:  Changed in version 22.3. Specifying a boolean ``false`` value for ``mode`` is deprecated. Use ``off`` instead.
[   40.824288] cloud-init[894]: 2023-05-17 10:01:05,515 - schema.py[WARNING]: Invalid cloud-config provided: Please run 'sudo cloud-init schema --system' to see the schema errors.
[   40.848291] cloud-init[894]: 2023-05-17 10:01:05,756 - cc_growpart.py[WARNING]: DEPRECATED: growpart mode 'False' is deprecated. Use 'off' instead.
[   42.588899] cloud-init[894]: Generating public/private rsa key pair.
[   42.604322] cloud-init[894]: Your identification has been saved in /etc/ssh/ssh_host_rsa_key
[   42.624637] cloud-init[894]: Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub
[   42.640343] cloud-init[894]: The key fingerprint is:
[   42.653224] cloud-init[894]: SHA256:g1cmQQL/SzUFxHuYFTqA4/38MFIKewzRZ39zcQ9GhxY root@ubuntu
[   42.672669] cloud-init[894]: The key's randomart image is:
[   42.684881] cloud-init[894]: +---[RSA 3072]----+
[   42.696688] cloud-init[894]: |    ...++oo.ooEo.|
[   42.712650] cloud-init[894]: |     .+..o+o. =o.|
[   42.724628] cloud-init[894]: |     ..+.oB* o .+|
[   42.736948] cloud-init[894]: |      +o.==oo o o|
[   42.748945] cloud-init[894]: |      .=S= . . o |
[   42.760943] cloud-init[894]: |      .o=o=      |
[   42.772935] cloud-init[894]: |       ... +     |
[   42.784923] cloud-init[894]: |            .    |
[   42.796697] cloud-init[894]: |                 |
[   42.812892] cloud-init[894]: +----[SHA256]-----+
[  OK  ] Finished Initial cloud-ini…ob (metadata service crawler).
[   42.813125] cloud-init[894]: Generating public/private dsa key pair.
[   42.852264] cloud-init[894]: Your identification has been saved in /etc/ssh/ssh_host_dsa_key
[  OK  ] Reached target Cloud-config availability.
[   42.852565] cloud-init[894]: Your public key has been saved in /etc/ssh/ssh_host_dsa_key.pub
[   42.900610] cloud-init[894]: The key fingerprint is:
[  OK  ] Started Periodic ext4 Onli…ata Check for All Filesystems.
[   42.900926] cloud-init[894]: SHA256:UKO/OaIM64iogSeEJpT1ZeKNIM7covX6ELKvdf6OX6I root@ubuntu
[   42.936568] cloud-init[894]: The key's randomart image is:
[  OK  ] Listening on cloud-init hotplug hook socket.
[   42.936846] cloud-init[894]: +---[DSA 1024]----+
[   43.012909] cloud-init[894]: | . o . oo        |
[   43.048856] cloud-init[894]: |+ = + *o .       |
[  OK  ] Reached target Basic System.
[   43.049151] cloud-init[894]: | B.. +o.         |
[   43.080966] cloud-init[894]: |+...   o         |
         Starting Start the snapd services from the snapd snap...
[   43.081301] cloud-init[894]: |=o. .   S        |
[   43.116932] cloud-init[894]: |=o o     o       |
         Starting Remove Stale Onli…t4 Metadata Check Snapshots...
[   43.144341] cloud-init[894]: |= * . o =        |
[   43.172290] cloud-init[894]: |oB O + + .       |
[   43.184329] cloud-init[894]: |Bo+ E++          |
[   43.212351] cloud-init[894]: +----[SHA256]-----+
[   43.224295] cloud-init[894]: Generating public/private ecdsa key pair.
[   43.240315] cloud-init[894]: Your identification has been saved in /etc/ssh/ssh_host_ecdsa_key
         Starting Generate sshd host keys...
[   43.260320] cloud-init[894]: Your public key has been saved in /etc/ssh/ssh_host_ecdsa_key.pub
[   43.292328] cloud-init[894]: The key fingerprint is:
[   43.292599] cloud-init[894]: SHA256:o7eTkKc2WP/4VLfQtYXpc3cuc2nXSRvpu48pU+cacMs root@ubuntu
[   43.321161] cloud-init[894]: The key's randomart image is:
[   43.344298] cloud-init[894]: +---[ECDSA 256]---+
[   43.356297] cloud-init[894]: |                 |
[   43.368350] cloud-init[894]: |               o |
[   43.392327] cloud-init[894]: |              o o|
[   43.404338] cloud-init[894]: |             o .+|
[   43.420315] cloud-init[894]: |       .S   + *=+|
[   43.432282] cloud-init[894]: |      +... . *+*X|
[   43.444345] cloud-init[894]: |     o.=...  .EO=|
[   43.456397] cloud-init[894]: |    . +.++  o .B+|
[   43.456795] cloud-init[894]: |     . .o+o  o+++|
[   43.492360] cloud-init[894]: +----[SHA256]-----+
[   43.504302] cloud-init[894]: Generating public/private ed25519 key pair.
[   43.520380] cloud-init[894]: Your identification has been saved in /etc/ssh/ssh_host_ed25519_key
[   43.556318] cloud-init[894]: Your public key has been saved in /etc/ssh/ssh_host_ed25519_key.pub
[   43.576335] cloud-init[894]: The key fingerprint is:
[   43.588874] cloud-init[894]: SHA256:Mjn3Chbj7nhQyeqja37uK6xHu4DHxPWRvxdlsZb/X1A root@ubuntu
[   43.608632] cloud-init[894]: The key's randomart image is:
[   43.620307] cloud-init[894]: +--[ED25519 256]--+
[   43.632326] cloud-init[894]: |            .    |
[   43.644299] cloud-init[894]: |       .     +   |
[   43.656311] cloud-init[894]: |    ..o.    *   E|
[   43.668308] cloud-init[894]: | . . .++   + . . |
[   43.680438] cloud-init[894]: |  o  oO S .   o  |
[   43.692329] cloud-init[894]: |.o. o. B o .   o |
[   43.704337] cloud-init[894]: |oooo .+ . o     o|
[   43.716329] cloud-init[894]: | o* ++.. o      o|
[   43.728379] cloud-init[894]: |.==O==o .       .|
[   43.756352] cloud-init[894]: +----[SHA256]-----+

With ubuntu-image command, I did not disable console-conf… is this the way, or am I missing something?

My custom gadget cloud config is copied over to the root of my gadget, with the name cloud.conf. gadget’s confinement is strict, with the model file’s grade set to “dangerous”.

I mounted the intalled image onto my linux box, and found my cloud-init config under 80_gadget-something.cfg, so I can confirm that it is on the filesystem, but never executed at boot time.

All-in-all, it seems that cloud-init is breaking with some outdated config, and seems to never execute mine afterwards?

Any feedback on this yet? @ijohnson I think that the cloud-init config is outdated or something?

Hi ruhan.vanderberg,

As you also suggest, this does not look right:

schema.py[WARNING]: Invalid cloud-config provided: Please run 'sudo cloud-init schema --system' to see the schema errors.

Perhaps the schema check will be useful. Do you have a simple reference to start from?

I could probably boot a Core20 image without a cloud-init config, and try from there. Will take a while to verify.