Pre-seeded cloud-init config is ignored for core18 image

I am creating a custom core18-amd64 image using ubuntu-image tool and ubuntu-core-18-amd64.model-assertion.

ubuntu-image snap --snap docker --snap avahi --snap myprivatesnap_amd64.snap --cloud-init cloud-config.yaml ubuntu-core-18-amd64.model-assertion

cloud-config.yaml is very simple. It just runs a few snap commands to configure myprivatesnap:

#cloud-config
snap:
  commands:
    - snap connect myprivatesnap:avahi-control avahi
    - snap connect myprivatesnap:docker-daemon docker
    - snap connect myprivatesnap:docker-executables docker
    - snap connect myprivatesnap:hardware-observe
    - snap set myprivatesnap foo=bar

I have verified that the cloud-config.yaml is correctly written to /var/lib/cloud/seed/nocloud-net in the resulting image. However after cloud-init runs, /var/lib/cloud/instance/user-data.txt is empty. From /var/log/cloud-init.log, it seems that /var/lib/cloud/seed/nocloud-net/user-data is read but somehow get dropped.

I also see the following errors in /var/log/cloud-init.log:
OSError: [Errno 30] Read-only file system: ‘/etc/pollinate’
OSError: [Errno 30] Read-only file system: ‘/etc/localtime’

/var/lib/cloud/data/status.json:

{
 "v1": {
  "datasource": "DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net,/dev/sr0][dsmode=net]",
  "init": {
   "errors": [
    "('write-files', OSError(30, 'Read-only file system'))"
   ],
   "finished": 1588981155.247145,
   "start": 1588981154.733196
  },
  "init-local": {
   "errors": [],
   "finished": 1588981152.377084,
   "start": 1588981151.9045477
  },
  "modules-config": {
   "errors": [
    "('timezone', OSError(30, 'Read-only file system'))"
   ],
   "finished": 1588981156.988874,
   "start": 1588981156.7807033
  },
  "modules-final": {
   "errors": [],
   "finished": 1588981157.4702873,
   "start": 1588981157.3532157
  },
  "modules-init": {
   "errors": [],
   "finished": null,
   "start": null
  },
  "stage": null
 }
}

However when I pass the cloud-config file when launching the image via multipass, it does get picked up and all cloud-init commands are correctly executed:

multipass launch --cloud-init cloud-config.yaml file:///pc.img 

I could use some help in troubleshooting why the pre-seeded cloud-init config is ignored while the same file gets correctly picked up by multipass. Thanks!

Hi @alok,

Multipass builds an .iso as the data source, that must be what makes it different. The OSErrors shouldn’t make the whole thing break. I’d suspect your config isn’t picked up at all.

You’ll have to retrace the cloud-init.log, especially wrt where it picks the data up.

1 Like

Thanks for the hint Michal.

I looked at the cloud-init logs again and noticed that DataSourceNoCloud is reading from two locations - seed directory and an iso (apparently written by multipass).

/run/cloud-init/result.json:

{
 "v1": {
  "datasource": "DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net,/dev/sr0][dsmode=net]",
  "errors": [
   "('write-files', OSError(30, 'Read-only file system'))",
   "('timezone', OSError(30, 'Read-only file system'))"
  ]
 }
}

Do you know if user-data from multiple sources is “merged” or “overwritten”? I could not find this information in cloud-init documentation. The behavior I am seeing is “overwrite”.

More relevant logs from cloud-init.log:

util.py[DEBUG]: Reading from /var/lib/cloud/seed/nocloud/user-data (quiet=False)
util.py[DEBUG]: Reading from /var/lib/cloud/seed/nocloud/meta-data (quiet=False)
util.py[DEBUG]: Reading from /var/lib/cloud/seed/nocloud/vendor-data (quiet=False)
util.py[DEBUG]: Reading from /var/lib/cloud/seed/nocloud/network-config (quiet=False)
util.py[DEBUG]: Reading from /var/lib/cloud/seed/nocloud-net/user-data (quiet=False)
util.py[DEBUG]: Read 64 bytes from /var/lib/cloud/seed/nocloud-net/user-data
util.py[DEBUG]: Reading from /var/lib/cloud/seed/nocloud-net/meta-data (quiet=False)
util.py[DEBUG]: Read 28 bytes from /var/lib/cloud/seed/nocloud-net/meta-data
util.py[DEBUG]: Reading from /var/lib/cloud/seed/nocloud-net/vendor-data (quiet=False)
util.py[DEBUG]: Reading from /var/lib/cloud/seed/nocloud-net/network-config (quiet=False)
DataSourceNoCloud.py[DEBUG]: Using seeded data from /var/lib/cloud/seed/nocloud-net
util.py[DEBUG]: Attempting to load yaml from string of length 28 with allowed root types (<class 'dict'>,)

DataSourceNoCloud.py[DEBUG]: Attempting to use data from /dev/sr0
util.py[DEBUG]: Running command ['mount', '-o', 'ro', '-t', 'auto', '/dev/sr0', '/run/cloud-init/tmp/tmpm0ydfqvf'] with allowed return codes [0] (shell=False, capture=True)
Reading from /run/cloud-init/tmp/tmpm0ydfqvf//user-data (quiet=False)
util.py[DEBUG]: Read 17 bytes from /run/cloud-init/tmp/tmpm0ydfqvf//user-data
util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpm0ydfqvf//meta-data (quiet=False)
util.py[DEBUG]: Read 63 bytes from /run/cloud-init/tmp/tmpm0ydfqvf//meta-data
util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpm0ydfqvf//vendor-data (quiet=False)
util.py[DEBUG]: Read 877 bytes from /run/cloud-init/tmp/tmpm0ydfqvf//vendor-data
util.py[DEBUG]: Reading from /run/cloud-init/tmp/tmpm0ydfqvf//network-config (quiet=False)
util.py[DEBUG]: Running command ['umount', '/run/cloud-init/tmp/tmpm0ydfqvf'] with allowed return codes [0] (shell=False, capture=True)
util.py[DEBUG]: Attempting to load yaml from string of length 63 with allowed root types (<class 'dict'>,)
DataSourceNoCloud.py[DEBUG]: Using data from /dev/sr0

atomic_helper.py[DEBUG]: Atomically writing to file /run/cloud-init/instance-data.json (via temporary file /run/cloud-init/tmp4lq18nbu) - w: [644] 755 bytes/chars
atomic_helper.py[DEBUG]: Atomically writing to file /run/cloud-init/instance-data-sensitive.json (via temporary file /run/cloud-init/tmpitaky9sw) - w: [600] 755 bytes/chars
handlers.py[DEBUG]: finish: init-local/search-NoCloud: SUCCESS: found local data from DataSourceNoCloud
stages.py[INFO]: Loaded datasource DataSourceNoCloud - DataSourceNoCloud [seed=/var/lib/cloud/seed/nocloud-net,/dev/sr0][dsmode=net]

Hi @alok,

nocloud-net is a “virtual” datasource that would require you to pass in details about the source (usually on kernel command line). Since that’s not happening, this does:

DataSourceNoCloud.py[DEBUG]: Using data from /dev/sr0

So it’s only the ISO file that is being read in - and as you say, everything worked in this case.

It would be interesting to see the cloud-init logs from when it fails. The write-files and timezone errors are due to how Core is set up, and cloud-init apparently doesn’t deal well with that. It may be worth filing a bug about the timezone module at least - write-files you’ll need to modify in a way that it writes to a writable location.

Hope this helps.