Ubuntu Core 18 - Rpi 3 not seeded

Hello everyone,

I have been trying for the last two days to get a Raspberry Pi 3 working under Ubuntu Core without console-conf by using cloud-init.

Sadly every single time, I face the same issue. After the Rpi boots and ssh is enabled, I am unable to use snap as apparently the setup process is not finished.

A simple sudo snap refresh will lead to the following message:

error: too early for operation, device not yet seeded or device model not acknowledged

Let me give you here more information. model.json

{
	"type": "model",
	"authority-id": "---acc-id--",
	"brand-id": "---acc-id--",
	"series": "16",
	"model": "pi3-arp",
	"architecture": "armhf",
	"base": "core18",
	"gadget": "pi=18-pi3",
	"kernel": "pi-kernel=18-pi3",
	"timestamp": "---proper_timestamp--",
	"required-snaps": ["docker"]
}

As you can see here, I am also requiring docker to be installed, but that is also not the case.

I have tried the following to create the image (also tried --extra-snaps before I tried required-snaps, I know that required snaps can’t be removed):

sudo ubuntu-image snap -c stable -O pi3 --cloud-init mycloud.conf myfilesigned.mode

The Rpi with this image boots up perfectly, but apparently the seeding of the device is not going well.

In case you wonder about the cloud-init it simply creates a system user with an SSH key and turns off console-conf cloud.conf

users:
  - name: someuser
    gecos: somefullname
    homedir: /home/someuser
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: users, admin
    lock_passwd: true
    shell: /bin/bash
    ssh_authorized_keys:
      - a_pub_ssh_key_here

bootcmd:
  - mkdir /var/lib/console-conf
  - touch /var/lib/console-conf/complete

That’s about it. If you have any tips or readings that could help solve my problem I would be very grateful. I have checked dmesg and journalctl -u snapd and haven’t found anything suspicious except one error about “PolicyKit1 was not provided by any .service files”

Thank you in advance, and sorry if this was already answered as I look through the forum and the web I could not find someone with a similar issue.

While not using cloud-init (i prefer the builtin config mechanisms of core), i doubt the user config you create works, the user creation will expect an email address for a store account to be handed to “snap create-user”, i’m not sure homedir, sudo, groups or shell are in any way valid, they will definitely not be configurable in core (/etc/passwd and /etc/group are readonly, so you cant add a user to an existing system group for example) …

i think the code (with a little bit of documentation) that originally enabled user creation on core via cloud-init is at:

Hi @orga
Thank you for your quick reply !

Yes, I doubted it too. But Since the documentation didn’t mention anything about R/O systems like Ubuntu Core, I figured they actually integrated the process properly.

That is an awesome link response thank you.

This definitely should be better documented.
Because of that I found this (Snappy and Snap-conf being deprecated) :

So for anyone having the same issue snap commands should be used instead or assertions blobs.
I’ll post an update here after I get it up and running.

Hello,

As promised here is a little update, as of now, I am facing the same issue.

I tried the following:

  • Installing the snap “make-system-user” to create system user assertions. Had to it from the “edge” repo because the “stable” make-system-user is borked and will complain that you are missing a python3 GLIBC version.

  • Removing all extras (users etc) from the cloud init conf and leaving only the necessary

    snap:
      assertions:
        - First assertion from make-system-user  (Type account)
        - Second "" (Type account-key)
        - Tird ""  (Type system-user)
      commands:
        - sudo snap create-user --sudoer --known email_sso@domain.ext
    

    The result is a local user being created but it is not possible to SSH into rpi

    • Also tried to provide the assertion in a single blob. Not separate into 3 assertions. Same result.

    • I also tried without “–know” in order to force the process to go through Ubuntu SSO. Same result. Unable to SSH into the Rpi even after waiting several minutes for something to happen… No errors, no nothing as usual. Back to the USB key to create a local user…

  • Instead of using the assertions from cloud-init, I provided the assertions on an USB stick

The result is that now the user is properly created and it is possible to SSH into the rpi

However, even after being able to SSH into the rpi and having the user created from an USB key the error is still the same. Somehow it is not seeded.

Note in relation to the command “sudo snap create-user”. The documentation doesn’t says it all, the URL https://docs.ubuntu.com/core/en/reference/assertions/system-user stipulates the following:

The simple addition of such assertions to a device assertion database should not be enough to trigger the user creation. This must be initiated explicitly (via snap create-user, or in the context of the auto-import mechanism for assertions from removable devices, which requires physical access to the device).

Which is exactly what we did. Now if we look at the cloud-init documentation: https://cloudinit.readthedocs.io/en/latest/topics/modules.html#snap

The assertions configuration option is a dictionary or list of properly-signed snap assertions which will run before any snap commands. They will be added to snapd’s assertion database by invoking snap ack <aggregate_assertion_file>.

snap:
    assertions:
      00: |
      signed_assertion_blob_here
      02: |
      signed_assertion_blob_here
    commands:
      00: snap create-user --sudoer --known <snap-user>@mydomain.com
      01: snap install canonical-livepatch
      02: canonical-livepatch enable <AUTH_TOKEN>

Which is very similar to what we did. In addition to this we can see the argument “–known” which is not documented properly if you do “snap create-user --help”. There is no info on that --known argument. So back to @ogra URL on launchpad and now we see this:

+If ``known`` is set to True, then it is expected the user also included
+an assertion of type ``system-user``.  When ``snap create-user`` is called
+cloud-init will append '--known' flag which instructs snapd to look for
+a system-user assertion with the details.  If ``known`` is not set, then
+``snap create-user`` will contact the Ubuntu SSO for validating and importing
+a system-user for the instance.

If we use --known, then our user will be created from the assertions rather than login to the Ubuntu SSO. Ok, I tried with and without, same story. Unable to ssh into the Rpi it asks for a password (The user was not created).

Something here is absolutely not right. I have tried all setups I could think about and there is no single option that work. Except of course login in from a monitor + keyboard. But that defeats the whole IOT thing IMHO…

I’m really out of ideas, if anyone wishes to help I would be very grateful.

i’m not sure why you try to go this overcomplicated route here … just using:

users:
  - name: bob
    snapuser: bob@gmail.com

should be totally enough, snap create-user will automatically download the ssh key associated with the user in the store and should give you ssh access if you have the correct private key on your host.

for pre-seeding snaps i’d really go the required-snaps path in the model assertion, this is well tested and often used in production.

Hello @ogra,

Thank you for that information.

So a few things here. I am simply trying to use the tools available to me that are included and more or less well documented.
I indeed missed the “snapuser” from the cloud-init users documentation, my apologies. This could indeed be a way to do it and I am going to try it out right now.

Concerning the “snapuser”, what happens if the device has unstable connectivity when you plug it in for the first time ?
By providing a signed assertion for users, it should avoid contacting ubuntu servers and by so avoid any connectivity issues during first boot.

Edit:
I am having trouble understanding why there is “snapuser”, “snap create-user” under “User and groups” since it is also under “snap:”

it will still try to run snap login AFAIK so the user does not need sudo for snap commands … (but i could be wrong perhaps that changed… in any case there are other issues when you are not connected on first boot, starting with a default network config delaying your boot and ending with snapd trying to talk to the store).

this is rather a question to the cloud-init guys, after all it is a third party config tool sitting on top of the default mechanisms core offers. i dont think anyone of the core team has had any involvement in picking option names or config defaults for cloud-init.

Thanks @ogra, yes the cloud-init guys should really clean that up.
Because between the deprecated methods, the supposed new methods (which I tried all) and the users groups method, there is many way were someone reading their documentation can get it totally wrong.
And there is no mention about “snap login” or what would be the difference between “snap create-user:” and “snapuser:”.

I see, so there is other issues related to boot connectivity then.
Would is also be the case after a reboot ? Lets say the kernel gets updated and the device is restarted.
But in the meantime for any reason you’re out of connectivity.

And another question since we are at it.
Do you recommend the use of “runcmd” module from cloud-init to bootstrap and enable certain services ?

Note: I am now testing this on a Rpi.
But it is very likely that this project will move on to another more custom setup (hardware)

how would you update the kernel without connectivity in the first place ? it comes as a snap from the store … the kernel snap (as well as the core snap (i.e the readonly rootfs)) by default does some self tests though and would automatically roll back to the former version if there were actual boot issues during or after upgrade.

no, i do not recommend to tinker with the system at all on that level. if you want particular services running, create a snap for them and pre-seed them via required-snaps in your image instead. i dont think you can even update the cloud-init config after first boot so if you ever need anything changed you are lost … use snaps on core instead :wink:

depending what this hardware is i’d consider leaving out cloud-init completely … cloud-init is a big pile of python, that runs fine on multi-core CPUs, but if you i.e. target some single core device like an IMX6ULL or a TI Sitara (beaglebone), it will bring your system to a crawl on boot.

I was not speaking of a connectivity problem during the update, but rather after. Lets say you reboot, you have a module that does GSM connectivity, and bad luck, it fails to connect for quite a while. What happens at that point ? Does the OS and apps still boot properly ?

Yes I saw that sadly… But right now it seems to be the only way to enable a user on the device without having physical access to it.

I will build some snaps that I will pre-install on the image, then if I understood correctly the following should work:

“If your application is intended to run as a service you simply add the line daemon: simple after the command keyword. This will automatically keep the service running on install, update, and reboot.” https://docs.snapcraft.io/go-applications/7818

Meaning that I will not need to use “runcmd” in order to enable the app service.

Is there a way to view all services and slots exposed for all snapcrafts ? The store at the moment doesn’t explicitly says what a snap provides without installing the snap locally.

Is there a github for the ones provided by “canonical” that we can access, so we can see the definitions ?

I just tried the following (no more, no less)

Still not able to ssh into it. I do have the proper SSH key which I even enforced by using -i ~/.ssh/…privatekey It works only when I provide the user assertion through a USB key.

Status:

  • The name of the user is the same as in the Ubuntu SSO account, same for “snapuser” which reflects the email registred to the Ubuntu SSO account
  • The Rpi model.json is using the correct account-id and brand-id associated with the Ubuntu SSO account
  • The SSH pub key has been properly added to the account in question
  • I checked all values to be certain nothing is missing
  • dmesg & journalct still gives no proper error
  • IPV4&IPV6 are setup properly (from journalctl logs)
  • After a reboot of the device I saw a “Sarting Automatically repair incorrect owner/permissions on core devices.” Don’t know if it is relevant or not.

Should I be seeing something related to the SSO login ?

I guess it is time to start filing bugs against cloud-init …

i’m pretty much out of ideas and sadly do not know anyone who actually uses cloud-init on core, so i cant ping anyone to comment here either … perhaps the cloud-init devs can help out (looking at the code it seems all their snap handling is essentially just for installing snap packages on classic (deb based) cloud images, not for Ubuntu Core at all… the few Core related bits like snapuser seem to be deprecated)

@ogra I can’t thank you enough for your help and patience.
Quite sad to see that provided tools do not work together.

I mean cloud-init is installed by default on Ubuntu Core.

Thank you very much anyway.
I’ll start to look at the problem from a different point of view.

Basically if I understand correctly, this is the way to go:

  • Ask for a brand account
  • Publish a private snap that will setup the account properly on device
  • Later on, create my own snaps and gadgets for the custom hardware.

Wishing you a great evening/day.

@yds probably too late, but here is example of cloud-init user data which works should do similar thing you are trying to do

#cloud-config
debug:
  verbose: true
users:
  - name: ubuntu
    lock_passwd: False
    groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video]
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/bash
    plain_text_passwd: 'ubuntu'
    ssh-authorized-keys:
    - |
      ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA4kFYKuGM9MulGU7OieZkd7VrA9gYcMx5+FN/qyOxGZyZHh/WWerPQ159ptFHxs032z+fSQrqbX8NQra8VbgGuSYaoX8wZeSsaOVq39+rBJespm+X68uF7NjX122f0QEhIBAMYN548yd6c8AHTQh8EL7NSYLudBmmJL+xhgfhgfhgfhgfhfghdtrssYT8zWnTlh32mHXHH9rOj8pSKKlnAorKYEMio6TW7svxV10t4LEKmz4Qeb0A4ZQNt2vxQLAENCjKWGEIsBjHGrrDkcX5F3mclHmau7P7mlb1DJ3WKXn+uBJejLe19c0hulnsqykQfP4xeF+bHU/8pOmOEKLfYi8TFkw== me.me@e5410-me01
snappy:
  email: my@email.com
bootcmd:
  - mkdir -p /var/lib/console-conf
  - touch /var/lib/console-conf/complete

Hi @ondra , Thanks for your reply. I didn’t give it a try yet. However, isn’t snappy deprecated ? https://cloudinit.readthedocs.io/en/latest/topics/modules.html#snappy

“Deprecated: Use Snap module instead. This module will not exist in cloud-init 18.3.”

sort of :slight_smile: it still works on Ubuntu core devices