Why snap_core and snap_kernel in boot cmdline

device
21

the failing ethernet should not have any impact on this, it should just come up with the config screen again next boot if you can not configure it …

on first boot snapd copies the seeded snaps from /var/lib/snapd/seed/snaps/ to /var/lib/snapd/snaps
this step happens asynchronous in the background, could be that you simply reboot to fast (try again and give it some time)… nontheless, you should make your network device work as well (i guess you need a board specific kernel driver or some such) since it will need to talk to the snap store to install the ssh key for the first user you create (and also verify some snap signatures)

22

@ogra: [quote=“ogra, post:21, topic:1190”]
on first boot snapd copies the seeded snaps from /var/lib/snapd/seed/snaps/ to /var/lib/snapd/snaps
this step happens asynchronous in the background, could be that you simply reboot to fast (try again and give it some time)…
[/quote]

@ogra: I tried with manually copy the core_2314.snap file from /var/lib/snapd/seed/snaps/ to /var/lib/snapd/snaps in initrmfs shell and then re-boot the system. System was able to boot up successfully but again when I try to re-boot then again I get the same error
mount: mounting /tmpmnt_writable/system-data/var/lib/snapd/snaps/core_2314.snap on /root failed: No such file or directory

It seems snapd daemon is not able to copy the core_2314.snap file successfully(What could be the issue?). When does it (snapd daemon) actually copies the core_2314.snap to /var/lib/snapd/snaps? Is it before restart?

23

the copying should theoretically happen during the boot, based on the contents of /var/lib/snapd/seed/seed.yaml, and /var/lib/snapd/seed/assertions/model …

the snaps get verified agaist the *.account-key files in /var/lib/snapd/seed/assertions using the *.snap-declaration files.

all these bits get put in place initially by ubuntu-image when it verifies your model assertion at build time.

24

@ogra: Correct this should be theoretically happen during the boot. As you have mentioned it is done by snapd daemon but after booting it is not started yet. First ramdisk image is loaded and ubuntu-core-rootfs script shall get executed which tried to mount core.snap to /root. Please correct me if I am wrong here.

First of all I am not able to figure it out that why /tmpmnt_writable/system-data/var/lib/snapd/snaps/core_2314.snap file is removed? Next after re-boot when system mount /tmpmnt_writable to writable partition in SD card then why core_2314.snap file is not available in “/tmpmnt_writable/system-data/var/lib/snapd/snaps” as it is available in writable partition.

Still I am not able to figure out why core_2314.snap is not copied to /tmpmnt_writable/system-data/var/lib/snapd/snaps/ on my system. Am I missing any configurations?

I also checked the content of /var/lib/snapd/seed/assertions directory. All files are present *.account-key, *.snap-declarations.

Please find below content of seed.yaml file

snaps:
- name: core
  snap-id: 99T7MUlRhtI3U0QFgl5mXXESAiSwt776
  channel: stable
  file: core_2314.snap
- name: olimex-a20-olinuxino-lime2
  devmode: true
  unasserted: true
  file: olimex-a20-olinuxino-lime2_x1.snap
- name: olimex-a20-olinuxino-lime2-linux
  devmode: true
  unasserted: true
  file: olimex-a20-olinuxino-lime2-linux_x1.snap
25

When ubuntu-image creates the image it puts the core snap and kernel snap into var/lib/snapd/snaps/ in the writable partition (this is used for the very first boot, but only for this)…

During first boot, when snapd starts it then walks over the seed directory, verifies each snap against the keys and copies it to /var/lib/snapd/snaps/. Somehow this step fails for you during the copying of the core snap and you end up with a corrupt or non-existing file there.

Did ubuntu-image show any errors (it should verify all keys against your model assertion when it copies the snaps in place) ?

Did you try using the beta channel instead of edge in ubuntu-image (the version of the core snap suggests you are using edge, there might be bugs in edge, that channel does not get any QA) ?

26

@ogra: ubuntu-image command didn’t show any errors. Below are the output of ubuntu-image command

Building Ubuntu Core Image
DEBUG:ubuntu-image:-> [ 0] make_temporary_directories
DEBUG:ubuntu-image:-> [ 1] prepare_image
Fetching core
Copying "olimex-a20-olinuxino-lime2/gadget-snap/olimex-a20-olinuxino-lime2_16.04-0.8_armhf.snap" (olimex-a20-olinuxino-lime2)
olimex-a20-olinuxino-lime2 already prepared, skipping
Copying "olimex-a20-olinuxino-lime2/olimex-a20-olinuxino-lime2-linux_4.4.19_armhf.snap" (olimex-a20-olinuxino-lime2-linux)
olimex-a20-olinuxino-lime2 already prepared, skipping
DEBUG:ubuntu-image:-> [ 2] load_gadget_yaml
DEBUG:ubuntu-image:-> [ 3] populate_rootfs_contents
DEBUG:ubuntu-image:-> [ 4] calculate_rootfs_size
DEBUG:ubuntu-image:-> [ 5] pre_populate_bootfs_contents
DEBUG:ubuntu-image:-> [ 6] populate_bootfs_contents
DEBUG:ubuntu-image:-> [ 7] prepare_filesystems
DEBUG:ubuntu-image:-> [ 8] populate_filesystems
DEBUG:ubuntu-image:-> [ 9] make_disk
DEBUG:ubuntu-image:-> [10] finish
DEBUG:ubuntu-image:-> [11] close

Initially I was using “stable” channel to download the core. Below command I used to build an Ubuntu-Core image.

ubuntu-image -d -c stable -O imagefile --extra-snaps olimex-a20-olinuxino-lime2-linux_4.4.19_armhf.snap --extra-snaps olimex-a20-olinuxino-lime2_16.04-0.8_armhf.snap model/Olimex-lime2.model -w prepare/

However I tried with building an image using beta channel. Still I am facing the same issues i.e. core_2314.snap is not present in /tmpmnt_writable/system-data/var/lib/snapd/snaps/ directory.
ubuntu-image -d -c beta -O imagefile --extra-snaps olimex-a20-olinuxino-lime2-linux_4.4.19_armhf.snap --extra-snaps olimex-a20-olinuxino-lime2_16.04-0.8_armhf.snap model/Olimex-lime2.model -w prepare/

Is there any pointers to check why “snapd” daemon is not able to copy the core snap appropriately?

27

I’m slowly running out of ideas :slight_smile:

You can add “systemd.debug-shell=1” to your kernel cmdline and will end up in a root shell in the running system (it will not go into configuration). from there you should be able to use journalctl to retrieve logs, perhaps there is a useful error message somewhere.

28

@ogra: First of all thanks for being supportive. I tried your above suggestions i.e. adding “systemd.debug-shell=1” to kernel command line but unfortunately end up in configuration window. I was not able to enter into root shell. Probably first time configurations are still required?

I have also observed that system was able to take an IP address dynamically (Ethernet drivers are working properly) but when I tried to manually enter the below settings then I got an error message for verifying settings:

    Network interface eth0 manual IPv4 configuration

            Subnet:  172.19.75.3/32                             [?]
           Address:  172.19.75.3
           Gateway:  172.19.75.1
      Name servers:  10.203.171.38, 10.203.171.39               [?]
    Search domains:                                             [?]

                               [ Done   ]
                               [ Cancel ]

Any pointers shall be helpful to check the things

29

it might be limited to actual tty’s i.e. you might need a monitor and kbd attached (iirc it spawns the debug shell on tty9 by default)

The ethernet issue sounds like a bug, does it work if you just hit enter to accept the defaults and let it use dhcp ? (finishing the configuration should get you a ssh login, that works indeed as well to collect log info)

30

@ogra I have a HDMI monitor set up available with me. Will I able to use debug shell with that?
Earlier I was using serial connection through UART interface.

Yes when I entered to accept the defaults and let it allowed to use DHCP then it was able to get an IP address but unfortunately I was sitting behind proxy servers that were restricted me to log-in to Ubuntu SSO account. I will try to get an access. Yeah it was able to take DHCP settings but it was failed to configure the manual settings.

I tried with manual settings because an open internet access is available with using these configurations.

31

yes, i’m not 100% sure which tty it is on, so you need to try with ctrl-alt+f* keys, one of the consoles should have the debug shell.

this is definitely a bug, you can file it on launchpad … @mwhudson will hopefully pick up on it then :slight_smile: it is either netplan (the network setup backend) or subiquity (the actual configuration tool) that is broken here …

32

@mwhudson Can you provide some feedback here, or point the conversation in the right direction? (see below)

@ogra Thanks for the great exchange on this topic digging into the issue.

33

@niemeyer @ogra Thanks for prompt response. I was able to find out why the issue was coming. Actually it was my mistake sorry for that. I did wrong configuration for “Subnet” field. It expect the subnet of IP address that one want to assign. When I entered correct value then I was able to move further.

Once again regret for any inconvenience here.

2 Likes
34

@ogra
So far I have managed to boot the system with ubuntu-core image. I already have an account at https://login.ubuntu.com/.
Last message I got from boot screen

Ubuntu Core 16 on 172.19.75.2 (ttyS0)

The host key fingerprints are:

RSA     SHA256:Ux8Ol6bbetJUoK1mst+LL15T9pdKMHxw53peoVOvvdQ
DSA     SHA256:c/gSyuIDZnLeYvgcSSsxYt0jILhliZJ1QL8r7YWehNA
ECDSA   SHA256:Q4DKhoX4OygCguP44c80PSpBHobU36BDGokvEdsgAj0
ED25519 SHA256:Lzouss4zYpW9q8Z60qnqjNcKZNrk4seZn2o9eoB8Unw

To login:

ssh arunmakkar@172.19.75.2

==========================================================================

Now I am trying to access machine from VM (virtual machine) on my system using “ssh arunmakkar@172.19.75.2”. It ask me for password. I have already uploaded public key of VM OS at my account. When I try to enter password that we provided during keys generation it is not accepted by server getting an message “Access Denied”.

Even I try to use ubuntu account password still I am getting same message “Access Denied”. Please provide me some pointers here.

Appreciate your support here.

35

If it asks for a password, something is wrong. The private key on the machine you try to log in from needs to match the public key that your account at login.ubuntu.com has.

Password login is not possible since the system explicitly does not set a password for the user, so only key based logins are possible.

Did you upload the key from the VM before actually running through the ubuntu core setup ? (else there is likely the old key installed on the Core system)

36

@ogra: Thanks for providing above info. Actually I was adding public keys after running through the ubuntu core setup. That was the reason for not being able to log-in. Now I added public keys before running through the setup and able to login successfully.

Now while installing hello-world snap. I am getting below error. Please provide some pointers here.

**root@localhost:~# snap install hello-world**
**error: cannot perform the following tasks:**
**- Make snap "core" (2314) available to the system (cannot set next boot: cannot determine bootloader)**
37

Looks like there is still an issue with your gadget, does the uboot.conf link exist (pointing to the generated uboot.env file) ? (see my template gadget snap branches)

The syslog of the first boot and the output of “snap changes” would be interesting … if you see any errors in snap changes, also the details of the chage that you can get with “snap change $number” might help.

38

@ogra: Yes the problem was with my gadget. I have made the modifications and build again. So far everything is running fine on my system. Thank you so much @ogra to provide me such a nice explanations. I will definitely take a help from you in future :). You have cleared my basic concepts related to booting of Ubuntu Core images. Thank you once again.

1 Like
39

@ogra: Sorry to bother you again. I am facing some problem after reset.

System was able to boot up successfully and Ubuntu Core configuration was set up properly. I was able to login the device and work with snap applications.
Now I restarted the system then it was not coming up and got stuck at “initrd: mounting /run” state.
Even it is not entering into ramdiskfs shell. It was stuck at this state for log time. Then again flash the image and followed the set-up procedure again. Everything worked fine for first time and then again after reset stuck at “initrd: mounting /run” state.

Please provide some pointers for this issue.
Appreciate your support here.

40

how do your console= entries on your cmdline look like ? this sounds like the board is simply switching to another console and redirecting the output there …