Accessing journal logs from 22.04 hosts when using older base snap?

snapd
1

The snap I maintain gathers info from the system to send bug reports to Launchpad.

After bringing a few fixes to make the snap able to run on 22.04 systems, I found that the journalctl log from previous boot was missing when running the snap on a 22.04 system. Normally, to gather this, the snap runs journalctl -b -1 -k.

The snap is built with core18 base snap, which bundles journalctl v237. Systems running 22.04 are running journalctl v249. When the snap tries to run journalctl 237 to retrieve journal logs created with journalctl 249, it fails with following error:

Journal file xxx.journal has unknown incompatible flags 0xc
Failed to open journal file xxx.journal: Protocol not supported
mmap cache statistics: 0 hit, 1 miss
Failed to open files: Protocol not supported

Same problem when trying to open the same journal log file using journalctl 245, so switching the snap to core20 base snap would not help.

I tried to run the host journalctl from within the snap, but it segfaults:

$ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/var/lib/snapd/hostfs/usr/lib/systemd/:/var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/ /var/lib/snapd/hostfs/bin/journalctl
Segmentation fault (core dumped)

And, for now, there is no official core22 base snap available…

I’m not exactly sure what the root cause is (I quickly went through systemd changelog, but there are just too many things…) and if there is any way I could somehow get the content of the previous boot log from within my snap, regardless of the host system version?

Thanks in advance!

2

There are two reasons journal files from 22.04 are not readable by systemd from 20.04:

  1. systemd now defaults to zstd compression
  2. the “keyed hash” feature is now the default (this has been true in systemd for quite a while, but a Debian/Ubuntu patch got dropped in jammy)

I don’t think there’s a direct way to do what you want. I guess you can build a newer journalctl file as part of your snap build?

3

i think the solution is a snapd interface that allows executing the hosts journalctl transparently if needed …

4

Are you able to get a strace out of this crash? I see that the log-observe interface grants you access to the journald in the host as well as to the systemd libraries, but you probably still lack access to the system libraries in /var/lib/snapd/hostfs/usr/lib/x86_64-linux-gnu/; do you see an AppArmor denial about this path?

I actually wonder if we shouldn’t add access to /var/lib/snapd/hostfs/usr/lib/** (or at least to the standard libraries in there) to all snaps from the base template.

EDIT: I now notice that your command line is actually wrong: you should place “$LD_LIBRARY_PATH” at the end of the path lists, otherwise you’ll probably still be using the libraries included in your snap or in the base snap.