Difficulties using logsync-james and logsync-receiver on arm64

snap
1

Hi, I have installed logsync-james and logsync-receiver on 2 arm64 devices. It’s not working but I wonder is it related to the following messages in dmesg:

[65163.644548] audit: type=1400 audit(1686212311.336:3355): apparmor=“DENIED” operation=“open” profile=“snap.logsync-james.upload” name="/proc/1/sched" pid=101283 comm=“systemd-journal” requested_mask=“r” denied_mask=“r” fsuid=0 ouid=0

and

[250784.084498] audit: type=1400 audit(1686225672.291:5540): apparmor=“DENIED” operation=“open” profile=“snap.logsync-receiver.remote” name="/proc/1/environ" pid=363256 comm=“systemd-journal” requested_mask=“r” denied_mask=“r” fsuid=0 ouid=0

[250784.084532] audit: type=1400 audit(1686225672.291:5541): apparmor=“DENIED” operation=“open” profile=“snap.logsync-receiver.remote” name="/proc/1/sched" pid=363256 comm=“systemd-journal” requested_mask=“r” denied_mask=“r” fsuid=0 ouid=0

I did connect the interfaces that were not automatically connected and set the url in the logsync-james snap.

2

seems it tries to read he env of systemd there … that should not be fatal … do you see any other errors beyond this ?

3

No I don’t. I am thinking to make my own version of these snaps but build them so they are not services, and instead be able to run the from a command line, so I might see the problem.

1 Like
4

After some changes to make it run as a command line program I get the following error:

$ epi-remoteloggingserver.remote

Received fewer sockets than expected

5

it might actually have a builtin requirement to be started by pid 1 (systemd), not sure …

6

Yes agreed, I might have created that problem by not running it as a daemon.

Looking at logsync-james on its own, snappy-debug diagnoses the following:

= AppArmor = Time: Jun 13 13:21:41 Log: apparmor=“DENIED” operation=“open” profile=“snap.logsync-james.upload” name="/proc/1/sched" pid=3030 comm=“systemd-journal” requested_mask=“r” denied_mask=“r” fsuid=0 ouid=0 File: /proc/1/sched (read)

Suggestion:

  • adjust program to not access ‘@{PROC}/@{pid}/sched’
7

I seem to have narrowed down the problem. logsync-receiver works on arm64 platform. logsync-james works on amd64 platform logsync-james not working on arm64 platform.

I will compare the system logs between amd64 and arm64 next.

8

this is very interesting indeed … i do not think we even tested cross arch back when we developed it (simply assuming it would behave the same) … it might be that different versions of journald could also play a role here (if they actually are different) …

9

Possibly I should create this as a separate issue, but it is more of a question than an issue. I put a my version of logsync-receiver and logsync-james into the same snapcraft file - with 2 apps (an upload and a download). This seems to not be allowed/appropriate as the upload parts repeatedly crashes and is restarted by the watchdog.

I see the following warnings from snappy debug

= AppArmor =

Time: Sep 07 11:00:12

Log: apparmor="DENIED" operation="sendmsg" profile="snap.epi-logstream.upload" name="/run/systemd/notify" pid=8265 comm="systemd-journal" requested_mask="w" denied_mask="w" fsuid=0 ouid=0

File: /run/systemd/notify (write)

Suggestions:

* adjust program to use $SNAP_DATA

* adjust program to use /run/shm/snap.$SNAP_NAME.*

* adjust program to use /run/snap.$SNAP_NAME.*

* adjust snap to use snap layouts (https://forum.snapcraft.io/t/snap-layouts/7207)

This is how I define the layout:

layout:
  /etc/systemd:
    bind: $SNAP_DATA/systemd

This is extract from my snapcraft.yaml

apps:

  download:
    command: lib/systemd/systemd-journal-remote --listen-http=-3 --output=$SNAP_COMMON
    plugs:
      - network-bind
      - network
      - time-control # this quietens log spam at access of /proc/1/environ
    daemon: simple
    sockets:
      systemd-journal-remote:
        listen-stream: 19532
  
  upload:
    command: lib/systemd/systemd-journal-upload --url=http://localhost --save-state=$SNAP_DATA/state
    plugs:
      - daemon-notify
      - network-bind
      - network-control
      - network
      - log-observe
      - time-control # this quietens log spam at access of /proc/1/environ
    daemon: simple
    watchdog-timeout: 3m

So I would like to understand the reason why they cannot co-exist within the same snap.

10

In the end I found the problem. It was actually due to the daemon-notify and network-control interfaces not being connected.

1 Like
11

I am back looking that the difficulties running on arm64 compared to amd64. I believe the problem is related to the version of systemd.

From inside the snap, I can see the version of journactl is as follows:

$ journalctl --version systemd 237 +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid

From outside the snap, the normal OS version is:

$ sudo journalctl --version systemd 249 (249.11-0ubuntu3.9) +PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified

When I run (inside the snap) journactl directly on a journal file produced by version 249 I see the following error:

$ journalctl --file=ARM/system.journal Failed to open files: Protocol not supported

So the version of journactl and systemd-journal-upload pulled down as part of the building of the snap cannot read the files produced by the version in the Ubuntu Core OS.

Is there any obvious way to work around this?

12

I believe my problems can be solves if I can build this snap for core22. I understand that there are some different requirements when building with core22.

My main problem is to do with missing libs as follows: