Requesting auto-connections for following interface declared for the waagent snap:
Rationale:
When the agent is entering the goal state, there is one non-blocking error reported in the logs:
Aug 25 19:40:09 jrtknauer-ixlsezavpe waagent.waagent[3960]: 2025-08-25T19:40:09.792678Z ERROR ExtHandler ExtHandler Unable to setup the persistent firewall rules: [Errno 30] Read-only file system: '/lib/systemd/system/waagent-network-setup.service'
While the error itself is non-blocking (the agent will still enter the goal state) and does not appear to be related to critical agent features (e.g. the log error is ignored in e2e tests for some environments) it is a regression compared to the Ubuntu Classic logs. Incorporating the hardware-observe interface remediates the issue as the error will no longer be present in the logs. In addition, from conversations with WALinuxAgent contributors there will be other functionalities (e.g. extensions) which will inevitably require the interface. As with previous auto-connection requests, it is necessary for all interfaces to auto-connect.
This request has been added to the queue for review by the @reviewers team.
Hello @jrtknauer!
As described in the documentation, the hardware-observe interface read-only access to many files and directories, primarily in /sys and /proc.
Based on your error log, it appears that waagent is attempting to write to a read-only directory.
Could you please clarify how granting the hardware-observe interface resolves this issue? (#askForInfo)
Stated plainly: I myself do not fully understand why the hardware-observe interface resolves the issue associated with this log error, only that it does. I was originally lead to incorporating hardware-observe:
- Interface recommendation via
snappy-debug.
- Trial-and-error testing when attempting to address other interface issues not captured in this request.
If this explanation is insufficient, please provide the investigation threshold necessary to proceed with this auto-connection request. Short of tracing the agent’s execution as part of goal-state acquisition, I am uncertain as to how to proceed.
Hey @jrtknauer
Could you please share the apparmor denials you find when running the snap without hardware-observe connected? That will help to understand why it silence the log errors
#askForInfo (automation purposes)
Apologies for the delay in getting these logs to you. Here is a dump of the agent apparmor denial logs on first boot:
Nov 05 08:30:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331435.352:119): apparmor="DENIED" operation="open" class="file" profile="snap.waagent.waagent" name="/etc/" pid=3035 comm="waagent" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 05 08:30:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331435.362:120): apparmor="DENIED" operation="open" class="file" profile="snap.waagent.waagent" name="/proc/973/cmdline" pid=3035 comm="waagent" requested_mask="r" denied_mask="r" fsuid=0 ouid=105
Nov 05 08:30:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331435.363:121): apparmor="DENIED" operation="open" class="file" profile="snap.waagent.waagent" name="/proc/977/cmdline" pid=3035 comm="waagent" requested_mask="r" denied_mask="r" fsuid=0 ouid=103
Nov 05 08:30:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331435.363:122): apparmor="DENIED" operation="open" class="file" profile="snap.waagent.waagent" name="/proc/1251/cmdline" pid=3035 comm="waagent" requested_mask="r" denied_mask="r" fsuid=0 ouid=104
Nov 05 08:30:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331435.964:123): apparmor="DENIED" operation="mknod" class="file" profile="snap.waagent.waagent" name="/var/log/waagent.log" pid=3035 comm="waagent" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Nov 05 08:30:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331435.965:124): apparmor="DENIED" operation="capable" class="cap" profile="snap.waagent.waagent" pid=3035 comm="waagent" capability=26 capname="sys_tty_config"
Nov 05 08:30:41 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331441.139:165): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="snap.waagent.waagent" name="/mnt/cdrom/secure/" pid=3146 comm="mount" fstype="udf" srcname="/dev/sr0" flags="ro, silent"
Nov 05 08:30:41 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331441.142:166): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="snap.waagent.waagent" name="/mnt/cdrom/secure/" pid=3146 comm="mount" fstype="iso9660" srcname="/dev/sr0" flags="ro, silent"
Nov 05 08:30:41 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331441.142:167): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="snap.waagent.waagent" name="/mnt/cdrom/secure/" pid=3146 comm="mount" fstype="vfat" srcname="/dev/sr0" flags="ro, silent"
Nov 05 08:30:41 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331441.149:168): apparmor="DENIED" operation="mknod" class="file" profile="snap.waagent.waagent" name="/var/log/waagent.log" pid=3035 comm="waagent" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Nov 05 08:30:41 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331441.149:169): apparmor="DENIED" operation="capable" class="cap" profile="snap.waagent.waagent" pid=3035 comm="waagent" capability=26 capname="sys_tty_config"
Nov 05 08:30:46 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331446.155:170): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="snap.waagent.waagent" name="/mnt/cdrom/secure/" pid=3222 comm="mount" fstype="udf" srcname="/dev/sr0" flags="ro, silent"
Nov 05 08:30:46 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331446.155:171): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="snap.waagent.waagent" name="/mnt/cdrom/secure/" pid=3222 comm="mount" fstype="iso9660" srcname="/dev/sr0" flags="ro, silent"
Nov 05 08:30:46 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331446.155:172): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="snap.waagent.waagent" name="/mnt/cdrom/secure/" pid=3222 comm="mount" fstype="vfat" srcname="/dev/sr0" flags="ro, silent"
Nov 05 08:30:46 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331446.155:173): apparmor="DENIED" operation="mknod" class="file" profile="snap.waagent.waagent" name="/var/log/waagent.log" pid=3035 comm="waagent" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Nov 05 08:30:46 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331446.155:174): apparmor="DENIED" operation="capable" class="cap" profile="snap.waagent.waagent" pid=3035 comm="waagent" capability=26 capname="sys_tty_config"
Nov 05 08:30:51 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331451.159:175): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="snap.waagent.waagent" name="/mnt/cdrom/secure/" pid=3224 comm="mount" fstype="udf" srcname="/dev/sr0" flags="ro, silent"
Nov 05 08:30:51 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331451.159:176): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="snap.waagent.waagent" name="/mnt/cdrom/secure/" pid=3224 comm="mount" fstype="iso9660" srcname="/dev/sr0" flags="ro, silent"
Nov 05 08:30:51 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331451.159:177): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="snap.waagent.waagent" name="/mnt/cdrom/secure/" pid=3224 comm="mount" fstype="vfat" srcname="/dev/sr0" flags="ro, silent"
Nov 05 08:30:51 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331451.160:178): apparmor="DENIED" operation="mknod" class="file" profile="snap.waagent.waagent" name="/var/log/waagent.log" pid=3035 comm="waagent" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Nov 05 08:30:51 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331451.160:179): apparmor="DENIED" operation="capable" class="cap" profile="snap.waagent.waagent" pid=3035 comm="waagent" capability=26 capname="sys_tty_config"
Nov 05 08:30:56 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331456.163:180): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="snap.waagent.waagent" name="/mnt/cdrom/secure/" pid=3225 comm="mount" fstype="udf" srcname="/dev/sr0" flags="ro, silent"
Nov 05 08:30:56 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331456.163:181): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="snap.waagent.waagent" name="/mnt/cdrom/secure/" pid=3225 comm="mount" fstype="iso9660" srcname="/dev/sr0" flags="ro, silent"
Nov 05 08:30:56 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331456.163:182): apparmor="DENIED" operation="mount" class="mount" info="failed flags match" error=-13 profile="snap.waagent.waagent" name="/mnt/cdrom/secure/" pid=3225 comm="mount" fstype="vfat" srcname="/dev/sr0" flags="ro, silent"
Nov 05 08:30:56 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331456.164:183): apparmor="DENIED" operation="mknod" class="file" profile="snap.waagent.waagent" name="/var/log/waagent.log" pid=3035 comm="waagent" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Nov 05 08:30:56 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331456.166:184): apparmor="DENIED" operation="capable" class="cap" profile="snap.waagent.waagent" pid=3035 comm="waagent" capability=26 capname="sys_tty_config"
Nov 05 08:31:01 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331461.167:185): apparmor="DENIED" operation="mknod" class="file" profile="snap.waagent.waagent" name="/var/log/waagent.log" pid=3035 comm="waagent" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Nov 05 08:31:01 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331461.167:186): apparmor="DENIED" operation="capable" class="cap" profile="snap.waagent.waagent" pid=3035 comm="waagent" capability=26 capname="sys_tty_config"
Nov 05 08:31:01 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331461.167:187): apparmor="DENIED" operation="mknod" class="file" profile="snap.waagent.waagent" name="/var/log/waagent.log" pid=3035 comm="waagent" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Nov 05 08:31:01 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331461.167:188): apparmor="DENIED" operation="mknod" class="file" profile="snap.waagent.waagent" name="/var/log/waagent.log" pid=3035 comm="waagent" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Nov 05 08:31:01 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331461.167:189): apparmor="DENIED" operation="mknod" class="file" profile="snap.waagent.waagent" name="/var/log/waagent.log" pid=3035 comm="waagent" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Nov 05 08:31:01 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331461.167:190): apparmor="DENIED" operation="mknod" class="file" profile="snap.waagent.waagent" name="/var/log/waagent.log" pid=3035 comm="waagent" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Nov 05 08:31:01 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331461.169:191): apparmor="DENIED" operation="mknod" class="file" profile="snap.waagent.waagent" name="/var/log/waagent.log" pid=3035 comm="waagent" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Nov 05 08:31:01 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331461.169:192): apparmor="DENIED" operation="capable" class="cap" profile="snap.waagent.waagent" pid=3035 comm="waagent" capability=26 capname="sys_tty_config"
Nov 05 08:31:01 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331461.170:193): apparmor="DENIED" operation="mknod" class="file" profile="snap.waagent.waagent" name="/var/log/waagent.log" pid=3035 comm="waagent" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Nov 05 08:31:01 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331461.191:194): apparmor="DENIED" operation="mknod" class="file" profile="snap.waagent.waagent" name="/var/log/waagent.log" pid=3035 comm="waagent" requested_mask="c" denied_mask="c" fsuid=0 ouid=0
Nov 05 08:31:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331495.522:413): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3312 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:31:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331495.522:414): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3312 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:31:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331495.712:415): apparmor="DENIED" operation="open" class="file" profile="snap.waagent.waagent" name="/sys/block/" pid=3280 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 05 08:32:05 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331525.525:416): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3344 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:32:05 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331525.525:417): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3344 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:32:05 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331525.720:418): apparmor="DENIED" operation="open" class="file" profile="snap.waagent.waagent" name="/sys/block/" pid=3280 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 05 08:32:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331555.525:419): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3356 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:32:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331555.525:420): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3356 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:32:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331555.728:421): apparmor="DENIED" operation="open" class="file" profile="snap.waagent.waagent" name="/sys/block/" pid=3280 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 05 08:33:05 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331585.526:422): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3360 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:33:05 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331585.526:423): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3360 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:33:05 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331585.738:424): apparmor="DENIED" operation="open" class="file" profile="snap.waagent.waagent" name="/sys/block/" pid=3280 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 05 08:33:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331615.528:425): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3369 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:33:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331615.528:426): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3369 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:33:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331615.746:427): apparmor="DENIED" operation="open" class="file" profile="snap.waagent.waagent" name="/sys/block/" pid=3280 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 05 08:34:05 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331645.529:428): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3400 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:34:05 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331645.529:429): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3400 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:34:05 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331645.754:430): apparmor="DENIED" operation="open" class="file" profile="snap.waagent.waagent" name="/sys/block/" pid=3280 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 05 08:34:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331675.530:431): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3428 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:34:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331675.530:432): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3428 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:34:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331675.764:433): apparmor="DENIED" operation="open" class="file" profile="snap.waagent.waagent" name="/sys/block/" pid=3280 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 05 08:35:05 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331705.531:434): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3436 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:35:05 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331705.531:435): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3436 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:35:05 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331705.772:436): apparmor="DENIED" operation="open" class="file" profile="snap.waagent.waagent" name="/sys/block/" pid=3280 comm="python3" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 05 08:35:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331735.532:437): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3448 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Nov 05 08:35:35 jrtknauer-core-logs kernel: audit: type=1400 audit(1762331735.532:438): apparmor="DENIED" operation="exec" class="file" profile="snap.waagent.waagent" name="/usr/sbin/killall5" pid=3448 comm="python3" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
Thanks for the info @jrtknauer.
I’m really struggling to see how hardware-observe is even related with this issue. Rules included in the hardware-observe interface are not related with the observed denials and/or error, so I guess I’m missing something…
Could you share the snappy-debug output chunk where it suggest to use hardware-observe to see which denial it is trying to fix?
Before closing this request I will summarize:
- Incorporation of the
hardware-observe interface eliminates the reported error in the agent’s goal state logs and is easily reproducable.
- More granular testing to satisfy the concerns about the relevance of the interface requires a more customized image, which presents non-deterministic testing challenges (e.g. the order in which connections are enabled during first boot).
The time required to reconcile these two issues has grown beyond the engineering time I can allocate to this request. While we would like to see these errors remediated eventually, for now it would be appropriate to either close this request thread or suspend it until a future date.
Let me know if you have any additional questions. Thanks.
@jrtknauer thanks for your answer. I’ll remove it from the review queue for now then.
