Networks are not working after disk full

snapd
1

The headless device runs on Ubuntu Core, and it be always accessed through networking by ssh.

When the disk be full by accident(i.e system collects logs from sensors frequently) . After system reboot, the networks on the device are no longer to work, since the network-manager snap cannot start up.
The device cannot access by ssh any more.

Here is the nm stopped log:

Feb 22 06:48:12 6767677 systemd[1]: Started Service for snap application network-manager.networkmanager.
Feb 22 06:48:12 6767677 network-manager.networkmanager[3371]: cannot execute snap-update-ns: Permission denied
Feb 22 06:48:12 6767677 network-manager.networkmanager[3371]: snap-update-ns failed with code 1: File exists
Feb 22 06:48:13 6767677 systemd[1]: snap.network-manager.networkmanager.service: Main process exited, code=exited, status=1/FAILURE
Feb 22 06:48:13 6767677 systemd[1]: snap.network-manager.networkmanager.service: Unit entered failed state.
Feb 22 06:48:13 6767677 systemd[1]: snap.network-manager.networkmanager.service: Failed with result ‘exit-code’.
Feb 22 06:48:13 6767677 systemd[1]: snap.network-manager.networkmanager.service: Service hold-off time over, scheduling restart.
Feb 22 06:48:14 6767677 systemd[1]: Stopped Service for snap application network-manager.networkmanager.

However, I checked the other snaps are not working as well.
The “snap list” command is no response.
It seems snapd is not working if no disk space left.

Is that any way to make the networks still work for headless device if disk full?

Thanks.
Darren

1 Like
2

Turn off rsyslog in such cases. Ubuntu Core ships both, rsyslog and journald, on devices with limited disk space (or when there is a likelyness that you could wear out the disk) you should turn off rsyslog.

We ship an option in the core snap to do so with:

snap set core  service.rsyslog.disable true

likewise you can set it as default in the gadget.yaml via:

defaults:
  99T7MUlRhtI3U0QFgl5mXXESAiSwt776:
    service:
      rsyslog:
        disable: true

here is a real-life example:

I also thought we fixed this long ago, but apparently this bug is still open: