Completely broken snapd environment in Manjaro (20.1.2)

Is the gnome session using wayland? If so please switch to Xorg one and try again.

If that does not help, please attach the output of:
SNAPD_DEBUG=1 SNAP_CONFINE_DEBUG=1 snap run <some-failing-snap>

Thanks for the response,

Manjaro is running in X11 and not Wayland. Since the last post I’ve had three different people confirm this issue and I see one reference to it on the Manjaro forums, but my presumption is that it might be an issue with Manjaro’s packaging that is only present on clean installations from the 20.1.2 install media and not exposed by upgrading from the older media, hence not being complained about more since Manjaro is a fairly popular user of snaps.

Here’s the requested output for chromium:

And hello-world:

The logs look fine. The segfault appears after control has been handed off to a process from the snap. I have a hunch that it may be related to fonts cache again. Can you wipe the following cache directories and try again?:

  • /var/cache/fontconfig
  • ~/.cache/fontconfig
  • ~/snap//current/.cache/fontconfig

I’ve removed the first 2 folders. The fontconfig folder in the snaps current/.cache did not exist (.cache itself did not exist). Chromium still segfaults on load. I explicitly triggered fc-cache, chromium failed again.

EDITED: there is a fontconfig in ~/snap/$SNAP/common/.cache/fontconfig, upon deleting this there was no change either, but the instructions were to check the versioned current folder rather than common and this wasn’t picked up on at the time.

While a lot of apps I’ve tested do appear to complain abouts fonts, Java explicitly, Firefox opens a crash dialogue with the []'s of corrupted font-cache, and VSCode crashes when opening a GTK dialogue which depends on fontconfigs to work, I do not think it’s expected that hello-world would have any snappy-debug output at all for such basic functionality of just echoing into the terminal, and it does not have any such AppArmor denials on Ubuntu. My leaning is that fonts are broken as a symptom, and the root cause is something to do with the AppArmor denials showing up in Manjaro but not elsewhere, but I’m no expert to say that with any real conviction

I did try installing the joplin-james-carroll package, which has been built on Snapcraft 4.4 with extensions, so benefits from the recently reworked fontconfig changes.

It actually seems to work functionally, including opening GTK dialogues such as the file browser. However, it has an absolutely ridiculous amount of AppArmor violations from snappy-debug, all along the lines of denials in the snap-confine apparmor profile, none of these exist in Ubuntu.

I see the fsetid denial on Arch and openSUSE with 5.9 kernel too. I did not try 5.8 kernels on those distros. However, I don’t see it on Ubuntu 20.10 which uses 5.8 kernel, so maybe it’s only 5.9+ and newer? (cc @jdstrand does it ring a bell?)

I’ve tried the failing snaps on Arch and they seem to work just fine (given workarounds for fonts). The problem with code is werid, it’s a classic snap after all and does not run under confinement or a new namespace.

Can you try an build snapd from AUR to see if it makes a difference?

I’ve removed Manjaro’s snapd package and replaced it with the AUR snapd package, currently 2.47.1-2.

Nothing appears to be different from the Manjaro provided snapd, snapcraft 4.4+ fonts seem functional (joplin-james-carroll), Firefox still has []'s in its crash reporter, Chromium segfaults and displays nothing, vscode (code) crashes upon opening a GTK file browser, Java 11 with Gnome extension built before Snapcraft 4.4 explicitly complains about fonts.

If you could try any of the simpler snaps, maybe trying with --experimental-gdb and looking at the backtrace would help finding the root cause.

For the snaps that segfaulted, you can also try checking coredumpctl. If it registered any of the core dumps, running coredumpctl show <pid|name> should show a bit of a backtrace.

Gnome calculator seems to be an ideal test for this scenario.

It opens with the fonts replaced with boxes but otherwise seems to work. However, when clicking where about-calculator appears to be, it crashes giving a error with Pango on line 5.

I’ve attached the output with the coredumpctl output:

I’d like to mention that above when you asked me to clear the fontconfig, I’d said there wasn’t even a .cache folder. This is because I think we’d typo’d and looked in current, whereas there is a fontconfig folder in common/.cache/fontconfig. I repeated the test properly deleting all 3 folders again and nothing changed, but I thought I’d bring it up again so that you don’t assume the problem is the folder isn’t being made at all, it is.

VSCode for reference from the classic snaps perspective:

KCalc crashes before displaying any GUI, included for the kde-neon perspective:

Thanks. So gnome-calculator points to pango/harfbuzz (fonts?), Kcalc points to Xcb, and code has a silly number of threads but those 2 calls stand out:

                #0  0x00007f7fa61d5438 raise (libc.so.6 + 0x35438)
                #1  0x00007f7fa61d703a abort (libc.so.6 + 0x3703a)
                #2  0x00007f7fabcbe955 g_assertion_message (libglib-2.0.so.0 + 0x73955)
                #3  0x00007f7fabcbe9ba g_assertion_message_expr (libglib-2.0.so.0 + 0x739ba)

Which would suggest g_assert() or g_error() somewhere. My bet remains on fonts/fontconfig causing the issue.

Can you run gnome-calculator again but this time setting the following in the environment:

G_MESSAGES_DEBUG=all
G_DEBUG=fatal-criticals

Please attach the terminal output.

~ >>> export G_MESSAGES_DEBUG=all                                                                                                                                                                           
~ >>> export G_DEBUG=fatal-criticals                                                                                                                                                                              
~ >>> snap run gnome-calculator                                                                                                                                                                                   
(gnome-calculator:2646): GLib-DEBUG: 14:36:06.769: Failed to set thread scheduler attributes: Operation not permitted
(gnome-calculator:2646): GLib-GIO-DEBUG: 14:36:06.777: _g_io_module_get_default: Found default implementation local (GLocalVfs) for ‘gio-vfs’
Gtk-Message: 14:36:06.853: Failed to load module "canberra-gtk-module"
Gtk-Message: 14:36:06.854: Failed to load module "canberra-gtk-module"
(gnome-calculator:2646): GLib-DEBUG: 14:36:06.854: unsetenv() is not thread-safe and should not be used after threads are created
(gnome-calculator:2646): Gtk-DEBUG: 14:36:06.854: Connecting to session manager
(gnome-calculator:2646): GLib-GIO-DEBUG: 14:36:06.855: _g_io_module_get_default: Found default implementation dconf (DConfSettingsBackend) for ‘gsettings-backend’

(gnome-calculator:2646): Pango-CRITICAL **: 14:36:13.486: pango_font_get_hb_font: assertion 'PANGO_IS_FONT (font)' failed
zsh: trace trap (core dumped)  snap run gnome-calculator

Thanks. Looks like it’s something about fonts again. Maybe @jamesh has some ideas about debugging this further.

@mborzecki and I discussed this on IRC. This is a denial for snap-confine (not the app itself) and should just be noise. It has to do with the ordering of security operations in a section of code of snap-confine that was refactored and (re)introduced this denial.

@mborzecki - this one is unexpected. Feel free to adjust the snap-update-ns profile for this access.

actually this is a bit weird, I see the following in /var/lib/snapd/apparmor/profiles/snap-update-ns.hello-world:

  # Allow reading the command line (snap-update-ns uses it in pre-Go bootstrap code).
  @{PROC}/@{pid}/cmdline r,

@James-Carroll is the snapd.apparmor.service enabled and running on Manjaro?

Edit: nvm didn’t notice it’s /proc/pid/cmdline vs /proc/cmdline.

Edit2: nevertheless, I initially suspected Go doing something weird, but I can’t reproduce this access on Arch with AppArmor enabled and Go 1.15.3. So it’s still unclear as to what may be causing this.

So I take that back. There’s this in the profile:

  # snapd logger.go checks /proc/cmdline
  @{PROC}/cmdline r,

The problem is fixed in 2.47+.

Yes, the @{PROC}/cmdline r, error was in the 2.46, I hadnt’ checked to see if it still existed in 2.47. There was a fair amount of other potentially other relevant AppArmor denials when running Joplin that didn’t exist in Ubuntu too but I unfortunately don’t have them logged anywhere at the moment and again these were snapd 2.46, I just remember they were vaguely about setting up sockets and related to the snapd parts not the app parts.

Unfortunately I can’t just pop on Manjaro and test right now as I have something personal to work and HyperV (Where Manjaro isn’t stable) and Virtualbox (where Manjaro is stable) don’t get along, preventing me from just simply starting it up and seeing. I can try poke around in a few hours though.

Otherwise incase anyone else is willing to spin up a VM to take a look sooner, there was literally nothing special about the Manjaro setup bar it was a fresh install from 20.1.2 media. It appears in the LiveCD environment, the installed environment with no updates, and an updated installed environment; and seems to be very reproducable. My only assumption is it isn’t valid on Manjaro installed from earlier media because otherwise I’d imagine it would be a much more pervasive and obvious problem; but there are some posts begining to crop up on Manjaro forums about what I presume is the same issue. (https://forum.manjaro.org/t/snap-corrupted-fonts-tofu-blocks-rectangles/5487/2, https://forum.manjaro.org/t/application-installed-via-snap-does-not-work/35313, https://forum.manjaro.org/t/snap-store-does-not-run/35169, https://forum.manjaro.org/t/snap-store-seg-fault/34430, possibly others)

This what happens when I do sudo snap run snap-store…:

mkdir: cannot create directory '/run/user/0': Permission denied
Warning: Schema “org.gnome.system.locale” has path “/system/locale/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy” has path “/system/proxy/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.http” has path “/system/proxy/http/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.https” has path “/system/proxy/https/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.ftp” has path “/system/proxy/ftp/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
Warning: Schema “org.gnome.system.proxy.socks” has path “/system/proxy/socks/”. Paths starting with “/apps/”, “/desktop/” or “/system/” are deprecated.
07:17:00:0619 GLib-GIO g_app_info_get_name: assertion 'G_IS_APP_INFO (appinfo)' failed
07:17:00:0819 dconf Unable to open dconf/profile/user: Permission denied
07:17:00:0819 dconf unable to create directory '/run/user/0/snap.snap-store/dconf': Permission denied. dconf will not work properly.
07:17:00:0821 dconf unable to create directory '/run/user/0/snap.snap-store/dconf': Permission denied. dconf will not work properly.
07:17:00:0822 dconf unable to create directory '/run/user/0/snap.snap-store/dconf': Permission denied. dconf will not work properly.
07:17:00:0825 dconf unable to create directory '/run/user/0/snap.snap-store/dconf': Permission denied. dconf will not work properly.
07:17:00:0850 dconf unable to create directory '/run/user/0/snap.snap-store/dconf': Permission denied. dconf will not work properly.
07:17:00:0850 dconf unable to create directory '/run/user/0/snap.snap-store/dconf': Permission denied. dconf will not work properly.
07:17:00:0850 dconf unable to create directory '/run/user/0/snap.snap-store/dconf': Permission denied. dconf will not work properly.
07:17:00:0853 Gs enabled plugins: odrs, rewrite-resource, snap, icons, key-colors, key-colors-metadata
07:17:00:0853 Gs disabled plugins: appstream, desktop-categories, desktop-menu-path, dpkg, dummy, fedora-langpacks, fedora-pkgdb-collections, fwupd, generic-updates, hardcoded-blacklist, hardcoded-popular, modalias, os-release, packagekit, packagekit-history, packagekit-local, packagekit-offline, packagekit-proxy, packagekit-refine, packagekit-refine-repos, packagekit-refresh, packagekit-upgrade, packagekit-url-to-app, provenance, provenance-license, repos, systemd-updates
Segmentation fault

… vs snap run snap-store:

07:18:11:0521 Gtk Failed to load module "canberra-gtk-module"
07:18:11:0522 Gtk Failed to load module "canberra-gtk-module"
07:18:11:0785 Gs enabled plugins: odrs, rewrite-resource, snap, icons, key-colors, key-colors-metadata
07:18:11:0785 Gs disabled plugins: appstream, desktop-categories, desktop-menu-path, dpkg, dummy, fedora-langpacks, fedora-pkgdb-collections, fwupd, generic-updates, hardcoded-blacklist, hardcoded-popular, modalias, os-release, packagekit, packagekit-history, packagekit-local, packagekit-offline, packagekit-proxy, packagekit-refine, packagekit-refine-repos, packagekit-refresh, packagekit-upgrade, packagekit-url-to-app, provenance, provenance-license, repos, systemd-updates
07:18:11:0904 Gs /etc/PackageKit/Vendor.conf file not found
07:18:12:0261 Pango pango_font_get_hb_font: assertion 'PANGO_IS_FONT (font)' failed
Segmentation fault (core dumped)

This is the log for a generic snap app I installed. This is what happens when I try running it.

Gtk-Message: 02:19:25.242: Failed to load module "canberra-gtk-module"
Gtk-Message: 02:19:25.242: Failed to load module "canberra-gtk-module"
QApplication: invalid style override passed, ignoring it.
Available styles: Windows, Fusion
Fontconfig warning: FcPattern object weight does not accept value [40 210)
Segmentation fault (core dumped)

I noticed that, indeed, I didn’t have the /etc/PackageKit/Vendor.conf file. I searched and found that that file is made thanks to packagekit. However, even after installing packagekit, the error persists. I’m running Manjaro 20.1.2 XFCE.

I don’t know how Manjaro is set up, but IIRC this will not launch the user session, so /run/user/0 will not be available. With all the desktoy dependencies, I’m not sure it’s supposed to work if you don’t have actual login session.

This is something I see on Arch too and AFAIK it’s caused by fonts cache. I can only get it working if I remove ~/snap/snap-store/common/.cache/* and wipe my fonts cache in ~/.cache/fontconfig.

There is some fallback handling which skips the global fonts cache from the host, but it only triggers when there’s arch or fedora in ID or ID_LIKE in /etc/os-release. What is the content of /etc/os-release on Manjaro?

1 Like

This is something I see on Arch too and AFAIK it’s caused by fonts cache. I can only get it working if I remove ~/snap/snap-store/common/.cache/* and wipe my fonts cache in ~/.cache/fontconfig .

Thank you!! That fixed it for me :slight_smile: Hopefully there is a permanent solution for this problem, since I don’t want to keep clearing my fonts cache whenever I want to run snap, and it’s still taking a long time to run these apps, but better late than never!

What is the content of /etc/os-release on Manjaro?

NAME="Manjaro Linux"
ID=manjaro
ID_LIKE=arch
BUILD_ID=rolling
PRETTY_NAME="Manjaro Linux"
ANSI_COLOR="32;1;24;144;200"
HOME_URL="https://manjaro.org/"
DOCUMENTATION_URL="https://wiki.manjaro.org/"
SUPPORT_URL="https://manjaro.org/"
BUG_REPORT_URL="https://bugs.manjaro.org/"
LOGO=manjarolinux