My pick snap (https://snapcraft.io/pick-colour-picker) has had a bug reported against the snap version (https://github.com/stuartlangridge/ColourPicker/issues/60). Apparently all the text glyphs are replaced with empty Unicode boxes, as per this screenshot from the bug:
https://i.imgur.com/PfN85fN.png
I have no idea what might cause this at all. Does anyone have any suggestions? I don’t know if it’s related to it running on Fedora; I can say that it doesn’t happen for me, on Ubuntu.
Off hand, I don’t know what is going on, but it would be helpful if you could find out whether any errors are printed on the user’s system when this happens. Get them to run something like:
# Ensure we are building the sandbox mount namespace from scratch
sudo /usr/libexec/snapd/snap-discard-ns pick-colour-picker
pick-colour-picker
(I think libexec is right for Fedora builds of snapd). That could help shed some light on what’s going on, more than just a screenshot.
Response from the user:
The source_remove stuff is a known bug in Pick, and unlikely to be relevant. The only other relevant thing in their output seems to be Failed to load module “pk-gtk-module” – I don’t know what that means, but perhaps it’s relevant to this issue of the fonts not loading?
The “Failed to load module” warning would be due to a GTK module listed in $GTK_MODULES not existing within the sandbox, which is generally harmless.
A quick Google indicates that this particular module is associated with PackageKit, intended to detect and install missing fonts:
This kind of functionality should be irrelevant to an app like yours though: if your GUI is using the user’s default font, it should already be available.
I’m not sure what to suggest next. Perhaps see if the problem exists in other graphical snaps, or if there are errors in dmesg. I’m not that familiar on the details of confinement on Fedora.
I entered this in the bug, but to repeat it here: I installed Fedora 30 Workstation, installed Pick, and it worked. There’s something more going on in the user’s installation; hoping to hear back about it.
FTR the issue was an /etc/fonts/fonts.conf that the fontconfig in the snaps couldn’t understand:
~$ snap run --shell pick-colour-picker
bash-4.4$ LD_LIBRARY_PATH=/snap/gnome-3-28-1804/current/usr/lib/x86_64-linux-gnu/ /snap/gnome-3-28-1804/current/usr/bin/fc-list
Fontconfig warning: "/etc/fonts/fonts.conf", line 5: unknown element "its:rules"
Fontconfig warning: "/etc/fonts/fonts.conf", line 6: unknown element "its:translateRule"
Fontconfig error: "/etc/fonts/fonts.conf", line 6: invalid attribute 'translate'
Fontconfig error: "/etc/fonts/fonts.conf", line 6: invalid attribute 'selector'
Fontconfig error: "/etc/fonts/fonts.conf", line 7: invalid attribute 'xmlns:its'
Fontconfig error: "/etc/fonts/fonts.conf", line 7: invalid attribute 'version'
Fontconfig warning: "/etc/fonts/fonts.conf", line 9: unknown element "description"
Fontconfig error: Cannot load config file from /etc/fonts/fonts.conf
the user’s fonts.conf had been fiddled with by either fontconfig-infinality or “fonts tweak tool” [sic]; reinstalling the default fonts.conf from fedora 30 resolved the issue.
maybe snap authors (or desktop-launch?) can detect and alert the user about this scenario?
I feel like this is a bit of a cross-platform problem, no? If I’m to have the same snap on every platform, but I bundle a fontconfig which may not know how to read the font configuration on a platform… what should I do about that? It doesn’t seem like a user problem to fix – in particular, telling a user “you can’t have this app because you’ve fiddled with your system fonts” is rather unfriendly, since all their other apps don’t fail.
(Also, does this break all snaps installed on a platform where they’ve added infiniality?)
Let me start with saying I’m not a fontconfig expert or anything like it. I know nothing about it.
But, you have two options: you give a snap access to the system’s fonts, or you don’t. You can’t have it both ways.
As I understand it fonts.conf is part of “the system’s fonts”. If this is so, then, yeah.
@jamesh would it be possible to patch fontconfig to not error on invalid attributes? Seems like it should be reasonable to warn on those rather than error. Thoughts?
TBH this looks like another evidence in favor of carrying our own /etc.
that would mean we woulnd’t find the system fonts, unless you’re saying we should read the system’s font config and translate it into something the snap understands. Which, … WAT.
From the sound of all this, the issue is that fontconfig and fonts.conf seem to be a matched pair, but in a snap world, one’s in the snap and the other’s in the system, so they can get mismatched. I don’t know why fontconfig lives in the snap – I know almost nothing about all this – but that may be an untenable situation. In the best case, patching fontconfig in the snap to ignore unknown attributes will mean that snapped apps have a different font configuration and thus look different from non-snapped apps, which is a bad place to be and causes people to reject snapped apps as looking out of place (as it does when a snapped app ignores the system theme, which I know some work has recently gone into). In the worst case, the promise of snapped apps (“build a snap once and it’ll work everywhere”) becomes “build a snap once and it’ll work some places, unless the people have customised their system, in which case it won’t work”, which makes me think “oh, then building a snap is not as useful as I thought”.
If libfontconfig failes to parse /etc/fonts/fonts.conf, it will instead load up a compiled in default configuration file:
https://gitlab.freedesktop.org/fontconfig/fontconfig/blob/master/src/fcinit.c#L38
This is essentially enough to find fonts in /usr/share/fonts and ~/.local/share/fonts, and cache files in /var/cache/fontconfig and ~/.cache/fontconfig. So while we’re definitely seeing errors related to unknown XML elements and attributes, it isn’t obvious that this is what is causing the font lookup failure.
I’ve definitely seen snaps exhibiting the same error messages when they ship an old libfontconfig and encounter new configuration files and still display fine, so there must be something else going on here.
As for patching fontconfig to make its config file parsing forward compatible, that’s still not going to get rid of all the snaps that have packaged ancient versions of the library. And if you are going to update a snap, you may as well upgrade to a newer fontconfig that will both fix the config file parsing and be compatible with the cache files on newer distros.
A user reported the same problem on Arch. I can reproduce it locally, here’s my gnome-logs: https://i.imgur.com/5sLAJTs.png
My fontconfig version is:
fontconfig 2:2.13.91+23+g65087ac-1
Looking at the change in PKGBUILD, it got updated just recently:
-pkgver=2.13.1+12+g5f5ec56
+pkgver=2.13.91+23+g65087ac
Anything I can do to debug this?
Edit:
this is the list of font related path the snap app touched: https://paste.ubuntu.com/p/C39dpbfyHC/ collected with snap run --strace='-e openat')
The log with FC_DEBUG=5 https://paste.ubuntu.com/p/83CNv4NncM/
So I’ve narrowed it down a bit.
In case of poedit and gnome-* snaps, fonts.conf comes from the content snap, which is gnome-3-26-1604 and gnome-3-28-1804 respectively. At this point it seems that the problem is more in the gnome-3-28-1804 snap than anything else. @jamesh @kenvandine any ideas what to try next?
I think this is more likely a problem related to the snaps using core18. Any snap using gnome-3-28-1804 has a base of core18. Maybe @mvo has some ideas?
I’m not sure. Krita is a core18 snap and the problem is not observed there. Besides, core18 does not deliver libfontconfig*, it’s part of gnome platform snaps:
maciek@galeon:~ find /snap/gnome-3-2*-*/current/ -name '*fontco*.so*'
/snap/gnome-3-26-1604/current/usr/lib/x86_64-linux-gnu/libfontconfig.so.1
/snap/gnome-3-26-1604/current/usr/lib/x86_64-linux-gnu/libfontconfig.so.1.11.1
/snap/gnome-3-28-1804/current/usr/lib/x86_64-linux-gnu/libfontconfig.so.1
/snap/gnome-3-28-1804/current/usr/lib/x86_64-linux-gnu/libfontconfig.so.1.10.1
Krita comes with the same version of libfontconfig as gnome-3-26-1604 though:
maciek@galeon:~ find /snap/krita/current/ -name '*fontco*.so*'
/snap/krita/current/usr/lib/x86_64-linux-gnu/libfontconfig.so.1
/snap/krita/current/usr/lib/x86_64-linux-gnu/libfontconfig.so.1.10.1
It is almost definitely related to the fontconfig cache files rather than access to the fonts themselves.
Presumably the core18 snaps are trying (and failing) to use the host system fontconfig cache files, while the core16 snaps ignore the host system caches as too new. I wonder if there have been any changes to the data structures between 2.13 and 2.13.91 without a corresponding bump in the cache version?
So, it looks like the big change in 2.13.91 is the revert of the UUID-based cache file manangement code introduced in 2.13:
https://gitlab.freedesktop.org/fontconfig/fontconfig/commit/c4324f54ee16e648ba91f3e9c66af13ab3b1754c
It looks like this is still “cache version 7”, which it has been since fontconfig 2.11.95. If anything, this should be more compatible with the older fontconfig found in gnome-3-28-1804.
Thanks for the tip. I’ve tweaked the mount namespace of the snap and mounted a clean tmpfs over /var/cache/fontconfig. At this point the fonts were rendered correctly, although the start time was obviously longer.
Now, the questions remains is how we can address this in most hands-off fashion for the users.