Lack of compiled locales breaks gettext based localisation

snapcraft
1

One of the issues that came up at the Snapcraft Summit is that many snaps of GTK based applications only displayed in English, even when the user’s locale was set to something else.

As an example, I looked at @kenvandine’s snap of iagno. The snap contains gettext message catalogues for many languages in $SNAP/usr/share/locale, and it looks like the executable is passing the correct path to gettext:

$ strings /snap/iagno/current/usr/bin/iagno | grep share/locale
/snap/iagno/current/usr/share/locale

However, when I try to run the app, I get a warning on the console and the game displays in English:

$ LANG=fr_FR.UTF-8 LANGUAGES=fr_FR:fr iagno

(iagno:489): Gtk-WARNING **: Locale not supported by C library.
    Using the fallback 'C' locale.

Looking into the source, this warning indicates that the C library setlocale call failed, so when gettext checks the current locale for translation it is still in the C locale.

This seems to be due to the core snap only containing a definition for C.UTF-8. If I manually compile a French locale and make it available in a location the snap can see it, and run the app again I get a correctly translated UI:

$ localedef -i fr_FR -f UTF-8 ~/snap/iagno/common/fr_FR.UTF-8
$ snap run --shell iagno
(inside snap environment)
$ export LOCPATH=$SNAP_USER_COMMON
$ export LANG=fr_FR.UTF-8 LANGUAGE=fr_FR
$ $SNAP/command-iagno.wrapper

To make the app work for all languages, we’d need access to compiled versions of all locales. That presents the question of who should provide them?

To get a feel for the size of this data, the Ubuntu locales-all package has an installed size of 130 MB, and compressed size of ~ 3.4 MB (which is probably a good measure of the squashfs size increase). It might slim down slightly with the data packaged into a locale-archive file, but probably not hugely.

That seems like quite a lot to add to each individual snap package that uses the C locale system. We might be able to avoid that for some apps by providing the data in the GNOME platform snap, but there are a lot of snaps that don’t make use of it.

Could this kind of thing be seen as important enough to put in the core snap? Or is it an impetus to start work on creating base snaps?

5 Likes
2

Note that the snapcraft-desktop-helpers parts for gtk2, gtk3, qt4 and qt5 all install locales-all in the stage-packages, thus enabling localization but also increasing the size of all the individual snaps.

1 Like
3

We also had some snaps with launcher scripts generating the currrent locale at first start, e.g

“# generate the locale definition for the current language
export I18NPATH=$SNAP/usr/share/i18n
export LOCPATH=$SNAP_USER_DATA/.local/share/locale
if [ ! -d $LOCPATH ]; then
mkdir $LOCPATH
fi
if [ ! -d $LOCPATH/$LANG ]; then
$SNAP/usr/bin/localedef -f UTF-8 -i $(echo $LANG | cut -d. -f1) $LOCPATH/$LANG
fi”

(with “locales” staged)

1 Like
4

Perhaps if we include locales-all in the platform snap and add that to the desktop helper it would be enough. I can give that a try.

1 Like
5

The gnome-platform snaps already include locales-all and the desktop-launcher script set the environment. We looked at bit more at the issue and debugged on IRC and it turns out a one liner fix to the env setting is needed and translations are working (doesn’t resolve the issue for snaps not using the launcher though)

6

This has now been fixed in the the desktop helpers, at least for the snaps that use the GNOME platform snap.

7

@kenvandine what is the plan for the other helpers? You said that the gnome platform helper is fixed, which leads me to wonder if that means the other helpers still need work?

It would be best to keep all these fixes in sync across all the helpers so that we don’t get into a case where it’s said that “locales are fixed in the desktop helpers” and someone then complains that their qt5 app isn’t correctly localised because they thought it was supposed to “just work” now.

Keeping the helpers in sync also means we’re not having to track which helper has which fix and which still need to be worked on.

8

I totally agree. The fix today was a bug in the script that would only affect the gnome platform runtime. I haven’t debugged the others yet. Do you have examples that aren’t working?

9

From what @oSoMoN said, the problem was limited to those packages using the gnome-platform snap via the content interface. The other desktop-* cloud parts would cause locales-all to be bundled with the snap. The problem was that desktop-launch would adjust $LOCPATH to find locales bundled with the snap, but not those made available via the GNOME platform snap.

I still think it’d be worth thinking of the best way to present this data to the application. If we’re shipping all locales, it would probably be best to do it in the form of a single file /usr/lib/locale/locale-archive provided by a base snap: that would allow the application to mmap a single file to use any locale. That would remove the need to set $LOCPATH (which also disables use of the archived locales).

2 Likes
10

Thanks for the explanation :slight_smile: I like the idea of pushing something as fundamental as localisation to the lowest possible point that we can. In this case I think that Core makes the most sense because almost every snap is going to want to access this data, including those that aren’t using a platform snap.

1 Like
11

desktop-gtk2, desktop-gtk3, desktop-qt4 and desktop-qt5 all stage the locales (locales-all). If any of those aren’t working, please file an issue.

I see some benefits of having a separate desktop base that includes such things as locales, we should definitely consider that.

1 Like
12

I brought up the idea of base snaps primarily because my understanding was that the plan was to reduce the core snap down to just what was needed to support snapd itself, and have everything else use a base snap. I don’t think Go code generally uses the C library locale, so it isn’t immediately obvious that they should be added to core going forward.

With that said, I think most base snaps would probably want to include locale data. While there are some high profile frameworks/languages that implement their own locale systems, there is a lot of code using libc for collation, number and date formatting, etc.

2 Likes
13

That is a most excellent point!

https://uproxx.files.wordpress.com/2015/06/bill_ted.jpg?quality=100&w=650&h=400

14

Could this happen with vlc.snap, because my regional language is spanish, but vlc come in english?

15

NOTE that command line interface snaps also requires localization, and has no reason to use the desktop helpers.

16

Is there any progress on including the compiled locale data into the core snap? I suppose snapd and other utilities provided in the core snap can also make use of the localization.

17

I’ve made a launcher based on @seb128’s script:

https://github.com/Lin-Buo-Ren/hello-snap/blob/master/snap/local/launchers/locales-launch

1 Like
18 
11210752 hello-snapcrafters_2.10-19-g4d0301f+pkg-4d03_amd64.locale-all.snap
 4567040 hello-snapcrafters_2.10-19-g4d0301f+pkg-4d03_amd64.snap

The squashed size is about ~7MiB greater than the one that doesn’t bundle the locale-all package (generated at runtime).

Though I still believe that it should belong to the core snap, as it is ~20MiB more disk usage per snap.

19

I’ve created a remote part for this issue: [OBSOLETED] The locales-launch remote part

1 Like
20

I noticed that in order to make Gettext localization work locale compilation isn’t necessary, simply set LANG and all LC_* environment variable to C.UTF-8 and voila! It works.

Here are the experiment details:

$ snap run --shell hello-snapcrafters
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

# Apply adapter
$ eval "$(head --lines=-1 $SNAP/command-hello-snapcrafters.wrapper)"
$ locale
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=zh_TW.UTF-8
LANGUAGE=zh_TW:zh_HK:zh_CN:en_US:en
LC_CTYPE="zh_TW.UTF-8"
LC_NUMERIC=zh_TW.UTF-8
LC_TIME=zh_TW.UTF-8
LC_COLLATE="zh_TW.UTF-8"
LC_MONETARY=zh_TW.UTF-8
LC_MESSAGES="zh_TW.UTF-8"
LC_PAPER=zh_TW.UTF-8
LC_NAME=zh_TW.UTF-8
LC_ADDRESS=zh_TW.UTF-8
LC_TELEPHONE=zh_TW.UTF-8
LC_MEASUREMENT=zh_TW.UTF-8
LC_IDENTIFICATION=zh_TW.UTF-8
LC_ALL=
$ hello-
hello-launch        hello-snapcrafters  
$ hello-snapcrafters 
Hello, world!

# Fix LANG
$ LANG=C.UTF-8
$ locale
locale: Cannot set LC_ALL to default locale: No such file or directory
LANG=C.UTF-8
LANGUAGE=zh_TW:zh_HK:zh_CN:en_US:en
LC_CTYPE="C.UTF-8"
LC_NUMERIC=zh_TW.UTF-8
LC_TIME=zh_TW.UTF-8
LC_COLLATE="C.UTF-8"
LC_MONETARY=zh_TW.UTF-8
LC_MESSAGES="C.UTF-8"
LC_PAPER=zh_TW.UTF-8
LC_NAME=zh_TW.UTF-8
LC_ADDRESS=zh_TW.UTF-8
LC_TELEPHONE=zh_TW.UTF-8
LC_MEASUREMENT=zh_TW.UTF-8
LC_IDENTIFICATION=zh_TW.UTF-8
LC_ALL=
$ hello-snapcrafters 
Hello, world!

# Fix all LC_* variables
$ for locale_variable_name in \
    $(printenv | grep ^LC_ | cut --delimiter== --fields=1); do \
    declare -n locale_variable="${locale_variable_name}"; \
    locale_variable=C.UTF-8; \
done
$ locale
LANG=C.UTF-8
LANGUAGE=zh_TW:zh_HK:zh_CN:en_US:en
LC_CTYPE="C.UTF-8"
LC_NUMERIC=C.UTF-8
LC_TIME=C.UTF-8
LC_COLLATE="C.UTF-8"
LC_MONETARY=C.UTF-8
LC_MESSAGES="C.UTF-8"
LC_PAPER=C.UTF-8
LC_NAME=C.UTF-8
LC_ADDRESS=C.UTF-8
LC_TELEPHONE=C.UTF-8
LC_MEASUREMENT=C.UTF-8
LC_IDENTIFICATION=C.UTF-8
LC_ALL=
$ hello-snapcrafters 
哈囉，大家好！
$ LANGUAGE=ja hello-snapcrafters 
世界よ、こんにちは！
1 Like