[SOLVED] UTF-8 not supported in Snap

snapd
1

The following script fails from within Snap if its argument contains non-ascii characters…:

#!/usr/bin/env python3
import sys
print(sys.argv[1])

Just run it as:

$ name.test á
Traceback (most recent call last):
  File "/snap/kml-maintenance/x14/usr/bin/test.py", line 10, in <module>
    print(args.filename)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udcc3' in position 43: surrogates not allowed

snapcraft.yml:

grade: stable
confinement: classic

apps:
  test:
    command: usr/bin/test.py

parts:
  python3:
    plugin: python
    python-version: python3
    install: |
      install app/* $SNAPCRAFT_PART_INSTALL/usr/bin/
4

Export the locale in environment. snaps default to a C locale.

snapcraft uses C.UTF-8, is python3 based and works fine fwiw.

5

Could you please link me to the documentation or give me a working snippet? https://docs.snapcraft.io/reference/env doesn’t teach how to change the environment, only how to read it.

6

The following doesn’t make any difference:

apps:
  aerial:
    command: usr/bin/aerial.py
    environment:
      LC_ALL: pt_BR.UTF-8
      LANG: pt_BR.UTF-8
7

This may be related to Click (http://click.pocoo.org/5/python3/), but print(sys.argv[1].encode(‘utf-8’)) does not work… I’m really lost on what to do…

8

The core snapd doesn’t ship locale data so the only working locale is C. This is a long standing issue that we plan to address in the core18 cycle (now) but it is not fixed yet.

For now applications need to ship their own locale data if they require it.

9

It should work provided that @teresaejunior is using the python plugin (that would bring in the minimal locales). It all depends on how usr/bin/test.py is making it onto the system, this is confinement classic which could bring in the additional confusion.

For what it’s worth, snapcraft also uses click to handle the cli. We solved the locale issue in code https://github.com/snapcore/snapcraft/blob/master/snapcraft/cli/main.py#L27 given that we also had issues with docker (and any other environment that does not default to utf-8).

10

This may be related to click+Docker indeed! The click website talks about “Python 3 Surrogate Handling”, although I haven’t yet found a working solution.

I have uploaded a minimal test example with the code and a pre-built snap here: https://gitlab.com/teresaejunior/snap-test

It was built using sudo ./install.sh, and can be tested with test.main €.

I have not tested without Docker yet, but from other problem I had posted here in these forums, I found that I should run snapcraft from Xenial, and my laptop runs Bionic. That is why Docker comes in very handy.

11

To simplify the issue, don’t use docker. Just use snapcraft cleanbuild. Snapcraft can spawn containers by itself and will do the right thing for you.

12

It is very unfortunate, but using the very same code uploaded to teresaejunior/snap-test, the same error appears with snapcraft cleanbuild:

$ snap install snapcraft --classic
$ snap install lxd
$ sudo usermod -g lxd ${USER}
$ newgrp lxd
$ sudo lxd init # yes to all
$ sudo snapcraft cleanbuild
$ sudo snap install --dangerous --classic ./*.snap
$ test.main €
Traceback (most recent call last):
  File "/snap/test/x4/usr/bin/test.py", line 3, in <module>
    print(sys.argv[1])
UnicodeEncodeError: 'utf-8' codec can't encode character '\udce2' in position 0: surrogates not allowed
13

So something got lost in translation. when I mentioned we had issues with docker, I was talking about the snapcraft codebase itself and not projects built with snapcraft.

Now that we can see your project, I am almost certain that everything will work if you either:

  • use the correct shebang in https://gitlab.com/teresaejunior/snap-test/blob/master/test.py, the current one resolves to what’s on the system (given that you are using classic confinement, it means you have no control over what’s on that system); to correct this you can create a proper setup.py and install it as a console_script or using entry_points; snapcraft will do the right thing and fix the shebang.
  • alternatively, in snapcraft.yaml, use command: $SNAP/usr/bin/python3 $SNAP/usr/bin/test.py
1 Like
14

Finally it worked! Thank you so much to both of you for your help!