Upgrading classic snap to core24 using snapcraft 8.3 causes python 3.12 errors at runtime

Hello. I am upgrading a core20 snap to core24, and python3.8 to 3.12.

I’ve changed the snapcraft.yaml syntax to the newer versions and updated the list of packages to stage, and now we build perfectly well, and run on the machine it was built on, but don’t run when moving the snap to another (earlier) ubuntu machine. This makes me think that the packages are not getting staged/linked/patched properly, along with the snapcraft linter saying that libpython3.12.so.1.0 is unused.

The error I get when running on another machine is Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding, which seems to usually be a PYTHONPATH/PYTHONHOME issue, but that doesn’t quite make sense here. Looking at the variables the error spits out, on a working (older) version, when running /snap/certbot/current/bin/python3, sys.prefix is /snap/certbot/current, but on the updated version it’s now /usr; same with exec_prefix. And sys.path is similarly pointing to the /usr/lib pythons instead of inside the snap as well. So maybe this is a problem with patching/rpath somehow? I have added build-attributes: enable-patchelf to the snapcraft.yaml, but did the actual patching get changed instead of just defaulting to no patching? Anything else I’m missing here? I’ve scoured the documentation but have run out of things to try.

All changes can be seen in this PR: Update python version in snaps by ohemorange · Pull Request #9956 · certbot/certbot · GitHub

(This was originally posted as an update to this post from 2020, but since I can’t update the title, I’ve created a new topic.) u p.s. I have read the post on updates to classic confinement but I don’t see anything there about what that means I would need to change in practice to get it working again.

Edit: further investigation indicates that something is wrong with /snap/certbot/current/pyvenv.cfg.

Contents on a working build:

home = /root/parts/certbot/install/bin
include-system-site-packages = false
version = 3.8.10

Contents on the broken build:

home = /root/parts/certbot/install/usr/bin
include-system-site-packages = false
version = 3.12.3
executable = /root/parts/certbot/install/usr/bin/python3.12
command = /root/parts/certbot/install/bin/python3 -m venv --upgrade /root/parts/certbot/install

And then the working build has the following python path variables:

>>> sys._base_executable
'/snap/certbot/x1/bin/python3'
>>> sys.base_prefix
'/snap/certbot/x1/bin/../usr'
>>> sys.base_exec_prefix
'/snap/certbot/x1/bin/../usr'
>>> sys.executable
'/snap/certbot/x1/bin/python3'
>>> sys.prefix
'/snap/certbot/x1'
>>> sys.exec_prefix
'/snap/certbot/x1'
>>> sys.path
['', '/snap/certbot/x1/usr/lib/python38.zip', '/snap/certbot/x1/usr/lib/python3.8', '/snap/certbot/x1/usr/lib/python3.8/lib-dynload', '/snap/certbot/x1/lib/python3.8/site-packages']

and the broken build has instead:

Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/usr/lib/python3.12'
  sys._base_executable = '/snap/certbot/x2/usr/bin/python3.12'
  sys.base_prefix = '/usr'
  sys.base_exec_prefix = '/usr'
  sys.platlibdir = 'lib'
  sys.executable = '/snap/certbot/x2/bin/python3'
  sys.prefix = '/usr'
  sys.exec_prefix = '/usr'
  sys.path = [
    '/usr/lib/python312.zip',
    '/usr/lib/python3.12',
    '/usr/lib/python3.12/lib-dynload',
  ]

Now, I can probably just go ahead and manually set PYTHONHOME and PYTHONPATH, but that seems unlikely to be addressing the root cause of these issues, which I would rather do.

I have the same problem on one of my classic snaps as well. We tried upgrading the snap from core22 to core24 to pull in a newer ROCm driver version, but our Python-based snap hooks would break if we attempted to install the updated snap on earlier releases of Ubuntu.

In the end we had to revert back to core22 since we didn’t have the time to debug the issues with upgrading to core24. Here’s our open issue that contains a similar stacktrace:

https://github.com/charmed-hpc/slurm-snap/issues/19

Now, I can probably just go ahead and manually set PYTHONHOME and PYTHONPATH, but that seems unlikely to be addressing the root cause of these issues, which I would rather do.

Update: When I manually set PYTHONPATH to $SNAP/lib/python3.12/site-packages:$SNAP/usr/lib/python3/dist-packages:$SNAP/usr/lib/python3.12:${PYTHONPATH} or variants and PYTHONHOME to $SNAP for runtime, I run into AttributeError: partially initialized module 'charset_normalizer' has no attribute 'md__mypyc' (most likely due to a circular import), which happens inside requests. I do not know why.

Similarly, I do not know why simply setting the PYTHONPATH works for these two classic snaps and not for us.

I also tried no longer manually creating the venv in override-build and just letting the python plugin create the venv by running craftctl default first instead, but that made no difference.

I went and opened a bug on craft-parts - the package that provides the python plugin - and included our respective error traces. Please feel free to comment any additional info that I might have left out on the issue thread on GitHub :smiley:

https://github.com/canonical/craft-parts/issues/786

cc @lengau, @mr_cal, @sergiusens :eyes: :eyes:

Thanks for the bug report - we’ll need to dedicate resources to investigating and solving this problem.

I was going to suggest running snap run --shell certbot and inspecting the environment there for LD_LIBRARY_PATH and PYTHON_PATH to see why it’s trying to load the system’s python. Based on your last comment and error message, I’m no longer sure what the problem is or where to look next.

@tigarmo has a good understanding of the Python plugin and may be able to help when he returns next week.

Exactly, PYTHONPATH is what I set it manually and otherwise unset, and also seems to need PYTHONHOME set. LD_LIBRARY_PATH is unset. I at one point also tried setting it to $SNAP/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/blas:$SNAP/usr/lib/$SNAPCRAFT_ARCH_TRIPLET/lapack:$LD_LIBRARY_PATH as in the removed code, and that did not seem to help.

I did manage to build a core22 version. That spit out even more linter errors than core24, but does seem to be working. Despite not setting PYTHONPATH manually, it is set in the env after running snap run --shell certbot.certbot. PYTHONHOME and LD_LIBRARY_PATH are unset. pyvenv.cfg is the same as in core20 version (obviously using 3.10 though). I still do not understand how that can be relevant, given that /root/parts/certbot/install/bin is not available after building the snap.

core22 linter:

Lint warnings:                                                                                                                                                         
- library: libexpat.so.1: unused library 'lib/x86_64-linux-gnu/libexpat.so.1.8.7'. (https://snapcraft.io/docs/linters-library)                                         
- library: libtirpc.so.3: unused library 'lib/x86_64-linux-gnu/libtirpc.so.3.0.0'. (https://snapcraft.io/docs/linters-library)                                         
- library: libz.so.1: unused library 'lib/x86_64-linux-gnu/libz.so.1.2.11'. (https://snapcraft.io/docs/linters-library)                                                
- library: libaugeas.so.0: unused library 'usr/lib/x86_64-linux-gnu/libaugeas.so.0.25.0'. (https://snapcraft.io/docs/linters-library)                                  
- library: libicuio.so.70: unused library 'usr/lib/x86_64-linux-gnu/libicuio.so.70.1'. (https://snapcraft.io/docs/linters-library)                                     
- library: libicutest.so.70: unused library 'usr/lib/x86_64-linux-gnu/libicutest.so.70.1'. (https://snapcraft.io/docs/linters-library)                                 
- library: libpython3.10.so.1.0: unused library 'usr/lib/x86_64-linux-gnu/libpython3.10.so.1.0'. (https://snapcraft.io/docs/linters-library) 

core24 linter:

Lint warnings:                                                                                                                                                         
- library: libaugeas.so.0: unused library 'usr/lib/x86_64-linux-gnu/libaugeas.so.0.25.0'. (https://snapcraft.io/docs/linters-library)                                  
- library: libicuio.so.74: unused library 'usr/lib/x86_64-linux-gnu/libicuio.so.74.2'. (https://snapcraft.io/docs/linters-library)                                     
- library: libicutest.so.74: unused library 'usr/lib/x86_64-linux-gnu/libicutest.so.74.2'. (https://snapcraft.io/docs/linters-library)                                 
- library: libpython3.12.so.1.0: unused library 'usr/lib/x86_64-linux-gnu/libpython3.12.so.1.0'. (https://snapcraft.io/docs/linters-library)   

Some updates from further investigation:

  • The two examples I found on github where it seems to be working with python plugin, core24, classic environment both don’t seem to care what version of python you use to run them, which is good because both of them use the system’s own python instead of the python packaged in the snap. So at least they’re not indications that we’re just missing something here.
  • After the release of python 3.10, cpython rewrote some of how it handles getting paths. So that could be related. Since python grabs the home from the malformed pyvenv.cfg, setting PYTHONHOME will have it look for libraries in the right place. Still not sure why it gets that circular import error though. I have that above but in case the full log helps:
Traceback (most recent call last):
  File "/snap/certbot/x1/bin/certbot", line 5, in <module>
    from certbot.main import main
  File "/snap/certbot/x1/lib/python3.12/site-packages/certbot/main.py", line 6, in <module>
    from certbot._internal import main as internal_main
  File "/snap/certbot/x1/lib/python3.12/site-packages/certbot/_internal/main.py", line 24, in <module>
    from acme import client as acme_client
  File "/snap/certbot/x1/lib/python3.12/site-packages/acme/client.py", line 20, in <module>
    import requests
  File "/snap/certbot/x1/lib/python3.12/site-packages/requests/__init__.py", line 48, in <module>
    from charset_normalizer import __version__ as charset_normalizer_version
  File "/snap/certbot/x1/lib/python3.12/site-packages/charset_normalizer/__init__.py", line 24, in <module>
    from .api import from_bytes, from_fp, from_path, is_binary
  File "/snap/certbot/x1/lib/python3.12/site-packages/charset_normalizer/api.py", line 5, in <module>
    from .cd import (
  File "/snap/certbot/x1/lib/python3.12/site-packages/charset_normalizer/cd.py", line 14, in <module>
    from .md import is_suspiciously_successive_range
AttributeError: partially initialized module 'charset_normalizer' has no attribute 'md__mypyc' (most likely due to a circular import)

In python3.10, after setting PYTHONPATH, os.py is found in /snap/certbot/x2/bin/../usr/lib/python3.10/os.py in the sanity check used to see if the “platform independent libraries” etc errors should shown, indicating that sys.base_exec_prefix had already by that point set to /snap/certbot/x2/bin/../usr/. In 3.12, it only searches the path mentioned in the malformed pyvenv.cfg and the host python install. This may indicate that these problems are being caused inside python.

Another update – I think we’ve found what’s going on here, and there’s a couple things.

  1. With the aforementioned python3.11/12 changes, some internal paths will be wrong, and since it uses those to check for the os.py install location, it will spit out those warnings. But they seem to be ignorable, as the actual paths needed are correct. It can be silenced by additionally setting PYTHONHOME to $SNAP.
  2. The circular import error is caused by charset_normalizer looking for the private module lib-dynload, which for some reason lives in usr/lib/python3.12/lib-dynload instead of lib/python3.12/lib-dynload. It was lying about being a circular import error, it’s actually just a module not found error. As such, adding $SNAP/usr/lib/python3.12/lib-dynload to PYTHONPATH should fix that error. (I haven’t investigated why, but just adding $SNAP/usr/lib/python3.12 won’t work.) Further info and some other workarounds in case anyone wants to actually fix this: https://stackoverflow.com/questions/74182807/no-module-named-charset-normalizer-md-mypyc
  3. For a bonus side note that I have not investigated deeply, for some reason if python3-venv is not specified in stage-packages (even if it is specified in build-packages), it will not be available at build time. That seems illogical.

Some other helpful links that came up in the investigation process that I’m saving for posterity:

2 Likes

Further further update: most things can be fixed nicely by simply unstaging pyvenv.cfg, which seems to just be confusing python at this point.

And then you can just set a short little PYTHONPATH and that’s it!

So the key bits:

...
confinement: classic
base: core24
...

# this could probably also be set in the apps but it does seem to be necessary
environment:
  PYTHONPATH: "$SNAP/lib/python3.12/site-packages:${PYTHONPATH}"

apps:
  certbot:
    # some people have issues if they don't call bin/python [appname]
    command: bin/python3 -s $SNAP/bin/certbot
    environment:
      PATH: "$SNAP/bin:$SNAP/usr/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
      # this may not be necessary for everything but we've had it forever

parts:
  certbot:
    plugin: python
    source: .
    ...
    stage:
      - -pyvenv.cfg # super important! this is the magic line!
    stage-packages:
      - libpython3.12-dev
      # added to stage python:
      - libpython3-stdlib
      - libpython3.12-stdlib
      - libpython3.12-minimal
      - python3-pip
      - python3-wheel
      - python3-venv
      - python3-minimal
      - python3-pkg-resources
      - python3.12-minimal
    build-packages:
      - python3-dev
    ...
    build-attributes:
      - enable-patchelf

Not all included packages may be necessary, but these work.

4 Likes

We can consider removing pyvenv.cfg through the plugin for classic if it does not provide major side effects. @tigarmo want to look into this?

@erica thanks for the report and investigation. I’m looking at a potential fix here: instead of removing pyvenv.cfg and setting PYTHONPATH manually we’ll set the home value to the expected final location of the venv on the snap. So in your override-build, replace home = /root/parts/certbot/install/usr/bin with home = /snap/certbot/current/usr/bin. Like this:

    # ...
    override-build: |
      craftctl default
      sed -i "${CRAFT_PART_INSTALL}/pyvenv.cfg" \
          -e 's@^home = '"${CRAFT_PART_INSTALL}"'/usr/bin$@home = /snap/certbot/current/usr/bin@g'

Can you try that out and report back on whether that fixes the issue? @nuccitheboss if you could try that out on the slurm snap that would also be great.

1 Like

@tigarmo When building on core22, the resulting pyvenv.cfg looked like this for imagecraft.

home = /build/imagecraft/stage/usr/bin
include-system-site-packages = false
version = 3.10.12

Now, on core24 by default I have:

home = /root/stage/usr/bin
include-system-site-packages = false
version = 3.12.3
executable = /root/stage/usr/bin/python3.12
command = /root/stage/usr/bin/python3 -m venv /root/parts/imagecraft/install

So I tried a couple of things and ended up with:

home = /snap/imagecraft/current/usr/bin
include-system-site-packages = false
version = 3.12.3
executable = /snap/imagecraft/current/usr/bin/python3.12

(note the command line removed)

But I still get the same error when executing the resulting snap:

 % imagecraft
Python path configuration:
  PYTHONHOME = '/usr/'
  PYTHONPATH = '/snap/imagecraft/x2/lib/python3.12/site-packages:'
  program name = '/snap/imagecraft/x2/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/usr/lib/python3.12'
  sys._base_executable = '/snap/imagecraft/x2/bin/python'
  sys.base_prefix = '/usr/'
  sys.base_exec_prefix = '/usr/'
  sys.platlibdir = 'lib'
  sys.executable = '/snap/imagecraft/x2/bin/python'
  sys.prefix = '/usr/'
  sys.exec_prefix = '/usr/'
  sys.path = [
    '/snap/imagecraft/x2/lib/python3.12/site-packages',
    '/home/paul/projects/imagecraft',
    '/usr/lib/python312.zip',
    '/usr/lib/python3.12',
    '/usr/lib/python3.12/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007d65b19d9300 (most recent call first):
  <no Python frame>

So either I still need to fix the pyvenv.cfg more or something else should be fixed.

I also tested the solution of @erica and this is not working in my case. I confirm I have no pyvenv.cfg in the resulting snap but this is failing the same way.

@tigarmo yes, that fix is working. You can see tests passing here and I double checked manually as well by downloading the snap artifact and running on a focal instance.

Would this be something that could be fixed in the underlying python snap or would this just be an alternative manual fix?