How do I correctly install a custom python version and a local wheel package?

I’m working on snapping an internal python tool which has a few requirements that need to be met. Specifically, it needs python 3.7 and based on the architecture that I’m building on, I need to install a different local wheel that I built ahead of time.

I seem to have figured out how to install python 3.7 correctly as a part but I’m having some difficulties installing the wheels. Specifically, they go through the install flow but for reasons that I don’t understand, their contents don’t end up in the resulting .snap file.

This is how my parts look like:

parts:
  python37:
    source: https://www.python.org/ftp/python/3.7.3/Python-3.7.3.tar.xz
    source-type: tar
    source-checksum: md5/93df27aec0cd18d6d42173e601ffbbfd
    plugin: autotools
    configflags:
      - --prefix=/usr
    build-packages:
      - libbz2-dev
      - libexpat1-dev
      - libffi-dev
      - libgdbm-dev
      - liblzma-dev
      - libncurses5-dev
      - libreadline-dev
      - libsqlite3-dev
      - libssl-dev
      - libzip-dev
      - uuid-dev
    stage-packages:
      - libbz2-1.0
      - libexpat1
      - libffi6
      - libgdbm3
      - liblzma5
      - libncurses5
      - libreadline6
      - libsqlite3-0
      - libssl1.0.0
      - libzip4
      - uuid-runtime
    stage:
      - -usr/lib/python3.7/test
      - -var

  my-app:
    plugin: python
    source: .
    stage-packages:
      - libboost-python1.58.0
      - libcurl3
      - libpython3.5
    stage:
      - -usr/bin/2to3
      - -usr/bin/pydoc3
      - -usr/bin/python3
    after:
      - python37
    override-stage: |
      case "$SNAPCRAFT_ARCH_TRIPLET" in
          *arm*)
              WHEEL_FILE="$SNAPCRAFT_PART_BUILD/3rdparty/wheels/azure_iothub_device_client-1.4.6-py3-none-linux_armv7l.whl" ;;
          *x86_64*)
              WHEEL_FILE="$SNAPCRAFT_PART_BUILD/3rdparty/wheels/azure_iothub_device_client-1.4.6-py3-none-manylinux1_x86_64.whl" ;;
          *)
              echo "Unsupported architecture!"
              exit 1 ;;
      esac

      python3.7 -m pip install -I $WHEEL_FILE
      snapcraftctl stage

The reason I need to install those wheels like this is because the package present on pypi comes with a precompiled binary that links against a different version of boost than what’s present in the base (on arm builds). So to solve it, I re-built the wheels locally and am overwriting the dependency that’s being pulled from pypi.

It’s not immediately clear from the available documentation that for this to work you don’t install packages as you would have done it normally outside of snapcraft (i.e. simple pip install <path to package>).

By studying the source code of the python plugin I came across these lines from which I learned that for this to work you would have to do something along the lines of: PYTHONUSERBASE=$SNAPCRAFT_STAGE python3.7 -m pip install --user -I $WHEEL_FILE.

So basically, to solve my issues I updated the my-app part as follows:

my-app:
  plugin: python
  source: .
  stage-packages:
    - libboost-python1.58.0
    - libcurl3
    - libpython3.5
  stage:
    - -usr/bin/2to3
    - -usr/bin/pydoc3
    - -usr/bin/python3
  after:
    - python37
  override-stage: |
    case "$SNAPCRAFT_ARCH_TRIPLET" in
        *arm*)
            WHEEL_FILE="$SNAPCRAFT_PART_BUILD/3rdparty/wheels/azure_iothub_device_client-1.4.6-py3-none-linux_armv7l.whl" ;;
        *x86_64*)
            WHEEL_FILE="$SNAPCRAFT_PART_BUILD/3rdparty/wheels/azure_iothub_device_client-1.4.6-py3-none-manylinux1_x86_64.whl" ;;
        *)
            echo "Unsupported architecture!"
            exit 1 ;;
    esac

    snapcraftctl stage
    PYTHONUSERBASE=$SNAPCRAFT_STAGE python3.7 -m pip install --user -I $WHEEL_FILE

Hi @zoopp
Since I see arm there, are you trying to cross compile?
looking at your snapcraft yaml, let’s first focus on python37 part,when I tested this, it does not cross compile, and seems like this is at the moment broken https://stackoverflow.com/questions/56178633/trouble-cross-compiling-python
Do you really need python3.7, would 3.5 do, or if you use base: core18, would 3.6 do?
If you can work with python 3.5/3.6, then check my modifications to your snapcraft:

base: core18
parts:
    python3:
        plugin: nil
        stage-packages:
            - to armhf:
                - libbz2-1.0:armhf
                - libexpat1:armhf
                - libffi6:armhf
                - libgdbm3:armhf
                - liblzma5:armhf
                - libncurses5:armhf
                - libreadline6:armhf
                - libsqlite3-0:armhf
                - libssl1.0.0:armhf
                - libzip4:armhf
                - uuid-runtime:armhf
                - python3:armhf
            - else:
                - libbz2-1.0
                - libexpat1
                - libffi6
                - libgdbm3
                - liblzma5
                - libncurses5
                - libreadline6
                - libsqlite3-0
                - libssl1.0.0
                - libzip4
                - uuid-runtime
                - python3
            - python3-pip
        stage:
            - -usr/lib/python3*/test
            - -var

    my-app:
        plugin: nil
        source:
          - to armhf: https://files.pythonhosted.org/packages/f0/29/e0f04410c27e0e0e67d226373406c618b364c5a61e406cd08cddd9ba2c3d/azure_iothub_device_client-1.4.6-py3-none-linux_armv7l.whl
          - to amd64: https://files.pythonhosted.org/packages/a3/77/cc635440dc06eebbd4363a3e50c3ef1b558c344e20059263868752e7979d/azure_iothub_device_client-1.4.6-py3-none-manylinux1_x86_64.whl
        source-type: zip
        stage-packages:
            - to amd64:
                - libboost-python1.58.0:amd64
                - libcurl3:amd64
                - libpython3.5:amd64
            - to armhf:
                - libboost-python1.58.0:armhf
                - libcurl3:armhf
                - libpython3.5:armhf
            - else:
                - libboost-python1.58.0
                - libcurl3
                - libpython3.5
        stage:
            - -usr/bin/2to3
            - -usr/bin/pydoc3
            - -usr/bin/python3
        after:
            - python3
        override-build: |
           case "$SNAPCRAFT_ARCH_TRIPLET" in
             *arm*)
               WHEEL_FILE=https://files.pythonhosted.org/packages/f0/29/e0f04410c27e0e0e67d226373406c618b364c5a61e406cd08cddd9ba2c3d/azure_iothub_device_client-1.4.6-py3-none-linux_armv7l.whl
               ;;
             *x86_64*)
               WHEEL_FILE=https://files.pythonhosted.org/packages/a3/77/cc635440dc06eebbd4363a3e50c3ef1b558c344e20059263868752e7979d/azure_iothub_device_client-1.4.6-py3-none-manylinux1_x86_64.whl
               ;;
             *)
               echo "Unsupported architecture!"
               exit 1
               ;;
            esac
            PYTHONUSERBASE=$SNAPCRAFT_PART_INSTALL python3 -m pip install -I $WHEEL_FILE
1 Like