Proposal: Catkin pip support

The Catkin plugin uses rosdep to determine dependencies of the workspace it’s building. However, the only type of dependency it supports right now are debs. Another type that would be useful and completely doable is pip. This is a proposal to achieve exactly that.

Really, the Catkin-specific work required is minimal. We simply need a way to hand packages to pip, which means this proposal is really discussing a reorganization of the Python plugin, where the pip functionality currently resides. The proposal is this:

  1. Extract pip (and associated helpers) from the Python plugin, and create a new snapcraft.plugins._python package for it.
  2. Update the Python plugin to use the extracted pip and helpers.
  3. Update the Catkin plugin to use the extracted pip and helpers to install pip dependencies as required
1 Like

Will snapcraft.plugins._python be a package or module?

A package, my apologies for not being clear. Similar to how we have _ros now.

It was clear, I was double checking :wink:

The final piece of this is here:

I’m wondering about what I can do if I require a pip package which is not in the python rosdeps yet. I think it makes sense to make a PR with the missing package, but is there a way I can use the pip package in the meantime?

From what I was reading in another post, adding the python plugin as a part would not work well together with the catkin plugin.

Hi there, have you already tried adding the python plugin with core20? Many of such issues are related to Python 2 / 3 conflicts (as noted in this thread. Maybe is it the one you are referring to?) but with core20 the ROS plugins are using Python 3 just like the Python plugin itself.
Please let us know how it goes.


FYI, you’ll have to do this if you ever want to depend upon that package in the package.xml. You probably know that but I wanted to make sure.

1 Like

Indeed, this is the post I was referring to. Since I’m just moving over to core20, this might not be an issue anymore if it is just caused by Python 2/3 conflicts. I thought the catkin-tools and Python plugin do not work well together in general.

In fact my PR was accepted in no time, so that I can just rely on my package.xml and there’s no need anymore for me to use the Python plugin.

Do I still need to add the pip-package as stage-package somehow? I added the package to the rosdeps (python3-influxdb-client-pip) and added the depend-tag in my package.xml, but still get an error when running it:

ModuleNotFoundError: No module named 'influxdb_client'

But during the build, I see:

executing command [pip3 install -U influxdb-client]
Collecting influxdb-client
  Downloading influxdb_client-1.11.0-py3-none-any.whl (478 kB)
     |████████████████████████████████| 478 kB 4.5 MB/s 

You will need to until the ROS plugins in core20 support pip dependencies, because of this:

Indeed, the build process will work because snapcraft DOES support pip dependencies during the build. Sadly, pip dependencies do not end up IN the snap, which means pip dependencies do not work at RUN time. Does that make any sense?

I’m having trouble explaining that very clearly, so let me try walking you through the process in case it’s unclear.

When you build a snap, snapcraft created a new build VM or container to use for the build. During the build process, it runs rosdep install, which installs both deb and pip dependencies on that VM/container. However, when creating the snap all those dependencies need to be in a particular directory that is then compressed into a squashfs image, and rosdep install doesn’t get them there. Snapcraft needs to specifically support fetching and unpacking each type of dependency supported by rosdep into that destination.

So this is just core18 and not yet implemented in core20?

I think I understand. Sounds similar to stage and build packages. But what is missing the functionality? The catkin-tools plugin or snapcraft in general? And what is the point of “Catkin pip support” if the pip package is not added to the snap?

Unfortunately, no success:

  plugin: python
    - influxdb-client
    - lib64/*

Still gives me “Oct 27 12:03:27 localhost nav.daemon[3947]: ModuleNotFoundError: No module named ‘influxdb_client’”. I added pip-dep with after to my catkin-tools part.

I have influxdb_client in ../prime/lib/python3.8/site-packages. But is this the python path catkin-tools is using as well?

I’ve created a minimal example to demonstrate the issue:

I realized that it is caused by the Python plugin’s stage: [lib64/*] parameter. But if I remove it, I get the following error:

Failed to stage: Parts 'core-dep' and 'test-pip' have the following files, but with different contents:

Can someone explain me what is going wrong there?

Edit: Can anyone help me or point me to a direction? I’m kind of blocked with my project. There really seems to be a problem with combining the python plugin with catkin in core20.

Or is there another way to bring a pip package to the right place?

Is it possible that lib64 is a symlink in one case and a directory in the other? I think that would cause that… If so, you can probably make it consistent before staging.

1 Like

Yes, this is the case for pip-dep:

../parts/pip-dep/install/lib64 -> lib

This is also the folder, which contains the missing dependency during runtime.

I tried what you proposed in this post. The minus didn’t work, so I was trying to stage the content of the folder:

  plugin: python
    - influxdb-client
    - lib64/*

In this case, the build works, but during runtime, I’m getting:

Oct 27 12:03:27 localhost nav.daemon[3947]: ModuleNotFoundError: No module named ‘influxdb_client

Am I doing something wrong with staging the content?

You may need to stage bin as well so it picks up the bin/python* symlinks.

Otherwise you could configure PYTHONPATH to find the libs.

This still didn’t work. The module is still missing during runtime.

  plugin: python
    - influxdb-client
    - lib64/*
    - bin

I realized that adding the stage keyword is causing the issue with the missing import. In case I remove the stage keyword because I don’t add the other package with the lib64 folder, it works fine.

Does this mean that if I don’t use stage, more files are staged automatically? But what would it be?

The following files are missing in the stage folder if I add the stage keyword:

  • include
  • pyvenv.cfg
  • ciso-requirements.txt
  • requirements.txt
  • extra-requirements.txt
  • lib64 -> lib (the symlink)
  • share
  • test-requirements.txt

The question is, how can I avoid the lib64 symlink without changing the stage-part so that I get all the files I usually get?


I think I got my answer over there: Error "while loading shared libraries" with BloodHound snap

A positive entry in stage would remove everything else. So I added the lib64 symlink as a negative entry.

However, there’s now a problem with lsb_release:

  File "/usr/lib/python3.8/", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('lsb_release', '-a')' returned non-zero exit status 1.
Failed to build 'workspace'.

I already added lsb-release and lsb-core to the build packages. If I execute lsb_release -a in the debug shell, it works.

Still trying to fix this…

I tried the following in addition to adding lsb-core and lsb-release as build-packages:

  • Adding a symlink:
    override-build: |
      mkdir -p /usr/lib/python3.8/site-packages
      ln -sfv /usr/share/pyshared/ /usr/lib/python3.8/site-packages/
      snapcraftctl build
  • Adding another symlink:
    override-build: |
      mkdir -p /usr/local/lib/python3.8/site-packages/
      ln -sfv /usr/share/pyshared/ /usr/local/lib/python3.8/site-packages/
      snapcraftctl build
  • Specifying the python version:
    override-build: |
      sed -i 's|#!/usr/bin/python3 -Es|#!/usr/bin/python3.8 -Es|g' /usr/bin/lsb_release
      snapcraftctl build
  • Deleting the lsb_release file:
    override-build: |
      rm /usr/bin/lsb_release
      snapcraftctl build

Still no solution in sight.