The Catkin plugin uses rosdep to determine dependencies of the workspace it’s building. However, the only type of dependency it supports right now are debs. Another type that would be useful and completely doable is pip. This is a proposal to achieve exactly that.
Really, the Catkin-specific work required is minimal. We simply need a way to hand packages to pip, which means this proposal is really discussing a reorganization of the Python plugin, where the pip functionality currently resides. The proposal is this:
Extract pip (and associated helpers) from the Python plugin, and create a new snapcraft.plugins._python package for it.
Update the Python plugin to use the extracted pip and helpers.
Update the Catkin plugin to use the extracted pip and helpers to install pip dependencies as required
I’m wondering about what I can do if I require a pip package which is not in the python rosdeps yet. I think it makes sense to make a PR with the missing package, but is there a way I can use the pip package in the meantime?
From what I was reading in another post, adding the python plugin as a part would not work well together with the catkin plugin.
Hi there, have you already tried adding the python plugin with core20? Many of such issues are related to Python 2 / 3 conflicts (as noted in this thread. Maybe is it the one you are referring to?) but with core20 the ROS plugins are using Python 3 just like the Python plugin itself.
Please let us know how it goes.
Indeed, this is the post I was referring to. Since I’m just moving over to core20, this might not be an issue anymore if it is just caused by Python 2/3 conflicts. I thought the catkin-tools and Python plugin do not work well together in general.
In fact my PR was accepted in no time, so that I can just rely on my package.xml and there’s no need anymore for me to use the Python plugin.
Do I still need to add the pip-package as stage-package somehow? I added the package to the rosdeps (python3-influxdb-client-pip) and added the depend-tag in my package.xml, but still get an error when running it:
ModuleNotFoundError: No module named 'influxdb_client'
You will need to until the ROS plugins in core20 support pip dependencies, because of this:
Indeed, the build process will work because snapcraft DOES support pip dependencies during the build. Sadly, pip dependencies do not end up IN the snap, which means pip dependencies do not work at RUN time. Does that make any sense?
I’m having trouble explaining that very clearly, so let me try walking you through the process in case it’s unclear.
When you build a snap, snapcraft created a new build VM or container to use for the build. During the build process, it runs rosdep install, which installs both deb and pip dependencies on that VM/container. However, when creating the snap all those dependencies need to be in a particular directory that is then compressed into a squashfs image, and rosdep install doesn’t get them there. Snapcraft needs to specifically support fetching and unpacking each type of dependency supported by rosdep into that destination.
So this is just core18 and not yet implemented in core20?
I think I understand. Sounds similar to stage and build packages. But what is missing the functionality? The catkin-tools plugin or snapcraft in general? And what is the point of “Catkin pip support” if the pip package is not added to the snap?
Still gives me “Oct 27 12:03:27 localhost nav.daemon[3947]: ModuleNotFoundError: No module named ‘influxdb_client’”. I added pip-dep with after to my catkin-tools part.
I have influxdb_client in ../prime/lib/python3.8/site-packages. But is this the python path catkin-tools is using as well?
I’ve created a minimal example to demonstrate the issue:
I realized that it is caused by the Python plugin’s stage: [lib64/*] parameter. But if I remove it, I get the following error:
Failed to stage: Parts 'core-dep' and 'test-pip' have the following files, but with different contents:
lib64
Can someone explain me what is going wrong there?
Edit: Can anyone help me or point me to a direction? I’m kind of blocked with my project. There really seems to be a problem with combining the python plugin with catkin in core20.
Or is there another way to bring a pip package to the right place?
Is it possible that lib64 is a symlink in one case and a directory in the other? I think that would cause that… If so, you can probably make it consistent before staging.
I realized that adding the stage keyword is causing the issue with the missing import. In case I remove the stage keyword because I don’t add the other package with the lib64 folder, it works fine.
Does this mean that if I don’t use stage, more files are staged automatically? But what would it be?
A positive entry in stage would remove everything else. So I added the lib64 symlink as a negative entry.
However, there’s now a problem with lsb_release:
File "/usr/lib/python3.8/subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('lsb_release', '-a')' returned non-zero exit status 1.
Failed to build 'workspace'.
I already added lsb-release and lsb-core to the build packages. If I execute lsb_release -a in the debug shell, it works.
