ROS snap Package 'python-dbus' isn't a valid system dependency

snapcraft
1

I’ve been reading all information I have found about this issue an a this point I’m kinda lost. I’m toying with snapcraft as part of some technology scouting. As an exercise I’m trying to snap a ROS application which has some dependencies on pip packages and other libraries.

I’m using core as a base (the application is based on ros kinetic) and since xenial has entered EOL that meant that I had to change to an older snapcraft version in order to be able to use such base (porting from kinectic to some other version is out of the question at this moment).

Luckily…

snap install snapcraft --channel 4.x

Did the trick. After this I tried to build the snap using a quite simple snapcraft.yaml

name: my-awesome-robot-snap
base: core 
version: '0.1' 
summary: A snap containing a ros app
description: |
  This is my-snap's description.

grade: devel # must be 'stable' to release into candidate/stable channels
confinement: devmode # use 'strict' once you have the right plugs and slots

parts:
  robot-workspace:
    plugin: catkin
    source: .

apps:
  system:
    command: roslaunch robot-app main.launch
    plugs: [network, network-bind]
    daemon: simple

Unfortunately when I try to create the snap I get the aforementioned error.

Initializing rosdep database...
Updating rosdep database...
Warning: running 'rosdep update' as root is not recommended.
  You should run 'sudo rosdep fix-permissions' and invoke 'rosdep update' again without sudo.
Determining system dependencies for Catkin packages...
ERROR: no rosdep rule for 'python-dbus'
Package 'python-dbus' isn't a valid system dependency. Did you forget to add it to catkin-packages? If not, add the Ubuntu package containing it to stage-packages until you can get it into the rosdep database.

Here is the first thing I find puzzling. The project has several python and non python dependencies. If I try to run again snapcraft, the error will re-appear but mention a different library even if I make no changes to the snapcraft.yaml. Sometimes it will always complain about the same X times in a row. But looks like the “source” library that triggers the error is random.

After googling a little found some similar errors being discussed but all of them referred to a missing functionality on the catkin plugin for pip that was fixed in catkin plugin: support rosdep pip dependencies by kyrofa · Pull Request #1581 · canonical/snapcraft · GitHub

I’ve checked the PR and it should be inside of 4.8 according to github.

Anyway, I went on and tried some of the alternatives I found in old threads:

  • putting apt based libraries inside build-packages:
  • create a pip-pkg part and add the python libraries there
  • force different build orders (pip-pkg before robot-workspace)

But nothing worked.

Next thing I tried was to check if rosdep, can actually resolve a couple of the failing dependencies. I fired a docker image based on xenial+kinetic and run

❯ rosdep resolve python-dbus --rosdistro kinetic
#apt
python-dbus

❯ rosdep resolve python-boto3-pip --rosdistro kinetic
#pip
boto3

Seeing that this was somewhat possible, I kept digging and realize that you can access the snapcraft VM right after the error with the --debug flag. Here comes my second puzzling moment, I tried the same and…

Initializing rosdep database...
Updating rosdep database...
Warning: running 'rosdep update' as root is not recommended.
  You should run 'sudo rosdep fix-permissions' and invoke 'rosdep update' again without sudo.
Determining system dependencies for Catkin packages...
ERROR: no rosdep rule for 'python-dbus'
Package 'python-dbus' isn't a valid system dependency. Did you forget to add it to catkin-packages? If not, add the Ubuntu package containing it to stage-packages until you can get it into the rosdep database.

snapcraft-my-awesome-robot-snap # rosdep resolve python-dbus --rosdistro kinetic
bash: rosdep: command not found

Anybody could explain me why is not rosdep available in the VM?

Anyway, thinking that it might be something that the catkin-plugin is using, and then disposing of, I tried to manually install rosdep within the VM

snapcraft-my-awesome-robot-snap # apt-cache search rosdep

Yielded python-rosdep as available. I installed it, went to /root/project, and then…

snapcraft-my-awesome-robot-snap ../project# sudo rosdep init
snapcraft-my-awesome-robot-snap ../project# rosdep update

snapcraft-my-awesome-robot-snap ../project# rosdep resolve python-dbus --rosdistro kinetic
WARNING: ROS_PYTHON_VERSION is unset. Defaulting to 2
ERROR: no rosdep rule for 'python-dbus'

snapcraft-my-awesome-baktus-snap ../project# rosdep resolve python-boto3-pip --rosdistro kinetic
WARNING: ROS_PYTHON_VERSION is unset. Defaulting to 2
ERROR: no rosdep rule for 'python-boto3-pip'

So it looks like there is definitely something fishy. At this point I’m out of ideas about how to fix this. Any suggestions will be really appreciated. Thanks in advance!

2

Hi @jagomo,

Looking at rosdistro I can’t seem to find a python-dbus entry; however there is a python-pydbus (introduced in this PR). Could that be it? But then I’m not sure how nor why rosdep could resolve it (as apt) in your Docker image.

Note also that python-boto3-pip is actually python-boto3.

3

Hi @artivis, first of all thank you for taking the time to reply!

I’m not really familiar with the internals of rosdep, but your comment gave me a VERY good hint of where to keep digging!

I checked the docker file and there I only saw that boto3 is installed with

RUN pip install --no-cache-dir \
  boto3\
  ...

So in principle nothing very suspicious. boto3 should be in the image, but I saw still no connection with rosdep behavior. Nevertheless, the docker image is created with some internal script (I’m still getting familiar with the codebase and tools we are using). And there I saw reference to a local rosdep.yaml! And bingo, all the dependencies that throw errors on snapcratf are defined there. For example:

python-boto3-pip:
  ubuntu:
    pip:
      packages: [boto3]
python-dbus:
  ubuntu:
    '*': [python-dbus]

From what I found here: https://answers.ros.org/question/207771/custom-rosdepyaml-in-my-package/

The usage of custom rosdep.yaml is not supported anymore (probably back on kinetic was). So my guess is that the snapcraft catkin plugin does not support this use case and ignores the local rosdep.yaml

Any idea about how to workaround this? The codebase is quite big, and many package.xml reference this libraries. So I guess that even if I install everything manually, the snap will still fail because rosdep won’t be able to resolve the dependencies without the local rosdep.yaml

4

If you are using custom rosdep sources it all make sense. And you are correct saying that installing these libraries ‘manually’ will not help since rosdep will still try (and fail) to resolve these entries.

Any idea about how to workaround this?

First of all, note that custom rosdep sources is not supported out-of-the-box in snapcraft at the moment. Furthermore, according to tfoote answer to the question you linked, it is still possible to use custom rosdep sources (see e.g. rosdep doc). Custom sources are now simply system-wide rather than per-packages.

This being said, I’d first encourage you to make sure that you do need custom sources. It may be a little tedious to fix your code base but at least the two dependencies that we are discussing are supported upstream and you may find several other benefits to going back to the ‘main’ rosdep sources. If, after evaluation, you are certain that you need custom rosdep sources, this may still be doable since snapcraft uses plain rosdep command line internally (emphasis since I haven’t tried myself). It’s likely a little tricky depending on how you build you snap (multipass, lxd, etc.) but the solution from the aforementioned rosdep doc should work in snapcraft too.

5

First of all, note that custom rosdep sources is not supported out-of-the-box in snapcraft at the moment. Furthermore, according to tfoote answer to the question you linked, it is still possible to use custom rosdep sources (see e.g. rosdep doc). Custom sources are now simply system-wide rather than per-packages.

You are absolutely right. I completely misunderstood that.

One of my colleagues pointed me to the same documentation you mentioned, and elaborated on how we manage that in the docker container. As could be expected, we follow the exact same method in the docker image. Currently I’m trying to replicate it by generate a file in

/root/parts/robot-workspace/rosdep/sources.list.d/

And make it point to the custom sources. I modified my snapcraft.yaml like this to be able to fiddle with the pull stage

parts:
  robot-workspace:
    # See 'snapcraft plugins'
    plugin: catkin
    source: .
    override-pull: |
      /root/project/src/inject-rosdep-yaml-snap.sh
      snapcraftctl pull

As the rosdep evaluation happens in the pull stage as far as I know. The content of the script is quite simple:

#!/bin/sh
echo 'yaml file:///root/parts/robot-workspace/src/src/robot-app/rosdep.yaml' > /root/parts/robot-workspace/rosdep/sources.list.d/10-rosdep.yaml.list
ls /root/parts/robot-workspace/rosdep/sources.list.d

And so far produces the desired results

Pulling robot-workspace 
+ /root/project/src/inject-rosdep-yaml-snap.sh
10-rosdep.yaml.list  20-default.list
+ snapcraftctl pull
Installing catkin...

Nevertheless looks like the plugin reverts the directory to its original state every time. After the failure ls will return this

snapcraft-my-awesome-robot-snap ../project# ls /root/parts/robot-workspace/rosdep/sources.list.d/
20-default.list

This being said, I’d first encourage you to make sure that you do need custom sources.

Well at some point we need to fix it, that is for sure. But I wanted to avoid it for the moment as it could take me a while and this was more of a PoC about how much effort would be to do the packing. Also, when the problem was somewhat clear, I wanted a go at it, just for fun :slight_smile:

But looks like this a dead end for the moment (since I can’t modify the sources without them being overwritten again before rosdep runs). So I might have a go at fixing the codebase.

Thank you very much for your time and suggestions, I’ll keep trying to get it done and document here the results :slight_smile:

6

Hi @jagomo,

I believe your inject-rosdep-yaml-snap.sh script should look like:

mkdir -p /etc/ros/rosdep/sources.list.d/
echo 'yaml file:///root/parts/robot-workspace/src/src/robot-app/rosdep.yaml' > /etc/ros/rosdep/sources.list.d/10-rosdep.yaml.list
ls /etc/ros/rosdep/sources.list.d/

Note that the rosdep sources path is changed and the destination directory is created by the script since it is called before rosdep initialization. Please let me know if this gets you any further.

> As the rosdep evaluation happens in the pull stage as far as I know.

rosdep is actually init during the build phase, so you could override-build rather than override-pull, but that wouldn’t change anything really.

Edit: Nevermind my comment above, you are using Core and this applies to Core20.

7

Just to document the issue further. Looks like I was right about the directory being wiped out and re-created. This happens when the catkin plugin configures rosdep. The exact point is this one for 4.8 version of snapcraft.

The call to rosdep.setup() is happening in the pull step of the plugin and, after the sources directory is wiped and recreated, rosdep init and rodep update are called right away. Thus, it is impossible to modify the sources directory content before rosdep starts doing its magic.