Catkin plugin & python conflicts

cratliff · June 27, 2017, 3:49pm

I have a Snapcraft file to build my project that uses catkin, one of the rosdep dependencies is only installed by pip and the catkin plugin doesn’t support pip dependencies currently. I attempted to install the pip dependency as a separate python part and received a number of file conflicts between what the catkin part python files and the python part files. I wanting to receive some comments on what the appropriate resolution to this issue might be.

From the perspective of using catkin, it would be best for Snapcraft to support pip dependencies. Modifying package dependencies and moving them to the snapcraft.yaml file wouldn’t be necessary then.

Would it also be good to attempt to resolve the inconsistencies between the python and catkin plugins to avoid using both parts requiring manual resolution of the conflicts?

@kyrofa

kyrofa · June 27, 2017, 4:15pm

Indeed, this is LP: #1683036. Definitely a feature that we need. I’ll try to get it into the next release.

Yes. The issue here is that the python plugin changes the python it places into the snap in a few ways that catkin does not, but this is something that should probably be done simply when the python packages are staged, not by the plugin itself.

cratliff · July 10, 2017, 7:03pm

Here is an example of what I am seeing. I have a module installed with pip and a ROS node using catkin. The files excluded from stage are what conflicts occur with the two plugins. When they are excluded using stage the snap builds, but the ds4drv module installed using pip will not run. The following error occurs:

Traceback (most recent call last):
  File "/snap/bar-base/x32/bin/ds4drv", line 7, in <module>
    from ds4drv.__main__ import main
  File "/snap/bar-base/x32/lib/python3.5/site-packages/ds4drv/__main__.py", line 5, in <module>
    from .actions import ActionRegistry
  File "/snap/bar-base/x32/lib/python3.5/site-packages/ds4drv/actions/__init__.py", line 1, in <module>
    from ..action import ActionRegistry
  File "/snap/bar-base/x32/lib/python3.5/site-packages/ds4drv/action.py", line 1, in <module>
    from .config import add_controller_option
  File "/snap/bar-base/x32/lib/python3.5/site-packages/ds4drv/config.py", line 15, in <module>
    from .uinput import parse_uinput_mapping
  File "/snap/bar-base/x32/lib/python3.5/site-packages/ds4drv/uinput.py", line 6, in <module>
    from evdev import UInput, UInputError, ecodes
  File "/snap/bar-base/x32/lib/python3.5/site-packages/evdev/__init__.py", line 7, in <module>
    from evdev.device import DeviceInfo, InputDevice, AbsInfo
  File "/snap/bar-base/x32/lib/python3.5/site-packages/evdev/device.py", line 10, in <module>
    from evdev.eventio_async import EventIO
  File "/snap/bar-base/x32/lib/python3.5/site-packages/evdev/eventio_async.py", line 3, in <module>
    import asyncio
  File "/snap/bar-base/x32/usr/lib/python3.5/asyncio/__init__.py", line 21, in <module>
    from .base_events import *
  File "/snap/bar-base/x32/usr/lib/python3.5/asyncio/base_events.py", line 18, in <module>
    import concurrent.futures
  File "/snap/bar-base/x32/usr/lib/python2.7/dist-packages/concurrent/futures/__init__.py", line 8, in <module>
    from concurrent.futures._base import (FIRST_COMPLETED,
  File "/snap/bar-base/x32/usr/lib/python2.7/dist-packages/concurrent/futures/_base.py", line 357
    raise type(self._exception), self._exception, self._traceback

The following is the snapcraft.yaml that produces this.

name: robot-base
version: '0.1'
summary: base snap
description: |
  This snap includes all of the ROS modules and other code to run the base of the
  robot.

grade: devel
confinement: devmode
parts:
  bar-pip:
    plugin: python
    python-packages: [ds4drv]
    stage-packages: [
        bluez,
    ]

  bar-base:
    plugin: catkin
    rosdistro: kinetic
    rosinstall-files: [ros_navigation.rosinstall]

    stage-packages: [
      fluid,
      python-boto3,
      python-requests,
      python-rosdep,
      python-ipdb,
      chrony,
      ros-kinetic-pcl-ros,
      ros-kinetic-desktop-full,
    ]
    catkin-packages: [
      amcl,
    ]
    stage:
      - -usr/bin/2to3-3.5
      - -usr/bin/py3clean
      - -usr/bin/py3compile
      - -usr/bin/pydoc3.5
      - -usr/lib/python3.5/base64.py
      - -usr/lib/python3.5/cProfile.py
      - -usr/lib/python3.5/cgi.py
      - -usr/lib/python3.5/encodings/rot_13.py
      - -usr/lib/python3.5/idlelib/PyShell.py
      - -usr/lib/python3.5/keyword.py
      - -usr/lib/python3.5/lib2to3/pgen2/token.py
      - -usr/lib/python3.5/pdb.py
      - -usr/lib/python3.5/platform.py
      - -usr/lib/python3.5/profile.py
      - -usr/lib/python3.5/pydoc.py
      - -usr/lib/python3.5/quopri.py
      - -usr/lib/python3.5/sitecustomize.py
      - -usr/lib/python3.5/smtpd.py
      - -usr/lib/python3.5/smtplib.py
      - -usr/lib/python3.5/symbol.py
      - -usr/lib/python3.5/tabnanny.py
      - -usr/lib/python3.5/tarfile.py
      - -usr/lib/python3.5/test/pystone.py
      - -usr/lib/python3.5/test/regrtest.py
      - -usr/lib/python3.5/timeit.py
      - -usr/lib/python3.5/trace.py
      - -usr/lib/python3.5/uu.py
      - -usr/lib/python3.5/webbrowser.py
      - -usr/share/apport/package-hooks/source_bluez.py
      - -usr/share/dh-python/dh_pypy
      - -usr/share/dh-python/dh_python2
      - -usr/share/dh-python/dh_python3
      - -usr/share/dh-python/dhpython/_defaults.py
      - -usr/share/dh-python/pybuild
      - -usr/share/python3/py3versions.py

apps:
  bash:
    command: bash

  ds4drv:
    command: ds4drv

Here is the .rosinstall file referenced:

- git:
    local-name: ros_navigation
    uri: https://github.com/ros-planning/navigation.git
    version: kinetic-devel

kyrofa · July 10, 2017, 11:07pm

@cratliff I’m afraid that example isn’t quite reproducible. I need the amcl project in my local tree in order to build, and I don’t. Is that available anywhere? I’d like an example I can actually run and poke at so I can figure out what’s happening here.

(also, your build-packages require the ROS repository to be configured on the host which isn’t ideal, but we can talk about that once we get you unblocked)

cratliff · July 12, 2017, 1:30pm

I updated the file in my original post. It should be reproducible now.

Yeah, occasionally I’ll do a quick edit to a Python script and it is quicker to leverage ROS being networked than integrating it into the snap immediately. I stripped them from the file above, but some of the other packages in the ROS repo take a very long time to build. I’ve been setting up ROS on my local machine to test small changes without repackaging. I’m open to suggestions though. It would be possible to have the node I’m working on be a different snap to not have to setup ROS on my local machine, but in prod having to keep multiple snaps on the same update together would probably be a headache worth avoiding.

kyrofa · July 13, 2017, 4:35pm

Ah, you know what you’re doing! No need to discuss it, then .

Alright, I’m able to reproduce your issue. While I continue poking at it, my first question is: is ds4drv supposed to be using python2 or 3? Because by default the python plugin uses 3. But ROS’s python bits are python2. I assume if this is supposed to be resolved by rosdep, it’s python2.

kyrofa · July 13, 2017, 4:58pm

Yep, that seems to be it. Explains the syntax error as well. The two changes necessary are to specify python2 as the python version for bar-pip, and to change the stage filter accordingly. Here’s the YAML that works for me:

name: robot-base
version: '0.1'
summary: base snap
description: |
  This snap includes all of the ROS modules and other code to run the base of the
  robot.

grade: devel
confinement: devmode
parts:
  bar-pip:
    plugin: python
    python-version: python2
    python-packages: [ds4drv]
    stage-packages: [
        bluez,
    ]

  bar-base:
    plugin: catkin
    rosdistro: kinetic
    rosinstall-files: [ros_navigation.rosinstall]

    stage-packages: [
      fluid,
      python-boto3,
      python-requests,
      python-rosdep,
      python-ipdb,
      chrony,
      ros-kinetic-pcl-ros,
      ros-kinetic-desktop-full,
    ]
    catkin-packages: [
      amcl,
    ]
    stage:
      - -usr/bin/2to3-2.7
      - -usr/bin/pydoc2.7
      - -usr/lib/python2.7/UserString.py
      - -usr/lib/python2.7/base64.py
      - -usr/lib/python2.7/cProfile.py
      - -usr/lib/python2.7/cgi.py
      - -usr/lib/python2.7/encodings/rot_13.py
      - -usr/lib/python2.7/ensurepip/__init__.py
      - -usr/lib/python2.7/keyword.py
      - -usr/lib/python2.7/lib2to3/pgen2/token.py
      - -usr/lib/python2.7/mimify.py
      - -usr/lib/python2.7/pdb.py
      - -usr/lib/python2.7/platform.py
      - -usr/lib/python2.7/profile.py
      - -usr/lib/python2.7/pydoc.py
      - -usr/lib/python2.7/quopri.py
      - -usr/lib/python2.7/sitecustomize.py
      - -usr/lib/python2.7/smtpd.py
      - -usr/lib/python2.7/smtplib.py
      - -usr/lib/python2.7/symbol.py
      - -usr/lib/python2.7/tabnanny.py
      - -usr/lib/python2.7/test/pystone.py
      - -usr/lib/python2.7/test/regrtest.py
      - -usr/lib/python2.7/timeit.py
      - -usr/lib/python2.7/trace.py
      - -usr/lib/python2.7/uu.py
      - -usr/lib/python2.7/webbrowser.py
      - -usr/share/python/dh_python2
      - -usr/share/python/pyversions.py

apps:
  bash:
    command: bash

  ds4drv:
    command: ds4drv

cratliff · July 13, 2017, 5:05pm

I’m a little surprised since ds4drv can use 2.7 or 3.3+. I’m not currently, but if I were to need a python part with python3 as the version would I run into the same problems? Thanks for the help.

kyrofa · July 13, 2017, 5:13pm

You’ve actually ran into a bit of a larger problem, LP: #1677002. It’s not specific to Catkin, although it’s what caused me to log it.

Basically, the issue is that both python2 and python3 use the variable $PYTHONPATH to determine where it searches for modules, etc. Which means you can only ever support EITHER python2 OR python3 using that variable. The Catkin plugin defines that path to be python2-specific. As soon as you throw python3 stuff in there, the $PYTHONPATH doesn’t work correctly anymore.

Note that you can work around this if you required a python3 app by writing a wrapper that re-defines that variable to be python3-specific. The environment keyword might work here, as well.

kyrofa · October 3, 2017, 5:50pm

Note that the final PR introducing pip support in the catkin plugin is here:

It was a bit of a longer road than I anticipated, I apologize for the delay.