Problems building using catkin plugin and underlays

Continuing the conversation from https://github.com/snapcore/snapcraft/pull/1354, in response to https://bugs.launchpad.net/snapcraft/+bug/1696014.

I’m having trouble snapping with the catkin plugin using an underlay of a prebuilt package that is not in rosdep. See example code here: https://bugs.launchpad.net/snapcraft/+bug/1696014/+attachment/4941502/+files/snapcraft_ros_example.zip

That example consists of using the dump plugin to add the already compiled package prebuilt_pkg and then using the catkin plugin with an underlay to build a package that depends on it. The output is:

$ snapcraft
"grade" property not specified: defaulting to "stable"
Preparing to pull underlay 
Pulling underlay 
'ros-project' has prerequisites that need to be staged: underlay
Preparing to build underlay 

-- snip --

Installing rosdep...
Initializing rosdep database...
Updating rosdep database...
Determining system dependencies for Catkin packages...
ERROR: no rosdep rule for 'prebuilt_pkg'
Package 'prebuilt_pkg' isn't a valid system dependency. Did you forget to add it to catkin-packages? If not, add the Ubuntu package containing it to stage-packages until you can get it into the rosdep database.

The error message is a little bit of a red herring because it is not expected for rosdep to find the package, but it only checked because the package was not found using the plugin’s _Catkin.find (which should have succeeded).

By stepping through the catkin plugin, at https://github.com/snapcore/snapcraft/blob/2.33/snapcraft/plugins/catkin.py#L954 running the command without swallowing STDERR results in:

(Pdb) n
> /usr/lib/python3/dist-packages/snapcraft/plugins/catkin.py(954)_run()
-> return subprocess.check_output(
(Pdb) subprocess.check_output(['/bin/bash', f.name, 'catkin_find'] + arguments).decode('utf8').strip()
Traceback (most recent call last):
  File "/home/ruddick/Downloads/snapcraft_ros_example/parts/ros-project/catkin/install/opt/ros/kinetic/bin/catkin_find", line 11, in <module>
    from catkin.find_in_workspaces import find_in_workspaces
ImportError: No module named catkin.find_in_workspaces
*** subprocess.CalledProcessError: Command '['/bin/bash', '/tmp/tmpzoki0751', 'catkin_find', '--first-only', 'prebuilt_pkg']' returned non-zero exit status 1

_Catkin.find interprets the non-zero return code as a failure to find the package, not an error, and happily continues. The solution I proposed in the PR was to add to the PYTHONPATH of the script so it knows where to find catkin.find_in_workspaces.

If you’re curious how I ran into this, I’m using zipped install directories as an easy way to internally distribute ROS packages (ie, not have to go through the hassle of building a debian)

Great catch, and thank you for the more detailed discussion! I was finally able to get to the bottom of this. The actual cause is outlined on the bug, but the proposed fix is here if you’re able to give it a test:

https://github.com/snapcore/snapcraft/pull/1526