ROS2 package fails staging "no rosdep rule for pkg"

spacegardener · June 18, 2021, 7:39pm

I am working on making a snap of my ROS2 project just like this and currently it appears that my ROS packages build but after this staging fails. Apologies for the paste dump but the output looks like this:

updated cache in /root/.ros/rosdep/sources.cache
+ rosdep install --default-yes --ignore-packages-from-source --from-paths /root/parts/ros-garden/src
#All required rosdeps installed successfully
+ colcon build --base-paths /root/parts/ros-garden/src --build-base /root/parts/ros-garden/build --merge-install --install-base /root/parts/ros-garden/install --parallel-workers 2
Starting >>> garden_interfaces
Finished <<< garden_interfaces [2.67s]                	 
Starting >>> garden_pkg
Starting >>> py_serial
Finished <<< py_serial [0.94s]                                     	 
Finished <<< garden_pkg [0.97s]
Starting >>> driver_pkg
Finished <<< driver_pkg [0.65s]     	 
Starting >>> garden_bringup
Finished <<< garden_bringup [0.54s]             	 

Summary: 5 packages finished [5.00s]
+ env -i LANG=C.UTF-8 LC_ALL=C.UTF-8 PATH=/snap/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin SNAP=/snap/snapcraft/6568 SNAP_ARCH=amd64 SNAP_NAME=snapcraft SNAP_VERSION=4.8.2 /snap/snapcraft/6568/usr/bin/python3 -I /snap/snapcraft/6568/lib/python3.6/site-packages/snapcraft/plugins/v2/_ros.py stage-runtime-dependencies --part-src /root/parts/ros-garden/src --part-install /root/parts/ros-garden/install --ros-distro foxy --target-arch amd64
Staging runtime dependencies...
Running ['rosdep', 'resolve', 'garden_pkg', '--rosdistro', 'foxy']
failed to run ['rosdep', 'resolve', 'garden_pkg', '--rosdistro', 'foxy']: b"ERROR: no rosdep rule for 'garden_pkg'\n"
Traceback (most recent call last):
  File "/snap/snapcraft/6568/lib/python3.6/site-packages/snapcraft/plugins/v2/_ros.py", line 198, in <module>
	plugin_cli()
  File "/snap/snapcraft/6568/lib/python3.6/site-packages/click/core.py", line 829, in __call__
	return self.main(*args, **kwargs)
  File "/snap/snapcraft/6568/lib/python3.6/site-packages/click/core.py", line 782, in main
	rv = self.invoke(ctx)
  File "/snap/snapcraft/6568/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
	return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/snap/snapcraft/6568/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
	return ctx.invoke(self.callback, **ctx.params)
  File "/snap/snapcraft/6568/lib/python3.6/site-packages/click/core.py", line 610, in invoke
	return callback(*args, **kwargs)
  File "/snap/snapcraft/6568/lib/python3.6/site-packages/snapcraft/plugins/v2/_ros.py", line 171, in stage_runtime_dependencies
	dep, proc.stdout.decode().strip()
UnboundLocalError: local variable 'proc' referenced before assignment
Failed to build 'ros-garden'.

the main part of interest being

Running ['rosdep', 'resolve', 'garden_pkg', '--rosdistro', 'foxy']
failed to run ['rosdep', 'resolve', 'garden_pkg', '--rosdistro', 'foxy']: b"ERROR: no rosdep rule for 'garden_pkg'\n"

i’m just not sure where to start as it everything seems to be going well up to this point and I am not super familiar with these tools. my snapcraft.yaml part looks like this

parts:
 ros-garden:
   plugin: colcon
   source: https://github.com/masynthetic/rospacegarden.git
   build-packages: [make, gcc, g++]
   stage-packages: [ros-foxy-ros2launch]

the git repo is just the entire src directory from within my ROS workspace

side note: before the ros/colcon build was working it was failing for very basic python library dependencies (serial, pickle, etc…) in my package.xml (where ros2 package dependencies are made) so I just removed these dependencies from the package.xml and it did not seem to affect the ROS functionality and the ros packages built from snapcraft so I have stuck with not having them.

ijohnson · July 2, 2021, 2:20am

I moved your post to the #snapcraft category where hopefully you will get more eyes on the issue

artivis · July 7, 2021, 7:45pm

Hi,

You are hitting a known issue where snapcraft tries to rosdep resolve all dependencies for each and every packages in your workspace. Thus if your package garden_bringup depends on garden_pkg then it tries to call rosdep resolve garden_pkg. It then fails since (I assume) the package garden_pkg is not release in rosdistro.
Luckily this issue was recently addressed (PR #3536 should be merge anytime soon) .

sergiusens · July 9, 2021, 12:11am

This can be tested from Snapcraft’s edge channel and if that works, just as an reminder, it would be part of Snapcraft 5.0 to be tagged for candidate this upcoming July 22: Snapcraft’s 21.10 Roadmap

artivis · July 12, 2021, 6:33pm

FYI, I just tested Snapcraft edge channel against a toy example and it works fine.
@spacegardener Let us know if that solve your use case.

spacegardener · July 13, 2021, 1:25am

thank you greatly for the updates, I have changed my snapcraft channel to edge and I am now on 4.8.3 but unfortunately it does not seem to be working for me. The error is identical.

can you confirm that just cloning the src directory from my ros2 workspace (combined with my part .yaml) should be ‘enough’ on that side to get the packages built? @artivis

artivis · July 13, 2021, 12:40pm

@spacegardener Are you still getting specifically the same error?
failed to run ['rosdep', 'resolve', 'garden_pkg', '--rosdistro', 'foxy']: b"ERROR: no rosdep rule for 'garden_pkg'\n".
Is this garden_pkg contained in your rospacegarden git repository (which seem to be private)? If so, then maybe you have some ROS packaging issue, possibly some incomplete/erroneous installation rules. At the point where it fails, snapcraft looks for packages in the ‘install’ folder of the workspace. If garden_pkg is not properly installed, it then falls back to calling rosdep.
Note that since you are using the colcon plugin, you can simply reproduce the build locally in order to iterate faster your debugging. Something along the lines of

colcon build --merge-install

Then calling something like

rosdep keys -i --rosdistro foxy --from-paths path/to/ws/install

Should not fail nor list garden_pkg.

spacegardener · July 13, 2021, 4:44pm

I think we are on the right track… but I still don’t quite have it. colcon build does not give me any errors running locally and everything works as expected but adding the --merge-install flag gives me an error that the install directory 'install' was created with the layout 'isolated' please remove the install directory, pick a different one, or remove the --merge install option I am failing to find much about this error. Running the second command you listed does work (after installing rosdep which I did not have).

The gap that I am seeing here is that it seems like you are saying my part should have an ‘install’ folder, correct? additionally the snapcraft log indicates it is looking in an install directory. however, my understanding is that the install folder is created by running colcon build which snapcraft seems to be doing on its own? This is why my repo is only the src directory as mentioned. Should I add my local install folder to the git repo as well? just a little hesitant as cloning more than src hasn’t gone smoothly for me before

artivis · July 13, 2021, 5:49pm

You are right about colcon automatically creating the install folder. This should not be part of your sources (hence no need to upload it to your git repository). Assuming you are following the usual ROS worskspace layout (see the catkin workspace documentation - the layout is similar for colcon), colcon will create the install folder next to the src folder which contains your ROS packages.

Concerning the --merge-install option, it shouldn’t matter much in your case. Actually, not using it may help you find ROS packaging issues in your workspace easier.
For completeness, colcon is complaining that “the install directory 'install' was created with the layout 'isolated'”, but it also tells you how to solve that: “please remove the install directory”. You can then simply delete it in order to run colcon again with the --merge-install.

I’m afraid I can’t help much more without seeing the garden_pkg source code. But your issue on the ROS answers website (which, I believe, is the same as this one) tends to confirm that you are facing a ROS issue rather than a snapcraft issue.

spacegardener · July 13, 2021, 6:52pm

Thank you again, Idk why I didn’t just try that but but removing the local install directory and running with --merge-install and I am getting a more usable warning: [0.587s] WARNING:colcon.colcon_ros.prefix_path.ament:The path '/home/mason/ros2_ws/install/garden_pkg' in the environment variable AMENT_PREFIX_PATH doesn't exist (same thing on all my packages and then same with CMAKE for a couple) again this is just on dev computer no snapcraft involved and that path does exist. i’m working my way through it using this but seems like I maybe just need to set these prefix paths to the correct place for the snap?

my guess right now would be it has something to do with all the source ing that has to happen in order to use ROS which happens in my .bashrc file that is never being touched by snapcraft. it looks like this, obvious culprit line 2. reproducing this in snap would be my best guess but not sure about how to do that yet.

source /opt/ros/foxy/setup.bash
source ~/ros2_ws/install/setup.bash
source /usr/share/colcon_argcomplete/hook/colcon-argcomplete.bash

I also added you to the repo if it helps, would be great to know how yours is set up differently.

sorry if the answer is obvious here, environment/path setup is always a bit of a pain for me and the cross between the two tools isn’t helping haha

spacegardener · July 16, 2021, 7:06am

I did not make any changes to my ROS code or really at all and the commands

colcon build --merge-install

and

rosdep keys -i --rosdistro foxy --from-paths path/to/ws/install

started working without error likely just due to reopening my terminal combined with the bashrc + source things mentioned above as I was getting various warnings from them when removing the install folder. Which I understand testing to match snapcrafts build command however I don’t believe that re-building would make any changes to the src directory? and I didn’t make any changes to it (I pushed it anyway and still getting stuck at the same error) I am not really sure what to test next.

Any suggestions or info you could share regarding environment setup for the snap?

Thanks

artivis · July 23, 2021, 9:27pm

Hi @spacegardener,

Just to let you know that I’m able to reproduce your issue locally and I’m working on a fix.

spacegardener · July 23, 2021, 10:14pm

great to hear, thanks! in the meantime I am just trying the launch file in the same pkg as its target

thom · July 30, 2021, 1:35pm

Hi, I was also affected by this issue and investigated it today. I’m trying to compile my ROS packages that have inter-dependencies using plugin colcon.

I found your recent fix @artivis , but it seems to be incomplete: https://github.com/snapcore/snapcraft/commit/034e38457f3d5d60c0009a2f5651ce010119c085. The problem I identified is that the function find_packages in
installed_pkgs = catkin_packages.find_packages(part_install).values()
actually ignores all the packages in the folder because a COLCON_IGNORE file is placed in the install folder by colcon in the build step before.

I commited my fix here https://github.com/snapcore/snapcraft/commit/6a30f04a039ab9e28cd1c5d0b483d1472aecbda4 and can confirm it resolves the issue for me, however as I delete the file COLCON_IGNORE you may want to come up with cleaner solutions. Let me know if I should open a PR.

artivis · July 30, 2021, 4:38pm

Hi @thom,

Thanks for the feedback. Indeed the fix isn’t working for ROS 2 and I arrived at the same conclusion (and solution) as you did. I’m also fixing another small issue with the plugin while at it. I’m currently wrapping up the unit testing and plan on pushing it upstream early next week.

thom · August 2, 2021, 8:33am

Glad to hear! Thanks @artivis.

artivis · August 5, 2021, 11:27pm

Hi there,

FYI the PR is opened upstream (#3566). It would be great if you could give it a shot, altho the additional tests should cover your use-case.

thom · October 8, 2021, 12:08pm

Hi @artivis,

we are now using the edge channel of snapcraft and I can confirm your fix works. Thanks a bunch!

rahul-tt · March 1, 2022, 8:38am

Hi @ijohnson ,

I still get that error. The only way to get rid of it is to clean the entire package and run in again. This takes soo long to do it every time. I just want to clean the part with the ros packages, is that possible to do? Or is there a solution to get rid of this error.

The edge channel and the stable channel of snapcraft snap contain the same revision. SO its no use switching to edge channel.

Please help!!

ijohnson · March 1, 2022, 12:45pm

Hi @rahul-tt I don’t know much about building snap packages with ROS in them, perhaps @artivis can assist you here?