rosdep is used in the catkin plugins to determine dependencies for ROS systems and used in both of the catkin plugins. Right now snapcraft uses the default rodep database, I was thinking it would be useful for snapcraft to be able to add custom rosdep sources. Rosdep supports being able to host your own rosdep repo or use other repos, but this isn’t in snapcraft. The reason being that if a user does use a custom rosdep database that has dependencies not listed in or different from the default repo the snap will not build the same when they switch to the default repo.
An instance of this is python-networkmanager. The default rosdep database specifies an apt package that is out of date from the pip package. The pip package has necessary bugfixes for our application. We are unable to specify this package in our rosdep as it will pull in the apt version instead of the version from pip. This adds complication to our build process.
Having a custom rosdep is also highly desirable because while rosdep is useful, there is the potential that package specifications are updated without your knowledge. This can break snaps when packages switch versions, from pip to apt, or remove dependencies.
Even though we’d only be interested in a single rosdep repository being added, I think it would make sense to keep the ability to add any number at different priorities to match rosdep behavior.
@cratliff the only time I’ve used my rosdep database fork is when testing changes that I shortly proposed back to the database, which was accepted within a few hours. I’ve never run long-term from one. In your experience, is that something people often do?
I’m not sure how often running a rosdep fork is done, I suspect it isn’t done too frequently. I’ve made similar updates to the database and they are usually pretty short, but sometimes it has been a few days.
I wouldn’t be surprised if many of the people who use the ROS Yocto layer fork the rosdep db though. If you’re goal is reliable, reproducible builds having the dependency structure changed by a third party at unspecified times is not compatible.
In the case I mentioned in my first post python-networkmanager from universe is v0.9.10 on pypy it is version 2.1. If this change were merged into the rosdep db, others who are using python-networkmanager from rosdep would have breaking changes picked up in their build without notification. In this case, if the update were to happen I would be fine. But I am concerned about being on the other end of this type of update and having a rosdep fork would ensure this doesn’t happen.
@cratliff and I chatted about this a bit on IRC. It doesn’t seem common to either of us, and seems to be an issue inherent in the rosdep design. Before adding support for this upstream, I’d like to see if anyone else does this, or if there are other (potentially better) ways of dealing with this issue. @cratliff said he’d check around (thank you!).
I have al alternate use-case for custom rosdep sources. In my group’s current workflow, we are generating debian files for some of our “custom” dependencies and hosting them privately on-premises. In order to use them in our CI/CD pipelines, we need to add both apt and rosdep sources.
I’ve seen examples of adding apt sources for snapcraft builds (although I haven’t tried it yet), but there doesn’t seem to be support for adding rosdep sources anywhere. I was going to add them manually using custom steps in my snapcraft.yaml, but a quick glance at the source code suggest anything I do will be wiped out right before updating rosdep.
Am I missing something here?
Otherwise, considering one of the selling points of snap as a packaging tool w/ respect to the ROS buildfarm is that it better supports private developments, I think this feature should be reconsidered.
I appreciate your input, @miguelprada. The idea of using rosdep for internal dependencies was not one I had considered, and makes a good case for a long-running fork. You’re right, manually twiddling things won’t work the way you hope. Let me mull this over a bit, see if we can come up with a decent way of supporting this use-case. A good first step might be similar to how apt sources are added (at the plugin level).
The solution we are using internally is to add a rosdep-sources keyword. When this is specified and it is a directory, the contents are copied into _rosdep_sources_path instead of calling rosdep init. This has been working well for us since you can still use the default by leaving it in the directory given or only use your custom ones.
This solution currently requires the sources to be in the same repo as snapcraft.yaml, which hasn’t been a problem for us, but I could see wanting to use something like the source keyword where you might want to be able to pull the sources from git/url/local directory without issue.
I forget, are rosdep packages only pulled in on stage packages or both stage and build packages? If this is being done for internal packages would you have an issue with adding a PPA for build packages?
To be honest, I’m just starting to play with snap(craft) and I’m not yet familiar with the different stages. Most (if not all) of the additional rosdeps we defined internally are hosted in a privately hosted server as well, and I would need to add that server to the apt sources for rosdep to properly resolve these dependencies. Not sure if this responds to your question.
