I’m using the catkin plugin for a large collection of packages. At the moment if I make a small change, I have to rebuild the complete set of packages which takes about 20mins. I am wondering if I am missing an argument on the snapcraft plugin in order to only rebuild packages that have changes.
cheers
Peter
Hey @peter.milani, I’m afraid there’s no trick here. Snapcraft, as a deployment tool, just isn’t quite smart enough to know that your source changed. It’s something we’d love to add, but it’s a large feature and it just hasn’t happened yet.
Cheers, I’m working around debugging, luckily scripts, in the prime space which at least doesn’t require a full build.
Hey @kyrofa I’m in a position to start thinking ahead. Do you have any idea how this feature could be enabled?
As far as I can understand the the catkin plugin is based on catkin_make_isolated? Whereas catkin tools in particular the catkin build does a similarly isolated build
Building a Workspace
When no packages are given as arguments, catkin build builds the entire workspace. It automatically creates directories for a build space and a devel space:
cd /tmp/ros_tutorials_ws # Navigate to workspace catkin build # Build all the packages in the workspace ls build # Show the resulting build space ls devel # Show the resulting devel spaceAfter the build finishes, the build space contains directories containing the intermediate build products for each package, and the devel space contains an FHS layout into which all the final build products are written.
Note
The products of catkin build differ significantly from the behavior of catkin_make, for example, which would have all of the build files and intermediate build products in a combined build space or catkin_make_isolated which would have an isolated FHS directory for each package in the devel space.
is my understanding of the catkin plugin true, is it feasible to build with catkin tools?
I’m also having trouble with the plugin halting for a long time at Preparing to build workspace after calling snapcraft
[…] is it feasible to build with catkin tools?
Indeed it is, check out the catkin-tools plugin. It uses the same keywords as the catkin plugin, but uses catkin tools instead.
I know we’ve talked about this before, but the caktin-tools plugin using a specified catkin config file instead of a some of the information in the snapcraft.yaml would be possible and not too much work.
@peter.milani If you use catkin tools it should be possible to set the install directory of your build to the directory that is an unsquashed snap that you are using with snap try. This would be something like catkin config -i squashfs-root/opt/ros/kinetic in your catkin ws. catkin tools can detect your changes and the install should be placed in your currently running snap. Let me know if this works for you.
Edit: This might be more work than I initially suggested, but if it worked it would be great.
Ah @kyrofa, I’d missed that the documentation didn’t mention it, but I saw it implemented in the repository. When I tried to run it, it still sequentially built the packages, whereas my normal build setup, a number of packages are usually built in parallel? Is there some flag set to limit the compile breadth?
Additionally the pause at Building Workspace seems to be due (from my own understanding of the code) to a wholescale rewrite of paths in .cmake files for all included packages. Would it be possible to cache this or speed it up. Its taking about 10 mins per snapcraft call which seems excessive.
@cratliff so you suggest building normally then essentially copying the install into the /opt/ros/kinetic directory in the squashfs. So mounting the snap somewhere and then copying install into it?
It’s seems like it should be possible, I haven’t got it to work quite yet.
Yeah, if you run unsquashfs [snap] you can write to files inside the newly created directory with the snap contents and if you install that directory with snap try it can run the directory as if it were a snap. Then set the install directory to /opt/ros/kinetic inside the snap you could run the newly build workspace as a snap. Snapcraft does some environment modification that you need to duplicate though. Mainly making sure your python path is set the same and #! is not set to /usr/bin/python. Going through the snapcraft code and making sure you make those same changes should get things to work like this.
I wouldn’t publish a snap that I’ve done this to, but it could speed up testing as you can immediately test changes in a snap. If nothing else you could use the unsquashed snap like you’re using prime now and you could skip the step of resnapping it.
That would be lovely, but Catkin writes a lot of absolute paths everywhere that we take great pains to fix in snapcraft. It also doesn’t include any system dependencies, of course.
When you say rebuild the complete set of packages do you mean this?
snapcraft clean
snapcraft
I’m also using the catkin plugin for snapping my ROS workspace and I have to do the above whenever I make a small change in the code. I thought I was missing something, so I tried to:
snapcraft clean ros_workspace -s build
snapcraft prime
Doing this, or just executing $snapcraft (without a $snapcraft clean before), I cannot see the changes in the code in the resulting snap. Is this the behaviour you are describing?
Hi @mbeneto I don’t have to clean between calls to snapcraft. As per the catkin plugin tutorial, I am referencing a source directory, so my changes have to go there if they are compiled. snapcraft should pull that directory (or update links to it – it doesn’t seem to pull a copy in the parts directory), and do a catkin_make_isolated on it. I dont need to clean the workspace between builds.
If its a script change, then you can edit the installed targets directly in prime, though this wont be reflected in your source.
Hope this helps.
Thanks for your fast reply @peter.milani. You are right, because I’m doing tests on different machines (armhf/amd64) I was snapping it from outside the workspace. Also, the fact that my src was a relative link wasn’t helping in keeping everything organised.
Now I configured it using the source-directory tag and an absolute path to my catkin src directory but is still not snapping the changes if I don’t do a $snapcraft clean. What am I missing here? (I’m of course compiling with catkin_make between $snapcraft calls)
You dont need to compile using catkin_make between calls to snapcraft. the snapcraft command will handle all compilation up to producing the snap. Try introducing some errors in your source that will trigger compilation errors to show that a snapcraft clean is not required between builds. If that works then probably the reason your change is not showing up is due to something else.
Hi @peter.milani , firstly thanks for the tip. On the other hand, although it is related, I’m afraid I’m kidnapping your thread, should I start a new one?
I tried to do what you suggested and I’m not able to see any change nor a failure during the snapping process. In order to discard any problem regarding my source configuration, I decided to start from scratch in a different computer and follow in a clean environment the very basic snapping tutorial as shown here.
Here, I’m experiencing the same behavior I explained in my first message. After generating a basic working snap, I’m introducing errors in the code. When executing $snapcraft this is the output:
Skipping pull workspace (already ran) Skipping build workspace (already ran) Skipping stage workspace (already ran) Skipping prime workspace (already ran) Snapping ‘publisher-subscriber’ -
Snapped publisher-subscriber_0.1_amd64.snap
Everything is generated properly without any error (too bad! 
). I searched for the file I had modified and I found it in different places. I took a look to them:
$ find -name listener.cpp ./src/beginner_tutorials/src/listener.cpp ==> Source where I introduced the bug ./parts/workspace/build/src/beginner_tutorials/src/listener.cpp ==> Doesn’t have the bug ./parts/workspace/src/src/beginner_tutorials/src/listener.cpp ==> Has the bug
I’m sure I’m missing something basic, but cannot find what is it. I’m using ROS Kinetic, snap v2.32.5 snapcraft v2.41.
I kept trying and this is the only way I could find so far to force a (erratic) rebuild.
snapcraft clean ros_workspace -s build
snapcraft
With this, it avoids fetching all the packages again (pull step) and then it builds the entire part, which is a big enough improvement comparing to $snapcraft clean and then $snapcraft. Nevertheless, I’m still concerned that I’m incorrectly configuring something as I cannot replicate the behaviour you explained.
