Part referencing other part?

[cg-bbi]

I’m installing two dependencies, one that relies on the other.

The first is our EtherCAT master kernel module and related libraries. I know I’m going to have issues with this later on but I’m going to defer that until it fails to run. In the meantime, I’ve managed to at least build and install it, so we’re good for now.

The second is the library that provides the interface to use the EtherCAT master. It requires an environment variable fed to cmake pointing to the permanent path of the EtherCAT master’s libraries. I’m unsure of what this should be and how to find it.

I defined each of these as parts, since they need to be downloaded from their respective repos and installed separately. My extremely crude first stab is to do this:

cmake -DETHERLAB_DIR=$SNAP/../ethercat/install/ ../

(ethercat being the name of my other part)

But that neither works nor seems acceptable in a civilized society. Using the install directory doesn’t seem right at all, I feel like it should point at the final share path… right? I’m guessing there must be some way of doing this or a better approach. Appreciate any advice!

[cg-bbi]

I could also approach this by moving the files that need to be accessible to the second part into some other shared area. I’m not sure what the best practices for something like this are. There are some other dependencies (to be dealt with) that would also benefit from it. Is creating folders within SNAP_LIBRARY_PATH considered an OK practice? Is it even possible from within snapcraft.yaml?

Sorry for all the Snapcraft 101 questions. Appreciate your help, as always.

[kyrofa]

At first blush it sounds like you have two parts, one that depends on the other. If A depends on B, then if you add after to A, B will be staged before A is built:

parts:
  A:
    plugin: cmake
    # ...
    after: [B]

Then you can use $SNAPCRAFT_STAGE in the YAML as the variable pointing into the staging area. However, there’s something that’s bothering me about this:

It requires an environment variable fed to cmake pointing to the permanent path of the EtherCAT master’s libraries.

What does this mean, exactly? Specifically, what do you mean by “permanent”? This causes my concern because once the snap is built, it’ll be mounted to another location.

[cg-bbi]

Yeah, the after part to control order of operations is already in place.

To answer the other question, my language my have been imprecise or weird. It’s moot now. I can give you details when I email you some other notes, but TLDR: hacky crap that predates my involvement with this.

My remaining related question, though, is this: dependency 2, once built, provides a C library that ROS needs to build. The ROS side looks like this.

FIND_LIBRARY(SIMPLECAT_LIBRARY simplecat /home/myuser/simplecat/lib/)

What is the Snapcraft best practice? This makes it look like I can maybe copy them into $SNAP/usr/lib/ manually and then modify our ROS install to find them there, but all instructions I can find apply to runtime, not buildtime.

[kyrofa]

Are you in control of dependency 2? Best practice (even ignoring snaps) would be for it to create a CMake config file or finder and install it into the right place. Something along the lines of this. Then it can be found in the staging area by simply telling CMake to look there for the config file.

[kyrofa]

Also:

TLDR: hacky crap that predates my involvement with this.

Man, been there .

[cg-bbi]

RE: control, yes and no. “Yes” in that we’re working from a fork of someone else’s repo. (Robotics engineer modified some files to do… something?) “No” in that none of us wrote the build process and I’m not entirely confident I can implement this without filling a lot of gaps in my understanding of CMake. (I wish this was all written in… practically any other language.) I do control where this dependency builds, so I can build straight to a shared location, I just need to know the Snapcraft best practice for creating and using shared paths between parts during buildtime so ROS can find it.

This make sense?

[kyrofa]

Makes total sense. Okay, let’s assume we can’t change dependency 2. If it’s its own part, and you run snapcraft stage, where does the simplecat library end up installed? stage/lib/? If so, honestly, this should just work because the Catkin plugin should set CMAKE_LIBRARY_PATH appropriately. Unfortunately, it does not, so I’ll log a bug for that. Until then, try this (which will also make sure my bug fix will work):

parts:
    # ... 
    ros-part:
        plugin: catkin
        # ...
        catkin-cmake-args: -DCMAKE_LIBRARY_PATH=$SNAPCRAFT_STAGE/lib
        after: [dependency-2] # you get the idea

[cg-bbi]

As far as I can tell, Simplecat doesn’t end up there. The only files I see in stage/lib/ belong to the Ethercat master, which is essentially the device driver for which Simplecat provides an interface. You can see the one we forked here. We skip building its examples, so our manual build is just…

# clone it, cd into it
mkdir build
cd build
cmake ../
make

We have one ROS package that relies on it. In that CMakeLists.txt, we have the FIND_LIBRARY command from my earlier post:

FIND_LIBRARY(SIMPLECAT_LIBRARY simplecat /home/myuser/simplecat/lib/)

…and…

include_directories(/home/myuser/simplecat/this-other-path-with-custom-crap)

Time constraints being what they are, I’m leaning towards to just kicking this whole thing way down the field, building an image that already has all this stuff installed manually, and continuing to use Ubuntu Server with our shiny new ROS snap for the time being. That at least works. I’m concerned that even if we can get this working, the Ethercat libraries and modules will present a much more significant issue.

[kyrofa]

Oh interesting, it seems that it’s missing install rules. Okay, in that case, you’ll need to install it by hand by using the install keyword, something like:

parts:
  # ...
  simplecat:
    plugin: cmake
    source: blahblah
    install: |
      # You're in the build dir now. We're essentially
      # writing install rules in shell.
      mkdir -p $SNAPCRAFT_PART_INSTALL/lib
      cp lib/* $SNAPCRAFT_PART_INSTALL/lib/
      # Assuming headers are required:
      mkdir -p $SNAPCRAFT_PART_INSTALL/include
      cp include/* $SNAPCRAFT_PART_INSTALL/include/

Assuming headers are required, you’ll also need to add something like -DCMAKE_INCLUDE_PATH=$SNAPCRAFT_STAGE/include to the catkin-cmake-args in the example a few posts above.

[kyrofa]

If you need to kick the can down the road no problem, but I’m here if you want to continue.

[cg-bbi]

Great, I’ll give that a shot. Thank you! This isn’t a catkin package, I’m using the nil plugin and setting source to my repo. Been building manually so far. Is that the right approach?

$SNAPCRAFT_PART_INSTALL/lib is specific to this part, right? If so, I’m still missing the correct way to reference it from my ros part, which is what you so generously helped with previously.

[kyrofa]

Been building manually so far. Is that the right approach?

Since the part in question uses CMake, I suggest trying the cmake plugin. Less you need to do manually. I actually suggest trying it without any hackery first-- maybe it has install rules I missed! So try this, and see how it works:

parts:
  # ...
  simplecat:
    plugin: cmake
    source: blahblah

If it errors out saying something that effectively means “what the heck, no install rules” try this (assuming the lib gets built in <build dir>/lib and the includes get placed into <build dir>/include):

parts:
  # ...
  simplecat:
    plugin: cmake
    source: blahblah
    artifacts: ['lib', 'include']

If that still doesn’t work, you can try the more manual shell hackery outlined above, or use the nil plugin as you’re doing and do EVERYTHING by hand. Yucky, but doable.

$SNAPCRAFT_PART_INSTALL/lib is specific to this part, right? If so, I’m still missing the correct way to reference it from my ros part, which is what you so generously helped with previously.

Bingo. Every part goes through a lifecycle of steps:

• pull: Pull the source code for this part
• build: Build/install this part (place things into $SNAPCRAFT_PART_INSTALL)
• stage: Place the installed stuff into the common staging area (i.e. migrate everything in $SNAPCRAFT_PART_INSTALL into staging area)
• prime: Place the final stuff into the priming area (which after all parts are complete, ends up being the snap)

So if you need to build part A that depends on part B, you need to make sure part B is already in the staging area before part A is built. That’s exactly what the after keyword does. By saying after: [B] you’re saying “please stage part B before building this part” which means you can use the $SNAPCRAFT_STAGE variable to access its libs, includes, etc. That’s what this post is covering.

[cg-bbi]

Ah, this is great. I think I’ve been misunderstanding the lifecycle a bit and didn’t realize there even was a shared staging area! facepalm

You anticipated the “what the heck, no install rules” correctly. Unfortunately, it looks like nothing winds up in build other than CMakeCache.txt, cmake_install.cmake, and the CMakeFiles directory. When building manually, I always have to run the cmake command from within my manually-created build dir.

mkdir build
cd build
cmake ../
make

You can see this is the process demonstrated in the project’s build_all.sh script, here. We don’t need the example files, so we just run those first commands.

[cg-bbi]

This feels like a new iteration of hacky crap but this populates the parts/simplecat/build/ directory:

simplecat:
  plugin: nil
  source: blah
  build-packages:
    - cmake
  prepare: |
    cd ../
    cd build
    cmake ../src
    make

[kyrofa]

Huh, the cmake plugin does an out-of-source build as well, not sure why that isn’t working. Regardless, yeah, if that works, might as well also add the “install rules” (I also suggest changing the “prepare” to “build”):

simplecat:
  plugin: nil
  source: blah
  build-packages:
    - cmake
  build: |
    cd ../
    cd build
    cmake ../src
    make
  install: |
    # You're in the build dir now. We're essentially
    # writing install rules in shell.
    mkdir -p $SNAPCRAFT_PART_INSTALL/lib
    cp lib/* $SNAPCRAFT_PART_INSTALL/lib/
    # Assuming headers are required:
    mkdir -p $SNAPCRAFT_PART_INSTALL/include
    cp include/* $SNAPCRAFT_PART_INSTALL/include/

[cg-bbi]

We’re in business! The installed files are copied into stage. I imagine I can then feed the simplecat lib path to catkin via catkin-cmake-args so it builds correctly? Going to poke at it some more now.

[kyrofa]

Yes exactly! The lib ended up in $SNAPCRAFT_STAGE/lib, then? And the includes?

[cg-bbi]

Just lib, there’s no includes. I also modified it to copy the src dir and its subdirectories, since there’s something in there that the ROS install wants.

[cg-bbi]

Just an update before I check out for the long weekend, I made it to a place where everything seems to build correctly, but I’m getting errors that suggest I need to mess with the install rules a bit more. I’m certainly equipped to do that at this point. Good things all around. Thanks for your help, once again!