Monitoring/ Debugging ROS snaps on Ubuntu Core over SSH

Hi there,

I just built my first ROS snap with two nodes and was surprised how convenient this is. However, I’m not sure if using a ROS snap on Ubuntu Core 18 is the right way during the development phase of our robot.

My main concern is debugging/ monitoring the robot in real-time during operation. ROS makes this very easy by giving access to the messages, output streams etc. I’m not talking about errors which I could figure out during development on my Ubuntu Desktop, but rather problems which arise during testing at our customers place because of hardware constraints, user interaction, overheating etc.

I would like to access the Ubuntu Core 18 over SSH to:

  1. Check the outputs of ROS in the terminal
  2. Using the ROS Command-line tools, e.g. rostopic echo, rostopic list, rosservice call

For the first point, I just can’t figure out how to see the output of a ROS snap which runs as a service. When I connect over SSH, I assume, I somehow need to bring the process to the foreground.

For the second point, it is somehow clear to me that I can not access these tools from the outside of the snap, since ROS is not installed there. A possible solution might be to install ROS in the classic mode to have these tools available (at least this works on my Desktop machine). But I have my doubts that this is a proper solution since there is also no way to source my workspace in the snap.

I’m thankful for any help.

Best,

prex

1 Like

Hi,

You can use snap logs <snap-name>.<snap-app>. You can also stop the service and run it manually with the full output sent to stdout with:

$ snap stop <snap-name>.<snap-app>
$ snap run <snap-name>.<snap-app>

I’m not sure what is best practice from the perspective of ROS in general, but could you bundle these commands inside the snap you are developing, since ROS must be inside the snap, no? I.e. add something like:

apps:
  rostopic:
    command: bin/rostopic
  rosservice:
    command: bin/rosservice

to your snapcraft.yaml.

This will not work on Ubuntu Core, since classic snaps cannot be installed on Ubuntu Core.

I don’t get any output when using snap logs. There is also no output when I manually start the service. It just tells me “Started.”

Just without the “bin/”. Works as expected.

Thanks!

The question about how to see the ROS output is not really answered for me. I tried different commands, but none of them are satisfying.

snap logs <snap_name>.<app_name>

-> Does not give me any output.

systemctl status -l snap.<snap_name>.<app_name>

-> On Ubuntu 18 Desktop, this gives me some of the latest outputs. But on Ubuntu Core, besides the information about the service, this does not show me the output streams from ROS.

sudo journalctl -fu snap.<snap_name>.<app_name>

-> This does more or less what it should, except that the order of the output is not correct anymore. The log timestamp does not match the ROS timestamp.

Did you try snap logs -n=all <snap_name>.<app_name> ?

For reference, I’m using the example with the ros-talker-listener: https://snapcraft.io/test-ros-talker-listener-msrl

test-ros-talker-listener-msrl.rostopic echo /chatter shows me what I’m expecting to see in the log. But this gives me now:

user@localhost:~$ snap logs -n=all test-ros-talker-listener.run
2020-01-08T17:40:27Z systemd[1]: Started Service for snap application test-ros-talker-listener.run.
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]: Unhandled exception in thread started by <bound method XmlRpcNode.run of <rosgraph.xmlrpc.XmlRpcNode object at 0x7faaa437da50>>
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]: ERROR: Unable to start XML-RPC server, port 11311 is already in use
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]: Traceback (most recent call last):
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]:   File "/snap/test-ros-talker-listener/3/opt/ros/melodic/lib/python2.7/dist-packages/rosgraph/xmlrpc.py", line 222, in run
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]:     self._run()
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]:   File "/snap/test-ros-talker-listener/3/opt/ros/melodic/lib/python2.7/dist-packages/rosgraph/xmlrpc.py", line 291, in _run
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]:     self._run_init()
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]:   File "/snap/test-ros-talker-listener/3/opt/ros/melodic/lib/python2.7/dist-packages/rosgraph/xmlrpc.py", line 241, in _run_init
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]:     self.server = ThreadingXMLRPCServer((bind_address, port), log_requests)
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]:   File "/snap/test-ros-talker-listener/3/opt/ros/melodic/lib/python2.7/dist-packages/rosgraph/xmlrpc.py", line 122, in __init__
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]:     SimpleXMLRPCServer.__init__(self, addr, SilenceableXMLRPCRequestHandler, log_requests)
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]:   File "/snap/test-ros-talker-listener/3/usr/lib/python2.7/SimpleXMLRPCServer.py", line 593, in __init__
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]:     SocketServer.TCPServer.__init__(self, addr, requestHandler, bind_and_activate)
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]:   File "/snap/test-ros-talker-listener/3/usr/lib/python2.7/SocketServer.py", line 420, in __init__
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]:     self.server_bind()
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]:   File "/snap/test-ros-talker-listener/3/usr/lib/python2.7/SocketServer.py", line 434, in server_bind
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]:     self.socket.bind(self.server_address)
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]:   File "/snap/test-ros-talker-listener/3/usr/lib/python2.7/socket.py", line 228, in meth
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]:     return getattr(self._sock,name)(*args)
2020-01-08T17:40:30Z test-ros-talker-listener.run[1070]: socket.error: [Errno 98] Address already in use

Sounds like you have roscore already running, there. I assume it’s being run by your service? Try stopping the service, first.

Regarding logs, are all of nodes in your launch file set to output="screen"? If not you’ll need to look in logfiles, which are going to $SNAP_USER_DATA/ros. If it’s a service, that will resolve to /root/snap/<snap name>/current/ros.

In retrospect I probably should have written one comment instead of three :stuck_out_tongue: . There are several good questions here, though!

As @ijohnson pointed out, while you’re correct in that the snap’s workspace cannot be directly sourced and used outside the snap, you can certainly export certain tools from the snap to be utilized for debugging purposes. That may be all you need, but you may also find yourself adding more and more debugging tools to the snap to account for this possibility, thereby bloating the snap beyond what you had originally hoped. If you find yourself doing that, keep in mind that snaps aren’t namespaced from the network. You can use that to your advantage, for example, by installing the lxd or docker snap on the device, firing up an ephemeral container, installing all the ROS debugging tools you want on it, and setting it up to use the roscore on the host (which is being run by the snap service). That may open up more possibilities for poking at things. When you’re done you can toast the container.

1 Like