Multitouch is not working properly on Ubuntu-Core

Hi everyone,

I’ve been struggling to integrate a multitouch film into my kiosk solution without success. For some reason, the multitouch does not work properly on the ubuntu-core (core20).

The snaps that I’m using in setup are: ubuntu-frame (latest/stable) and this custom electron application. The device I’m using is an Intel NUC (so pc-kernel is being used as well).

After some research I realized the electron aplication reports the maxTouchPoints property (navigator.maxTouchPoints) as 0 even when the multitouch device is connected. This seems to be causing some issues to the libraries used on the frontend application and make them stop working as expected.

On the other hand, the same hardware setup (same computer and same multitouch film) running Ubuntu 20.04 and using the latest version of chrome works perfectly. In this case maxTouchPoints property reports a value grater than 0.

Tips to overcome this situation are welcome.

Cheers

Hi @joao.belo,

I’m not familiar with Electron or where it derives navigator.maxTouchPoints from. But there are a lot of differences between the scenarios you describe.

On Core you are using a Wayland compositor (ubuntu-frame) to (amongst other things) provide input through the Wayland protocol. I don’t think that protocol ever notifies the capabilities of the touch device.

You don’t state which compositor you are using to test on Ubuntu 20.04, but I’ll guess you are using GNOME and connecting using X11.

I would suggest testing with a Wayland compositor: GNOME can do that, but it isn’t the default for 20.04 and needs to be selected at the login screen. Then force Chrome to connect using Wayland using something like:

chromium --enable-features=UseOzonePlatform,WaylandWindowDecorations --ozone-platform=wayland --enable-wayland-ime

That won’t solve the problem (yet), but will allow you to narrow down the reason for the differences.

Hi @alan_g,

Thanks for your reply.

I’ve just repeated the test following your suggestions and the result is nearly the same. The multitouch continues to work as it is expected, but now the navigator.maxTouchPoints property is returned as 0 as it is in the Ubuntu Core setup.

To be honest I do not know what I can do more to make the touch film working properly in my solution. Do you have any additional tip? @ogra, do you have anything to add?

Cheers

Hi @joao.belo,

thanks for the update. That does narrow things down a lot.

Mir’s underlying handling of touch events has had support for multitouch “forever” (it is used, for example, with UBport’s “Ubuntu Touch”).

But I’m afraid that “multitouch does not work properly” doesn’t give me much clue what might be going wrong. Are there some specific scenarios you could describe? (E.g. “two finger pinch not recognised”). And are there other things that do work properly?

Anyway, if Electron works with GNOME’s Wayland compositor, then I can think of two likely possibilities:

  1. There can be differences in the way touch devices behave. There’s an example found by UBports being discussed at the moment where strange behavior from the device leads to problems

  2. Electron’s understanding of the relevant Wayland protocol differs from Mir’s. As Electron works with GNOME, then it is likely that Mir has it wrong

In either case, it will likely be necessary change Mir code to accommodate your touch device and/or Electron. But that can’t be done without a detailed understanding of what is going wrong and why.

There are ways to extract logs of the Wayland messages and the udev input to identify the differences, but the Forum isn’t the best place to discuss that. (A Mir issue is).

Hi @alan_g,

Thanks again for your reply.

When I say the multitouch doesn’t work properly is exactly because some gestures are not recognized, although the touch film supports them. Two fingers pinch is precisely the gesture that is not working. I can share with you a video showing the situation I’m facing.

In the meantime, I have repeated all the tests using a touch screen instead of the touch film I was using before, and the results are the same. The two fingers pinch does not work.

I going to open a Mir issue in order to get more help as well.

Does this additional information narrow things down a bit?

Hopefully yes: The fact you see it with a touchscreen too suggests it isn’t device specific.

And “two finger pinch” gives us something specific to test with Electron apps we have access to.

Good! So, will you perform some tests on your side?

Let me know if need any additional details and let me know when you have a conclusion.