Connecting plugs and slots in a GUI

#1

I’m looking for some clarification in the current thinking on the user experience connecting plugs and slots in a graphical environment.

At one of the previous sprints we decided on implementing a dialog in gnome-software that would give the user an option connect plugs as necessary [1]. Does this still match up with the current thinking? It also seems likely to modify these settings after installation we will need a dialog in the control center.

One case of an interfaces in a graphical snap is VLC. This has the camera interface disconnected by default. However, I think the correct long term solution for this is to prompt for this interface to be connected in the shell at usage time. So the prompt at install time is not required long term - is this correct?

Are there any cases that do require prompting at installation long term? Do any snaps exist in the store now with one of these interfaces?

[1] https://bugs.launchpad.net/bugs/1597314

Undocumented interfaces
Access USB stick from snap desktop application
Should we support channels in GNOME Software?
Ubuntu Desktop Team Priorities
Access camera with OpenCV over /dev/video0
#2

I realised today that a snapd search doesn’t return any interface information - this is only available once a snap is installed. This means we can’t prompt the user until after installation - is this the desired behaviour?

#3

Thanks for bringing this topic up, Robert.

From what I recall of our conversation quite some time ago, we were talking about a stop-gap feature, so I suggest we discuss this over again to see how we would like this to look like long term instead.

If possible, it’d be nice to start from the more general feature, allowing people to see what interfaces are connected, and connecting or disconnecting any of them. The APIs supporting those operations are already in place, as we use them to implement the respective features in the command line. We also have some improvements coming to those (@zyga is leading that) which will allow a more interface-centric view of the status quo, instead of a system-wide one.

The feature which prompts for interfaces just-in-time will come, and some of the building blocks are already in progress. For example, calling of snapctl out of hooks will soon be possible, which will allow us to use that as the backend of the prompting feature. That said, this seems like a future step after the basics work. Right now we can’t even tell what is connected or not, or to connect/disconnect, so I’d start there.

#4

This is the type of system I’ve been prototyping with the VLC snap. It scans the available plugs and shows those that a user would understand. It doesn’t show the plugs that are too complex / wouldn’t make sense to disconnect e.g. opengl, unity7, x11. We may even consider the network plug should be hidden too, as a user disconnecting network access from an app is likely to break it.

This could be shared with other GNOME Software plugins as the permission types can be shared.

Screenshot from 2017-06-08 16-43-25.jpg1200x847 131 KB

#5

That looks nice, but it’s also extremely limited compared to our interface system, and thus will offer a tremendous impedance mismatch that will be a constant pain and prevent things from working at all. These interfaces you mention, x11, opengl, etc, are fundamental and trivial to handle rather than too complex. What happens when somebody needs to decide whether to give X11 access to something that wasn’t auto-connected? Or what happens when there are multiple choices (e.g. two serial ports)? Interfaces aren’t just on/off… they connect something to something else, and we need to somehow be able to represent that.

#6

I think that while our interface system, in general, is a graph, we could get started with a UI tailored for something that only involves unambiguous connections with the core snap in much the same form as depicted above. For other things snapd could provide connection opportunities but the UI would have to be different. As a simple approximation of a graph UI (which I personally think would be unwieldy for most users and is not necessary in simple desktop software) we could offer either on/off toggles or one-of-many selection/combo boxes with available choices supplied by snapd.

#7

That seems to paint the system as more complex than it really is. If you are looking at a snap, it has a number of endpoints (plugs and slots) that may be connected to other things. For any one of those endpoints, there’s a well defined and very limited set of things that it can connect to. We should be able to make a pretty straightforward user interface for that without involving graphs.

#8

As discussed at the Snappy Sprint (June 2017, London) the current plan forward is:

  • Make an interface editor tool that provides a graphical means of connecting and disconnecting interfaces.
  • Snap this tool so that it can be shared amongst distributions.
  • Have GNOME Software install this tool when you install a snap.
  • Have the interface editor launch from an app details page in GNOME Software.
  • Provide labels in GNOME Software that show what interfaces are connected / disconnected (similar to above prototype, but not editable). This aligns with the current upstream thinking of how to show “permissions”. The current upstream thinking is that “permission” controls will be in GNOME Control Center. We can investigate how this applies to snaps as this work is done.
#9

Here are some screenshots from the current interface tool prototype (obviously more iterations required). See http://github.com/robert-ancell/snap-interface-editor:

oh my giraffe is a graphical game. It requires network, graphics and sound and has these interfaces auto-connected:

Tents is an Ubuntu Phone app. It makes use of graphics and the Ubuntu Platform snap for libraries.

VLC uses many interfaces:

Screenshot from 2017-06-30 14-42-18.png334x567 30.2 KB

The core snap has one plug:

From this initial prototype a number of things stand out:

  • There are a few common interfaces used by graphical apps - x11, unity7, opengl, pulseaudio, network. Disconnecting the first three would cause these apps to not be able to launch. The latter two may be useful to disable, though they could render the app effectively broken (some care might need to be taken displaying these).
  • Platform plugs also make an app unusable. Unfortunately since these are content interfaces it may be hard to determine when/how it is appropriate to show these controls. More metadata may be required from snapd / developers to make use of these.
  • When an app uses many interfaces (e.g. VLC) it gets very hard to see which controls are useful. For this reason we probably want to have some knowledge of common interfaces to sort and layout in a clear manner.
  • The core snap has one plug that seems dangerous to disconnect. This suggests either the core snap should not be shown here or it should hide this plug (are there plugs that may be useful to a user in the future?).
#10

Thanks for putting these mock ups together.

Without trying to rehash the long discussions in the sprint, when interfaces are disconnected certain things will not work indeed, and that’s part of the point of having interfaces in the first place. The network example provided is a good one: disconnecting it allows the user to make an application local. A proper application will just report that the network is off, as that often happens for many other reasons.

For cases which are indeed less usual to disconnect for a normal user (say, “opengl”), I think what we should have is an ability to flag an interface as “advanced” in the interface model itself, so that GUIs can learn which of those should be separated out.

It’s good to remember that interfaces won’t necessarily come up connected, though. Some interfaces deemed dangerous and not fundamental for an application may in certain cases be left out of the auto-connection case. So we need to also think in terms of “Would someone want to connect it?” instead of just “Why would someone disconnect it?”. Having no good means of connecting such interfaces may easily backfire, as it increases the pressure of having the interface connected in the first place.

Finally, there are indeed interfaces that pretty much never make sense to show up. “core-support” would be one of them. That might be a different value in the same field that flags interfaces as advanced.

For example, something along the lines of:

visibility → default / advanced / hidden

This would be defined by the interface type itself (snapd code) instead of being a per-snap setting. Lack of a value would imply “default”.

#11

Having additional metadata about how to show interfaces would be a huge help here. As per the content interface example a snap client can’t be 100% sure if this is something that is essential or optional.

We should make snapcraft help here so app developers set these appropriately and we don’t get a bad default experience.

For example, if you add an “x11” plug to your app snapcraft could warn when building the snap:

“You have used the x11 interface without a visibility tag, we recommend setting this to advanced so users don’t accidentally disconnect it”

#12

Additional features we’d like to add to this GUI for safety:

  • We’d like to have a “reset to defaults” button to allow users to get their snap back to the installed state. This would require a “default” field or similar that would show what the default slot a plug is auto-connected to.
  • We want to warn the user about common mistakes. e.g. if the network plug is disconnected we’ll show some information in GNOME Software to say something like “This app doesn’t have network access and may have limited functionality”. This will hint to users if they’ve forgotten this is disabled or made a mistake how to correct it.
#13

I don’t think @niemeyer is suggesting this has a plug property in snapcraft.yaml for the snap application, but rather part of interface definition in snapd.

I think if we add that has an interface property, the result will be that a lot of snap app will set all their plugs to hidden, resulting in no possibility to disconnect anything.

#14

Ah, right. This solution is mostly just moving the policy from the client to snapd (which has advantages and disadvantages).

There’s two cases which this solution doesn’t work well:

  • Non-core interfaces which might be essential / advanced (e.g. platform content interfaces).
  • Interfaces that may be essential for one snap (e.g. network access in a network scanner) but not another (e.g. network access to the calculator).
#15

Good use case on the content interface for platform snaps. Right now, if the app is ran as a CLI tool, it will warn about it, but we don’t have any graphical equivalent when run from their .desktop file.
Let’s see what Gustavo thinks about it…

#16

This is not moving from the client to snapd, since that information was never in the client. This is simply storing the data where it belongs, next to everything else about that interface.

Network would definitely not be “advanced” whether for a scanner or for anything else. If you cut out network access to an application, the exact same would happen as if your laptop was offline.

I don’t mind too much whether the content interface shows up in “default” or “advanced”, as long as it shows up. There will likely be one option in most cases, or a couple if one installs two or three themes on the same system for example.

#17

A client has to interpret the interfaces to display them in an appropriate way in a user interface. Some decisions that can be made are:

  • Is this interface appropriate to the user in this context (e.g. should the core core-support-plug be configurable?).
  • What does this interface mean in the context of the snaps it connects? (e.g. what does it mean to disconnect the network interface on a network scanner or a calculator?).
  • What is the best means of controlling this interface (switch, combo box, drag lines between boxes?)
  • What is the best way of describing this interface (where to position it? what text and icons to show?)

Interfaces currently contain a label and @niemeyer is proposing they contain a flag indicating their suitability for display (for core interfaces only).

The advantage of such a system is that clients are relatively simple to write and don’t need a table of labels and flags to display them. The disadvantage is snapd doesn’t have the context of the client so it can’t tell what wording is appropriate for the label and what is advanced for the user. The client still contains information about what controls and icons to use. Any of this information could be moved between snapd and the client . There is no exact place where the data belongs; we have options with advantages and disadvantages.

The content interface can be used for many things including:

  • A set of core libraries shared between snaps.
  • Optional data files, e.g. themes.

I don’t think this can be easily categorised into either both “default” or both “advanced”.

Brainstorming some possible ways forward:

  • We decide the content interface is too general and make some more specialist content like interfaces with limitations, e.g. “theme”, “platform”.
  • We allow the developer to provide hinting information about some or all plugs so that they can be more easily displayed.
  • We make clients have a small table of known common interfaces (e.g. the Ubuntu platform). This would cover the majority of cases where users could be confused.
  • We decide the content interface is not suitable for platforms, and use one of the new snap types in development.
#18

I can see the value in an interface type specifying a default visibility, but it isn’t clear to me that every app would treat an interface the same way.

One app might be well behaved and continue functioning when you disable the network interface. A second one might essentially become non-functional. In the first case, it might be safe to show the toggle to the user, and in the second you might want it set to advanced or hidden.

#19

This is about all interfaces rather than just for core.

Quite the opposite: snapd is the only place that has context about what is appropriate or not. It’s the place where interfaces are coded, and assertions evaluated. If we put that in the client tomorrow there will be a new interface and the client will have no clue about what to do.

#20

A concern I have about developer hinting is that different developers will make different choices which could be confusing for users. “Why is ‘network’ in advanced for the calculator but in default for the scanner?” (I turned this example around from the expectation that the calculator shouldn’t require networking but the scanner should to illustrate potential confusion).

Also, interfaces give access to the system and users can decide what to connect/disconnect so another concern I have is that a mal-intentioned developer might try to manipulate hinting to confuse users into making ill-informed choices. If we pursue developer hinting, lets keep this in mind.

I agree with @niemeyer for these reasons and the ones he listed.