Experimental flag for hiding ~/snap

[jamesh]

@diddledani

This worries me. People will read “snap data is not placed in ~/snap any more” and think that applies to all snap data. They will be very shocked when most of the snaps they use are still putting their data there because they’re not core22-based.

My reading is that the move from ~/snap to ~/.snap would happen for all snaps on the system. It’s the environment changes that would be limited to core22+ snaps.

That makes sense, since the environment changes are very likely to break existing snaps. For example, imagine I have a program that writes its configuration to $HOME/.foorc and I’ve created a core20 based snap out of it. With the current environment, it will write that config to $SNAP_USER_DATA/.foorc. If we leave $HOME unchanged for snaps, the program will start getting AppArmor errors when trying to access its config file.

There’s no core22 snaps in existence right now, so making the environment change contingent on the base snap won’t break any existing published snaps.

[Coeur-Noir]

What are the flaws of :

⋅ a file named hidden

⋅ at the root of folder /home/$USER

⋅ which contains names of folder to hide, one name by line

?

Nothing else to change, just check if this file exists when installing snapd and append snap to it ? Or create it if not already.

[alexmurray]

@miguelpires

That’s not what is described in the main post. The XDG_* variables will point to directories under ~/.snap/data/<snap_name>/<revision>/.

Hmm so traditionally XDG_* variables live under $HOME - and with the current situation where $HOME == $SNAP_USER_DATA this is still respected.

However with this new approach, since $HOME is ~/Snap/<snap_name> then the XDG_* variables are now under $SNAP_USER_DATA instead (ie ~/.snap/data/<snap_name>) which seems extremely confusing to me. How does a user / developer know which location to expect their snap will use when storing data? Why is “user-facing data” seen any different than non-user facing data? This is breaking a pretty well-established convention.

Also I agree this doesn’t solve the majority of user complaints which are that the ~/snap directory exists in the first place - we are trading that problem for a similar one (ie. now ~/Snap exists too) as @mardy suggests above.

[jamesh]

Adding to what Alex said, I think the part of the proposal to add a ~/Snap directory will be deeply unpopular: it’d feel like a bait-and-switch when combined with moving the existing ~/snap directory.

One of the main complaints about the existing directory hierarchy is that people expect to have control over the non-hidden files in their home directory. But if they exercise that control over the contents of the ~/snap directory and move files/directories around, things break. The ~/Snap directory would have the same restrictions. The imposed structure might be simpler (i.e. no subdirectories with revision numbers), but it is still constrained.

I suspect much of the complaints come from desktop users, and in most of those cases they’d want to read and write documents in standard directories like ~/Documents or in other locations custom to the user. This seems much better handled by the document portal or home interface rather than the addition of a ~/Snap directory.

[Coeur-Noir]

As a desktop user I already hide the ~/snap folder which I don’t need to see permanently.

[niemeyer]

Hello all,

Just to add some color to the conversation, as it looks like there’s some confusion about what we’re trying to do and why.

@jamesh

Adding to what Alex said, I think the part of the proposal to add a ~/Snap directory will be deeply unpopular: it’d feel like a bait-and-switch when combined with moving the existing ~/snap directory.

There are two different points at play around this discussion. One of them is about the fact people are unhappy that there is a directory inside $HOME that is related to snaps at all. We’re not trying to address this argument, because the point of this directory is to offer the snap a safe and isolated space that is still convenient for the user to see. Note that this is exactly the same reason why we have Pictures, Downloads, etc, inside the home directory. So hiding the directory from view goes against its very purpose, which is to be visible.

There’s a second argument about it which is related to the fact the shape of that directory today is unpolished, and that’s true. Back when we created the system there was too much in our minds, and we simply copied an internal pattern that we’ve used in a few different places, which is a correct pattern, but it’s not a proper user interface. We’re trying to address that, not bait-and-switch, and not take blind criticism at face value either, as that will continue to exist regardless of what we do.

We’re not yet completely sure that the approach we’re going for is the right one, though, which is why instead of making the change visible we’ll first have a feature flag that enables experimenting with the concept privately. Once that’s available, we’d encourage experimenting with it and providing feedback to the team. We certainly don’t want to get that one wrong again.

[jamesh]

I do think the proposed layout of ~/Snap is better than the existing ~/snap directory, but if I’ve got 50 snaps installed it’s still going to feel a bit unwieldy. If many of those directories are empty (or appear empty), then it’s not going to feel like something the user owns. And that is one of the main complaints about the existing ~/snap directory.

Maybe one solution would be to make creation of the directory opt-in? If I’ve got 50 snaps installed, but the only subdirectories under ~/Snap are for the music app and photo manager then the directory will be much more manageable in a graphical file manager. It’d also mean that don’t need this facility (e.g. a game or a simple command line utility) don’t impose visible complexity on the user.

If the we’re only showing these directories to the user when they have a purpose, then they are more likely to accept it.

[niemeyer]

That’s certainly an option, with the down side that all those snap authors that don’t realize that they need to opt into it will just have their files hidden under a ~/.local/share directory or similar. So with that in mind, what’s the real strategy? Are we going to be nagging developers to enable the flag so that users can see content that would typically be exposed anyway for normal applications? How do we do that realistically? Or is it worth hiding every application $HOME away from people’s view regardless, because people are offended by the very notion of a directory with application names under it? Note that this is literally $HOME for those applications, not just $XDG_SOMETHING.

but if I’ve got 50 snaps installed it’s still going to feel a bit unwieldy.

Having 50 snaps should not create the directories. Using 50 snaps would.

[James-Carroll]

@niemeyer

Or is it worth hiding every application $HOME away from people’s view regardless

On the desktop GUI level, aren’t we already at this point? The desktop-launch scripts configure filepickers to show the real Downloads/Pictures/etc, the one they fail at changing is $HOME itself because there isn’t a good technical solution for it that doesn’t cause permissions issues. On the other hand, extensions go one step further and enable XDG portals, which also shows the real $HOME, and the community (and Snaps own documentation) considers that proper and expected behaviour. I can’t imagine many people actually expect or appreciate the behaviour of a snap saving a file in what it claims is $HOME but isn’t, it’s a recipe losing data!

I do think it makes sense for some apps, but IMO, it’s very much balanced towards the majority of apps not needing to make any of the snap folders visible, to the point making it opt in makes sense to me.

On Ubuntu Core where the home interface doesn’t get autoconnected and the portals won’t work, I can understand the need for a user-visible folder for the snaps internal $HOME, but elsewhere, I’d consider it an implementation detail that’s best kept hidden as far as reasonably possible.

[niemeyer]

@James-Carroll

On the other hand, extensions go one step further and enable XDG portals,

That’s a fine response for the cases where it applies, but snaps are not limited to applications that make use of XDG portals. If that was the case then indeed there would be no point in having any other isolated home directories for the application at all.

[James-Carroll]

Even the most basic CLI app will have inconsistencies with home. Relative paths based on current working directory means that users often don’t notice the fact the snap has a different internal home to the host; so the illusion is mostly the same as the portals case. But it gets worse when you consider e.g:

snap run example $HOME/data

Bash will substitute $HOME with what the host sees, not what the snap sees.

I personally think it’s asking too much of the users for them to understand the technical justifications. If the packages documentation doesn’t explicitly mention to use ~/Snap, then they shouldn’t even need to know it’s a thing, and we should encourage hiding the technical details as much as possible. The opt in approach makes most sense to me here.

[jamesh]

@niemeyer

Having 50 snaps should not create the directories. Using 50 snaps would.

My point is that many of the snaps I actively use don’t need this. For example, I actively use the lxd snap and the current data it’s stored in my home directory is a config file:

/home/james/snap/lxd/common/config/config.yml
/home/james/snap/lxd/23037/.config/lxc/config.yml
/home/james/snap/lxd/23001/.config/lxc/config.yml

I suspect they’d be absolutely fine with those files being moved to ~/.snap/data/lxd, and have nothing of interest to place in the new ~/Snap/lxd directory. The majority of the snaps I use are in a similar situation, since the only user data they’re storing is stuff that they’d stash in ~/.config or ~/.local/share if unconfined.

These snaps would just be adding noise under ~/Snap, and making it harder to find the data from snaps that do want to produce user visible documents. If ~/Snap is only populated by directories with a purpose, then it is going to be more useful to the user and the snaps that opt in to using it.

[niemeyer]

I understand your idea: only have it when it’s relevant. That sounds like the perfect solution indeed. Except, that’s a bit thin as an actual design plan. Hiding everything until people find out that they cannot see the files? Okay… so people enable the flag once someone files a bug that they cannot find normal application files that are supposed to be visible unless they dig inside ~/.config. What’s next? File a bug, and wait for the package maintainer to notice and the app to be updated? Eventually, app gets updated, then how would that work? Do we move files around under the application to expose them? What do we do with the paths that were previously set in the application configuration? How do we tell the user that application files are moving around? What if the application hides it again?

Again, I can definitely feel empathy for the idea of only displaying things when they need to be displayed. But we need something a bit more clear in terms of plan.

[Lin-Buo-Ren]

TLDR, I like to put some of my thoughts and probably unrelated preferences here while skimming the thread…

• In my expectations user’s snap data should be under ~/.local/snap, following the existing file hierarchy traditions
• I don’t really like applications that believe creating a non-hidden directory for their own purpose in my home folder is a good idea, a ls ~ command should only show the standard user directories and items that I created on my own, nothing more.
• If there’s a need to create such a folder for accessibility or other reasons, it should be an opted-in option and the path should be customizable by the users.
• Perhaps making an extension of some sort in the file manager application that presents the data without actually making the directory structures would be preferable
• Perhaps a FUSE filesystem specifically used to present such information on an arbitrary directory would be preferable, without making the actual directory structures.

My 2 cents.

[niemeyer]

@Lin-Buo-Ren

I don’t really like applications that believe creating a non-hidden directory for their own purpose in my home folder is a good idea, a ls ~ command should only show the standard user directories and items that I created on my own, nothing more.

Most people commonly have visible folders in their home directories that applications make use of all the time. Hopefully we can agree on that? Okay, so with that aside, when we attempt to isolate applications from one another, the reasons why such folders are relevant are still there. What we’re trying to do is to have a middle ground where we bridge a world where applications were commonly reading and writing on such visible space, with a world where applications are isolated from one another. Eventually we’ll have better ways to handle this situation, and we’re working on that as well, but that’s a separate topic. Right now we’re trying to find that middle ground so we can improve security and privacy while having these use cases into account still.

[RefreshingSnapper]

~/snap vs. ~/Snap

Uh oh! My case-insensitive filesystem just fell apart!

[miguelpires]

Update

Summary

In January, we wrote this post to explain the introduction of an experimental hidden-snap-folder flag that allows users to opt-in to move their ~/snap folder into ~/.snap/data. Our post also described a plan to eventually relocate snap home directories to a location under ~/Snap.

During our 22.10 roadmap planning sprint, we reconsidered parts of our ~/snap strategy, and this means that snaps refreshing to the new and latest core22 base will not automatically migrate ~/snap into either ~/.snap/data or ~/Snap, at least for now. More details on this can be found below.

Our new plan is to introduce an additional experimental flag to enable migration to ~/Snap only for a list of specific snaps. This isn’t an automatic migration, but we still wanted to let users easily test the potential migration feature. With that in mind, we’ve also added a debug command to safely migrate snaps’ home directories.

Note that the original experimental hidden-snap-folder flag remains and still allows users to move ~/snap for core20- and core18-based snaps to ~/.snap/data.

Previous plan

Our original plan was to migrate snap home directories into a sub-directory of ~/Snap. Each new directory would be initialised with the contents of a snap’s revisioned directory, used previously as its home directory. However, there would be no revisioning in the new location going forward. This change would meet our objectives of 1) hiding the data that a snap may place in its internal directories, while 2) still providing a cleaner and tidier location for users to interact with snap user-facing data.

Another objective that we had was to minimise migration pains for both snap users and developers. To help here, we decided to migrate snaps to the new home directory during their first refresh to a core22 base. Each snap is obviously configured differently, and some might do things that affect the home migration or its subsequent operation (e.g., mapping the HOME environment variable to a custom location, using SNAP_USER_DATA and HOME interchangeably). Therefore, it was crucial that migrated snaps were tested under the new environment. Migrating during a refresh to a new base was intended to minimise the effort required by snap developers, since they’re most likely already testing their snaps with the new base anyway.

New plan in detail

Considering we want to provide users and developers with the time and opportunity to experiment with the new snap home location, we have decided to not migrate snap home directories with the core22 base refresh.

Instead, we have introduced a new experimental flag and debug command that allow users to opt-in to the new environment. The feedback we get from snap users and developers will inform our decisions on how we deal with the new home directory in the future. Options we want to consider include:

• Allowing snaps to opt-in to having their home directory under ~/Snap via a setting in their snap.yamlor snapcraft.yaml. This leaves the decision to migrate in a snap developers’ hands and allows them to test their snaps with the new home location before releasing them to users.
• Automatically migrate snap home directories to a location under ~/Snap when a snap refreshes to a revision that uses the core24 base. This would also allow snap developers to test their snaps with the new home location before they reach users.
• Automatically migrate all snaps at some yet-to-be-determined point. Before this becomes feasible, we’d need to know that there was very little risk of snaps breaking due to the home directory change.
• Don’t migrate snap home directories at all.

None of this affects the first part of the plan. The experimental hidden-snap-folder flag will remain for the foreseeable future, allowing users the option to move ~/snap to ~/.snap/data, effectively hiding it. This will eventually be made into the default behaviour, unless a compelling reason not to do it is discovered during the ongoing experimental phase.

Using the new experimental flag

For the new experimental flag and debug command, there are two steps required to manually trigger the migration of the home directory to a location under ~/Snap.

First, as the feature is experimental, you must first acknowledge this by enabling a feature flag. You can do that with the following command:

sudo snap set system experimental.move-snap-home-dir=true

Second, initiate the migration by running the following command debug command:

sudo snap debug migrate-home <snap1> <snap2>...

This process will generate a change that creates a directory under ~/Snap and initialises it with the contents of the previous (versioned) home directory.

Exceptions which are not copied include some of the default directories in the XDG Base Directory Specification, in particular .config, .local/share and .config. These files remain in the original revisioned location and their respective environment variables will remain set to these original locations after the migration has completed.

The exceptions are to avoid the new home directories becoming overwhelmed with configuration files, making them confusing for users who wish to interact with their application data.

Note that this can be done with or without the hidden-snap-folder feature as the two experimental options are wholly independent of each other.

[James-Carroll]

Hi Miguel, can I just confirm the following two points?

@miguelpires

The experimental hidden-snap-folder flag will remain for the foreseeable future, allowing users the option to move ~/snap to ~/.snap/data,

Contrasted with

@miguelpires

Exceptions which are not copied include some of the default directories in the XDG Base Directory Specification, in particular .config, .local/share and .config. These files remain in the original revisioned location and their respective environment variables will remain set to these original locations after the migration has completed.

I’d read the second paragraph as implying that the folders at $HOME/snap/$snap_name/ still exist, containing the $XDG_ dirs. However given the context, I’d have thought these XDG dirs would be migrated to $HOME/.snap/data and the variables pointed there, rather than “set to these original locations”, which I’d read as $HOME/snap/. Can you confirm which behaviour is expected with the setting?

[miguelpires]

Hi James,

Your second statement is correct, the XDG dirs will be left under ~/.snap/data, if the hidden-snap-folder is turned on. I think the ambiguity is in the “original locations” part of the statement. To clarify, I meant that as the locations where they were before the home migration, which is totally independent from the hidden migration. So depending on whether hidden-snap-folder is on or not, that’s either ~/.snap/data or ~/snap, respectively.

[SYaslem]

guys, please use $HOME/.local/share/snap as standard then create system link in $HOME that easy and clean right? If I don’t want snap or .snap folder in my desktop I can easily remove the symlink!

other idea for $HOME/.local/share/snap is to add data folder which contain applications user data