Experimental flag for hiding ~/snap

snapd
1

Experimental support for hidden snap directory

Snap user data is currently stored under ~/snap where each snap has a self-named subdirectory that contains:

  • numbered directories for recent and current revisions, used to store versioned data
  • a symlink called ‘current’ that points to the installed revision directory
  • a ‘common’ directory to store non-versioned data accessible across revisions

This means the directory structure for snap user data currently looks something like the following:

~/snap
├── snap_name
│   ├── 152
│   │   └── options.cfg
│   ├── 159
│   │   └── options.cfg
│   ├── common
│   │   ├── client.crt
│   │   └── client.key
│   └── current -> 159

Snaps use these directories to store application data, caches, and configuration files, but having them visible to users may result in a poor user experience.

To help with this, we’re happy to announce a new experimental flag, experimental.hidden-snap-folder, that allows snap directories to be moved from ~/snap to ~/.snap/data (note the dot before ‘snap’ to hide the directory). This location was chosen because ~/.snap is already being used to store user credentials.

Snaps installed after the flag has been set will store their data in ~/.snap/data by default, while previously installed snaps will have their data migrated to ~/.snap/data when they next refresh.

After data has been migrated to the new location, the SNAP_USER_DATA and SNAP_USER_COMMON environment variables are updated to the subdirectories of the new hidden location, ~/.snap/data/<snap_name>/<rev> and ~/.snap/data/<snap_name>/common, respectively.

Enabling hidden snap directory

To enable the experimental hidden snap directory feature, snapd must currently be installed from its ‘edge’ channel. You can check your current version with:

$ snap info snapd

The installed version (which you can see at the bottom of the output) should be the same as the version in the ‘latest/edge’ channel. If it’s not, you can install the latest snapd version from edge with:

$ sudo snap refresh --channel=latest/edge snapd

You can then enable the experimental hidden snap directory feature with:

$ sudo snap set system experimental.hidden-snap-folder=true

To disable the hidden snap directory feature you can run:

$ sudo snap unset system experimental.hidden-snap-folder

Note that disabling the flag does not currently move the data back to ~/snap. This must instead be done manually. The 2.55 snapd release will include the ability to automatically revert the migration with the next refresh by disabling the flag. With the snapd version currently on edge, however, if you wish to revert the snap directory, you must:

  1. disable the experimental.hidden-snap-folder flag
  2. manually backup any data under ~/.snap/data you want to keep
  3. remove the affected snaps
  4. restore their backed up data to ~/snap
  5. re-install the snaps

Planned Work

One of the problems with the current state of snap user data storage is that snaps map both the HOME and SNAP_USER_DATA environment variables to the versioned directory under ~/snap/<snap_name> or ~/.snap/data/<snap_name>.

This means that user-facing data written to $HOME will often be stored alongside application data, such as caches and configuration files, written to $SNAP_USER_DATA. Note that this isn’t universally true, since snaps can override environment variables through their snapcraft.yaml.

In order to address these shortcomings, we intend to introduce a new directory, ~/Snap (note the upper-case S), meant solely for user-facing data. This new directory will be pointed to by a new SNAP_USER_HOME environment variable as well as the HOME environment variable.

To further encourage snaps to separate user-facing from application data, some of the environment variables defined by the XDG Base Directory Specification will be mapped to directories under $SNAP_USER_DATA. In particular, the XDG_DATA_HOME, XDG_CONFIG_HOME and XDG_CACHE_HOME environment variables will be mapped to $SNAP_USER_DATA/data, $SNAP_USER_DATA/config and $SNAP_USER_DATA/cache, respectively.

Since snaps may override environment variables or use arbitrary locations to write application data, snap developers need to ensure that the snap application is writing to the correct location. To make this transition as easy as possible, we intend to couple the new ~/Snap directory migration with changes being made for the upcoming core22 base. This should hopefully minimise the effort required by snap developers as they’ll already be testing their snaps against the new base.

8 Likes
2

Hi, Your post states that if my snap users download the the latest snapd and set system experimental.hidden-snap-folder=true , app data will be mapped to a different snap oriented top-level directory in /HOME.

How will this work for those developers such as myself who have Users with significant user data, ( a database, thousands of files, GB of data) built up in the current /Home/snap/“myApp”/common directory, which for “myApp” contains three directories: the myApp data directory, a .cache directory and a .openjfx caching directory. It looks like I will need to supply a data migration tool with my app for existing users who make this switch.

Similarly, the “Planned Work” element of your post post implies that if I upgrade my snap to Core22 I will also need to supply a data migration routine for my exisiting users. This would be a right royal pain!

Have I misunderstood something here?

cheers Alan

3

Hi @apdw. Your app’s data, including ~/snap/<my_app>/common, will be migrated by snapd to the new location under ~/.snap/data/<my_app>. After the migration is done, the environment variables (including SNAP_USER_COMMON) will point to the new locations. To be clear the migration isn’t performed as soon as the user sets the experimental flag, it will be performed during the next refresh (when your snap isn’t running). If your app is using the environment variables as they are set by snapd, it shouldn’t have to do anything. If you map environment variables to custom values or something similar, then some changes might be necessary depending on the specifics. I hope that answers your question. Please let me know if can clarify anything or if I misunderstood your concern.

Thanks,

Miguel

4

This change sounds like a great idea. The visibility of this directory has been a point of friction for years: many users consider the visible files/directories in their home directory to be under their control, and run into problems when they move or delete files in this directory.

With that said, I do have a few concerns about this:

Apps expanding $SNAP_USER_COMMON/DATA in config files

I noticed the Chromium snap references the absolute path to its $SNAP_USER_COMMON in a number of its config/data files. Most seem to be log files, but the ~/snap/chromium/common/chromium/Local State file looks to be configuration and starts with:

{"autofill":{"states_data_dir":"/home/james/snap/chromium/common/chromium/AutofillStates/...

It doesn’t look like we did any special migration of this file when moving from a deb, so I suspect this will work. But it makes me wonder how many other snaps end up writing out absolute paths like this.

Migration performed outside of user context

This is basically the same issue as other home directory access performed on refresh: on some systems a user’s home directory will only be available while they are logged in. The main situations where this can occur are encrypted home directories and automounted NFS home directories.

These are both fairly niche use cases at the moment (e.g. I don’t think we prompt people to set up an ecryptfs home dir for a while), so it might be easier to say they are unsupported. But if systemd-homed takes off on some distro, that could change.

Has any thought been put into whether the migration could be completed at snap run time, if we find that it hasn’t happened for a particular user?

Bind mount shenanigans

I’m not sure how much of an issue this one is in practice for ~/snap, but there are a number of threads on this forum where we’ve suggested people create bind mounts when snaps are trying to place data on volumes that have run out of space (e.g. this one about LXD).

If ~/.snap is on a different file system to the files in ~/snap, then at best the migration is going to be quite slow as it copies things over, and at worst might fail after running out of space.

Again, it might be best to say these setups are unsupported. But it’d be nice if snapd doesn’t fall over if it encounters them.

3 Likes
5

Thanks for the feedback @jamesh, this is really great. I’ll try to address your concerns below:

It’s true that some snaps do things that won’t play nice with the migration (e.g., writing an expanded env var, using HOME and SNAP_USER_DATA interchangeably, etc). This is one of the reasons why this change is being tied to core22, so that snap developers testing their snaps for core22 can simultaneously test it with these changes and, if necessary, make some adjustments.

Regarding your second point, you’re right that those use cases pose an issue for the migration (which hasn’t been addressed yet). However, they were already an issue for a “normal” refresh, since we copy data from the previous revision’s directory to the new one, so I think at least we’re not excluding use cases that were previously possible. I could see this being an issue in other situations as well so we might have to think of a more general solution that can be reused. One thing to note is that this issue currently doesn’t prevent updates from proceeding, the unavailable home dir is just ignored (regardless of whether the migration occurs or not).

Regarding the bind mount, that’s also a good point which I think we haven’t considered yet. I agree, we should take care to address this even if it’s just by detecting it and failing as nicely as possible. Thank you for pointing this out, I’ll keep it in mind.

6

If you are intent to use new directory ~/Snap why not just use /.Snap instead of ./snap/data?

7

This worries me. People will read “snap data is not placed in ~/snap any more” and think that applies to all snap data. They will be very shocked when most of the snaps they use are still putting their data there because they’re not core22-based.

Why don’t we “just” (sorry for the trivialisation) move all data wholesale and create a mount-namespace-isolated bind-mount for each snap’s world view to put the ~/snap folder back again. That way the data can be found at either (i.e. at the same time) ~/snap or ~/.snap/data - this is the backwards compatible version that won’t break people yet still achieves the “remove the ~/snap folder from my eyes!” premise.

1 Like
8

Hiding ~/snap is not all we were asked to do. One of the issues that this change must address is to provide snaps with a HOME where user-facing data can be placed without exposing users to application data that isn’t relevant to them (like config and log files). Just moving ~/snap to ~/.snap/data doesn’t accomplish that, even if it would make everything much simpler.

9

The two directories will serve different purposes, ~/Snap is meant as a new $HOME for snaps where they can put user-facing data and ~/.snap/data is meant to be a place for app data that is not user-facing (e.g., logs, configs, etc).

1 Like
10

Then for core22:

  • ~/Snap for user-facing data
  • ~/.Snap for non user-facing data (instead of ~/.snap/data dir)

and everything related to older pre-core22:

  • ~/snap
  • ~/.snap

I am somehow used to some consistency in naming that reduces confusion what is older snap vs. new one, because we be living with both for years (until all snaps are recreated with core22, some may never be). Just a suggestion…

11

The reason why ~/.snap/data was chosen as a place to hide non user-facing app data was because ~/.snap already exists. It’s currently used to store user credentials. Reusing it avoids adding another top-level directory in users’ HOME directories.

because we be living with both for years (until all snaps are recreated with core22, some may never be)

I think you’re right, there will be a transition period where ~/snap and ~/Snap will have to coexist. However, I just want to underline that this can be mitigated by setting the ‘experimental.hidden-snap-folder’ flag. That will cause each snap’s directory under ~/snap to move to ~/.snap/data during the next refresh instead of when the snap moves to the core22 base. Then when the snap updates to the core22 base, the ~/Snap part of the migration will happen automatically and without being affected by the previous flag-triggered migration.

1 Like
12

Hi Miguel, and thanks for sharing the plan! I have to say that I’m concerned about the introduction of the ~/Snap/ folder, because the feedback we had so far is that user visible folder in the home directory are not very welcome. :slight_smile:

If our plan is to use it only for files that are relevant to the end user, then I don’t think we should remap the XDG_* variables to point at it, otherwise it will easily get filled with data not relevant to the user. I’d rather have the snaps explicitly use this folder if they support it — that way it’s more likely that the directory will remain mostly clean and its contents relevant to the user.

Ideally, when the snap itself does not use the XDG desktop portals, it would be nice to have a mechanism to lend it a partial view over the main user directories (Documents, Videos, Music) consisting only of those files that the snap itself has created. Whether bind mounts, a fuse FS, a combination of the two or even something else altogether need to be used, I don’t know, but it feels weird to solve a long standing problem and immediately introduce a new one. :slight_smile:

Maybe, until we get to the perfect solution, the ~/Snap/ folder can be just a symbolic link to some place under ~/.snap/, that the user can easily delete without losing any data in case he/she doesn’t want to see it?

3 Likes
13

@mardy I could not agree more about the decision to introduce the ~/Snap directory. Isn’t the main complaint over the years that this bug has existed is that users don’t want un-moveable clutter in their home directories? I do not understand why Canonical would refuse to listen their users on this point and instead add more to the home directory

1 Like
14

Hi Alberto. I understand your point and I even agree, personally. However, the requirements of what we were asked to do are not just “hide ~/snap”, they include having a visible place where snaps can place user-facing data that is separate from app data and other things that aren’t relatable to the user (e.g., the ‘common’ or revision directories). The current plan was made with those requirements and the challenges of migrating without breaking snaps in mind. Personally, I would also prefer to simply hide ~/snap into ~/.snap/data, so I can’t argue against it, but I can recognise that it’s just one opinion among many and that we can’t decide what to build unilaterally.

That’s not what is described in the main post. The XDG_* variables will point to directories under ~/.snap/data/<snap_name>/<revision>/.

One issue is that changing what HOME is mapped to can break snaps in unpredictable ways (e.g., if it uses HOME and SNAP_USER_DATA interchangeably) so tying this change to an upgrade to a new core is currently the best way we have to mitigate that. Since developers will already be testing their snaps with the new core, they’ll also be testing it with this migration. Having an interim solution like the one you mentioned and taking more time to discuss this pushes this problem to a later date where we won’t have an upcoming core upgrade to tie this change to.

15

Apart from introducing yet another folder in $HOME, i.e. $HOME/snap is separate to $HOME/Snap, and the inevitable backlash you’re going to receive for such a user-hostile move that has already had the most prolific complaints for just $HOME/snap. What benefits are there of “not simply hiding” $HOME/snap in $HOME/.snap/data? If the snap author wants to allow users to save files then that is what the home plug and removable-media plug are for, and they work just fine for everyone. So again, what benefit is there to “having a visible place where snaps can place user-facing data that is separate from app data” that cannot be accomplished by simply moving ~/snap to ~/.snap/data and introducing home and removable-media plugs?

1 Like
16

Also, it’s worth noting that not every application will respect XDG_* variables, and will place configuration data in what is pointed to by $HOME in their runtime space. If this change will make $HOME point to $HOME/Snap for the snap then they’ll be placing their configuration in the folder that you’ve explicitly said is not for that!

1 Like
17

they do not work on Ubuntu Core where home is never auto-connected, they will not work in enterprise env’s where an admin decided that (certain) snaps should not have access to the users general data and disconnects the home plug explicitly … in such cases $SNAP_USER_DATA is still needed for the snaps to function properly …

18

And how is $SNAP_USER_DATA or $SNAP_USER_COMMON improved by this change? You’re simply rearranging deckchairs.

19

i’m not talking in support of that move here, just pointing out a flaw in the argument :slight_smile:

20

a badly articulated argument is just as valid as a cited research thesis. ;-p

1 Like