So 13 GiB and 78% full in / is more than I had expected but 9.4 GiB and 99% full in /var is quite unexpected.
/dev/sda5 is swap (1.0 GiB, with over 60 GB of ram I didn’t think I would need a ‘standard’ sized swap).
So it looks like snapd + lxd is somehow hogging a lot of space which I can’t really account for with snap somehow being at the root of the issue. I’m running ‘core’ and presently 10 containers only one of which I have been trying to load software onto so this space usage seems quite ‘out there’.
Reading through your suggestion it seems that the proposal actually removes a container.
As I don’t really want to do that I’m not sure how to use the idea. Seems to be connected but is there a way to say ‘de-frag the drive’ in snap?
That might fix the issue.
I need to fix this issue so that I can actually install software on the containers so that I can use those same containers. Hopefully soon!!
Well, what made you partition your HD in such a fragmented manner ?
i guess your only proper solution would be to re-size /var or to do a re-install with proper partitioning (the values picked for these gazillion of /dev/sda* partitions look rather randomly chosen (/var as the partition for “variable application data” (caches, all actual app data…) being so small in your setup is simply a problem, while you picked twice the space for /usr which typically only holds the application binaries and does typically not change much in size))
If you have any unallocated space on your disk you could also try to partition it for /var/snap and mount it there, but that will eventually make you hit that limitation again, the lxd snap does install its containers in the lxd snaps home which is /var/snap/lxd (using the lxd deb instead would likely also not help since it will install its payload data in /var/lib/lxd i think).
Could you paste the output of du -ah /var/snap | sort -h? The easiest way is probably to pipe the output of that to pastebinit, or to xsel -i -b and then Ctrl-V that in here.
I am not super familiar with the internal structure of lxd, but it seem like a very big storage device was added (existing containers on my system show up under /var/snap/lxd/common/lxd/containers).
@stgraber might be able to provide a more concrete answer to what that lxd2.img is.
It looks like you have a storage pool called lxd2 which uses loop-backed storage.
This creates a sparse file under /var/snap/lxd/common/lxd/disks which can grow up to the size you’ve provided at the time this was created (or if no size was provided, up to a reasonable amount of space determined at creation time).
lxc storage show lxd2 should give some more details on that storage pool.
Hmmmmmmmmmmmm - - - these sparse files are then problematic.
My suggestion is that something needs to be changed so that a minimal content sparse file does not ‘lock up’ a directory. This means one of two things - - - either the sparse files also trigger space ‘claimed’ (not necessarily taken but still claimed) or that the space ‘claimed’ also registers with ‘all’ file manipulation and checking tools both cli and gui.
I was able to affect a fix to the issue but in the process have found a documentation lacuna.
Solution first.
(At a mentor’s (an old time unix geek) suggestion!)
I moved /var/snap to a directory with lots of space and then set up a sym-link between /var/snap and the new location. Process was as follows (for anyone else needing the solution).
First make sure that ALL of your containers have been stopped.
Copy /var/snap to another directory that has lots (LOTS!!!) of space in it.
Then you need to $ rm -r the original (/var/snap).
Lastly you need to create a hard link between /var/snap to /mynewdirectory/snap - - - I used $ ln -s /mynewdirectory/snap /var/snap .
I now did a reboot using shutdown -r now to clear any caches and to set things (as advised by my mentor).
(After the reboot completes.) You now need to start all your containers.
Should be working (at least in my case everything is working - - - grin!).
This issue is caused, at least in part, by the assumption that the system is installed with no partitions. There is no mention of where the ‘snaps’ will be stored and any space requirements that could be needed by those secondary, to snapd, tools. Therefore there should be something in the documentation that informs the user that this possible level of space requirement will be there or there should be direction that clearly states that snapd should only be used in a system install that doesn’t use partitions.
Useful would be having the sparse files ‘claimed space’ (not necessarily the ‘used’ space (which is vastly smaller)) visible and also reflected in file manager tool responses. It may be necessary to have two responses, one for ‘used’ space and another for ‘claimed’ space - - this is likely far more problematic for implementation.
well, rather that the system is installed with properly picked sizes for partitions you created, this is not snap specific at all … if you install a database server (or mail or webserver) on your system and store 10GB data in the DB it will also end up in /var due to the FHS . /var is typically the most growing directory on a system next to /home and should be sized accordingly, this is not a snap specific demand at all, snapd just follows the FHS here.
As you @ogra are insisting that a database server is going to also be installed in /var I thought I would check. You are actually incorrect - - - a database is installed in both /usr AND /var where snapd only has a single file in /usr seeming to rely on /var as its major repository.
Secondly, when I ask my system to tell me what space is being used - - - those database servers or whatever else actually show what space is being used. On the other hand snapd doesn’t show up in all of the tools because snapd is a ‘claimed’ space not a ‘used’ space.
Thirdly, I have seen suggestions in the documentation of at least some of these database servers that it may be necessary to provision space for large files - - - there is no mention of anything like that that I have been able to find for snapd. In fact a dev team leader that uses snapd as a tool had this for a comment on my question as to what was happening:
"That’s certainly a weird behavior… It doesn’t really make sense that du thinks the directory is using 6GB but when listing its content it goes down to a few kBs…
Could it be du not accounting for some hidden directory or something in this case?"
So perhaps I’m not the only one that finds this behavior ‘unusual’.
As to ‘properly picked sizes for partitions’ - - - no guidance is even hinted at nor are there any suggestions. So instead of barking out this kind of response ('proper picked sizes ’ just previously and “Well, what made you partition your HD in such a fragmented manner ?” earlier) you might want to actually note in the documentation that space could be an issue and give guidance as to possible solutions. That would be useful.
Binaries for a database (or even web-) server application usually go to /usr/bin or /usr/sbin, libraries such a server ships along with its packages normally go to /usr/lib… all variable data for such applications typically go to /var (check any webservers on debian based systems, their payload goes to /var/www for example, postgres or mysql created their databases in /var/lib/<database> …) which you typically pick big enough according to these standards. how did you verify that the database server you installed did not use /var when you injected gigabytes of database data into it ?
If you install snapd and do not isntall any data (snaps) it will also only occupy space in /usr with the few binaries the package ships,.Only if you make it utilize any snap data (i,e. by installing a snap) it will occupy space in /var … pretty much like any other server application that follows the FHS, this is a standard since decades and snapd simply follows it like any other server application … feel free to read in more detail about the FHS at:
Alternatively your server could utilize /srv instead of /var, but since this is a relatively new addition to the FHS not all distros may support it which is why snapd uses the more common /var.
Applications can indeed use an arbitrary amount of space, depending only on their purpose and behavior. This is unrelated to snaps or any other packaging technology. We can and eventually will implement constraining of space used in various ways, but if you use that feature, you’d just get the exact same problem earlier, with the application running out of space.
OK - - so getting other tools to understand ‘claimed’ space is going to be problematic - - fair enough.
Even so - - - guidance that any use of the package is going to cause ‘claimed’ space and where that space is is something that could and should be in the documentation rather than leaving it for the non-specialist to find the hard way.
well, one could argue that fragmented partitioning is a highly specialist task (… which it is after all, you usually only do such partitinioning for a very special purpose and it typically requires some experience to not actually run out of space on the various partitions… i.e. my mom wouldnt do it to run a desktop (and there would be no valid reason for her to do so))
Again, this is not specific to snaps or any packaging technology. Almost every application that you install will cause some “claimed” space, and in Linux that claimed space usually goes into /var if it’s a system-level application, or your home otherwise.
With that said, you have a good point @dabeegmon. We could definitely be helpful in providing convenient means of listing disk space used, and even warning when particular snaps are consuming relevant volumes of disk. Marking this thread as backlog so we can look into this when we have a moment.
