Applications inherently depend on disk space

backlog

#1

Running debian containers (mainly) on snapd running on debian stretch (stable or also 9).

I ran the command
$ apt update
and I was told that there was insufficient memory to complete the task (there were a few lines before that).

With some assistance from a friend I’m been trying to find where the space hog is

root@debianserver:/# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
udev 74254448 0 74254448 0% /dev
tmpfs 14853476 255724 14597752 2% /run
/dev/sda2 18011420 13298284 3775152 78% /
/dev/sda5 20027216 5944576 13042256 32% /usr
tmpfs 74267372 24696 74242676 1% /dev/shm
tmpfs 5120 0 5120 0% /run/lock
tmpfs 74267372 0 74267372 0% /sys/fs/cgroup
/dev/sda7 3023760 5044 2865116 1% /tmp
/dev/sda6 15053152 41868 14226900 1% /usr/local
/dev/sda4 10013512 9353460 131668 99% /var
/dev/sda8 5759082496 17018792 5740155304 1% /home
/dev/sda9 510984 144 510840 1% /boot/efi
tmpfs 14853472 8 14853464 1% /run/user/1000
tmpfs 14853472 4 14853468 1% /run/user/114
/dev/loop3 209715200 4530288 203402800 3% /media/darald/lxd2
/dev/loop12 48256 48256 0 100% /snap/lxd/5785
/dev/loop14 48256 48256 0 100% /snap/lxd/5841
/dev/loop13 48256 48256 0 100% /snap/lxd/5866
/dev/loop19 83840 83840 0 100% /snap/core/4189
/dev/loop18 83840 83840 0 100% /snap/core/4194
/dev/loop16 83840 83840 0 100% /snap/core/4201

So 13 GiB and 78% full in / is more than I had expected but 9.4 GiB and 99% full in /var is quite unexpected.
/dev/sda5 is swap (1.0 GiB, with over 60 GB of ram I didn’t think I would need a ‘standard’ sized swap).

root@debianserver:/var# du -d 1 -BM /var
12M /var/tmp
1M /var/www
7M /var/mail
13M /var/backups
1M /var/opt
6205M /var/snap
1M /var/local
131M /var/log
1M /var/lost+found
656M /var/cache
1409M /var/lib
4M /var/spool
8434M /var

yet:
/var/snap properties
39 files
76.3 KiB (file size)
208.0 KiB (size on disk)

/var/snap/lxd
32 files
76.3 KiB
184.0 KiB
and
/var/snap/core
6 files
4 bytes
20.0 KiB

There’s some 6.20 GiB not accounted for!

Not such a large discrepancy but:

root@debianserver:/var# du -d 1 -BM /var/cache
1M /var/cache/cracklib
1M /var/cache/lightdm
308M /var/cache/apt
3M /var/cache/man
1M /var/cache/system-tools-backends
330M /var/cache/lxc
1M /var/cache/bind
8M /var/cache/debconf
1M /var/cache/dbconfig-common
1M /var/cache/snapd
1M /var/cache/libvirt
5M /var/cache/locate
1M /var/cache/dictionaries-common
1M /var/cache/cups
1M /var/cache/ldconfig
1M /var/cache/localepurge
1M /var/cache/PackageKit
1M /var/cache/tcpdf
1M /var/cache/awstats
1M /var/cache/samba
1M /var/cache/postgresql
2M /var/cache/fontconfig
1M /var/cache/lxd
1M /var/cache/apparmor
1M /var/cache/apache2
1M /var/cache/apt-cacher-ng
656M /var/cache

So it looks like snapd + lxd is somehow hogging a lot of space which I can’t really account for with snap somehow being at the root of the issue. I’m running ‘core’ and presently 10 containers only one of which I have been trying to load software onto so this space usage seems quite ‘out there’.

Please advise.

TIA


#2

could be this:


#3

Reading through your suggestion it seems that the proposal actually removes a container.
As I don’t really want to do that I’m not sure how to use the idea. Seems to be connected but is there a way to say ‘de-frag the drive’ in snap?
That might fix the issue.
I need to fix this issue so that I can actually install software on the containers so that I can use those same containers. Hopefully soon!!


#4

Well, what made you partition your HD in such a fragmented manner ?

i guess your only proper solution would be to re-size /var or to do a re-install with proper partitioning (the values picked for these gazillion of /dev/sda* partitions look rather randomly chosen (/var as the partition for “variable application data” (caches, all actual app data…) being so small in your setup is simply a problem, while you picked twice the space for /usr which typically only holds the application binaries and does typically not change much in size))

If you have any unallocated space on your disk you could also try to partition it for /var/snap and mount it there, but that will eventually make you hit that limitation again, the lxd snap does install its containers in the lxd snaps home which is /var/snap/lxd (using the lxd deb instead would likely also not help since it will install its payload data in /var/lib/lxd i think).


#5

Could you paste the output of du -ah /var/snap | sort -h? The easiest way is probably to pipe the output of that to pastebinit, or to xsel -i -b and then Ctrl-V that in here.


#6

root@debianserver:/var# du -ah /var/snap | sort -h
0 /var/snap/core/current
0 /var/snap/lxd/common/lxd/unix.socket
0 /var/snap/lxd/common/mntns
0 /var/snap/lxd/current
4.0K /var/snap/core/4201
4.0K /var/snap/core/4213
4.0K /var/snap/core/4217
4.0K /var/snap/core/common
4.0K /var/snap/lxd/5785
4.0K /var/snap/lxd/5841
4.0K /var/snap/lxd/5866
4.0K /var/snap/lxd/common/config
4.0K /var/snap/lxd/common/lxcfs.pid
4.0K /var/snap/lxd/common/lxc/local.conf
4.0K /var/snap/lxd/common/lxd/containers/my-debian
4.0K /var/snap/lxd/common/lxd/containers/my-debian-calendar
4.0K /var/snap/lxd/common/lxd/containers/my-debian-horde-server
4.0K /var/snap/lxd/common/lxd/containers/my-debian-internal-business-server
4.0K /var/snap/lxd/common/lxd/containers/my-debian-postgresql
4.0K /var/snap/lxd/common/lxd/containers/my-debian-recordkeeping
4.0K /var/snap/lxd/common/lxd/containers/my-debian-sid
4.0K /var/snap/lxd/common/lxd/containers/my-debian-testing
4.0K /var/snap/lxd/common/lxd/containers/my-debian-testing-gui
4.0K /var/snap/lxd/common/lxd/containers/my-ubuntu-trusty
4.0K /var/snap/lxd/common/lxd/devices/my-debian
4.0K /var/snap/lxd/common/lxd/devices/my-debian-calendar
4.0K /var/snap/lxd/common/lxd/devices/my-debian-horde-server
4.0K /var/snap/lxd/common/lxd/devices/my-debian-internal-business-server
4.0K /var/snap/lxd/common/lxd/devices/my-debian-postgresql
4.0K /var/snap/lxd/common/lxd/devices/my-debian-recordkeeping
4.0K /var/snap/lxd/common/lxd/devices/my-debian-sid
4.0K /var/snap/lxd/common/lxd/devices/my-debian-testing
4.0K /var/snap/lxd/common/lxd/devices/my-debian-testing-gui
4.0K /var/snap/lxd/common/lxd/devices/my-ubuntu-trusty
4.0K /var/snap/lxd/common/lxd/devlxd
4.0K /var/snap/lxd/common/lxd/images
4.0K /var/snap/lxd/common/lxd/logs/my-debian-calendar/lxc.conf
4.0K /var/snap/lxd/common/lxd/logs/my-debian-horde-server/lxc.conf
4.0K /var/snap/lxd/common/lxd/logs/my-debian-internal-business-server/lxc.conf
4.0K /var/snap/lxd/common/lxd/logs/my-debian/lxc.conf
4.0K /var/snap/lxd/common/lxd/logs/my-debian-postgresql/lxc.conf
4.0K /var/snap/lxd/common/lxd/logs/my-debian-recordkeeping/lxc.conf
4.0K /var/snap/lxd/common/lxd/logs/my-debian-sid/lxc.conf
4.0K /var/snap/lxd/common/lxd/logs/my-debian-testing-gui/lxc.conf
4.0K /var/snap/lxd/common/lxd/logs/my-debian-testing/lxc.conf
4.0K /var/snap/lxd/common/lxd/logs/my-ubuntu-trusty/lxc.conf
4.0K /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/my-debian
4.0K /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/my-debian-calendar
4.0K /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/my-debian-horde-server
4.0K /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/my-debian-internal-business-server
4.0K /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/my-debian-postgresql
4.0K /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/my-debian-recordkeeping
4.0K /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/my-debian-sid
4.0K /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/my-debian-testing
4.0K /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/my-debian-testing-gui
4.0K /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/my-ubuntu-trusty
4.0K /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases
4.0K /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.pid
4.0K /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.raw
4.0K /var/snap/lxd/common/lxd.pid
4.0K /var/snap/lxd/common/lxd/security/apparmor/cache/.features
4.0K /var/snap/lxd/common/lxd/security/apparmor/cache/lxd-my-debian-postgresql
4.0K /var/snap/lxd/common/lxd/security/apparmor/cache/lxd-my-debian-recordkeeping
4.0K /var/snap/lxd/common/lxd/security/apparmor/cache/lxd-my-debian-sid
4.0K /var/snap/lxd/common/lxd/security/apparmor/cache/lxd-my-debian-testing
4.0K /var/snap/lxd/common/lxd/security/apparmor/cache/lxd-my-debian-testing-gui
4.0K /var/snap/lxd/common/lxd/security/apparmor/cache/lxd-my-ubuntu-trusty
4.0K /var/snap/lxd/common/lxd/security/seccomp/my-debian
4.0K /var/snap/lxd/common/lxd/security/seccomp/my-debian-calendar
4.0K /var/snap/lxd/common/lxd/security/seccomp/my-debian-horde-server
4.0K /var/snap/lxd/common/lxd/security/seccomp/my-debian-internal-business-server
4.0K /var/snap/lxd/common/lxd/security/seccomp/my-debian-postgresql
4.0K /var/snap/lxd/common/lxd/security/seccomp/my-debian-recordkeeping
4.0K /var/snap/lxd/common/lxd/security/seccomp/my-debian-sid
4.0K /var/snap/lxd/common/lxd/security/seccomp/my-debian-testing
4.0K /var/snap/lxd/common/lxd/security/seccomp/my-debian-testing-gui
4.0K /var/snap/lxd/common/lxd/security/seccomp/my-ubuntu-trusty
4.0K /var/snap/lxd/common/lxd/server.crt
4.0K /var/snap/lxd/common/lxd/server.key
4.0K /var/snap/lxd/common/lxd/shmounts
4.0K /var/snap/lxd/common/lxd/snapshots
4.0K /var/snap/lxd/common/lxd/storage-pools/lxd2
4.0K /var/snap/lxd/common/var/lib/lxcfs
8.0K /var/snap/lxd/common/lxc
8.0K /var/snap/lxd/common/lxd/cache/instance_types.yaml
8.0K /var/snap/lxd/common/lxd/logs/my-debian
8.0K /var/snap/lxd/common/lxd/logs/my-debian-calendar
8.0K /var/snap/lxd/common/lxd/logs/my-debian-horde-server
8.0K /var/snap/lxd/common/lxd/logs/my-debian-internal-business-server
8.0K /var/snap/lxd/common/lxd/logs/my-debian-postgresql
8.0K /var/snap/lxd/common/lxd/logs/my-debian-recordkeeping
8.0K /var/snap/lxd/common/lxd/logs/my-debian-sid
8.0K /var/snap/lxd/common/lxd/logs/my-debian-testing
8.0K /var/snap/lxd/common/lxd/logs/my-debian-testing-gui
8.0K /var/snap/lxd/common/lxd/logs/my-ubuntu-trusty
8.0K /var/snap/lxd/common/lxd/security/apparmor/cache/lxd-my-debian
8.0K /var/snap/lxd/common/lxd/security/apparmor/cache/lxd-my-debian-calendar
8.0K /var/snap/lxd/common/lxd/security/apparmor/cache/lxd-my-debian-horde-server
8.0K /var/snap/lxd/common/lxd/security/apparmor/cache/lxd-my-debian-internal-business-server
8.0K /var/snap/lxd/common/lxd/storage-pools
8.0K /var/snap/lxd/common/var/lib
12K /var/snap/lxd/common/lxd/security/apparmor/profiles/lxd-my-debian
12K /var/snap/lxd/common/lxd/security/apparmor/profiles/lxd-my-debian-calendar
12K /var/snap/lxd/common/lxd/security/apparmor/profiles/lxd-my-debian-horde-server
12K /var/snap/lxd/common/lxd/security/apparmor/profiles/lxd-my-debian-internal-business-server
12K /var/snap/lxd/common/lxd/security/apparmor/profiles/lxd-my-debian-postgresql
12K /var/snap/lxd/common/lxd/security/apparmor/profiles/lxd-my-debian-recordkeeping
12K /var/snap/lxd/common/lxd/security/apparmor/profiles/lxd-my-debian-sid
12K /var/snap/lxd/common/lxd/security/apparmor/profiles/lxd-my-debian-testing
12K /var/snap/lxd/common/lxd/security/apparmor/profiles/lxd-my-debian-testing-gui
12K /var/snap/lxd/common/lxd/security/apparmor/profiles/lxd-my-ubuntu-trusty
12K /var/snap/lxd/common/var
20K /var/snap/core
44K /var/snap/lxd/common/lxd/containers
44K /var/snap/lxd/common/lxd/devices
44K /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts
44K /var/snap/lxd/common/lxd/security/seccomp
60K /var/snap/lxd/common/lxd/networks/lxdbr0
64K /var/snap/lxd/common/lxd/networks
64K /var/snap/lxd/common/lxd/security/apparmor/cache
68K /var/snap/lxd/common/lxd/cache/simplestreams.yaml
72K /var/snap/lxd/common/lxd/logs/lxd.log
72K /var/snap/lxd/common/lxd/lxd.db
80K /var/snap/lxd/common/lxd/cache
124K /var/snap/lxd/common/lxd/security/apparmor/profiles
156K /var/snap/lxd/common/lxd/logs
192K /var/snap/lxd/common/lxd/security/apparmor
240K /var/snap/lxd/common/lxd/security
6.1G /var/snap
6.1G /var/snap/lxd
6.1G /var/snap/lxd/common
6.1G /var/snap/lxd/common/lxd
6.1G /var/snap/lxd/common/lxd/disks
6.1G /var/snap/lxd/common/lxd/disks/lxd2.img


#7

I am not super familiar with the internal structure of lxd, but it seem like a very big storage device was added (existing containers on my system show up under /var/snap/lxd/common/lxd/containers).

@stgraber might be able to provide a more concrete answer to what that lxd2.img is.


#8

It looks like you have a storage pool called lxd2 which uses loop-backed storage.
This creates a sparse file under /var/snap/lxd/common/lxd/disks which can grow up to the size you’ve provided at the time this was created (or if no size was provided, up to a reasonable amount of space determined at creation time).

lxc storage show lxd2 should give some more details on that storage pool.


#9

memyself@debianserver:~$ lxc storage show lxd2
config:
size: 200GB
source: /var/snap/lxd/common/lxd/disks/lxd2.img
description: ""
name: lxd2
driver: btrfs
used_by:

  • /1.0/containers/my-debian
  • /1.0/containers/my-debian-calendar
  • /1.0/containers/my-debian-horde-server
  • /1.0/containers/my-debian-internal-business-server
  • /1.0/containers/my-debian-postgresql
  • /1.0/containers/my-debian-recordkeeping
  • /1.0/containers/my-debian-sid
  • /1.0/containers/my-debian-testing
  • /1.0/containers/my-debian-testing-gui
  • /1.0/containers/my-ubuntu-trusty
  • /1.0/profiles/default

Hmmmmmmmmmmmm - - - these sparse files are then problematic.

My suggestion is that something needs to be changed so that a minimal content sparse file does not ‘lock up’ a directory. This means one of two things - - - either the sparse files also trigger space ‘claimed’ (not necessarily taken but still claimed) or that the space ‘claimed’ also registers with ‘all’ file manipulation and checking tools both cli and gui.


#10

I was able to affect a fix to the issue but in the process have found a documentation lacuna.
Solution first.

(At a mentor’s (an old time unix geek) suggestion!)
I moved /var/snap to a directory with lots of space and then set up a sym-link between /var/snap and the new location. Process was as follows (for anyone else needing the solution).
First make sure that ALL of your containers have been stopped.
Copy /var/snap to another directory that has lots (LOTS!!!) of space in it.
Then you need to $ rm -r the original (/var/snap).
Lastly you need to create a hard link between /var/snap to /mynewdirectory/snap - - - I used $ ln -s /mynewdirectory/snap /var/snap .
I now did a reboot using shutdown -r now to clear any caches and to set things (as advised by my mentor).
(After the reboot completes.) You now need to start all your containers.

Should be working (at least in my case everything is working - - - grin!).

This issue is caused, at least in part, by the assumption that the system is installed with no partitions. There is no mention of where the ‘snaps’ will be stored and any space requirements that could be needed by those secondary, to snapd, tools. Therefore there should be something in the documentation that informs the user that this possible level of space requirement will be there or there should be direction that clearly states that snapd should only be used in a system install that doesn’t use partitions.
Useful would be having the sparse files ‘claimed space’ (not necessarily the ‘used’ space (which is vastly smaller)) visible and also reflected in file manager tool responses. It may be necessary to have two responses, one for ‘used’ space and another for ‘claimed’ space - - this is likely far more problematic for implementation.


#11

well, rather that the system is installed with properly picked sizes for partitions you created, this is not snap specific at all … if you install a database server (or mail or webserver) on your system and store 10GB data in the DB it will also end up in /var due to the FHS . /var is typically the most growing directory on a system next to /home and should be sized accordingly, this is not a snap specific demand at all, snapd just follows the FHS here.


#12

As you @ogra are insisting that a database server is going to also be installed in /var I thought I would check. You are actually incorrect - - - a database is installed in both /usr AND /var where snapd only has a single file in /usr seeming to rely on /var as its major repository.
Secondly, when I ask my system to tell me what space is being used - - - those database servers or whatever else actually show what space is being used. On the other hand snapd doesn’t show up in all of the tools because snapd is a ‘claimed’ space not a ‘used’ space.
Thirdly, I have seen suggestions in the documentation of at least some of these database servers that it may be necessary to provision space for large files - - - there is no mention of anything like that that I have been able to find for snapd. In fact a dev team leader that uses snapd as a tool had this for a comment on my question as to what was happening:
"That’s certainly a weird behavior… It doesn’t really make sense that du thinks the directory is using 6GB but when listing its content it goes down to a few kBs…

Could it be du not accounting for some hidden directory or something in this case?"

So perhaps I’m not the only one that finds this behavior ‘unusual’.

As to ‘properly picked sizes for partitions’ - - - no guidance is even hinted at nor are there any suggestions. So instead of barking out this kind of response ('proper picked sizes ’ just previously and “Well, what made you partition your HD in such a fragmented manner ?” earlier) you might want to actually note in the documentation that space could be an issue and give guidance as to possible solutions. That would be useful.


#13

Binaries for a database (or even web-) server application usually go to /usr/bin or /usr/sbin, libraries such a server ships along with its packages normally go to /usr/lib… all variable data for such applications typically go to /var (check any webservers on debian based systems, their payload goes to /var/www for example, postgres or mysql created their databases in /var/lib/<database> …) which you typically pick big enough according to these standards. how did you verify that the database server you installed did not use /var when you injected gigabytes of database data into it ?

If you install snapd and do not isntall any data (snaps) it will also only occupy space in /usr with the few binaries the package ships,.Only if you make it utilize any snap data (i,e. by installing a snap) it will occupy space in /var … pretty much like any other server application that follows the FHS, this is a standard since decades and snapd simply follows it like any other server application … feel free to read in more detail about the FHS at:

Alternatively your server could utilize /srv instead of /var, but since this is a relatively new addition to the FHS not all distros may support it which is why snapd uses the more common /var.


#14

Applications can indeed use an arbitrary amount of space, depending only on their purpose and behavior. This is unrelated to snaps or any other packaging technology. We can and eventually will implement constraining of space used in various ways, but if you use that feature, you’d just get the exact same problem earlier, with the application running out of space.


#15

OK - - so getting other tools to understand ‘claimed’ space is going to be problematic - - fair enough.
Even so - - - guidance that any use of the package is going to cause ‘claimed’ space and where that space is is something that could and should be in the documentation rather than leaving it for the non-specialist to find the hard way.


#16

well, one could argue that fragmented partitioning is a highly specialist task (… which it is after all, you usually only do such partitinioning for a very special purpose and it typically requires some experience to not actually run out of space on the various partitions… i.e. my mom wouldnt do it to run a desktop (and there would be no valid reason for her to do so))

:wink:


#17

Again, this is not specific to snaps or any packaging technology. Almost every application that you install will cause some “claimed” space, and in Linux that claimed space usually goes into /var if it’s a system-level application, or your home otherwise.

With that said, you have a good point @dabeegmon. We could definitely be helpful in providing convenient means of listing disk space used, and even warning when particular snaps are consuming relevant volumes of disk. Marking this thread as backlog so we can look into this when we have a moment.


#18

Thank you for understanding and for your assistance.