Incrementing loop device names breaking datadog monitor


#1

Metric queries cannot filter out future device names and all /dev/loop# devices are mounted 100% full RO devices.


#2

What exactly is the issue here?
Snaps are implemented using squashfs files which are then mounted using loopback devices. I don’t know of a way to control or somehow influence which snap gets which loopback device in /dev/loop* as it depends on what loopback devices have already been mounted.


#3

Datadog metrics are based on known tags so until a /dev/loop* device has been seen it can’t be excluded from the monitor. That means whenever a new loopback device gets mounted with a new number at the end, it triggers the monitor that alerts on 100% volume usage. More on datadog metrics here: https://docs.datadoghq.com/graphing/faq/when-i-query-can-i-use-wildcards-in-metric-names-and-events/

CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it, may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED.


#4

Getting a warning for volume usage on something that is obviously readonly sounds like a bug, to me.


#5

Yeah I agree with @chipaca that this is an issue with the datadog metrics application and there’s not a problem with snapd here.
I think that there’s some special option added to the mount options used with snap files where the snap mount has x-gdu.hide in the options. Perhaps datadog could also learn to ignore mounts with that option as well


#6

Just querying the loop device (like what losetup does, as a regular user) would be enough to figure out you shouldn’t be alerting about it.


#7

Thanks for the replies. I didn’t see any obvious way to adjust this on the datadog side. Why do the loop devices need to have a changing number at the end?

CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it, may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this message is STRICTLY PROHIBITED.


#8

because they grow dynamically, as they are needed?