This situation is different from the other Snap refresh posts in the #snapd category.
It is not working, while most of the system has connectivity.
# snap refresh
error: cannot refresh: cannot refresh snap-declaration for "core": Get
https://api.snapcraft.io/api/v1/snaps/assertions/snap-declaration/16/99T7MUlRhtI3U0QFgl5mXXESAiSwt776?max-format=3:
net/http: request canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)
There is a Launchpad issue #1639981 which describes a similar situation.
https://bugs.launchpad.net/snappy/+bug/1639981
The journal outputs merely the same message
# journalctl --no-pager -n 1 -u snapd
-- Logs begin at Tue 2018-05-22 16:58:55 CEST, end at Mon 2019-02-18 21:24:08 CET. --
Feb 18 21:17:18 pons snapd[846]: stateengine.go:102: state ensure error: cannot refresh snap-declaration for "core": Get https://api.snapcraft.io/api/v1/snaps/assertions/snap-declaration/16/99T7MUlRhtI3U0QFgl5mXXESAiSwt776?max-format=3: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
while the resource is accessible to other processes.
# curl -I 'https://api.snapcraft.io/api/v1/snaps/assertions/snap-declaration/16/99T7MUlRhtI3U0QFgl5mXXESAiSwt776?max-format=3'
HTTP/1.1 200 OK
Server: gunicorn/19.7.1
Date: Mon, 18 Feb 2019 17:57:06 GMT
Content-Type: application/x.ubuntu.assertion
Content-Length: 986
Snap-Store-Version: 11
X-View-Name: snapdevicegw.webapi_assertions.assertions_find
X-VCS-Revision: 9cd1a3f
X-Request-Id: 7DD4D03ACB830A325C7F01BB5C6AF1F1106C537
Age: 75
X-Cache: HIT from juju-7794b8-prod-ols-snap-store-indep-1375
X-Cache-Lookup: HIT from juju-7794b8-prod-ols-snap-store-indep-1375:3128
Via: 1.1 juju-7794b8-prod-ols-snap-store-indep-1375 (squid/3.5.27)
# curl 'https://api.snapcraft.io/api/v1/snaps/assertions/snap-declaration/16/99T7MUlRhtI3U0QFgl5mXXESAiSwt776?max-format=3'
type: snap-declaration
authority-id: canonical
series: 16
snap-id: 99T7MUlRhtI3U0QFgl5mXXESAiSwt776
publisher-id: canonical
snap-name: core
timestamp: 2016-09-28T17:33:53.740774Z
sign-key-sha3-384: BWDEoaqyr25nF5SNCvEv2v7QnM9QsfCc0PBMYD_i2NGSQ32EF2d4D0hqUel3m8ul
AcLBUgQAAQoABgUCV+v/AQAA2tsQAFHkr0inUpkgz3HpLoRfZ47iiRg76DniS/LQhxufoOeALzmB
TaOU91po50zhIvU8gYAUaFOvW948xGm6I+7HllGXpb5HswNMPdAhjMOuKJON0De3jvwuYoFcDncY
AHM9EVfl2QdWSlVMD09mBEoVVI9jRCSoHBtqLjDwk7woJ+VEGjiL6X1XRuelV8IYKAVR6RywoyTd
hlPGMk/qmIjzgvMBw3Bh/bS+15P4Iv0ylCllLbizlsVd7x/W1S5v4BMo5zrWClg96aZ0HrWnbxqC
yctpWh8XsWXViqhF5hvlzIszaXfqtSLRVXrGXGxEu3SmJYcae+CLan0jZDU7G7Qn+z5JhqxEuAph
K1TW8hDHjjDNhNO47dpMEp4WfNvhC0gu0l1jhpo/ylp4vIW6Wb2SOFxkz+rCjh9OiV+1zuIzrv/X
BROWHSDEQb9XwzkoLeroUUVsLqBuB0sVdreqp9VPdic6z07uMLVTQzJs/Od4zdnBlIxF1hVsumxZ
VJeJ63toKOJ7rhzB7tz1dwSLJ+iiX2o3evbc/5LP5hMZfyH5QIh9X7EtEIYc8PT5+nOLyUMmUJSY
m9c8a7+qltq3FjT+OiTsONpKedKr/rnV1Nppf9g+VVGy6kL+Z0CiI8FFOsGm1G84CLPQ9sLbuBnr
pl/dxHMTsng80JoGcppES6cz0gLI
I also had to run
# ausearch -c 'snapd' --raw | audit2allow -M my-snapd
******************** WICHTIG ***********************
To make this policy package active, execute:
semodule -i my-snapd.pp
# semodule -X 300 -i my-snapd.pp
several times before snapd would be initially working with SELinux enabled.
How would one debug this kind of connectivity issue further?
This is a
# snap version
snap 2.37.2-1.fc29
snapd 2.37.2-1.fc29
series 16
fedora 29
kernel 4.20.8-200.fc29.x86_64
system, and no proxy is in place.
It uses a manually configured /etc/nsswitch.conf (circumventing authselect) setting it to use resolve before dns, and having /etc/resolv.conf symlinked to /lib/systemd/resolv.conf for NetworkManager, making it recognise the systemd-resolved stub resolver on 127.0.0.53:53.
This was introduced in setting up update-systemd-resolved scripts for OpenVPN.
https://github.com/jonathanio/update-systemd-resolved
Now my intuition suggests me that this has to do with how Go binaries resolve DNS, by either mobilising Go or Cgo, indirectly via /etc/resolv.conf through the systemd-resolved stub resolver.
https://golang.org/src/net/dnsclient_unix.go
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1624320/comments/1
https://golang.org/pkg/net/#hdr-Name_Resolution
It’s been reported that the fast transition to systemd-resolved on Ubuntu and Fedora alike causes headaches for the maintainers of applications that rely on DNS resolution, like NetworkManager or OpenVPN. Is this maybe also the cause for snap stopping to work as expected, or are other forces at work here?