Error when joining cohort

cory_fu · September 30, 2019, 3:06pm

It seems that for classic snaps, while refreshing to a different channel using snap refresh does not require specifying --classic, joining a cohort using snap refresh does require the --classic flag. Furthermore, the error message that is returned if --classic is omitted is confusing, stating that the snap is not available in that channel at all.

It turns out that the use of --classic did not help. The actual issue is that it’s failing as below when on a LXD Juju unit, while directly on the host of the same machine, it works fine.

ubuntu@juju-faa4c5-4:~$ sudo snap refresh kubectl --cohort MSBaZ0cyVVJ5Y0RndnhTVnNrZm9aeG40NHVhUk13MGl3ZSAxNTY5ODU2NDQzIDA2MTNhNGM2MmE1MWQxNjA0YTQ3ZmQ0YzhkMGI1MmJjMDZkMzE4YTQxYzlhZjExOWMyOGYwNWY4NjQ5ZTU2YzE= --classic
error: snap "kubectl" is not available on 1.16/stable but is available to install on the following
       tracks:

       latest/stable  snap refresh --stable kubectl
       1.15/stable    snap refresh --channel=1.15 kubectl
       1.14/stable    snap refresh --channel=1.14 kubectl
       1.13/stable    snap refresh --channel=1.13 kubectl
       1.12/stable    snap refresh --channel=1.12 kubectl
       1.11/stable    snap refresh --channel=1.11 kubectl
       1.10/stable    snap refresh --channel=1.10 kubectl
       1.9/stable     snap refresh --channel=1.9 kubectl
       1.8/stable     snap refresh --channel=1.8 kubectl
       1.7/stable     snap refresh --channel=1.7 kubectl
       1.6/stable     snap refresh --channel=1.6 kubectl
       1.5/stable     snap refresh --channel=1.5 kubectl

       Please be mindful that different tracks may include different features. Get more information
       with 'snap info kubectl'.

ubuntu@juju-faa4c5-4:~$ snap info kubectl
name:      kubectl
summary:   Command line client for controlling a Kubernetes cluster.
publisher: Canonical✓
contact:   https://www.ubuntu.com/kubernetes
license:   unset
description: |
  kubectl is a command line client for running commands against Kubernetes
  clusters.
  
  For more information about kubectl, including syntax, descriptions of command operations, and
  common examples, see the [overview](https://kubernetes.io/docs/reference/kubectl/overview/). For
  details about each command, including all the supported flags and subcommands, see the [kubectl
  reference
  documentation](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands/).
commands:
  - kubectl
snap-id:      ZgG2URycDgvxSVskfoZxn44uaRMw0iwe
tracking:     1.16/stable
refresh-date: today at 14:19 UTC
channels:
  stable:         1.16.0         2019-09-27 (1249) 11MB classic
  candidate:      1.16.0         2019-09-27 (1249) 11MB classic
  beta:           1.16.0         2019-09-27 (1249) 11MB classic
  edge:           1.16.0         2019-09-27 (1249) 11MB classic
  1.17/stable:    –                                     
  1.17/candidate: –                                     
  1.17/beta:      –                                     
  1.17/edge:      1.17.0-alpha.0 2019-09-18 (1235) 11MB classic
  1.16/stable:    1.16.0         2019-09-19 (1249) 11MB classic
  1.16/candidate: 1.16.0-rc.2    2019-09-13 (1202) 11MB classic
  1.16/beta:      1.16.1-beta.0  2019-09-18 (1228) 11MB classic
  1.16/edge:      1.16.1-beta.0  2019-09-18 (1228) 11MB classic
  1.15/stable:    1.15.4         2019-09-18 (1234) 10MB classic
  1.15/candidate: 1.15.3         2019-08-21 (1189) 10MB classic
  1.15/beta:      1.15.5-beta.0  2019-09-18 (1208) 10MB classic
  1.15/edge:      1.15.5-beta.0  2019-09-18 (1208) 10MB classic
  1.14/stable:    1.14.7         2019-09-18 (1233) 10MB classic
  1.14/candidate: 1.14.6         2019-08-21 (1188) 10MB classic
  1.14/beta:      1.14.8-beta.0  2019-09-18 (1207) 10MB classic
  1.14/edge:      1.14.8-beta.0  2019-09-18 (1207) 10MB classic
  1.13/stable:    1.13.11        2019-09-18 (1243) 10MB classic
  1.13/candidate: 1.13.10        2019-08-21 (1187) 10MB classic
  1.13/beta:      1.13.12-beta.0 2019-09-18 (1206) 10MB classic
  1.13/edge:      1.13.12-beta.0 2019-09-18 (1206) 10MB classic
  1.12/stable:    1.12.9         2019-06-05 (1004) 11MB classic
  1.12/candidate: 1.12.9         2019-05-29 (1004) 11MB classic
  1.12/beta:      1.12.9         2019-05-29 (1004) 11MB classic
  1.12/edge:      1.12.9         2019-05-29 (1004) 11MB classic
  1.11/stable:    1.11.9         2019-03-29  (864) 11MB classic
  1.11/candidate: 1.11.9         2019-03-26  (864) 11MB classic
  1.11/beta:      1.11.9         2019-03-26  (864) 11MB classic
  1.11/edge:      1.11.9         2019-03-26  (864) 11MB classic
  1.10/stable:    1.10.13        2019-02-27  (743) 10MB classic
  1.10/candidate: 1.10.13        2019-03-21  (836) 10MB classic
  1.10/beta:      1.10.13        2019-03-21  (836) 10MB classic
  1.10/edge:      1.10.13        2019-03-21  (836) 10MB classic
  1.9/stable:     1.9.11         2018-10-08  (454) 12MB classic
  1.9/candidate:  1.9.11         2018-10-17  (466) 12MB classic
  1.9/beta:       1.9.11         2018-10-17  (466) 12MB classic
  1.9/edge:       1.9.11         2018-10-17  (466) 12MB classic
  1.8/stable:     1.8.15         2018-07-11  (435) 10MB classic
  1.8/candidate:  1.8.15         2018-10-17  (465) 10MB classic
  1.8/beta:       1.8.15         2018-10-17  (465) 10MB classic
  1.8/edge:       1.8.15         2018-10-17  (465) 10MB classic
  1.7/stable:     1.7.16         2018-06-06  (394) 13MB classic
  1.7/candidate:  1.7.16         2018-10-17  (464) 13MB classic
  1.7/beta:       1.7.16         2018-10-17  (464) 13MB classic
  1.7/edge:       1.7.16         2018-10-17  (464) 13MB classic
  1.6/stable:     1.6.13         2017-11-30  (233) 12MB classic
  1.6/candidate:  1.6.13         2018-10-17  (463) 12MB classic
  1.6/beta:       1.6.13         2018-10-17  (463) 12MB classic
  1.6/edge:       1.6.13         2018-10-17  (463) 12MB classic
  1.5/stable:     1.5.5          2017-05-17    (3)  9MB classic
  1.5/candidate:  1.5.5          2017-05-17    (3)  9MB classic
  1.5/beta:       1.5.5          2017-05-17    (3)  9MB classic
  1.5/edge:       1.5.5          2017-05-17    (3)  9MB classic
installed:        1.16.0                    (1249) 11MB classic,in-cohort

cory_fu · September 30, 2019, 3:28pm

In that output, I notice that it did somehow get into the cohort, per the installed line of the snap info, but I’m not sure how and after leaving the cohort, I can’t replicate it to get back in.

cory_fu · September 30, 2019, 3:35pm

More info:

ubuntu@juju-faa4c5-4:~$ sudo snap switch kubectl --cohort MSBaZ0cyVVJ5Y0RndnhTVnNrZm9aeG40NHVhUk13MGl3ZSAxNTY5ODU3MTM3IDYzOTE1YWVmYWRiNzY2NTEwM2ZkMWI2ODNlNWViNmYyNzg0MDM3YWVlYjg2NWY4OWRmMjEzNGExNjc2Nzc2ZDA=
"kubectl" switched to the "…2Nzc2ZDA=" cohort
ubuntu@juju-faa4c5-4:~$ sudo snap refresh kubectl
snap "kubectl" has no updates available
ubuntu@juju-faa4c5-4:~$ sudo snap switch kubectl --channel=1.15/stable
"kubectl" switched to the "1.15/stable" channel

ubuntu@juju-faa4c5-4:~$ sudo snap refresh
kubectl (1.15/stable) 1.15.4 from Canonical✓ refreshed
ubuntu@juju-faa4c5-4:~$ sudo snap refresh kubectl --cohort MSBaZ0cyVVJ5Y0RndnhTVnNrZm9aeG40NHVhUk13MGl3ZSAxNTY5ODU3MTM3IDYzOTE1YWVmYWRiNzY2NTEwM2ZkMWI2ODNlNWViNmYyNzg0MDM3YWVlYjg2NWY4OWRmMjEzNGExNjc2Nzc2ZDA=
snap "kubectl" has no updates available
ubuntu@juju-faa4c5-4:~$ sudo snap refresh kubectl --cohort MSBaZ0cyVVJ5Y0RndnhTVnNrZm9aeG40NHVhUk13MGl3ZSAxNTY5ODU3MTM3IDYzOTE1YWVmYWRiNzY2NTEwM2ZkMWI2ODNlNWViNmYyNzg0MDM3YWVlYjg2NWY4OWRmMjEzNGExNjc2Nzc2ZDA= --channel=1.16/stable
kubectl (1.16/stable) 1.16.0 from Canonical✓ refreshed
ubuntu@juju-faa4c5-4:~$ sudo snap refresh kubectl --cohort MSBaZ0cyVVJ5Y0RndnhTVnNrZm9aeG40NHVhUk13MGl3ZSAxNTY5ODU3MTM3IDYzOTE1YWVmYWRiNzY2NTEwM2ZkMWI2ODNlNWViNmYyNzg0MDM3YWVlYjg2NWY4OWRmMjEzNGExNjc2Nzc2ZDA= --channel=1.16/stable
error: snap "kubectl" is not available on 1.16/stable but is available to install on the following
       tracks:

       latest/stable  snap refresh --stable kubectl
       1.15/stable    snap refresh --channel=1.15 kubectl
       1.14/stable    snap refresh --channel=1.14 kubectl
       1.13/stable    snap refresh --channel=1.13 kubectl
       1.12/stable    snap refresh --channel=1.12 kubectl
       1.11/stable    snap refresh --channel=1.11 kubectl
       1.10/stable    snap refresh --channel=1.10 kubectl
       1.9/stable     snap refresh --channel=1.9 kubectl
       1.8/stable     snap refresh --channel=1.8 kubectl
       1.7/stable     snap refresh --channel=1.7 kubectl
       1.6/stable     snap refresh --channel=1.6 kubectl
       1.5/stable     snap refresh --channel=1.5 kubectl

       Please be mindful that different tracks may include different features. Get more information
       with 'snap info kubectl'.

cory_fu · September 30, 2019, 5:41pm

I setup the snap-store-proxy to see if the issue had to do with no updates being available when issuing the refresh, but now I can’t even create a cohort key:

root@loved-crawdad:~# snap install kubectl --classic --channel=1.16/stable
kubectl (1.16/stable) 1.16.0-rc.2 from Canonical✓ installed
root@loved-crawdad:~# snap refresh --list
All snaps up to date.
root@loved-crawdad:~# snap create-cohort kubectl
error: cannot create cohorts: snap not found

chipaca · September 30, 2019, 7:35pm

I’m having a little trouble figuring out exactly what it is that isn’t working for you. This topic seems to start in the middle of something, with no intro…

What are you trying to do? How is it failing?

cory_fu · September 30, 2019, 7:59pm

Yeah, sorry. I started this while in the middle of testing my charm logic thinking I was just hitting some edge case, but at the end I can’t seem to get anything related to cohorts working. The basic steps my charm is following are:

Install a snap (kubectl on channel 1.16/stable)
Create / update cohort
2.a) Leader of master creates a cohort key and sends it to peers & workers
2.b) Workers get new cohort key from relation
Join that cohort
Refresh
Leader watches for new revs of snap, and goes to 2

I can do step 3 using snap switch --cohort but using snap refresh --cohort fails with the "not available on " error, as does separately calling snap refresh kubectl after joining the cohort.

I thought I saw some inconsistent results early on, with some attempts working, which added to the confusion, but at this point it consistently fails.

I thought maybe it would only give that error if there were no new revs available, and I needed to test with the snap store proxy anyway, so I went and set that up. However, with a snap store proxy configured (with an override; haven’t tested without the override), I always just get “snap not found” when trying to create the cohort.

cory_fu · September 30, 2019, 8:06pm

The relevant errors are:

error: snap "kubectl" is not available on 1.16/stable but is available to install on the following
       tracks:

       latest/stable  snap refresh --stable kubectl
       1.15/stable    snap refresh --channel=1.15 kubectl
       1.14/stable    snap refresh --channel=1.14 kubectl
       1.13/stable    snap refresh --channel=1.13 kubectl
       1.12/stable    snap refresh --channel=1.12 kubectl
       1.11/stable    snap refresh --channel=1.11 kubectl
       1.10/stable    snap refresh --channel=1.10 kubectl
       1.9/stable     snap refresh --channel=1.9 kubectl
       1.8/stable     snap refresh --channel=1.8 kubectl
       1.7/stable     snap refresh --channel=1.7 kubectl
       1.6/stable     snap refresh --channel=1.6 kubectl
       1.5/stable     snap refresh --channel=1.5 kubectl

       Please be mindful that different tracks may include different features. Get more information
       with 'snap info kubectl'.

And:

error: cannot create cohorts: snap not found

cory_fu · September 30, 2019, 9:56pm

And here’s a start-to-finish session of installing the snap, creating a cohort, attempting to join (and failing, both ways), and finally setting up the snap store proxy and having that break creating cohort keys until the store proxy is removed: https://pastebin.canonical.com/p/FrG9dWxVWy/

chipaca · September 30, 2019, 11:08pm

Yeah, I’m able to reproduce this. Something weird is going on.

Weirder still: wait a few minutes after creating the cohort, and it starts working.

I’ll file a bug.

chipaca · September 30, 2019, 11:17pm

Filed as

wgrant · October 1, 2019, 4:12am

You weren’t wrong: it was inconsistently failing. I’ve deployed a fix, and it should behave properly now. Thanks for the report.

cory_fu · October 2, 2019, 3:23pm

I still get this when using the snap store proxy and trying to create a cohort. Cohorts start working again when I remove the store ID config to disable the store proxy.

chipaca · October 2, 2019, 3:25pm

does the store proxy support cohorts?

cory_fu · October 2, 2019, 3:26pm

I suppose it may not, but that seems like a pretty significant use-case for cohorts. For example, being able to manage the snap revisions is pretty much required for us to actually test the snap coherence support in Charmed Kubernetes (see lp:1845559).

bloodearnest · October 2, 2019, 3:38pm

The bugfix that @wgrant applied to production has not yet been released in the snap-store-proxy. We’re working on it, will let you know when it’s released.

bloodearnest · October 15, 2019, 3:06pm

A release of the snap store proxy with the bug fix in is currently on the candidate channel.

@cory_fu, could you possibly refresh to that and check if it solves the issue you had with cohorts?

Thanks.