Kubectl: wrong gcloud path when doing any operations

snap
1

This started happening recently:

~ kubectl version      
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.5", GitCommit:"e6503f8d8f769ace2f338794c914a96fc335df0f", GitTreeState:"clean", BuildDate:"2020-07-07T14:04:52Z", GoVersion:"go1.13.12", Compiler:"gc", Platform:"linux/amd64"}
Unable to connect to the server: error executing access token command "/snap/google-cloud-sdk/138/bin/gcloud config config-helper --format=json": err=fork/exec /snap/google-cloud-sdk/138/bin/gcloud: no such file or directory output= stderr=

In fact no kubectl commands are working at all at the moment.

~ ll /snap/google-cloud-sdk/
total 0
drwxr-xr-x 14 root root 624 Jun 30 14:54 139
drwxr-xr-x 14 root root 624 Jul  8 15:34 140
lrwxrwxrwx  1 root root   3 Jul  8 20:15 current -> 140

~ snap list    (removed irrelevant entries)                                       
Name                  Version                     Rev   Tracking         Publisher          Notes
google-cloud-sdk      300.0.0                     140   latest/stable    google-cloud-sdk✓  classic
helm                  3.2.4                       256   latest/stable    snapcrafters       classic
kubectl               1.18.5                      1559  latest/stable    canonical✓         classic

The kubectl command should not use hardcoded paths to utils but instead rely either on path, or the binaries at /snap/bin

2

This is better asked in the #snap category, but perhaps @joedborg knows who maintains the kubectl snap.

3

Thank you, I forgot to select the category when I created the post. Whoops.

4

Hey @copong. Apologies, I saw this last week but got sidetracked. I’m trying to find the relevant place to log this.

I’m surprised you’re getting this error though because this is a classic snap, it should be able to find gcloud on the system. You could try with:

PATH=/path/to/gcloud/bin:${PATH} kubectl ...

that will force it to look in the relevant place (you can find this by running which gcloud and removing gcloud from the end).

5

Hey, thanks for your reply. Gcloud is already on my path:

~ which gcloud
/snap/bin/gcloud

The problem must be something else perhaps?

6

@ijohnson off the top of your head, is there any reason that two classic snaps wouldn’t be able to “see” eachother?

7

No, there shouldn’t be. The error here looks like kubelet has either cached the location to the gcloud binary or otherwise inspected the system to find that and stored it in some configuration file. Beyond that without looking at the kubelet snap I can’t say what might be the cause of this.

BTW, where is the kubelet snap source ?

8

After some sleuthing on my home folder, it looks like my kube config is reflecting the real path to the gcloud binary as it was when the context was created, instead of the symlink at /snap/bin/gcloud

~ cat ~/.kube/config
- name: gke_FOOBAR
  user:
    auth-provider:
      config:
        access-token: FOOBARFOOBAR
        cmd-args: config config-helper --format=json
        cmd-path: /snap/google-cloud-sdk/138/bin/gcloud
        expiry: "2020-07-27T16:11:13Z"
        expiry-key: '{.credential.token_expiry}'
        token-key: '{.credential.access_token}'
      name: gcp

I’m unsure why it’s doing this, but these files are managed by kubectl itself are they not?

9

I’ve encountered this issue, myself. The problem is that when you use the gcloud cli to save details of a google-cloud-hosted kubernetes into ~/.kube/config so that you can use kubectl to access it the gcloud cli introspects itself to determine its own executable’s path. It then saves that full path into the kube config so that the kubectl can call gcloud as a credential helper to assist with the authorisation process to access the google-cloud gke cluster. Once the gcloud snap is updated three times from the original invokation that saved the config to kubectl the path it saved will be removed from the system because it used the full path to the executable instead of the path to /snap/bin/gcloud so it included the snap revision number as part of the path. Three updates and the snap revision referenced is removed from the system to maintain the maximum of three available revisions (the active one and two previous revisions for rollback if required).

The solution to this will be for the gcloud project to include a code-path that recognises when it should use the /snap/bin/gcloud path or rely on searching the $PATH. This likley means we need Google to fix this. In the meantime you may be able to use /snap/bin/gcloud in place of the full path or change the revision number to the magic name current.

Thinking further on using /snap/bin/gcloud this will potentially fall foul of apparmor weirdness when calling a different classic snap’s executable (gcloud) from the current classic snap (kubectl)…

10

Thank you. Yes, for now I’ve symlinked current into 138, but certainly needs fixing properly at the base.