Published: Thursday, 02 September 2021 15:46
Written by Lars Berntzon
Are you getting below errors when running: kubeadm upgrade node
[upgrade] Reading configuration from the cluster...
[upgrade] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
unable to fetch the kubeadm-config ConfigMap: failed to get config map: Unauthorized
To see the stack trace of this error execute with --v=5 or higher
Chances are that the kubelet certificate is expired and can not be used to upgrade the node. A strace of that command reveals it does not really use the current users config but the file /etc/kubernetes/kubelet.conf. That file does in turn point to the key and cert to be used. In my installation both cert and key refered to /var/lib/kubelet/pki/kubelet-client-current.pem as follows:
That file is in turn a symlink to the real file, in my case was /var/lib/kubelet/pki/kubelet-client-2020-06-04-18-38-46.pem - which for some reason was not updated recently.
You need to create an updated cert/key file there and change /var/lib/kubelet/pki/kubelet-client-current.pem to this new file instead.
Published: Thursday, 15 April 2021 12:02
Written by Lars Berntzon
For some reason the kublet selfsigned certificate was expired in my cluster. That is the kubelets own API-service, running on port 10250 (i.e. not the client cert that kubelet uses to talk with api-servers). Its supposed to be a self-signed certificate but it was not renewed.
The problem was not very obvious but we saw it when the metrics-service did not work properly. It complained about expired certificates on for port 10250 on nodes.
I could not find any article about how to re-create this certificate. Sure, kubeadm certs has a lot of renewal options, but not for the actual kublet https port as far as I could find out.
The solution showed up to be quite simple. Just remove the two files /var/lib/kubelet/pki/kubelet.crt and /var/lib/kubelet/pki/kubelet.key and restart the kublet service with systemctl restart kublet.
The kubelet will then generate new self-signed certs.
In the end though, this was shown not to be the problem. First, the metrics service deployment needs to be run with the container argument: --kubelet-insecure-tls
at least if the kubelets run with self-signed certs.
Our root problem was that one api-server was running with a faulty proxy settings which caused its internal call to the metrics server to fail.