Did your kubelet certificate expire in k8s
- Details
- Category: Blog
- Published: Thursday, 15 April 2021 12:02
- Written by Lars Berntzon
- Hits: 2764
For some reason the kublet selfsigned certificate was expired in my cluster. That is the kubelets own API-service, running on port 10250 (i.e. not the client cert that kubelet uses to talk with api-servers). Its supposed to be a self-signed certificate but it was not renewed.
The problem was not very obvious but we saw it when the metrics-service did not work properly. It complained about expired certificates on for port 10250 on nodes.
I could not find any article about how to re-create this certificate. Sure, kubeadm certs has a lot of renewal options, but not for the actual kublet https port as far as I could find out.
The solution showed up to be quite simple. Just remove the two files /var/lib/kubelet/pki/kubelet.crt and /var/lib/kubelet/pki/kubelet.key and restart the kublet service with systemctl restart kublet.
The kubelet will then generate new self-signed certs.
In the end though, this was shown not to be the problem. First, the metrics service deployment needs to be run with the container argument: --kubelet-insecure-tls
at least if the kubelets run with self-signed certs.
Our root problem was that one api-server was running with a faulty proxy settings which caused its internal call to the metrics server to fail.