Unusable cluster after upgrading to 1.34.1

[davidnich]

Summary

After upgrading from microk8s 1.33.0 to 1.34.1, the cluster is unstable - I have many pods with CrashLoopBackOff status as well as Pending status with no events or other errors to explain why from what I could find. For pods that were running, updating a deployment with an annotation that should trigger a rolling upgrade / restart does nothing. When I try to remove Pending pods, the command reports that the pod was remooved and then hangs, and the pods stay in status Terminating indefinitely as well.

What Should Happen Instead?

None of the above should happen, and if it does, there should be a clearly-visible reason, so the root cause can be identified and addressed

Reproduction Steps

I do not know how to reproduce - I have been running my microk8s cluster for more than 2 years, and now I think I have to rebuild it from scratch

Introspection Report

inspection-report-20251023_110810.tar.gz

Can you suggest a fix?

I do not know

Are you interested in contributing with a fix?

If I could I would

[davidnich]

I have a way to temporarily get back to somewhat normal for a short period, but it requires downtime.

If I drain all nodes (some with --force --grace-period=0 after they hang) then do microk8s stop on all nodes, then microk8s start, everything starts up fine and looks good for short time. Then the problems start again - currently I have 1 node put in NotReady status and a series of pods terminated for unknown reasons - where the deployments & stateful sets also have no error info in them - I'm unable to find any errors anywhere to explain this behavior and am considering wiping the cluster and recreating it from scratch

I can see events where the pods and nodes were taken offline but cannot find any reasons for it

[davidnich]

I was able to solve my problems (I believe - looking good) by updating the certificates on each node with:
sudo /snap/bin/microk8s refresh-certs --cert server.crt

To check if the certs on each node are out of date, you can run:
sudo /snap/bin/microk8s refresh-certs -c

Doc ref: https://microk8s.io/docs/command-reference#heading--microk8s-refresh-certs

Another user who ran into this problem: https://www.davidpuplava.com/coding-craft/kubernetes-refresh-certs-with-microk8s-cluster

There were other problems as well, but I may well have caused some of the them while working on the cluster when it was down.

Is this not something that can / should be automated on each node in microk8s itself?

At least the docs should be updated so that users can put in place their own automation for this on each node to avoid surprises