From 1ca86d09d10ead20a04c9fa37ea75b386d908fac Mon Sep 17 00:00:00 2001 From: jpgouin Date: Mon, 30 Jun 2025 16:54:13 +0200 Subject: [PATCH] add troubleshoot how to guide (#390) * add troubleshoot how to guide Co-authored-by: Enrico Candino --- docs/howtos/troubleshooting.md | 147 +++++++++++++++++++++++++++++++++ 1 file changed, 147 insertions(+) create mode 100644 docs/howtos/troubleshooting.md diff --git a/docs/howtos/troubleshooting.md b/docs/howtos/troubleshooting.md new file mode 100644 index 00000000..403a4f6f --- /dev/null +++ b/docs/howtos/troubleshooting.md @@ -0,0 +1,147 @@ +# Troubleshooting + +This guide walks through common troubleshooting steps for working with K3K virtual clusters. + +--- + +## `too many open files` error + +The `k3k-kubelet` or `k3kcluster-server-` run into the following issue: + +```sh +E0604 13:14:53.369369 1 leaderelection.go:336] error initially creating leader election record: Post "https://k3k-http-proxy-k3kcluster-service/apis/coordination.k8s.io/v1/namespaces/kube-system/leases": context canceled +{"level":"fatal","timestamp":"2025-06-04T13:14:53.369Z","logger":"k3k-kubelet","msg":"virtual manager stopped","error":"too many open files"} +``` + +This typically indicates a low limit on inotify watchers or file descriptors on the host system. + +To increase the inotify limits connect to the host nodes and run: + +```sh +sudo sysctl -w fs.inotify.max_user_watches=2099999999 +sudo sysctl -w fs.inotify.max_user_instances=2099999999 +sudo sysctl -w fs.inotify.max_queued_events=2099999999 +``` + +You can persist these settings by adding them to `/etc/sysctl.conf`: + +```sh +fs.inotify.max_user_watches=2099999999 +fs.inotify.max_user_instances=2099999999 +fs.inotify.max_queued_events=2099999999 +``` + +Apply the changes: + +```sh +sudo sysctl -p +``` + +You can find more details in this [KB document](https://www.suse.com/support/kb/doc/?id=000020048). + +--- + +## Inspect Controller Logs for Failure Diagnosis + +To view logs for a failed virtual cluster: + +```sh +kubectl logs -n k3k-system -l app.kubernetes.io/name=k3k +``` + +This retrieves logs from K3k controller components. + +--- + +## Inspect Cluster Logs for Failure Diagnosis + +To view logs for a failed virtual cluster: + +```sh +kubectl logs -n -l cluster= +``` + +This retrieves logs from K3k cluster components (`agents, server and virtual-kubelet`). + +> 💡 You can also use `kubectl describe cluster ` to check for recent events and status conditions. + +--- + +## Virtual Cluster Not Starting or Stuck in Pending + +Some of the most common causes are related to missing prerequisites or wrong configuration. + +### Storage class not available + +When creating a Virtual Cluster with `dynamic` persistence, a PVC is needed. You can check if the PVC was claimed but not bound with `kubectl get pvc -n `. If you see a pending PVC you probably don't have a default storage class defined, or you have specified a wrong one. + +#### Example with wrong storage class + +The `pvc` is pending: + +```bash +kubectl get pvc -n k3k-test-storage +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE +varlibrancherk3s-k3k-test-storage-server-0 Pending not-available 4s +``` + +The `server` is pending: + +```bash +kubectl get po -n k3k-test-storage +NAME READY STATUS RESTARTS AGE +k3k-test-storage-kubelet-j4zn5 1/1 Running 0 54s +k3k-test-storage-server-0 0/1 Pending 0 54s +``` + +To fix this you should use a valid storage class, you can list existing storage class using: + +```bash +kubectl get storageclasses.storage.k8s.io +NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE +local-path (default) rancher.io/local-path Delete WaitForFirstConsumer false 3d6h +``` + +### Wrong node selector + +When creating a Virtual Cluster with `defaultNodeSelector`, if the selector is not valid all pods will be pending. + +#### Example + +The `server` is pending: + +```bash +kubectl get po +NAME READY STATUS RESTARTS AGE +k3k-k3kcluster-node-placed-server-0 0/1 Pending 0 58s +``` + +The description of the pod provide the reason: + +```bash +kubectl describe po k3k-k3kcluster-node-placed-server-0 +... +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Warning FailedScheduling 84s default-scheduler 0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling. +``` + +To fix this you should use a valid node affinity/selector. + +### Image pull issues (airgapped setup) + +When creating a Virtual Cluster in air-gapped environment, images need to be available in the configured registry. You can check for `ImagePullBackOff` status when getting the pods in the virtual cluster namespace. + +#### Example + +The `server` is failing: + +```bash +kubectl get po -n k3k-test-registry +NAME READY STATUS RESTARTS AGE +k3k-test-registry-kubelet-r4zh5 1/1 Running 0 54s +k3k-test-registry-server-0 0/1 ImagePullBackOff 0 54s +``` + +To fix this make sure the failing image is available. You can describe the failing pod to get more details.