Enrico CandinoandGitHub
3a4d93f5fb
* Add tests asserting GetPods is scoped to the node's own Pods Adds a unit test (fake host + virtual clients, Pods across multiple nodes plus a dangling one) and a multi-node e2e test (a Deployment with one nginx Pod per node via required anti-affinity; restart every k3k-kubelet agent Pod; assert no workload Pod is deleted from the host or virtual cluster). Both pin the intended behavior and fail against the current code; the fix follows in the next commit. * Scope Provider.GetPods to the node's own Pods to prevent cross-node dangling-pod deletion GetPods() listed host Pods cluster-wide (by the k3k.io/clusterName label only). The vendored virtual-kubelet library's deleteDanglingPods reconciliation deletes any Pod returned here that is missing from this instance's virtual Pod lister, which is scoped to spec.nodeName == agentHostname. On a multi-node host cluster, every restarting k3k-kubelet instance therefore saw Pods owned by other nodes as 'dangling' and deleted them from the host (and, in turn, the virtual cluster). Scope GetPods by the *virtual* Pod's spec.nodeName -- the same ownership signal the framework uses -- excluding Pods owned by other nodes while still returning own-node Pods and genuinely dangling ones. The host Pod's physical node is not a reliable owner (it is scheduled with only a soft, sometimes-absent affinity), so it must not be used. Virtual Pods are read live to avoid a startup cache-sync race. Makes the previous commit's tests pass. * Refactor GetPods to scope to the cluster namespace and update tests accordingly * Add AgentNameLabel to track Pods synced by the k3k-kubelet agent and update tests accordingly * Add failing test for updatePod
Virtual Kubelet
This package provides an impelementation of a virtual cluster node using virtual-kubelet.
The implementation is based on several projects, including:
Overview
This project creates a node that registers itself in the virtual cluster. When workloads are scheduled to this node, it simply creates/updates the workload on the host cluster.
Usage
Build/Push the image using (from the root of rancher/k3k):
make build
docker buildx build -f package/Dockerfile . -t $REPO/$IMAGE:$TAG
When running, it is recommended to deploy a k3k cluster with 1 server (with --disable-agent as a server arg) and no agents (so that the workloads can only be scheduled on the virtual node/host cluster).
After the image is built, it should be deployed with the following ENV vars set:
CLUSTER_NAMEshould be the name of the cluster.CLUSTER_NAMESPACEshould be the namespace the cluster is running in.HOST_KUBECONFIGshould be the path on the local filesystem (in container) to a kubeconfig for the host cluster (likely stored in a secret/mounted as a volume).VIRT_KUBECONFIGshould be the path on the local filesystem (in container) to a kubeconfig for the virtual cluster (likely stored in a secret/mounted as a volume).VIRT_POD_IPshould be the IP that the container is accessible from.
This project is still under development and there are many features yet to be implemented, but it can run a basic nginx pod.