github/k3k
1
0
Fork 0
mirror of https://github.com/rancher/k3k.git synced 2026-08-19 04:16:16 +00:00
Code Issues Packages Projects Releases Wiki Activity

Added HCP (Hosted Control Plane) mode (#876)

Browse Source 
* Add HCP (Hosted Control Plane) support

Introduce hosted control plane mode for k3k virtual clusters, including
API types, controller logic, server endpoint handling, CLI flags,
CRD updates, kubeconfig generation, and examples.

Co-Authored-By: RuFlo <ruv@ruv.net>

* removed hcpRegitration command

added HCP conformance tests

warning for hcp

fix multi-VM HCP conformance test networking

  Both QEMU workers booted with `-net user` and ended up registering the
  same InternalIP (10.0.2.15) because each VM gets its own isolated NAT
  slirp. Flannel propagated this to `public-ip` on both nodes, so VXLAN
  could not tunnel between workers and any cross-node pod traffic broke
  (89 failed / 335 passed of 424 conformance specs).

  Replace user-mode networking with a Linux bridge (k3kbr0,
  192.168.100.0/24) and one TAP device per VM, so the two workers share
  an L2 segment with unique routable IPs. NAT outbound from the bridge
  keeps internet access working for image pulls.

  Also set unique hostnames via cloud-init (worker-1/worker-2) and drop
  the `--node-name` flag from INSTALL_K3S_EXEC, since k3s now picks the
  correct node name from the OS hostname on its own.

  Bump hydrophone back to `--parallel 4` to match the single-VM job
  (parallelism was reduced earlier when the failure was thought to be
  resource-related).

added HCP print command

updated crds

adding e2e tests

Refactor selectNonLoopbackSAN function to accept SANs directly and update related logic in ensureHCPRegistration

* Update agent flag validation and enhance ingress host check with a warning log

Refactor descriptions for cluster provisioning mode and role in CRDs and documentation

Refactor logging in ServerURL function to use controller-runtime logger

Rename selectNonLoopbackSAN to findNonLoopbackSAN for clarity and update references

Refactor ServerURL function and related code to remove unused parameters and improve clarity

Remove unused imports from kubeconfig.go to improve code clarity

* suggested changes

* fix comment

* fix test

---------

Co-authored-by: jpgouin <jeanphilippe.gouin@suse.com>
Co-authored-by: RuFlo <ruv@ruv.net>
This commit is contained in:
Enrico Candino
2026-07-07 14:12:19 +02:00
committed by GitHub
co-authored by RuFlo jpgouin
parent 33216d49ae
commit 39cc69f3e3
28 changed files with 1355 additions and 212 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cli/k3kcli.adoc
+1 -1
View File
@@ -54,7 +54,7 @@ k3kcli cluster create [command options] NAME
--kubeconfig-server string override the kubeconfig server host
--labels stringArray Labels to add to the cluster object (e.g. key=value)
--mirror-host-nodes Mirror Host Cluster Nodes
--mode string k3k mode type (shared, virtual) (default "shared")
--mode string k3k mode type (shared, virtual, hcp) (default "shared")
-n, --namespace string namespace of the k3k cluster
--persistence-type string persistence mode for the nodes (dynamic, ephemeral) (default "dynamic")
--policy string The policy to create the cluster in
docs/cli/k3kcli_cluster_create.md
+1 -1
View File
@@ -25,7 +25,7 @@ k3kcli cluster create [command options] NAME
--kubeconfig-server string override the kubeconfig server host
--labels stringArray Labels to add to the cluster object (e.g. key=value)
--mirror-host-nodes Mirror Host Cluster Nodes
--mode string k3k mode type (shared, virtual) (default "shared")
--mode string k3k mode type (shared, virtual, hcp) (default "shared")
-n, --namespace string namespace of the k3k cluster
--persistence-type string persistence mode for the nodes (dynamic, ephemeral) (default "dynamic")
--policy string The policy to create the cluster in
docs/crds/crds.adoc
+9 -9
View File
@@ -133,8 +133,9 @@ _Underlying type:_ _string_
ClusterMode is the possible provisioning mode of a Cluster.
_Validation:_
- Enum: [shared virtual]
Supported values: `shared`, `virtual`, `hcp`.
_Appears In:_
@@ -177,8 +178,7 @@ _Appears In:_
| *`version`* __string__ | Version is the K3s version to use for the virtual nodes. +
It should follow the K3s versioning convention (e.g., v1.28.2-k3s1). +
If not specified, the Kubernetes version of the host node will be used. + | |
| *`mode`* __xref:{anchor_prefix}-github-com-rancher-k3k-pkg-apis-k3k-io-v1beta1-clustermode[$$ClusterMode$$]__ | Mode specifies the cluster provisioning mode: "shared" or "virtual". +
Defaults to "shared". This field is immutable. + | shared | Enum: [shared virtual] +
| *`mode`* __xref:{anchor_prefix}-github-com-rancher-k3k-pkg-apis-k3k-io-v1beta1-clustermode[$$ClusterMode$$]__ | Mode specifies the cluster provisioning mode. This field is immutable. + | shared | Enum: [shared virtual hcp] +
| *`servers`* __integer__ | Servers specifies the number of K3s pods to run in server (control plane) mode. +
Must be at least 1. Defaults to 1. + | 1 |
@@ -548,8 +548,9 @@ _Underlying type:_ _string_
PodSecurityAdmissionLevel is the policy level applied to the pods in the namespace.
_Validation:_
- Enum: [privileged baseline restricted]
Supported values: `privileged`, `baseline`, `restricted`.
_Appears In:_
@@ -624,8 +625,7 @@ secret contents will be mounted. + | |
| *`subPath`* __string__ | SubPath is an optional path within the secret to mount instead of the root. +
When specified, only the specified key from the secret will be mounted as a file +
at MountPath, keeping the parent directory writable. + | |
| *`role`* __string__ | Role is the type of the k3k pod that will be used to mount the secret. +
This can be 'server', 'agent', or 'all' (for both). + | | Enum: [server agent all] +
| *`role`* __string__ | Role is the type of the k3k pod that will be used to mount the secret. + | | Enum: [server agent all] +
|===
@@ -797,7 +797,7 @@ to set defaults and constraints (min/max) + | |
This includes both node affinity and pod affinity/anti-affinity rules. + | |
| *`defaultAgentAffinity`* __link:https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#affinity-v1-core[$$Affinity$$]__ | DefaultAgentAffinity specifies the affinity rules applied to agent pods of all clusters in the target Namespace. +
This includes both node affinity and pod affinity/anti-affinity rules. + | |
| *`allowedMode`* __xref:{anchor_prefix}-github-com-rancher-k3k-pkg-apis-k3k-io-v1beta1-clustermode[$$ClusterMode$$]__ | AllowedMode specifies the allowed cluster provisioning mode. Defaults to "shared". + | shared | Enum: [shared virtual] +
| *`allowedMode`* __xref:{anchor_prefix}-github-com-rancher-k3k-pkg-apis-k3k-io-v1beta1-clustermode[$$ClusterMode$$]__ | AllowedMode specifies the allowed cluster provisioning mode. + | shared | Enum: [shared virtual hcp] +
| *`disableNetworkPolicy`* __boolean__ | DisableNetworkPolicy indicates whether to disable the creation of a default network policy for cluster isolation. + | |
| *`podSecurityAdmissionLevel`* __xref:{anchor_prefix}-github-com-rancher-k3k-pkg-apis-k3k-io-v1beta1-podsecurityadmissionlevel[$$PodSecurityAdmissionLevel$$]__ | PodSecurityAdmissionLevel specifies the pod security admission level applied to the pods in the namespace. + | | Enum: [privileged baseline restricted] +
docs/crds/crds.md
+9 -7
View File
@@ -102,8 +102,9 @@ _Underlying type:_ _string_
ClusterMode is the possible provisioning mode of a Cluster.
_Validation:_
- Enum: [shared virtual]
Supported values: `shared`, `virtual`, `hcp`.
_Appears in:_
- [ClusterSpec](#clusterspec)
@@ -138,7 +139,7 @@ _Appears in:_
| Field | Description | Default | Validation |
| --- | --- | --- | --- |
| `version` _string_ | Version is the K3s version to use for the virtual nodes.<br />It should follow the K3s versioning convention (e.g., v1.28.2-k3s1).<br />If not specified, the Kubernetes version of the host node will be used. | | |
| `mode` _[ClusterMode](#clustermode)_ | Mode specifies the cluster provisioning mode: "shared" or "virtual".<br />Defaults to "shared". This field is immutable. | shared | Enum: [shared virtual] <br /> |
| `mode` _[ClusterMode](#clustermode)_ | Mode specifies the cluster provisioning mode. This field is immutable. | shared | Enum: [shared virtual hcp] <br /> |
| `servers` _integer_ | Servers specifies the number of K3s pods to run in server (control plane) mode.<br />Must be at least 1. Defaults to 1. | 1 | |
| `agents` _integer_ | Agents specifies the number of K3s pods to run in agent (worker) mode.<br />Must be 0 or greater. Defaults to 0.<br />This field is ignored in "shared" mode. | 0 | |
| `clusterCIDR` _string_ | ClusterCIDR is the CIDR range for pod IPs.<br />Defaults to 10.42.0.0/16 in shared mode and 10.52.0.0/16 in virtual mode.<br />This field is immutable. | | |
@@ -410,8 +411,9 @@ _Underlying type:_ _string_
PodSecurityAdmissionLevel is the policy level applied to the pods in the namespace.
_Validation:_
- Enum: [privileged baseline restricted]
Supported values: `privileged`, `baseline`, `restricted`.
_Appears in:_
- [VirtualClusterPolicySpec](#virtualclusterpolicyspec)
@@ -456,7 +458,7 @@ _Appears in:_
| `name` _string_ | Name is the name of the secret mount volume that will be used<br />as the name of volume and volume mount for the server or agent pod<br />if empty then the secret name will be used instead. | | MaxLength: 63 <br />MinLength: 1 <br />Pattern: `^[a-z0-9]([-a-z0-9]*[a-z0-9])?$` <br /> |
| `mountPath` _string_ | MountPath is the path within server and agent pods where the<br />secret contents will be mounted. | | |
| `subPath` _string_ | SubPath is an optional path within the secret to mount instead of the root.<br />When specified, only the specified key from the secret will be mounted as a file<br />at MountPath, keeping the parent directory writable. | | |
| `role` _string_ | Role is the type of the k3k pod that will be used to mount the secret.<br />This can be 'server', 'agent', or 'all' (for both). | | Enum: [server agent all] <br /> |
| `role` _string_ | Role is the type of the k3k pod that will be used to mount the secret. | | Enum: [server agent all] <br /> |
#### SecretSyncConfig
@@ -592,7 +594,7 @@ _Appears in:_
| `defaultPriorityClass` _string_ | DefaultPriorityClass specifies the priorityClassName applied to all pods of all clusters in the target Namespace. | | |
| `defaultServerAffinity` _[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#affinity-v1-core)_ | DefaultServerAffinity specifies the affinity rules applied to server pods of all clusters in the target Namespace.<br />This includes both node affinity and pod affinity/anti-affinity rules. | | |
| `defaultAgentAffinity` _[Affinity](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.31/#affinity-v1-core)_ | DefaultAgentAffinity specifies the affinity rules applied to agent pods of all clusters in the target Namespace.<br />This includes both node affinity and pod affinity/anti-affinity rules. | | |
| `allowedMode` _[ClusterMode](#clustermode)_ | AllowedMode specifies the allowed cluster provisioning mode. Defaults to "shared". | shared | Enum: [shared virtual] <br /> |
| `allowedMode` _[ClusterMode](#clustermode)_ | AllowedMode specifies the allowed cluster provisioning mode. | shared | Enum: [shared virtual hcp] <br /> |
| `disableNetworkPolicy` _boolean_ | DisableNetworkPolicy indicates whether to disable the creation of a default network policy for cluster isolation. | | |
| `podSecurityAdmissionLevel` _[PodSecurityAdmissionLevel](#podsecurityadmissionlevel)_ | PodSecurityAdmissionLevel specifies the pod security admission level applied to the pods in the namespace. | | Enum: [privileged baseline restricted] <br /> |
| `sync` _[SyncConfig](#syncconfig)_ | Sync specifies the resources types that will be synced from virtual cluster to host cluster. | \{ \} | |