feat: add support for multiple Datastores (#961)

* feat: add support for multiple Datastores

* docs: add guide for datastore overrides

* feat(datastore): add e2e test for dataStoreOverrides

* ci: reclaim disk space from runner to fix flaky tests

docs/content/guides/datastore-overrides.md

@@ -0,0 +1,78 @@
+# Datastore Overrides
+
+Kamaji offers the possibility of having multiple ETCD clusters backing different resources of the k8s api server by configuring the [`--etcd-servers-overrides`](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/#:~:text=%2D%2Detcd%2Dservers%2Doverrides%20strings) flag. This feature can be useful for massive clusters to store resources with high churn in a dedicated ETCD cluster.
+
+## Install Datastores
+
+Create a self-signed cert-manager `ClusterIssuer`.
+```bash
+echo 'apiVersion: cert-manager.io/v1
+kind: ClusterIssuer
+metadata:
+  name: self-signed
+spec:
+  selfSigned: {}
+' | kubectl apply -f -
+```
+
+Install two Datastores, a primary and a secondary that will be used for `/events` resources.
+```bash
+ helm install etcd-primary clastix/kamaji-etcd -n kamaji-etcd --create-namespace \
+   --set selfSignedCertificates.enabled=false \
+   --set certManager.enabled=true \
+   --set certManager.issuerRef.kind=ClusterIssuer \
+   --set certManager.issuerRef.name=self-signed
+```
+
+For the secondary Datastore, use the cert-manager CA created by the `etcd-primary` helm release.
+```bash
+ helm install etcd-secondary clastix/kamaji-etcd -n kamaji-etcd --create-namespace \
+   --set selfSignedCertificates.enabled=false \
+   --set certManager.enabled=true \
+   --set certManager.ca.create=false \
+   --set certManager.ca.nameOverride=etcd-primary-kamaji-etcd-ca \
+   --set certManager.issuerRef.kind=ClusterIssuer \
+   --set certManager.issuerRef.name=self-signed
+```
+
+## Create a Tenant Control Plane
+
+Using the `spec.dataStoreOverrides` field, Datastores different from the one used in `spec.dataStore` can be used to store specific resources.
+
+```bash
+echo 'apiVersion: kamaji.clastix.io/v1alpha1
+kind: TenantControlPlane
+metadata:
+  name: k8s-133
+  labels:
+    tenant.clastix.io: k8s-133
+spec:
+  controlPlane:
+    deployment:
+      replicas: 2
+    service:
+      serviceType: LoadBalancer
+  kubernetes:
+    version: "v1.33.1"
+    kubelet:
+      cgroupfs: systemd
+  dataStore: etcd-primary-kamaji-etcd
+  dataStoreOverrides:
+    - resource: "/events" # Store events in the secondary ETCD
+      dataStore: etcd-secondary-kamaji-etcd
+  networkProfile:
+    port: 6443
+  addons:
+    coreDNS: {}
+    kubeProxy: {}
+    konnectivity:
+      server:
+        port: 8132
+      agent:
+        mode: DaemonSet
+' | k apply -f -
+```
+
+## Considerations
+
+Only built-in resources can be tagetted by `--etcd-servers-overrides`, it is currently not possible to target Custom Resources.

docs/content/reference/api.md

@@ -28500,6 +28500,13 @@ Migration from one DataStore to another backed by the same Driver is possible. S
 Migration from one DataStore to another backed by a different Driver is not supported.<br/>
         </td>
         <td>false</td>
+      </tr><tr>
+        <td><b><a href="#tenantcontrolplanespecdatastoreoverridesindex">dataStoreOverrides</a></b></td>
+        <td>[]object</td>
+        <td>
+          DataStoreOverride defines which kubernetes resources will be stored in dedicated datastores.<br/>
+        </td>
+        <td>false</td>
       </tr><tr>
         <td><b>dataStoreSchema</b></td>
         <td>string</td>
@@ -42214,6 +42221,38 @@ In case this value is set, kubeadm does not change automatically the version of
 </table>
 
 
+<span id="tenantcontrolplanespecdatastoreoverridesindex">`TenantControlPlane.spec.dataStoreOverrides[index]`</span>
+
+
+DataStoreOverride defines which kubernetes resource will be stored in a dedicated datastore.
+
+<table>
+    <thead>
+        <tr>
+            <th>Name</th>
+            <th>Type</th>
+            <th>Description</th>
+            <th>Required</th>
+        </tr>
+    </thead>
+    <tbody><tr>
+        <td><b>dataStore</b></td>
+        <td>string</td>
+        <td>
+          DataStore specifies the DataStore that should be used to store the Kubernetes data for the given Resource.<br/>
+        </td>
+        <td>false</td>
+      </tr><tr>
+        <td><b>resource</b></td>
+        <td>string</td>
+        <td>
+          Resource specifies which kubernetes resource to target.<br/>
+        </td>
+        <td>false</td>
+      </tr></tbody>
+</table>
+
+
 <span id="tenantcontrolplanespecnetworkprofile">`TenantControlPlane.spec.networkProfile`</span>
 
 

docs/mkdocs.yml

@@ -76,6 +76,7 @@ nav:
   - guides/pausing.md
   - guides/write-permissions.md
   - guides/datastore-migration.md
+  - guides/datastore-overrides.md
   - guides/gitops.md
   - guides/console.md
   - guides/kubeconfig-generator.md