mirror of
https://github.com/clastix/kamaji.git
synced 2026-08-26 00:47:20 +00:00
Dario Tranchitella
3230a70475
* feat(migration): customising timeout via tcp annotation Signed-off-by: Dario Tranchitella <dario@tranchitella.eu> * docs: customising migration timeout via tcp annotation Signed-off-by: Dario Tranchitella <dario@tranchitella.eu> * fix(migrate): delete job in case of timeout change This will delete the failed job due to an incorrect timeout and performs the creation of a new object rather than updating it, since its immutability in the API specification. Signed-off-by: Dario Tranchitella <dario@tranchitella.eu> --------- Signed-off-by: Dario Tranchitella <dario@tranchitella.eu>
205 lines
8.7 KiB
Markdown
205 lines
8.7 KiB
Markdown
# Datastore Migration
|
|
|
|
On the Management Cluster, you can deploy one or more multi-tenant datastores as `etcd`, `PostgreSQL`, `MySQL`, and `NATS` to save the state of the Tenant Clusters.
|
|
A Tenant Control Plane can be migrated from a datastore to another one without service disruption or without complex and error-prone backup & restore procedures.
|
|
|
|
This guide will assist you to live migrate Tenant's data from a datastore to another one having the same `etcd` driver.
|
|
|
|
## Prerequisites
|
|
|
|
Assume you have a Tenant Control Plane using the default datastore:
|
|
|
|
``` shell
|
|
kubectl get tcp
|
|
NAME VERSION STATUS CONTROL-PLANE ENDPOINT KUBECONFIG DATASTORE AGE
|
|
tenant-00 v1.25.2 Ready 192.168.32.200:6443 tenant-00-admin-kubeconfig default 8d
|
|
```
|
|
|
|
You can check a custom resource called `DataStore` providing a declarative description of the `default` datastore:
|
|
|
|
```yaml
|
|
apiVersion: kamaji.clastix.io/v1alpha1
|
|
kind: DataStore
|
|
metadata:
|
|
annotations:
|
|
labels:
|
|
name: default
|
|
spec:
|
|
driver: etcd
|
|
endpoints:
|
|
- etcd-0.etcd.kamaji-system.svc.cluster.local:2379
|
|
- etcd-1.etcd.kamaji-system.svc.cluster.local:2379
|
|
- etcd-2.etcd.kamaji-system.svc.cluster.local:2379
|
|
tlsConfig:
|
|
certificateAuthority:
|
|
certificate:
|
|
secretReference:
|
|
keyPath: ca.crt
|
|
name: etcd-certs
|
|
namespace: kamaji-system
|
|
privateKey:
|
|
secretReference:
|
|
keyPath: ca.key
|
|
name: etcd-certs
|
|
namespace: kamaji-system
|
|
clientCertificate:
|
|
certificate:
|
|
secretReference:
|
|
keyPath: tls.crt
|
|
name: etcd-root-client-certs
|
|
namespace: kamaji-system
|
|
privateKey:
|
|
secretReference:
|
|
keyPath: tls.key
|
|
name: etcd-root-client-certs
|
|
namespace: kamaji-system
|
|
status:
|
|
usedBy:
|
|
- default/tenant-00
|
|
```
|
|
|
|
The `default` datastore is installed by Kamaji Helm chart in the same namespace hosting the controller:
|
|
|
|
```shell
|
|
kubectl -n kamaji-system get pods
|
|
NAME READY STATUS RESTARTS AGE
|
|
etcd-0 1/1 Running 0 23d
|
|
etcd-1 1/1 Running 0 23d
|
|
etcd-2 1/1 Running 0 23d
|
|
kamaji-5d6cdfbbb9-bn27f 1/1 Running 0 2d19h
|
|
```
|
|
|
|
## Install a new datastore
|
|
A managed datastore is highly recommended in production. The [kamaji-etcd](https://github.com/clastix/kamaji-etcd) project provides a viable option to setup a managed multi-tenant `etcd` running as StatefulSet made of three replicas:
|
|
|
|
```bash
|
|
helm repo add clastix https://clastix.github.io/charts
|
|
helm repo update
|
|
helm install dedicated clastix/kamaji-etcd -n dedicated --create-namespace --set datastore.enabled=true
|
|
```
|
|
|
|
You should end up with a new datastore `dedicated` provided by an `etcd` cluster:
|
|
|
|
```yaml
|
|
# kubectl get datastore dedicated -o yaml
|
|
apiVersion: kamaji.clastix.io/v1alpha1
|
|
kind: DataStore
|
|
metadata:
|
|
annotations:
|
|
labels:
|
|
name: dedicated
|
|
spec:
|
|
driver: etcd
|
|
endpoints:
|
|
- dedicated-0.dedicated.dedicated.svc.cluster.local:2379
|
|
- dedicated-1.dedicated.dedicated.svc.cluster.local:2379
|
|
- dedicated-2.dedicated.dedicated.svc.cluster.local:2379
|
|
tlsConfig:
|
|
certificateAuthority:
|
|
certificate:
|
|
secretReference:
|
|
keyPath: ca.crt
|
|
name: dedicated-certs
|
|
namespace: dedicated
|
|
privateKey:
|
|
secretReference:
|
|
keyPath: ca.key
|
|
name: dedicated-certs
|
|
namespace: dedicated
|
|
clientCertificate:
|
|
certificate:
|
|
secretReference:
|
|
keyPath: tls.crt
|
|
name: dedicated-root-client-certs
|
|
namespace: dedicated
|
|
privateKey:
|
|
secretReference:
|
|
keyPath: tls.key
|
|
name: dedicated-root-client-certs
|
|
namespace: dedicated
|
|
status: {}
|
|
```
|
|
|
|
Check the `etcd` cluster:
|
|
|
|
```bash
|
|
kubectl -n dedicated get sts,pods,pvc
|
|
NAME READY AGE
|
|
statefulset.apps/dedicated 3/3 25h
|
|
|
|
NAME READY STATUS RESTARTS AGE
|
|
pod/dedicated-0 1/1 Running 0 25h
|
|
pod/dedicated-1 1/1 Running 0 25h
|
|
pod/dedicated-2 1/1 Running 0 25h
|
|
|
|
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
|
|
persistentvolumeclaim/data-dedicated-0 Bound pvc-a5c66737-ef78-4689-b863-037f8382ed78 10Gi RWO local-path 25h
|
|
persistentvolumeclaim/data-dedicated-1 Bound pvc-1e9f77eb-89f3-4256-9508-c18b71fca7ea 10Gi RWO local-path 25h
|
|
persistentvolumeclaim/data-dedicated-2 Bound pvc-957c4802-1e7c-4f37-ac01-b89ad1fa9fdb 10Gi RWO local-path 25h
|
|
```
|
|
|
|
## Migrate data
|
|
To migrate data from current `default` datastore to the new dedicated one, patch the Tenant Control Plane `tenant-00` to use the new `dedicated` datastore:
|
|
|
|
```shell
|
|
kubectl patch --type merge tcp tenant-00 -p '{"spec": {"dataStore": "dedicated"}}'
|
|
```
|
|
|
|
and check the process happening in real time:
|
|
|
|
```shell
|
|
kubectl get tcp -w
|
|
NAME VERSION STATUS CONTROL-PLANE ENDPOINT KUBECONFIG DATASTORE AGE
|
|
tenant-00 v1.25.2 Ready 192.168.32.200:6443 tenant-00-admin-kubeconfig default 9d
|
|
tenant-00 v1.25.2 Migrating 192.168.32.200:6443 tenant-00-admin-kubeconfig default 9d
|
|
tenant-00 v1.25.2 Migrating 192.168.32.200:6443 tenant-00-admin-kubeconfig default 9d
|
|
tenant-00 v1.25.2 Migrating 192.168.32.200:6443 tenant-00-admin-kubeconfig dedicated 9d
|
|
tenant-00 v1.25.2 Migrating 192.168.32.200:6443 tenant-00-admin-kubeconfig dedicated 9d
|
|
tenant-00 v1.25.2 Ready 192.168.32.200:6443 tenant-00-admin-kubeconfig dedicated 9d
|
|
```
|
|
|
|
During the datastore migration, the Tenant Control Plane is put in read-only mode to avoid misalignments between source and destination datastores. If tenant users try to update the data, an admission controller denies the request with the following message:
|
|
|
|
```shell
|
|
Error from server (the current Control Plane is in freezing mode due to a maintenance mode,
|
|
all the changes are blocked: removing the webhook may lead to an inconsistent state upon its completion):
|
|
admission webhook "catchall.migrate.kamaji.clastix.io" denied the request
|
|
```
|
|
|
|
After a while, depending on the amount of data to migrate, the Tenant Control Plane is put back in full operating mode by the Kamaji controller.
|
|
|
|
Migration is expected to complete in 5 minutes.
|
|
However, that timeout can be customized at the `TenantControlPlane` level with the annotation `kamaji.clastix.io/migration-timeout` with a Go-duration value (e.g.: `5m`).
|
|
|
|
!!! info "Leftover"
|
|
Please, note the datastore migration leaves the data on the default datastore, so you have to remove it manually.
|
|
|
|
!!! info "Avoiding stale DataStore content"
|
|
When migrating `TenantControlPlane` across DataStore, a collision with the __schema__ name could happen,
|
|
leading to unexpected results such as old data still available.
|
|
The annotation `kamaji.clastix.io/cleanup-prior-migration=true` allows to enforce the clean-up of the target `DataStore` schema in case of collision.
|
|
|
|
## Post migration
|
|
After migrating data to the new datastore, complete the migration procedure by restarting the `kubelet.service` on all the tenant worker nodes.
|
|
|
|
## Troubleshooting
|
|
|
|
### Migration Job Image Version
|
|
When migrating between datastores, the Kamaji controller automatically creates a migration job to transfer data from the source to the destination datastore. By default, this job uses the same image version as the running Kamaji controller. If you need to use a different image version for the migration job, you can specify it by passing extra arguments to the controller:
|
|
|
|
```shell
|
|
helm upgrade kamaji clastix/kamaji --version ${CHART_VERSION} -n kamaji-system
|
|
--set extraArgs[0]=--migrate-image=custom/kamaji:version`
|
|
```
|
|
|
|
### Handling Private Registry Images
|
|
If the Kamaji controller images are stored in a private registry that requires authentication, the migration job will fail because it does not use any `ImagePullSecret` by default. You need to attach your registry secret to the `kamaji-controller-manager` service account, which is used by the migration job. You can do this with the following command:
|
|
|
|
```shell
|
|
kubectl -n kamaji-system patch serviceaccount kamaji-controller-manager \
|
|
-p '{"imagePullSecrets": [{"name": "myregistry-credentials"}]}'
|
|
```
|
|
|
|
This command patches the kamaji-controller-manager service account to include your registry secret, allowing the migration job to pull images from the private registry successfully.
|
|
|