github/kamaji
1
0
Fork 0
mirror of https://github.com/clastix/kamaji.git synced 2026-04-15 06:56:47 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(docs): refactoring (#784)

Browse Source 
* feat(docs): add landing page

* feat(docs): refactoring
This commit is contained in:
Adriano Pezzuto
2025-04-16 11:13:35 +02:00
committed by GitHub
parent 223aa6d4c9
commit 69141e5765
42 changed files with 828 additions and 323 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
docs/content/getting-started/index.md
View File

@@ -1,6 +1,16 @@
# Getting started
This section contains the information on how to get started with Kamaji
This section contains how to get started with Kamaji on different environments:
!!! success "Slow Start"
The material provided in this section is intended to be a slow start to Kamaji.
It is intended to be a deep learning experience, and to help you getting started with Kamaji while understanding the components involved and the core concepts behind it. We do not provide any "one-click" deployment here.
- [Getting started with Kamaji on Kind](./kamaji-kind.md)
- [Getting started with Kamaji on generic infra](./kamaji-generic.md)
- [Getting started with Kamaji on EKS](./kamaji-aws.md)
- [Getting started with Kamaji on AKS](./kamaji-azure.md)
- [Getting started with Kamaji](getting-started.md): install the required components and Kamaji on any Kubernetes cluster
- [Kamaji: Getting started on Kind](kind.md): useful for development environments, create a Kamaji environment on `kind`

415
docs/content/getting-started/kamaji-aws.md Normal file
View File

@@ -0,0 +1,415 @@
# Kamaji on AWS
This guide will lead you through the process of creating a working Kamaji setup on on AWS.
The guide requires:
- a bootstrap machine
- a Kubernetes cluster (EKS) to run the Management and Tenant Control Planes
- an arbitrary number of machines to host Tenant workloads.
## Summary
* [Prepare the bootstrap workspace](#prepare-the-bootstrap-workspace)
* [Access Management Cluster](#access-management-cluster)
* [Install Kamaji](#install-kamaji)
* [Create Tenant Cluster](#create-tenant-cluster)
* [Cleanup](#cleanup)
## Prepare the bootstrap workspace
On the bootstrap machine, clone the repo and prepare the workspace directory:
```bash
git clone https://github.com/clastix/kamaji
cd kamaji/deploy
```
We assume you have installed on the bootstrap machine:
- [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)
- [helm](https://helm.sh/docs/intro/install/)
- [jq](https://stedolan.github.io/jq/)
- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
- [eksctl](https://eksctl.io/installation/)
- [clusterawsadm](https://github.com/kubernetes-sigs/cluster-api-provider-aws/releases)
Make sure you have a valid AWS Account, and login to AWS:
```bash
aws configure
```
## Access Management cluster
In Kamaji, a Management Cluster is a regular Kubernetes cluster which hosts zero to many Tenant Cluster Control Planes. The Management Cluster acts as a cockpit for all the Tenant clusters and implements monitoring, logging, and governance of all the Kamaji setups, including all Tenant Clusters. For this guide, we're going to use an instance of AWS Kubernetes Service (EKS) as a Management Cluster.
Throughout the following instructions, shell variables are used to indicate values that you should adjust to your own AWS environment:
### Create EKS cluster
In order to create quickly an EKS cluster, we will use `eksctl` provided by AWS. `eksctl` is a simple CLI tool for creating and managing clusters on EKS
`eksctl` will provision for you:
- A dedicated VPC on `192.168.0.0/16` CIDR
- 3 private subnets and 3 public subnets in 3 different availability zones
- NAT Gateway for the private subnets, An internet gateway for the public ones
- The required route tables to associate the subnets with the IGW and the NAT gateways
- Provision the EKS cluster
- Provision worker nodes and associate them to your cluster
- Optionally creates the required IAM policies for your addons and attach them to the node
- Optionally, install the EKS add-ons to your cluster
For our use case, we will create an EKS cluster with the following configuration:
```bash
source kamaji-aws.env
cat > eks-cluster.yaml <<EOF
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: ${KAMAJI_CLUSTER}
region: ${KAMAJI_REGION}
version: ${KAMAJI_CLUSTER_VERSION}
iam:
withOIDC: true
vpc:
clusterEndpoints:
privateAccess: true
publicAccess: true
managedNodeGroups:
- name: ${KAMAJI_NODE_NG}
labels: { role: workers }
instanceType: ${KAMAJI_NODE_TYPE}
desiredCapacity: 1
privateNetworking: true
availabilityZones: [${KAMAJI_AZ}]
iam:
withAddonPolicies:
certManager: true
ebs: true
externalDNS: true
addons:
- name: aws-ebs-csi-driver
EOF
eks create cluster -f eks-cluster.yaml
```
Please note :
- The `aws-ebs-csi-driver` addon is required to use EBS volumes as persistent volumes. This will be mainly used to store the tenant control plane data using the _default_ `etcd` DataStore.
- We created a node group with 1 node in one availability zone to simplify the setup.
### Access to the management cluster
And check you can access:
```bash
aws eks update-kubeconfig --region ${KAMAJI_REGION} --name ${KAMAJI_CLUSTER}
kubectl cluster-info
# make ebs as a default storage class
kubectl patch storageclass gp2 -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
```
### Add route 53 domain
In order to easily access tenant clusters, it is recommended to create a Route53 domain or use an existing one if it exists
```bash
# for within VPC
aws route53 create-hosted-zone --name "$TENANT_DOMAIN" --caller-reference $(date +%s) --vpc "VPCRegion=$KAMAJI_REGION,VPCId=$KAMAJI_VPC_ID"
```
## Install Kamaji
Follow the [Getting Started](kamaji-generic.md) to install Cert Manager and the Kamaji Controller.
### Install Cert Manager
Kamaji takes advantage of the [dynamic admission control](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/), such as validating and mutating webhook configurations. These webhooks are secured by a TLS communication, and the certificates are managed by [`cert-manager`](https://cert-manager.io/), making it a prerequisite that must be installed:
```bash
helm repo add jetstack https://charts.jetstack.io
helm repo update
helm install \
cert-manager jetstack/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.11.0 \
--set installCRDs=true
```
### Install ExternalDNS (optional)
ExternalDNS allows updating your DNS records dynamically from an annotation that you add in the service within EKS. Run the following commands to install the ExternalDNS Helm chart:
```bash
helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/
helm repo update
helm install external-dns external-dns/external-dns \
--namespace external-dns \
--create-namespace \
--version 1.15.1
```
### Install Kamaji Controller
Installing Kamaji via Helm charts is the preferred way. Run the following commands to install a stable release of Kamaji:
```bash
helm repo add clastix https://clastix.github.io/charts
helm repo update
helm install kamaji clastix/kamaji -n kamaji-system --create-namespace
```
## Create Tenant Cluster
Now that our management cluster is up and running, we can create a Tenant Cluster. A Tenant Cluster is a Kubernetes cluster that is managed by Kamaji.
### Tenant Control Plane
A tenant cluster is made of a `Tenant Control Plane` and an arbitrary number of worker nodes. The `Tenant Control Plane` is a Kubernetes Control Plane managed by Kamaji and responsible for running the Tenant's workloads.
Before creating a Tenant Control Plane, you need to define some variables:
```bash
export KAMAJI_VPC_ID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=$KAMAJI_VPC_NAME" --query "Vpcs[0].VpcId" --output text)
export KAMAJI_PUBLIC_SUBNET_ID=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=$KAMAJI_VPC_ID" --filters "Name=tag:Name,Values=$KAMAJI_PUBLIC_SUBNET_NAME" --query "Subnets[0].SubnetId" --output text)
export TENANT_EIP_ID=$(aws ec2 allocate-address --query 'AllocationId' --output text)
export TENANT_PUBLIC_IP=$(aws ec2 describe-addresses --allocation-ids $TENANT_EIP_ID --query 'Addresses[0].PublicIp' --output text)
```
In the next step, we will create a Tenant Control Plane with the following configuration:
```yaml
cat > ${TENANT_NAMESPACE}-${TENANT_NAME}.yaml <<EOF
apiVersion: v1
kind: Namespace
metadata:
name: ${TENANT_NAMESPACE}
---
apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
name: ${TENANT_NAME}
namespace: ${TENANT_NAMESPACE}
labels:
tenant.clastix.io: ${TENANT_NAME}
spec:
dataStore: default
controlPlane:
deployment:
replicas: 1
nodeSelector:
topology.kubernetes.io/zone: ${KAMAJI_AZ}
additionalMetadata:
labels:
tenant.clastix.io: ${TENANT_NAME}
extraArgs:
apiServer: []
controllerManager: []
scheduler: []
resources:
apiServer:
requests:
cpu: 250m
memory: 512Mi
limits: {}
controllerManager:
requests:
cpu: 125m
memory: 256Mi
limits: {}
scheduler:
requests:
cpu: 125m
memory: 256Mi
limits: {}
service:
additionalMetadata:
labels:
tenant.clastix.io: ${TENANT_NAME}
annotations:
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
service.beta.kubernetes.io/aws-load-balancer-subnets: ${KAMAJI_PUBLIC_SUBNET_ID}
service.beta.kubernetes.io/aws-load-balancer-eip-allocations: ${TENANT_EIP_ID}
service.beta.kubernetes.io/aws-load-balancer-type: nlb
external-dns.alpha.kubernetes.io/hostname: ${TENANT_NAME}.${TENANT_DOMAIN}
serviceType: LoadBalancer
kubernetes:
version: ${TENANT_VERSION}
kubelet:
cgroupfs: systemd
admissionControllers:
- ResourceQuota
- LimitRanger
networkProfile:
address: ${TENANT_PUBLIC_IP}
port: ${TENANT_PORT}
certSANs:
- ${TENANT_NAME}.${TENANT_DOMAIN}
serviceCidr: ${TENANT_SVC_CIDR}
podCidr: ${TENANT_POD_CIDR}
dnsServiceIPs:
- ${TENANT_DNS_SERVICE}
addons:
coreDNS: {}
kubeProxy: {}
konnectivity:
server:
port: ${TENANT_PROXY_PORT}
resources:
requests:
cpu: 100m
memory: 128Mi
limits: {}
EOF
kubectl -n ${TENANT_NAMESPACE} apply -f ${TENANT_NAMESPACE}-${TENANT_NAME}.yaml
```
Make sure:
- Tenant Control Plane will expose the API server using a public IP address through a network load balancer.
it is important to provide a static public IP address for the API server in order to make it reachable from the outside world.
- The following annotation: `external-dns.alpha.kubernetes.io/hostname` is set to create the DNS record. It tells AWS to expose the Tenant Control Plane with a public domain name: `${TENANT_NAME}.${TENANT_DOMAIN}`.
Since AWS load Balancer does not support setting LoadBalancerIP, you will get the following warning on the service created for the control plane tenant `Error syncing load balancer: failed to ensure load balancer: LoadBalancerIP cannot be specified for AWS ELB`. you can ignore it for now.
### Working with Tenant Control Plane
Check the access to the Tenant Control Plane:
```bash
curl -k https://${TENANT_PUBLIC_IP}:${TENANT_PORT}/version
curl -k https://${TENANT_NAME}.${TENANT_DOMAIN}:${TENANT_PORT}/healthz
curl -k https://${TENANT_NAME}.${TENANT_DOMAIN}:${TENANT_PORT}/version
```
!!! warning "Using Private Domains"
If the domain you used is a private __Route 53__ domain make sure to map the public IP of the LoadBalancer to `${TENANT_NAME}.${TENANT_DOMAIN}` in your `/etc/hosts`. Otherwise, `kubectl` will fail to check SSL certificates
Let's retrieve the `kubeconfig` in order to work with it:
```bash
kubectl get secrets -n ${TENANT_NAMESPACE} ${TENANT_NAME}-admin-kubeconfig -o json \
| jq -r '.data["admin.conf"]' \
| base64 --decode \
> ${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig
kubectl --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig config \
set-cluster ${TENANT_NAME} \
--server https://${TENANT_NAME}.${TENANT_DOMAIN}:${TENANT_PORT}
```
and let's check it out:
```bash
kubectl --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 38h
```
Check out how the Tenant Control Plane advertises itself:
```bash
kubectl --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig get ep
NAME ENDPOINTS AGE
kubernetes 13.37.33.12:6443 3m22s
```
### Join worker nodes
The Tenant Control Plane is made of pods running in the Kamaji Management Cluster. At this point, the Tenant Cluster has no worker nodes. So, the next step is to join some worker nodes to the Tenant Control Plane.
Kamaji does not provide any helper for the creation of tenant worker nodes, instead, it leverages the [Cluster Management API](https://github.com/kubernetes-sigs/cluster-api). This allows you to create the Tenant Clusters, including worker nodes, in a completely declarative way. Currently, a Cluster API `ControlPlane` provider for AWS is available: check the [official documentation](https://github.com/clastix/cluster-api-control-plane-provider-kamaji/blob/master/docs/providers-aws.md).
An alternative approach to create and join worker nodes in AWS is to manually create the VMs, turn them into Kubernetes worker nodes and then join through the `kubeadm` command.
### Generate kubeadm join command
To join the worker nodes to the Tenant Control Plane, you need to generate the `kubeadm join` command from the Management cluster:
```bash
TENANT_ADDR=$(kubectl -n ${TENANT_NAMESPACE} get svc ${TENANT_NAME} -o json | jq -r ."spec.loadBalancerIP")
JOIN_CMD=$(echo "sudo kubeadm join ${TENANT_ADDR}:6443 ")$(kubeadm --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig token create --ttl 0 --print-join-command |cut -d" " -f4-)
```
!!! tip "Token expiration"
Setting `--ttl=0` on the `kubeadm token create` will guarantee that the token will never expire and can be used every time. It's not intended for production-grade setups.
### Create tenant worker nodes
In this section, we will use AMI provided by CAPA (Cluster API Provider AWS) to create the worker nodes. Those AMIs are built using [image builder](https://github.com/kubernetes-sigs/image-builder/tree/main) and contain all the necessary components to join the cluster.
```bash
export KAMAJI_PRIVATE_SUBNET_ID=$(aws ec2 describe-subnets --filters "Name=vpc-id,Values=$KAMAJI_VPC_ID" --filters "Name=tag:Name,Values=$KAMAJI_PRIVATE_SUBNET_NAME" --query "Subnets[0].SubnetId" --output text)
export WORKER_AMI=$(clusterawsadm ami list --kubernetes-version=$TENANT_VERSION --os=ubuntu-24.04 --region=$KAMAJI_REGION -o json | jq -r .items[0].spec.imageID)
cat <<EOF >> worker-user-data.sh
#!/bin/bash
$JOIN_CMD
EOF
aws ec2 run-instances --image-id $WORKER_AMI --instance-type "t2.medium" --user-data $(cat worker-user-data.sh | base64 -w0) --network-interfaces '{"SubnetId":'"'${KAMAJI_PRIVATE_SUBNET_ID}'"',"AssociatePublicIpAddress":false,"DeviceIndex":0,"Groups":["<REPLACE_WITH_SG>"]}' --count "1"
```
We have used user data to run the `kubeadm join` command on the instance boot. This will make sure that the worker node will join the cluster automatically.
Make sure to replace `<REPLACE_WITH_SG>` with the security group id that allows the worker nodes to communicate with the public IP of the tenant control plane
Checking the nodes in the Tenant Cluster:
```bash
kubectl --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-153-94 NotReady <none> 56m v1.30.2
```
The cluster needs a [CNI](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) plugin to get the nodes ready. In this guide, we are going to install [calico](https://projectcalico.docs.tigera.io/about/about-calico), but feel free to use one of your taste.
Download the latest stable Calico manifest:
```bash
curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.1/manifests/calico.yaml -O
```
As per [documentation](https://projectcalico.docs.tigera.io/reference/public-cloud/AWS), Calico in VXLAN mode is supported on AWS while IPIP packets are blocked by the AWS network fabric. Make sure you edit the manifest above and set the following variables:
- `CLUSTER_TYPE="k8s"`
- `CALICO_IPV4POOL_IPIP="Never"`
- `CALICO_IPV4POOL_VXLAN="Always"`
Apply to the Tenant Cluster:
```bash
kubectl --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig apply -f calico.yaml
```
And after a while, nodes will be ready
```bash
kubectl --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig get nodes
NAME STATUS ROLES AGE VERSION
ip-192-168-153-94 Ready <none> 59m v1.30.2
```
## Cleanup
To get rid of the whole Kamaji infrastructure, remove the EKS cluster:
```bash
eksctl delete cluster -f eks-cluster.yaml
```
That's all folks!

367
docs/content/getting-started/kamaji-azure.md Normal file
View File

@@ -0,0 +1,367 @@
# Kamaji on Azure
This guide will lead you through the process of creating a working Kamaji setup on on MS Azure.
The guide requires:
- a bootstrap machine
- a Kubernetes cluster (AKS) to run the Management and Tenant Control Planes
- an arbitrary number of machines to host Tenant workloads.
## Summary
* [Prepare the bootstrap workspace](#prepare-the-bootstrap-workspace)
* [Access Management Cluster](#access-management-cluster)
* [Install Kamaji](#install-kamaji)
* [Create Tenant Cluster](#create-tenant-cluster)
* [Cleanup](#cleanup)
## Prepare the bootstrap workspace
On the bootstrap machine, clone the repo and prepare the workspace directory:
```bash
git clone https://github.com/clastix/kamaji
cd kamaji/deploy
```
We assume you have installed on the bootstrap machine:
- [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl)
- [kubeadm](https://kubernetes.io/docs/tasks/tools/#kubeadm)
- [helm](https://helm.sh/docs/intro/install/)
- [jq](https://stedolan.github.io/jq/)
- [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli)
Make sure you have a valid Azure subscription, and login to Azure:
```bash
az account set --subscription "MySubscription"
az login
```
## Access Management Cluster
In Kamaji, a Management Cluster is a regular Kubernetes cluster which hosts zero to many Tenant Cluster Control Planes. The Management Cluster acts as cockpit for all the Tenant clusters and implements Monitoring, Logging, and Governance of all the Kamaji setup, including all Tenant Clusters. For this guide, we're going to use an instance of Azure Kubernetes Service (AKS) as Management Cluster.
Throughout the following instructions, shell variables are used to indicate values that you should adjust to your own Azure environment:
```bash
source kamaji-azure.env
az group create \
--name $KAMAJI_RG \
--location $KAMAJI_REGION
az network vnet create \
--resource-group $KAMAJI_RG \
--name $KAMAJI_VNET_NAME \
--location $KAMAJI_REGION \
--address-prefix $KAMAJI_VNET_ADDRESS
az network vnet subnet create \
--resource-group $KAMAJI_RG \
--name $KAMAJI_SUBNET_NAME \
--vnet-name $KAMAJI_VNET_NAME \
--address-prefixes $KAMAJI_SUBNET_ADDRESS
KAMAJI_SUBNET_ID=$(az network vnet subnet show \
--resource-group ${KAMAJI_RG} \
--vnet-name ${KAMAJI_VNET_NAME} \
--name ${KAMAJI_SUBNET_NAME} \
--query id --output tsv)
az aks create \
--resource-group $KAMAJI_RG \
--name $KAMAJI_CLUSTER \
--location $KAMAJI_REGION \
--vnet-subnet-id $KAMAJI_SUBNET_ID \
--zones 1 2 3 \
--node-count 3 \
--nodepool-name $KAMAJI_CLUSTER
```
Once the cluster formation succedes, get credentials to access the cluster as admin
```bash
az aks get-credentials \
--resource-group $KAMAJI_RG \
--name $KAMAJI_CLUSTER
```
And check you can access:
```bash
kubectl cluster-info
```
## Install Kamaji
Follow the [Getting Started](kamaji-generic.md) to install Cert Manager and the Kamaji Controller.
## Create Tenant Cluster
### Tenant Control Plane
With Kamaji on AKS, the tenant control plane is accessible:
- from tenant worker nodes through an internal loadbalancer
- from tenant admin user through an external loadbalancer responding to `https://${TENANT_NAME}.${TENANT_NAME}.${TENANT_DOMAIN}:443`
Create a tenant control plane of example:
```yaml
cat > ${TENANT_NAMESPACE}-${TENANT_NAME}-tcp.yaml <<EOF
apiVersion: kamaji.clastix.io/v1alpha1
kind: TenantControlPlane
metadata:
name: ${TENANT_NAME}
namespace: ${TENANT_NAMESPACE}
labels:
tenant.clastix.io: ${TENANT_NAME}
spec:
dataStore: default
controlPlane:
deployment:
replicas: 3
additionalMetadata:
labels:
tenant.clastix.io: ${TENANT_NAME}
extraArgs:
apiServer: []
controllerManager: []
scheduler: []
resources:
apiServer:
requests:
cpu: 250m
memory: 512Mi
limits: {}
controllerManager:
requests:
cpu: 125m
memory: 256Mi
limits: {}
scheduler:
requests:
cpu: 125m
memory: 256Mi
limits: {}
service:
additionalMetadata:
labels:
tenant.clastix.io: ${TENANT_NAME}
annotations:
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
serviceType: LoadBalancer
kubernetes:
version: ${TENANT_VERSION}
kubelet:
cgroupfs: systemd
admissionControllers:
- ResourceQuota
- LimitRanger
networkProfile:
port: ${TENANT_PORT}
certSANs:
- ${TENANT_NAME}.${TENANT_DOMAIN}
serviceCidr: ${TENANT_SVC_CIDR}
podCidr: ${TENANT_POD_CIDR}
dnsServiceIPs:
- ${TENANT_DNS_SERVICE}
addons:
coreDNS: {}
kubeProxy: {}
konnectivity:
server:
port: ${TENANT_PROXY_PORT}
resources:
requests:
cpu: 100m
memory: 128Mi
limits: {}
---
apiVersion: v1
kind: Service
metadata:
name: ${TENANT_NAME}-public
namespace: ${TENANT_NAMESPACE}
annotations:
service.beta.kubernetes.io/azure-dns-label-name: ${TENANT_NAME}
spec:
ports:
- port: 443
protocol: TCP
targetPort: ${TENANT_PORT}
selector:
kamaji.clastix.io/name: ${TENANT_NAME}
type: LoadBalancer
EOF
kubectl -n ${TENANT_NAMESPACE} apply -f ${TENANT_NAMESPACE}-${TENANT_NAME}-tcp.yaml
```
Make sure:
- the following annotation: `service.beta.kubernetes.io/azure-load-balancer-internal=true` is set on the `tcp` service. It tells Azure to expose the service within an internal loadbalancer.
- the following annotation: `service.beta.kubernetes.io/azure-dns-label-name=${TENANT_NAME}` is set the public loadbalancer service. It tells Azure to expose the Tenant Control Plane with public domain name: `${TENANT_NAME}.${TENANT_DOMAIN}`.
### Working with Tenant Control Plane
Check the access to the Tenant Control Plane:
```bash
curl -k https://${TENANT_NAME}.${KAMAJI_REGION}.cloudapp.azure.com/healthz
curl -k https://${TENANT_NAME}.${KAMAJI_REGION}.cloudapp.azure.com/version
```
Let's retrieve the `kubeconfig` in order to work with it:
```bash
kubectl get secrets -n ${TENANT_NAMESPACE} ${TENANT_NAME}-admin-kubeconfig -o json \
| jq -r '.data["admin.conf"]' \
| base64 --decode \
> ${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig
kubectl --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig config \
set-cluster ${TENANT_NAME} \
--server https://${TENANT_NAME}.${KAMAJI_REGION}.cloudapp.azure.com
```
and let's check it out:
```
kubectl --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig get svc
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.32.0.1 <none> 443/TCP 6m
```
Check out how the Tenant Control Plane advertises itself:
```
kubectl --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig get ep
NAME ENDPOINTS AGE
kubernetes 10.240.0.100:6443 57m
```
### Join worker nodes
The Tenant Control Plane is made of pods running in the Kamaji Management Cluster. At this point, the Tenant Cluster has no worker nodes. So, the next step is to join some worker nodes to the Tenant Control Plane.
Kamaji does not provide any helper for creation of tenant worker nodes, instead it leverages the [Cluster Management API](https://github.com/kubernetes-sigs/cluster-api). This allows you to create the Tenant Clusters, including worker nodes, in a completely declarative way. Currently, a Cluster API `ControlPlane` provider for Azure is not yet available: check the road-map on the [official repository](https://github.com/clastix/cluster-api-control-plane-provider-kamaji).
An alternative approach to create and join worker nodes in Azure is to manually create the VMs, turn them into Kubernetes worker nodes and then join through the `kubeadm` command.
Create an Azure VM Stateful Set to host worker nodes
```bash
az network vnet subnet create \
--resource-group $KAMAJI_RG \
--name $TENANT_SUBNET_NAME \
--vnet-name $KAMAJI_VNET_NAME \
--address-prefixes $TENANT_SUBNET_ADDRESS
az vmss create \
--name $TENANT_VMSS \
--resource-group $KAMAJI_RG \
--image $TENANT_VM_IMAGE \
--vnet-name $KAMAJI_VNET_NAME \
--subnet $TENANT_SUBNET_NAME \
--computer-name-prefix $TENANT_NAME- \
--load-balancer "" \
--instance-count 0
az vmss update \
--resource-group $KAMAJI_RG \
--name $TENANT_VMSS \
--set virtualMachineProfile.networkProfile.networkInterfaceConfigurations[0].enableIPForwarding=true
az vmss scale \
--resource-group $KAMAJI_RG \
--name $TENANT_VMSS \
--new-capacity 3
```
Once all the machines are ready, follow the related [documentation](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/) in order to:
- install `containerd` as container runtime
- install `crictl`, the command line for working with `containerd`
- install `kubectl`, `kubelet`, and `kubeadm` in the desired version
After the installation is complete on all the nodes, store the entire command of joining in a variable:
```bash
TENANT_ADDR=$(kubectl -n ${TENANT_NAMESPACE} get svc ${TENANT_NAME} -o json | jq -r ."spec.loadBalancerIP")
JOIN_CMD=$(echo "sudo kubeadm join ${TENANT_ADDR}:6443 ")$(kubeadm --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig token create --print-join-command |cut -d" " -f4-)
```
Use a loop to log in to and run the join command on each node:
```bash
VMIDS=($(az vmss list-instances \
--resource-group $KAMAJI_RG \
--name $TENANT_VMSS \
--query [].instanceId \
--output tsv))
for i in ${!VMIDS[@]}; do
VMID=${VMIDS[$i]}
az vmss run-command create \
--name join-tenant-control-plane \
--vmss-name $TENANT_VMSS \
--resource-group $KAMAJI_RG \
--instance-id ${VMID} \
--script "${JOIN_CMD}"
done
```
Checking the nodes:
```bash
kubectl --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig get nodes
NAME STATUS ROLES AGE VERSION
tenant-00-000000 NotReady <none> 112s v1.25.0
tenant-00-000002 NotReady <none> 92s v1.25.0
tenant-00-000003 NotReady <none> 71s v1.25.0
```
The cluster needs a [CNI](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/) plugin to get the nodes ready. In this guide, we are going to install [calico](https://projectcalico.docs.tigera.io/about/about-calico), but feel free to use one of your taste.
Download the latest stable Calico manifest:
```bash
curl https://raw.githubusercontent.com/projectcalico/calico/v3.24.1/manifests/calico.yaml -O
```
As per [documentation](https://projectcalico.docs.tigera.io/reference/public-cloud/azure), Calico in VXLAN mode is supported on Azure while IPIP packets are blocked by the Azure network fabric. Make sure you edit the manifest above and set the following variables:
- `CLUSTER_TYPE="k8s"`
- `CALICO_IPV4POOL_IPIP="Never"`
- `CALICO_IPV4POOL_VXLAN="Always"`
Apply to the Tenant Cluster:
```bash
kubectl --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig apply -f calico.yaml
```
And after a while, nodes will be ready
```bash
kubectl --kubeconfig=${TENANT_NAMESPACE}-${TENANT_NAME}.kubeconfig get nodes
NAME STATUS ROLES AGE VERSION
tenant-00-000000 Ready <none> 3m38s v1.25.0
tenant-00-000002 Ready <none> 3m18s v1.25.0
tenant-00-000003 Ready <none> 2m57s v1.25.0
```
## Cleanup
To get rid of the Kamaji infrastructure, remove the RESOURCE_GROUP:
```
az group delete --name $KAMAJI_RG --yes --no-wait
```
That's all folks!

17
docs/content/getting-started/getting-started.md → docs/content/getting-started/kamaji-generic.md
View File

@@ -1,14 +1,11 @@
# Getting started with Kamaji
# Kamaji on generic infra
This guide will lead you through the process of creating a working Kamaji setup on a generic infrastructure.
!!! info "Slow Start"
The material here is relatively dense. We strongly encourage you to dedicate time to walk through these instructions, with a mind to learning how Kamaji works. We do NOT provide any "one-click" deployment here. However, once you've understood the components involved it is encouraged that you build suitable, auditable GitOps deployment processes around your final infrastructure.
The guide requires:
- a bootstrap machine
- a Kubernetes cluster to run the Admin and Tenant Control Planes
- an arbitrary number of machines to host `Tenant`s' workloads
- a Kubernetes cluster to run the Management and Tenant Control Planes
- an arbitrary number of machines to host Tenant workloads.
## Summary
@@ -75,9 +72,9 @@ helm install \
Installing Kamaji via Helm charts is the preferred way to deploy the Kamaji controller. The Helm chart is available in the `charts` directory of the Kamaji repository.
!!! info "Stable Releases"
As of July 2024 [Clastix Labs](https://github.com/clastix) does no longer publish stable release artifacts. Stable releases are offered on a subscription basis by [CLASTIX](https://clastix.io), the main Kamaji project contributor.
As of July 2024 [Clastix Labs](https://github.com/clastix) no longer publish stable release artifacts. Stable releases are offered on a subscription basis by [CLASTIX](https://clastix.io), the main Kamaji project contributor.
Run the following commands to install latest edge release of Kamaji:
Run the following commands to install the latest edge release of Kamaji:
```bash
git clone https://github.com/clastix/kamaji
@@ -190,7 +187,7 @@ service/tenant-00 LoadBalancer 10.32.132.241 192.168.32.240 6443:32152/T
The regular Tenant Control Plane containers: `kube-apiserver`, `kube-controller-manager`, `kube-scheduler` are running unchanged in the `tcp` pods instead of dedicated machines and they are exposed through a service on the port `6443` of worker nodes in the Management Cluster.
The `LoadBalancer` service type is used to expose the Tenant Control Plane on the assigned `loadBalancerIP` acting as `ControlPlaneEndpoint` for the worker nodes and other clients as, for example, `kubectl`. Service types `NodePort` and `ClusterIP` are still viable options to expose the Tenant Control Plane, depending on the case. High Availability and rolling updates of the Tenant Control Planes are provided by the `tcp` Deployment and all the resources reconcilied by the Kamaji controller.
The `LoadBalancer` service type is used to expose the Tenant Control Plane on the assigned `loadBalancerIP` acting as `ControlPlaneEndpoint` for the worker nodes and other clients as, for example, `kubectl`. Service types `NodePort` and `ClusterIP` are still viable options to expose the Tenant Control Plane, depending on the case. High Availability and rolling updates of the Tenant Control Planes are provided by the `tcp` Deployment and all the resources reconciled by the Kamaji controller.
### Assign a Specific Address to the Tenant Control Plane
@@ -283,7 +280,7 @@ The Tenant Control Plane is made of pods running in the Kamaji Management Cluste
!!! warning "Opening Ports"
To make sure worker nodes can join the Tenant Control Plane, you must allow incoming connections to: `${TENANT_ADDR}:${TENANT_PORT}` and `${TENANT_ADDR}:${TENANT_PROXY_PORT}`
Kamaji does not provide any helper for creation of tenant worker nodes, instead it leverages the [Cluster API](https://github.com/kubernetes-sigs/cluster-api). This allows you to create the Tenant Clusters, including worker nodes, in a completely declarative way. Refer to the [Cluster API guide](guides/cluster-api/index.md) to learn more about Cluster API support in Kamaji.
Kamaji does not provide any helper for creation of tenant worker nodes, instead it leverages the [Cluster API](https://github.com/kubernetes-sigs/cluster-api). This allows you to create the Tenant Clusters, including worker nodes, in a completely declarative way. Refer to the section [Cluster API](../cluster-api/index.md) to learn more about Cluster API support in Kamaji.
An alternative approach for joining nodes is to use the `kubeadm` command on each node. Follow the related [documentation](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/) in order to:

87
docs/content/getting-started/kind.md → docs/content/getting-started/kamaji-kind.md
View File

@@ -1,19 +1,19 @@
# Kamaji: Getting started on Kind
This guide will lead you through the process of creating a setup of a working Kamaji setup using Kind clusters.
# Kamaji on Kind
This guide will lead you through the process of creating a working Kamaji setup using Kind cluster. The guide requires the following installed on your workstation: `docker`, `kind`, `helm`, and `kubectl`.
The guide requires the following installed:
- Docker
- Kind
- Helm
!!! warning "Development Only"
Run Kamaji on kind only for development or learning purposes.
Kamaji is designed to be run on production-grade Kubernetes clusters, such as those provided by cloud providers or on-premises solutions. Kind is not a production-grade Kubernetes cluster, and it is not recommended to run in production environments.
## Summary
* [Creating Kind Cluster](#creating-kind-cluster)
* [Installing Dependencies: Cert-Manager](#installing-dependencies-cert-manager)
* [Installing Cert-Manager](#installing-cert-manager)
* [Installing MetalLb](#installing-metallb)
* [Creating IP Address Pool](#creating-ip-address-pool)
* [Installing Kamaji](#installing-kamaji)
* [Creating Tenant Control Plane](#creating-tenant-control-plane)
## Creating Kind Cluster
@@ -23,37 +23,37 @@ Create a kind cluster.
kind create cluster --name kamaji
```
This will take a short while for the kind cluster to created.
This will take a short while for the kind cluster to be created.
## Installing Dependencies: Cert-Manager
## Installing Cert-Manager
Kamaji has a dependency on Cert Manager, as it uses dynamic admission control, validating and mutating webhook configurations which are secured by a TLS communication, these certificates are managed by `cert-manager`. Hence, it needs to be added.
Add the Bitnami Repo to the Helm Manager.
Add the Bitnami Repo to the Helm Manager.
```
helm repo add bitnami https://charts.bitnami.com/bitnami
```
Install Cert Manager to the cluster using the bitnami charts using Helm --
Install Cert Manager using Helm
```
helm upgrade --install cert-manager bitnami/cert-manager --namespace certmanager-system --create-namespace --set "installCRDs=true"
helm upgrade --install cert-manager bitnami/cert-manager \
--namespace certmanager-system \
--create-namespace \
--set "installCRDs=true"
```
This will install cert-manager to the cluster. You can watch the progress of the installation on the cluster using the command -
This will install cert-manager to the cluster. You can watch the progress of the installation on the cluster using the command
```
kubectl get pods -Aw
```
!!! Info ""
Another pre-requisite is to have a __storage provider__.
Kind by default provides `local-path-provisioner`, but one can have any other CSI Drivers. Since there are ETCD and Control-Planes running, having persistent volumes is essential for the cluster.
## Installing MetalLb
MetalLB is used in order to dynamically assign IP addresses to the components, and also define custom IP Address Pools.
MetalLB is used in order to dynamically assign IP addresses to the components, and also define custom IP Address Pools. Install MetalLb using the `kubectl` command for apply the manifest:
Install MetalLb using the `kubectl` manifest apply command --
```
kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml
```
@@ -63,11 +63,13 @@ This will install MetalLb onto the cluster with all the necessary resources.
## Creating IP Address Pool
Extract the Gateway IP of the network Kind is running on.
```
GW_IP=$(docker network inspect -f '{{range .IPAM.Config}}{{.Gateway}}{{end}}' kind)
```
Modify the IP Address, and create the resource to be added to the cluster to create the IP Address Pool.
Modify the IP Address, and create the resource to be added to the cluster to create the IP Address Pool
```
NET_IP=$(echo ${GW_IP} | sed -E 's|^([0-9]+\.[0-9]+)\..*$|\1|g')
cat << EOF | sed -E "s|172.19|${NET_IP}|g" | kubectl apply -f -
@@ -90,51 +92,68 @@ EOF
## Installing Kamaji
- Add the Clastix Repo in the Helm Repo lists.
- Clone the Kamaji repository
```
helm repo add clastix https://clastix.github.io/charts
helm repo update
git clone https://github.com/clastix/kamaji
cd kamaji
```
- Install Kamaji
- Install Kamaji with Helm
```
helm upgrade --install kamaji clastix/kamaji --namespace kamaji-system --create-namespace --set 'resources=null'
helm upgrade --install kamaji charts/kamaji \
--namespace kamaji-system \
--create-namespace \
--set image.tag=latest \
--set 'resources=null'
```
- Watch the progress of the deployments --
- Watch the progress of the deployments
```
kubectl get pods -Aw
```
- Verify by first checking Kamaji CRDs.
- Verify by first checking Kamaji CRDs
```
kubectl get crds | grep -i kamaji
```
- Install a Tenant Control Plane using the command --
!!! Info "CSI Drivers"
Kamaji requires a __storage provider__ installed on the management cluster. Kind by default provides `local-path-provisioner`, but one can have any other CSI Drivers.
## Creating Tenant Control Plane
- Create a Tenant Control Plane using the command
```
kubectl apply -f https://raw.githubusercontent.com/clastix/kamaji/master/config/samples/kamaji_v1alpha1_tenantcontrolplane.yaml
```
- Watch the progress of the Tenant Control Plane by ---
- Watch the progress of the Tenant Control Plane by
```
kubectl get tcp -w
```
- You can attempt to get the details of the control plane by downloading the kubeconfig file ---
- You can attempt to get the details of the control plane by downloading the `kubeconfig` file
```
# Set the SECRET as KUBECONFIG column listed in the tcp output.
SECRET=""
kubectl get secret $SECRET -o jsonpath='{.data.admin\.conf}'|base64 -d > /tmp/kamaji.conf
```
- Export the KUBECONFIG
- Export the `kubeconfig` file to the environment variable `KUBECONFIG`
```
export KUBECONFIG=/tmp/kamaji.conf
```
- Notice that the `kubectl` version changes, and there is no nodes now.
- Notice that the `kubectl` version changes, and there are no nodes now.
```
kubectl version
kubectl get nodes