12 KiB
R01- Configuring 🎸ROCKY deployment with Flux
The ⚙️OPS team manages 2 distinct envs: ⚗️TEST et 🚜PROD
Thanks to Kustomize
- it creates a base common config
- this common config is overwritten with a ⚗️TEST tenant-specific configuration
- the same applies with a 🚜PROD-specific configuration
💡 This seems complex, but no worries: Flux's CLI handles most of it.
Creating the 🎸ROCKY-dedicated tenant in ⚗️TEST env
- Using the
fluxCLI, we create the file configuring the 🎸ROCKY team's dedicated tenant… - … this file takes place in the
basecommon configuration for both envs
.lab[
k8s@shpod:~/fleet-config-using-flux-XXXXX$ \
mkdir -p ./tenants/base/rocky && \
flux create tenant rocky \
--with-namespace=rocky-test \
--cluster-role=rocky-full-access \
--export > ./tenants/base/rocky/rbac.yaml
]
class: extra-details
📂 ./tenants/base/rocky/rbac.yaml
Let's see our file…
3 resources are created: Namespace, ServiceAccount, and ClusterRoleBinding
Flux impersonates as this ServiceAccount when it applies any resources found in this tenant-dedicated source(s)
- By default, the
ServiceAccountis bound to thecluster-adminClusterRole - The team maintaining the sourced
Githubrepository is almighty at cluster scope
A not that much isolated tenant! 😕
That's why the ⚙️OPS team enforces specific ClusterRoles with restricted permissions
Let's create these permissions!
namespace isolation for 🎸ROCKY
.lab[
- Here are the restricted permissions to use in the
rocky-testNamespace
k8s@shpod:~/fleet-config-using-flux-XXXXX$ \
cp ~/container.training/k8s/M6-rocky-cluster-role.yaml ./tenants/base/rocky/
]
💡 Note that some resources are managed at cluster scope (like
PersistentVolumes). We need specific permissions, then…
Creating Github source in Flux for 🎸ROCKY app repository
A specific branch of the Github repository is monitored by the Flux source
.lab[
- ⚠️ you may change the repository URL to the one of your own clone
k8s@shpod:~/fleet-config-using-flux-XXXXX$ flux create source git rocky-app \
--namespace=rocky-test \
--url=https://github.com/Musk8teers/container.training-spring-music/ \
--branch=rocky --export > ./tenants/base/rocky/sync.yaml
]
Creating kustomization in Flux for 🎸ROCKY app repository
.lab[
k8s@shpod:~/fleet-config-using-flux-XXXXX$ flux create kustomization rocky \
--namespace=rocky-test \
--service-account=rocky \
--source=GitRepository/rocky-app \
--path="./k8s/" --export >> ./tenants/base/rocky/sync.yaml
k8s@shpod:~/fleet-config-using-flux-XXXXX$ \
cd ./tenants/base/rocky/ && \
kustomize create --autodetect && \
cd -
]
class: extra-details
📂 Flux config files
Let's review our Flux configuration files
.lab[
k8s@shpod:~/fleet-config-using-flux-XXXXX$ \
cat ./tenants/base/rocky/sync.yaml && \
cat ./tenants/base/rocky/kustomization.yaml
]
Adding a kustomize patch for ⚗️TEST cluster deployment
💡 Remember the DRY strategy!
-
The
Fluxtenant-dedicated configuration is looking for this file:.tenants/test/rocky/kustomization.yaml -
It has been configured here:
clusters/CLOUDY/tenants.yaml -
All the files we just created are located in
.tenants/base/rocky -
So we have to create a specific kustomization in the right location
k8s@shpod:~/fleet-config-using-flux-XXXXX$ \
mkdir -p ./tenants/test/rocky && \
cp ~/container.training/k8s/M6-rocky-test-patch.yaml ./tenants/test/rocky/ && \
cp ~/container.training/k8s/M6-rocky-test-kustomization.yaml ./tenants/test/rocky/kustomization.yaml
Synchronizing Flux config with its Github repo
Locally, our Flux config repo is ready
The ⚙️OPS team has to push it to Github for Flux controllers to watch and catch it!
.lab[
k8s@shpod:~/fleet-config-using-flux-XXXXX$ \
git add . && \
git commit -m':wrench: :construction_worker: add ROCKY tenant configuration' && \
git push
]
class: pic
class: pic
class: extra-details
Flux resources for ROCKY tenant 1/2
.lab[
k8s@shpod:~$ flux get all -A
NAMESPACE NAME REVISION SUSPENDED
READY MESSAGE
flux-system gitrepository/flux-system main@sha1:8ffd72cf False
True stored artifact for revision 'main@sha1:8ffd72cf'
rocky-test gitrepository/rocky-app rocky@sha1:ffe9f3fe False
True stored artifact for revision 'rocky@sha1:ffe9f3fe'
(…)
]
class: extra-details
Flux resources for ROCKY tenant 2/2
.lab[
k8s@shpod:~$ flux get all -A
(…)
NAMESPACE NAME REVISION SUSPENDED
READY MESSAGE
flux-system kustomization/flux-system main@sha1:8ffd72cf False
True Applied revision: main@sha1:8ffd72cf
flux-system kustomization/tenant-prod False
False kustomization path not found: stat /tmp/kustomization-1164119282/tenants/prod: no such file or directory
flux-system kustomization/tenant-test main@sha1:8ffd72cf False
True Applied revision: main@sha1:8ffd72cf
rocky-test kustomization/rocky False
False StatefulSet/db dry-run failed (Forbidden): statefulsets.apps "db" is forbidden: User "system:serviceaccount:rocky-test:rocky" cannot patch resource "statefulsets" in API group "apps" at the cluster scope
]
And here is our 2nd Flux error(s)! 😅
class: extra-details
Flux Kustomization, mutability, …
🔍 Notice that none of the expected resources is created:
the whole kustomization is rejected, even if the StatefulSet is this only resource that fails!
🔍 Flux Kustomization uses the dry-run feature to templatize the resources and then applying patches onto them
Good but some resources are not completely mutable, such as StatefulSets
We have to fix the mutation by applying the change without having to patch the resource.
🔍 Simply add the spec.targetNamespace: rocky-test to the Kustomization named rocky
class: extra-details
And then it's deployed 1/2
You should see the following resources in the rocky-test namespace
.lab[
k8s@shpod-578d64468-tp7r2 ~/$ k get pods,svc,deployments -n rocky-test
NAME READY STATUS RESTARTS AGE
pod/db-0 1/1 Running 0 47s
pod/web-6c677bf97f-c7pkv 0/1 Running 1 (22s ago) 47s
pod/web-6c677bf97f-p7b4r 0/1 Running 1 (19s ago) 47s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/db ClusterIP 10.32.6.128 <none> 5432/TCP 48s
service/web ClusterIP 10.32.2.202 <none> 80/TCP 48s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/web 0/2 2 0 47s
]
class: extra-details
And then it's deployed 2/2
You should see the following resources in the rocky-test namespace
.lab[
k8s@shpod-578d64468-tp7r2 ~/$ k get statefulsets,pvc,pv -n rocky-test
NAME READY AGE
statefulset.apps/db 1/1 47s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
persistentvolumeclaim/postgresql-data-db-0 Bound pvc-c1963a2b-4fc9-4c74-9c5a-b0870b23e59a 1Gi RWO sbs-default <unset> 47s
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE
persistentvolume/postgresql-data 1Gi RWO,RWX Retain Available <unset> 47s
persistentvolume/pvc-150fcef5-ebba-458e-951f-68a7e214c635 1G RWO Delete Bound shpod/shpod sbs-default <unset> 4h46m
persistentvolume/pvc-c1963a2b-4fc9-4c74-9c5a-b0870b23e59a 1Gi RWO Delete Bound rocky-test/postgresql-data-db-0 sbs-default <unset> 47s
]
class: extra-details
PersistentVolumes are using a default StorageClass
💡 This managed cluster comes with custom StorageClasses leveraging on Cloud IaaS capabilities (i.e. block devices)
- a default
StorageClassis applied if none is specified (like here) - for 🏭PROD purpose, ops team might enforce a more performant
StorageClass - on a bare-metal cluster, 🏭PROD team has to configure and provide
StorageClasseson its own
class: pic
Upgrading ROCKY app
The Git source named rocky-app is pointing at
- a Github repository named Musk8teers/container.training-spring-music
- on its branch named
rocky
This branch deploy the v1.0.0 of the Web app:
spec.template.spec.containers.image: ghcr.io/musk8teers/container.training-spring-music:1.0.0
What happens if the 🎸ROCKY team upgrades its branch to deploy v1.0.1 of the Web app?
tenant 🏭PROD
💡 🏭PROD tenant is still waiting for its Flux configuration, but don't bother for it right now.
🗺️ Where are we in our scenario?
%%{init:
{
"theme": "default",
"gitGraph": {
"mainBranchName": "OPS",
"mainBranchOrder": 0
}
}
}%%
gitGraph
commit id:"0" tag:"start"
branch ROCKY order:3
branch MOVY order:4
branch YouRHere order:5
checkout OPS
commit id:'Flux install on CLOUDY cluster' tag:'T01'
branch TEST-env order:1
commit id:'FLUX install on TEST' tag:'T02' type: HIGHLIGHT
checkout OPS
commit id:'Flux config. for TEST tenant' tag:'T03'
commit id:'namespace isolation by RBAC'
checkout TEST-env
merge OPS id:'ROCKY tenant creation' tag:'T04'
checkout OPS
commit id:'ROCKY deploy. config.' tag:'R01'
checkout TEST-env
merge OPS id:'TEST ready to deploy ROCKY' type: HIGHLIGHT tag:'R02'
checkout ROCKY
commit id:'ROCKY' tag:'v1.0.0'
checkout TEST-env
merge ROCKY tag:'ROCKY v1.0.0'
checkout YouRHere
commit id:'x'
checkout OPS
merge YouRHere id:'YOU ARE HERE'
checkout OPS
commit id:'Ingress-controller config.' tag:'T05'
checkout TEST-env
merge OPS id:'Ingress-controller install' type: HIGHLIGHT tag:'T06'
checkout OPS
commit id:'ROCKY patch for ingress config.' tag:'R03'
checkout TEST-env
merge OPS id:'ingress config. for ROCKY app'
checkout ROCKY
commit id:'blue color' tag:'v1.0.1'
checkout TEST-env
merge ROCKY tag:'ROCKY v1.0.1'
checkout ROCKY
commit id:'pink color' tag:'v1.0.2'
checkout TEST-env
merge ROCKY tag:'ROCKY v1.0.2'
checkout OPS
commit id:'FLUX config for MOVY deployment' tag:'M01'
checkout TEST-env
merge OPS id:'FLUX ready to deploy MOVY' type: HIGHLIGHT tag:'M02'
checkout MOVY
commit id:'MOVY' tag:'v1.0.3'
checkout TEST-env
merge MOVY tag:'MOVY v1.0.3' type: REVERSE
checkout OPS
commit id:'Network policies'
checkout TEST-env
merge OPS type: HIGHLIGHT