diff --git a/slides/k8s/flux.md b/slides/k8s/flux.md new file mode 100644 index 00000000..895f7105 --- /dev/null +++ b/slides/k8s/flux.md @@ -0,0 +1,504 @@ +# FluxCD + +- We're going to implement a basic GitOps workflow with Flux + +- Pushing to `main` will automatically deploy to the clusters + +- There will be two clusters (`dev` and `prod`) + +- The two clusters will have similar (but slightly different) workloads + +--- + +## Repository structure + +This is (approximately) what we're going to do: + +``` +@@INCLUDE[slides/k8s/gitopstree.txt] +``` + +--- + +## Getting ready + +- Let's make sure we have two clusters + +- It's OK to use local clusters (kind, minikube...) + +- We might run into resource limits, though + + (pay attention to `Pending` pods!) + +- We need to install the Flux CLI ([packages], [binaries]) + +- **Highly recommended:** set up CLI completion! + +- Of course we'll need a Git service, too + + (we're going to use GitHub here) + +[packages]: https://fluxcd.io/flux/get-started/ +[binaries]: https://github.com/fluxcd/flux2/releases + +--- + +## GitHub setup + +- Generate a GitHub token: + + https://github.com/settings/tokens/new + +- Give it "repo" access + +- This token will be used by the `flux bootstrap github` command later + +- It will create a repository and configure it (SSH key...) + +- The token can be revoked afterwards + +--- + +## Flux bootstrap + +.lab[ + +- Let's set a few variables for convenience, and create our repository: + ```bash + export GITHUB_TOKEN=... + export GITHUB_USER=changeme + export GITHUB_REPO=alsochangeme + export FLUX_CLUSTER=dev + + flux bootstrap github \ + --owner=$GITHUB_USER \ + --repository=$GITHUB_REPO \ + --branch=main \ + --path=./clusters/$FLUX_CLUSTER \ + --personal --public + ``` + +] + +Problems? check next slide! + +--- + +## What could go wrong? + +- `flux bootstrap` will create or update the repository on GitHub + +- Then it will install Flux controllers to our cluster + +- Then it waits for these controllers to be up and running and ready + +- Check pod status in `flux-system` + +- If pods are `Pending`, check that you have enough resources on your cluster + +- For testing purposes, it should be fine to lower or remove Flux `requests`! + + (but don't do that in production!) + +- If anything goes wrong, don't worry, we can just re-run the bootstrap + +--- + +class: extra-details + +## Idempotence + +- It's OK to run that same `flux bootstrap` command multiple times! + +- If the repository already exists, it will re-use it + + (it won't destroy or empty it) + +- If the path `./clusters/$FLUX_CLUSTER` already exists, it will update it + +- It's totally fine to re-run `flux bootstrap` if something fails + +- It's totally fine to run it multiple times on different clusters + +- Or even to run it multiple times for the *same* cluster + + (to reinstall Flux on that cluster after a cluster wipe / reinstall) + +--- + +## What do we get? + +- Let's look at what `flux bootstrap` installed on the cluster + +.lab[ + +- Look inside the `flux-system` namespace: + ```bash + kubectl get all --namespace flux-system + ``` + +- Look at `kustomizations` custom resources: + ```bash + kubectl get kustomizations --all-namespaces + ``` + +- See what the `flux` CLI tells us: + ```bash + flux get all + ``` + +] + +--- + +## Deploying with GitOps + +- We'll need to add/edit files on the repository + +- We can do it by using `git clone`, local edits, `git commit`, `git push` + +- Or by editing online on the GitHub website + +.lab[ + +- Create a manifest; for instance `clusters/dev/flux-system/blue.yaml` + +- Add that manifest to `cluseters/dev/kustomization.yaml` + +- Commit and push both changes to the repository + +] + +--- + +## Waiting for reconciliation + +- Compare the git hash that we pushed and the one show with `kubectl get ` + +- Option 1: wait for Flux to pick up the changes in the repository + + (the default interval for git repositories is 1 minute, so that's fast) + +- Option 2: use `flux reconcile source git flux-system` + + (this puts an annotation on the appropriate resource, triggering an immediate check) + +--- + +## Checking progress + +- `flux logs` + +- `kubectl get gitrepositories --all-namespaces` + +- `kubectl get kustomizations --all-namespaces` + +--- + +## Did it work? + +-- + +- No! + +-- + +- Why? + +-- + +- We need to indicate the namespace where the app should be deployed + +- Either in the YAML manifests + +- Or in the `kustomization` custom resource + + (using field `spec.targetNamespace`) + +- Add the namespace to the manifest and try again! + +--- + +## Adding an app in a reusable way + +- Let's see a technique to add a whole app + + (with multiple resource manifets) + +- We want that to be reusable + + (i.e. easy to add on multiple clusters with minimal changes) + +--- + +## The plan + +- Add the app manifests in a directory + + (e.g.: `apps/myappname/manifests`) + +- Create a kustomization manifest for the app and its namespace + + (e.g.: `apps/myappname/flux.yaml`) + +- The kustomization manifest will refer to the app manifest + +- Add the kustomization manifest to the top-level `flux-system` kustomization + +--- + +## Creating the manifests + +- All commands below should be executed at the root of the repository + +.lab[ + +- Put application manifests in their directory: + ```bash + mkdir -p apps/dockercoins + cp ~/container.training/k8s/dockercoins.yaml apps/dockercoins/ + ``` + +- Create kustomization manifest: + ```bash + flux create kustomization dockercoins \ + --source=GitRepository/flux-system \ + --path=./apps/dockercoins/manifests/ \ + --target-namespace=dockercoins \ + --prune=true --export > apps/dockercoins/flux.yaml + ``` + +] + +--- + +## Creating the target namespace + +- When deploying *helm releases*, it is possible to automatically create the namespace + +- When deploying *kustomizations*, we need to create it explicitly + +- Let's put the namespace with the kustomization manifest + + (so that the whole app can be mediated through a single manifest) + +.lab[ + +- Add the target namespace to the kustomization manifest: + ```bash + echo "--- + kind: Namespace + apiVersion: v1 + metadata: + name: dockercoins" >> apps/dockercoins/flux.yaml + ``` + +] + +--- + +## Linking the kustomization manifest + +- Edit `clusters/dev/flux-system/kustomization.yaml` + +- Add a line to reference the kustomization manifest that we created: + ```yaml + - ../../../apps/dockercoins/flux.yaml + ``` + +- `git add` our manifests, `git commit`, `git push` + + (check with `git status` that we haven't forgotten anything!) + +- `flux reconcile` or wait for the changes to be picked up + +--- + +## Installing with Helm + +- We're going to see two different workflows: + + - installing a third-party chart +
+ (e.g. something we found on the Artifact Hub) + + - installing one of our own charts +
+ (e.g. a chart with authored ourselves) + +- The procedures are very similar + +--- + +## Installing from a public Helm repository + +- Let's install [kube-prometheus-stack][kps] + +.lab[ + +- Create the Flux manifests: + ```bash + mkdir -p apps/kube-prometheus-stack + flux create source helm kube-prometheus-stack \ + --url=https://prometheus-community.github.io/helm-charts \ + --export >> apps/kube-prometheus-stack/flux.yaml + flux create helmrelease kube-prometheus-stack \ + --source=HelmRepository/kube-prometheus-stack \ + --chart=kube-prometheus-stack --release-name=kube-prometheus-stack \ + --target-namespace=kube-prometheus-stack --create-target-namespace \ + --export >> apps/kube-prometheus-stack/flux.yaml + ``` + +] + +[kps]: https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack + +--- + +## Enable the app + +- Just like before, link the manifest from the top-level kustomization + + (`flux-system` in namespace `flux-system`) + +- `git add` / `git commit` / `git push` + +- We should now have a Prometheus+Grafana observability stack! + +--- + +## Installing from a Helm chart in a git repo + +- In this example, the chart will be in the same repo + +- In the real world, it will typically be in a different repo! + +.lab[ + +- Generate a basic Helm chart: + ```bash + mkdir -p charts + helm create charts/myapp + ``` + +] + +(This generates a chart which installs NGINX. A lot of things can be customized, though.) + +--- + +## Creating the Flux manifests + +- The invocation is very similar to our first example + +.lab[ + +- Generate the Flux manifest for the Helm release: + ```bash + mkdir apps/myapp + flux create helmrelease myapp \ + --source=GitRepository/flux-system \ + --chart=charts/myapp \ + --target-namespace=myapp --create-target-namespace \ + --export > apps/myapp/flux.yaml + ``` + +- Add that manifest to the top-level kustomization + +- `git add` / `git commit` / `git push` the chart, manifest, and kustomization + +] + +--- + +## Passing values + +- We can also configure our Helm releases with values + +- Using an existing `myvalues.yaml` file: + + `flux create helmrelease ... --values=myvalues.yaml` + +- Referencing an existing ConfigMap or Secret with a `values.yaml` key: + + `flux create helmrelease ... --values-from=ConfigMap/myapp` + +--- + +## Gotchas + +- When creating a HelmRelease using a chart stored in a git repository, you must: + + - either bump the chart version (in `Chart.yaml`) after each change, + + - or set `spec.chart.spec.reconcileStrategy` to `Revision` + +- Why? + +- Flux installs helm releases using packaged artifacts + +- Artifacts are updated only when the Helm chart version changes + +- Unless `reoncileStrategy` is set to `Revision` (instead of the default `ChartVersion`) + +--- + +## More gotchas + +- There is a bug in Flux that prevents using identical subcharts with aliases + +- See [fluxcd/flux2#2505][flux2505] for details + +[flux2505]: https://github.com/fluxcd/flux2/discussions/2505 + +--- + +## Things that we didn't talk about... + +- Bucket sources + +- Image automation controller + +- Image reflector controller + +- And more! + +??? + +:EN:- Implementing gitops with Flux +:FR:- Workflow gitops avec Flux + + + diff --git a/slides/k8s/gitopstree.txt b/slides/k8s/gitopstree.txt new file mode 100644 index 00000000..b913e206 --- /dev/null +++ b/slides/k8s/gitopstree.txt @@ -0,0 +1,13 @@ +├── charts/ <--- could also be in separate app repos +│ ├── dockercoins/ +│ └── color/ +├── apps/ <--- YAML manifests for GitOps resources +│ ├── dockercoins/ (might reference the "charts" above, +│ ├── blue/ and/or include environment-specific +│ ├── green/ manifests to create e.g. namespaces, +│ ├── kube-prometheus-stack/ configmaps, secrets...) +│ ├── cert-manager/ +│ └── traefik/ +└── clusters/ <--- per-cluster; will typically reference + ├── prod/ the "apps" above, possibly extending + └── dev/ or adding configuration resources too diff --git a/slides/k8s/gitworkflows.md b/slides/k8s/gitworkflows.md index 36b2c053..6b7f4465 100644 --- a/slides/k8s/gitworkflows.md +++ b/slides/k8s/gitworkflows.md @@ -1,4 +1,4 @@ -# Git-based workflows +# Git-based workflows (GitOps) - Deploying with `kubectl` has downsides: @@ -40,223 +40,210 @@ ## Enabling git-based workflows -- There are a few tools out there to help us do that +- There are a many tools out there to help us do that -- We'll see demos of two of them: [Flux] and [Gitkube] - -- There are *many* other tools, some of them with even more features + (examples: [ArgoCD], [FluxCD]...) - There are also *many* integrations with popular CI/CD systems - (e.g.: GitLab, Jenkins, ...) + (e.g.: GitHub Actions, GitLab, Jenkins...) -[Flux]: https://www.weave.works/oss/flux/ -[Gitkube]: https://gitkube.sh/ +[ArgoCD]: https://argoproj.github.io/cd/ +[Flux]: https://fluxcd.io/ --- -## Flux overview +## The road to production -- We put our Kubernetes resources as YAML files in a git repository +In no specific order, we need to at least: -- Flux polls that repository regularly (every 5 minutes by default) +- Choose a tool -- The resources described by the YAML files are created/updated automatically +- Choose a cluster / app / namespace layout +
+ (one cluster per app, different clusters for prod/staging...) -- Changes are made by updating the code in the repository +- Choose a repository layout +
+ (different repositories, directories, branches per app, env, cluster...) + +- Choose an installation / bootstrap method + +- Choose how new apps / environments / versions will be deployed + +- Choose how new images will be built --- -## Preparing a repository for Flux +## FluxCD -- We need a repository with Kubernetes YAML files +- Nice bootstrap -- I have one: https://github.com/jpetazzo/kubercoins + (CLI tool can automatically install, create git repos...) -- Fork it to your GitHub account +- Self-hosted -- Create a new branch in your fork; e.g. `prod` + (flux controllers are managed by flux itself) - (e.g. with "branch" dropdown through the GitHub web UI) +- Many CRDs -- This is the branch that we are going to use for deployment + (Kustomization, HelmRelease, GitRepository...) + +- No web UI out of the box + +- CLI relies on Kubernetes API access --- -## Setting up Flux with kustomize +## ArgoCD -- Clone the Flux repository: - ```bash - git clone https://github.com/fluxcd/flux - cd flux - ``` +- Simple bootstrap -- Edit `deploy/flux-deployment.yaml` + (just apply YAMLs / install Helm chart) -- Change the `--git-url` and `--git-branch` parameters: - ```yaml - - --git-url=git@github.com:your-git-username/kubercoins - - --git-branch=prod - ``` +- Few CRDs -- Apply all the YAML: - ```bash - kubectl apply -k deploy/ - ``` + (basic workflow can be done with a single "Application" resource) + +- Comes with a web UI + +- CLI relies on separate API and authentication system --- -## Setting up Flux with Helm +## Cluster, app, namespace layout -- Add Flux helm repo: - ```bash - helm repo add fluxcd https://charts.fluxcd.io - ``` +- One cluster per app, different namespaces for environments? -- Install Flux: - ```bash - kubectl create namespace flux - helm upgrade --install flux \ - --set git.url=git@github.com:your-git-username/kubercoins \ - --set git.branch=prod \ - --namespace flux \ - fluxcd/flux - ``` +- One cluster per environment, different namespaces for apps? + +- Everything on a single cluster? One cluster per combination? + +- Something in between: + + - prod cluster, database cluster, dev/staging/etc cluster + + - prod+db cluster per app, shared dev/staging/etc cluster + +- And more! + +Note: this decision isn't really tied to GitOps! --- -## Allowing Flux to access the repository +## Repository layout -- When it starts, Flux generates an SSH key +So many different possibilities! -- Display that key: - ```bash - kubectl -n flux logs deployment/flux | grep identity.pub | cut -d '"' -f2 - ``` +- Source repos -- Then add that key to the repository, giving it **write** access +- Cluster/infra repos/branches/directories - (some Flux features require write access) +- "Deployment" repos (with manifests, charts) -- After a minute or so, DockerCoins will be deployed to the current namespace +- Different repos/branches/directories for environments + +🤔 How to decide? --- -## Making changes +## Permissions -- Make changes (on the `prod` branch), e.g. change `replicas` in `worker` +- Different teams/companies = different repos -- After a few minutes, the changes will be picked up by Flux and applied + - separate platform team → separate "infra" vs "apps" repos + + - teams working on different apps → different repos per app + +- Branches can be "protected" (`production`, `main`...) + + (don't need separate repos for separate environments) + +- Directories will typically have the same permissions + +- Managing directories is easier than branches + +- But branches are more "powerful" (cherrypicking, rebasing...) --- -## Other features +## Resource hierarchy -- Flux can keep a list of all the tags of all the images we're running +- Git-based deployments are managed by Kubernetes resources -- The `fluxctl` tool can show us if we're running the latest images + (e.g. Kustomization, HelmRelease with Flux; Application with ArgoCD) -- We can also "automate" a resource (i.e. automatically deploy new images) +- These resources need to be managed like any other Kubernetes resource -- And much more! + (YAML manifests, Kustomizations, Helm charts) + +- They can be managed with Git workflows too! --- -## Gitkube overview +## Cluster / infra management -- We put our Kubernetes resources as YAML files in a git repository +- How do we provision clusters? -- Gitkube is a git server (or "git remote") +- Manual "one-shot" provisioning (CLI, web UI...) -- After making changes to the repository, we push to Gitkube +- Automation with Terraform, Ansible... -- Gitkube applies the resources to the cluster +- Kubernetes-driven systems (Crossplane, CAPI) + +- Infrastructure can also be managed with GitOps --- -## Setting up Gitkube +## Example 1 -- Install the CLI: - ```bash - sudo curl -L -o /usr/local/bin/gitkube \ - https://github.com/hasura/gitkube/releases/download/v0.2.1/gitkube_linux_amd64 - sudo chmod +x /usr/local/bin/gitkube - ``` +- Managed with YAML/Charts: -- Install Gitkube on the cluster: - ```bash - gitkube install --expose ClusterIP - ``` + - core components (CNI, CSI, Ingress, logging, monitoring...) + + - GitOps controllers + + - critical application foundations (database operator, databases) + + - GitOps manifests + +- Managed with GitOps: + + - applications + + - staging databases --- -## Creating a Remote +## Example 2 -- Gitkube provides a new type of API resource: *Remote* +- Managed with YAML/Charts: - (this is using a mechanism called Custom Resource Definitions or CRD) + - essential components (CNI, CoreDNS) -- Create and apply a YAML file containing the following manifest: - ```yaml - apiVersion: gitkube.sh/v1alpha1 - kind: Remote - metadata: - name: example - spec: - authorizedKeys: - - `ssh-rsa AAA...` - manifests: - path: "." - ``` + - initial installation of GitOps controllers - (replace the `ssh-rsa AAA...` section with the content of `~/.ssh/id_rsa.pub`) +- Managed with GitOps: + + - upgrades of GitOps controllers + + - core components (CSI, Ingress, logging, monitoring...) + + - operators, databases + + - more GitOps manifests for applications! --- -## Pushing to our remote +## Concrete example -- Get the `gitkubed` IP address: - ```bash - kubectl -n kube-system get svc gitkubed - IP=$(kubectl -n kube-system get svc gitkubed -o json | - jq -r .spec.clusterIP) - ``` +- Source code repository (not shown here) -- Get ourselves a sample repository with resource YAML files: - ```bash - git clone git://github.com/jpetazzo/kubercoins - cd kubercoins - ``` +- Infrastructure repository (shown below), single branch -- Add the remote and push to it: - ```bash - git remote add k8s ssh://default-example@$IP/~/git/default-example - git push k8s master - ``` - ---- - -## Making changes - -- Edit a local file - -- Commit - -- Push! - -- Make sure that you push to the `k8s` remote - ---- - -## Other features - -- Gitkube can also build container images for us - - (see the [documentation](https://github.com/hasura/gitkube/blob/master/docs/remote.md) for more details) - -- Gitkube can also deploy Helm charts - - (instead of raw YAML files) +``` +@@INCLUDE[slides/k8s/gitopstree.txt] +``` ??? diff --git a/slides/kube-selfpaced.yml b/slides/kube-selfpaced.yml index 3da8be0f..a42438da 100644 --- a/slides/kube-selfpaced.yml +++ b/slides/kube-selfpaced.yml @@ -124,6 +124,9 @@ content: - k8s/portworx.md - k8s/openebs.md - k8s/stateful-failover.md +- + - k8s/gitworkflows.md + - k8s/flux.md - - k8s/logs-centralized.md - k8s/prometheus.md @@ -164,7 +167,6 @@ content: - k8s/cluster-upgrade.md - k8s/cluster-backup.md - k8s/cloud-controller-manager.md - - k8s/gitworkflows.md - - k8s/lastwords.md - k8s/links.md