diff --git a/slides/k8s/cert-manager.md b/slides/k8s/cert-manager.md index c9b577fe..996c2a87 100644 --- a/slides/k8s/cert-manager.md +++ b/slides/k8s/cert-manager.md @@ -84,13 +84,15 @@ ## Creating the ClusterIssuer -- The manifest shown on the previous slide is in @@LINK[k8s/cm-clusterissuer.yaml] +- Download the file @@LINK[k8s/cm-clusterissuer.yaml] + + (or copy-paste from the previous slide) .exercise[ - Create the ClusterIssuer: ```bash - kubectl apply -f ~/container.training/k8s/cm-clusterissuer.yaml + kubectl apply cm-clusterissuer.yaml ``` ] @@ -113,7 +115,9 @@ ## Creating the Certificate -- The manifest shown on the previous slide is in @@LINK[k8s/cm-certificate.yaml] +- Download the file @@LINK[k8s/cm-certificate.yaml] + + (or copy-paste from the previous slide) .exercise[ @@ -123,7 +127,7 @@ - Create the Certificate: ```bash - kubectl apply -f ~/container.training/k8s/cm-certificate.yaml + kubectl apply -f cm-certificate.yaml ``` ] @@ -170,25 +174,14 @@ --- -## What's missing ? +## And then... --- - -An Ingress Controller! 😅 - -.exercise[ - -- Install an Ingress Controller: - ```bash - kubectl apply -f ~/container.training/k8s/traefik-v2.yaml - ``` - -- Wait a little bit, and check that we now have a `kubernetes.io/tls` Secret: +- A little bit later, we will have a `kubernetes.io/tls` Secret: ```bash kubectl get secrets ``` -] +- Note that this might take a few minutes, because of the DNS integration! --- @@ -232,25 +225,6 @@ class: extra-details - Note: the Ingress still needs the `tls` section with `secretName` and `hosts` ---- - -class: extra-details - -## Let's Encrypt and nip.io - -- Let's Encrypt has [rate limits](https://letsencrypt.org/docs/rate-limits/) per domain - - (the limits only apply to the production environment, not staging) - -- There is a limit of 50 certificates per registered domain - -- If we try to use the production environment, we will probably hit the limit - -- It's fine to use the staging environment for these experiments - - (our certs won't validate in a browser, but we can always check - the details of the cert to verify that it was issued by Let's Encrypt!) - ??? :EN:- Obtaining certificates with cert-manager diff --git a/slides/k8s/gitlab.md b/slides/k8s/gitlab.md index 8f198b52..6c7312e1 100644 --- a/slides/k8s/gitlab.md +++ b/slides/k8s/gitlab.md @@ -164,154 +164,493 @@ class: extra-details --- -## Ingress - -- We will assume that we have a domain name pointing to our cluster - - (i.e. with a wildcard record pointing to at least one node of the cluster) - -- We will get traffic in the cluster by leveraging `ExternalIPs` services - - (but it would be easy to use `LoadBalancer` services instead) - -- We will use Traefik as the ingress controller - - (but any other one should work too) - -- We will use cert-manager to obtain certificates with Let's Encrypt - ---- - -## Other details +## Install GitLab itself - We will deploy GitLab with its official Helm chart - It will still require a bunch of parameters and customization -- We also need a Storage Class - - (unless our cluster already has one, of course) - -- We suggest the [Rancher local path provisioner](https://github.com/rancher/local-path-provisioner) +- Brace! --- -## Setting everything up +## Installing the GitLab chart -1. `git clone https://github.com/jpetazzo/kubecoin` +```bash +helm repo add gitlab https://charts.gitlab.io/ +DOMAIN=`cloudnative.party` +ISSUER=letsencrypt-production +helm upgrade --install gitlab gitlab/gitlab \ + --create-namespace --namespace gitlab \ + --set global.hosts.domain=$DOMAIN \ + --set certmanager.install=false \ + --set nginx-ingress.enabled=false \ + --set global.ingress.class=traefik \ + --set global.ingress.provider=traefik \ + --set global.ingress.configureCertmanager=false \ + --set global.ingress.annotations."cert-manager\.io/cluster-issuer"=$ISSUER \ + --set gitlab.webservice.ingress.tls.secretName=gitlab-gitlab-tls \ + --set registry.ingress.tls.secretName=gitlab-registry-tls \ + --set minio.ingress.tls.secretName=gitlab-minio-tls +``` -2. `export EMAIL=xxx@example.com DOMAIN=awesome-kube-ci.io` - - (we need a real email address and a domain pointing to the cluster!) - -3. `. setup-gitlab-on-k8s.rc` - - (this doesn't do anything, but defines a number of helper functions) - -4. Execute each helper function, one after another - - (try `do_[TAB]` to see these functions) +😰 Can we talk about all these parameters? --- -## Local Storage +## Breaking down all these parameters -`do_1_localstorage` +- `certmanager.install=false` -Applies the YAML directly from Rancher's repository. + do not install cert-manager, we already have it -Annotate the Storage Class so that it becomes the default one. +- `nginx-ingress.enabled=false` + + do not install the NGINX ingress controller, we already have Traefik + +- `global.ingress.class=traefik`, `global.ingress.provider=traefik` + + these merely enable creation of Ingress resources + +- `global.ingress.configureCertmanager=false` + + do not create a cert-manager Issuer or ClusterIssuer, we have ours --- -## Traefik +## More parameters -`do_2_traefik_with_externalips` +- `global.ingress.annotations."cert-manager\.io/cluster-issuer"=$ISSUER` -Install the official Traefik Helm chart. + this annotation tells cert-manager to automatically issue certs -Instead of a `LoadBalancer` service, use a `ClusterIP` with `ExternalIPs`. +- `gitlab.webservice.ingress.tls.secretName=gitlab-gitlab-tls`, +
+ `registry.ingress.tls.secretName=gitlab-registry-tls`, +
+ `minio.ingress.tls.secretName=gitlab-minio-tls` -Automatically infer the `ExternalIPs` from `kubectl get nodes`. - -Enable TLS. + these annotations enable TLS in the Ingress controller --- -## cert-manager +## Wait for GitLab to come up -`do_3_certmanager` +- Let's watch what's happening in the GitLab namespace: + ```bash + watch kubectl get all --namespace gitlab + ``` -Install cert-manager using their official YAML. +- We want to wait for all the Pods to be "Running" or "Completed" -Easy-peasy. +- This will take a few minutes (10-15 minutes for me) + +- Don't worry if you see Pods crashing and restarting + + (it happens when they are waiting on a dependency which isn't up yet) --- -## Certificate issuers +## Things that could go wrong -`do_4_issuers` +- Symptom: Pods remain "Pending" or "ContainerCreating" for a while -Create a couple of `ClusterIssuer` resources for cert-manager. +- Investigate these pods (with `kubectl describe pod ...`) -(One for the staging Let's Encrypt environment, one for production.) +- Also look at events: + ```bash + kubectl get events \ + --field-selector=type=Warning --sort-by=metadata.creationTimestamp + ``` -Note: this requires to specify a valid `$EMAIL` address! +- Make sure your cluster is big enough -Note: if this fails, wait a bit and try again (cert-manager needs to be up). + (I use 3 `g6-standard-4` nodes) --- -## GitLab +## Log into GitLab -`do_5_gitlab` +- First, let's check that we can connect to GitLab (with TLS): -Deploy GitLab using their official Helm chart. + `https://gitlab.$DOMAIN` -We pass a lot of parameters to this chart: -- the domain name to use -- disable GitLab's own ingress and cert-manager -- annotate the ingress resources so that cert-manager kicks in -- bind the shell service (git over SSH) to port 222 to avoid conflict -- use ExternalIPs for that shell service +- It's asking us for a login and password! -Note: on modest cloud instances, it can take 10 minutes for GitLab to come up. - -We can check the status with `kubectl get pods --namespace=gitlab` +- The login is `root`, and the password is stored in a Secret: + ```bash + kubectl get secrets --namespace=gitlab gitlab-gitlab-initial-root-password \ + -o jsonpath={.data.password} | base64 -d + ``` --- -## Log into GitLab and configure it +## Configure GitLab -`do_6_showlogin` +- For simplicity, we're going to use that "root" user -This will get the GitLab root password (stored in a Secret). + (but later, you can create multiple users, teams, etc.) -Then we need to: -- log into GitLab -- add our SSH key (top-right user menu → settings, then SSH keys on the left) -- create a project (using the + menu next to the search bar on top) -- go to project configuration (on the left, settings → CI/CD) -- add a `KUBECONFIG` file variable with the content of our `.kube/config` file -- go to settings → access tokens to create a read-only registry token -- add variables `REGISTRY_USER` and `REGISTRY_PASSWORD` with that token -- push our repo (`git remote add gitlab ...` then `git push gitlab ...`) +- First, let's add our SSH key + + (top-right user menu → settings, then SSH keys on the left) + +- Then, create a project + + (using the + menu next to the search bar on top) + +- Let's call it `kubecoin` + + (you can change it, but you'll have to adjust Git paths later on) --- -## Monitoring progress and troubleshooting +## Try to push our repository -- Click on "CI/CD" in the left bar to view pipelines +- This is the repository that we're going to use: -- If you see a permission issue mentioning `system:serviceaccount:gitlab:...`: + https://github.com/jpetazzo/kubecoin - *make sure you did set `KUBECONFIG` correctly!* +- Let's clone that repository locally first: + ```bash + git clone https://github.com/jpetazzo/kubecoin + ``` -- GitLab will create namespaces named `gl--` +- Add our GitLab instance as a remote: + ```bash + git remote add gitlab git@gitlab.$DOMAIN:root/kubecoin.git + ``` -- At the end of the deployment, the web UI will be available on some unique URL +- Try to push: + ```bash + git push -u gitlab + ``` - (`http://---gitlab.`) +--- + +## Connection refused? + +- Normally, we get the following error: + + `port 22: Connection refused` + +- Why? 🤔 + +-- + +- What does `gitlab.$DOMAIN` point to? + +-- + +- Our Ingress Controller! (i.e. Traefik) 💡 + +- Our Ingress Controller has nothing to do with port 22 + +- So how do we solve this? + +--- + +## Routing port 22 + +- Whatever is on `gitlab.$DOMAIN` needs to have the following "routing": + + - port 80 → GitLab web service + + - port 443 → GitLab web service, with TLS + + - port 22 → GitLab shell service + +- Currently, Traefik is managing `gitlab.$DOMAIN` + +- We are going to tell Traefik to: + + - accept connections on port 22 + + - send them to GitLab + +--- + +## TCP routing + +- The technique that we are going to use is specific to Traefik + +- Other Ingress Controllers may or may not have similar features + +- When they have similar features, they will be enabled very differently + +--- + +## Telling Traefik to open port 22 + +- Let's reconfigure Traefik: + ```bash + helm upgrade --install traefik traefik/traefik \ + --create-namespace --namespace traefik \ + --set "ports.websecure.tls.enabled=true" \ + --set "providers.kubernetesIngress.publishedService.enabled=true" \ + --set "ports.ssh.port=2222" \ + --set "ports.ssh.exposedPort=22" \ + --set "ports.ssh.expose=true" \ + --set "ports.ssh.protocol=TCP" + ``` + +- This creates a new "port" on Traefik, called "ssh", listening on port 22 + +- Internally, Traefik listens on port 2222 (for permission reasons) + +- Note: Traefik docs also call these ports "entrypoints" + + (these entrypoints are totally unrelated to the `ENTRYPOINT` in Dockerfiles) + +--- + +## Knocking on port 22 + +- What happens if we try to connect to that port 22 right now? + ```bash + curl gitlab.$DOMAIN:22 + ``` + +- We hit GitLab's web service! + +- We need to tell Traefik what to do with connections to that port 22 + +- For that, we will create a "TCP route" + +--- + +## Traefik TCP route + +The following custom resource tells Traefik to route the `ssh` port that we +created earlier, to the `gitlab-gitlab-shell` service belonging to GitLab. + +```yaml +apiVersion: traefik.containo.us/v1alpha1 +kind: IngressRouteTCP +metadata: + name: gitlab-shell + namespace: gitlab +spec: + entryPoints: + - ssh + routes: + - match: HostSNI(\`*`) + services: + - name: gitlab-gitlab-shell + port: 22 +``` + +The `HostSNI` wildcard is the magic option to define a "default route". + +--- + +## Creating the TCP route + +Since our manifest has backticks, we must pay attention to quoting: + +```bash +kubectl apply -f- << "EOF" +apiVersion: traefik.containo.us/v1alpha1 +kind: IngressRouteTCP +metadata: + name: gitlab-shell + namespace: gitlab +spec: + entryPoints: + - ssh + routes: + - match: HostSNI(\`*`) + services: + - name: gitlab-gitlab-shell + port: 22 +EOF +``` + +--- + +## Knocking on port 22, again + +- Let's see what happens if we try port 22 now: + ```bash + curl gitlab.$DOMAIN:22 + ``` + +- This should tell us something like `Received HTTP/0.9 when not allowed` + + (because we're no longer talking to an HTTP server, but to SSH!) + +- Try with SSH: + ```bash + ssh git@gitlab.$DOMAIN + ``` + +- After accepting the key fingerprint, we should see `Welcome to GitLab, @root!` + +--- + +## Pushing again + +- Now we can try to push our repository again: + ```bash + git push -u gitlab + ``` + +- Reload the project page in GitLab + +- We should see our repository! + +--- + +## CI/CD + +- Click on the CI/CD tab on the left + + (the one with the shuttle / space rocket icon) + +- Our pipeline was detected... + +- But it failed 😕 + +- Let's click on one of the failed jobs + +- This is a permission issue! + +--- + +## Fixing permissions + +- GitLab needs to do a few of things in our cluster: + + - create Pods to build our container images with BuildKit + + - create Namespaces to deploy staging and production versions of our app + + - create and update resources in these Namespaces + +- For the time being, we're going to grant broad permissions + + (and we will revisit and discuss what to do later) + +--- + +## Granting permissions + +- Let's give `cluster-admin` permissions to the GitLab ServiceAccount: + ```bash + kubectl create clusterrolebinding gitlab \ + --clusterrole=cluster-admin --serviceaccount=gitlab:default + ``` + +- Then retry the CI/CD pipeline + +- The build steps will now succeed; but the deploy steps will fail + +- We need to set the `REGISTRY_USER` and `REGISTRY_PASSWORD` variables + +- Let's explain what this is about! + +--- + +## GitLab container registry access + +- A registry access token is created for the duration of the CI/CD pipeline + + (it is exposed through the `$CI_JOB_TOKEN` environment variable) + +- This token gives access only to a specific repository in the registry + +- It is valid only during the execution of the CI/CD pipeline + +- We can (and we do!) use it to *push* images to the registry + +- We cannot use it to *pull* images when running in staging or production + + (because Kubernetes might need to pull images *after* the token expires) + +- We need to create a separate read-only registry access token + +--- + +## Creating the registry access token + +- Let's go to "Settings" (the cog wheel on the left) / "Access Tokens" + +- Create a token with `read_registry` permission + +- Save the token name and the token value + +- Then go to "Settings" / "CI/CD" + +- In the "Variables" section, add two variables: + + - `REGISTRY_USER` → token name + - `REGISTRY_PASSWORD` → token value + +- Make sure that they are **not** protected! + + (otherwise, they won't be available in non-default tags and branches) + +--- + +## Trying again + +- Go back to the CI/CD pipeline view, and hit "Retry" + +- The deploy stage should now work correctly! 🎉 + +--- + +## Our CI/CD pipeline + +- Let's have a look at the [.gitlab-ci.yml](https://github.com/jpetazzo/kubecoin/blob/107dac5066087c52747e557babc97e57f42dd71d/.gitlab-ci.yml) file + +- We have multiple *stages*: + + - lint (currently doesn't do much, it's mostly as an example) + + - build (currently uses BuildKit) + + - deploy + +- "Deploy" behaves differently in staging and production + +- Let's investigate that! + +--- + +## Staging vs production + +- In our pipeline, "production" means "a tag or branch named `production`" + + (see the `except:` and `only:` sections) + +- Everything else is "staging" + +- In "staging": + + - we build and push images + - we create a staging Namespace and deploy a copy of the app there + +- In "production": + + - we do not build anything + - we deploy (or update) a copy of the app in the production Namespace + +--- + +## Namespace naming + +- GitLab will create Namespaces named `gl---` + +- At the end of the deployment, the web UI will be available at: + + `http://---gitlab.` + +- The "production" Namespace will be `-` + +- And it will be available on its own domain as well: + + `http://--gitlab.` --- @@ -325,7 +664,7 @@ Then we need to: - It will do it *only* if that same git commit was pushed to staging first - (look in the pipeline configuration file to see how it's done!) + (because the "production" pipeline skips the build phase) --- @@ -411,35 +750,15 @@ Then we need to: --- -## Pros +## Why not use GitLab's Kubernetes integration? -- GitLab is an amazing, open source, all-in-one platform +- "All-in-one" approach -- Available as hosted, community, or enterprise editions + (deploys its own Ingress, cert-manager, Prometheus, and much more) -- Rich ecosystem, very customizable +- I wanted to show you something flexible and customizable instead -- Can run on Kubernetes, or somewhere else - ---- - -## Cons - -- It can be difficult to use components separately - - (e.g. use a different registry, or a different job runner) - -- More than one way to configure it - - (it's not an opinionated platform) - -- Not "Kubernetes-native" - - (for instance, jobs are not Kubernetes jobs) - -- Job latency could be improved - -*Note: most of these drawbacks are the flip side of the "pros" on the previous slide!* +- But feel free to explore it now that we have shown the basics! ??? diff --git a/slides/k8s/helm-intro.md b/slides/k8s/helm-intro.md index 0ba16d85..66d7a2f6 100644 --- a/slides/k8s/helm-intro.md +++ b/slides/k8s/helm-intro.md @@ -504,8 +504,7 @@ The `readme` may or may not have (accurate) explanations for the values. - Update `my-juice-shop`: ```bash - helm upgrade my-juice-shop juice/my-juice-shop \ - --set service.type=NodePort + helm upgrade my-juice-shop juice/juice-shop --set service.type=NodePort ``` ] diff --git a/slides/lke.yml b/slides/lke.yml index 2b5330e1..dcfe117c 100644 --- a/slides/lke.yml +++ b/slides/lke.yml @@ -50,6 +50,7 @@ content: #- k8s/helm-values-schema-validation.md #- k8s/helm-secrets.md #- k8s/exercise-helm.md +- - shared/thankyou.md diff --git a/slides/lke/external-dns.md b/slides/lke/external-dns.md index 24bdff64..234e0b4c 100644 --- a/slides/lke/external-dns.md +++ b/slides/lke/external-dns.md @@ -72,7 +72,7 @@ - First, let's add the Bitnami repo: ```bash - kubectl repo add bitnami https://charts.bitnami.com/bitnami + helm repo add bitnami https://charts.bitnami.com/bitnami ``` - Then, install ExternalDNS: @@ -100,7 +100,7 @@ - Check ExternalDNS logs: ```bash - kubectl logs --n external-dns -l app.kubernetes.io/name=external-dns + kubectl logs -n external-dns -l app.kubernetes.io/name=external-dns ``` - It might take a few minutes for ExternalDNS to start, patience! diff --git a/slides/lke/prometheus.md b/slides/lke/prometheus.md index d4b6cbca..86d220ae 100644 --- a/slides/lke/prometheus.md +++ b/slides/lke/prometheus.md @@ -26,7 +26,7 @@ - a collection of Grafana dashboards (building them from scratch is tedious) -- The Helm char `kube-prometheus-stack` combines all these elements +- The Helm chart `kube-prometheus-stack` combines all these elements - ... So we're going to use it to deploy our metrics stack! @@ -45,7 +45,9 @@ --repo https://prometheus-community.github.io/helm-charts ``` -- Check what was installed: +- This will take a minute... + +- Then check what was installed: ```bash kubectl get all --namespace kube-prometheus-stack ``` @@ -81,8 +83,8 @@ - Decode the Secret: ```bash - kubectl get secret kube-prometheus-stack-grafana -o json \ - | jq '.data | map_values(@base64d)' + kubectl get secret --namespace kube-prometheus-stack \ + kube-prometheus-stack-grafana -o json | jq '.data | map_values(@base64d)' ``` - If you don't have the `jq` tool mentioned above, don't worry... diff --git a/slides/lke/traefik.md b/slides/lke/traefik.md index d3e5b818..ab2faa24 100644 --- a/slides/lke/traefik.md +++ b/slides/lke/traefik.md @@ -11,7 +11,7 @@ - Then, install the chart: ```bash - helm upgrade --install treafik trafik/traefik \ + helm upgrade --install traefik traefik/traefik \ --create-namespace --namespace traefik \ --set "ports.websecure.tls.enabled=true" ``` @@ -107,7 +107,7 @@ - Let's update our Traefik install: ```bash - helm upgrade --install treafik trafik/traefik \ + helm upgrade --install traefik traefik/traefik \ --create-namespace --namespace traefik \ --set "ports.websecure.tls.enabled=true" \ --set "providers.kubernetesIngress.publishedService.enabled=true"