🏭️ Replace Weave with Cilium

Initially we were using Weave because it was super easy to install (single kubectl apply -f command), but we now have multiple problems: - it is not maintained anymore - its network policy controller seems to have subtle bugs (I had multiple classes in 2026 where we ran into policies that didn't work as intended, but worked fine when trying on another cluster that used Cilium) - it doesn't support IPV6 We did a test run with Cilium on a few IPv6-only clusters. It worked fine. Then a test run on dual stack clusters. Now we're using Cilium on all kubeadm clusters (removing support for Weave) and we've updated all materials to either remove or update mentions to Weave.
2026-07-14 18:41:35 +00:00 · 2026-06-08 15:33:00 +02:00
parent 8a31ab8298
commit c1971aa30b
8 changed files with 58 additions and 43 deletions
--- a/prepare-labs/lib/commands.sh
+++ b/prepare-labs/lib/commands.sh
@@ -602,18 +602,41 @@ _cmd_kubeadm() {

    # Initialize kube control plane
    pssh --timeout 200 "
-    IPV6=\$(ip -json a | jq -r '.[].addr_info[] | select(.scope==\"global\" and .family==\"inet6\") | .local' | head -n1)
-    if [ \"\$IPV6\" ]; then
-      ADVERTISE=\"advertiseAddress: \$IPV6\"
-      SERVICE_SUBNET=\"serviceSubnet: fdff::/112\"
-      touch /tmp/install-cilium-ipv6-only
-      touch /tmp/ipv6-only
+    IPV4=\$(ip -json route get 1.1.1.1 | jq -r .[0].prefsrc)
+    IPV6=\$(ip -json route get 2600::  | jq -r .[0].prefsrc)
+    if [ \"\$IPV4\" ] && [ \"\$IPV6\" ]; then
+      KUBEADM_ADVERTISE=
+      SERVICE_SUBNET=10.96.0.0/12,fdff::/112
+      CILIUM_IPV6_ENABLED=true
+      CILIUM_IPV4_ENABLED=true
+      CILIUM_UNDERLAYPROTOCOL=ipv4
+    elif [ \"\$IPV6\" ]; then
+      KUBEADM_ADVERTISE=\"advertiseAddress: \$IPV6\"
+      SERVICE_SUBNET=fdff::/112
+      CILIUM_IPV6_ENABLED=true
+      CILIUM_IPV4_ENABLED=false
+      CILIUM_UNDERLAYPROTOCOL=ipv6
+      # Hack to skip krew and ngrok install
+      # (krew is broken on IPV6-only hosts, as it relies on GitHub)
+      sudo -u $USER_LOGIN mkdir /home/$USER_LOGIN/.krew
+      # (in 2025, Ngrok didn't install cleanly on IPV6-only hosts)
+      sudo ln -s /bin/false /usr/local/bin/ngrok
    else
-      ADVERTISE=
-      SERVICE_SUBNET=
-      touch /tmp/install-weave
+      KUBEADM_ADVERTISE=
+      KUBEADM_SERVICE_SUBNET=10.96.0.0/12
+      CILIUM_IPV6_ENABLED=false
+      CILIUM_IPV4_ENABLED=true
+      CILIUM_UNDERLAYPROTOCOL=ipv4
    fi
-    echo IPV6=\$IPV6 ADVERTISE=\$ADVERTISE
+    cat >/tmp/cilium.yaml <<EOF
+cni:
+  chainingMode: portmap
+ipv4:
+  enabled: \$CILIUM_IPV4_ENABLED
+ipv6:
+  enabled: \$CILIUM_IPV6_ENABLED
+underlayProtocol: \$CILIUM_UNDERLAYPROTOCOL
+EOF
    if i_am_first_node && [ ! -f /etc/kubernetes/admin.conf ]; then
        kubeadm token generate > /tmp/token &&
        cat >/tmp/kubeadm-config.yaml <<EOF
@@ -622,7 +645,7 @@ apiVersion: kubeadm.k8s.io/v1beta4
 bootstrapTokens:
 - token: \$(cat /tmp/token)
 localAPIEndpoint:
-  \$ADVERTISE
+  \$KUBEADM_ADVERTISE
 nodeRegistration:
  ignorePreflightErrors:
  - NumCPU
@@ -664,7 +687,7 @@ apiServer:
    readOnly: true
    pathType: File  
 networking:
-  \$SERVICE_SUBNET
+  serviceSubnet: \$SERVICE_SUBNET
 $CLUSTER_CONFIGURATION_KUBERNETESVERSION
 EOF
 	sudo kubeadm init --config=/tmp/kubeadm-config.yaml
@@ -680,23 +703,13 @@ EOF
        sudo chown -R $USER_LOGIN /home/$USER_LOGIN/.kube
    fi"

-    # Install weave as the pod network
+    # Set up CNI
    pssh "
    if i_am_first_node; then
-        if [ -f /tmp/install-weave ]; then
-            curl -fsSL \$GITHUB/weaveworks/weave/releases/download/v2.8.1/weave-daemonset-k8s-1.11.yaml |
-            sed s,weaveworks/weave,quay.io/rackspace/weave, |
-            kubectl apply -f-
-        fi
-        if [ -f /tmp/install-cilium-ipv6-only ]; then
-            helm upgrade -i cilium cilium --repo https://helm.cilium.io/ \
-            --namespace kube-system \
-            --set cni.chainingMode=portmap \
-            --set ipv6.enabled=true \
-            --set ipv4.enabled=false \
-            --set underlayProtocol=ipv6 \
-            --version 1.18.3
-        fi
+        helm upgrade -i cilium cilium --repo https://helm.cilium.io/ \
+        --namespace kube-system \
+        --values /tmp/cilium.yaml \
+        --version 1.18.3
    fi"

    # FIXME this is a gross hack to add the deployment key to our SSH agent,
@@ -878,7 +891,7 @@ EOF

    # Install the krew package manager
    pssh "
-    if [ ! -d /home/$USER_LOGIN/.krew ] && [ ! -f /tmp/ipv6-only ]; then
+    if [ ! -d /home/$USER_LOGIN/.krew ]; then
        cd /tmp &&
        KREW=krew-linux_$ARCH
        curl -fsSL \$GITHUB/kubernetes-sigs/krew/releases/latest/download/\$KREW.tar.gz |
@@ -997,7 +1010,7 @@ EOF
    # Ngrok. Note that unfortunately, this is the x86_64 binary.
    # We might have to rethink how to handle this for multi-arch environments.
    pssh "
-    if [ ! -x /usr/local/bin/ngrok ] && [ ! -f /tmp/ipv6-only ]; then
+    if [ ! -x /usr/local/bin/ngrok ]; then
        curl -fsSL https://bin.equinox.io/c/bNyj1mQVY4c/ngrok-v3-stable-linux-amd64.tgz |
        sudo tar -zxvf- -C /usr/local/bin ngrok
    fi"
--- a/slides/k8s/cni.md
+++ b/slides/k8s/cni.md
@@ -48,7 +48,7 @@ Look in each plugin's directory for its documentation.

 ## CNI configuration in practice

- When we set up the "pod network" (like Calico, Weave...) it ships a CNI configuration
+- When we set up the "pod network" (like Calico, Cilium...) it ships a CNI configuration

  (and sometimes, custom CNI plugins)

--- a/slides/k8s/daemonset.md
+++ b/slides/k8s/daemonset.md
@@ -34,9 +34,9 @@

 - Daemon sets are great for cluster-wide, per-node processes:

-  - `kube-proxy`
+  - network components like `kube-proxy`, `calico`, `cilium`...

-  - `weave` (our overlay network)
+  - storage components (e.g. CSI plugins)

  - monitoring agents

--- a/slides/k8s/kubectlget.md
+++ b/slides/k8s/kubectlget.md
@@ -310,13 +310,13 @@ class: extra-details

 - `coredns` provides DNS-based service discovery ([replacing kube-dns as of 1.11](https://kubernetes.io/blog/2018/07/10/coredns-ga-for-kubernetes-cluster-dns/))

- `kube-proxy` is the (per-node) component managing port mappings and such
+- `kube-proxy` is the (per-node) component managing internal service access

- `weave` is the (per-node) component managing the network overlay
+- `cilium` is the (per-node) component managing the network overlay

 - the `READY` column indicates the number of containers in each pod

-  (1 for most pods, but `weave` has 2, for instance)
+  (which will often be 1 for most pods!)

 ---

--- a/slides/k8s/logs-cli.md
+++ b/slides/k8s/logs-cli.md
@@ -117,13 +117,13 @@ Exactly what we need!

 .lab[

- View what's up with the `weave` system containers:
+- View what's up with the `coredns` system containers:
  ```bash
-  stern --tail 1 --timestamps --all-namespaces weave
+  stern --tail 1 --timestamps --all-namespaces coredns
  ```

 <!--
-```wait weave-npc```
+```wait coredns-```
 ```key ^C```
 -->

--- a/slides/k8s/netpol.md
+++ b/slides/k8s/netpol.md
@@ -240,7 +240,9 @@ The second command will fail and time out after 3 seconds.

 - For instance, Weave added support for egress rules [in version 2.4](https://github.com/weaveworks/weave/pull/3313) (released in July 2018)

- But only recently added support for ipBlock [in version 2.5](https://github.com/weaveworks/weave/pull/3367) (released in Nov 2018)
+- And added support for ipBlock [in version 2.5](https://github.com/weaveworks/weave/pull/3367) (released in Nov 2018)
+
+- Before 2023, AWS EKS didn't install a network policy controller at all

 - Unsupported features might be silently ignored

--- a/slides/k8s/setup-managed.md
+++ b/slides/k8s/setup-managed.md
@@ -367,11 +367,11 @@ class: extra-details

 - To see available versions and CNI plugins, run `scw k8s version list`

- As of May 2020, Kapsule supports:
+- As of May 2026, Kapsule supports:

-  - multiple CNI plugins, including: cilium, calico, weave, flannel
+  - multiple CNI plugins, including: calico, cilium, kilo

-  - Kubernetes versions 1.15 to 1.18
+  - Kubernetes versions 1.32 to 1.35

  - multiple container runtimes, including: Docker, containerd, CRI-O

--- a/slides/k8s/setup-overview.md
+++ b/slides/k8s/setup-overview.md
@@ -293,7 +293,7 @@ Don't hesitate to hire help before going to production with your first K8S app!

    3. Run `kubeadm init` on the first node (it deploys the control plane on that node)

-    4. Set up  Weave (the overlay network) with a single `kubectl apply` command
+    4. Set up  Cilium (the overlay network) with a single `helm install` command

    5. Run `kubeadm join` on the other nodes (with the token produced by `kubeadm init`)