k3k/docs/howtos/using-kata-containers.md

# Using Kata Containers

> **Experimental:** Kata Containers support is in early development. Expect rough edges.

Kata Containers runs each pod inside a lightweight QEMU VM, providing stronger isolation than runc. When used with k3k, this means each virtual cluster's server and agent pods run as hardware-virtualized guests on the host node.

This guide covers the full setup: installing Kata, configuring the QEMU runtime and devmapper snapshotter, preparing a custom K3s image, and deploying a cluster.

# Prerequisites

## 1. Requirements

- KVM available on the host node (`/dev/kvm` accessible)
- `vhost_net` and `vhost_vsock` kernel modules available
- `dmsetup` and `losetup` available (device-mapper utilities)
- K3s or RKE2 installed on the host

Load the required kernel modules:

```bash
sudo modprobe vhost_net
sudo modprobe vhost_vsock
```

To persist these across reboots, add them to `/etc/modules-load.d/kata.conf`.

## 2. Set Up the devmapper Snapshotter

The default snapshotter will use virtio-fs as the container filesystem, the default overlayfs snapshotter does not work with this type, so devmapper is required in order to provide a block device.

The script below creates a thin-pool backed by loopback devices. This is adequate for development and testing. The [device-mapper thin provisioning documentation](https://www.kernel.org/doc/html/latest/admin-guide/device-mapper/thin-provisioning.html) covers the full lifecycle, which is more involved than the example here. `erofs` may be a suitable replacement in future to eliminate the need to manage this pool entirely.

```bash
#!/bin/bash

DIR=/opt/devmapper
CONTAINERD_SOCK="/run/k3s/containerd/containerd.sock"

set -euxo pipefail

name="k3s"

if [ "$EUID" -ne 0 ]; then
  echo "Please run as root."
  exit 1
fi

create_loopback_device() {
    local path=$1
    local size=$2

    if [[ ! -f "$path" ]]; then
        touch "$path"
        truncate -s "$size" "$path"
    fi

    local dev=$(losetup --output NAME --noheadings --associated "$path")
    if [[ -z "$dev" ]]; then
        dev=$(losetup --find --show $path)
    fi
    echo $dev
}

pool_create() {
    mkdir -p $DIR

    local datadev=$(create_loopback_device "$DIR/data" '10G')
    local metadev=$(create_loopback_device "$DIR/metadata" '1G')

    if dmsetup info "$name" &>/dev/null; then
        echo "Thin pool '$name' already exists, ensuring it is active."
        dmsetup resume "$name" 2>/dev/null || true
        return
    fi

    local sectorsize=512
    local datasize="$(blockdev --getsize64 -q ${datadev})"
    local length_sectors=$(bc <<< "${datasize}/${sectorsize}")
    local thinp_table="0 ${length_sectors} thin-pool ${metadev} ${datadev} 128 32768 1 skip_block_zeroing"
    dmsetup create "$name" --table "${thinp_table}"
}

pool_create
```

The pool does not survive a reboot. Consider wrapping `pool_create` in a systemd service that runs before `rke2-server.service`.

## 3. Handling emptyDir mounts

The standard K3s image declares `emptyDir` volumes in its image manifest. Kata detects these as tmpfs mounts and fails to start the container. The k3k controller automatically strips `emptyDir` volumes from the pod spec at runtime when the cluster's `runtimeClassName` starts with `kata-`, but the declarations baked into the image manifest will still be in place by default.

We can avoid emptyDir mounts from image manifests by either of the following options:

### Option A: Ignore the image manifest volume declaration (recommended)

Configuring containerd to ignore the image manifest volumes is a simpler option, but note that it is a global setting and will apply to all runtimes on a particular node. **If you use the recommended Helm installation below, this configuration is handled automatically via the `ignore_image_defined_volumes` setting in the `kata-values.yaml`.**

### Option B: Build a custom K3s image

Build a custom image that omits those declarations:

```dockerfile
FROM rancher/k3s:v1.35.3-k3s1 AS rancher

FROM scratch

COPY --from=rancher / /

ENV PATH=/var/lib/rancher/k3s/data/cni:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin/aux
ENV CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
```

Push this image to a registry accessible from the host cluster and configure the controller to use it for Kata-based clusters.

# Helm Installation (Recommended)

Using Helm is the quickest and simplest way to install Kata Containers. It automates the distribution of Kata binaries, configures containerd, and registers the necessary RuntimeClasses across your cluster. It has been tested with RKE2 1.35 and Kata 3.32.

Ensure you are using a v3 containerd config. This will be the default on 1.35 if no template is provided, or you can create a blank v3 template at `/var/lib/rancher/rke2/agent/etc/containerd/config-v3.toml.tmpl`:

```toml
{{ template "base" . }}
```

Install the `kata-deploy` Helm chart:

```bash
helm upgrade --install kata-deploy oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy \
  --version 3.32.0 \
  --namespace kata-deploy \
  --create-namespace \
  --values=kata-values.yaml
```

Contents of `kata-values.yaml`:

```yaml
k8sDistribution: "rke2"

containerd:
  userDropIn: |
    version = 3

    [plugins.'io.containerd.cri.v1.runtime']
      ignore_image_defined_volumes = true

    [plugins.'io.containerd.snapshotter.v1.devmapper']
      pool_name = "k3s"
      root_path = "/opt/devmapper"
      base_image_size = "4GB"
      discard_blocks = true

snapshotter:
  setup: []

shims:
  disableAll: true

  qemu:
    enabled: true
    supportedArches:
      - amd64
    allowedHypervisorAnnotations: []
    containerd:
      snapshotter: "devmapper"
    dropIn: |
      [hypervisor.qemu]
      disable_block_device_use = false

debug: true

defaultShim:
  amd64: qemu
```

# Manual Installation (Advanced)

> **Warning:** Manual installation is highly dependent on your host OS and specific environment. The steps below should be treated as a general guide. Helm is strongly recommended for a simpler and more reliable setup.

## 1. Install Kata Containers

Download and extract the Kata static binary bundle, then symlink the containerd shim:

```bash
curl -sSL https://github.com/kata-containers/kata-containers/releases/download/3.32.0/kata-static-3.32.0-amd64.tar.zst \
  | sudo tar --zstd -xvf - -C /

sudo ln -s /opt/kata/runtime-rs/bin/containerd-shim-kata-v2 /usr/local/bin/containerd-shim-kata-v2
```

## 2. Configure the QEMU Runtime

The default Kata QEMU config requires a few changes. The key ones switch the rootfs and block device transports from `virtio-pmem`/`virtio-scsi` to `virtio-blk-pci`, which is necessary because the devmapper snapshotter provides block devices rather than overlay filesystems. `shared_fs` is also disabled since `virtio-fs` is not needed when using block-backed storage.

```bash
sudo sed -i \
  -e 's/vm_rootfs_driver = "virtio-pmem"/vm_rootfs_driver = "virtio-blk-pci"/' \
  -e 's/shared_fs = "virtio-fs"/shared_fs = "none"/' \
  -e 's/block_device_driver = "virtio-scsi"/block_device_driver = "virtio-blk-pci"/' \
  -e 's/disable_image_nvdimm = false/disable_image_nvdimm = true/' \
  /opt/kata/share/defaults/kata-containers/runtime-rs/configuration-qemu-runtime-rs.toml
```

## 3. Configure containerd

Place the following template at `/var/lib/rancher/rke2/agent/etc/containerd/config-v3.toml.d/custom.toml`. It registers the `kata-qemu` runtime handler and configures the devmapper snapshotter.

```toml
version = 3

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.kata-qemu]
  runtime_type = "io.containerd.kata.v2"
  snapshotter = "devmapper"
  privileged_without_host_devices = true
  pod_annotations = ["io.katacontainers.*"]

[plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.kata-qemu.options]
  ConfigPath = "/opt/kata/share/defaults/kata-containers/runtime-rs/configuration-qemu-runtime-rs.toml"

[plugins.'io.containerd.snapshotter.v1.devmapper']
  pool_name = "k3s"
  root_path = "/opt/devmapper"
  base_image_size = "4GB"
  discard_blocks = true
```

If you are using Option A from the "Handling emptyDir mounts" section, you must also add the following to `/var/lib/rancher/rke2/agent/etc/containerd/config-v3.toml.d/volumes.toml`:

```toml
version = 3

[plugins.'io.containerd.cri.v1.runtime']
  ignore_image_defined_volumes = true
```

### 5.2 Build a custom K3S image

Build a custom image that omits those declarations:

```dockerfile
FROM rancher/k3s:v1.36.2-k3s1 AS rancher

FROM scratch

COPY --from=rancher / /

ENV PATH=/var/lib/rancher/k3s/data/cni:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/bin/aux
ENV CRI_CONFIG_FILE=/var/lib/rancher/k3s/agent/etc/crictl.yaml
```

Push this image to a registry accessible from the host cluster and configure the controller to use it for Kata-based clusters.

An alternative would be an OCI hook to strip the mounts at runtime, but that falls outside the scope of using the Kata project's supplied build artifacts.


## 6. Load Kernel Modules

`vhost_net` and `vhost_vsock` are required for VM networking and host-to-guest communication over vsock:

```bash
modprobe vhost_net
modprobe vhost_vsock
```

To persist these across reboots, add them to `/etc/modules-load.d/kata.conf`.

## 7. Create the RuntimeClass

The `RuntimeClass` name **must start with `kata-`**. This prefix is how the k3k controller identifies Kata-based clusters and applies the necessary pod spec adjustments: stripping `emptyDir` volumes, injecting a `/dev/kmsg` host path mount, and configuring cgroup handling in the K3s startup script.

```yaml
apiVersion: node.k8s.io/v1
kind: RuntimeClass
handler: kata-qemu
metadata:
  name: kata-qemu
```

# Create a Cluster

Kata-based clusters require `mode: virtual` and `persistence.type: ephemeral`:

- **Virtual mode** is required because Kata needs a full K3s agent with its own container runtime running on the host. Shared mode uses a virtual kubelet that bypasses node-level runtimes entirely.
- **Ephemeral persistence** is required because dynamic persistence creates a PVC that conflicts with how Kata manages block device storage inside the guest VM.

The example below uses an external PostgreSQL datastore via `secretMounts` to provide persistence.

```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: test-k3k
---
apiVersion: v1
kind: Secret
metadata:
  name: datastore-config
  namespace: test-k3k
type: Opaque
stringData:
  config.yaml: |
    datastore-endpoint: postgres://username:password@host:5432/dbname
    cluster-init: false
    server: ""
---
apiVersion: k3k.io/v1beta1
kind: Cluster
metadata:
  name: test
  namespace: test-k3k
spec:
  mode: virtual
  runtimeClassName: kata-qemu
  persistence:
    type: ephemeral
  servers: 3
  secretMounts:
    - name: externaldb-init-config
      secretName: datastore-config
      mountPath: /opt/rancher/k3s/init/config.yaml.d/
      role: server
    - name: externaldb-server-config
      secretName: datastore-config
      mountPath: /opt/rancher/k3s/server/config.yaml.d/
```

# Notes

- Currently `runtime-rs` is not supported when SELinux is enabled.