diff --git a/helm-chart/PF_RING.md b/helm-chart/PF_RING.md new file mode 100644 index 000000000..bae39df34 --- /dev/null +++ b/helm-chart/PF_RING.md @@ -0,0 +1,189 @@ +# PF_RING + + + +- [PF_RING](#pf_ring) + - [Overview](#overview) + - [Provisioning mode](#provisioning-mode) + - [Selection of Provisioning Mode](#selection-of-provisioning-mode) + - [Pre-built kernel module exists and external egress allowed](#pre-built-kernel-module-exists-and-external-egress-allowed) + - [Pre-built kernel module doesn't exist or external egress isn't allowed](#pre-built-kernel-module-doesnt-exist-or-external-egress-isnt-allowed) + - [Steps to Use kmm with Custom Containers](#steps-to-use-kmm-with-custom-containers) + - [Appendix A: PF_RING kernel module compilation](#appendix-a-pf_ring-kernel-module-compilation) + - [Automated complilation](#automated-complilation) + - [Manual compilation](#manual-compilation) + + + +## Overview + +PF_RING™ is an advanced Linux kernel module and user-space framework designed for high-speed packet processing. It offers a uniform API for packet processing applications, enabling efficient handling of large volumes of network data. + +For comprehensive information on PF_RING™, please visit the [User's Guide]((https://www.ntop.org/guides/pf_ring) and access detailed [API Documentation](http://www.ntop.org/guides/pf_ring_api/files.html). + +## Provisioning mode + +There are two approaches for loading the PF_RING kernel module on nodes: + +1. `auto` + +In this mode, the Kubeshark worker retrieves the necessary PF_RING kernel module version from an S3 bucket and loads it onto the node. + +> mode=auto requires an active internet connection and is not suitable for air-gapped environments. + +2. `kmm` + +The Kernel Module Management controller ([KMM](https://kmm.sigs.k8s.io/documentation/deploy_kmod/)) acquires the required PF_RING kernel module version from a Docker container and loads it onto the node + +> mode=kmm is suitable for air-gapped environments. + +## Selection of Provisioning Mode + +> This step is optional. In case mode=auto and no PF_RING kernel module is found Kubeshark falls back to `libpcap` if `PF_RING` kernel module not available + +Prior to choosing a method, it is essential to verify if a PF_RING kernel module is already built for your kernel version. +Kubeshark provides additional CLI tool for this purpose - [pf-ring-compiler](https://github.com/kubeshark/pf-ring-compiler). + +Compatibility verification can be done by running: + +``` +pfring-compiler compatibility +``` + +This command checks for the availability of kernel modules for the kernel versions running across all nodes in the Kubernetes cluster. + +Example output for a compatible cluster: +``` +Node Kernel Version Supported +ip-192-168-77-230.us-west-2.compute.internal 5.10.199-190.747.amzn2.x86_64 true +ip-192-168-34-216.us-west-2.compute.internal 5.10.199-190.747.amzn2.x86_64 true + +Cluster is compatible +``` + + +### Pre-built kernel module exists and external egress allowed + +If PF_RING kernel modules are already available for the target nodes (cluster is compatible), both `auto` and `kmm` modes are applicable. + +|auto|kmm| +|----|---| +| `SYS_MODULE` capability required for Kubeshark | `SYS_MODULE` capability is **not** required for Kubeshark| +| no additional dependencies | (!)requires `cert-manager` and `KMM` installed (follow [instructions](https://kmm.sigs.k8s.io/documentation/install/) or a specific [cloud platform guide](https://kmm.sigs.k8s.io/lab/)) | +| Kubeshark falls back to `libpcap` if `PF_RING` kernel module not available | Kubshark waits until PF_RING is loaded with KMM| +| module is downloaded from S3 bucket in AWS | module is loaded from `kubeshark/pf-ring-module:` container| +| requires egress connectivity to AWS S3 endpoints | can work in an air-gapped environment when the docker images are stored in a local container registry| + + +### Pre-built kernel module doesn't exist or external egress isn't allowed + +In cases where PF_RING kernel modules are not yet available for the target nodes, or if external egress is restricted, the `kmm` mode is the only viable option (`auto` mode would start Kubeshark with libpcap, not PF_RING). +This approach enables the use of custom container images as the source for PF_RING kernel modules and allows leveraging private container registries. + +#### Steps to Use kmm with Custom Containers + +1. Compile the pf_ring.ko kernel module for your target kernel version (see [Appendix B](#appendix-b-pf_ring-kernel-module-compilation) for instructions). + +After building the module with kubeshark pfring compile, you will obtain a `pf-ring-.ko` file. +If manually built, rename the kernel module to this format. + +2. Build and push Docker container(-s) with the kernel module file from stage 1. + +Create `Dockerfile` in the folder with PF_RING kernel module: + +``` +FROM alpine:3.18 +ARG KERNEL_VERSION + +COPY pf-ring-${KERNEL_VERSION}.ko /opt/lib/modules/${KERNEL_VERSION}/pf_ring.ko +RUN apk add kmod + +RUN depmod -b /opt ${KERNEL_VERSION} +``` + +Run build&command: + +``` +docker build --build-arg /: +docker push :/: +``` + +It is recommended to use kernel version as a container tag for consistency. + + +3. Configure Helm values + +``` +tap: + kernelModule: + mode: kmm + kernelMappings: + - regexp: '' + containerImage: '/:' + imageRepoSecret: +``` + + +## Appendix A: PF_RING kernel module compilation + +PF_RING kernel module compilation can be completed automatically or manually. + +### Automated complilation + +In case your Kubernetes workers run supported Linux distribution, `kubeshark` CLI can be used to build PF_RING module: + +``` +pfring-compiler compile --target +``` + +This command requires: +- kubectl to be installed and configured with a proper context +- egress connection to Internet available + +This command: +1. Runs Kubernetes job with build container +2. Waits for job to be completed +3. Downloads `pf-ring-.ko` file into the current folder. +4. Cleans up created job. + +Currently supported distros: +- Ubuntu +- RHEL 9 +- Amazon Linux 2 + +### Manual compilation + +The process description is based on Ubuntu 22.04 distribution. + +1. Get terminal access to the node with target kernel version +This can be done either via SSH directly to node or with debug container running on the target node: + +``` +kubectl debug node/ -it --attach=true --image=ubuntu:22.04 +``` + +2. Install build tools and kernel headers + +``` +apt update +apt install -y gcc build-essential make git wget tar gzip +apt install -y linux-headers-$(uname -r) +``` + +3. Download PF_RING source code + +``` +wget https://github.com/ntop/PF_RING/archive/refs/tags/8.4.0.tar.gz +tar -xf 8.4.0.tar.gz +cd PF_RING-8.4.0/kernel +``` + +4. Compile the kernel module + +``` +make KERNEL_SRC=/usr/src/linux-headers-$(uname -r) +``` + +5. Copy `pf_ring.ko` to the local file system. + +Use `scp` or `kubectl cp` depending on type of access(SSH or debug pod). diff --git a/helm-chart/README.md b/helm-chart/README.md index 3673325e0..f39e6f7e5 100644 --- a/helm-chart/README.md +++ b/helm-chart/README.md @@ -94,7 +94,7 @@ For example, change from the default 500Mi to 1Gi: ```shell --set tap.storageLimit=1Gi ``` - + ## Disabling IPV6 Not all have IPV6 enabled, hence this has to be disabled as follows: @@ -152,7 +152,10 @@ helm install kubeshark kubeshark/kubeshark \ | `tap.ingress.annotations` | `Ingress` annotations | `{}` | | `tap.ipv6` | Enable IPv6 support for the front-end | `true` | | `tap.debug` | Enable debug mode | `false` | -| `tap.noKernelModule` | Do not install `PF_RING` kernel module | `false` | +| `tap.kernelModule.enabled` | Use PF_RING kernel module([details](PF_RING.md)) | `true` | +| `tap.kernelModule.mode` | PF_RING kernel module loading approach([details](PF_RING.md)) | `auto` | +| `tap.kernelModule.imageRepoSecret` | ImageRepoSecret is an optional secret that is used to pull both the module loader container([details](PF_RING.md)) | "" | +| `tap.kernelModule.kernelMappings` |List of mappings between kernel version and container loader([details](PF_RING.md)) | `[{'regexp': '.+$', 'containerImage': 'kubehq/pf-ring-module:${KERNEL_FULL_VERSION}'}]` | | `tap.telemetry.enabled` | Enable anonymous usage statistics collection | `true` | | `logs.file` | Logs dump path | `""` | | `kube.configPath` | Path to the `kubeconfig` file (`$HOME/.kube/config`) | `""` | @@ -163,3 +166,7 @@ helm install kubeshark kubeshark/kubeshark \ | `scripting.env` | Environment variables for the scripting | `{}` | | `scripting.source` | Source directory of the scripts | `""` | | `scripting.watchScripts` | Enable watch mode for the scripts in source directory | `true` | + +KernelMapping pairs kernel versions with a + DriverContainer image. Kernel versions can be matched + literally or using a regular expression \ No newline at end of file diff --git a/helm-chart/templates/09-worker-daemon-set.yaml b/helm-chart/templates/09-worker-daemon-set.yaml index 4702df8c7..41ce0f878 100644 --- a/helm-chart/templates/09-worker-daemon-set.yaml +++ b/helm-chart/templates/09-worker-daemon-set.yaml @@ -25,6 +25,23 @@ spec: name: kubeshark-worker-daemon-set namespace: kubeshark spec: + {{ if and (eq .Values.tap.kernelModule.enabled true) (eq .Values.tap.kernelModule.mode "kmm")}} + initContainers: + - name: wait-for-pf-ring + image: alpine:3.18 + command: ["/bin/sh", "-c"] + args: + - > + while true; do + if lsmod | grep -q "pf_ring"; then + echo "pf_ring module is loaded."; + break; + else + echo "Waiting for pf_ring module to be loaded..."; + sleep 5; + fi + done + {{ end }} containers: - command: - ./worker @@ -37,12 +54,12 @@ spec: {{- end }} - -procfs - /hostproc + {{- if or (eq .Values.tap.kernelModule.enabled false) (eq .Values.tap.kernelModule.mode "kmm") }} + - -no-kernel-module + {{ end }} {{- if .Values.tap.debug }} - -debug {{- end }} - {{- if .Values.tap.noKernelModule }} - - -no-kernel-module - {{- end }} image: '{{ .Values.tap.docker.registry }}/worker:{{ not (eq .Values.tap.docker.tag "") | ternary .Values.tap.docker.tag (printf "v%s" .Chart.Version) }}' imagePullPolicy: {{ .Values.tap.docker.imagePullPolicy }} name: sniffer @@ -68,7 +85,7 @@ spec: {{- range .Values.tap.capabilities.networkCapture }} {{ print "- " . }} {{- end }} - {{- if not .Values.tap.noKernelModule }} + {{- if and (.Values.tap.kernelModule.enabled) (eq .Values.tap.kernelModule.mode "auto") }} {{- range .Values.tap.capabilities.kernelModule }} {{ print "- " . }} {{- end }} diff --git a/helm-chart/templates/15-pf-ring-kernel-module.yaml b/helm-chart/templates/15-pf-ring-kernel-module.yaml new file mode 100644 index 000000000..ae53a4609 --- /dev/null +++ b/helm-chart/templates/15-pf-ring-kernel-module.yaml @@ -0,0 +1,84 @@ +--- +{{- if .Values.tap.kernelModule.enabled }} +{{- if not (or (eq .Values.tap.kernelModule.mode "auto") (eq .Values.tap.kernelModule.mode "kmm")) }} +{{- fail "Invalid value for tap.kernelModule.mode; must be 'auto' or 'kmm'" }} +{{- end }} +{{- if eq .Values.tap.kernelModule.mode "kmm" }} +apiVersion: kmm.sigs.x-k8s.io/v1beta1 +kind: Module +metadata: + name: pf-ring +spec: + moduleLoader: + container: + modprobe: + moduleName: pf_ring + dirName: /opt + kernelMappings: + - + {{- range .Values.tap.kernelModule.kernelMappings }} + {{- toYaml . | nindent 8 }} + {{- end }} + selector: + kubernetes.io/hostname: + {{- if ne .Values.tap.kernelModule.imageRepoSecret "" }} + imageRepoSecret: + name: {{ .Values.tap.kernelModule.imageRepoSecret }} + {{- end }} +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: pfring-kmm-status-check-script +data: + check_status.sh: | + #!/bin/bash + + timeout=300 + interval=10 + elapsed=0 + + while [ $elapsed -lt $timeout ]; do + desired=$(kubectl get modules pf-ring -o json | jq -r .status.moduleLoader.desiredNumber) + available=$(kubectl get modules pf-ring -o json | jq -r .status.moduleLoader.availableNumber) + + echo "Checking desired and available module load: $desired vs $available" + + if [ "$desired" = "$available" ]; then + echo "PF_RING loaded an all nodes." + exit 0 + else + sleep $interval + elapsed=$((elapsed + interval)) + fi + done + + echo "Timeout reached. PF_RING module didn't reach desired status." + exit 1 + +--- +apiVersion: batch/v1 +kind: Job +metadata: + name: pfring-kmm-status-check + annotations: + "helm.sh/hook": post-install,post-upgrade + "helm.sh/hook-weight": "1" + "helm.sh/hook-delete-policy": hook-succeeded +spec: + template: + spec: + containers: + - name: verify-pfring-kmm-status + image: alpine/k8s:1.27.8 + command: ["/bin/bash", "/scripts/check_status.sh"] + volumeMounts: + - name: script-volume + mountPath: /scripts + volumes: + - name: script-volume + configMap: + name: pfring-kmm-status-check-script + restartPolicy: Never +{{- end }} +{{- end }} diff --git a/helm-chart/values.yaml b/helm-chart/values.yaml index e06c051c5..76cf4a9e0 100644 --- a/helm-chart/values.yaml +++ b/helm-chart/values.yaml @@ -63,7 +63,13 @@ tap: annotations: {} ipv6: true debug: false - noKernelModule: false + kernelModule: + enabled: true + mode: auto + kernelMappings: + - regexp: '^.+$' + containerImage: 'kubeshark/pf-ring-module:${KERNEL_FULL_VERSION}' + imageRepoSecret: "" telemetry: enabled: true defaultFilter: ""