mirror of
https://github.com/kubeshark/kubeshark.git
synced 2026-07-27 01:03:34 +00:00
Add PF_RING related changes to docs and helm (#1471)
* Install pf-ring KMM Module and wait for it * Add mode configuration * save * Update doc * upd * toc * adjust template * upd * Add module cr verification job * upd doc * Fix binary name * Add disable mode * Update PF_RING.md Some adjustments to the instructions. * Update 15-pf-ring-kernel-module.yaml Small syntax err * upd * merge master --------- Co-authored-by: Alon Girmonsky <1990761+alongir@users.noreply.github.com>
This commit is contained in:
Volodymyr Stoiko
GitHub
189
helm-chart/PF_RING.md
Normal file
189
helm-chart/PF_RING.md
Normal file
@@ -0,0 +1,189 @@
|
||||
# PF_RING
|
||||
|
||||
<!-- TOC -->
|
||||
|
||||
- [PF_RING](#pf_ring)
|
||||
- [Overview](#overview)
|
||||
- [Provisioning mode](#provisioning-mode)
|
||||
- [Selection of Provisioning Mode](#selection-of-provisioning-mode)
|
||||
- [Pre-built kernel module exists and external egress allowed](#pre-built-kernel-module-exists-and-external-egress-allowed)
|
||||
- [Pre-built kernel module doesn't exist or external egress isn't allowed](#pre-built-kernel-module-doesnt-exist-or-external-egress-isnt-allowed)
|
||||
- [Steps to Use kmm with Custom Containers](#steps-to-use-kmm-with-custom-containers)
|
||||
- [Appendix A: PF_RING kernel module compilation](#appendix-a-pf_ring-kernel-module-compilation)
|
||||
- [Automated complilation](#automated-complilation)
|
||||
- [Manual compilation](#manual-compilation)
|
||||
|
||||
<!-- /TOC -->
|
||||
|
||||
## Overview
|
||||
|
||||
PF_RING™ is an advanced Linux kernel module and user-space framework designed for high-speed packet processing. It offers a uniform API for packet processing applications, enabling efficient handling of large volumes of network data.
|
||||
|
||||
For comprehensive information on PF_RING™, please visit the [User's Guide]((https://www.ntop.org/guides/pf_ring) and access detailed [API Documentation](http://www.ntop.org/guides/pf_ring_api/files.html).
|
||||
|
||||
## Provisioning mode
|
||||
|
||||
There are two approaches for loading the PF_RING kernel module on nodes:
|
||||
|
||||
1. `auto`
|
||||
|
||||
In this mode, the Kubeshark worker retrieves the necessary PF_RING kernel module version from an S3 bucket and loads it onto the node.
|
||||
|
||||
> mode=auto requires an active internet connection and is not suitable for air-gapped environments.
|
||||
|
||||
2. `kmm`
|
||||
|
||||
The Kernel Module Management controller ([KMM](https://kmm.sigs.k8s.io/documentation/deploy_kmod/)) acquires the required PF_RING kernel module version from a Docker container and loads it onto the node
|
||||
|
||||
> mode=kmm is suitable for air-gapped environments.
|
||||
|
||||
## Selection of Provisioning Mode
|
||||
|
||||
> This step is optional. In case mode=auto and no PF_RING kernel module is found Kubeshark falls back to `libpcap` if `PF_RING` kernel module not available
|
||||
|
||||
Prior to choosing a method, it is essential to verify if a PF_RING kernel module is already built for your kernel version.
|
||||
Kubeshark provides additional CLI tool for this purpose - [pf-ring-compiler](https://github.com/kubeshark/pf-ring-compiler).
|
||||
|
||||
Compatibility verification can be done by running:
|
||||
|
||||
```
|
||||
pfring-compiler compatibility
|
||||
```
|
||||
|
||||
This command checks for the availability of kernel modules for the kernel versions running across all nodes in the Kubernetes cluster.
|
||||
|
||||
Example output for a compatible cluster:
|
||||
```
|
||||
Node Kernel Version Supported
|
||||
ip-192-168-77-230.us-west-2.compute.internal 5.10.199-190.747.amzn2.x86_64 true
|
||||
ip-192-168-34-216.us-west-2.compute.internal 5.10.199-190.747.amzn2.x86_64 true
|
||||
|
||||
Cluster is compatible
|
||||
```
|
||||
|
||||
|
||||
### Pre-built kernel module exists and external egress allowed
|
||||
|
||||
If PF_RING kernel modules are already available for the target nodes (cluster is compatible), both `auto` and `kmm` modes are applicable.
|
||||
|
||||
|auto|kmm|
|
||||
|----|---|
|
||||
| `SYS_MODULE` capability required for Kubeshark | `SYS_MODULE` capability is **not** required for Kubeshark|
|
||||
| no additional dependencies | (!)requires `cert-manager` and `KMM` installed (follow [instructions](https://kmm.sigs.k8s.io/documentation/install/) or a specific [cloud platform guide](https://kmm.sigs.k8s.io/lab/)) |
|
||||
| Kubeshark falls back to `libpcap` if `PF_RING` kernel module not available | Kubshark waits until PF_RING is loaded with KMM|
|
||||
| module is downloaded from S3 bucket in AWS | module is loaded from `kubeshark/pf-ring-module:<kernel version>` container|
|
||||
| requires egress connectivity to AWS S3 endpoints | can work in an air-gapped environment when the docker images are stored in a local container registry|
|
||||
|
||||
|
||||
### Pre-built kernel module doesn't exist or external egress isn't allowed
|
||||
|
||||
In cases where PF_RING kernel modules are not yet available for the target nodes, or if external egress is restricted, the `kmm` mode is the only viable option (`auto` mode would start Kubeshark with libpcap, not PF_RING).
|
||||
This approach enables the use of custom container images as the source for PF_RING kernel modules and allows leveraging private container registries.
|
||||
|
||||
#### Steps to Use kmm with Custom Containers
|
||||
|
||||
1. Compile the pf_ring.ko kernel module for your target kernel version (see [Appendix B](#appendix-b-pf_ring-kernel-module-compilation) for instructions).
|
||||
|
||||
After building the module with kubeshark pfring compile, you will obtain a `pf-ring-<kernel version>.ko` file.
|
||||
If manually built, rename the kernel module to this format.
|
||||
|
||||
2. Build and push Docker container(-s) with the kernel module file from stage 1.
|
||||
|
||||
Create `Dockerfile` in the folder with PF_RING kernel module:
|
||||
|
||||
```
|
||||
FROM alpine:3.18
|
||||
ARG KERNEL_VERSION
|
||||
|
||||
COPY pf-ring-${KERNEL_VERSION}.ko /opt/lib/modules/${KERNEL_VERSION}/pf_ring.ko
|
||||
RUN apk add kmod
|
||||
|
||||
RUN depmod -b /opt ${KERNEL_VERSION}
|
||||
```
|
||||
|
||||
Run build&command:
|
||||
|
||||
```
|
||||
docker build --build-arg <kernel version> <your registry>/<image>:<kernel version>
|
||||
docker push <your registry>:/<image>:<kernel version>
|
||||
```
|
||||
|
||||
It is recommended to use kernel version as a container tag for consistency.
|
||||
|
||||
|
||||
3. Configure Helm values
|
||||
|
||||
```
|
||||
tap:
|
||||
kernelModule:
|
||||
mode: kmm
|
||||
kernelMappings:
|
||||
- regexp: '<kernel version>'
|
||||
containerImage: '<your-registry>/<image>:<kernel version>'
|
||||
imageRepoSecret: <optional secret with credentials for private registry>
|
||||
```
|
||||
|
||||
|
||||
## Appendix A: PF_RING kernel module compilation
|
||||
|
||||
PF_RING kernel module compilation can be completed automatically or manually.
|
||||
|
||||
### Automated complilation
|
||||
|
||||
In case your Kubernetes workers run supported Linux distribution, `kubeshark` CLI can be used to build PF_RING module:
|
||||
|
||||
```
|
||||
pfring-compiler compile --target <distro>
|
||||
```
|
||||
|
||||
This command requires:
|
||||
- kubectl to be installed and configured with a proper context
|
||||
- egress connection to Internet available
|
||||
|
||||
This command:
|
||||
1. Runs Kubernetes job with build container
|
||||
2. Waits for job to be completed
|
||||
3. Downloads `pf-ring-<kernel version>.ko` file into the current folder.
|
||||
4. Cleans up created job.
|
||||
|
||||
Currently supported distros:
|
||||
- Ubuntu
|
||||
- RHEL 9
|
||||
- Amazon Linux 2
|
||||
|
||||
### Manual compilation
|
||||
|
||||
The process description is based on Ubuntu 22.04 distribution.
|
||||
|
||||
1. Get terminal access to the node with target kernel version
|
||||
This can be done either via SSH directly to node or with debug container running on the target node:
|
||||
|
||||
```
|
||||
kubectl debug node/<target node> -it --attach=true --image=ubuntu:22.04
|
||||
```
|
||||
|
||||
2. Install build tools and kernel headers
|
||||
|
||||
```
|
||||
apt update
|
||||
apt install -y gcc build-essential make git wget tar gzip
|
||||
apt install -y linux-headers-$(uname -r)
|
||||
```
|
||||
|
||||
3. Download PF_RING source code
|
||||
|
||||
```
|
||||
wget https://github.com/ntop/PF_RING/archive/refs/tags/8.4.0.tar.gz
|
||||
tar -xf 8.4.0.tar.gz
|
||||
cd PF_RING-8.4.0/kernel
|
||||
```
|
||||
|
||||
4. Compile the kernel module
|
||||
|
||||
```
|
||||
make KERNEL_SRC=/usr/src/linux-headers-$(uname -r)
|
||||
```
|
||||
|
||||
5. Copy `pf_ring.ko` to the local file system.
|
||||
|
||||
Use `scp` or `kubectl cp` depending on type of access(SSH or debug pod).
|
||||
@@ -94,7 +94,7 @@ For example, change from the default 500Mi to 1Gi:
|
||||
```shell
|
||||
--set tap.storageLimit=1Gi
|
||||
```
|
||||
|
||||
|
||||
## Disabling IPV6
|
||||
|
||||
Not all have IPV6 enabled, hence this has to be disabled as follows:
|
||||
@@ -152,7 +152,10 @@ helm install kubeshark kubeshark/kubeshark \
|
||||
| `tap.ingress.annotations` | `Ingress` annotations | `{}` |
|
||||
| `tap.ipv6` | Enable IPv6 support for the front-end | `true` |
|
||||
| `tap.debug` | Enable debug mode | `false` |
|
||||
| `tap.noKernelModule` | Do not install `PF_RING` kernel module | `false` |
|
||||
| `tap.kernelModule.enabled` | Use PF_RING kernel module([details](PF_RING.md)) | `true` |
|
||||
| `tap.kernelModule.mode` | PF_RING kernel module loading approach([details](PF_RING.md)) | `auto` |
|
||||
| `tap.kernelModule.imageRepoSecret` | ImageRepoSecret is an optional secret that is used to pull both the module loader container([details](PF_RING.md)) | "" |
|
||||
| `tap.kernelModule.kernelMappings` |List of mappings between kernel version and container loader([details](PF_RING.md)) | `[{'regexp': '.+$', 'containerImage': 'kubehq/pf-ring-module:${KERNEL_FULL_VERSION}'}]` |
|
||||
| `tap.telemetry.enabled` | Enable anonymous usage statistics collection | `true` |
|
||||
| `logs.file` | Logs dump path | `""` |
|
||||
| `kube.configPath` | Path to the `kubeconfig` file (`$HOME/.kube/config`) | `""` |
|
||||
@@ -163,3 +166,7 @@ helm install kubeshark kubeshark/kubeshark \
|
||||
| `scripting.env` | Environment variables for the scripting | `{}` |
|
||||
| `scripting.source` | Source directory of the scripts | `""` |
|
||||
| `scripting.watchScripts` | Enable watch mode for the scripts in source directory | `true` |
|
||||
|
||||
KernelMapping pairs kernel versions with a
|
||||
DriverContainer image. Kernel versions can be matched
|
||||
literally or using a regular expression
|
||||
@@ -25,6 +25,23 @@ spec:
|
||||
name: kubeshark-worker-daemon-set
|
||||
namespace: kubeshark
|
||||
spec:
|
||||
{{ if and (eq .Values.tap.kernelModule.enabled true) (eq .Values.tap.kernelModule.mode "kmm")}}
|
||||
initContainers:
|
||||
- name: wait-for-pf-ring
|
||||
image: alpine:3.18
|
||||
command: ["/bin/sh", "-c"]
|
||||
args:
|
||||
- >
|
||||
while true; do
|
||||
if lsmod | grep -q "pf_ring"; then
|
||||
echo "pf_ring module is loaded.";
|
||||
break;
|
||||
else
|
||||
echo "Waiting for pf_ring module to be loaded...";
|
||||
sleep 5;
|
||||
fi
|
||||
done
|
||||
{{ end }}
|
||||
containers:
|
||||
- command:
|
||||
- ./worker
|
||||
@@ -37,12 +54,12 @@ spec:
|
||||
{{- end }}
|
||||
- -procfs
|
||||
- /hostproc
|
||||
{{- if or (eq .Values.tap.kernelModule.enabled false) (eq .Values.tap.kernelModule.mode "kmm") }}
|
||||
- -no-kernel-module
|
||||
{{ end }}
|
||||
{{- if .Values.tap.debug }}
|
||||
- -debug
|
||||
{{- end }}
|
||||
{{- if .Values.tap.noKernelModule }}
|
||||
- -no-kernel-module
|
||||
{{- end }}
|
||||
image: '{{ .Values.tap.docker.registry }}/worker:{{ not (eq .Values.tap.docker.tag "") | ternary .Values.tap.docker.tag (printf "v%s" .Chart.Version) }}'
|
||||
imagePullPolicy: {{ .Values.tap.docker.imagePullPolicy }}
|
||||
name: sniffer
|
||||
@@ -68,7 +85,7 @@ spec:
|
||||
{{- range .Values.tap.capabilities.networkCapture }}
|
||||
{{ print "- " . }}
|
||||
{{- end }}
|
||||
{{- if not .Values.tap.noKernelModule }}
|
||||
{{- if and (.Values.tap.kernelModule.enabled) (eq .Values.tap.kernelModule.mode "auto") }}
|
||||
{{- range .Values.tap.capabilities.kernelModule }}
|
||||
{{ print "- " . }}
|
||||
{{- end }}
|
||||
|
||||
84
helm-chart/templates/15-pf-ring-kernel-module.yaml
Normal file
84
helm-chart/templates/15-pf-ring-kernel-module.yaml
Normal file
@@ -0,0 +1,84 @@
|
||||
---
|
||||
{{- if .Values.tap.kernelModule.enabled }}
|
||||
{{- if not (or (eq .Values.tap.kernelModule.mode "auto") (eq .Values.tap.kernelModule.mode "kmm")) }}
|
||||
{{- fail "Invalid value for tap.kernelModule.mode; must be 'auto' or 'kmm'" }}
|
||||
{{- end }}
|
||||
{{- if eq .Values.tap.kernelModule.mode "kmm" }}
|
||||
apiVersion: kmm.sigs.x-k8s.io/v1beta1
|
||||
kind: Module
|
||||
metadata:
|
||||
name: pf-ring
|
||||
spec:
|
||||
moduleLoader:
|
||||
container:
|
||||
modprobe:
|
||||
moduleName: pf_ring
|
||||
dirName: /opt
|
||||
kernelMappings:
|
||||
-
|
||||
{{- range .Values.tap.kernelModule.kernelMappings }}
|
||||
{{- toYaml . | nindent 8 }}
|
||||
{{- end }}
|
||||
selector:
|
||||
kubernetes.io/hostname:
|
||||
{{- if ne .Values.tap.kernelModule.imageRepoSecret "" }}
|
||||
imageRepoSecret:
|
||||
name: {{ .Values.tap.kernelModule.imageRepoSecret }}
|
||||
{{- end }}
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: pfring-kmm-status-check-script
|
||||
data:
|
||||
check_status.sh: |
|
||||
#!/bin/bash
|
||||
|
||||
timeout=300
|
||||
interval=10
|
||||
elapsed=0
|
||||
|
||||
while [ $elapsed -lt $timeout ]; do
|
||||
desired=$(kubectl get modules pf-ring -o json | jq -r .status.moduleLoader.desiredNumber)
|
||||
available=$(kubectl get modules pf-ring -o json | jq -r .status.moduleLoader.availableNumber)
|
||||
|
||||
echo "Checking desired and available module load: $desired vs $available"
|
||||
|
||||
if [ "$desired" = "$available" ]; then
|
||||
echo "PF_RING loaded an all nodes."
|
||||
exit 0
|
||||
else
|
||||
sleep $interval
|
||||
elapsed=$((elapsed + interval))
|
||||
fi
|
||||
done
|
||||
|
||||
echo "Timeout reached. PF_RING module didn't reach desired status."
|
||||
exit 1
|
||||
|
||||
---
|
||||
apiVersion: batch/v1
|
||||
kind: Job
|
||||
metadata:
|
||||
name: pfring-kmm-status-check
|
||||
annotations:
|
||||
"helm.sh/hook": post-install,post-upgrade
|
||||
"helm.sh/hook-weight": "1"
|
||||
"helm.sh/hook-delete-policy": hook-succeeded
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: verify-pfring-kmm-status
|
||||
image: alpine/k8s:1.27.8
|
||||
command: ["/bin/bash", "/scripts/check_status.sh"]
|
||||
volumeMounts:
|
||||
- name: script-volume
|
||||
mountPath: /scripts
|
||||
volumes:
|
||||
- name: script-volume
|
||||
configMap:
|
||||
name: pfring-kmm-status-check-script
|
||||
restartPolicy: Never
|
||||
{{- end }}
|
||||
{{- end }}
|
||||
@@ -63,7 +63,13 @@ tap:
|
||||
annotations: {}
|
||||
ipv6: true
|
||||
debug: false
|
||||
noKernelModule: false
|
||||
kernelModule:
|
||||
enabled: true
|
||||
mode: auto
|
||||
kernelMappings:
|
||||
- regexp: '^.+$'
|
||||
containerImage: 'kubeshark/pf-ring-module:${KERNEL_FULL_VERSION}'
|
||||
imageRepoSecret: ""
|
||||
telemetry:
|
||||
enabled: true
|
||||
defaultFilter: ""
|
||||
|
||||
Reference in New Issue
Block a user