Add rh-1.0 (#863)

* Bring Go to 1.16

* Bring Go to 1.16

* Bump actions/setup-go to v2

* Bump actions/setup-go to v2

.github/ISSUE_TEMPLATE/bug_report.md

@@ -3,15 +3,35 @@ name: Bug report
 about: Tell us about a problem you are experiencing
 ---
 
-**What steps did you take and what happened:**
+**Overview**
 
-[A clear and concise description of what the bug is, and what commands you ran. If possible please supply logs generated with the `-v 3` parameter.)
+[A clear and concise description of what the bug is] 
+
+**How did you run kube-bench?**
+
+[Please specify exactly how you ran kube-bench, including details of command parameters and/or job file that you used to run it]
+
+**What happened?**
+
+[Please include output from the report to illustrate the problem. If possible please supply logs generated with the `-v 3` parameter.]
 
 **What did you expect to happen:**
 
+[Please describe what you expected to happen differently.]
+
 **Environment** 
 
-[Please specify the version of kube-bench and Kubernetes]
+[What is your version of kube-bench? (run `kube-bench version`)]
+
+[What is your version of Kubernetes? (run `kubectl version` or `oc version` on OpenShift.)]
+
+**Running processes**
+
+[Please include the output from running `ps -eaf | grep kube` on the affected node. This will allow us to check what Kubernetes processes are running, and how this compares to what kube-bench detected.]
+
+**Configuration files**
+
+[If kube-bench is reporting an issue related to the settings defined in a config file, please attach the file, or include an extract showing the settings that are being detected incorrectly.]
 
 **Anything else you would like to add:**
 

.github/workflows/build.yml

@@ -1,3 +1,4 @@
+---
 name: Build
 on:
   push:
@@ -18,9 +19,9 @@ jobs:
     runs-on: ubuntu-18.04
     steps:
       - name: Setup Go
-        uses: actions/setup-go@v1
+        uses: actions/setup-go@v2
         with:
-          go-version: 1.13
+          go-version: 1.16
       - name: Checkout code
         uses: actions/checkout@v2
       - name: yaml-lint

.github/workflows/publish.yml

@@ -1,11 +1,13 @@
 ---
 name: Publish
 on:
+  workflow_dispatch:
   push:
     tags:
       - "v*"
 env:
   ALIAS: aquasecurity
+  DOCKERHUB_ALIAS: aquasec
   REP: kube-bench
 jobs:
   publish:
@@ -54,9 +56,9 @@ jobs:
           builder: ${{ steps.buildx.outputs.name }}
           push: true
           tags: |
-            ${{ secrets.DOCKERHUB_USER }}/${{ env.REP }}:${{ steps.get_version.outputs.version }}
+            ${{ env.DOCKERHUB_ALIAS }}/${{ env.REP }}:${{ steps.get_version.outputs.version }}
             public.ecr.aws/${{ env.ALIAS }}/${{ env.REP }}:${{ steps.get_version.outputs.version }}
-            ${{ secrets.DOCKERHUB_USER }}/${{ env.REP }}:latest
+            ${{ env.DOCKERHUB_ALIAS }}/${{ env.REP }}:latest
             public.ecr.aws/${{ env.ALIAS }}/${{ env.REP }}:latest
           cache-from: type=local,src=/tmp/.buildx-cache/release
           cache-to: type=local,mode=max,dest=/tmp/.buildx-cache/release

.github/workflows/release.yml

@@ -1,3 +1,4 @@
+---
 name: Release
 on:
   push:
@@ -9,9 +10,9 @@ jobs:
     runs-on: ubuntu-18.04
     steps:
       - name: Setup Go
-        uses: actions/setup-go@v1
+        uses: actions/setup-go@v2
         with:
-          go-version: 1.13
+          go-version: 1.16
       - name: Checkout code
         uses: actions/checkout@v2
       - name: Run unit tests

.goreleaser.yml

@@ -24,6 +24,7 @@ archives:
     name_template: '{{ .Binary }}_{{.Version}}_{{ .Os }}_{{ .Arch }}{{ if .Arm }}v{{.Arm }}{{ end }}'
     files:
       - "cfg/**/*"
+      - "cfg/config.yaml"
 nfpms:
   -
     vendor: Aqua Security
@@ -32,6 +33,7 @@ nfpms:
     homepage: https://github.com/aquasecurity/kube-bench
     files:
       "cfg/**/*": "/etc/kube-bench/cfg"
+      "cfg/config.yaml": "/etc/kube-bench/cfg"
     formats:
       - deb
       - rpm

Dockerfile

@@ -1,4 +1,4 @@
-FROM golang:1.15 AS build
+FROM golang:1.16 AS build
 WORKDIR /go/src/github.com/aquasecurity/kube-bench/
 COPY go.mod go.sum ./
 COPY main.go .
@@ -10,7 +10,7 @@ ARG GOOS=linux
 ARG GOARCH=amd64
 RUN GO111MODULE=on CGO_ENABLED=0 GOOS=$GOOS GOARCH=$GOARCH go build -a -ldflags "-X github.com/aquasecurity/kube-bench/cmd.KubeBenchVersion=${KUBEBENCH_VERSION} -w" -o /go/bin/kube-bench
 
-FROM alpine:3.12 AS run
+FROM alpine:3.13 AS run
 WORKDIR /opt/kube-bench/
 # add GNU ps for -C, -o cmd, and --no-headers support
 # https://github.com/aquasecurity/kube-bench/issues/109

README.md

@@ -40,6 +40,8 @@ Table of Contents
   - [CIS Kubernetes Benchmark support](#cis-kubernetes-benchmark-support)
   - [Installation](#installation)
   - [Running kube-bench](#running-kube-bench)
+    - [Specifying the benchmark or Kubernetes version](#specifying-the-benchmark-or-kubernetes-version)
+    - [Specifying Benchmark sections](#specifying-benchmark-sections)
     - [Running inside a container](#running-inside-a-container)
     - [Running in a Kubernetes cluster](#running-in-a-kubernetes-cluster)
     - [Running in an AKS cluster](#running-in-an-aks-cluster)
@@ -76,6 +78,8 @@ kube-bench supports the tests for Kubernetes as defined in the [CIS Kubernetes B
 
 By default, kube-bench will determine the test set to run based on the Kubernetes version running on the machine, but please note that kube-bench does not automatically detect OpenShift and GKE - see the section below on [Running kube-bench](https://github.com/aquasecurity/kube-bench#running-kube-bench).
 
+The test files for the various versions of CIS Benchmark can be found in directories
+with same name as the CIS Benchmark versions under `cfg/`, for example `cfg/cis-1.5`.
 ## Installation
 
 You can choose to
@@ -88,45 +92,48 @@ You can choose to
 
 If you run kube-bench directly from the command line you may need to be root / sudo to have access to all the config files.
 
-kube-bench automatically selects which `controls` to use based on the detected
-node type and the version of Kubernetes a cluster is running. This behavior
-can be overridden by specifying the `master` or `node` subcommand and the
-`--version` flag on the command line.
+By default kube-bench attempts to auto-detect the running version of Kubernetes, and map this to the corresponding CIS Benchmark version. For example, Kubernetes version 1.15 is mapped to CIS Benchmark version `cis-1.15` which is the benchmark version valid for Kubernetes 1.15.
 
+kube-bench also attempts to identify the components running on the node, and uses this to determine which tests to run (for example, only running the master node tests if the node is running an API server). 
 
-The Kubernetes version can also be set with the `KUBE_BENCH_VERSION` environment variable.
-The value of `--version` takes precedence over the value of `KUBE_BENCH_VERSION`.
+### Specifying the benchmark or Kubernetes version
 
-For example, run kube-bench against a master with version auto-detection:
+kube-bench uses the Kubernetes API, or access to the `kubectl` or `kubelet` executables to try to determine the Kubernetes version, and hence which benchmark to run. If you wish to override this, or if none of these methods are available, you can specify either the Kubernetes version or CIS Benchmark as a command line parameter.  
+
+You can specify a particular version of Kubernetes by setting the `--version` flag or with the `KUBE_BENCH_VERSION` environment variable. The value of `--version` takes precedence over the value of `KUBE_BENCH_VERSION`.
+
+For example, run kube-bench using the tests for Kubernetes version 1.13:
 
 ```
-kube-bench master
+kube-bench --version 1.13
 ```
 
-Or run kube-bench against a worker node using the tests for Kubernetes version 1.13:
+
+You can specify `--benchmark` to run a specific CIS Benchmark version:
 
 ```
-kube-bench node --version 1.13
+kube-bench --benchmark cis-1.5
 ```
 
-`kube-bench` will map the `--version` to the corresponding CIS Benchmark version as indicated by the mapping table above. For example, if you specify `--version 1.15`, this is mapped to CIS Benchmark version `cis-1.15`.
+**Note:**  It is an error to specify both `--version` and `--benchmark` flags together
 
-Alternatively, you can specify `--benchmark` to run a specific CIS Benchmark version:
+### Specifying Benchmark sections
 
-```
-kube-bench node --benchmark cis-1.5
-```
-
-If you want to target specific CIS Benchmark `target` (i.e master, node, etcd, etc...)
+If you want to run specific CIS Benchmark sections (i.e master, node, etcd, etc...)
 you can use the `run --targets` subcommand.
+
 ```
-kube-bench --benchmark cis-1.5 run --targets master,node
+kube-bench run --targets master,node
 ```
+
 or
+
 ```
-kube-bench --benchmark cis-1.5 run --targets master,node,etcd,policies
+kube-bench run --targets master,node,etcd,policies
 ```
 
+Check the contents of the benchmark directory under `cfg` to see which targets are available for that benchmark. Each file except `config.yaml` represents a target (also known as a `control` in other parts of this documentation). 
+
 The following table shows the valid targets based on the CIS Benchmark version.
 | CIS Benchmark | Targets |
 |---|---|
@@ -135,40 +142,31 @@ The following table shows the valid targets based on the CIS Benchmark version.
 | gke-1.0| master, controlplane, node, etcd, policies, managedservices |
 | eks-1.0| controlplane, node, policies, managedservices |
 
-If no targets are specified, `kube-bench` will determine the appropriate targets based on the CIS Benchmark version.
-
-`controls` for the various versions of CIS Benchmark can be found in directories
-with same name as the CIS Benchmark versions under `cfg/`, for example `cfg/cis-1.5`.
-
-**Note:**  **`It is an error to specify both --version and --benchmark flags together`**
-
+If no targets are specified, `kube-bench` will determine the appropriate targets based on the CIS Benchmark version and the components detected on the node. The detection is done by verifying which components are running, as defined in the config files (see [Configuration](#configuration).
 ### Running inside a container
 
 You can avoid installing kube-bench on the host by running it inside a container using the host PID namespace and mounting the `/etc` and `/var` directories where the configuration and other files are located on the host so that kube-bench can check their existence and permissions.
 
 ```
-docker run --pid=host -v /etc:/etc:ro -v /var:/var:ro -t aquasec/kube-bench:latest [master|node] --version 1.13
+docker run --pid=host -v /etc:/etc:ro -v /var:/var:ro -t aquasec/kube-bench:latest --version 1.13
 ```
 
 > Note: the tests require either the kubelet or kubectl binary in the path in order to auto-detect the Kubernetes version. You can pass `-v $(which kubectl):/usr/local/mount-from-host/bin/kubectl` to resolve this. You will also need to pass in kubeconfig credentials. For example:
 
 ```
-docker run --pid=host -v /etc:/etc:ro -v /var:/var:ro -v $(which kubectl):/usr/local/mount-from-host/bin/kubectl -v ~/.kube:/.kube -e KUBECONFIG=/.kube/config -t aquasec/kube-bench:latest [master|node]
+docker run --pid=host -v /etc:/etc:ro -v /var:/var:ro -v $(which kubectl):/usr/local/mount-from-host/bin/kubectl -v ~/.kube:/.kube -e KUBECONFIG=/.kube/config -t aquasec/kube-bench:latest 
 ```
 
 You can use your own configs by mounting them over the default ones in `/opt/kube-bench/cfg/`
 
 ```
-docker run --pid=host -v /etc:/etc:ro -v /var:/var:ro -t -v path/to/my-config.yaml:/opt/kube-bench/cfg/config.yam -v $(which kubectl):/usr/local/mount-from-host/bin/kubectl -v ~/.kube:/.kube -e KUBECONFIG=/.kube/config aquasec/kube-bench:latest [master|node]
+docker run --pid=host -v /etc:/etc:ro -v /var:/var:ro -t -v path/to/my-config.yaml:/opt/kube-bench/cfg/config.yam -v $(which kubectl):/usr/local/mount-from-host/bin/kubectl -v ~/.kube:/.kube -e KUBECONFIG=/.kube/config aquasec/kube-bench:latest
 ```
 
 ### Running in a Kubernetes cluster
 
 You can run kube-bench inside a pod, but it will need access to the host's PID namespace in order to check the running processes, as well as access to some directories on the host where config files and other files are stored.
 
-Master nodes are automatically detected by kube-bench and will run master checks when possible.
-The detection is done by verifying that mandatory components for master, as defined in the config files, are running (see [Configuration](#configuration)).
-
 The supplied `job.yaml` file can be applied to run the tests as a job. For example:
 
 ```bash
@@ -191,18 +189,14 @@ kubectl logs kube-bench-j76s9
 ...
 ```
 
-You can still force to run specific master or node checks using respectively `job-master.yaml` and `job-node.yaml`.
-
-To run the tests on the master node, the pod needs to be scheduled on that node. This involves setting a nodeSelector and tolerations in the pod spec.
+To run tests on the master node, the pod needs to be scheduled on that node. This involves setting a nodeSelector and tolerations in the pod spec.
 
 The default labels applied to master nodes has changed since Kubernetes 1.11, so if you are using an older version you may need to modify the nodeSelector and tolerations to run the job on the master node.
-
-
 ### Running in an AKS cluster
 
 1. Create an AKS cluster(e.g. 1.13.7) with RBAC enabled, otherwise there would be 4 failures
 
-1. Use the [kubectl-enter plugin] (https://github.com/kvaps/kubectl-enter) to shell into a node
+1. Use the [kubectl-enter plugin](https://github.com/kvaps/kubectl-enter) to shell into a node
 `
 kubectl-enter {node-name}
 `
@@ -212,7 +206,7 @@ could open nsg 22 port and assign a public ip for one agent node (only for testi
 1. Run CIS benchmark to view results:
 ```
 docker run --rm -v `pwd`:/host aquasec/kube-bench:latest install
-./kube-bench node
+./kube-bench 
 ```
 kube-bench cannot be run on AKS master nodes
 
@@ -282,7 +276,7 @@ This command copies the kube-bench binary and configuration files to your host f
 docker run --rm -v `pwd`:/host aquasec/kube-bench:latest install
 ```
 
-You can then run `./kube-bench [master|node]`.
+You can then run `./kube-bench`.
 
 ### Download and Install binaries
 
@@ -318,12 +312,12 @@ tar -xvf kube-bench_0.3.1_linux_amd64.tar.gz
 
 You can then run kube-bench directly:
 ```
-kube-bench [master|node]
+kube-bench
 ```
 
 If you manually downloaded the kube-bench binary (using curl command above), you have to specify the location of configuration directory and file. For example:
 ```
-./kube-bench --config-dir `pwd`/cfg --config `pwd`/cfg/config.yaml [master|node]
+./kube-bench --config-dir `pwd`/cfg --config `pwd`/cfg/config.yaml 
 ```
 
 See previous section on [Running kube-bench](#running-kube-bench) for further details on using the kube-bench binary.

cfg/cis-1.5/policies.yaml

@@ -131,7 +131,7 @@ groups:
         text: "Minimize the admission of containers with capabilities assigned (Not Scored)"
         type: "manual"
         remediation: |
-          Review the use of capabilites in applications runnning on your cluster. Where a namespace
+          Review the use of capabilites in applications running on your cluster. Where a namespace
           contains applicaions which do not require any Linux capabities to operate consider adding
           a PSP which forbids the admission of containers which do not drop all capabilities.
         scored: false

cfg/cis-1.6/master.yaml

@@ -120,8 +120,16 @@ groups:
 
       - id: 1.1.9
         text: "Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual)"
-        audit: "stat -c permissions=%a <path/to/cni/files>"
-        type: "manual"
+        audit: |
+          ps -ef | grep $kubeletbin | grep -- --cni-conf-dir | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs stat -c permissions=%a
+          find /var/lib/cni/networks -type f | xargs --no-run-if-empty stat -c permissions=%a
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "permissions"
+              compare:
+                op: bitmask
+                value: "644"
         remediation: |
           Run the below command (based on the file location on your system) on the master node.
           For example,
@@ -130,8 +138,13 @@ groups:
 
       - id: 1.1.10
         text: "Ensure that the Container Network Interface file ownership is set to root:root (Manual)"
-        audit: "stat -c %U:%G <path/to/cni/files>"
-        type: "manual"
+        audit: |
+          ps -ef | grep $kubeletbin | grep -- --cni-conf-dir | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs stat -c %U:%G
+          find /var/lib/cni/networks -type f | xargs --no-run-if-empty stat -c %U:%G
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "root:root"
         remediation: |
           Run the below command (based on the file location on your system) on the master node.
           For example,
@@ -819,8 +832,15 @@ groups:
 
       - id: 1.2.34
         text: "Ensure that encryption providers are appropriately configured (Manual)"
-        audit: "/bin/ps -ef | grep $apiserverbin | grep -v grep"
-        type: "manual"
+        audit: |
+          ENCRYPTION_PROVIDER_CONFIG=$(ps -ef | grep $apiserverbin | grep -- --encryption-provider-config | sed 's%.*encryption-provider-config[= ]\([^ ]*\).*%\1%')
+          if test -e $ENCRYPTION_PROVIDER_CONFIG; then grep -A1 'providers:' $ENCRYPTION_PROVIDER_CONFIG | tail -n1 | grep -o "[A-Za-z]*" | sed 's/^/provider=/'; fi
+        tests:
+          test_items:
+            - flag: "provider"
+              compare:
+                op: valid_elements
+                value: "aescbc,kms,secretbox"
         remediation: |
           Follow the Kubernetes documentation and configure a EncryptionConfig file.
           In this file, choose aescbc, kms or secretbox as the encryption provider.

cfg/cis-1.6/policies.yaml

@@ -131,7 +131,7 @@ groups:
         text: "Minimize the admission of containers with capabilities assigned (Manual)"
         type: "manual"
         remediation: |
-          Review the use of capabilites in applications runnning on your cluster. Where a namespace
+          Review the use of capabilites in applications running on your cluster. Where a namespace
           contains applicaions which do not require any Linux capabities to operate consider adding
           a PSP which forbids the admission of containers which do not drop all capabilities.
         scored: false

cfg/config.yaml

@@ -15,6 +15,7 @@ master:
     - flanneld
     # kubernetes is a component to cover the config file /etc/kubernetes/config that is referred to in the benchmark
     - kubernetes
+    - kubelet
 
   kubernetes:
     defaultconf: /etc/kubernetes/config
@@ -25,12 +26,15 @@ master:
       - "hyperkube apiserver"
       - "hyperkube kube-apiserver"
       - "apiserver"
+      - "openshift start master api"
+      - "hypershift openshift-kube-apiserver"
     confs:
       - /etc/kubernetes/manifests/kube-apiserver.yaml
       - /etc/kubernetes/manifests/kube-apiserver.yml
       - /etc/kubernetes/manifests/kube-apiserver.manifest
       - /var/snap/kube-apiserver/current/args
       - /var/snap/microk8s/current/args/kube-apiserver
+      - /etc/origin/master/master-config.yaml
     defaultconf: /etc/kubernetes/manifests/kube-apiserver.yaml
 
   scheduler:
@@ -39,15 +43,19 @@ master:
       - "hyperkube scheduler"
       - "hyperkube kube-scheduler"
       - "scheduler"
+      - "openshift start master controllers"
     confs:
       - /etc/kubernetes/manifests/kube-scheduler.yaml
       - /etc/kubernetes/manifests/kube-scheduler.yml
       - /etc/kubernetes/manifests/kube-scheduler.manifest
       - /var/snap/kube-scheduler/current/args
       - /var/snap/microk8s/current/args/kube-scheduler
+      - /etc/origin/master/scheduler.json
     defaultconf: /etc/kubernetes/manifests/kube-scheduler.yaml
     kubeconfig:
       - /etc/kubernetes/scheduler.conf
+      - /var/lib/kube-scheduler/kubeconfig
+      - /var/lib/kube-scheduler/config.yaml
     defaultkubeconfig: /etc/kubernetes/scheduler.conf
 
   controllermanager:
@@ -57,6 +65,8 @@ master:
       - "hyperkube controller-manager"
       - "hyperkube kube-controller-manager"
       - "controller-manager"
+      - "openshift start master controllers"
+      - "hypershift openshift-controller-manager"
     confs:
       - /etc/kubernetes/manifests/kube-controller-manager.yaml
       - /etc/kubernetes/manifests/kube-controller-manager.yml
@@ -66,12 +76,14 @@ master:
     defaultconf: /etc/kubernetes/manifests/kube-controller-manager.yaml
     kubeconfig:
       - /etc/kubernetes/controller-manager.conf
+      - /var/lib/kube-controller-manager/kubeconfig
     defaultkubeconfig: /etc/kubernetes/controller-manager.conf
 
   etcd:
     optional: true
     bins:
       - "etcd"
+      - "openshift start etcd"
     confs:
       - /etc/kubernetes/manifests/etcd.yaml
       - /etc/kubernetes/manifests/etcd.yml
@@ -81,7 +93,6 @@ master:
       - /var/snap/etcd/common/etcd.conf.yaml
       - /var/snap/microk8s/current/args/etcd
       - /usr/lib/systemd/system/etcd.service
-      - /etc/kubernetes/manifests
     defaultconf: /etc/kubernetes/manifests/etcd.yaml
 
   flanneld:
@@ -90,6 +101,12 @@ master:
       - flanneld
     defaultconf: /etc/sysconfig/flanneld
 
+  kubelet:
+    optional: true
+    bins:
+      - "hyperkube kubelet"
+      - "kubelet"
+
 node:
   components:
     - kubelet
@@ -114,6 +131,8 @@ node:
       - "/lib/systemd/system/kubelet.service"
       - "/etc/systemd/system/snap.kubelet.daemon.service"
       - "/etc/systemd/system/snap.microk8s.daemon-kubelet.service"
+      - "/etc/systemd/system/atomic-openshift-node.service"
+      - "/etc/systemd/system/origin-node.service"
     bins:
       - "hyperkube kubelet"
       - "kubelet"
@@ -154,6 +173,7 @@ node:
       - "hyperkube proxy"
       - "hyperkube kube-proxy"
       - "proxy"
+      - "openshift start network"
     confs:
       - /etc/kubernetes/proxy
       - /etc/kubernetes/addons/kube-proxy-daemonset.yaml
@@ -215,6 +235,7 @@ version_mapping:
   "gke-1.0": "gke-1.0"
   "ocp-3.10": "rh-0.7"
   "ocp-3.11": "rh-0.7"
+  "ocp-4.0": "rh-1.0"
   "aks-1.0": "aks-1.0"
 
 target_mapping:
@@ -252,3 +273,9 @@ target_mapping:
     - "controlplane"
     - "policies"
     - "managedservices"
+  "rh-1.0":
+    - "master"
+    - "node"
+    - "controlplane"
+    - "policies"
+    - "etcd"

cfg/gke-1.0/policies.yaml

@@ -131,7 +131,7 @@ groups:
         text: "Minimize the admission of containers with capabilities assigned (Scored) "
         type: "manual"
         remediation: |
-          Review the use of capabilites in applications runnning on your cluster. Where a namespace
+          Review the use of capabilites in applications running on your cluster. Where a namespace
           contains applications which do not require any Linux capabities to operate consider adding
           a PSP which forbids the admission of containers which do not drop all capabilities.
         scored: true

cfg/rh-0.7/config.yaml

@@ -1,32 +1,2 @@
 ---
 ## Version-specific settings that override the values in cfg/config.yaml
-
-master:
-  apiserver:
-    bins:
-      - openshift start master api
-      - hypershift openshift-kube-apiserver
-
-  scheduler:
-    bins:
-      - "openshift start master controllers"
-      - "hyperkube kube-scheduler"
-    confs:
-      - /etc/origin/master/scheduler.json
-
-  controllermanager:
-    bins:
-      - "openshift start master controllers"
-      - "hypershift openshift-controller-manager"
-
-  etcd:
-    bins:
-      - openshift start etcd
-
-node:
-  svcs:
-    - /etc/systemd/system/atomic-openshift-node.service
-    - /etc/systemd/system/origin-node.service
-  proxy:
-    bins:
-      - openshift start network

cfg/rh-0.7/master.yaml

@@ -16,13 +16,10 @@ groups:
 
       - id: 1.2
         text: "Verify that the basic-auth-file method is not enabled"
-        audit: "grep -A2 basic-auth-file /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "--basic-auth-file"
-              compare:
-                op: eq
-                value: ""
+            - path: "{.kubernetesMasterConfig.apiServerArguments.basic-auth-file}"
               set: false
         remediation: |
           Edit the kubernetes master config file /etc/origin/master/master-config.yaml and
@@ -41,35 +38,27 @@ groups:
 
       - id: 1.4
         text: "Secure communications between the API server and master nodes"
-        audit: "grep -A4 kubeletClientInfo /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           bin_op: and
           test_items:
-            - flag: "kubeletClientInfo:"
-              compare:
-                op: eq
-                value: "kubeletClientInfo:"
-              set: true
-            - flag: "ca"
+            - path: "{.kubeletClientInfo.ca}"
               compare:
                 op: has
                 value: "ca-bundle.crt"
-              set: true
-            - flag: "certFile"
+            - path: "{.kubeletClientInfo.certFile}"
               compare:
                 op: has
                 value: "master.kubelet-client.crt"
-              set: true
-            - flag: "keyFile"
+            - path: "{.kubeletClientInfo.keyFile}"
               compare:
                 op: has
                 value: "master.kubelet-client.key"
               set: true
-            - flag: "port: 10250"
+            - path: "{.kubeletClientInfo.port}"
               compare:
                 op: eq
-                value: "port: 10250"
-              set: true
+                value: "10250"
         remediation: |
           Edit the kubernetes master config file /etc/origin/master/master-config.yaml
           and change it to match the below.
@@ -83,10 +72,10 @@ groups:
 
       - id: 1.5
         text: "Prevent insecure bindings"
-        audit: "grep -A2 insecure-bind-address /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "insecure-bind-address"
+            - path: "{.kubernetesMasterConfig.insecure-bind-address}"
               set: false
         remediation: |
           Edit the kubernetes master config file /etc/origin/master/master-config.yaml
@@ -100,10 +89,10 @@ groups:
 
       - id: 1.6
         text: "Prevent insecure port access"
-        audit: "grep -A2 insecure-port /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "insecure-port"
+            - path: "{.kubernetesMasterConfig.apiServerArguments.insecure-port}"
               set: false
         remediation: |
          Edit the kubernetes master config file /etc/origin/master/master-config.yaml
@@ -117,17 +106,16 @@ groups:
 
       - id: 1.7
         text: "Use Secure Ports for API Server Traffic"
-        audit: "grep -A2 secure-port /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           bin_op: or
           test_items:
-            - flag: "secure-port"
+            - path: "{.kubernetesMasterConfig.apiServerArguments.secure-port}"
               set: false
-            - flag: "secure-port"
+            - path: "{.kubernetesMasterConfig.apiServerArguments.secure-port}"
               compare:
-                op: nothave
+                op: noteq
                 value: "0"
-              set: true
         remediation: |
          Edit the kubernetes master config file /etc/origin/master/master-config.yaml
          and either remove the secure-port parameter or set it to a different (non-zero)
@@ -146,17 +134,16 @@ groups:
 
       - id: 1.9
         text: "Verify repair-malformed-updates argument for API compatibility"
-        audit: "grep -A2 repair-malformed-updates /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           bin_op: or
           test_items:
-            - flag: "repair-malformed-updates"
+            - path: "{.repair-malformed-updates}"
               set: false
-            - flag: "repair-malformed-updates"
+            - path: "{.repair-malformed-updates}"
               compare:
-                op: has
+                op: eq
                 value: "true"
-              set: true
         remediation: |
          Edit the kubernetes master config file /etc/origin/master/master-config.yaml
          and remove the repair-malformed-updates entry or set repair-malformed-updates=true.
@@ -164,10 +151,10 @@ groups:
 
       - id: 1.10
         text: "Verify that the AlwaysAdmit admission controller is disabled"
-        audit: "grep -A4 AlwaysAdmit /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "AlwaysAdmit"
+            - path: "{.AlwaysAdmit}"
               set: false
         remediation: |
           Edit the kubernetes master config file /etc/origin/master/master-config.yaml
@@ -182,14 +169,13 @@ groups:
 
       - id: 1.11
         text: "Manage the AlwaysPullImages admission controller"
-        audit: "grep -A4 AlwaysPullImages /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "disable"
+            - path: "{.admissionConfig.pluginConfig.configuration.disable}"
               compare:
-                op: has
+                op: eq
                 value: "false"
-              set: true
         remediation: |
           Edit the kubernetes master config file /etc/origin/master/master-config.yaml
           and add the entry below.
@@ -215,10 +201,10 @@ groups:
 
       - id: 1.14
         text: "Manage the NamespaceLifecycle admission controller"
-        audit: "grep -A4 NamespaceLifecycle /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "NamespaceLifecycle"
+            - path: "{.NamespaceLifecycle}"
               set: false
         remediation: |
           Edit the kubernetes master config file /etc/origin/master/master-config.yaml
@@ -233,14 +219,13 @@ groups:
 
       - id: 1.15
         text: "Configure API server auditing - audit log file path"
-        audit: "grep -A5 auditConfig /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "enabled"
+            - path: "{.auditConfig.enabled}"
               compare:
-                op: has
+                op: eq
                 value: "true"
-              set: true
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml, update the following entry and restart the API server.
 
@@ -257,14 +242,13 @@ groups:
 
       - id: 1.16
         text: "Configure API server auditing - audit log retention"
-        audit: "grep -A5 auditConfig /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "maximumFileRetentionDays: 30"
+            - path: "{.auditConfig.maximumFileRetentionDays}"
               compare:
-                op: has
-                value: "maximumFileRetentionDays"
-              set: true
+                op: gte
+                value: 30
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml,
           update the maximumFileRetentionDays entry and restart the API server.
@@ -282,14 +266,13 @@ groups:
 
       - id: 1.17
         text: "Configure API server auditing - audit log backup retention"
-        audit: "grep -A5 auditConfig /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "maximumRetainedFiles: 10"
+            - path: "{.auditConfig.maximumRetainedFiles}"
               compare:
-                op: has
-                value: "maximumRetainedFiles"
-              set: true
+                op: eq
+                value: "10"
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml, update the maximumRetainedFiles entry,
           set enabled to true and restart the API server.
@@ -307,14 +290,13 @@ groups:
 
       - id: 1.18
         text: "Configure audit log file size"
-        audit: "grep -A5 auditConfig /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "maximumFileSizeMegabytes: 30"
+            - path: "{.auditConfig.maximumFileSizeMegabytes}"
               compare:
-                op: has
-                value: "maximumFileSizeMegabytes"
-              set: true
+                op: eq
+                value: "30"
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml, update the maximumFileSizeMegabytes entry,
           set enabled to true and restart the API server.
@@ -332,10 +314,10 @@ groups:
 
       - id: 1.19
         text: "Verify that authorization-mode is not set to AlwaysAllow"
-        audit: "grep -A1 authorization-mode /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "authorization-mode"
+            - path: "{.kubernetesMasterConfig.apiServerArguments.authorization-mode}"
               set: false
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml and remove the authorization-mode
@@ -349,10 +331,10 @@ groups:
 
       - id: 1.20
         text: "Verify that the token-auth-file flag is not set"
-        audit: "grep token-auth-file /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "token-auth-file"
+            - path: "{.kubernetesMasterConfig.apiServerArguments.token-auth-file}"
               set: false
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml and remove the token-auth-file
@@ -366,10 +348,10 @@ groups:
 
       - id: 1.21
         text: "Verify the API server certificate authority"
-        audit: "grep -A1 kubelet-certificate-authority /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "kubelet-certificate-authority"
+            - path: "{.kubernetesMasterConfig.apiServerArguments.kubelet-certificat-authority}"
               set: false
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml and remove the following
@@ -383,20 +365,18 @@ groups:
 
       - id: 1.22
         text: "Verify the API server client certificate and client key"
-        audit: "grep -A4 kubeletClientInfo /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           bin_op: and
           test_items:
-            - flag: "keyFile"
+            - path: "{.kubeletClientInfo. keyFile}"
               compare:
-                op: has
+                op: eq
                 value: "master.kubelet-client.key"
-              set: true
-            - flag: "certFile"
+            - path: "{.kubeletClientInfo.certFile}"
               compare:
-                op: has
+                op: eq
                 value: "master.kubelet-client.crt"
-              set: true
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml and add the following
           configuration under kubeletClientInfo
@@ -420,20 +400,18 @@ groups:
 
       - id: 1.25
         text: "Verify that the service account key file argument is not set"
-        audit: "grep -A9 serviceAccountConfig /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           bin_op: and
           test_items:
-            - flag: "privateKeyFile"
+            - path: "{.serviceAccountConfig.privateKeyFile}"
               compare:
-                op: has
+                op: eq
                 value: "serviceaccounts.private.key"
-              set: true
-            - flag: "serviceaccounts.public.key"
+            - path: "{.serviceAccountConfig. publicKeyFiles}"
               compare:
-                op: has
+                op: eq
                 value: "serviceaccounts.public.key"
-              set: true
         remediation: |
           OpenShift API server does not use the service-account-key-file argument.
           Even if value is set in master-config.yaml, it will not be used to verify
@@ -460,20 +438,18 @@ groups:
 
       - id: 1.26
         text: "Verify the certificate and key used for communication with etcd"
-        audit: "grep -A3 etcdClientInfo /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           bin_op: and
           test_items:
-            - flag: "certFile"
+            - path: "{.etcdClientInfo.certFile}"
               compare:
-                op: has
+                op: eq
                 value: "master.etcd-client.crt"
-              set: true
-            - flag: "keyFile"
+            - path: "{.etcdClientInfo.keyFile}"
               compare:
-                op: has
+                op: eq
                 value: "master.etcd-client.key"
-              set: true
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml and set keyFile and certFile
           under etcdClientInfo like below.
@@ -486,17 +462,16 @@ groups:
 
       - id: 1.27
         text: "Verify that the ServiceAccount admission controller is enabled"
-        audit: "grep -A4 ServiceAccount /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           bin_op: or
           test_items:
-            - flag: "ServiceAccount"
+            - path: "{.ServiceAccount.configuration.disable}"
               set: false
-            - flag: "disable"
+            - path: "{.ServiceAccount.configuration.disable}"
               compare:
-                op: has
+                op: eq
                 value: "false"
-              set: true
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml and enable ServiceAccount
           admission control policy.
@@ -510,20 +485,18 @@ groups:
 
       - id: 1.28
         text: "Verify the certificate and key used to encrypt API server traffic"
-        audit: "grep -A7 servingInfo /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           bin_op: and
           test_items:
-            - flag: "certFile"
+            - path: "{.servingInfo.certFile}"
               compare:
-                op: has
+                op: eq
                 value: "master.server.crt"
-              set: true
-            - flag: "keyFile"
+            - path: "{.servingInfo.keyFile}"
               compare:
-                op: has
+                op: eq
                 value: "master.server.key"
-              set: true
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml and set keyFile and certFile under servingInfo.
 
@@ -539,10 +512,10 @@ groups:
 
       - id: 1.29
         text: "Verify that the --client-ca-file argument is not set"
-        audit: "grep client-ca-file /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "clientCA: ca.crt"
+            - path: "{.servingInfo.clientCA}"
               set: false
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml and set clientCA under servingInfo.
@@ -559,14 +532,13 @@ groups:
 
       - id: 1.30
         text: "Verify the CA used for communication with etcd"
-        audit: "grep -A3 etcdClientInfo /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "ca"
+            - path: "{.etcdClientInfo.ca}"
               compare:
-                op: has
+                op: eq
                 value: "master.etcd-ca.crt"
-              set: true
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml and set ca under etcdClientInfo.
 
@@ -583,17 +555,16 @@ groups:
 
       - id: 1.32
         text: "Verify that the NodeRestriction admission controller is enabled"
-        audit: "grep -A4 NodeRestriction /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           bin_op: or
           test_items:
-            - flag: "NodeRestriction"
+            - path: "{.NodeRestriction.configuration.disable}"
               set: false
-            - flag: "disable"
+            - path: "{.NodeRestriction.configuration.disable}"
               compare:
-                op: has
+                op: eq
                 value: "false"
-              set: true
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml and enable NodeRestriction ca under etcdClientInfo.
 
@@ -606,14 +577,10 @@ groups:
 
       - id: 1.33
         text: "Configure encryption of data at rest in etcd datastore"
-        audit: "grep -A1 experimental-encryption-provider-config /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "experimental-encryption-provider-config:"
-              compare:
-                op: has
-                value: "experimental-encryption-provider-config:"
-              set: true
+            - path: "{.kubernetesMasterConfig.apiServerArguments.experimental-encryption-provider-config}"
         remediation: |
           Follow the instructions in the documentation to configure encryption.
           https://docs.openshift.com/container-platform/3.10/admin_guide/encrypting_data.html
@@ -621,14 +588,14 @@ groups:
 
       - id: 1.34
         text: "Set the encryption provider to aescbc for etcd data at rest"
-        audit: "grep -A1 experimental-encryption-provider-config /etc/origin/master/master-config.yaml | sed -n '2p' | awk '{ print $2 }' | xargs grep -A1 providers"
+
+        audit_config: "grep -A1 experimental-encryption-provider-config /etc/origin/master/master-config.yaml | sed -n '2p' | awk '{ print $2 }' | xargs cat"
         tests:
           test_items:
-            - flag: "aescbc:"
+            - path: "{.providers.aescbc.experimental-encryption-provider-config}"
               compare:
                 op: has
-                value: "aescbc:"
-              set: true
+                value: "aescbc"
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml and set aescbc as the first provider in encryption provider config.
           See https://docs.openshift.com/container-platform/3.10/admin_guide/encrypting_data.html.
@@ -636,14 +603,13 @@ groups:
 
       - id: 1.35
         text: "Enable the EventRateLimit plugin"
-        audit: "grep -A4 EventRateLimit /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "disable"
+            - path: "{.admissionConfig.pluginConfig.AlwaysPullImages.configuration.disable}"
               compare:
-                op: has
+                op: eq
                 value: "false"
-              set: true
         remediation: |
           Follow the documentation to enable the EventRateLimit plugin.
           https://docs.openshift.com/container-platform/3.10/architecture/additional_concepts/admission_controllers.html#admission-controllers-general-admission-rules
@@ -651,16 +617,15 @@ groups:
 
       - id: 1.36
         text: "Configure advanced auditing"
-        audit: "grep AdvancedAuditing /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           bin_op: or
           test_items:
-            - flag: "AdvancedAuditing"
+            - path: "{.kubernetesMasterConfig.apiServerArguments.feature-gates.AdvancedAuditing}"
               compare:
                 op: eq
                 value: "true"
-              set: true
-            - flag: "AdvancedAuditing"
+            - path: "{.kubernetesMasterConfig.apiServerArguments.feature-gates.AdvancedAuditing}"
               set: false
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml and enable AdvancedAuditing,
@@ -696,14 +661,13 @@ groups:
     checks:
       - id: 3.1
         text: "Adjust the terminated-pod-gc-threshold argument as needed"
-        audit: "grep terminated-pod-gc-threshold -A1 /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "terminated-pod-gc-threshold:"
+            - path: "{.kubernetesMasterConfig.controllerArguments.terminated-pod-gc-threshold}"
               compare:
-                op: has
+                op: eq
                 value: "12500"
-              set: true
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml  and enable terminated-pod-gc-threshold.
 
@@ -722,17 +686,16 @@ groups:
 
       - id: 3.3
         text: "Verify that the --use-service-account-credentials argument is set to true"
-        audit: "grep -A2 use-service-account-credentials /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           bin_op: or
           test_items:
-            - flag: "use-service-account-credentials"
+            - path: "{.kubernetesMasterConfig.controllerArguments.use-service-account-credentials}"
               set: false
-            - flag: "true"
+            - path: "{.kubernetesMasterConfig.controllerArguments.use-service-account-credentials}"
               compare:
-                op: has
+                op: eq
                 value: "true"
-              set: true
         remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml and set use-service-account-credentials
           to true under controllerArguments section.
@@ -743,45 +706,45 @@ groups:
                  - true
         scored: true
 
-      # Review 3.4
+
       - id: 3.4
         text: "Verify that the --service-account-private-key-file argument is set as appropriate"
         audit: |
           grep -A9 serviceAccountConfig /etc/origin/master/master-config.yaml | grep privateKeyFile;
           grep -A2 service-account-private-key-file /etc/origin/master/master-config.yaml
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           bin_op: and
           test_items:
             - flag: "privateKeyFile: serviceaccounts.private.key"
+              path: "{.serviceAccountConfig.privateKeyFile}"
               compare:
-                op: has
-                value: "privateKeyFile"
+                op: eq
+                value: "serviceaccounts.private.key"
             - flag: "service-account-private-key-file"
               set: false
-        remediation:
+        remediation: |
           Edit the Openshift master config file /etc/origin/master/master-config.yaml and remove service-account-private-key-file
         scored: true
 
-      # Review 3.5
+
       - id: 3.5
         text: "Verify that the --root-ca-file argument is set as appropriate"
-        audit: "/bin/sh -c 'grep root-ca-file /etc/origin/master/master-config.yaml; grep -A9 serviceAccountConfig /etc/origin/master/master-config.yaml'"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           bin_op: and
           test_items:
-            - flag: "root-ca-file=/etc/origin/master/ca-bundle.crt"
+            - path: "{.serviceAccountConfig.root-ca-file}"
               compare:
-                op: has
+                op: eq
                 value: "/etc/origin/master/ca-bundle.crt"
-              set: true
-            - flag: "masterCA"
+            - path: "{.serviceAccountConfig.masterCA}"
               compare:
-                op: has
+                op: eq
                 value: "ca-bundle.crt"
-              set: true
-        remediation:
+        remediation: |
           Reset to OpenShift defaults OpenShift starts kube-controller-manager with
-          root-ca-file=/etc/origin/master/ca-bundle.crt by default.  OpenShift Advanced
+          root-ca-file=/etc/origin/master/ca-bundle.crt by default. OpenShift Advanced
           Installation creates this certificate authority and configuration without any
           configuration required.
 
@@ -795,15 +758,14 @@ groups:
 
       - id: 3.7
         text: "Manage certificate rotation"
-        audit: "grep -B3 RotateKubeletServerCertificate=true /etc/origin/master/master-config.yaml"
+        audit_config: "cat /etc/origin/master/master-config.yaml"
         tests:
           test_items:
-            - flag: "RotateKubeletServerCertificate"
+            - path: "{.kubeletArguments.feature-gates.RotateKubeletServerCertificate}"
               compare:
                 op: eq
                 value: "true"
-              set: true
-        remediation:
+        remediation: |
           If you decide not to enable the RotateKubeletServerCertificate feature,
           be sure to use the Ansible playbooks provided with the OpenShift installer to
           automate re-deploying certificates.
@@ -822,7 +784,6 @@ groups:
               compare:
                 op: bitmask
                 value: "600"
-              set: true
         remediation: |
           Run the below command.
 
@@ -835,10 +796,6 @@ groups:
         tests:
           test_items:
             - flag: "root:root"
-              compare:
-                op: eq
-                value: "root:root"
-              set: true
         remediation: |
           Run the below command on the master node.
 
@@ -854,7 +811,6 @@ groups:
               compare:
                 op: bitmask
                 value: "600"
-              set: true
         remediation: |
           Run the below command on the master node.
 
@@ -867,10 +823,6 @@ groups:
         tests:
           test_items:
             - flag: "root:root"
-              compare:
-                op: eq
-                value: "root:root"
-              set: true
         remediation: |
           Run the below command on the master node.
 
@@ -886,7 +838,6 @@ groups:
               compare:
                 op: bitmask
                 value: "600"
-              set: true
         remediation: |
           Run the below command.
 
@@ -899,10 +850,6 @@ groups:
         tests:
           test_items:
             - flag: "root:root"
-              compare:
-                op: eq
-                value: "root:root"
-              set: true
         remediation: |
           Run the below command on the master node.
 
@@ -918,7 +865,6 @@ groups:
               compare:
                 op: bitmask
                 value: "600"
-              set: true
         remediation: |
           Run the below command.
 
@@ -931,10 +877,6 @@ groups:
         tests:
           test_items:
             - flag: "root:root"
-              compare:
-                op: eq
-                value: "root:root"
-              set: true
         remediation: |
           Run the below command on the master node.
 
@@ -944,13 +886,13 @@ groups:
       - id: 4.9
         text: "Verify the default OpenShift Container Network Interface file permissions"
         audit: "stat -c permissions=%a /etc/origin/openvswitch/ /etc/cni/net.d/"
+        use_multiple_values: true
         tests:
           test_items:
             - flag: "permissions"
               compare:
                 op: bitmask
                 value: "644"
-              set: true
         remediation: |
           Run the below command.
 
@@ -960,13 +902,10 @@ groups:
       - id: 4.10
         text: "Verify the default OpenShift Container Network Interface file ownership"
         audit: "stat -c %U:%G /etc/origin/openvswitch/ /etc/cni/net.d/"
+        use_multiple_values: true
         tests:
           test_items:
             - flag: "root:root"
-              compare:
-                op: eq
-                value: "root:root"
-              set: true
         remediation: |
           Run the below command on the master node.
 
@@ -982,7 +921,6 @@ groups:
               compare:
                 op: bitmask
                 value: "700"
-              set: true
         remediation: |
           On the etcd server node, get the etcd data directory, passed as an argument --data-dir ,
           from the below command:
@@ -997,10 +935,6 @@ groups:
         tests:
           test_items:
             - flag: "etcd:etcd"
-              compare:
-                op: eq
-                value: "etcd:etcd"
-              set: true
         remediation: |
           Run the below command on the master node.
 
@@ -1016,7 +950,6 @@ groups:
               compare:
                 op: bitmask
                 value: "644"
-              set: true
         remediation: |
           Run the below command.
 
@@ -1029,10 +962,6 @@ groups:
         tests:
           test_items:
             - flag: "root:root"
-              compare:
-                op: eq
-                value: "root:root"
-              set: true
         remediation: |
           Run the below command on the master node.
 
@@ -1048,7 +977,6 @@ groups:
               compare:
                 op: bitmask
                 value: "644"
-              set: true
         remediation: |
           Run the below command.
 
@@ -1061,10 +989,6 @@ groups:
         tests:
           test_items:
             - flag: "root:root"
-              compare:
-                op: eq
-                value: "root:root"
-              set: true
         remediation: |
           Run the below command on the master node.
 
@@ -1080,7 +1004,6 @@ groups:
               compare:
                 op: bitmask
                 value: "644"
-              set: true
         remediation: |
           Run the below command.
 
@@ -1114,40 +1037,29 @@ groups:
           bin_op: and
           test_items:
             - flag: "Binary file /proc/1/environ matches"
+            - flag: "ETCD_CERT_FILE"
               compare:
-                op: has
-                value: "Binary file /proc/1/environ matches"
-              set: true
-            - flag: "ETCD_CERT_FILE=/etc/etcd/server.crt"
+                op: eq
+                value: "/etc/etcd/server.crt"
+            - flag: "ETCD_KEY_FILE"
               compare:
-                op: has
-                value: "ETCD_CERT_FILE=/etc/etcd/server.crt"
-              set: true
-            - flag: "ETCD_KEY_FILE=/etc/etcd/server.key"
-              compare:
-                op: has
-                value: "ETCD_KEY_FILE=/etc/etcd/server.key"
-              set: true
+                op: eq
+                value: "/etc/etcd/server.key"
         remediation: |
           Reset to the OpenShift default configuration.
         scored: true
 
       - id: 5.2
         text: "Verify the default OpenShift setting for the client-cert-auth argument"
-        audit: "/bin/sh -c'/usr/local/bin/master-exec etcd etcd grep ETCD_CLIENT_CERT_AUTH=true /proc/1/environ; grep ETCD_CLIENT_CERT_AUTH /etc/etcd/etcd.conf'"
+        audit: "/bin/sh -c '/usr/local/bin/master-exec etcd etcd grep ETCD_CLIENT_CERT_AUTH=true /proc/1/environ; grep ETCD_CLIENT_CERT_AUTH /etc/etcd/etcd.conf'"
         tests:
           bin_op: and
           test_items:
             - flag: "Binary file /proc/1/environ matches"
+            - flag: "ETCD_CLIENT_CERT_AUTH"
               compare:
-                op: has
-                value: "Binary file /proc/1/environ matches"
-              set: true
-            - flag: "ETCD_CLIENT_CERT_AUTH=true"
-              compare:
-                op: has
-                value: "ETCD_CLIENT_CERT_AUTH=true"
-              set: true
+                op: eq
+                value: "true"
         remediation: |
           Reset to the OpenShift default configuration.
         scored: true
@@ -1158,41 +1070,29 @@ groups:
         tests:
           bin_op: or
           test_items:
-            - flag: "ETCD_AUTO_TLS=false"
+            - flag: "ETCD_AUTO_TLS"
               compare:
-                op: has
-                value: "ETCD_AUTO_TLS=false"
-              set: true
-            - flag: "#ETCD_AUTO_TLS"
-              compare:
-                op: has
-                value: "#ETCD_AUTO_TLS"
-              set: true
+                op: eq
+                value: "false"
         remediation: |
           Reset to the OpenShift default configuration.
         scored: true
 
       - id: 5.4
         text: "Verify the OpenShift default peer-cert-file and peer-key-file arguments for etcd"
-        audit: "/bin/sh -c'/usr/local/bin/master-exec etcd etcd grep ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt /proc/1/environ; /usr/local/bin/master-exec etcd etcd grep ETCD_PEER_KEY_FILE=/etc/etcd/peer.key /proc/1/environ; grep ETCD_PEER_CERT_FILE /etc/etcd/etcd.conf; grep ETCD_PEER_KEY_FILE /etc/etcd/etcd.conf'"
+        audit: "/bin/sh -c '/usr/local/bin/master-exec etcd etcd grep ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt /proc/1/environ; /usr/local/bin/master-exec etcd etcd grep ETCD_PEER_KEY_FILE=/etc/etcd/peer.key /proc/1/environ; grep ETCD_PEER_CERT_FILE /etc/etcd/etcd.conf; grep ETCD_PEER_KEY_FILE /etc/etcd/etcd.conf'"
         tests:
           bin_op: and
           test_items:
             - flag: "Binary file /proc/1/environ matches"
+            - flag: "ETCD_PEER_CERT_FILE"
               compare:
-                op: has
-                value: "Binary file /proc/1/environ matches"
-              set: true
-            - flag: "ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt"
+                op: eq
+                value: "/etc/etcd/peer.crt"
+            - flag: "ETCD_PEER_KEY_FILE"
               compare:
-                op: has
-                value: "ETCD_PEER_CERT_FILE=/etc/etcd/peer.crt"
-              set: true
-            - flag: "ETCD_PEER_KEY_FILE=/etc/etcd/peer.key"
-              compare:
-                op: has
-                value: "ETCD_PEER_KEY_FILE=/etc/etcd/peer.key"
-              set: true
+                op: eq
+                value: "/etc/etcd/peer.key"
         remediation: |
           Reset to the OpenShift default configuration.
         scored: true
@@ -1204,15 +1104,10 @@ groups:
           bin_op: and
           test_items:
             - flag: "Binary file /proc/1/environ matches"
+            - flag: "ETCD_PEER_CLIENT_CERT_AUTH"
               compare:
-                op: has
-                value: "Binary file /proc/1/environ matches"
-              set: true
-            - flag: "ETCD_PEER_CLIENT_CERT_AUTH=true"
-              compare:
-                op: has
-                value: "ETCD_PEER_CLIENT_CERT_AUTH=true"
-              set: true
+                op: eq
+                value: "true"
         remediation: |
           Reset to the OpenShift default configuration.
         scored: true
@@ -1221,18 +1116,11 @@ groups:
         text: "Verify the OpenShift default configuration for the peer-auto-tls argument"
         audit: "/bin/sh -c '/usr/local/bin/master-exec etcd etcd grep ETCD_PEER_AUTO_TLS /proc/1/environ; grep ETCD_PEER_AUTO_TLS /etc/etcd/etcd.conf'"
         tests:
-          bin_op: and
           test_items:
-            - flag: "Binary file /proc/1/environ matches"
+            - flag: "ETCD_PEER_AUTO_TLS"
               compare:
-                op: has
-                value: "Binary file /proc/1/environ matches"
-              set: true
-            - flag: "#ETCD_PEER_AUTO_TLS=false"
-              compare:
-                op: has
-                value: "#ETCD_PEER_AUTO_TLS=false"
-              set: true
+                op: noteq
+                value: "true"
         remediation: |
           Reset to the OpenShift default configuration.
         scored: true
@@ -1253,10 +1141,6 @@ groups:
         tests:
           test_items:
             - flag: "issuer= /CN=etcd-signer"
-              compare:
-                op: has
-                value: "issuer= /CN=etcd-signer"
-              set: true
         remediation: |
           Reset to the OpenShift default configuration.
         scored: false

cfg/rh-0.7/node.yaml

@@ -19,17 +19,16 @@ groups:
 
       - id: 7.3
         text: "Verify that the --authorization-mode argument is set to WebHook"
-        audit: "grep -A1 authorization-mode /etc/origin/node/node-config.yaml"
+        audit_config: "cat /etc/origin/node/node-config.yaml"
         tests:
           bin_op: or
           test_items:
-            - flag: "authorization-mode"
+            - path: "{.kubeletArguments.authorization-mode}"
               set: false
-            - flag: "authorization-mode"
+            - path: "{.kubeletArguments.authorization-mode}"
               compare:
                 op: has
                 value: "Webhook"
-              set: true
         remediation: |
           Edit the Openshift node config file /etc/origin/node/node-config.yaml and remove authorization-mode under
           kubeletArguments in /etc/origin/node/node-config.yaml or set it to "Webhook".
@@ -37,10 +36,10 @@ groups:
 
       - id: 7.4
         text: "Verify the OpenShift default for the client-ca-file argument"
-        audit: "grep -A1 client-ca-file /etc/origin/node/node-config.yaml"
+        audit_config: "cat /etc/origin/node/node-config.yaml"
         tests:
           test_items:
-            - flag: "client-ca-file"
+            - path: "{.PodManifestConfig.client-ca-file}"
               set: false
         remediation: |
           Edit the Openshift node config file /etc/origin/node/node-config.yaml and remove any configuration returned by the following:
@@ -53,31 +52,32 @@ groups:
 
       - id: 7.5
         text: "Verify the OpenShift default setting for the read-only-port argument"
-        audit: "grep -A1 read-only-port /etc/origin/node/node-config.yaml"
+        audit_config: "cat /etc/origin/node/node-config.yaml"
         tests:
           bin_op: or
           test_items:
-            - flag: "read-only-port"
+            - path: "{.kubeletArguments.read-only-port}"
               set: false
-            - flag: "read-only-port"
+            - path: "{.kubeletArguments.read-only-port}"
               compare:
-                op: has
+                op: eq
                 value: "0"
-              set: true
         remediation: |
           Edit the Openshift node config file /etc/origin/node/node-config.yaml and removed so that the OpenShift default is applied.
         scored: true
 
       - id: 7.6
         text: "Adjust the streaming-connection-idle-timeout argument"
-        audit: "grep -A1 streaming-connection-idle-timeout /etc/origin/node/node-config.yaml"
+        audit_config: "cat /etc/origin/node/node-config.yaml"
         tests:
           bin_op: or
           test_items:
-            - flag: "streaming-connection-idle-timeout"
-              set: false
-            - flag: "5m"
+            - path: "{.kubeletArguments.streaming-connection-idle-timeout}"
               set: false
+            - path: "{.kubeletArguments.streaming-connection-idle-timeout}"
+              compare:
+                op: eq
+                value: "5m"
         remediation: |
           Edit the Openshift node config file /etc/origin/node/node-config.yaml and set the streaming-connection-timeout
           value like the following in node-config.yaml.
@@ -94,17 +94,16 @@ groups:
 
       - id: 7.8
         text: "Verify the OpenShift default value of true for the make-iptables-util-chains argument"
-        audit: "grep -A1 make-iptables-util-chains /etc/origin/node/node-config.yaml"
+        audit_config: "cat /etc/origin/node/node-config.yaml"
         tests:
           bin_op: or
           test_items:
-            - flag: "make-iptables-util-chains"
+            - path: "{.kubeletArguments.make-iptables-util-chains}"
               set: false
-            - flag: "make-iptables-util-chains"
+            - path: "{.kubeletArguments.make-iptables-util-chains}"
               compare:
-                op: has
+                op: eq
                 value: "true"
-              set: true
         remediation: |
           Edit the Openshift node config file /etc/origin/node/node-config.yaml and reset make-iptables-util-chains to the OpenShift
           default value of true.
@@ -112,14 +111,16 @@ groups:
 
       - id: 7.9
         text: "Verify that the --keep-terminated-pod-volumes argument is set to false"
-        audit: "grep -A1 keep-terminated-pod-volumes /etc/origin/node/node-config.yaml"
+        audit_config: "cat /etc/origin/node/node-config.yaml"
         tests:
+          bin_op: or
           test_items:
-            - flag: "keep-terminated-pod-volumes"
+            - path: "{.kubeletArguments.keep-terminated-pod-volumes}"
+              set: false
+            - path: "{.kubeletArguments.keep-terminated-pod-volumes}"
               compare:
-                op: has
+                op: eq
                 value: "false"
-              set: true
         remediation: |
           Reset to the OpenShift defaults
         scored: true
@@ -131,17 +132,16 @@ groups:
 
       - id: 7.11
         text: "Set the --event-qps argument to 0"
-        audit: "grep -A1 event-qps /etc/origin/node/node-config.yaml"
+        audit_config: "cat /etc/origin/node/node-config.yaml"
         tests:
           bin_op: or
           test_items:
-            - flag: "event-qps"
+            - path: "{.kubeletArguments.event-qps}"
               set: false
-            - flag: "event-qps"
+            - path: "{.kubeletArguments.event-qps}"
               compare:
-                op: has
+                op: eq
                 value: "0"
-              set: true
         remediation: |
           Edit the Openshift node config file /etc/origin/node/node-config.yaml set the event-qps argument to 0 in
           the kubeletArguments section of.
@@ -149,31 +149,29 @@ groups:
 
       - id: 7.12
         text: "Verify the OpenShift cert-dir flag for HTTPS traffic"
-        audit: "grep -A1 cert-dir /etc/origin/node/node-config.yaml"
+        audit_config: "cat /etc/origin/node/node-config.yaml"
         tests:
           test_items:
-            - flag: "/etc/origin/node/certificates"
+            - path: "{.kubeletArguments.cert-dir}"
               compare:
                 op: has
                 value: "/etc/origin/node/certificates"
-              set: true
         remediation: |
           Reset to the OpenShift default values.
         scored: true
 
       - id: 7.13
         text: "Verify the OpenShift default of 0 for the cadvisor-port argument"
-        audit: "grep -A1 cadvisor-port /etc/origin/node/node-config.yaml"
+        audit_config: "cat /etc/origin/node/node-config.yaml"
         tests:
           bin_op: or
           test_items:
-            - flag: "cadvisor-port"
+            - path: "{.kubeletArguments.cadvisor-port}"
               set: false
-            - flag: "cadvisor-port"
+            - path: "{.kubeletArguments.cadvisor-port}"
               compare:
-                op: has
+                op: eq
                 value: "0"
-              set: true
         remediation: |
           Edit the Openshift node config file /etc/origin/node/node-config.yaml and remove the cadvisor-port flag
           if it is set in the  kubeletArguments section.
@@ -181,28 +179,26 @@ groups:
 
       - id: 7.14
         text: "Verify that the RotateKubeletClientCertificate argument is set to true"
-        audit: "grep -B1 RotateKubeletClientCertificate=true /etc/origin/node/node-config.yaml"
+        audit_config: "cat /etc/origin/node/node-config.yaml"
         tests:
           test_items:
-            - flag: "RotateKubeletClientCertificate=true"
+            - path: "{.kubeletArguments.feature-gates}"
               compare:
                 op: has
-                value: "true"
-              set: true
+                value: "RotateKubeletClientCertificate=true"
         remediation: |
           Edit the Openshift node config file /etc/origin/node/node-config.yaml and set RotateKubeletClientCertificate to true.
         scored: true
 
       - id: 7.15
         text: "Verify that the RotateKubeletServerCertificate argument is set to true"
-        audit: "grep -B1 RotateKubeletServerCertificate=true /etc/origin/node/node-config.yaml"
+        audit_config: "cat /etc/origin/node/node-config.yaml"
         tests:
           test_items:
-            - flag: "RotateKubeletServerCertificate=true"
+            - path: "{.kubeletArguments.feature-gates}"
               compare:
                 op: has
-                value: "true"
-              set: true
+                value: "RotateKubeletServerCertificate=true"
         remediation: |
           Edit the Openshift node config file /etc/origin/node/node-config.yaml and set RotateKubeletServerCertificate to true.
         scored: true
@@ -220,7 +216,6 @@ groups:
               compare:
                 op: bitmask
                 value: "644"
-              set: true
         remediation: |
           Run the below command on each worker node.
           chmod 644 /etc/origin/node/node.kubeconfig
@@ -232,10 +227,6 @@ groups:
         tests:
           test_items:
             - flag: "root:root"
-              compare:
-                op: eq
-                value: root:root
-              set: true
           remediation: |
             Run the below command on each worker node.
             chown root:root /etc/origin/node/node.kubeconfig
@@ -243,32 +234,27 @@ groups:
 
       - id: 8.3
         text: "Verify the kubelet service file permissions of 644"
-        audit: "stat -c permissions=%a $nodesvc"
+        audit: "stat -c permissions=%a $kubeletsvc"
         tests:
           test_items:
             - flag: "permissions"
               compare:
                 op: bitmask
                 value: "644"
-              set: true
         remediation: |
           Run the below command on each worker node.
-          chmod 644 $nodesvc
+          chmod 644 $kubeletsvc
         scored: true
 
       - id: 8.4
         text: "Verify the kubelet service file ownership of root:root"
-        audit: "stat -c %U:%G $nodesvc"
+        audit: "stat -c %U:%G $kubeletsvc"
         tests:
           test_items:
             - flag: "root:root"
-              compare:
-                op: eq
-                value: root:root
-              set: true
           remediation: |
             Run the below command on each worker node.
-            chown root:root $nodesvc
+            chown root:root $kubeletsvc
           scored: true
 
       - id: 8.5
@@ -280,7 +266,6 @@ groups:
               compare:
                 op: bitmask
                 value: "644"
-              set: true
         remediation: |
           Run the below command on each worker node.
           chmod 644 /etc/origin/node/node.kubeconfig
@@ -292,10 +277,6 @@ groups:
         tests:
           test_items:
             - flag: "root:root"
-              compare:
-                op: eq
-                value: root:root
-              set: true
           remediation: |
             Run the below command on each worker node.
             chown root:root /etc/origin/node/node.kubeconfig
@@ -310,7 +291,6 @@ groups:
               compare:
                 op: bitmask
                 value: "644"
-              set: true
         remediation: |
           Run the below command on each worker node.
           chmod 644 /etc/origin/node/client-ca.crt
@@ -322,10 +302,6 @@ groups:
         tests:
           test_items:
             - flag: "root:root"
-              compare:
-                op: eq
-                value: root:root
-              set: true
           remediation: |
             Run the below command on each worker node.
             chown root:root /etc/origin/node/client-ca.crt

cfg/rh-1.0/config.yaml

@@ -0,0 +1,2 @@
+---
+## Version-specific settings that override the values in cfg/config.yaml

cfg/rh-1.0/controlplane.yaml

@@ -0,0 +1,62 @@
+---
+controls:
+version: rh-1.0
+id: 3
+text: "Control Plane Configuration"
+type: "controlplane"
+groups:
+  - id: 3.1
+    text: "Authentication and Authorization"
+    checks:
+      - id: 3.1.1
+        text: "Client certificate authentication should not be used for users (Manual)"
+        audit: |
+          # To verify user authentication is enabled
+          oc describe authentication
+          # To verify that an identity provider is configured
+          oc get identity
+          # To verify that a custom cluster-admin user exists
+          oc get clusterrolebindings -o=custom-columns=NAME:.metadata.name,ROLE:.roleRef.name,SUBJECT:.subjects[*].kind | grep cluster-admin | grep User
+          # To verity that kbueadmin is removed, no results should be returned
+          oc get secrets kubeadmin -n kube-system
+        type: manual
+        remediation: |
+          Configure an identity provider for the OpenShift cluster.
+          Understanding identity provider configuration | Authentication | OpenShift
+          Container Platform 4.5. Once an identity provider has been defined,
+          you can use RBAC to define and apply permissions.
+          After you define an identity provider and create a new cluster-admin user,
+          remove the kubeadmin user to improve cluster security.
+        scored: false
+
+  - id: 3.2
+    text: "Logging"
+    checks:
+      - id: 3.2.1
+        text: "Ensure that a minimal audit policy is created (Manual)"
+        audit: |
+          #To view kube apiserver log files
+          oc adm node-logs --role=master --path=kube-apiserver/
+          #To view openshift apiserver log files
+          oc adm node-logs --role=master --path=openshift-apiserver/
+          #To verify kube apiserver audit config
+          oc get configmap config -n openshift-kube-apiserver -ojson | jq -r '.data["config.yaml"]' | jq '.auditConfig[]'
+          #To verify openshift apiserver audit config
+          oc get configmap config -n openshift-apiserver -ojson | jq -r '.data["config.yaml"]' | jq '.auditConfig[]'
+        type: manual
+        remediation: |
+          No remediation required.
+        scored: false
+
+      - id: 3.2.2
+        text: "Ensure that the audit policy covers key security concerns (Manual)"
+        audit: |
+          #To verify openshift apiserver audit config
+          oc get configmap config -n openshift-kube-apiserver -ojson | jq -r '.data["config.yaml"]' | jq '.auditConfig.policyConfiguration.rules[]'
+          #To verify kube apiserver audit config
+          oc get configmap config -n openshift-apiserver -ojson | jq -r '.data["config.yaml"]' | jq '.auditConfig.policyConfiguration.rules[]'
+        type: manual
+        remediation: |
+          In OpenShift 4.6 and higher, if appropriate for your needs,
+          modify the audit policy.
+        scored: false

cfg/rh-1.0/etcd.yaml

@@ -0,0 +1,154 @@
+---
+controls:
+version: rh-1.0
+id: 2
+text: "Etcd Node Configuration"
+type: "etcd"
+groups:
+  - id: 2
+    text: "Etcd Node Configuration Files"
+    checks:
+      - id: 2.1
+        text: "Ensure that the --cert-file and --key-file arguments are set as appropriate (Manual)"
+        audit: |
+          # For --cert-file
+          for i in $(oc get pods -oname -n openshift-etcd)
+          do
+            oc exec -n openshift-etcd -c etcd $i -- ps -o command= -C etcd | sed 's/.*\(--cert-file=[^ ]*\).*/\1/'
+          done 2>/dev/null
+          # For --key-file
+          for i in $(oc get pods -oname -n openshift-etcd)
+          do
+            oc exec -n openshift-etcd -c etcd $i -- ps -o command= -C etcd | sed 's/.*\(--key-file=[^ ]*\).*/\1/'
+          done 2>/dev/null
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "file"
+              compare:
+                op: regex
+                value: '\/etc\/kubernetes\/static-pod-certs\/secrets\/etcd-all-serving\/etcd-serving-.*\.(?:crt|key)'
+        remediation: |
+          OpenShift does not use the etcd-certfile or etcd-keyfile flags.
+          Certificates for etcd are managed by the etcd cluster operator.
+        scored: false
+
+      - id: 2.2
+        text: "Ensure that the --client-cert-auth argument is set to true (Manual)"
+        audit: |
+          for i in $(oc get pods -oname -n openshift-etcd)
+          do
+            oc exec -n openshift-etcd -c etcd $i -- ps -o command= -C etcd | sed 's/.*\(--client-cert-auth=[^ ]*\).*/\1/'
+          done 2>/dev/null
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "--client-cert-auth"
+              compare:
+                op: eq
+                value: true
+        remediation: |
+          This setting is managed by the cluster etcd operator. No remediation required."
+        scored: false
+
+      - id: 2.3
+        text: "Ensure that the --auto-tls argument is not set to true (Manual)"
+        audit: |
+          # Returns 0 if found, 1 if not found
+          for i in $(oc get pods -oname -n openshift-etcd)
+          do
+            oc exec -n openshift-etcd -c etcd $i -- ps -o command= -C etcd | grep -- --auto-tls=true 2>&1>/dev/null ; echo exit_code=$?
+          done 2>/dev/null
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "exit_code"
+              compare:
+                op: eq
+                value: "1"
+        remediation: |
+          This setting is managed by the cluster etcd operator. No remediation required.e
+        scored: false
+
+      - id: 2.4
+        text: "Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Manual)"
+        audit: |
+          # For --peer-cert-file
+          for i in $(oc get pods -oname -n openshift-etcd)
+          do
+            oc exec -n openshift-etcd -c etcd $i -- ps -o command= -C etcd | sed 's/.*\(--peer-cert-file=[^ ]*\).*/\1/'
+          done 2>/dev/null
+          # For --peer-key-file
+          for i in $(oc get pods -oname -n openshift-etcd)
+          do
+            oc exec -n openshift-etcd -c etcd $i -- ps -o command= -C etcd | sed 's/.*\(--peer-key-file=[^ ]*\).*/\1/'
+          done 2>/dev/null
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "file"
+              compare:
+                op: regex
+                value: '\/etc\/kubernetes\/static-pod-certs\/secrets\/etcd-all-peer\/etcd-peer-.*\.(?:crt|key)'
+        remediation: |
+          None. This configuration is managed by the etcd operator.
+        scored: false
+
+      - id: 2.5
+        text: "Ensure that the --peer-client-cert-auth argument is set to true (Manual)"
+        audit: |
+          for i in $(oc get pods -oname -n openshift-etcd)
+          do
+            oc exec -n openshift-etcd -c etcd $i -- ps -o command= -C etcd | sed 's/.*\(--peer-client-cert-auth=[^ ]*\).*/\1/'
+          done 2>/dev/null
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "--peer-client-cert-auth"
+              compare:
+                op: eq
+                value: true
+        remediation: |
+          This setting is managed by the cluster etcd operator. No remediation required.
+        scored: false
+
+      - id: 2.6
+        text: "Ensure that the --peer-auto-tls argument is not set to true (Manual)"
+        audit: |
+          # Returns 0 if found, 1 if not found
+          for i in $(oc get pods -oname -n openshift-etcd)
+          do
+            oc exec -n openshift-etcd -c etcd $i -- ps -o command= -C etcd | grep -- --peer-auto-tls=true 2>&1>/dev/null ; echo exit_code=$?
+          done 2>/dev/null
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "exit_code"
+              compare:
+                op: eq
+                value: "1"
+        remediation: |
+          This setting is managed by the cluster etcd operator. No remediation required.
+        scored: false
+
+      - id: 2.7
+        text: "Ensure that a unique Certificate Authority is used for etcd (Manual)"
+        audit: |
+          for i in $(oc get pods -oname -n openshift-etcd)
+          do
+            oc exec -n openshift-etcd -c etcd $i -- ps -o command= -C etcd | sed 's/.*\(--trusted-ca-file=[^ ]*\).*/\1/'
+          done 2>/dev/null
+          for i in $(oc get pods -oname -n openshift-etcd)
+          do
+            oc exec -n openshift-etcd -c etcd $i -- ps -o command= -C etcd | sed 's/.*\(--peer-trusted-ca-file=[^ ]*\).*/\1/'
+          done 2>/dev/null
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "file"
+              compare:
+                op: regex
+                value: '\/etc\/kubernetes\/static-pod-certs\/configmaps\/etcd-(?:serving|peer-client)-ca\/ca-bundle\.(?:crt|key)'
+        remediation: |
+          None required. Certificates for etcd are managed by the OpenShift cluster etcd operator.
+        scored: false

cfg/rh-1.0/node.yaml

@@ -0,0 +1,453 @@
+---
+controls:
+version: rh-1.0
+id: 4
+text: "Worker Node Security Configuration"
+type: "node"
+groups:
+  - id: 4.1
+    text: "Worker Node Configuration Files"
+    checks:
+      - id: 4.1.1
+        text: "Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated)"
+        audit: |
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}')
+          do
+            oc debug node/${node} -- chroot /host stat -c "$node %n permissions=%a" /etc/systemd/system/kubelet.service
+          done 2> /dev/null
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "permissions"
+              compare:
+                op: bitmask
+                value: "644"
+        remediation: |
+          By default, the kubelet service file has permissions of 644.
+        scored: true
+
+      - id: 4.1.2
+        text: "Ensure that the kubelet service file ownership is set to root:root (Automated)"
+        audit: |
+          # Should return root:root for each node
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}')
+          do
+            oc debug node/${node} -- chroot /host stat -c "$node %n %U:%G" /etc/systemd/system/kubelet.service
+          done 2> /dev/null
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: root:root
+        remediation: |
+          By default, the kubelet service file has ownership of root:root.
+        scored: true
+
+      - id: 4.1.3
+        text: "If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual)"
+        audit: |
+          for i in $(oc get pods -n openshift-sdn -l app=sdn -oname)
+          do
+            oc exec -n openshift-sdn $i -- stat -Lc "$i %n permissions=%a" /config/kube-proxy-config.yaml
+          done 2> /dev/null
+        use_multiple_values: true
+        tests:
+          bin_op: or
+          test_items:
+            - flag: "permissions"
+              set: true
+              compare:
+                op: bitmask
+                value: "644"
+        remediation: |
+          None needed.
+        scored: false
+
+      - id: 4.1.4
+        text: "Ensure that the proxy kubeconfig file ownership is set to root:root (Manual)"
+        audit: |
+          for i in $(oc get pods -n openshift-sdn -l app=sdn -oname)
+          do
+            oc exec -n openshift-sdn $i -- stat -Lc "$i %n %U:%G" /config/kube-proxy-config.yaml
+          done 2> /dev/null
+        use_multiple_values: true
+        tests:
+          bin_op: or
+          test_items:
+            - flag: root:root
+        remediation: |
+          None required. The configuration is managed by OpenShift operators.
+        scored: false
+
+      - id: 4.1.5
+        text: "Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Manual)"
+        audit: |
+          # Check permissions
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}')
+          do
+            oc debug node/${node} -- chroot /host stat -c "$node %n permissions=%a" /etc/kubernetes/kubelet.conf
+          done 2> /dev/null
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "permissions"
+              compare:
+                op: bitmask
+                value: "644"
+        remediation: |
+          None required.
+        scored: false
+
+      - id: 4.1.6
+        text: "Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Manual)"
+        audit: |
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}')
+          do
+            oc debug node/${node} -- chroot /host stat -c "$node %n %U:%G" /etc/kubernetes/kubelet.conf
+          done 2> /dev/null
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: root:root
+        remediation: |
+          None required.
+        scored: false
+
+      - id: 4.1.7
+        text: "Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Automated)"
+        audit: |
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}')
+          do
+            oc debug node/${node} -- chroot /host stat -c "$node %n permissions=%a" /etc/kubernetes/kubelet-ca.crt
+          done 2> /dev/null
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "permissions"
+              compare:
+                op: bitmask
+                value: "644"
+        remediation: |
+          None required.
+        scored: true
+
+      - id: 4.1.8
+        text: "Ensure that the client certificate authorities file ownership is set to root:root (Automated)"
+        audit: |
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}')
+          do
+            oc debug node/${node} -- chroot /host stat -c "$node %n %U:%G" /etc/kubernetes/kubelet-ca.crt
+          done 2> /dev/null
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: root:root
+        remediation: |
+          None required.
+        scored: true
+
+      - id: 4.1.9
+        text: "Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated)"
+        audit: |
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}')
+          do
+            oc debug node/${node} -- chroot /host stat -c "$node %n permissions=%a" /var/lib/kubelet/kubeconfig
+          done 2> /dev/null
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "permissions"
+              compare:
+                op: bitmask
+                value: "644"
+        remediation: |
+          None required.
+        scored: true
+
+      - id: 4.1.10
+        text: "Ensure that the kubelet configuration file ownership is set to root:root (Automated)"
+        audit: |
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}')
+          do
+            oc debug node/${node} -- chroot /host stat -c "$node %n %U:%G" /var/lib/kubelet/kubeconfig
+          done 2> /dev/null
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: root:root
+        remediation: |
+          None required.
+        scored: true
+
+  - id: 4.2
+    text: "Kubelet"
+    checks:
+      - id: 4.2.1
+        text: "Ensure that the --anonymous-auth argument is set to false (Automated)"
+        audit: |
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}')
+          do
+            oc debug node/${node} -- chroot /host grep -B4 -A1 anonymous: /etc/kubernetes/kubelet.conf
+          done
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "enabled: true"
+              set: false
+        remediation: |
+          Follow the instructions in the documentation to create a Kubelet config CRD
+          and set the anonymous-auth is set to false.
+        scored: true
+
+      - id: 4.2.2
+        text: "Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated)"
+        type: manual
+        # Takes a lot of time for connection to fail and
+        audit: |
+          POD=$(oc -n openshift-kube-apiserver get pod -l app=openshift-kube-apiserver -o jsonpath='{.items[0].metadata.name}')
+          TOKEN=$(oc whoami -t)
+          for name in $(oc get nodes -ojsonpath='{.items[*].metadata.name}')
+          do
+            oc exec -n openshift-kube-apiserver $POD -- curl -sS https://172.25.0.1/api/v1/nodes/$name/proxy/configz -k -H "Authorization:Bearer $TOKEN" | jq -r '.kubeletconfig.authorization.mode'
+          done
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "Connection timed out"
+        remediation: |
+          None required. Unauthenticated/Unauthorized users have no access to OpenShift nodes.
+        scored: true
+
+      - id: 4.2.3
+        text: "Ensure that the --client-ca-file argument is set as appropriate (Automated)"
+        audit: |
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}')
+          do
+            oc debug node/${node} -- chroot /host grep clientCAFile: /etc/kubernetes/kubelet.conf
+          done 2> /dev/null
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "clientCAFile"
+              compare:
+                op: eq
+                value: "/etc/kubernetes/kubelet-ca.crt"
+        remediation: |
+          None required. Changing the clientCAFile value is unsupported.
+        scored: true
+
+      - id: 4.2.4
+        text: "Verify that the read only port is not used or is set to 0 (Automated)"
+        audit: |
+          echo `oc -n openshift-kube-apiserver get cm kube-apiserver-pod -o yaml | grep --color read-only-port` 2> /dev/null
+          echo `oc -n openshift-kube-apiserver get cm config -o yaml | grep --color "read-only-port"` 2> /dev/null
+        tests:
+          bin_op: or
+          test_items:
+            - flag: "read-only-port"
+              compare:
+                op: has
+                value: "[\"0\"]"
+            - flag: "read-only-port"
+              set: false
+        remediation: |
+          In earlier versions of OpenShift 4, the read-only-port argument is not used.
+          Follow the instructions in the documentation to create a Kubelet config CRD
+          and set the --read-only-port is set to 0.
+        scored: true
+
+      - id: 4.2.5
+        text: "Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Automated)"
+        audit: |
+          # Should return 1 for each node
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}')
+          do
+            oc debug node/${node} -- chroot /host ps -ef | grep kubelet | grep streaming-connection-idle-timeout
+            echo exit_code=$?
+          done 2>/dev/null
+          # Should return 1 for each node
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}')
+          do
+            oc debug node/${node} -- chroot /host grep streamingConnectionIdleTimeout /etc/kubernetes/kubelet.conf
+            echo exit_code=$?
+          done 2>/dev/null
+        use_multiple_values: true
+        tests:
+          bin_op: or
+          test_items:
+            - flag: --streaming-connection-idle-timeout
+              compare:
+                op: noteq
+                value: 0
+            - flag: "exit_code"
+              compare:
+                op: eq
+                value: 1
+        remediation: |
+          Follow the instructions in the documentation to create a Kubelet config CRD and set
+          the --streaming-connection-idle-timeout to the desired value. Do not set the value to 0.
+        scored: true
+
+      - id: 4.2.6
+        text: "Ensure that the --protect-kernel-defaults argument is not set (Manual)"
+        audit: |
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}');
+          do
+            oc debug node/${node} -- chroot /host more /etc/kubernetes/kubelet.conf;
+          done
+        tests:
+          test_items:
+            - flag: protectKernelDefaults
+              set: false
+        remediation: |
+          None required. The OpenShift 4 kubelet modifies the system tunable;
+          using the protect-kernel-defaults flag will cause the kubelet to fail on start if the tunables
+          don't match the kubelet configuration and the OpenShift node will fail to start.
+        scored: false
+
+      - id: 4.2.7
+        text: "Ensure that the --make-iptables-util-chains argument is set to true (Manual)"
+        audit: |
+          /bin/bash
+          flag=make-iptables-util-chains
+          opt=makeIPTablesUtilChains
+          # look at each machineconfigpool
+          while read -r pool nodeconfig; do
+            # true by default
+            value='true'
+            # first look for the flag
+            oc get machineconfig $nodeconfig -o json | jq -r '.spec.config.systemd[][] | select(.name=="kubelet.service") | .contents' | sed -n "/^ExecStart=/,/^\$/ { /^\\s*--$flag=false/ q 100 }"
+            # if the above command exited with 100, the flag was false
+            [ $? == 100 ] && value='false'
+            # now look in the yaml KubeletConfig
+            yamlconfig=$(oc get machineconfig $nodeconfig -o json | jq -r '.spec.config.storage.files[] | select(.path=="/etc/kubernetes/kubelet.conf") | .contents.source ' | sed 's/^data:,//' | while read; do echo -e ${REPLY//%/\\x}; done)
+            echo "$yamlconfig" | sed -n "/^$opt:\\s*false\\s*$/ q 100"
+            [ $? == 100 ] && value='false'
+            echo "Pool $pool has $flag ($opt) set to $value"
+          done < <(oc get machineconfigpools -o json | jq -r '.items[] | select(.status.machineCount>0) | .metadata.name + " " + .spec.configuration.name')
+        use_multiple_values: true
+        tests:
+          test_items:
+            - flag: "set to true"
+        remediation: |
+          None required. The --make-iptables-util-chains argument is set to true by default.
+        scored: false
+
+      - id: 4.2.8
+        text: "Ensure that the --hostname-override argument is not set (Manual)"
+        audit: |
+          echo `oc get machineconfig 01-worker-kubelet -o yaml | grep hostname-override`
+          echo `oc get machineconfig 01-master-kubelet -o yaml | grep hostname-override`
+        tests:
+          test_items:
+            - flag: hostname-override
+              set: false
+        remediation: |
+          By default, --hostname-override argument is not set.
+        scored: false
+
+      - id: 4.2.9
+        text: "Ensure that the kubeAPIQPS [--event-qps] argument is set to 0 or a level which ensures appropriate event capture (Automated)"
+        audit: |
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}');
+          do
+            oc debug node/${node} -- chroot /host more /etc/kubernetes/kubelet.conf;
+          done
+          oc get machineconfig 01-worker-kubelet -o yaml | grep --color kubeAPIQPS%3A%2050
+          oc get machineconfig 01-master-kubelet -o yaml | grep --color kubeAPIQPS%3A%2050
+        type: "manual"
+        remediation: |
+          Follow the documentation to edit kubelet parameters
+          https://docs.openshift.com/container-platform/4.5/scalability_and_performance/recommended-host-practices.html#create-a-kubeletconfig-crd-to-edit-kubelet-parameters
+          KubeAPIQPS: <QPS>
+        scored: true
+
+      - id: 4.2.10
+        text: "Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated)"
+        audit: |
+          oc get configmap config -n openshift-kube-apiserver -ojson | jq -r '.data["config.yaml"]' | jq '.kubeletClientInfo'
+        tests:
+          bin_op: and
+          test_items:
+            - flag: "/etc/kubernetes/static-pod-certs/secrets/kubelet-client/tls.crt"
+            - flag: "/etc/kubernetes/static-pod-certs/secrets/kubelet-client/tls.key"
+        remediation: |
+          OpenShift automatically manages TLS authentication for the API server communication with the node/kublet.
+          This is not configurable.
+        scored: true
+
+      - id: 4.2.11
+        text: "Ensure that the --rotate-certificates argument is not set to false (Manual)"
+        audit: |
+          #Verify the rotateKubeletClientCertificate feature gate is not set to false
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}')
+          do
+            oc debug node/${node} -- chroot /host cat /etc/kubernetes/kubelet.conf | grep RotateKubeletClientCertificate
+          done 2> /dev/null
+          # Verify the rotateCertificates argument is set to true
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}')
+          do
+            oc debug node/${node} -- chroot host grep rotate /etc/kubernetes/kubelet.conf;
+          done 2> /dev/null
+        use_multiple_values: true
+        tests:
+          bin_op: or
+          test_items:
+            - flag: rotateCertificates
+              compare:
+                op: eq
+                value: true
+            - flag: rotateKubeletClientCertificates
+              compare:
+                op: noteq
+                value: false
+            - flag: rotateKubeletClientCertificates
+              set: false
+        remediation: |
+          None required.
+        scored: false
+
+      - id: 4.2.12
+        text: "Verify that the RotateKubeletServerCertificate argument is set to true (Manual)"
+        audit: |
+          #Verify the rotateKubeletServerCertificate feature gate is on
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}');
+          do
+            oc debug node/${node} -- chroot /host grep RotateKubeletServerCertificate /etc/kubernetes/kubelet.conf;
+          done 2> /dev/null
+          # Verify the rotateCertificates argument is set to true
+          for node in $(oc get nodes -o jsonpath='{.items[*].metadata.name}')
+          do
+            oc debug node/${node} -- chroot host grep rotate /etc/kubernetes/kubelet.conf;
+          done 2> /dev/null
+        use_multiple_values: true
+        tests:
+          bin_op: or
+          test_items:
+            - flag: RotateKubeletServerCertificate
+              compare:
+                op: eq
+                value: true
+            - flag: rotateCertificates
+              compare:
+                op: eq
+                value: true
+        remediation: |
+          By default, kubelet server certificate rotation is disabled.
+        scored: false
+
+      - id: 4.2.13
+        text: "Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual)"
+        audit: |
+          # needs verification
+          # verify cipher suites
+          oc describe --namespace=openshift-ingress-operator ingresscontroller/default
+          oc get kubeapiservers.operator.openshift.io cluster -o json |jq .spec.observedConfig.servingInfo
+          oc get openshiftapiservers.operator.openshift.io cluster -o json |jq .spec.observedConfig.servingInfo
+          oc get cm -n openshift-authentication v4-0-config-system-cliconfig -o jsonpath='{.data.v4\-0\-config\-system\-cliconfig}' | jq .servingInfo
+          #check value for tlsSecurityProfile; null is returned if default is used
+          oc get kubeapiservers.operator.openshift.io cluster -o json |jq .spec.tlsSecurityProfile
+        type: manual
+        remediation: |
+          Follow the directions above and in the OpenShift documentation to configure the tlsSecurityProfile.
+          Configuring Ingress
+        scored: false

cfg/rh-1.0/policies.yaml

@@ -0,0 +1,283 @@
+---
+controls:
+version: rh-1.0
+id: 5
+text: "Kubernetes Policies"
+type: "policies"
+groups:
+  - id: 5.1
+    text: "RBAC and Service Accounts"
+    checks:
+      - id: 5.1.1
+        text: "Ensure that the cluster-admin role is only used where required (Manual)"
+        type: "manual"
+        remediation: |
+          Identify all clusterrolebindings to the cluster-admin role. Check if they are used and
+          if they need this role or if they could use a role with fewer privileges.
+          Where possible, first bind users to a lower privileged role and then remove the
+          clusterrolebinding to the cluster-admin role :
+          kubectl delete clusterrolebinding [name]
+        scored: false
+
+      - id: 5.1.2
+        text: "Minimize access to secrets (Manual)"
+        type: "manual"
+        remediation: |
+          Where possible, remove get, list and watch access to secret objects in the cluster.
+        scored: false
+
+      - id: 5.1.3
+        text: "Minimize wildcard use in Roles and ClusterRoles (Manual)"
+        type: "manual"
+        remediation: |
+          Where possible replace any use of wildcards in clusterroles and roles with specific
+          objects or actions.
+        scored: false
+
+      - id: 5.1.4
+        text: "Minimize access to create pods (Manual)"
+        type: "manual"
+        remediation: |
+          Where possible, remove create access to pod objects in the cluster.
+        scored: false
+
+      - id: 5.1.5
+        text: "Ensure that default service accounts are not actively used. (Manual)"
+        type: "manual"
+        remediation: |
+          None required.
+        scored: false
+
+      - id: 5.1.6
+        text: "Ensure that Service Account Tokens are only mounted where necessary (Manual)"
+        type: "manual"
+        remediation: |
+          Modify the definition of pods and service accounts which do not need to mount service
+          account tokens to disable it.
+        scored: false
+
+  - id: 5.2
+    text: "Pod Security Policies"
+    checks:
+      - id: 5.2.1
+        text: "Minimize the admission of privileged containers (Manual)"
+        audit: |
+          # needs verification
+          for i in `oc get scc --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'`;
+          do
+            echo "$i"; oc describe scc $i | grep "Allow Privileged";
+          done
+        tests:
+          test_items:
+            - flag: "false"
+        remediation: |
+          Create a SCC as described in the OpenShift documentation, ensuring that the Allow
+          Privileged field is set to false.
+        scored: false
+
+      - id: 5.2.2
+        text: "Minimize the admission of containers wishing to share the host process ID namespace (Manual)"
+        audit: |
+          for i in `oc get scc --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'`;
+          do
+            echo "$i"; oc describe scc $i | grep "Allow Host PID";
+          done
+        tests:
+          test_items:
+            - flag: "false"
+        remediation: |
+          Create a SCC as described in the OpenShift documentation, ensuring that the Allow Host
+          PID field is set to false.
+        scored: false
+
+      - id: 5.2.3
+        text: "Minimize the admission of containers wishing to share the host IPC namespace (Manual)"
+        audit: |
+          for i in `oc get scc --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'`;
+          do
+            echo "$i"; oc describe scc $i | grep "Allow Host IPC";
+          done
+        tests:
+          test_items:
+            - flag: "false"
+        remediation: |
+          Create a SCC as described in the OpenShift documentation, ensuring that the Allow Host
+          IPC field is set to false.
+        scored: false
+
+      - id: 5.2.4
+        text: "Minimize the admission of containers wishing to share the host network namespace (Manual)"
+        audit: |
+          for i in `oc get scc --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'`;
+          do
+            echo "$i"; oc describe scc $i | grep "Allow Host Network";
+          done
+        tests:
+          test_items:
+            - flag: "false"
+        remediation: |
+          Create a SCC as described in the OpenShift documentation, ensuring that the Allow Host
+          Network field is omitted or set to false.
+        scored: false
+
+      - id: 5.2.5
+        text: "Minimize the admission of containers with allowPrivilegeEscalation (Manual)"
+        audit: |
+          for i in `oc get scc --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'`;
+          do
+            echo "$i"; oc describe scc $i | grep "Allow Privilege Escalation";
+          done
+        tests:
+          test_items:
+            - flag: "false"
+        remediation: |
+          Create a SCC as described in the OpenShift documentation, ensuring that the Allow
+          Privilege Escalation field is omitted or set to false.
+        scored: false
+
+      - id: 5.2.6
+        text: "Minimize the admission of root containers (Manual)"
+        audit: |
+          # needs verification
+          for i in `oc get scc --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'`;
+          do
+            echo "$i";
+            oc describe scc $i | grep "Run As User Strategy";
+          done
+          #For SCCs with MustRunAs verify that the range of UIDs does not include 0
+          for i in `oc get scc --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'`;
+          do
+            echo "$i";
+            oc describe scc $i | grep "\sUID";
+          done
+        tests:
+          bin_op: or
+          test_items:
+            - flag: "MustRunAsNonRoot"
+            - flag: "MustRunAs"
+              compare:
+                op: nothave
+                value: 0
+        remediation: |
+          None required. By default, OpenShift includes the non-root SCC with the the Run As User
+          Strategy is set to either MustRunAsNonRoot. If additional SCCs are appropriate, follow the
+          OpenShift documentation to create custom SCCs.
+        scored: false
+
+      - id: 5.2.7
+        text: "Minimize the admission of containers with the NET_RAW capability (Manual)"
+        audit: |
+          # needs verification
+          for i in `oc get scc --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}'`;
+          do
+            echo "$i";
+            oc describe scc $i | grep "Required Drop Capabilities";
+          done
+        tests:
+          bin_op: or
+          test_items:
+            - flag: "ALL"
+            - flag: "NET_RAW"
+        remediation: |
+          Create a SCC as described in the OpenShift documentation, ensuring that the Required
+          Drop Capabilities is set to include either NET_RAW or ALL.
+        scored: false
+
+      - id: 5.2.8
+        text: "Minimize the admission of containers with added capabilities (Manual)"
+        type: "manual"
+        remediation: |
+          Ensure that Allowed Capabilities is set to an empty array for every SCC in the cluster
+          except for the privileged SCC.
+        scored: false
+
+      - id: 5.2.9
+        text: "Minimize the admission of containers with capabilities assigned (Manual)"
+        type: "manual"
+        remediation: |
+          Review the use of capabilites in applications running on your cluster. Where a namespace
+          contains applicaions which do not require any Linux capabities to operate consider
+          adding a SCC which forbids the admission of containers which do not drop all capabilities.
+        scored: false
+
+  - id: 5.3
+    text: "Network Policies and CNI"
+    checks:
+      - id: 5.3.1
+        text: "Ensure that the CNI in use supports Network Policies (Manual)"
+        type: "manual"
+        remediation: |
+          None required.
+        scored: false
+
+      - id: 5.3.2
+        text: "Ensure that all Namespaces have Network Policies defined (Manual)"
+        type: "manual"
+        remediation: |
+          Follow the documentation and create NetworkPolicy objects as you need them.
+        scored: false
+
+  - id: 5.4
+    text: "Secrets Management"
+    checks:
+      - id: 5.4.1
+        text: "Prefer using secrets as files over secrets as environment variables (Manual)"
+        type: "manual"
+        remediation: |
+          If possible, rewrite application code to read secrets from mounted secret files, rather than
+          from environment variables.
+        scored: false
+
+      - id: 5.4.2
+        text: "Consider external secret storage (Manual)"
+        type: "manual"
+        remediation: |
+          Refer to the secrets management options offered by your cloud provider or a third-party
+          secrets management solution.
+        scored: false
+
+  - id: 5.5
+    text: "Extensible Admission Control"
+    checks:
+      - id: 5.5.1
+        text: "Configure Image Provenance using ImagePolicyWebhook admission controller (Manual)"
+        type: "manual"
+        remediation: |
+          Follow the OpenShift documentation: [Image configuration resources](https://docs.openshift.com/container-platform/4.5/openshift_images/image-configuration.html
+        scored: false
+
+  - id: 5.7
+    text: "General Policies"
+    checks:
+      - id: 5.7.1
+        text: "Create administrative boundaries between resources using namespaces (Manual)"
+        type: "manual"
+        remediation: |
+          Follow the documentation and create namespaces for objects in your deployment as you need
+          them.
+        scored: false
+
+      - id: 5.7.2
+        text: "Ensure that the seccomp profile is set to docker/default in your pod definitions (Manual)"
+        type: "manual"
+        remediation: |
+          To enable the default seccomp profile, use the reserved value /runtime/default that will
+          make sure that the pod uses the default policy available on the host.
+        scored: false
+
+      - id: 5.7.3
+        text: "Apply Security Context to Your Pods and Containers (Manual)"
+        type: "manual"
+        remediation: |
+          Follow the Kubernetes documentation and apply security contexts to your pods. For a
+          suggested list of security contexts, you may refer to the CIS Security Benchmark for Docker
+          Containers.
+        scored: false
+
+      - id: 5.7.4
+        text: "The default namespace should not be used (Manual)"
+        type: "manual"
+        remediation: |
+          Ensure that namespaces are created to allow for appropriate segregation of Kubernetes
+          resources and that all new resources are created in a specific namespace.
+        scored: false

check/check.go

@@ -107,12 +107,14 @@ func (r *defaultRunner) Run(c *Check) State {
 // Run executes the audit commands specified in a check and outputs
 // the results.
 func (c *Check) run() State {
+	glog.V(3).Infof("-----   Running check %v   -----", c.ID)
 	// Since this is an Scored check
 	// without tests return a 'WARN' to alert
 	// the user that this check needs attention
 	if c.Scored && strings.TrimSpace(c.Type) == "" && c.Tests == nil {
 		c.Reason = "There are no tests"
 		c.State = WARN
+		glog.V(3).Info(c.Reason)
 		return c.State
 	}
 
@@ -120,6 +122,7 @@ func (c *Check) run() State {
 	if c.Type == SKIP {
 		c.Reason = "Test marked as skip"
 		c.State = INFO
+		glog.V(3).Info(c.Reason)
 		return c.State
 	}
 
@@ -127,6 +130,7 @@ func (c *Check) run() State {
 	if c.Type == MANUAL {
 		c.Reason = "Test marked as a manual test"
 		c.State = WARN
+		glog.V(3).Info(c.Reason)
 		return c.State
 	}
 
@@ -138,6 +142,7 @@ func (c *Check) run() State {
 		} else {
 			c.State = WARN
 		}
+		glog.V(3).Info(c.Reason)
 		return c.State
 	}
 
@@ -172,12 +177,13 @@ func (c *Check) run() State {
 		} else {
 			c.State = WARN
 		}
+		glog.V(3).Info(c.Reason)
 	}
 
 	if finalOutput != nil {
-		glog.V(3).Infof("Check.ID: %s Command: %q TestResult: %t State: %q \n", c.ID, lastCommand, finalOutput.testResult, c.State)
+		glog.V(3).Infof("Command: %q TestResult: %t State: %q \n", lastCommand, finalOutput.testResult, c.State)
 	} else {
-		glog.V(3).Infof("Check.ID: %s Command: %q TestResult: <<EMPTY>> \n", c.ID, lastCommand)
+		glog.V(3).Infof("Command: %q TestResult: <<EMPTY>> \n", lastCommand)
 	}
 
 	if c.Reason != "" {
@@ -212,7 +218,7 @@ func (c *Check) execute() (finalOutput *testOutput, err error) {
 	res := make([]testOutput, len(ts.TestItems))
 	expectedResultArr := make([]string, len(res))
 
-	glog.V(3).Infof("%d tests", len(ts.TestItems))
+	glog.V(3).Infof("Running %d test_items", len(ts.TestItems))
 	for i, t := range ts.TestItems {
 
 		t.isMultipleOutput = c.IsMultiple
@@ -236,6 +242,7 @@ func (c *Check) execute() (finalOutput *testOutput, err error) {
 			t.auditUsed = AuditEnv
 			result = *(t.execute(c.AuditEnvOutput))
 		}
+		glog.V(2).Infof("Used %s", t.auditUsed)
 		res[i] = result
 		expectedResultArr[i] = res[i].ExpectedResult
 	}
@@ -289,8 +296,8 @@ func runAudit(audit string) (output string, err error) {
 	if err != nil {
 		err = fmt.Errorf("failed to run: %q, output: %q, error: %s", audit, output, err)
 	} else {
-		glog.V(3).Infof("Command %q\n - Output:\n %q", audit, output)
-
+		glog.V(3).Infof("Command: %q", audit)
+		glog.V(3).Infof("Output:\n %q", output)
 	}
 	return output, err
 }

check/test.go

@@ -126,6 +126,9 @@ func (t flagTestItem) findValue(s string) (match bool, value string, err error)
 		// flag: somevalue
 		// --flag
 		// somevalue
+		// DOESN'T COVER - use pathTestItem implementation of findValue() for this
+		// flag:
+		//	 - wehbook
 		pttn := `(` + t.Flag + `)(=|: *)*([^\s]*) *`
 		flagRe := regexp.MustCompile(pttn)
 		vals := flagRe.FindStringSubmatch(s)
@@ -145,7 +148,7 @@ func (t flagTestItem) findValue(s string) (match bool, value string, err error)
 			err = fmt.Errorf("invalid flag in testItem definition: %s", s)
 		}
 	}
-	glog.V(3).Infof("In flagTestItem.findValue %s, match %v, s %s, t.Flag %s", value, match, s, t.Flag)
+	glog.V(3).Infof("In flagTestItem.findValue %s", value)
 
 	return match, value, err
 }
@@ -183,6 +186,7 @@ func (t envTestItem) findValue(s string) (match bool, value string, err error) {
 			value = ""
 		}
 	}
+	glog.V(3).Infof("In envTestItem.findValue %s", value)
 	return match, value, nil
 }
 
@@ -232,10 +236,22 @@ func (t testItem) evaluate(s string) *testOutput {
 	}
 
 	result.flagFound = match
-	glog.V(3).Info(fmt.Sprintf("found %v", result.flagFound))
-
-
-	return result
+	var isExist = "exists"
+	if !result.flagFound{
+		isExist = "does not exist"
+	}
+	switch t.auditUsed {
+        case "auditCommand": 
+			glog.V(3).Infof("Flag '%s' %s", t.Flag, isExist)
+        case "auditConfig":
+			glog.V(3).Infof("Path '%s' %s", t.Path, isExist)
+		case "auditEnv":
+			glog.V(3).Infof("Env '%s' %s", t.Env, isExist)
+		default: 
+		glog.V(3).Infof("Error with identify audit used %s", t.auditUsed)
+    	}
+	
+    	return result
 }
 
 func compareOp(tCompareOp string, flagVal string, tCompareValue string, flagName string) (string, bool) {

cmd/common.go

@@ -313,12 +313,12 @@ func loadTargetMapping(v *viper.Viper) (map[string][]string, error) {
 	return benchmarkVersionToTargetsMap, nil
 }
 
-func getBenchmarkVersion(kubeVersion, benchmarkVersion string, v *viper.Viper) (bv string, err error) {
+func getBenchmarkVersion(kubeVersion, benchmarkVersion, platformName string, v *viper.Viper) (bv string, err error) {
 	if !isEmpty(kubeVersion) && !isEmpty(benchmarkVersion) {
 		return "", fmt.Errorf("It is an error to specify both --version and --benchmark flags")
 	}
-	if isEmpty(benchmarkVersion) && isEmpty(kubeVersion) {
-		benchmarkVersion = getPlatformBenchmarkVersion(getPlatformName())
+	if isEmpty(benchmarkVersion) && isEmpty(kubeVersion) && !isEmpty(platformName){
+		benchmarkVersion = getPlatformBenchmarkVersion(platformName)
 	}
 
 	if isEmpty(benchmarkVersion) {

cmd/common_test.go

@@ -195,17 +195,19 @@ func TestIsMaster(t *testing.T) {
 	defer restore()
 
 	for _, tc := range testCases {
-		cfgFile = tc.cfgFile
-		initConfig()
+		func() {
+			cfgFile = tc.cfgFile
+			initConfig()
 
-		oldGetBinariesFunc := getBinariesFunc
-		getBinariesFunc = tc.getBinariesFunc
-		defer func() {
-			getBinariesFunc = oldGetBinariesFunc
-			cfgFile = ""
+			oldGetBinariesFunc := getBinariesFunc
+			getBinariesFunc = tc.getBinariesFunc
+			defer func() {
+				getBinariesFunc = oldGetBinariesFunc
+				cfgFile = ""
+			}()
+
+			assert.Equal(t, tc.isMaster, isMaster(), tc.name)
 		}()
-
-		assert.Equal(t, tc.isMaster, isMaster(), tc.name)
 	}
 }
 
@@ -320,11 +322,11 @@ func TestGetBenchmarkVersion(t *testing.T) {
 		t.Fatalf("Unable to load config file %v", err)
 	}
 
-	type getBenchmarkVersionFnToTest func(kubeVersion, benchmarkVersion string, v *viper.Viper) (string, error)
+	type getBenchmarkVersionFnToTest func(kubeVersion, benchmarkVersion, platformName string, v *viper.Viper) (string, error)
 
-	withFakeKubectl := func(kubeVersion, benchmarkVersion string, v *viper.Viper, fn getBenchmarkVersionFnToTest) (string, error) {
+	withFakeKubectl := func(kubeVersion, benchmarkVersion, platformName string, v *viper.Viper, fn getBenchmarkVersionFnToTest) (string, error) {
 		execCode := `#!/bin/sh
-		echo '{"serverVersion": {"major": "1", "minor": "15", "gitVersion": "v1.15.10"}}'
+		echo '{"serverVersion": {"major": "1", "minor": "18", "gitVersion": "v1.18.10"}}'
 		`
 		restore, err := fakeExecutableInPath("kubectl", execCode)
 		if err != nil {
@@ -332,39 +334,40 @@ func TestGetBenchmarkVersion(t *testing.T) {
 		}
 		defer restore()
 
-		return fn(kubeVersion, benchmarkVersion, v)
+		return fn(kubeVersion, benchmarkVersion, platformName, v)
 	}
 
-	withNoPath := func(kubeVersion, benchmarkVersion string, v *viper.Viper, fn getBenchmarkVersionFnToTest) (string, error) {
+	withNoPath := func(kubeVersion, benchmarkVersion, platformName string, v *viper.Viper, fn getBenchmarkVersionFnToTest) (string, error) {
 		restore, err := prunePath()
 		if err != nil {
 			t.Fatal("Failed when calling prunePath ", err)
 		}
 		defer restore()
 
-		return fn(kubeVersion, benchmarkVersion, v)
+		return fn(kubeVersion, benchmarkVersion, platformName, v)
 	}
 
-	type getBenchmarkVersionFn func(string, string, *viper.Viper, getBenchmarkVersionFnToTest) (string, error)
+	type getBenchmarkVersionFn func(string, string, string, *viper.Viper, getBenchmarkVersionFnToTest) (string, error)
 	cases := []struct {
 		n                string
 		kubeVersion      string
 		benchmarkVersion string
+		platformName     string
 		v                *viper.Viper
 		callFn           getBenchmarkVersionFn
 		exp              string
 		succeed          bool
 	}{
-		{n: "both versions", kubeVersion: "1.11", benchmarkVersion: "cis-1.3", exp: "cis-1.3", callFn: withNoPath, v: viper.New(), succeed: false},
-		{n: "no version-missing-kubectl", kubeVersion: "", benchmarkVersion: "", v: viperWithData, exp: "", callFn: withNoPath, succeed: false},
-		{n: "no version-fakeKubectl", kubeVersion: "", benchmarkVersion: "", v: viperWithData, exp: "cis-1.5", callFn: withFakeKubectl, succeed: true},
-		{n: "kubeVersion", kubeVersion: "1.15", benchmarkVersion: "", v: viperWithData, exp: "cis-1.5", callFn: withNoPath, succeed: true},
-		{n: "ocpVersion310", kubeVersion: "ocp-3.10", benchmarkVersion: "", v: viperWithData, exp: "rh-0.7", callFn: withNoPath, succeed: true},
-		{n: "ocpVersion311", kubeVersion: "ocp-3.11", benchmarkVersion: "", v: viperWithData, exp: "rh-0.7", callFn: withNoPath, succeed: true},
-		{n: "gke10", kubeVersion: "gke-1.0", benchmarkVersion: "", v: viperWithData, exp: "gke-1.0", callFn: withNoPath, succeed: true},
+		{n: "both versions", kubeVersion: "1.11", benchmarkVersion: "cis-1.3", platformName: "", exp: "cis-1.3", callFn: withNoPath, v: viper.New(), succeed: false},
+		{n: "no version-missing-kubectl", kubeVersion: "", benchmarkVersion: "", platformName: "", v: viperWithData, exp: "cis-1.6", callFn: withNoPath, succeed: true},
+		{n: "no version-fakeKubectl", kubeVersion: "", benchmarkVersion: "", platformName: "", v: viperWithData, exp: "cis-1.6", callFn: withFakeKubectl, succeed: true},
+		{n: "kubeVersion", kubeVersion: "1.15", benchmarkVersion: "", platformName: "", v: viperWithData, exp: "cis-1.5", callFn: withNoPath, succeed: true},
+		{n: "ocpVersion310", kubeVersion: "ocp-3.10", benchmarkVersion: "", platformName: "", v: viperWithData, exp: "rh-0.7", callFn: withNoPath, succeed: true},
+		{n: "ocpVersion311", kubeVersion: "ocp-3.11", benchmarkVersion: "", platformName: "", v: viperWithData, exp: "rh-0.7", callFn: withNoPath, succeed: true},
+		{n: "gke10", kubeVersion: "gke-1.0", benchmarkVersion: "", platformName: "", v: viperWithData, exp: "gke-1.0", callFn: withNoPath, succeed: true},
 	}
 	for _, c := range cases {
-		rv, err := c.callFn(c.kubeVersion, c.benchmarkVersion, c.v, getBenchmarkVersion)
+		rv, err := c.callFn(c.kubeVersion, c.benchmarkVersion, c.platformName, c.v, getBenchmarkVersion)
 		if c.succeed {
 			if err != nil {
 				t.Errorf("[%q]-Unexpected error: %v", c.n, err)
@@ -506,17 +509,19 @@ func TestIsEtcd(t *testing.T) {
 	defer restore()
 
 	for _, tc := range testCases {
-		cfgFile = tc.cfgFile
-		initConfig()
+		func() {
+			cfgFile = tc.cfgFile
+			initConfig()
 
-		oldGetBinariesFunc := getBinariesFunc
-		getBinariesFunc = tc.getBinariesFunc
-		defer func() {
-			getBinariesFunc = oldGetBinariesFunc
-			cfgFile = ""
+			oldGetBinariesFunc := getBinariesFunc
+			getBinariesFunc = tc.getBinariesFunc
+			defer func() {
+				getBinariesFunc = oldGetBinariesFunc
+				cfgFile = ""
+			}()
+
+			assert.Equal(t, tc.isEtcd, isEtcd(), tc.name)
 		}()
-
-		assert.Equal(t, tc.isEtcd, isEtcd(), tc.name)
 	}
 }
 

cmd/kubernetes_version.go

@@ -33,24 +33,28 @@ func (k *KubeVersion) BaseVersion() string {
 }
 
 func getKubeVersionFromRESTAPI() (*KubeVersion, error) {
+	glog.V(2).Info("Try to get version from Rest API")
 	k8sVersionURL := getKubernetesURL()
 	serviceaccount := "/var/run/secrets/kubernetes.io/serviceaccount"
 	cacertfile := fmt.Sprintf("%s/ca.crt", serviceaccount)
 	tokenfile := fmt.Sprintf("%s/token", serviceaccount)
 
-	tlsCert, err := loadCertficate(cacertfile)
+	tlsCert, err := loadCertificate(cacertfile)
 	if err != nil {
+		glog.V(2).Infof("Failed loading certificate Error: %s", err)
 		return nil, err
 	}
 
 	tb, err := ioutil.ReadFile(tokenfile)
 	if err != nil {
+		glog.V(2).Infof("Failed reading token file Error: %s", err)
 		return nil, err
 	}
 	token := strings.TrimSpace(string(tb))
 
 	data, err := getWebDataWithRetry(k8sVersionURL, token, tlsCert)
 	if err != nil {
+		glog.V(2).Infof("Failed to get data Error: %s", err)
 		return nil, err
 	}
 
@@ -143,7 +147,7 @@ func getWebData(srvURL, token string, cacert *tls.Certificate) ([]byte, error) {
 	return ioutil.ReadAll(resp.Body)
 }
 
-func loadCertficate(certFile string) (*tls.Certificate, error) {
+func loadCertificate(certFile string) (*tls.Certificate, error) {
 	cacert, err := ioutil.ReadFile(certFile)
 	if err != nil {
 		return nil, err

cmd/kubernetes_version_test.go

@@ -11,8 +11,8 @@ import (
 	"testing"
 )
 
-func TestLoadCertficate(t *testing.T) {
-	tmp, err := ioutil.TempDir("", "TestFakeLoadCertficate")
+func TestLoadCertificate(t *testing.T) {
+	tmp, err := ioutil.TempDir("", "TestFakeLoadCertificate")
 	if err != nil {
 		t.Fatalf("unable to create temp directory: %v", err)
 	}
@@ -58,7 +58,7 @@ FAjB57z2NcIgJuVpQnGRYtr/JcH2Qdsq8bLtXaojUIWOOqoTDRLYozdMOOQ=
 
 	for id, c := range cases {
 		t.Run(strconv.Itoa(id), func(t *testing.T) {
-			tlsCert, err := loadCertficate(c.file)
+			tlsCert, err := loadCertificate(c.file)
 			if !c.fail {
 				if err != nil {
 					t.Errorf("unexpected error: %v", err)

cmd/master.go

@@ -28,7 +28,7 @@ var masterCmd = &cobra.Command{
 	Short: "Run Kubernetes benchmark checks from the master.yaml file.",
 	Long:  `Run Kubernetes benchmark checks from the master.yaml file in cfg/<version>.`,
 	Run: func(cmd *cobra.Command, args []string) {
-		bv, err := getBenchmarkVersion(kubeVersion, benchmarkVersion, viper.GetViper())
+		bv, err := getBenchmarkVersion(kubeVersion, benchmarkVersion, getPlatformName(), viper.GetViper())
 		if err != nil {
 			exitWithError(fmt.Errorf("unable to determine benchmark version: %v", err))
 		}
@@ -37,6 +37,7 @@ var masterCmd = &cobra.Command{
 		runChecks(check.MASTER, filename)
 		writeOutput(controlsCollection)
 	},
+	Deprecated: "this command will be retired soon. Please use the `run` command with `--targets=master` instead.",
 }
 
 func init() {

cmd/node.go

@@ -28,7 +28,7 @@ var nodeCmd = &cobra.Command{
 	Short: "Run Kubernetes benchmark checks from the node.yaml file.",
 	Long:  `Run Kubernetes benchmark checks from the node.yaml file in cfg/<version>.`,
 	Run: func(cmd *cobra.Command, args []string) {
-		bv, err := getBenchmarkVersion(kubeVersion, benchmarkVersion, viper.GetViper())
+		bv, err := getBenchmarkVersion(kubeVersion, benchmarkVersion, getPlatformName(), viper.GetViper())
 		if err != nil {
 			exitWithError(fmt.Errorf("unable to determine benchmark version: %v", err))
 		}
@@ -37,6 +37,7 @@ var nodeCmd = &cobra.Command{
 		runChecks(check.NODE, filename)
 		writeOutput(controlsCollection)
 	},
+	Deprecated: "this command will be retired soon. Please use the `run` command with `--targets=node` instead.",
 }
 
 func init() {

cmd/root.go

@@ -68,7 +68,7 @@ var RootCmd = &cobra.Command{
 	Short: "Run CIS Benchmarks checks against a Kubernetes deployment",
 	Long:  `This tool runs the CIS Kubernetes Benchmark (https://www.cisecurity.org/benchmark/kubernetes/)`,
 	Run: func(cmd *cobra.Command, args []string) {
-		bv, err := getBenchmarkVersion(kubeVersion, benchmarkVersion, viper.GetViper())
+		bv, err := getBenchmarkVersion(kubeVersion, benchmarkVersion, getPlatformName(), viper.GetViper())
 		if err != nil {
 			exitWithError(fmt.Errorf("unable to determine benchmark version: %v", err))
 		}

cmd/run.go

@@ -32,7 +32,7 @@ var runCmd = &cobra.Command{
 			exitWithError(fmt.Errorf("unable to get `targets` from command line :%v", err))
 		}
 
-		bv, err := getBenchmarkVersion(kubeVersion, benchmarkVersion, viper.GetViper())
+		bv, err := getBenchmarkVersion(kubeVersion, benchmarkVersion, getPlatformName(), viper.GetViper())
 		if err != nil {
 			exitWithError(fmt.Errorf("unable to get benchmark version. error: %v", err))
 		}
@@ -52,7 +52,10 @@ var runCmd = &cobra.Command{
 
 		// Merge version-specific config if any.
 		path := filepath.Join(cfgDir, bv)
-		mergeConfig(path)
+		err = mergeConfig(path)
+		if err != nil {
+			fmt.Printf("Error in mergeConfig: %v\n", err)
+		}
 
 		err = run(targets, bv)
 		if err != nil {

cmd/util.go

@@ -291,17 +291,22 @@ func getKubeVersion() (*KubeVersion, error) {
 	_, err := exec.LookPath("kubectl")
 
 	if err != nil {
+		glog.V(3).Infof("Error locating kubectl: %s", err)
 		_, err = exec.LookPath("kubelet")
 		if err != nil {
+			glog.V(3).Infof("Error locating kubelet: %s", err)
 			// Search for the kubelet binary all over the filesystem and run the first match to get the kubernetes version
 			cmd := exec.Command("/bin/sh", "-c", "`find / -type f -executable -name kubelet 2>/dev/null | grep -m1 .` --version")
 			out, err := cmd.CombinedOutput()
 			if err == nil {
+				glog.V(3).Infof("Found kubelet and query kubernetes version is: %s", string(out))
 				return getVersionFromKubeletOutput(string(out)), nil
 			}
 
 			glog.Warning(missingKubectlKubeletMessage)
-			return nil, fmt.Errorf("unable to find the programs kubectl or kubelet in the PATH")
+			glog.V(1).Info("unable to find the programs kubectl or kubelet in the PATH")
+			glog.V(1).Infof("Cant detect version, assuming default %s", defaultKubeVersion)
+			return &KubeVersion{baseVersion: defaultKubeVersion}, nil 
 		}
 		return getKubeVersionFromKubelet(), nil
 	}
@@ -313,6 +318,7 @@ func getKubeVersionFromKubectl() *KubeVersion {
 	cmd := exec.Command("kubectl", "version", "-o", "json")
 	out, err := cmd.CombinedOutput()
 	if err != nil {
+		glog.V(2).Infof("Failed to query kubectl: %s", err)
 		glog.V(2).Info(err)
 	}
 
@@ -324,6 +330,7 @@ func getKubeVersionFromKubelet() *KubeVersion {
 	out, err := cmd.CombinedOutput()
 
 	if err != nil {
+		glog.V(2).Infof("Failed to query kubelet: %s", err)
 		glog.V(2).Info(err)
 	}
 
@@ -331,7 +338,7 @@ func getKubeVersionFromKubelet() *KubeVersion {
 }
 
 func getVersionFromKubectlOutput(s string) *KubeVersion {
-	glog.V(2).Info(s)
+	glog.V(2).Infof("Kubectl output: %s", s)
 	type versionResult struct {
 		ServerVersion VersionResponse
 	}
@@ -354,7 +361,7 @@ func getVersionFromKubectlOutput(s string) *KubeVersion {
 }
 
 func getVersionFromKubeletOutput(s string) *KubeVersion {
-	glog.V(2).Info(s)
+	glog.V(2).Infof("Kubelet output: %s", s)
 	serverVersionRe := regexp.MustCompile(`Kubernetes v(\d+.\d+)`)
 	subs := serverVersionRe.FindStringSubmatch(s)
 	if len(subs) < 2 {
@@ -421,6 +428,12 @@ These program names are provided in the config.yaml, section '%s.%s.bins'
 }
 
 func getPlatformName() string {
+
+	openShiftVersion := getOpenShiftVersion()
+	if openShiftVersion != ""{
+		return openShiftVersion
+	}
+
 	kv, err := getKubeVersion()
 	if err != nil {
 		glog.V(2).Info(err)
@@ -439,11 +452,67 @@ func getPlatformNameFromVersion(s string) string {
 }
 
 func getPlatformBenchmarkVersion(platform string) string {
+	glog.V(3).Infof("getPlatformBenchmarkVersion platform: %s", platform)
 	switch platform {
 	case "eks":
 		return "eks-1.0"
 	case "gke":
 		return "gke-1.0"
+	case "ocp-3.10":
+		return "rh-0.7"
+	case "ocp-4.1":
+		return "rh-1.0"
 	}
 	return ""
 }
+
+func getOpenShiftVersion() string{
+	glog.V(1).Info("Checking for oc")
+	_, err := exec.LookPath("oc")
+
+	if err == nil {
+		cmd := exec.Command("oc", "version")
+		out, err := cmd.CombinedOutput()
+
+		if err == nil {
+			versionRe := regexp.MustCompile(`oc v(\d+\.\d+)`)
+			subs := versionRe.FindStringSubmatch(string(out))
+			if len(subs) < 1 {
+				versionRe = regexp.MustCompile(`Client Version:\s*(\d+\.\d+)`)
+				subs = versionRe.FindStringSubmatch(string(out))
+			}
+			if len(subs) > 1 {
+				glog.V(2).Infof("OCP output '%s' \nplatform is %s \nocp %v",string(out),getPlatformNameFromVersion(string(out)),subs[1])
+				ocpBenchmarkVersion, err := getOcpValidVersion(subs[1])
+				if err == nil{
+					return fmt.Sprintf("ocp-%s", ocpBenchmarkVersion)
+				} else {
+					glog.V(1).Infof("Can't get getOcpValidVersion: %v", err)
+				}
+			} else {
+				glog.V(1).Infof("Can't parse version output: %v", subs)
+			}
+		} else {
+			glog.V(1).Infof("Can't use oc command: %v", err)
+		}
+	} else {
+		glog.V(1).Infof("Can't find oc command: %v", err)
+	}
+	return ""
+}
+
+func getOcpValidVersion(ocpVer string) (string, error) {
+	ocpOriginal := ocpVer
+
+	for (!isEmpty(ocpVer)) {
+		glog.V(3).Info(fmt.Sprintf("getOcpBenchmarkVersion check for ocp: %q \n", ocpVer))
+		if ocpVer == "3.10" || ocpVer == "4.1"{
+			glog.V(1).Info(fmt.Sprintf("getOcpBenchmarkVersion found valid version for ocp: %q \n", ocpVer))
+			return ocpVer, nil
+		}
+		ocpVer = decrementVersion(ocpVer)		
+	}
+
+	glog.V(1).Info(fmt.Sprintf("getOcpBenchmarkVersion unable to find a match for: %q", ocpOriginal))
+	return "", fmt.Errorf("unable to find a matching Benchmark Version match for ocp version: %s", ocpOriginal)
+}

cmd/util_test.go

@@ -593,6 +593,20 @@ func Test_getPlatformBenchmarkVersion(t *testing.T) {
 			},
 			want: "",
 		},
+		{
+			name: "openshift3",
+			args: args{
+				platform: "ocp-3.10",
+			},
+			want: "rh-0.7",
+		},
+		{
+			name: "openshift4",
+			args: args{
+				platform: "ocp-4.1",
+			},
+			want: "rh-1.0",
+		},
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
@@ -602,3 +616,32 @@ func Test_getPlatformBenchmarkVersion(t *testing.T) {
 		})
 	}
 }
+
+func Test_getOcpValidVersion(t *testing.T) {
+
+	cases := []struct {
+		openShiftVersion string
+		succeed          bool
+		exp              string
+	}{
+		{openShiftVersion: "3.11", succeed: true, exp: "3.10"},
+		{openShiftVersion: "3.10", succeed: true, exp: "3.10"},
+		{openShiftVersion: "2.9", succeed: false, exp: ""},
+		{openShiftVersion: "4.1", succeed: true, exp: "4.1"},
+		{openShiftVersion: "4.5", succeed: true, exp: "4.1"},
+		{openShiftVersion: "4.6", succeed: true, exp: "4.1"},
+		{openShiftVersion: "invalid", succeed: false, exp: ""},
+	}
+	for _, c := range cases {
+		ocpVer,_ := getOcpValidVersion(c.openShiftVersion)
+		if c.succeed {
+			if c.exp != ocpVer {
+				t.Errorf("getOcpValidVersion(%q) - Got %q expected %s", c.openShiftVersion, ocpVer, c.exp)
+			}
+		} else {
+			if len(ocpVer) > 0 {
+				t.Errorf("getOcpValidVersion(%q) - Expected empty string but Got %s", c.openShiftVersion, ocpVer)
+			}
+		}
+	}
+}

docs/asff.md

@@ -15,7 +15,7 @@ You can configure kube-bench with the `--asff` to send findings to AWS Security
 
 ## Configure permissions in an IAM Role
 
-* Grant these permissions to the IAM Role that the kube-bench pod will be associated with. There are two potions:
+* Grant these permissions to the IAM Role that the kube-bench pod will be associated with. There are two options:
   * You can run the kube-bench pod under a specific [service account associated with an IAM role](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html) that has these permissions to write Security Hub findings.
   * Alternatively the pod can be granted permissions specified by the Role that your [EKS node group uses](https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html).
   
@@ -36,16 +36,14 @@ Here is an example IAM Policy that you can attach to your EKS node group's IAM R
 }
 ```
 
-## Configure and rebuild kube-bench
+### Modify the job configuration
 
-You will need to download, build and push the kube-bench container image to your ECR repo as described in Step 3 of the [EKS instructions][eks-instructions], except that before you build the container image, you need to edit `cfg/eks-1.0/config.yaml` to specify the AWS account, AWS region, and the EKS Cluster ARN.
+* Modify the kube-bench Configmap in `job-eks-asff.yaml` to specify the AWS account, AWS region, and the EKS Cluster ARN.
+* In the same file, modify the image specifed in the Job to use the kube-bench image pushed to your ECR
+* [Optional] - If you have created a dedicated IAM role to be used with kube-bench as described above in [Configure permissions in an IAM Role](#configure-permissions-in-an-iam-role), you will need to add the IAM role arn to the kube-bench ServiceAccount in `job-eks-asff.yaml`.
+* Make sure that `job-eks-asff.yaml` specifies the container image you just pushed to your ECR registry.
 
-## Modify the job configuration
-
-* Modify `job-eks.yaml` to specify the `--asff` flag, so that kube-bench writes output in ASFF format to Security Hub
-* Make sure that `job-eks.yaml` specifies the container image you just pushed to your ECR registry.
-
-You can now run kube-bench as a pod in your cluster: `kubectl apply -f job-eks.yaml`
+You can now run kube-bench as a pod in your cluster: `kubectl apply -f job-eks-asff.yaml`
 
 Findings will be generated for any kube-bench test that generates a `[FAIL]` or `[WARN]` output. If all tests pass, no findings will be generated. However, it's recommended that you consult the pod log output to check whether any findings were generated but could not be written to Security Hub.
 

integration/testdata/cis-1.5/job.data

@@ -353,7 +353,7 @@ UIDs not including 0.
 5.2.8 Ensure that allowedCapabilities is not present in PSPs for the cluster unless
 it is set to an empty array.
 
-5.2.9 Review the use of capabilites in applications runnning on your cluster. Where a namespace
+5.2.9 Review the use of capabilites in applications running on your cluster. Where a namespace
 contains applicaions which do not require any Linux capabities to operate consider adding
 a PSP which forbids the admission of containers which do not drop all capabilities.
 

integration/testdata/cis-1.6/job.data

@@ -356,7 +356,7 @@ UIDs not including 0.
 5.2.8 Ensure that allowedCapabilities is not present in PSPs for the cluster unless
 it is set to an empty array.
 
-5.2.9 Review the use of capabilites in applications runnning on your cluster. Where a namespace
+5.2.9 Review the use of capabilites in applications running on your cluster. Where a namespace
 contains applicaions which do not require any Linux capabities to operate consider adding
 a PSP which forbids the admission of containers which do not drop all capabilities.
 
@@ -416,4 +416,4 @@ resources and that all new resources are created in a specific namespace.
 72 checks PASS
 11 checks FAIL
 39 checks WARN
-0 checks INFO
+0 checks INFO

job-eks-asff.yaml

@@ -0,0 +1,67 @@
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: kube-bench
+  # If using a dedicated IAM role for kube-bench, uncomment the annotations
+  # block below and replace the ROLE_ARN
+  # annotations:
+  #   eks.amazonaws.com/role-arn: "<ROLE_ARN>"
+
+---
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: kube-bench-eks-config
+data:
+  config.yaml: |
+    AWS_ACCOUNT: "<AWS_ACCT_NUMBER>"
+    AWS_REGION: "<AWS_REGION>"
+    CLUSTER_ARN: "<AWS_CLUSTER_ARN>"
+
+---
+apiVersion: batch/v1
+kind: Job
+metadata:
+  name: kube-bench
+spec:
+  template:
+    spec:
+      hostPID: true
+      containers:
+        - name: kube-bench
+          # Push the image to your ECR and then refer to it here
+          image: <ID.dkr.ecr.region.amazonaws.com/aquasec/kube-bench:ref>
+          command: ["kube-bench", "node", "--benchmark", "eks-1.0", "--asff"]
+          volumeMounts:
+            - name: var-lib-kubelet
+              mountPath: /var/lib/kubelet
+              readOnly: true
+            - name: etc-systemd
+              mountPath: /etc/systemd
+              readOnly: true
+            - name: etc-kubernetes
+              mountPath: /etc/kubernetes
+              readOnly: true
+            - name: kube-bench-eks-config
+              mountPath: "/opt/kube-bench/cfg/eks-1.0/config.yaml"
+              subPath: config.yaml
+              readOnly: true
+      restartPolicy: Never
+      serviceAccountName: kube-bench
+      volumes:
+        - name: var-lib-kubelet
+          hostPath:
+            path: "/var/lib/kubelet"
+        - name: etc-systemd
+          hostPath:
+            path: "/etc/systemd"
+        - name: etc-kubernetes
+          hostPath:
+            path: "/etc/kubernetes"
+        - name: kube-bench-eks-config
+          configMap:
+            name: kube-bench-eks-config
+            items:
+              - key: config.yaml
+                path: config.yaml

job-eks.yaml

@@ -10,11 +10,9 @@ spec:
       containers:
         - name: kube-bench
           # Push the image to your ECR and then refer to it here
-          image: <ID.dkr.ecr.region.amazonaws.com/aquasec/kube-bench:ref>
-          # Use the --asff flag if you would like to send findings to AWS Security Hub
-          # Note that this requires you to rebuild a version of the kube-bench image
-          # after editing the cfg/eks-1.0/config.yaml with your account information
-          # command: ["kube-bench", "node", "--benchmark", "eks-1.0", "--asff"]
+          # image: <ID.dkr.ecr.region.amazonaws.com/aquasec/kube-bench:ref>
+          image: aquasec/kube-bench:latest
+          # To send findings to AWS Security Hub, refer to `job-eks-asff.yaml` instead
           command: ["kube-bench", "node", "--benchmark", "eks-1.0"]
           volumeMounts:
             - name: var-lib-kubelet

job-master.yaml

@@ -16,11 +16,29 @@ spec:
       containers:
         - name: kube-bench
           image: aquasec/kube-bench:latest
-          command: ["kube-bench", "master"]
+          command: ["kube-bench", "run", "--targets=master"]
           volumeMounts:
             - name: var-lib-etcd
               mountPath: /var/lib/etcd
               readOnly: true
+            - name: var-lib-kubelet
+              mountPath: /var/lib/kubelet
+              readOnly: true
+            - name: var-lib-kube-scheduler
+              mountPath: /var/lib/kube-scheduler
+              readOnly: true
+            - name: var-lib-kube-controller-manager
+              mountPath: /var/lib/kube-controller-manager
+              readOnly: true
+            - name: etc-systemd
+              mountPath: /etc/systemd
+              readOnly: true
+            - name: lib-systemd
+              mountPath: /lib/systemd/
+              readOnly: true
+            - name: srv-kubernetes
+              mountPath: /srv/kubernetes/
+              readOnly: true
             - name: etc-kubernetes
               mountPath: /etc/kubernetes
               readOnly: true
@@ -29,14 +47,44 @@ spec:
             - name: usr-bin
               mountPath: /usr/local/mount-from-host/bin
               readOnly: true
+            - name: etc-cni-netd
+              mountPath: /etc/cni/net.d/
+              readOnly: true
+            - name: opt-cni-bin
+              mountPath: /opt/cni/bin/
+              readOnly: true
       restartPolicy: Never
       volumes:
         - name: var-lib-etcd
           hostPath:
             path: "/var/lib/etcd"
+        - name: var-lib-kubelet
+          hostPath:
+            path: "/var/lib/kubelet"
+        - name: var-lib-kube-scheduler
+          hostPath:
+            path: "/var/lib/kube-scheduler"
+        - name: var-lib-kube-controller-manager
+          hostPath:
+            path: "/var/lib/kube-controller-manager"
+        - name: etc-systemd
+          hostPath:
+            path: "/etc/systemd"
+        - name: lib-systemd
+          hostPath:
+            path: "/lib/systemd"
+        - name: srv-kubernetes
+          hostPath:
+            path: "/srv/kubernetes"
         - name: etc-kubernetes
           hostPath:
             path: "/etc/kubernetes"
         - name: usr-bin
           hostPath:
             path: "/usr/bin"
+        - name: etc-cni-netd
+          hostPath:
+            path: "/etc/cni/net.d/"
+        - name: opt-cni-bin
+          hostPath:
+            path: "/opt/cni/bin/"

job-node.yaml

@@ -10,14 +10,29 @@ spec:
       containers:
         - name: kube-bench
           image: aquasec/kube-bench:latest
-          command: ["kube-bench", "node"]
+          command: ["kube-bench", "run", "--targets=node"]
           volumeMounts:
+            - name: var-lib-etcd
+              mountPath: /var/lib/etcd
+              readOnly: true
             - name: var-lib-kubelet
               mountPath: /var/lib/kubelet
               readOnly: true
+            - name: var-lib-kube-scheduler
+              mountPath: /var/lib/kube-scheduler
+              readOnly: true
+            - name: var-lib-kube-controller-manager
+              mountPath: /var/lib/kube-controller-manager
+              readOnly: true
             - name: etc-systemd
               mountPath: /etc/systemd
               readOnly: true
+            - name: lib-systemd
+              mountPath: /lib/systemd/
+              readOnly: true
+            - name: srv-kubernetes
+              mountPath: /srv/kubernetes/
+              readOnly: true
             - name: etc-kubernetes
               mountPath: /etc/kubernetes
               readOnly: true
@@ -26,17 +41,44 @@ spec:
             - name: usr-bin
               mountPath: /usr/local/mount-from-host/bin
               readOnly: true
+            - name: etc-cni-netd
+              mountPath: /etc/cni/net.d/
+              readOnly: true
+            - name: opt-cni-bin
+              mountPath: /opt/cni/bin/
+              readOnly: true
       restartPolicy: Never
       volumes:
+        - name: var-lib-etcd
+          hostPath:
+            path: "/var/lib/etcd"
         - name: var-lib-kubelet
           hostPath:
             path: "/var/lib/kubelet"
+        - name: var-lib-kube-scheduler
+          hostPath:
+            path: "/var/lib/kube-scheduler"
+        - name: var-lib-kube-controller-manager
+          hostPath:
+            path: "/var/lib/kube-controller-manager"
         - name: etc-systemd
           hostPath:
             path: "/etc/systemd"
+        - name: lib-systemd
+          hostPath:
+            path: "/lib/systemd"
+        - name: srv-kubernetes
+          hostPath:
+            path: "/srv/kubernetes"
         - name: etc-kubernetes
           hostPath:
             path: "/etc/kubernetes"
         - name: usr-bin
           hostPath:
             path: "/usr/bin"
+        - name: etc-cni-netd
+          hostPath:
+            path: "/etc/cni/net.d/"
+        - name: opt-cni-bin
+          hostPath:
+            path: "/opt/cni/bin/"

job.yaml

@@ -21,9 +21,21 @@ spec:
             - name: var-lib-kubelet
               mountPath: /var/lib/kubelet
               readOnly: true
+            - name: var-lib-kube-scheduler
+              mountPath: /var/lib/kube-scheduler
+              readOnly: true
+            - name: var-lib-kube-controller-manager
+              mountPath: /var/lib/kube-controller-manager
+              readOnly: true
             - name: etc-systemd
               mountPath: /etc/systemd
               readOnly: true
+            - name: lib-systemd
+              mountPath: /lib/systemd/
+              readOnly: true
+            - name: srv-kubernetes
+              mountPath: /srv/kubernetes/
+              readOnly: true
             - name: etc-kubernetes
               mountPath: /etc/kubernetes
               readOnly: true
@@ -32,6 +44,12 @@ spec:
             - name: usr-bin
               mountPath: /usr/local/mount-from-host/bin
               readOnly: true
+            - name: etc-cni-netd
+              mountPath: /etc/cni/net.d/
+              readOnly: true
+            - name: opt-cni-bin
+              mountPath: /opt/cni/bin/
+              readOnly: true
       restartPolicy: Never
       volumes:
         - name: var-lib-etcd
@@ -40,12 +58,30 @@ spec:
         - name: var-lib-kubelet
           hostPath:
             path: "/var/lib/kubelet"
+        - name: var-lib-kube-scheduler
+          hostPath:
+            path: "/var/lib/kube-scheduler"
+        - name: var-lib-kube-controller-manager
+          hostPath:
+            path: "/var/lib/kube-controller-manager"
         - name: etc-systemd
           hostPath:
             path: "/etc/systemd"
+        - name: lib-systemd
+          hostPath:
+            path: "/lib/systemd"
+        - name: srv-kubernetes
+          hostPath:
+            path: "/srv/kubernetes"
         - name: etc-kubernetes
           hostPath:
             path: "/etc/kubernetes"
         - name: usr-bin
           hostPath:
             path: "/usr/bin"
+        - name: etc-cni-netd
+          hostPath:
+            path: "/etc/cni/net.d/"
+        - name: opt-cni-bin
+          hostPath:
+            path: "/opt/cni/bin/"