Commit Graph

24 Commits

Author SHA1 Message Date
Simon Croome
977fc438ea Remote host collectors (#392)
* Add collect command and remote host collectors

Adds the ability to run a host collector on a set of remote k8s nodes.
Target nodes can be filtered using the --selector flag, with the same
syntax as kubectl.  Existing flags for --collector-image,
--collector-pullpolicy and --request-timeout are used.  To run on a
specified node, --selector="kubernetes.io/hostname=kind-worker2" could
be used.

The collect command is used by the remote collector to output the
results using a "raw" format, which uses the filename as the key, and
the value the output as a escaped json string.  When run manually it
defaults to fully decoded json. The existing block devices,
ipv4interfaces and services host collectors don't decode properly - the
fix is to convert their slice output to a map (fix not included as
unsure what depends on the existing format).

The collect command is also useful for troubleshooting preflight issues.

Examples are included to show remote collector usage.

```
bin/collect --collector-image=croomes/troubleshoot:latest  examples/collect/remote/memory.yaml --namespace test
{
  "kind-control-plane": {
    "system/memory.json": {
      "total": 1304207360
    }
  },
  "kind-worker": {
    "system/memory.json": {
      "total": 1695780864
    }
  },
  "kind-worker2": {
    "system/memory.json": {
      "total": 1726353408
    }
  }
}
```

The preflight command has been updated to run remote collectors.  To run
a host collector remotely it must be specified in the spec as a
`remoteCollector`:

```
apiVersion: troubleshoot.sh/v1beta2
kind: HostPreflight
metadata:
  name: memory
spec:
  remoteCollectors:
    - memory:
        collectorName: memory
  analyzers:
    - memory:
        outcomes:
          - fail:
              when: "< 8Gi"
              message: At least 8Gi of memory is required
          - warn:
              when: "< 32Gi"
              message: At least 32Gi of memory is recommended
          - pass:
              message: The system has as sufficient memory
```

Results for each node are analyzed separately, with the node name
appended to the title:

```
bin/preflight --interactive=false --collector-image=croomes/troubleshoot:latest examples/preflight/remote/memory.yaml --format=json
{memory running 0 1}
{memory completed 1 1}
{
  "fail": [
    {
      "title": "Amount of Memory (kind-worker2)",
      "message": "At least 8Gi of memory is required"
    },
    {
      "title": "Amount of Memory (kind-worker)",
      "message": "At least 8Gi of memory is required"
    },
    {
      "title": "Amount of Memory (kind-control-plane)",
      "message": "At least 8Gi of memory is required"
    }
  ]
}
```

Also added a host collector to allow preflight checks of required kernel
modules, which is the main driver for this change.
2021-10-06 09:03:53 -05:00
John Murphy
e0f6cab5b3 Fix removes control characters from non interactive preflight runs (#394) 2021-07-23 09:46:36 -05:00
emosbaugh
8dcfa9886d Copy from host collector (#391)
* Copy from host collector

* namespace improvements

* better support for multiple nodes
2021-07-22 12:25:59 -07:00
emosbaugh
39350b5722 ConfigMap collector and secrets can be collected by selectors (#384)
* ConfigMap collector and secrets can be collected by selectors

* follow docs

* Pass context and kubernetes client to collectors

* collect tests

* analyze tests

* fix tests

* improvements
2021-07-08 16:30:26 -07:00
Ethan Mosbaugh
9357d5ac96 Include result if not nil regardless of error 2021-04-28 02:58:59 +00:00
divolgin
62afc87af8 Add progress percentage 2021-03-18 22:29:27 +00:00
divolgin
5f2525b663 Report back some basic progress 2021-03-18 18:56:27 +00:00
Ethan Mosbaugh
4b78c430ca Host preflight ux improvements 2021-03-02 17:27:01 +00:00
Ethan Mosbaugh
09d16ff185 Host preflights exclude 2021-03-01 22:45:16 +00:00
Ethan Mosbaugh
d6acd6d906 Conditional Analyzers 2021-02-26 04:33:11 +00:00
Andrew Reed
10a34c2e58 Host preflight (#311)
* Add HostPreflight v1beta2

* Work on TCP Load Balancer

* Host disk usage collector and analyzer

* Host memory analyzer

* TCP port status

* TCP load balancer

* Review changes

Co-authored-by: Marc Campbell <marc.e.campbell@gmail.com>
2021-02-08 16:09:01 -05:00
Andrew Lavery
0126fb7d7a Merge remote-tracking branch 'origin/master' into laverya/allow-analyzing-all-files-in-dir 2020-09-08 14:27:26 -04:00
divolgin
a0ce85ae1e Adding troubleshoot.sh/v1beta2 2020-09-01 19:57:11 +00:00
Marc Campbell
65f957db81 Refactor to support K8s 1.18 2020-06-12 09:28:49 -07:00
Andrew Lavery
d016e3269c add global and per-collector redactors
add redact type, and begin wiring global redactors

use per-collector redactors

add a test of the 'data' collector and redaction

handle literal string replacements

remove redundant types and redact calls

add proper redactor type, foundations of global redactors

accept global redactors from the CLI, include sample redaction spec
2020-04-16 14:03:00 -04:00
Andrew Lavery
2bc9938a2a use filepath.Glob and filepath.Match 2020-04-15 18:59:17 -04:00
Andrew Lavery
5050f46b1e use gobwas/glob to match globs 2020-04-15 18:47:16 -04:00
Andrew Lavery
222a93162d allow textAnalyze to run on all files in a dir or matching a prefix 2020-04-15 18:08:08 -04:00
Andrew Lavery
527fe16bcc reduce marshal/unmarshal passes by returning map[string][]byte
WIP
2020-04-14 16:37:51 -04:00
Marc Campbell
983aaaacea Defining types in a package 2020-03-10 17:32:16 +00:00
Marc Campbell
e74101070d Refactor 2020-03-10 01:07:57 +00:00
Marc Campbell
ed0ad7ef01 Image pull secrets 2019-07-22 19:20:42 +00:00
Marc Campbell
df4edcb80d Running collectors without the CRD 2019-07-18 02:46:42 +00:00
Marc Campbell
4b68be509f Run preflight from CLI 2019-07-17 14:48:36 +00:00