* Add CLUSTER_RESOURCES_INGRESS_CLASS constant
* Collect IngressClass resources in cluster resources
* Add IngressClass analyzer API type
* Regenerate deepcopy for IngressClass type
* Update client-gen output from make generate
* Add IngressClass analyzer tests
* Implement IngressClass analyzer
* Register IngressClass analyzer in dispatcher
* Restore v1beta3 import in clientset scheme registration
The v1beta3 import was accidentally removed during client-gen
regeneration, causing a compile error since the SchemeBuilder
still references troubleshootv1beta3.AddToScheme.
Change to stop re-analysing preflight results when uploadResultsTo is present leading to duplicate results
Signed-off-by: Evans Mungai <evans@replicated.com>
* feat: node metrics analyser
The analyser only checks PVC usage at the moment. More analysers
can be added on a need to have basis
* Add tests
* Fix flaky test by waiting for goldpinger pods to start
* Fix how outcomes get checked
* Fix catch all outcome condition
* Fix test
* feat: node metrics analyser
The analyser only checks PVC usage at the moment. More analysers
can be added on a need to have basis
* Add tests
* Fix flaky test by waiting for goldpinger pods to start
* Fix how outcomes get checked
* Fix catch all outcome condition
* Fix test
* Regenerate schemas
* Fix failing test
---------
Co-authored-by: Dexter Yan <yanshaocong@gmail.com>
* feat: goldpinger analyser
Analyser to generate a report from goldpinger results
* Add goldpinger testdata
* Goldpinger collector
* Improvements after running tests
* More minor updates after further testing
* Better error message if a container fails to start
* A few more updates
* Add goldpinger e2e test
* Update schemas
* Clean up help installs in e2e tests
* Add resource limits to goldpinger pods
* Some minor improvements
* Some more changes noted when writing docs
* Update schemas
* A few more updates when testing with kURL
* Log goldpinger tests
* Tests before exit code
* feat: add velero analyzer (#806)
* updated schema
* analyzer without collector
* tests
* covers deprecated Restic repository type
* velero version from deployment image to check deprecated type
* read for both velero pod kinds (velero*, node-agent*)
---------
Signed-off-by: Archit Sharma <archit@pm.me>
* feat: Add regular expressions host anaylser
This anaylser is the same as the in-cluster text anaylser. You pass in
search expressions to find values in files collected in a bundle
* additional test assertion to check analyser warn
When running a support bundle, we want to know how long each operation
(collect, redact, analyze) takes. This commit adds a new trace exporter
that records the start and end times of each operation, and then prints
a summary of the execution. The summary is also stored in the support
bundle.
Related to #923
Have all in-cluster analysers implement the same interface. This
will help with the implementation of code that requires making
calls to all analysers
Fixes#995
* First draft of a generic cluster-resource analyzer
* Add more resource mappings
* Support some cluster-scoped resources
the structure of this could probably be a bit tidyer, but this now
allows us to target non-namespaced resources simply by not specifying
the namespace in the analyzer.
* General tidy up
* pull resource selection into it's own function
* remove pointless pointer to string
* Export findResource function
This lets other analyzers use it.
* Add tests for cluster resources analyzer
* Update schemas
* Address some of @banjoh's comments
* rework resource selection
thanks @banjoh
* Replace FindFiles with GetFile
Since we already know where we're looking for files,
it doesn't make sense to have to loop over a single item slice.
* Use assert instead of require
* format
* Change default behaviour for no namespace
Now not providing a namespace causes us to default to "default", with an
explicit bool to toggle cluster-scoped resource checking.
This should feel somewhat more intuitive when writing analyzers that use
this function
* Generate schemas
* Value → expectedValue
* feat(analyze): add ExcludeFiles field to textAnazlye
* feat(analyze): fix test for getFiles
* feat(analyze): change function name to excludeFilePaths
* feat(analyze): fix preflight test fail
* feat(analyze): add tests for excludeFiles
* feat(schemas): run make schemas
* feat(analyze): use getChildCollectedFileContents function prototype
* feat(analyze): reduce time complexity
* feat(longhorn): add getFileContents as getCollectedFileContents
* add strict flag to Analyzer/AnalyzerMeta
and regenerate schemas and controller-gen code
* map analyzer strict to result
* Update stdout for human and json format
* fix review comment
* update interactive result
* update interactive results
* Update types.go
* Update upload_results.go
* print strict when only true
* Add collect command and remote host collectors
Adds the ability to run a host collector on a set of remote k8s nodes.
Target nodes can be filtered using the --selector flag, with the same
syntax as kubectl. Existing flags for --collector-image,
--collector-pullpolicy and --request-timeout are used. To run on a
specified node, --selector="kubernetes.io/hostname=kind-worker2" could
be used.
The collect command is used by the remote collector to output the
results using a "raw" format, which uses the filename as the key, and
the value the output as a escaped json string. When run manually it
defaults to fully decoded json. The existing block devices,
ipv4interfaces and services host collectors don't decode properly - the
fix is to convert their slice output to a map (fix not included as
unsure what depends on the existing format).
The collect command is also useful for troubleshooting preflight issues.
Examples are included to show remote collector usage.
```
bin/collect --collector-image=croomes/troubleshoot:latest examples/collect/remote/memory.yaml --namespace test
{
"kind-control-plane": {
"system/memory.json": {
"total": 1304207360
}
},
"kind-worker": {
"system/memory.json": {
"total": 1695780864
}
},
"kind-worker2": {
"system/memory.json": {
"total": 1726353408
}
}
}
```
The preflight command has been updated to run remote collectors. To run
a host collector remotely it must be specified in the spec as a
`remoteCollector`:
```
apiVersion: troubleshoot.sh/v1beta2
kind: HostPreflight
metadata:
name: memory
spec:
remoteCollectors:
- memory:
collectorName: memory
analyzers:
- memory:
outcomes:
- fail:
when: "< 8Gi"
message: At least 8Gi of memory is required
- warn:
when: "< 32Gi"
message: At least 32Gi of memory is recommended
- pass:
message: The system has as sufficient memory
```
Results for each node are analyzed separately, with the node name
appended to the title:
```
bin/preflight --interactive=false --collector-image=croomes/troubleshoot:latest examples/preflight/remote/memory.yaml --format=json
{memory running 0 1}
{memory completed 1 1}
{
"fail": [
{
"title": "Amount of Memory (kind-worker2)",
"message": "At least 8Gi of memory is required"
},
{
"title": "Amount of Memory (kind-worker)",
"message": "At least 8Gi of memory is required"
},
{
"title": "Amount of Memory (kind-control-plane)",
"message": "At least 8Gi of memory is required"
}
]
}
```
Also added a host collector to allow preflight checks of required kernel
modules, which is the main driver for this change.
The IPAM pool analyzer checks that utilization of the pod IP subnet is
less than 85%. For example, if using 10.32.0.0/12, this analyzer will
warn if 3,482 IPs are currently allocated to pods.
The pending allocation analyzer checks that the IPAM status in the
report has no items for the PendingAllocates field. This indicates the
IPAM service is not ready according to the code in the weave status
template
https://github.com/weaveworks/weave/blob/e3712152d2a0fe3bc998964c948e45bdf8ff6144/prog/weaver/http.go#L186.
The weave connections analyzer checks that all connections to remote
peers are in the established state. The state will be "pending" if UDP
is blocked between nodes and will be "failed" if the weave pod on the
remote node is in a crash loop. To force a pending state for testing,
run the commands `iptables -A INPUT -p udp --dport 6784 -j REJECT` and
`iptables -A INPUT -p udp --dport 6783 -j REJECT` on a peer.
The weave connections analyzer also checks that all connections are
using the fastdp protocol. A commopn issue seen in the field on
CentOS/RHEL 7 is that some sides of a connection are using fastdp and
other sides have fallen back to sleeve. Set the WEAVE_NO_FASTDP env var
on the weave daemonset to "true" to test this analyzer.
The included example found P99 latency of 2.6ms.
Fio reported P99 latency of 2.5ms with this command:
fio --rw=write --ioengine=sync --fdatasync=1 --directory=/var/lib/etcd
--size=220m --bs=2300
Andrew Lavery
ada mancini
Diamon Wiggins
Gerard Nguyen
Evans Mungai
Archit Sharma
danj-replicated
divolgin
Craig O'Donnell
Ethan Mosbaugh
divolgin
Salah Aldeen Al Saleh
Simon Croome
Andrew Reed
John Murphy
Andrew Reed