When running a support bundle, we want to know how long each operation
(collect, redact, analyze) takes. This commit adds a new trace exporter
that records the start and end times of each operation, and then prints
a summary of the execution. The summary is also stored in the support
bundle.
Related to #923
introduces a new option to limit the size of a pod log when added to the bundle. This will make sure the support bundle will not grow to an unacceptable size and thus might contain information that is too old.
The maximum size of a pod log in a bundle is set by default to 5MB, and can be changed if we decide upon the need.
BREAKING CHANGE: any logs that are collected by the logs collector are now limited by default to 5MB unless a different size limit is specified. Folks expecting log files larger than that to be collected without truncation will need to adjust their support bundle specs.
Fixes: #878
* feat(redactors): Run redactors on an existing support bundle
Add redact subcommand to support-bundle to allow running redactors on an
existing bundle to creating a new redacted bundle.
The command will be launched like so
support-bundle redact <redactor urls> --bundle support-bundle.tar.gz
Fixes: #705
* filter on cpu architecture
* filter by cpu architecture
* fail if we dont have a label match too
* add tests for cpu arch filter
* update for make schemas
For a postgres collector spec targeting a server configured to accept
(m)TLS connections we need to pass in the necessary parameters in order
to successfully connect to the server. Both preflight and support bundle
specs use this collector.
This change allows us to pass in the necessary TLS parameters via inlined
TLS configuration or via a secret reference.
Fixes#747
feat(collectors): Add mTLS parameters to the redis collector
For a redis collector spec targeting a redis server configured to accept
(m)TLS connections we need to pass in the necessary TLS parameters in order
to successfully connect to the server. Both preflight and support bundle
specs use this collector.
This change allows us to pass in the necessary TLS parameters via inlined
TLS configuration or via a secret reference.
Fixes#746
* feat(collectors): Store all pod logs in cluster-resources directory
All pod logs collected by the logs collector will now be stored in
/cluster-resources/pods/logs/[namespace]/[pod]/[container].log. This
will provide consistency and allow sbctl to find the logs when we run
`kubectl logs <pod>`. To allow backwards compatibility, symlinks of the
log files will be created in the current expected locations.
Closes: #744
## Description:
Currently, when we install an k8s with kurl and we run kubectl preflight https://preflight.replicated.com it will fail:
> ------------
> Check FAIL
> Title: Ingress
> Message: Contour ingress not found!
Therefore, Contour ingress does not seems a pre-requirement to kURL. So, should we have this check in the default example/test?
* collector/analyzer for host operating system
* address cr comments
* cleanup
* fix invoking the analyzer
code cleanup
* fix cr comments
* add corner case unit-test
* fix kernel version parsing
* address review comments
* add default case
* parse using regex
* added more testcases and fixed the bug found in cr
* few small things
* Add collect command and remote host collectors
Adds the ability to run a host collector on a set of remote k8s nodes.
Target nodes can be filtered using the --selector flag, with the same
syntax as kubectl. Existing flags for --collector-image,
--collector-pullpolicy and --request-timeout are used. To run on a
specified node, --selector="kubernetes.io/hostname=kind-worker2" could
be used.
The collect command is used by the remote collector to output the
results using a "raw" format, which uses the filename as the key, and
the value the output as a escaped json string. When run manually it
defaults to fully decoded json. The existing block devices,
ipv4interfaces and services host collectors don't decode properly - the
fix is to convert their slice output to a map (fix not included as
unsure what depends on the existing format).
The collect command is also useful for troubleshooting preflight issues.
Examples are included to show remote collector usage.
```
bin/collect --collector-image=croomes/troubleshoot:latest examples/collect/remote/memory.yaml --namespace test
{
"kind-control-plane": {
"system/memory.json": {
"total": 1304207360
}
},
"kind-worker": {
"system/memory.json": {
"total": 1695780864
}
},
"kind-worker2": {
"system/memory.json": {
"total": 1726353408
}
}
}
```
The preflight command has been updated to run remote collectors. To run
a host collector remotely it must be specified in the spec as a
`remoteCollector`:
```
apiVersion: troubleshoot.sh/v1beta2
kind: HostPreflight
metadata:
name: memory
spec:
remoteCollectors:
- memory:
collectorName: memory
analyzers:
- memory:
outcomes:
- fail:
when: "< 8Gi"
message: At least 8Gi of memory is required
- warn:
when: "< 32Gi"
message: At least 32Gi of memory is recommended
- pass:
message: The system has as sufficient memory
```
Results for each node are analyzed separately, with the node name
appended to the title:
```
bin/preflight --interactive=false --collector-image=croomes/troubleshoot:latest examples/preflight/remote/memory.yaml --format=json
{memory running 0 1}
{memory completed 1 1}
{
"fail": [
{
"title": "Amount of Memory (kind-worker2)",
"message": "At least 8Gi of memory is required"
},
{
"title": "Amount of Memory (kind-worker)",
"message": "At least 8Gi of memory is required"
},
{
"title": "Amount of Memory (kind-control-plane)",
"message": "At least 8Gi of memory is required"
}
]
}
```
Also added a host collector to allow preflight checks of required kernel
modules, which is the main driver for this change.
The IPAM pool analyzer checks that utilization of the pod IP subnet is
less than 85%. For example, if using 10.32.0.0/12, this analyzer will
warn if 3,482 IPs are currently allocated to pods.
The pending allocation analyzer checks that the IPAM status in the
report has no items for the PendingAllocates field. This indicates the
IPAM service is not ready according to the code in the weave status
template
e3712152d2/prog/weaver/http.go (L186).
The weave connections analyzer checks that all connections to remote
peers are in the established state. The state will be "pending" if UDP
is blocked between nodes and will be "failed" if the weave pod on the
remote node is in a crash loop. To force a pending state for testing,
run the commands `iptables -A INPUT -p udp --dport 6784 -j REJECT` and
`iptables -A INPUT -p udp --dport 6783 -j REJECT` on a peer.
The weave connections analyzer also checks that all connections are
using the fastdp protocol. A commopn issue seen in the field on
CentOS/RHEL 7 is that some sides of a connection are using fastdp and
other sides have fallen back to sleeve. Set the WEAVE_NO_FASTDP env var
on the weave daemonset to "true" to test this analyzer.
Fixed bug caused by host preflights not handling empty when clauses, this cropped up because we now handle multiple host preflight results. Also expanded test coverage and added integration test script.
Add a background IOPS feature to the filesystem performance collector
that specifies separate read and write background IOPS to perform while
measuring latency. This allows for better assessment of whether etcd
will be stable when running alongside other workloads on the same
cluster.
Also add templating to the outcome message of the filesystem performance
analyzers to allow printing individual latency percentiles or the entire
table.
Remove the random IOPS benchmark since it was attempting to perform
unaligned direct I/O.
The random IOPS benchmark attempts to replicate the results of this
fio command:
fio --ioengine=psync --direct=1 --bs=4k --size=1Gi --readwrite=randread --serialize_overlap=1
Across three tests the fio command reported 1877 IOPS and the preflight
1822 IOPS with the same block and file size.
The included example found P99 latency of 2.6ms.
Fio reported P99 latency of 2.5ms with this command:
fio --rw=write --ioengine=sync --fdatasync=1 --directory=/var/lib/etcd
--size=220m --bs=2300
Add all supported analyzers to host preflight sample.
Don't log transient errors waiting for TCP connection.
Begin human stdout results on new line after spinner.
