* feat: add loader APIs to load specs from a list of yaml docs
The change introduces a loader package that will contain loader
public APIs. The aim of these APIs will be to, given any source of
troubleshoot specs, the loaders will fetch the specs and parse out
all troubleshoot objects that can be extracted.
* Some refactoring
* Some more changes
* More changes caught when testing vendor portal
* Add tests and rename Troubleshoot kinds struct
* Additional test
* Handle ConfigMap and Secrets with multiple specs in them
* Fix failing test
* Revert multidoc split implementation
* Fix merge conflict
* Change LoadFromXXX functions to a single LoadSpecs function
* ignore some more stuff
* watchrsync make command like kurl project
* document watchrsync
* the actual watchrsync script
* dont need these for npm dependencies
* Revert "dont need these for npm dependencies"
This reverts commit bdb4e62c38.
* install gaze-run-interrupt with make
* watchrsync instructions
* similar to the textAnalyze analyzer, adds support to evaluate the value at the specified yamlPath with regex or regexGroups
* fixes a typo in the shorthand name used for searching for cluster resource PVC objects
* we can now read preflight specs out of secrets, either from stdin or file input
* moved spec read logic out into its own function so it can be unit
tested easier
* added more comprehensive unit testing on the different ways we can read in specs
* only process when conditional if specified
* adding tests for cluster pod status analyzer
* use klog instead of fmt for logging
* add additional tests for warn and more operators
---------
Co-authored-by: Camila Macedo <7708031+camilamacedo86@users.noreply.github.com>
* Boiler plate for host copy collector
* feat: Add copy host collector
* Add tests
* No need to handle symlinks in a special way
System libraries (os.ReadAll, os.ReadDir) already handle symlinks
* fix: Discover specs from namespaces user is allowed
If a user has limited access to read secrets and config maps
from certain namespaces in a cluster, we'd need to gracefully
fail when forbidden errors are caught. We'll log them and continue
searching for specs in other namespaces.
The fetch-depth parameter to the checkout action will provide a full
checkout instead of a shallow one that only has the last commit. Using
this makes the workflow files a bit more concise and removes an extra
step.
* Add arm to GOARCHes in .goreleaser.yaml and exclude the windows/arm
build
* Add arm/linux to goreleaser-test GitHub workflow job
* Add the linux/arm binaries to krew plugin manifests
Currently, there's no debug logs whatsoever when running a command like
troubleshoot --debug --interactive=false
Tackle this by printing similar log statements as presented in
interactive mode to the debug logger. Refactor the code a bit so there's
no need for a dedicated finished channel and to exit the goroutines
properly.
Co-authored-by: Evans Mungai <evans@replicated.com>
The which binary is used to detect if client-gen is installed, and if
it's not, the Makefile will install it. The initial detection prints
an error if it's not found. This is misleading, as it is actually an
expected situation.
When running a support bundle, we want to know how long each operation
(collect, redact, analyze) takes. This commit adds a new trace exporter
that records the start and end times of each operation, and then prints
a summary of the execution. The summary is also stored in the support
bundle.
Related to #923
Have all in-cluster analysers implement the same interface. This
will help with the implementation of code that requires making
calls to all analysers
Fixes#995
* adding dedup for in cluster collectors
* add tests
* return collector as is whenever marshalling to json fails
---------
Co-authored-by: Evans Mungai <evans@replicated.com>
* adding test coverage for preflight.RunPreflights()
TDD to work on https://github.com/replicatedhq/troubleshoot/issues/906
and verify the fix is successful
* go.mod/go.sum: removing gnomock stuff since it's not in use (yet)
* Makefile: try running the preflight integration test with the e2e tests,
since there's a K3s instance in place already
* Makefile add a dedicated test-integration task, which runs as it's own
github action job
* Makefile: exclude a few things from test-integration that break the
github action job
* WIP on preflight tests, addressing some of @banjoh's feedback, more to
go though (specifically changing over to using assert)
* preflight tests: use the testify libraries, restructure code to be
formatted more like other tests in this project
introduces a new option to limit the size of a pod log when added to the bundle. This will make sure the support bundle will not grow to an unacceptable size and thus might contain information that is too old.
The maximum size of a pod log in a bundle is set by default to 5MB, and can be changed if we decide upon the need.
BREAKING CHANGE: any logs that are collected by the logs collector are now limited by default to 5MB unless a different size limit is specified. Folks expecting log files larger than that to be collected without truncation will need to adjust their support bundle specs.
Fixes: #878
Allow collecting of CPU and memory diagnostics when running troubleshoot CLI applications using --memprofile and --cpuprofile flags. These flags accept file paths if where to store the collected runtime data
* feat(redactors): Run redactors on an existing support bundle
Add redact subcommand to support-bundle to allow running redactors on an
existing bundle to creating a new redacted bundle.
The command will be launched like so
support-bundle redact <redactor urls> --bundle support-bundle.tar.gz
Fixes: #705
* github build-test-deploy workflow: split goreleaser-test into 3 actions,
one for each ID/component (sharing a single config file)
rationale: speeding up github actions, goreleaser-test was taking
12-14mins and is the outlier longest to execute right now on this repo
* Revert "gitlab build-test-deploy workflow: split goreleaser-test into 3 actions,"
This reverts commit 61083dbbb8.
* github build-test-deploy workflow: split into 6 actions, divided by
GOARCH/GOOS
rationale: improve performance of checks on PR's
* .goreleaser.yaml - add a breadcrumb to ensure we keep the action jobs
list aligned
* filter on cpu architecture
* filter by cpu architecture
* fail if we dont have a label match too
* add tests for cpu arch filter
* update for make schemas
* First draft of a generic cluster-resource analyzer
* Add more resource mappings
* Support some cluster-scoped resources
the structure of this could probably be a bit tidyer, but this now
allows us to target non-namespaced resources simply by not specifying
the namespace in the analyzer.
* General tidy up
* pull resource selection into it's own function
* remove pointless pointer to string
* Export findResource function
This lets other analyzers use it.
* Add tests for cluster resources analyzer
* Update schemas
* Address some of @banjoh's comments
* rework resource selection
thanks @banjoh
* Replace FindFiles with GetFile
Since we already know where we're looking for files,
it doesn't make sense to have to loop over a single item slice.
* Use assert instead of require
* format
* Change default behaviour for no namespace
Now not providing a namespace causes us to default to "default", with an
explicit bool to toggle cluster-scoped resource checking.
This should feel somewhat more intuitive when writing analyzers that use
this function
* Generate schemas
* Value → expectedValue
To keep both the Support Bundle and Preflight CLIs similar, this PR adds the ability for the Preflight binary to allow multiple specs be provided as CLI args and for them all to be run.
The symlinking logs feature led to a regression where symlinks of
unhealthy pods were overwritting logs in the support bundle. This
fix allows the cluster resources collector to instruct the logs
collector not to symlink logs, which in turn ensures logs are not
overwritten.
Fixes: #908
* add dedup for cluster resources collector
* restructure both collect.go in both pkg/supportbundle and pkg/preflight to be more similar for eventual refactor
For a postgres collector spec targeting a server configured to accept
(m)TLS connections we need to pass in the necessary parameters in order
to successfully connect to the server. Both preflight and support bundle
specs use this collector.
This change allows us to pass in the necessary TLS parameters via inlined
TLS configuration or via a secret reference.
Fixes#747
This change removes the IPv4 address redaction which previously ran by default on all
support bundle collections.
Folks that want to redact IPv4 addresses will need to add that redactor manually to their redactor specs.
feat(collectors): Add mTLS parameters to the redis collector
For a redis collector spec targeting a redis server configured to accept
(m)TLS connections we need to pass in the necessary TLS parameters in order
to successfully connect to the server. Both preflight and support bundle
specs use this collector.
This change allows us to pass in the necessary TLS parameters via inlined
TLS configuration or via a secret reference.
Fixes#746
TL;DR
-----
Updates Sysctl collector and analyzer for virtual memory parameters
Details
-------
Adds supoort for virtual memory parameters to the Sysctl collector and
analyzers. I uncovered this writing a pre-flight for a Helm chart that
includes ECK as a subchart. Since ECK requires a specific minimum value
for `vm.max_map_count` I wanted to use the Sysctl analyzer to check for
the expected value, but wasn't able to because of the limited values it
collected. I also learned that Sonarqube expects the same parameter to
be increased, so it seemed like a general enough requirement to add it
in.
The code updates the collector to collect values under `/proc/sys/vm`
and adds tests to the analyzer to based on the ECK requirements. Making
the tests pass required adding operators to the when expression, since
the existing code only allowed for `=`, `==`, and `===`. The when
expression now supports `>`, `<`, `>=`, and `<=`.
All tests pass.
* feat(analyze): add ExcludeFiles field to textAnazlye
* feat(analyze): fix test for getFiles
* feat(analyze): change function name to excludeFilePaths
* feat(analyze): fix preflight test fail
* feat(analyze): add tests for excludeFiles
* feat(schemas): run make schemas
* feat(analyze): use getChildCollectedFileContents function prototype
* feat(analyze): reduce time complexity
* feat(longhorn): add getFileContents as getCollectedFileContents
* feat(collectors): Store all pod logs in cluster-resources directory
All pod logs collected by the logs collector will now be stored in
/cluster-resources/pods/logs/[namespace]/[pod]/[container].log. This
will provide consistency and allow sbctl to find the logs when we run
`kubectl logs <pod>`. To allow backwards compatibility, symlinks of the
log files will be created in the current expected locations.
Closes: #744
* fix(flag): fix wrong output filename
* fix(flag): add reset flag function
* fix(flag): add output flag test cases
* fix(flag): move resetFlags function into private go test
* fix(flag): restructure flag tests with testify
* fix(flag): remove resetFlags function
* fix(flag): remove duplicated test and rewrite test names
## Description:
Currently, when we install an k8s with kurl and we run kubectl preflight https://preflight.replicated.com it will fail:
> ------------
> Check FAIL
> Title: Ingress
> Message: Contour ingress not found!
Therefore, Contour ingress does not seems a pre-requirement to kURL. So, should we have this check in the default example/test?
This change ensures that the clusterResources collector runs prior to any others
in order to not collect info on pods that collectors run during collection.
Additionally centralizes functions that are common to all collection to make future
maintenance simpler.
Fixes: #767
Due to deprecation of the API at `policy/v1beta1` for `PodDisruptionBudgets` and `batch/v1beta1` for `CronJobs`, updated cluster_resources.go to accommodate using either apiVersion v1 & v1beta1
Add --follow-uri flag to support-bundle to ignore the uri field when set in a spec
Implement the new CLI flag in `root.go` so we can pass it if we need to ignore the `uri` field in a spec. This also serves as a minimal documentation effort when running `support-bundle --help`.
Fixes: #695
When troubleshooting an issue we want to see and check the `PodDisruptionBudget` for any application. As we currently do not have that 'standard' output in our support bundle, this PR serves as an implementation of providing one with an overview of `PodDisruptionBudget` on a per-namespace basis.
After creating a support bundle, the output can be reviewed by checking out `../cluster-resources/pdb/*.json`
- [Release notes](https://github.com/google/periph/releases)
updated-dependencies:
- dependency-name: periph.io/x/periph
dependency-type: direct:production
update-type: version-update:semver-minor
Module periph.io/x/periph has moved from 3.6.8 to re-arrange locations
in 3.7.2. This reworks to take advantage of the new format.
* Add Mysql variables to collector
* Cleanup row scanning and a few updates based on feedback
* Close db connection
* Move defer db.close
* Updates based on feedback
* Use vars in loop instead of struct
* Only pull parameters specified in collector config
Co-authored-by: Ethan Mosbaugh <ethan@replicated.com>
* fix: return true when one of analyzers is true
* fix: return true when one of analyzers is true
* update: add unit test
* update: log errors while processing analyzers
* fix: return parse error instead of logging
* fix: update err message
* fix: evaluate strict check err separately
* add strict flag to Analyzer/AnalyzerMeta
and regenerate schemas and controller-gen code
* map analyzer strict to result
* Update stdout for human and json format
* fix review comment
* update interactive result
* update interactive results
* Update types.go
* Update upload_results.go
* print strict when only true
* collector/analyzer for host operating system
* address cr comments
* cleanup
* fix invoking the analyzer
code cleanup
* fix cr comments
* add corner case unit-test
* fix kernel version parsing
* address review comments
* add default case
* parse using regex
* added more testcases and fixed the bug found in cr
* few small things
* Add collect command and remote host collectors
Adds the ability to run a host collector on a set of remote k8s nodes.
Target nodes can be filtered using the --selector flag, with the same
syntax as kubectl. Existing flags for --collector-image,
--collector-pullpolicy and --request-timeout are used. To run on a
specified node, --selector="kubernetes.io/hostname=kind-worker2" could
be used.
The collect command is used by the remote collector to output the
results using a "raw" format, which uses the filename as the key, and
the value the output as a escaped json string. When run manually it
defaults to fully decoded json. The existing block devices,
ipv4interfaces and services host collectors don't decode properly - the
fix is to convert their slice output to a map (fix not included as
unsure what depends on the existing format).
The collect command is also useful for troubleshooting preflight issues.
Examples are included to show remote collector usage.
```
bin/collect --collector-image=croomes/troubleshoot:latest examples/collect/remote/memory.yaml --namespace test
{
"kind-control-plane": {
"system/memory.json": {
"total": 1304207360
}
},
"kind-worker": {
"system/memory.json": {
"total": 1695780864
}
},
"kind-worker2": {
"system/memory.json": {
"total": 1726353408
}
}
}
```
The preflight command has been updated to run remote collectors. To run
a host collector remotely it must be specified in the spec as a
`remoteCollector`:
```
apiVersion: troubleshoot.sh/v1beta2
kind: HostPreflight
metadata:
name: memory
spec:
remoteCollectors:
- memory:
collectorName: memory
analyzers:
- memory:
outcomes:
- fail:
when: "< 8Gi"
message: At least 8Gi of memory is required
- warn:
when: "< 32Gi"
message: At least 32Gi of memory is recommended
- pass:
message: The system has as sufficient memory
```
Results for each node are analyzed separately, with the node name
appended to the title:
```
bin/preflight --interactive=false --collector-image=croomes/troubleshoot:latest examples/preflight/remote/memory.yaml --format=json
{memory running 0 1}
{memory completed 1 1}
{
"fail": [
{
"title": "Amount of Memory (kind-worker2)",
"message": "At least 8Gi of memory is required"
},
{
"title": "Amount of Memory (kind-worker)",
"message": "At least 8Gi of memory is required"
},
{
"title": "Amount of Memory (kind-control-plane)",
"message": "At least 8Gi of memory is required"
}
]
}
```
Also added a host collector to allow preflight checks of required kernel
modules, which is the main driver for this change.
Most of the code here is borrowed from this [Microsoft repo of base images](https://github.com/microsoft/vscode-dev-containers), except for replicated specific things.
## Notes
* k3d *DOES NOT* work with DinD. You have to use the docker with docker install instead.
* Might be faster to install kubectl plugins on the `$PATH` in the `Dockerfile` instead of downloading them `onCreate.sh`.
if [ -z "$(git config --get core.editor)" ] && [ -z "${GIT_EDITOR}" ]; then
if [ "${TERM_PROGRAM}" = "vscode" ]; then
if [[ -n $(command -v code-insiders) && -z $(command -v code) ]]; then
export GIT_EDITOR="code-insiders --wait"
else
export GIT_EDITOR="code --wait"
fi
fi
fi
EOF
)"
# code shim, it fallbacks to code-insiders if code is not available
cat << 'EOF' > /usr/local/bin/code
#!/bin/sh
get_in_path_except_current() {
which -a "$1" | grep -A1 "$0" | grep -v "$0"
}
code="$(get_in_path_except_current code)"
if [ -n "$code" ]; then
exec "$code" "$@"
elif [ "$(command -v code-insiders)" ]; then
exec code-insiders "$@"
else
echo "code or code-insiders is not installed" >&2
exit 127
fi
EOF
chmod +x /usr/local/bin/code
# systemctl shim - tells people to use 'service' if systemd is not running
cat << 'EOF' > /usr/local/bin/systemctl
#!/bin/sh
set -e
if [ -d "/run/systemd/system" ]; then
exec /bin/systemctl/systemctl "$@"
else
echo '\n"systemd" is not running in this container due to its overhead.\nUse the "service" command to start services intead. e.g.: \n\nservice --status-all'
fi
EOF
chmod +x /usr/local/bin/systemctl
# Codespaces bash and OMZ themes - partly inspired by https://github.com/ohmyzsh/ohmyzsh/blob/master/themes/robbyrussell.zsh-theme
echo -e '\n(*) Warning: The docker command was not found.\n\nYou can use one of the following scripts to install it:\n\nhttps://github.com/microsoft/vscode-dev-containers/blob/main/script-library/docs/docker-in-docker.md\n\nor\n\nhttps://github.com/microsoft/vscode-dev-containers/blob/main/script-library/docs/docker.md'
echo -e "\nForward port ${SSHD_PORT} to your local machine and run:\n\n ssh -p ${SSHD_PORT} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o GlobalKnownHostsFile=/dev/null ${USERNAME}@localhost\n"
about:Create a tracking issue for a change that is larger than a single task
labels:'epic'
---
# Design Proposal
Link to the [proposal](https://github.com/replicatedhq/troubleshoot/tree/main/design/template.md)
# Definition of done
Describe what specific goals can measure if this overall task is considered completed. Things to consider are documentation, high level description of the feature working, and tests.
* [ ]
# Subtasks
Create a list of the smaller tasks to implement this design. Task can start as simple descriptions but should be converted to issues before work is started.
**Describe the rationale for the suggested feature.**
Insert a description of the rationale for the new feature here. For example, you could describe a problem that Troubleshoot doesn't address, a limitation, or an idea to improve Troubleshoot.
Insert a description of the rationale for the new feature here. For example, you could describe a problem that Troubleshoot doesn't address, a limitation, or an idea to improve Troubleshoot.
**Describe the feature**
Insert a description of the feature here. Be specific about how it addresses the any problems, limitations, or suggested improvements outlined in the rationale for the feature.
Insert a description of the feature here. Be specific about how it addresses the problem. Mention any limitations, or suggested improvements outlined in the rationale for the feature.
**Describe alternatives you've considered**
@@ -19,4 +19,4 @@ Describe alternative solutions here. Include any workarounds you've considered.
**Additional context**
Add additional context about the feature request. If the change is substantial, consider attaching files to the issue outlining architectural changes, data flows, file formats etc., anything that helps describe the requested change.
Add additional context about the feature request. If the change is substantial, consider attaching files to the issue outlining architectural changes, data flows, file formats etc., anything that helps describe the requested change.
Please include a summary of the change or what problem it solves. Please also include relevant motivation and context.
<!--- If it relates to an open issue, please link to the issue here.
e.g.
Fixes: #414
-->
## Checklist
- [ ] New and existing tests pass locally with introduced changes.
- [ ] Tests for the changes have been added (for bug fixes / features)
- [ ] The commit message(s) are informative and highlight any breaking changes
- [ ] Any documentation required has been added/updated. For changes to https://troubleshoot.sh/ create a PR [here](https://github.com/replicatedhq/troubleshoot.sh/pulls)
## Does this PR introduce a breaking change?
- [ ] Yes
- [ ] No
<!-- If this PR contains a breaking change, please describe the impact and migration path for existing applications below. -->
Thank you for your interest in Troubleshoot, we welcome your participation. Please familiarize yourself with our [Code of Conduct](https://github.com/replicatedhq/troubleshoot/blob/master/CODE_OF_CONDUCT.md) prior to contributing. There are a number of ways to participate in Troubleshoot as outlined below:
Thank you for your interest in Troubleshoot, we welcome your participation. Please familiarize yourself with our [Code of Conduct](https://github.com/replicatedhq/troubleshoot/blob/main/CODE_OF_CONDUCT.md) prior to contributing. There are a number of ways to participate in Troubleshoot as outlined below:
# Community
For discussions about developing Troubleshoot, there's an [#app-troubleshoot channel in Kubernetes Slack](https://kubernetes.slack.com/channels/app-troubleshoot), plus IRC using [Libera](ircs://irc.libera.chat:6697/#troubleshoot) (#troubleshoot).
There are [community meetings](https://calendar.google.com/calendar/u/0?cid=Y19mMGx1aGhiZGtscGllOGo5dWpicXMwNnN1a0Bncm91cC5jYWxlbmRhci5nb29nbGUuY29t) on a regular basis, with a shared calendar and [public notes](https://hackmd.io/yZbotEHdTg6TfRZBzb8Tcg)
## Issues
- [Request a New Feature](https://github.com/replicatedhq/troubleshoot/issues/new?assignees=&labels=feature&template=feature_enhancement.md) Create an issue to add functionality that addresses a problem or adds an enhancement.
- [Report a Bug](https://github.com/replicatedhq/troubleshoot/issues/new?assignees=&labels=bug&template=bug_report.md) Report a problem or unexpected behaviour with Troubleshoot.
## Pull Requests
- [Request a New Feature](https://github.com/replicatedhq/troubleshoot/issues/new?assignees=&labels=feature&template=feature_enhancement.md) Create an issue to add functionality that addresses a problem or adds an enhancement.
- [Report a Bug](https://github.com/replicatedhq/troubleshoot/issues/new?assignees=&labels=bug&template=bug_report.md) Report a problem or unexpected behaviour with Troubleshoot.
If you are interested in contributing a change to the code or documentation please open a pull request with your set of changes. The pull request will be reviewed in a timely manner.
## Design Principles
When implementing a new feature please review the [design principles](./docs/design/design-principles.md) to help guide the approach.
## Development Environment
To get started we recommend:
1. Go (v1.19 or later)
2. A Kubernetes cluster (we recommend <https://k3d.io/>. This requires Docker v20.10.5 or later)
3. Fork and clone the repo to $GOPATH/src/github.com/replicatedhq/
4. Run `make support-bundle preflight` to generate binaries
5. Run `make run-troubleshoot` to generate a support bundle with the `sample-troubleshoot.yaml` in the root of the repo
> Note: recent versions of Go support easy cross-compilation. For example, to cross-compile a Linux binary from MacOS:
> `GOOS=linux GOARCH=amd64 make support-bundle preflight`
6. Install [golangci-lint] linter and run `make lint` to execute additional code linters.
### Syncing to a test cluster with `watchrsync`
1. Install `npm`
2. Export `REMOTES=<user>@<ip>` so that `watchrsync` knows where to sync.
3. Maybe run `export GOOS=linux` and `export GOARCH=amd64` so that you build Linux binaries.
4. run `make watchrsync` to build and sync binaries automatically on saving.
A running Kubernetes cluster as well as `jq` are required to run e2e tests.
### Profiling
You are able to collect CPU & memory runtime properties and store the data for analysis in a file. To do so, pass in the file paths using `--cpuprofile` and `--memprofile` flags in the CLI. Once you have your data collected, you can analyse it using [pprof visualization tool](https://github.com/google/pprof/blob/main/doc/README.md). Here is how
Run support bundle and with CPU & memory profile flags
- Push your changes to a topic branch in your fork of the repository.
- Submit a pull request to the original repository. It will be reviewed in a timely manner.
### Pull Requests
A pull request should address a single issue, feature or bug. For example, lets say you've written code that fixes two issues. That's great! However, you should submit two small pull requests, one for each issue as opposed to combining them into a single larger pull request. In general the size of the pull request should be kept small in order to make it easy for a reviewer to understand, and to minimize risks from integrating many changes at the same time. For example, if you are working on a large feature you should break it into several smaller PRs by implementing the feature as changes to several packages and submitting a separate pull request for each one. Squash commit history when preparing your PR so it merges as 1 commit.
Code submitted in pull requests must be properly documented, formatted and tested in order to be approved and merged. The following guidelines describe the things a reviewer will look for when they evaluate your pull request. Here's a tip. If your reviewer doesn't understand what the code is doing, they won't approve the pull request. Strive to make code clear and well documented. If possible, request a reviewer that has some context on the PR.
### Commit messages
Commit messages should follow the general guidelines:
- Breaking changes should be highlighted in the heading of the commit message.
- Commits should be clear about their purpose (and a single commit per thing that changed)
- Messages should be descriptive:
- First line, 50 chars or less, as a heading/title that people can find
- Then a paragraph explaining things
- Consider a footer with links to which bugs they fix etc, bearing in mind that Github does some of this magic already
cd longhorn-manager && git checkout v1.2.2 &&cd ..
rm -rf pkg/longhorn
mv longhorn-manager/k8s/pkg pkg/longhorn
mv longhorn-manager/types pkg/longhorn/types
mv longhorn-manager/util pkg/longhorn/util
rm -rf pkg/longhorn/util/daemon
rm -rf pkg/longhorn/util/server
find pkg/longhorn -type f | xargs sed -i "s/github.com\/longhorn\/longhorn-manager\/k8s\/pkg/github.com\/replicatedhq\/troubleshoot\/pkg\/longhorn/g"
find pkg/longhorn -type f | xargs sed -i "s/github.com\/longhorn\/longhorn-manager\/types/github.com\/replicatedhq\/troubleshoot\/pkg\/longhorn\/types/g"
find pkg/longhorn -type f | xargs sed -i "s/github.com\/longhorn\/longhorn-manager\/util/github.com\/replicatedhq\/troubleshoot\/pkg\/longhorn\/util/g"
rm -rf longhorn-manager
.PHONY:scan
scan:
trivy fs \
--scanners vuln \
--exit-code=1\
--severity="HIGH,CRITICAL"\
--ignore-unfixed \
./
.PHONY:lint
lint:fmtvet
golangci-lint run --new -c .golangci.yaml ${BUILDPATHS}
.PHONY:lint-and-fix
lint-and-fix:fmtvet
golangci-lint run --new --fix -c .golangci.yaml ${BUILDPATHS}
## Syncronize the code with a remote server. More info: CONTRIBUTING.md
.PHONY:watchrsync
watchrsync:npm-install
bin/watchrsync.js
.PHONY:npm-install
npm-install:
npm --version 2>&1 >/dev/null ||(echo"npm not installed; install npm to set up watchrsync"&&exit1)
npm list gaze-run-interrupt || npm install install gaze-run-interrupt@~2.0.0
For a details on creating the custom resource files that drive preflight checks, visit [creating preflight checks](https://troubleshoot.sh/docs/preflight/introduction/).
**NOTE** this is an example. Do **not** use to validate real scenarios.
For more details on creating the custom resource files that drive preflight checks, visit [creating preflight checks](https://troubleshoot.sh/docs/preflight/introduction/).
## Support Bundle
@@ -26,29 +30,45 @@ A support bundle is an archive that's created in-cluster, by collecting logs and
To collect a sample support bundle, install the troubleshoot kubectl plugin:
```shell
```
curl https://krew.sh/support-bundle | bash
```
and run:
```shell
and run, where https://support-bundle.replicated.com provides an **example** support bundle spec:
For details on creating the custom resource files that drive support-bundle collection, visit [creating collectors](https://troubleshoot.sh/docs/collect/) and [creating analyzers](https://troubleshoot.sh/docs/analyze/).
**NOTE** this is an example. Do **not** use to validate real scenarios.
For more details on creating the custom resource files that drive support-bundle collection, visit [creating collectors](https://troubleshoot.sh/docs/collect/) and [creating analyzers](https://troubleshoot.sh/docs/analyze/).
And see our other tool [sbctl](https://github.com/replicatedhq/sbctl) that makes it easier to interact with support bundles using `kubectl` commands you already know
# Community
For questions about using Troubleshoot, there's a [Replicated Community](https://help.replicated.com/community) forum, and a [#app-troubleshoot channel in Kubernetes Slack](https://kubernetes.slack.com/channels/app-troubleshoot).
For questions about using Troubleshoot, how to contribute and engaging with the project in any other way, please refer to the following resources and channels.
# Software Bill of Materials
A signed SBOM that includes Troubleshoot dependencies is included in each release.
-**troubleshoot-sbom.tgz** contains a software bill of materials for Troubleshoot.
- [Replicated Community](https://help.replicated.com/community) forum
- [#app-troubleshoot channel in Kubernetes Slack](https://kubernetes.slack.com/channels/app-troubleshoot)
-[#Community meetings calendar](https://calendar.google.com/calendar/u/0?cid=Y19mMGx1aGhiZGtscGllOGo5dWpicXMwNnN1a0Bncm91cC5jYWxlbmRhci5nb29nbGUuY29t). This happen monthly but dates may change and would be kept upto date in the calendar.
# Software Bill of Materials
A signed SBOM that includes Troubleshoot dependencies is included in each release.
- **troubleshoot-sbom.tgz** contains a software bill of materials for Troubleshoot.
- **troubleshoot-sbom.tgz.sig** is the digital signature for troubleshoot-sbom.tgz
- **key.pub** is the public key from the key pair used to sign troubleshoot-sbom.tgz
The following example illustrates using [cosign](https://github.com/sigstore/cosign) to verify that **troubleshoot-sbom.tgz** has
If you were to get an error similar to the one below, it means you are verifying an SBOM signed using cosign `v1` using a newer `v2` of the binary. This version introduced [breaking changes](https://github.com/sigstore/cosign/blob/main/CHANGELOG.md#breaking-changes) which require an additional flag `--insecure-ignore-tlog=true` to successfully verify SBOMs like so.
returnerrors.Wrap(err,"failed to create support bundle archive")
}
fmt.Println("Redacted support bundle:",output)
returnnil
},
}
cmd.Flags().String("bundle","","file path of the support bundle archive to redact")
cmd.MarkFlagRequired("bundle")
cmd.Flags().BoolP("quiet","q",false,"enable/disable error messaging and only show parseable output")
cmd.Flags().StringP("output","o","","file path of where to save the redacted support bundle archive (default \"redacted-support-bundle-YYYY-MM-DDTHH_MM_SS.tar.gz\")")
cmd.Flags().Bool("collect-without-permissions",true,"always generate a support bundle, even if it some require additional permissions")
cmd.Flags().StringSliceP("selector","l",[]string{"troubleshoot.sh/kind=support-bundle"},"selector to filter on for loading additional support bundle specs found in secrets within the cluster")
cmd.Flags().Bool("load-cluster-specs",false,"enable/disable loading additional troubleshoot specs found within the cluster. required when no specs are provided on the command line")
cmd.Flags().String("since-time","","force pod logs collectors to return logs after a specific date (RFC3339)")
cmd.Flags().String("since","","force pod logs collectors to return logs newer than a relative duration like 5s, 2m, or 3h.")
cmd.Flags().StringP("output","o","","specify the output file path for the support bundle")
cmd.Flags().Bool("debug",false,"enable debug logging. This is equivalent to --v=0")
// hidden in favor of the `insecure-skip-tls-verify` flag
cmd.Flags().Bool("allow-insecure-connections",false,"when set, do not verify TLS certs when retrieving spec and reporting results")
// `no-uri` references the `followURI` functionality where we can use an upstream spec when creating a support bundle
// This flag makes sure we can also disable this and fall back to the default spec.
cmd.Flags().Bool("no-uri",false,"When this flag is used, Troubleshoot does not attempt to retrieve the bundle referenced by the uri: field in the spec.`")
// loadClusterSpecs loads the support bundle and redactor specs from the cluster
// based on the provided labels. By default this will be troubleshoot.io/kind=support-bundle and troubleshoot.sh/kind=support-bundle label selectors. We search for secrets
// and configmaps with the label selector and parse the data as a support bundle. If the
// user does not have sufficient permissions to list & read secrets and configmaps from
// all namespaces, we will fallback to trying each namespace individually, and eventually
// default to the configured kubeconfig namespace.
description:RemoteCollector is the Schema for the remote collectors API
properties:
apiVersion:
description:'APIVersion defines the versioned schema of this representation
of an object. Servers should convert recognized schemas to the latest
internal value, and may reject unrecognized values. More info:https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
type:string
kind:
description:'Kind is a string value representing the REST resource this
object represents. Servers may infer this from the endpoint the client
submits requests to. Cannot be updated. In CamelCase. More info:https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
type:string
metadata:
type:object
spec:
description:RemoteCollectorSpec defines the desired state of the RemoteCollector
Currently all collectors run sequentially, in the order in which they are specified in the spec. This is implied, and has not been guaranteed. With the addition of the ability to supply multiple specs, the order is no longer predicatble.
Although sequence is not guaranteed for any collectors, #768 ensures that the `clusterResources` collector runs first, so that the list of resources does not include those that are started during the Troubleshoot collection sequence.
The performance for support-bundle and preflight runs is important to the end user experience of the product. There are examples where a support bundle takes multiple minutes to complete, and folks are left wondering if the process has hung or crashed. Preflights that take too long will simply be skipped, which leads to problems with installation being difficult to identify.
## High-Level Design
Switch collectors and host collectors to use goroutines to collect (concurrently). This should run the list of collectors of `clusterResources` type first, then the remainder concurrently when `clusterResources` is complete.
## Detailed Design
Tasks that can be split to separate merges (in order):
* Switch all collectors to use goroutines to collect (concurrently).
* In `supportbundle.runCollectors()`, refactor to use a `go func()` pattern to run all the collectors that are not `clusterResources` concurrently. Check that the first in the list is `clusterResources` and run that separately, prior to the others.
* This task should introduce an optional concurrency limit field, default to, say, 10, which limits the number of goroutines/collectors that can run at any given time.
* This task should improve the collection speed of in-cluster collectors and preflights
* Switch Host Collectors to use goroutines for concurrent collection.
* In `supportbundle.runHostCollectors()`, refactor to use a `go func()` pattern for each collector.
* No need to honor the same concurrency limit as host collectors do not load the Kubernetes API
* Switch Host Preflights to use goroutines - `preflight.CollectHost()` runs `go func()` for each collector.
* Switch in-cluster Preflights to use goroutines - `preflight.Collect()` runs `go func()` for each collector.
* Ensure `clusterResources` runs in advance of the others so that pods started by collectors do not produce an error in the results.
## Limitations
Relies on splitting up the list of collectors into `clusterResources` and others.
Collisions between specs (e.g. if `runPod` has multiple specs with the same name) are not handled. At the current stage with collectors running sequentially, if two collectors write to the same file the second overwrites the data from the first. If collectors run concurrently, there can be issues creating multiple pods with the same name, and issues with concurrent access to the target file. This issue is tracked in [#895](https://github.com/replicatedhq/troubleshoot/issues/895)
This proposal excludes Remote Collectors.
## Assumptions
* The Kubernetes API can handle the load of collectors running concurrently, including when the pod logs collector runs.
## Testing
Any new function will need unit tests.
The existing Collect functions will need unit tests altered (or added).
We will need to ensure that `clusterResources` is collected prior to others.
## Documentation
There should be a note in https://troubleshoot.sh/docs/collect/ ensuring that folks do not expect the collectors to run in any particular order.
Folks that have not specified the IP address regex above will, on support bundle creation, share their IP address information in that bundle by default.
When the `preflight` binary, or pacakge, is called, it supports only one spec definition at a time.
Recent changes in Troubleshoot allow the `support-bundle` binary to be called with multiple specs at a time. This allows cluster components to contribute independant Troubleshoot specs for their scope, and have Troubleshoot assemble them at run time.
Tools such as kURL have components which are maintained with a degree of separation to one another. It would be helpful to the maintainers of such projects to be able to call `preflight` specifying a number of specs at runtime, allowing Troubleshoot to assemble them into one spec for collection/analysis.
Currently if the `preflight` binary is called with multiple specs, it simply ignores all after the first.
## Decision
Modify the `preflight` CLI and package to be able to read multiple args rather than just one.
Introduce a merge mechanism in the same way that the `support-bundle` binary runs, to merge and deduplicate Preflight specs.
## Status
Proposed
## Consequences
There are no backward compatibility consequences or breaking changes in this proposal.
The project benefits:
* Folks maintaining kURL add-ons can contribute unique preflight specs for their add-on (and the same for other projects simlarly structured)
* Folks using `preflight` from the CLI without other applications (e.g. for a Helm install) are able to specify a list of preflights for their application rather than having to assemble one spec for each environment.
## Design notes
This proposal does not include adding the `uri:` field to `kind: Preflight`.
The file `cmd/preflight/cli/root.go` calls `preflight.RunPreflights` with `args[0]` which is likely to need to change to just `args`.
Func `RunPreflights` takes a single string arg (`arg string`) for the spec definition. This is likely to need to change to `arg []string`.
File `cmd/troubleshoot/cli/run.go` loops through the list of args, concatenating them together. A similar process is suitable for this change.
Describe the resulting context, after applying the decision. All consequences should be listed here,
not just the "positive" ones. A particular decision may have positive, negative, and neutral consequences,
but all of them affect the team and project in the future.
```
## Tips and Hints
* Titles should be descriptive, concise, and precise
* The whole document should be one or two pages long at most.
* Think of the document as a conversation with a future developer. This means write well and use full
sentences.
* Update consequences as they become known. The ADR becomes like a diary for seeing how the design
decisions we make impact the system over time.
* Include diagrams as necessary.
## References
Nygard, Michael. Documenting Architecture Decisions, from _Think Relevance_ blog. [Web](http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions)
Kruchten, Philippe. _The Decision View's Role in Software Architecture Practice_, IEEE Software 26:36-42, February 2009
Tyree, J. and Akerman, A. _Architecture Decisions: Demystifying Architecture_, IEEE Software 22:2:19-27, March-April 2005 [PDF](http://www.utdallas.edu/~chung/SA/zz-Impreso-architecture_decisions-tyree-05.pdf)
This document captures design principles that the Troubleshoot project abides by. This is intended to provide new contributors with guidance on how to approach problems and a better understanding of what to consider and address while implementing features.
## Client not Cluster based
Troubleshoot has to communicate with the Kubernetes API server to gather information. However, Troubleshoot should interact with the cluster as little as reasonably possible and has no intention of having a persistent in-cluster presence. There are several reasons for this approach, in no particular order those include:
* Users experiencing an issue in their cluster may discover Troubleshoot after they have a problem and requiring installed components may exclude them from solving their problem.
* A user may have limited cluster access and requiring cluster wide installation like a CRD or Operator can prevent them for using tools to recover from their errors.
* By definition Troubleshoot is being used because there is an issue with the cluster, to prevent exasperating the issue Troubleshoot should avoid writing to the cluster.
## Tools not specs
The Troubleshoot project should include tools which can be used to diagnose issues but is not attempting to include Specs built into the project itself. The number of issues and projects which could benefit from Troubleshoot is very large and we intended to enable those projects to better support their project. It is unreasonable and undesirable for Troubleshoot to be the source of truth for all possible cluster and software issues.
## Fail forward
Clusters with issues can be unpredictable, the scope of error conditions is likely unbounded. Troubleshoot should keep this in mind when doing error handling and fail forward proceeding with as much of the intended operation as possible while logging the errors. For example if a collector fails to collect information that should not prevent other collectors from running. Any condition that causes Troubleshoot to hang or not complete a run is considered a bug.
## Provide a predictable user experience
When things go wrong people can be stressed and stressed people aren't likely to thoroughly read documentation. Whenever possible default values, command line flags, etc should be set to provide a user the best experience possible with the input provided. Some examples of this include:
* When accepting flags for things like Namespaces, as much as possible, use the same syntax that `kubectl` accepts which users are likely to try instinctively.
* Default to gathering more information not less, rarely is it easier to diagnose an issue with less information.
Improve consistency between preflight and support-bundle.
## Non Goals
## Background
In the current Troubleshoot code base, there are three separate paths to running collectors. These include preflights, support-bundle, and the collect package. The three have diverged over time and are different to one another, but do not appear to have any need to be separate. This is confusing and likely to introduce errors in the future.
## High-Level Design
* Add a `collect` package public API that can be called to run the collect logic, from any other package
* change the preflight, support-bundle and collect binaries to call that API rather than their own collect routines
## Detailed Design
Package `preflight`:
* Remove `CollectHost`, `Collect` and `CollectRemote`
Package `supportbundle`:
* Remove `runCollectors`, `CollectSupportBundleFromSpec`, and associated code
Package `collect`:
* Add a replacement for `runCollectors`, `CollectSupportBundleFromSpec` taken from `supportbundle`
* Add replacements for `CollectHost`, `Collect` and `CollectRemote` taken from `preflight`
* Where the above duplicate functionality, alter the `supportbundle` or `preflight` packages to ensure that a single new function handles the requirement
CLI packages:
* Alter to use the public functions from `collect`
## Limitations
Breaking change - KOTS at least imports `CollectSupportBundleFromSpec` plus potentially others.
This does not affect the analysis and redaction portions of code.
## Assumptions
* there is no need to run collectors differently between preflight and support-bundle
## Testing
Any new function will need unit tests.
Existing tests will need to be altered, and possibly consolidated.
## Documentation
The public function to call collection should be documented and at that point regarded stable.
## Alternatives Considered
## Security Considerations
None identified.
## Related changes
TODO:
* make an API for analyze and redact
* create an API that calls collect, redact, and analyze - and decide if that can be called by preflight and support-bundle
# Provide the ability to specify multiple Troubleshoot specs in one run of support-bundle or preflight
Troubleshoot doesn't have a modular way for different components to specify specifications. If a software project wanted to include project specific items the end user would have a hard time knowing where to find those and how to collect them. Furthermore if multiple projects were to do this the user would have to run troubleshoot multiple times.
Ideally troubleshoot would allow merging of specs to allow building a spec either automatically or influenced by user input to target specific needs.
## Goals
Primary goal: Allow folks that develop a particular component to maintain Troubleshoot specs for that component, including Vendors for their application.
Long term goal: allow folks to update the component specs without needing to run any other upgrades.
This proposal initially adds the ability to supply multiple support bundle specs for a single run of support-bundle and/or preflight so that:
* when Troubleshoot runs, the spec it uses is an aggregate of multiple specs from the same source - yaml file, URL, configMap, or (new) CRD
* each software component can contribute a Troubleshoot spec that is specific to that software
* Ownership of each individual spec per component transfers to the owner of that component
This will:
* Ensure support bundles generated by users are comprehensive and contain all needed information for a project maintainer to action
* Allow multiple specs from different projects, should a cluster have multiple projects using Troubleshoot
* Software developers to continually update specs for components to easily identify known issues and reduce support noise.
## Non Goals
* Compatibility with previous URL, configMap and yaml specs, both Redactors and SupportBundle types
* Maintain the ability to read secrets from the Kubernetes API
## Background
When users of Troubleshoot create a support bundle from the CLI, they use a single spec that is provided from:
* A secret installed in the cluster by an application such as [KOTS](kots.io)
* A url like <http://kots.io>
* some other single example spec, e.g. <https://github.com/replicatedhq/troubleshoot-specs>
When KOTS collects a support bundles from the KOTS UI, the spec is a [merged](https://github.com/replicatedhq/kots/blob/main/pkg/supportbundle/spec.go#L426) combination of the following:
* The [default spec](https://github.com/replicatedhq/kots/blob/main/pkg/supportbundle/defaultspec/spec.yaml) in the kots code
* The spec provided by the Vendor’s application bundle
* This merged spec is deduplicated by kots [here](https://github.com/replicatedhq/kots/blob/main/pkg/supportbundle/spec.go#L217).
Redactors are [hard coded](https://github.com/replicatedhq/troubleshoot/blob/main/pkg/redact/redact.go#L160) into Troubleshoot as well as supplied in the spec.
Although we’re currently making strides in improving the Troubleshoot project by creating new collectors, analyzers, and specs, we have no way to more quickly deliver Troubleshoot improvements to installations in the field without updating the kots application and pushing a new release, or upgrading kots to a new version.
When folks find a support issue that could have been identified by Troubleshoot, we ask them to write a new collector, and/or analyzer, for that information. However, if we do that, the new collector/analyzer is not easily available to end users.
Some of the useful features of Troubleshoot are actually implemented as part of KOTS, while both open source Troubleshoot should address this independently.
## High-Level Design
Add CRD support to Troubleshoot:
* Design a custom resource (CRD) that allows adding spec(s) to the Kubernetes cluster using `kubectl`. There is no need to extend this to use [API server aggregation](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#api-server-aggregation).
* Update Troubleshoot to allow consuming the first object found of the new custom resource, by default - i.e. if there are no specs provided on the CLI or entrypoint, use CRD
* Once merge is implemented, update Troubleshoot to consume and merge all the instances of the CRD.
* To minimize code changes in Troubleshoot, we could implement two CRDs, of `type: SupportBundle` and `type: Redactor`. However, there is no specific need to separate the two and it maybe advantageous to keep things simple, combining them.
Allow multiple specs to be merged by Troubleshoot:
* create an interface for collectors, analyzers, and redactors to merge
* each collector/analyzer/redactor can use a generic implementation of the interface for the merge, or can use a specific one for that task if that particular collector will benefit from a more intelligent merge.
* the spec merge code in kots can, at this stage, be removed once kots ships CRDs for the Troubleshoot specs
Spec sources:
* Alter Troubleshoot to accept multiple yaml files on the CLI
* Create a CRD containing a spec, have Troubleshoot search for spec CRDs and combine them all when run
* maintain the URL compatibility as is
* maintain the configMap compatibility as is
## Detailed Design
CRD: to be designed
Merge:
* define a new interface that provides the merge functionality for collectors, analyzers, and redactors
* there should be a generic implementation of that interface which is used by default for all objects. This can simply use `append()`.
* specific collectors may have alternative implementations of the interface where overrides are required.
Particular overrides known at this point in time:
* clusterResources has an option namespaces config, these should be merged and deduplicated
* Two runPod collectors with the same name and different commands are unmergable. Only one should be run and an error logged.
* Two configMap collectors can be deduplicated and merged depending on the configurations
* Two logs collectors likely only need deduplication to prevent collecting the same thing twice
## Limitations
### Visibility
Users may collect more than intended with the new implementation if all available CRD objects are run. We therefore need to be sure the Troubleshoot client allows for selection of what to run. While the default would be to run everything, a user should be able to just run specific SupportBundle specs based on labels, namespaces, etc. By doing this no functionality is lost.
### Airgap
Some of the spec locations such as URL wouldn’t work here. There may be other considerations around airgap as well.
Airgap installs are intentionally feature limited. Failure to retrieve new versions of specs should fail gracefully and proceed with the run.
As a follow up feature to improve support bundle collection Troubleshoot could gain some abilities to help manage and upgrade SupportBundles found in a cluster in a fashion that allows the user to split the discovery, download, and upgrade so they could be run on different machines.
An example implementation would be:
* A command to find, deduplicate, and output a list of SupportBundles in the cluster. The list would include at least the upstream URL and version currently installed in the cluster.
* A command to take the output of the above command and check for updates, downloading all updated SupportBundles into a single .tgz file.
* A command to apply the above .tgz file to the cluster upgrading any existing SupportBundles with the newer versions.
## Assumptions
* all deployments are able to deploy CRDs to the Kubernetes cluster
## Testing
## Alternatives Considered
### Use the existing URL functionality in Troubleshoot
Proposal: Provide a custom URL for each install, that when called by Troubleshoot returns a spec composed of the ‘latest’ for every component spec specified in that installation, much like we do with <https://kurl.sh/latest> now.
Pros:
* No changes required to Troubleshoot
* Always use the latest specs for components other than the Vendor’s application
Cons:
* Would not work for airgap installations
* Software developers need to update their application in order to get new specs from all dependencies
* Replicated would need to provide a new API/web service to host the specs, reducing the general community applicability of troubleshoot as a stand alone project
* Discourages individual projects from maintaining Troubleshoot specs for that project
## Security Considerations
Passing control of component specs to individual projects presents a possibility of reducing the amount of review a spec goes through for each update, allowing a spec provided by, say, a kURL addon to run collectors. The current implementation relies on the kots review process for default specs, plus the review of individual example specs. Depending on the final implementation, it is unlikely that this change itself increases any risk since the delivery of automated and default specs to Troubleshoot is maintained within kots, and/or Troubleshoot itself much as it is already.
Make sbctl easier to maintain by consolidating code bases.
Reduce code duplication.
Improve end-user experience and improve discoverability of sbctl.
## Non Goals
## Background
sbctl is our command line tool for examining K8s resources inside of Support Bundles generated by our troubleshoot project. With both projects having separate code bases we see the following problems:
* Most users are familiar with the binaries from troubleshoot (support-bundle, preflight, etc.), but are unaware of the existince of sbctl because it's a separate project
* sbctl and troubleshoot are naturally coupled together due to sbctl's reliance on support bundles to operate. Them being in separate repos leads to duplication of code in both projects increasing maintenance burden.
## High-Level Design
Because the functionality of sbctl and Troubleshoot are so tightly coupled together, they should live in the same repository. Moreover, as opposed to maintaing a separate binary for sbctl, all sbctl commands should become subcommands of `support-bundle`
## Detailed Design
The `serve` and `shell` commands become subcommands of the `support-bundle` CLI.
```
./bin/support-bundle --help
A support bundle is an archive of files, output, metrics and state
from a server that can be used to assist when troubleshooting a Kubernetes cluster.
Usage:
support-bundle [url] [flags]
support-bundle [command]
Available Commands:
analyze analyze a support bundle
serve Start API server
shell Start interractive shell
completion Generate the autocompletion script for the specified shell
...
```
Similiar to the [Analyze](https://github.com/replicatedhq/troubleshoot/blob/main/cmd/troubleshoot/cli/analyze.go) subcommand, we add `serve.go` and `shell.go` to `cmd/troubleshoot/cli`
### Standardize Cluster Resources Collector
* In `pkg/collect` standardize the root directory location of the Cluster Resources collector with a variable instead of repeatedly specifying it as a string - https://github.com/replicatedhq/troubleshoot/blob/main/pkg/collect/cluster_resources.go#L121.
* Import the directory the Cluster Resources collector uses from `pkg/collect` for usage in the APIs - https://github.com/replicatedhq/sbctl/blob/main/pkg/api/server.go#L220. This is an example of the way we can start making the sbctl code more efficient with this change.
tests # API tests, integration tests etc reside here.
serve/ # sbctl BDD tests.
# all tests and test fixtures moved here
```
### Makefile Changes
We'll need to add make targets specific to these commands similar to the current ones
```
make serve run-serve
make shell run-shell
make ginkgo
```
### Docs
In troubleshoot.sh docs, add a new sub section under "Support Bundle" describing then new serve & shell subcommands.
### Other Changes
We would need to merge some of the support bundle functionality we see in both projects like so
- Create a new sbutil module to store reusable support bundle functionality. At least `analyze` & `serve` modules will depend on this module
```
pkg
sbutil/
```
- Merge `troubleshoot/pkg/analyze/download.ExtractTroubleshootBundle` & `sbctl/pkg/sbctl/support-bundle.ExtractBundle` that duplicate support bundle archive extraction
- Build on `sbctl/pkg/sbctl/support-bundle.ClusterData` to have it expose well known paths (ClusterResourcesDir, Version files, analysis results etc) and unmarshalled resources (this bit can be built on a need to have basis) in a support bundle especially once [Add generic kubernetes resource analyzer #780 PR](https://github.com/replicatedhq/troubleshoot/pull/780) is merged.
- Build on `sbctl/pkg/sbctl/support-bundle.FindClusterData` to perform pre-validation of a support bundle to ensure it is valid before running any applications.
## Limitations
## Assumptions
## Testing
We will need to have a separate directory for sbctl (ginkgo BDD) tests. The test harness used is ginkgo cli instead of go test and there is no straight forward way of separating these tests. Also, from a directory structure point of view, such integration tests usually reside in modules in a root level tests directory. Changes to the test code need to be minimal, ideally no change, cause they will be used to ensure any refactored code does not break. We could now easily introduce tests which ensure that changes to collectors don't break APIs in sbctl.
## Alternatives Considered
* Have troubleshoot as a library dependency in sbctl keeping it as a separate binary and repo
* Move sbctl to the Troubleshoot library "as is" and continue maintaing a separate binary
# Provide an extendable API for accessing bundle information
## Goals
* Provide API based access to collected information, decoupling other projects from troubleshoot on-disk format
* Reuse existing APIs when they exist to make the bundle compatible with existing tools without modification
* Plan for extensibility for accessing information beyond just the standard kubernetes api
## Non Goals
* Compatibility with previous on-disk formats
* Compatibility with existing collectors without modification
* There should be a plan to allow the implementation of existing collectors
## Background
While using the information gathered in a support bundle users were finding it hard to find information while manually reviewing the various files collected in the bundle. Users have to understand the folder structure, files structure, and process JSON files to find information about the cluster. The [sbctl](https://github.com/replicatedhq/sbctl) project was created to prove out the utility of providing api based access so that users could use existing tools which they already understood. This has been a very successful experiment with feedback being that most users now use this utility as their primary, or only, interface to the support bundle information.
There are some drawbacks to the current approach. The `sbctl` project is tightly coupled to troubleshoot on-disk formats, each kubernetes API must be implemented in `sbctl` individually and will require being kept up to date as APIs change, and `sbctl` has no plan today to provide similar API based access to information other than the standard kubernetes api.
This proposal is meant to take the learnings from `sbctl` and consider implementing it as a first class feature of troubleshoot while attempting to address the current maintenance and expandability limitations.
## High-Level Design
To help address standard access to API data, troubleshoot will start an etcd instance which collectors can then use to store collected information rather than writing to custom on-disk locations. Collectors that only need to collect Kubernetes API information then do not need to serve up data as the API server will be used to provide access to the collected data. Storing data directly in etcd will allow troubleshoot to later serve up this same information by again starting an api-server and etcd using the previously collected etcd data store. This should remove almost all maintenance from troubleshoot for implementing API calls to access the collected information.
There will also be information that is desired to be stored that doesn't have any native representation in the api-server today. Examples could include custom collectors which execute into containers and extensions like [metrics server](https://github.com/kubernetes-sigs/metrics-server). These components can be addressed using the built in [Kubernetes API Aggregation Layer](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/) which is how `metrics server` itself works to extend and provide api based access to node information. By registering additional API extensions troubleshoot plugins can implement both a collection and an API for retrieving custom information which is accessible in the same fashion as the rest of the api-server.
The additional benefit to this, which can be demonstrated with `metrics-server`, is that collectors can be written for any other extension API and be compatible with existing tools the same way using the api-server provides compatibility with `kubectl`. Using `metrics-server` as an example, a collector can be written to collect node information which can be stored locally. The collector can then implement the standard [GET Endpoints](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/) from the metrics-server and reply to them with the results it collected. By doing so, all tools that work with metrics-server today, like `kubectl top` will also work with information served from a support bundle. Additionally, the collector implementation handles any custom file formats without exposing that to any other tool making it easy to maintain.
Any other access to the filesystem directly will be modified to instead use the provided api-server. This means analyzers will not directly reference files on disk and will instead run against the api-server to analyze the collected information. This decouples analyzers from how collectors store data providing clean separation of concerns for maintaining both collectors and analyzers. Decoupling these also creates a natural way to implement analyzers that use data from multiple collectors. This could also allow analyzers to run against existing clusters providing a use for analyzers independent of support bundle collection. This could encourage additional community contributions of analyzers.
An initially unintended benefit of using the Aggregation Layer is that any HostCollector using this implementation would be very close to an implementation of an extension which you could install in clusters. This could make HostCollectors also useful to install as a service in a live cluster for operations information about hosts.
## Detailed Design
### Outstanding design questions
1. How reasonable is it to start an api-server as part of troubleshoot? Consider the following known implementations that do something like this:
* [envtest](https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/envtest) - requires binaries present on the machine
* [k0s uses upstream binaries statically compiled](https://docs.k0sproject.io/v1.23.8+k0s.0/architecture/) - bundles statically compiled binaries that self extract and uses a process monitor to run them
2. Can you in fact push metadat like "Status" into an api-server or do we have to write directly to etcd?
* If we can't push to the api-server is just writing the information directly into etcd something we can do and have a reasonable expectation of compatibility?
3. Is the overhead to write an Aggregation API going to add an unnecessary burden to writing new collector plugins? Can these be templated into a reasonably to ease collector creation?
## Limitations
Using the actual API server will provide limitations on the version skew which can be collected/displayed. This could be addressed by including multiple versions of the kubernetes-api server to allow serving a wide range of support bundles. This limitation likely already exists today but would exist in the tooling that is trying to collect, analyzer, or server the data.
## Assumptions
* Serving logs hasn't been designed yet, and presumably can be addressed in the detailed design to provide logs to the api-server in place of kubelet. Ideally this can be done using the standard upstream kubernetes-api server it is undesirable to fork it.
* Running the api-server, etcd, and any other supporting services (like kubelet) as go routines while adding some overhead to the collection process won't cause a significant burden on ram or cpu to collect support bundles.
## Testing
## Alternatives Considered
### Keep sbctl separate
The current `sbctl` project could be left to run it's course independent of this project. This leaves the troubleshoot project reliant on a separate project to provide a good user experience for anything other than analyzers.
## Security Considerations
Consideration to how Redactors are implemented needs to be considered.
## References
Original PR discussion found [here](https://github.com/replicatedhq/troubleshoot/pull/611)
# Provide a facility to automatically obtain updated specs
## Background
Troubleshoot is limited in that when folks write an update or a new spec for data collection and analysis, that upgrade is not available when Troubleshoot is invoked if specs for it are stored in a secret or a file which was deployed by an application. Updating the local spec is simple enough, but may require an upgrade to the application (e.g. KOTS) in order to collect updates to the spec supplied by that application. This update requirement delays folks from accessing updated Troubleshoot specs and therefore discourages people from writing new/updated specs.
## Goals
* Provide a means for a spec to optionally specify a location for a spec that can replace it if successfully reached.
## Other requirements
* Maintain the ability for Troubleshoot to run in an airgapped environment, though with some feature limitations.
* Maintain compatilbility with existing Troubleshoot specs.
## High-Level Design
Add a new field to the Troubleshoot spec definition, which includes a URI used to locate a replacement spec.
If the field is populated, Troubleshoot is to attempt to collect the spec from the location provided, and if successful, ignore the remainder of the spec provided and use the spec listed in the new field.
If the additional spec is not found at the location (or, if there is no network access to that location), Troubleshoot is to continue processing the remainder of the spec provided, with a log message describing the failure to process the URI.
Add a CLI flag (e.g. `--no-updates`) to disable the URI specified location from being accessed. This would be useful in airgap environments and/or when we simply do not want to have the external spec used, without having to update the yaml, secret, or remote source.
URI's provided may be any of the URI types already accepted by Troubleshoot, including web addresses (https://), secrets, or files.
## Detailed Design
Current spec format is like this example:
```yaml
apiVersion: troubleshoot.sh/v1beta2
kind: SupportBundle
metadata:
name: default
spec:
collectors:
- clusterInfo: {}
- clusterResources: {}
analyzers:
- cephStatus: {}
- longhorn: {}
```
We could add a new type, `specURI`, which contains one or more URIs from which to obtain specs. If the spec is retrieved successfully then the remainder of the provided spec is ignored, and replaced with the one downloaded:
When a spec is parsed from the initial call to Troubleshoot, the content should be read and if a `specURI` exists which is not blank, the code should attempt to download that URI. Should that be successful, the collectors and analyzers sections of the downloaded content should be used to replace the remainder of the spec provided via the initial call.
## Impact on kURL and KOTS
This section is included because KOTS is a significant consumer of the Troubleshoot codebase, and as such deserves consideration when significant changes are made.
The code changes proposed in this document do not require any changes to kURL or KOTS, and any changes to those repos are excluded from the scope of work described here. The following notes are suggestions for how the kURL and KOTS repos could benefit from the changes in this document, and could relate to any project which calls Troubleshoot or includes Troubleshoot specs.
For kURL, each component/addon installed could include adding an individual spec with a URI pointing at a specific spec for that component, which could be updated independently of other components. Invoking troubleshoot with all the individual specs required would allow updated specs to be collected without needing to upgrade any particular component in the cluster.
KOTS currently provides a spec to Troubleshoot which is a merge of a default spec provided by the KOTS codebase, and the spec provided by an application vendor. Should either of those provide a URI, that would mean the downloaded spec would replace the entire spec. This is undesirable since the URI would need to contain the application vendor's updated spec as well as the generic cluster-wide collectors and analyzers, which is a significant maintenance challenge. To resolve this, it is desirable that the following is completed before KOTS uses the URI field:
* Troubleshoot should accept multiple specs (see #650)
* KOTS would need to be modified to provide the application vendor and the default specs independently of one another.
## Limitations
Redactors are currently excluded from this proposal, though it would be desirable to add a similar feature in the future.
If a URI provided includes a spec which includes another URI, it is possible to get into some kind of recursion if the downloaded spec contains another URI. For this reason, and to ensure that the results are predictable, we should only parse the URI field once per spec for specs passed to Troubleshoot on initiation via the CLI or entrypoint.
## Assumptions
## Testing
## Alternatives Considered
* Some mechanism to trigger a process to update secrets stored in Kubernetes. This was rejected due to the need to run a manual update spec for something installed in the cluster.
* Altering KOTS to provide a URL for the spec rather than storing it in code. This was rejected to allow airgap compatibility.
## Security Considerations
This change will make additional network calls with default usage of Troubleshoot. To ensure a user has control of limiting these calls an argument to disable this feature entirely will be provided. This allows users with different security postures to select what's appropriate for their environment.
--kubeconfig string Path to the kubeconfig file to use for CLI requests.
--memprofile string File path to write memory profiling data
-n, --namespace string If present, the namespace scope for this CLI request
-o, --output string specify the output file path for the preflight checks
--request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0")
--selector string selector (label query) to filter remote collection nodes on.
-s, --server string The address and port of the Kubernetes API server
--since string force pod logs collectors to return logs newer than a relative duration like 5s, 2m, or 3h.
--since-time string force pod logs collectors to return logs after a specific date (RFC3339)
--tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used
--token string Bearer token for authentication to the API server
--user string The name of the kubeconfig user to use
-v, --v Level number for the log level verbosity
```
### SEE ALSO
* [preflight oci-fetch](preflight_oci-fetch.md) - Fetch a preflight from an OCI registry and print it to standard out
* [preflight version](preflight_version.md) - Print the current version and exit
###### Auto generated by spf13/cobra on 8-Jun-2023
--kubeconfig string Path to the kubeconfig file to use for CLI requests.
--load-cluster-specs enable/disable loading additional troubleshoot specs found within the cluster. required when no specs are provided on the command line
--memprofile string File path to write memory profiling data
-n, --namespace string If present, the namespace scope for this CLI request
--no-uri When this flag is used, Troubleshoot does not attempt to retrieve the bundle referenced by the uri: field in the spec.`
-o, --output string specify the output file path for the support bundle
--redactors strings names of the additional redactors to use
--request-timeout string The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0")
-l, --selector strings selector to filter on for loading additional support bundle specs found in secrets within the cluster (default [troubleshoot.sh/kind=support-bundle])
-s, --server string The address and port of the Kubernetes API server
--since string force pod logs collectors to return logs newer than a relative duration like 5s, 2m, or 3h.
--since-time string force pod logs collectors to return logs after a specific date (RFC3339)
--tls-server-name string Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used
--token string Bearer token for authentication to the API server
--user string The name of the kubeconfig user to use
-v, --v Level number for the log level verbosity
```
### SEE ALSO
* [support-bundle analyze](support-bundle_analyze.md) - analyze a support bundle
* [support-bundle redact](support-bundle_redact.md) - Redact information from a generated support bundle archive
* [support-bundle version](support-bundle_version.md) - Print the current version and exit
###### Auto generated by spf13/cobra on 8-Jun-2023
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Dexter Yan
Evans Mungai
ada mancini
Xav Paice
Andrew Lavery
David Rohnow
Ahmed Mousa
danj-replicatedandNathan Sullivan
Martin Hrabovcin
Ethan Mosbaugh
Ricardo Maraschini
Nick Meyer
Tom Wieczorek
Andrew Storms
Edgar Lanting
Nathan SullivanandEvans Mungai
Craig O'Donnell
stefanrepl
Diamon Wiggins
Dan Jones
Edgar Lanting
Nick Meyer
Salah Al Saleh
divolgin
Yue Tan
Pavan Sokke Nagaraj
Kira C Boyle
Martin Hrabovcin
ada mancini
Xav Paice
Tarun Gupta Akirala
Alex Parker
Alexander
Marc Campbell
Nils Seip
Ethan Mosbaugh
Edgar OchoaandEthan Mosbaugh
Jalaja Ganapathy
Craig O'Donnell
Jeff Golden
divolgin
Sean Rester
deepsource-autofix[bot]
Andrew Reed
Dimitri KoshkinandSalah Aldeen Al Saleh
Vera Harless
Simon Croome
Simon Croome
Dan Stough
danbudris