Commit Graph
68 Commits
Author SHA1 Message Date
Benjamin Yang c67775c1a0 feat: clean tokenization system implementation (#1874)
Core tokenization functionality with minimal file changes:

 Core Features:
- Intelligent tokenization engine (tokenizer.go)
- Context-aware secret classification (PASSWORD, APIKEY, DATABASE, etc.)
- Cross-file correlation with deterministic HMAC-SHA256 tokens
- Optional encrypted mapping for token→original value resolution

 Integration:
- CLI flags: --tokenize, --redaction-map, --encrypt-redaction-map
- Updated all redactor types: literal, single-line, multi-line, YAML
- Support bundle integration with auto-upload compatibility
- Backward compatibility: preserves ***HIDDEN*** when disabled

 Production Ready:
- Only 11 essential files (vs 31 in original PR)
- No excessive test files or documentation
- Clean build, all functionality verified
- Maintains existing redaction behavior by default

Token format: ***TOKEN_<TYPE>_<HASH>*** (e.g., ***TOKEN_PASSWORD_A1B2C3***)
2025-09-30 15:04:23 -05:00
Marc Campbell 007ef32e77 fmt: 2025-09-29 10:43:51 -07:00
Noah Campbell 885819ea9a Auto support bundle upload (#1860)
* basic auto uploading support bundles

* added upload command

* added default vendor endpoint

* added auth system from replicated cli

* fixed case sensitivity issue in YAML parsing

* support bundle uploads for end customers

* app slug flag and detection without licenseID
2025-09-17 12:59:41 -05:00
Diamon Wiggins 7a6bffeff5 chore: fix noisy info logs (#1808)
* refine logging

* keep progress message at level 0
2025-07-09 20:58:47 -04:00
Ethan Mosbaugh a4a387eb0e chore: CVE-2024-0406 remove github.com/mholt/archiver/v3 dependency (#1793) 2025-06-06 11:35:56 -07:00
9bca9c5245 chore(deps): bump github.com/distribution/distribution/v3 from 3.0.0-rc.3 to 3.0.0 (#1771)
* chore(deps): bump github.com/distribution/distribution/v3

Bumps [github.com/distribution/distribution/v3](https://github.com/distribution/distribution) from 3.0.0-rc.3 to 3.0.0.
- [Release notes](https://github.com/distribution/distribution/releases)
- [Commits](https://github.com/distribution/distribution/compare/v3.0.0-rc.3...v3.0.0)

---
updated-dependencies:
- dependency-name: github.com/distribution/distribution/v3
  dependency-version: 3.0.0
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* update go

* use constant format strings

* f

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Andrew Lavery <laverya@umich.edu>
2025-04-21 16:45:47 +00:00
Xav Paice 86b7e54466 Revert "feat: save YAML spec used to generate support bundle/preflight" (#1715)
Revert "feat: save YAML spec used to generate support bundle/preflight (#1713)"

This reverts commit f6f51acbd5.
2025-01-06 09:42:58 +11:00
Gerard Nguyen f6f51acbd5 feat: save YAML spec used to generate support bundle/preflight (#1713)
* save YAML spec of support bundle

* save YAML spec of preflight

* add unit test

* redact TLS private key by default in output spec

* update YAML path for HTTP TLS redactor
2025-01-04 11:35:43 +11:00
Ash ecc92b1e3e [bug] Quick fix for handling non 200 status codes when loading specs from URI (#1695)
* Quick fix for handling non 200 status codes when loading specs from URI

Go http client already handles 3xx responses for us

* note
2024-11-25 15:04:38 +00:00
Gerard Nguyen 7bb88e6b83 feat: ensure Copy collector run last (#1688)
* ensure Copy collector run last

* * add unit test
* reorder in Preflight as well
2024-11-15 10:59:38 +11:00
deeeea7cec exec remote host collectors in a daemonset (#1671)
Co-authored-by: Gerard Nguyen <gerard@replicated.com>
Co-authored-by: Dexter Yan <yanshaocong@gmail.com>
2024-11-12 08:47:24 +13:00
Evans MungaiandDexter Yan d25aa7d0ea fix: Do not fail analysis if node list does not exist (#1678)
* fix: Do not error if node list does not exist

Signed-off-by: Evans Mungai <evans@replicated.com>

* fix test fail

---------

Signed-off-by: Evans Mungai <evans@replicated.com>
Co-authored-by: Dexter Yan <yanshaocong@gmail.com>
2024-11-08 09:53:03 +13:00
Diamon Wiggins 06506ed95d Fix remote host collection RBAC checks (#1672)
* fix remote host collection rbac checks

* move saveNodeList into collectRemoteHost function

* fix resource attribute list and retrieve namespace from kubeconfig

* revert change to set a default namespace from kubeconfig

* remove duplicate code
2024-11-07 10:07:27 -05:00
Dexter Yan 350418c6e9 feat(host-collector): add progress for host collector (#1659) 2024-10-25 15:34:09 +13:00
Dexter Yan 0d21eed5f8 fix(support): add missing host collectors for ParseSupportBundle (#1656)
* fix(support): add missing host collectors for ParseSupportBundle

* update

* add host ananlyers
2024-10-22 13:07:44 +13:00
b88bc8ddf7 Refactor Multi Node Analyzers (#1646)
* initial refactor of host os analyzer

* refactor remote collect analysis

---------

Signed-off-by: Evans Mungai <evans@replicated.com>
Co-authored-by: Gerard Nguyen <gerard@replicated.com>
Co-authored-by: Evans Mungai <evans@replicated.com>
2024-10-22 10:45:50 +13:00
Evans Mungai 9c24ab6067 chore: Remove preempted deprecation warnings (#1655)
Signed-off-by: Evans Mungai <evans@replicated.com>
2024-10-22 08:35:36 +11:00
Gerard Nguyen 289102f16d bug: fix nil check in host collector filter (#1653)
* add nil check in filter host collector
2024-10-18 15:58:32 +11:00
Diamon WigginsandDexter Yan 8105fa00e9 Refactor Remote Host Collection (#1633)
* refactor remote collectors

* add remotecollect params struct

* remove commented checkrbac function

* removed unused function

* add temp comments

* refactor to not require RemoteCollect method per collector

* removed unneeded param

* removed unneeded param

* more refactor

* more refactor

* remove unneeded function

* remove debug print

* fix analyzer results

* move rbac to separate file

* be more specific with rbac function name

* fix imports

* fix node list file

* make k8s rest client config consistent with in cluster collection

* add ctx and otel tracing

* add test for allCollectedData

* move runHostCollectorsInPod to spec instead of metadata

* make generate

* fix broken references to supportbundle metadata

* add e2e tests

* update loader tests

* fix tests

* fix hostos remote collector spec

* update remoteHostCollectrs.yaml

---------

Co-authored-by: Dexter Yan <yanshaocong@gmail.com>
2024-10-09 18:38:49 +13:00
Gerard Nguyen c1c4b612a4 feat: [sc-113128] Create node list file before running remote host collector (#1632)
* create node list
2024-10-01 14:43:24 +10:00
Dexter Yan e97b9613a5 feat(support-bundle): add runHostCollectorsInPod in spec (#1608) 2024-09-20 11:57:58 -05:00
Evans Mungai aea4f7c87c feat: Optionally save preflight bundles to disk (#1612)
* feat: Optionally save preflight bundles to disk

Signed-off-by: Evans Mungai <evans@replicated.com>

* Add e2e test of saving preflight bundle

Signed-off-by: Evans Mungai <evans@replicated.com>

* Update cli docs

Signed-off-by: Evans Mungai <evans@replicated.com>

* Expose GetVersionFile function publicly

Signed-off-by: Evans Mungai <evans@replicated.com>

* Store analysis.json file in preflight bundle

Signed-off-by: Evans Mungai <evans@replicated.com>

* Run go fmt when running lint fixers

Signed-off-by: Evans Mungai <evans@replicated.com>

* Always generate a preflight bundle in CLI

Signed-off-by: Evans Mungai <evans@replicated.com>

* Print saving bundle message to stderr

Signed-off-by: Evans Mungai <evans@replicated.com>

* Revert changes in docs directory

Signed-off-by: Evans Mungai <evans@replicated.com>

* Use NewResult constructor

Signed-off-by: Evans Mungai <evans@replicated.com>

* Log always when preflight bundle is saved to disk

Signed-off-by: Evans Mungai <evans@replicated.com>

---------

Signed-off-by: Evans Mungai <evans@replicated.com>
2024-09-16 23:36:52 +01:00
Evans Mungai e4363a1e50 fix (supportbundle): Add default collectors when expected (#1418)
* fix (supportbundle): Add default collectors when expected

* Remove unnecessary change

* Add default collectors to a empty spec

* Add more tests
2024-01-08 11:32:04 +00:00
Evans Mungai e5e26eea14 fix(support-bundle): default in-cluster collectors in host support bundle (#1394)
* fix(support-bundle): default in-cluster collectors in host support bundle

Ensure cluster-resources and cluster-info collectors are present only
when a support bundle spec contains in-cluster collectors.

* Various improvements

* Improve error messages
* Util function appending elements to a nil slice that allows adding
  specs to an empty slice of collectors/analysers/redactors

* Fix failing test
2023-11-27 18:33:02 +00:00
Archit Sharma 7038da85b1 Velero analyzer (#1366)
* feat: add velero analyzer (#806)

  * updated schema
  * analyzer without collector
  * tests
  * covers deprecated Restic repository type
  * velero version from deployment image to check deprecated type
  * read for both velero pod kinds (velero*, node-agent*)

---------

Signed-off-by: Archit Sharma <archit@pm.me>
2023-11-03 18:41:17 +05:30
Evans Mungai 15a4802cd2 feat: Add dry run flag to print support bundle specs to std out (#1337)
* Add dry-run flag

* No traces on dry run

* More refactoring

* More updates to support bundle binary

* More refactoring changes

* Different approach of loading specs from URIs

* Self review

* More changes after review and testing

* fix how we parse oci image uri

* Remove unnecessary comment

* Add missing file

* Fix failing tests

* Better error check for no collectors

* Add default collectors when parsing support bundle specs

* Add missed test fixture

* Download specs with correct headers

* Fix typo
2023-10-10 18:43:32 +01:00
Pavan Sokke Nagaraj 39314ef200 chore: export error ErrInsufficientPermissionsToRun and func ShowTextResultsStructured (#1297)
* chore: move ErrInsufficientPermissions to collect

* chore: export func ShowTextResultsStructured

* chore: rename to ErrInsufficientPermissionsToRun
2023-08-08 14:17:15 -04:00
Evans Mungai 401dfe2c57 feat: add loader APIs to load specs from raw troubleshoot spec (#1202)
* feat: add loader APIs to load specs from a list of yaml docs

The change introduces a loader package that will contain loader
public APIs. The aim of these APIs will be to, given any source of
troubleshoot specs, the loaders will fetch the specs and parse out
all troubleshoot objects that can be extracted.

* Some refactoring

* Some more changes

* More changes caught when testing vendor portal

* Add tests and rename Troubleshoot kinds struct

* Additional test

* Handle ConfigMap and Secrets with multiple specs in them

* Fix failing test

* Revert multidoc split implementation

* Fix merge conflict

* Change LoadFromXXX functions to a single LoadSpecs function
2023-06-06 16:48:29 -04:00
Dexter Yan 8fe5bffae4 feat(analyzer): remove duplicated analyzers (#1056)
feat(analyzer): add DedupAnalyzers
2023-03-22 11:45:41 -04:00
Evans Mungai 546ffde14b feat: use klog as the default logging library (#1008) 2023-02-24 18:24:51 +00:00
Evans Mungai 100f9a13b6 feat: Record summary of execution times of support bundle operations (collect/redact/analyse) (#935)
When running a support bundle, we want to know how long each operation
(collect, redact, analyze) takes. This commit adds a new trace exporter
that records the start and end times of each operation, and then prints
a summary of the execution. The summary is also stored in the support
bundle.

Related to #923
2023-02-07 09:50:21 +00:00
yunju.lly 1114902707 refactor: keep support bundle concat logic to be consistent with Preflight concat (#1002)
* refactor: keep support bundle concat logic to be consistent with Preflight
* test: add tests for support bundle spec concat function
2023-02-03 11:05:11 +00:00
Diamon WigginsandEvans Mungai 4fca6aff98 Deduplication for In-Cluster Collectors (#972)
* adding dedup for in cluster collectors

* add tests

* return collector as is whenever marshalling to json fails

---------

Co-authored-by: Evans Mungai <evans@replicated.com>
2023-02-01 14:14:43 -05:00
Diamon Wiggins 2fcdc77cd3 Standardize Cluster Resources Collector File Paths (#971)
* using const for cluster resources k8s objects to standardize directories and files
2023-01-25 13:34:15 -05:00
Evans Mungai a523551da9 feat(redactors): Run redactors on an existing support bundle (#887)
* feat(redactors): Run redactors on an existing support bundle

Add redact subcommand to support-bundle to allow running redactors on an
existing bundle to creating a new redacted bundle.

The command will be launched like so

support-bundle redact <redactor urls> --bundle support-bundle.tar.gz

Fixes: #705
2023-01-03 18:05:15 +00:00
Dexter Yan be26462c19 feat(cluster_resources): increase default client burst and qps (#920)
* feat(collect): add client burst and qps
2022-12-22 09:49:42 +13:00
Diamon Wiggins a4c4b24056 Deduplication for Cluster Resources Collector (#832)
* add dedup for cluster resources collector
* restructure both collect.go in both pkg/supportbundle and pkg/preflight to be more similar for eventual refactor
2022-12-07 15:10:31 -04:00
Evans Mungai fbbcf87405 feat(collectors): Store all pod logs in cluster-resources directory (#821)
* feat(collectors): Store all pod logs in cluster-resources directory

All pod logs collected by the logs collector will now be stored in
/cluster-resources/pods/logs/[namespace]/[pod]/[container].log. This
will provide consistency and allow sbctl to find the logs when we run
`kubectl logs <pod>`. To allow backwards compatibility, symlinks of the
log files will be created in the current expected locations.

Closes: #744
2022-11-22 07:10:34 +13:00
Diamon Wiggins c34d80c300 Discover Redactors in Cluster (#827)
Adds the ability to search for support bundle specs and redactors, in both configmaps and secrets
2022-11-10 17:36:51 +13:00
Xav Paice 3513eeca19 Ensure clusterResources is added prior to other collectors (#768)
This change ensures that the clusterResources collector runs prior to any others
in order to not collect info on pods that collectors run during collection.

Additionally centralizes functions that are common to all collection to make future
maintenance simpler.

Fixes: #767
2022-11-01 12:16:01 +13:00
Diamon Wiggins 04c7a18da3 Fix Progress Callback for Support Bundle Collection (#781)
fix progress callback for support bundle and revert collector title changes
2022-10-14 12:29:59 -04:00
ada mancini eb40b9422f implement uri: field (#702) 2022-10-05 15:35:55 +13:00
Diamon Wiggins c7b84ad1e5 Refactor in-clusters collectors to use struct per collector (#670)
refactor in-clusters collectors to use struct per collector
2022-10-03 13:53:05 -04:00
Diamon Wiggins e53871b4dc adding tests 2022-09-13 23:00:57 -04:00
Diamon Wiggins ed34afc51b adding back default collectors 2022-09-11 18:33:35 -04:00
Diamon Wiggins d4e5c0dba4 discovery support bundle specs from secrets matching label selector 2022-09-11 18:28:24 -04:00
Dan Jones d70ecf7e5f Add a rough test for bundle concatenation
Signed-off-by: Dan Jones <danj@replicated.com>
2022-09-08 17:16:38 +01:00
Dan Jones b5a56e5239 Remove redundant loops
Signed-off-by: Dan Jones <danj@replicated.com>
2022-09-07 09:14:09 +01:00
Dan Jones b6f8ac211e Format
Signed-off-by: Dan Jones <danj@replicated.com>
2022-09-05 15:54:54 +01:00
Dan Jones 63510fc89d Moved ConcatSpec to supportbundle package and clean up dead code
Signed-off-by: Dan Jones <danj@replicated.com>
2022-09-05 14:12:29 +01:00