41 Commits
Author SHA1 Message Date
Benjamin YangandGitHub c67775c1a0 feat: clean tokenization system implementation (#1874)
Core tokenization functionality with minimal file changes:

 Core Features:
- Intelligent tokenization engine (tokenizer.go)
- Context-aware secret classification (PASSWORD, APIKEY, DATABASE, etc.)
- Cross-file correlation with deterministic HMAC-SHA256 tokens
- Optional encrypted mapping for token→original value resolution

 Integration:
- CLI flags: --tokenize, --redaction-map, --encrypt-redaction-map
- Updated all redactor types: literal, single-line, multi-line, YAML
- Support bundle integration with auto-upload compatibility
- Backward compatibility: preserves ***HIDDEN*** when disabled

 Production Ready:
- Only 11 essential files (vs 31 in original PR)
- No excessive test files or documentation
- Clean build, all functionality verified
- Maintains existing redaction behavior by default

Token format: ***TOKEN_<TYPE>_<HASH>*** (e.g., ***TOKEN_PASSWORD_A1B2C3***)
2025-09-30 15:04:23 -05:00
Ethan MosbaughandGitHub b80f38a9a0 fix(redact): multi-line redactors strip empty lines (#1742) 2025-02-20 21:55:05 -05:00
Xav PaiceandGitHub 86b7e54466 Revert "feat: save YAML spec used to generate support bundle/preflight" (#1715)
Revert "feat: save YAML spec used to generate support bundle/preflight (#1713)"

This reverts commit f6f51acbd5.
2025-01-06 09:42:58 +11:00
Gerard NguyenandGitHub f6f51acbd5 feat: save YAML spec used to generate support bundle/preflight (#1713)
* save YAML spec of support bundle

* save YAML spec of preflight

* add unit test

* redact TLS private key by default in output spec

* update YAML path for HTTP TLS redactor
2025-01-04 11:35:43 +11:00
Evans MungaiandGitHub e2adfa3774 fix: Correct selector used to find installer specs when redacting (#1456) 2024-02-07 10:36:15 +00:00
Evans MungaiandGitHub 86279b4ec4 chore(redactors): memory consumption improvements (#1332)
* Document additional go tool profiling flags

* Add a regex cache to avoid compiling regular expressions all the time

* Reduce max buffer capacity

* Prefer bytes to strings

Strings are immutable and hence we need to create a new one
all the time when operation on them

* Some more changes

* More bytes

* Use writer.Write instead of fmt.FPrintf

* Clear regex cache when resetting redactors

* Logs errors when redactors error since they get swallowed

* Add an improvement comment

* Limit the number of goroutines spawned when redacting

* Minor improvement

* Write byte slices one at a time instead of concatenating them first

* Add a test for writeBytes

* Additional tests
2023-09-15 13:09:21 -04:00
Dexter YanandGitHub d730be0fae feat(redact): use a scan regex for default redact rule of lines to improve cpu usage and reduce time cost (#1291) 2023-08-25 15:11:04 +12:00
Evans MungaiandGitHub 546ffde14b feat: use klog as the default logging library (#1008) 2023-02-24 18:24:51 +00:00
Diamon WigginsandGitHub 2fcdc77cd3 Standardize Cluster Resources Collector File Paths (#971)
* using const for cluster resources k8s objects to standardize directories and files
2023-01-25 13:34:15 -05:00
Xav PaiceandGitHub c85bf9a9a6 BREAKING: remove IP address redaction (#734)
This change removes the IPv4 address redaction which previously ran by default on all
support bundle collections.

Folks that want to redact IPv4 addresses will need to add that redactor manually to their redactor specs.
2022-11-30 08:42:42 +13:00
Xav Paice 4acd7e4951 Add Ceph auth redactors
Add automatic redactors for Ceph auth_dump json which includes auth keys.
2022-06-27 17:50:38 +12:00
Salah Aldeen Al SalehandGitHub c7c21e88fb fix custom resources redaction file path (#480)
* fix custom resources redaction file path
2021-11-02 12:36:15 -07:00
cf03503216 feat: Collect custom resources (#447)
* feat: Collect custom resources
Co-authored-by: Martin Hrabovcin<mhrabovcin@users.noreply.github.com>

Co-authored-by: Andrew Reed <andrew@replicated.com>
2021-10-21 16:49:59 -05:00
divolgin 0e8bedc281 Save collector data to disk directly 2021-09-29 00:15:02 +00:00
divolgin 53e8ee5495 Add a default redactor for postgres db connection string 2021-06-29 22:38:32 +00:00
Marc Campbell c7fdec0291 Removing Scopeagent 2021-01-28 18:22:48 +00:00
divolgin a0ce85ae1e Adding troubleshoot.sh/v1beta2 2020-09-01 19:57:11 +00:00
Andrew Lavery 6e874483b6 update yaml key name, fixup example troubleshoot 2020-06-15 13:55:07 -04:00
Andrew Lavery e66d12dab2 combine multiline and single line regex 2020-06-15 11:15:50 -04:00
Andrew Lavery a561254756 break apart redactor type 2020-06-09 18:43:44 -04:00
Marc Campbell 661d7fcfc4 Update names 2020-05-27 13:10:46 -07:00
Andrew Lavery b9c3ad2635 better default redactor names 2020-05-27 14:36:40 -04:00
Andrew Lavery f03916cfd2 include whether a redaction is from a default redactor in reports 2020-05-27 13:46:39 -04:00
Andrew Lavery 553718eada fully spell out default redactor names 2020-05-26 19:33:40 -04:00
Andrew Lavery 171c440f27 improve generated names for custom redactors 2020-05-26 18:13:48 -04:00
Andrew Lavery 869a9eafd3 actually increment withinRedactNum 2020-05-26 18:07:23 -04:00
Andrew Lavery 6c61040f74 default redactor name improvements
for instance, defaultRegex.SECRET_ACCESS_KEY instead of the raw regex
2020-05-26 17:15:30 -04:00
Andrew Lavery 0d84c62c78 add yaml/json annotations for redaction reports 2020-05-20 14:58:37 -04:00
Andrew Lavery 5a2c153026 add tests for redaction reports 2020-05-20 14:36:07 -04:00
Andrew Lavery fb2f028fb5 add methods to get and clear redactions 2020-05-20 10:28:46 -04:00
Andrew Lavery d37ace6e51 better naming scheme that does not leak secrets 2020-05-19 17:21:55 -04:00
Andrew Lavery b6e92e1bdf record yaml redactions 2020-05-14 18:49:30 -04:00
Andrew Lavery 27369acbf4 plumbing for literal/regex/multiline reports 2020-05-14 16:00:58 -04:00
Andrew Lavery 15b100e0ee support * when redacting yaml maps 2020-04-27 22:01:04 -04:00
Andrew Lavery 56150c3345 add support for redacting items within yaml documents 2020-04-27 20:46:52 -04:00
Andrew Lavery 8f594e8764 add support for custom multiline redactors, add yaml redactors to spec 2020-04-27 16:50:25 -04:00
Andrew Lavery 37f75c1a3f use gobwas/glob, not filepath, file matching 2020-04-23 13:13:46 -04:00
Andrew Lavery d016e3269c add global and per-collector redactors
add redact type, and begin wiring global redactors

use per-collector redactors

add a test of the 'data' collector and redaction

handle literal string replacements

remove redundant types and redact calls

add proper redactor type, foundations of global redactors

accept global redactors from the CLI, include sample redaction spec
2020-04-16 14:03:00 -04:00
Frank Shotwell f8870e5fc0 move scope instrumentation to handle table-driven tests 2020-01-31 06:25:07 +00:00
Frank Shotwell 10b2941c57 Add scope test instrumentation 2020-01-30 21:01:06 +00:00
divolgin 9942f4bf65 Adding basic redaction functionality 2019-07-17 22:57:19 +00:00