From a46548c1194aadfac325d33c9b3231f81953aef3 Mon Sep 17 00:00:00 2001 From: Xav Paice Date: Fri, 12 Aug 2022 14:28:10 +1200 Subject: [PATCH 1/2] Add design proposal for mergeable specs --- design/proposal-mergeable-spec.md | 143 ++++++++++++++++++++++++++++++ 1 file changed, 143 insertions(+) create mode 100644 design/proposal-mergeable-spec.md diff --git a/design/proposal-mergeable-spec.md b/design/proposal-mergeable-spec.md new file mode 100644 index 00000000..84f82549 --- /dev/null +++ b/design/proposal-mergeable-spec.md @@ -0,0 +1,143 @@ +# Provide the ability to specify multiple Troubleshoot specs in one run of support-bundle or preflight + +Troubleshoot doesn't have a modular way for different components to specify specifications. If a software project wanted to include project specific items the end user would have a hard time knowing where to find those and how to collect them. Furthermore if multiple projects were to do this the user would have to run troubleshoot multiple times. + +Ideally troubleshoot would allow merging of specs to allow building a spec either automatically or influenced by user input to target specific needs. + +## Goals + +Primary goal: Allow folks that develop a particular component to maintain Troubleshoot specs for that component, including Vendors for their application. + +Long term goal: allow folks to update the component specs without needing to run any other upgrades. + +This proposal initially adds the ability to supply multiple support bundle specs for a single run of support-bundle and/or preflight so that: + +* when Troubleshoot runs, the spec it uses is an aggregate of multiple specs from the same source - yaml file, URL, configMap, or (new) CRD +* each software component can contribute a Troubleshoot spec that is specific to that software +* Ownership of each individual spec per component transfers to the owner of that component + +This will: + +* Ensure support bundles generated by Vendor’s end customers are comprehensive and contain all needed information for a Replicated engineer to investigate +* Allow multiple specs from a Vendor, should they run multiple applications maintained by different teams +* Allow contribution of specific specs for each addon to be maintained by either kURL, or even upstream (e.g. in the Prometheus operator project sources). +* Empower Replicated and the open source community to continually update specs for components to prevent issues recurring across multiple customers/vendors. + +## Non Goals + +* Compatibility with previous URL, configMap and yaml specs, both Redactors and SupportBundle types +* Maintain the ability to read secrets from the Kubernetes API + +## Background + +When end users of Troubleshoot create a support bundle from the CLI, they use a single spec that is provided from: + +* the Vendor’s application, or +* http://kots.io, or +* some other single example spec, e.g. https://github.com/replicatedhq/troubleshoot-specs + +When Replicated customers collect support bundles from the KOTS UI, the spec is a [merged](https://github.com/replicatedhq/kots/blob/main/pkg/supportbundle/spec.go#L426) combination of the following: + +* The [default spec](https://github.com/replicatedhq/kots/blob/main/pkg/supportbundle/defaultspec/spec.yaml) in the kots code +* The spec provided by the Vendor’s application bundle +* This merged spec is deduplicated by kots [here](https://github.com/replicatedhq/kots/blob/main/pkg/supportbundle/spec.go#L217). + +Redactors are [hard coded](https://github.com/replicatedhq/troubleshoot/blob/main/pkg/redact/redact.go#L160) into Troubleshoot as well as supplied in the spec. + +Although we’re currently making strides in improving the Troubleshoot project by creating new collectors, analyzers, and specs, we have no way to more quickly deliver Troubleshoot improvements to installations in the field without updating the kots application and pushing a new release, or upgrading kots to a new version. + +When folks find a support issue that could have been identified by Troubleshoot, we ask them to write a new collector, and/or analyzer, for that information. However, if we do that, the new collector/analyzer is not easily available to end users. + +Similar to the issues we see with vendors defining their own kURL specs, we often find that vendors omit key collectors and analyzers from their spec making the troubleshooting process inefficient when an engineer needs to request additional information after the creation of a support case. + +## High-Level Design + +Add CRD support to Troubleshoot: + +* Design a custom resource (CRD) that allows adding spec(s) to the Kubernetes cluster using `kubectl`. There is no need to extend this to use [API server aggregation](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#api-server-aggregation). +* Update Troubleshoot to allow consuming the first object found of the new custom resource, by default - i.e. if there are no specs provided on the CLI or entrypoint, use CRD +* Once merge is implemented, update Troubleshoot to consume and merge all the objects in that CRD. +* To minimize code changes in Troubleshoot, we could implement two CRDs, of `type: SupportBundle` and `type: Redactor`. However, there is no specific need to separate the two and it maybe advantageous to keep things simple, combining them. + +Allow multiple specs to be merged by Troubleshoot: + +* create an interface for collectors, analyzers and redactors to merge +* each collector/analyzer/redactor can use a generic implementation of the interface for the merge, or can use a specific one for that task if there are 'special' rules for that particular collector. +* the spec merge code in kots can, at this stage, be removed once kots ships CRDs for the Troubleshoot specs + +Spec sources: + +* Alter Troubleshoot to accept multiple yaml files on the CLI +* Create a CRD containing a spec, have Troubleshoot search for spec CRDs and combine them all when run +* maintain the URL compatibility as is +* maintain the configMap compatibilty as is + +## Detailed Design + +CRD: to be designed + +Merge: + +* define a new interface that provides the merge functionality for collectors, analyzers and redactors +* there should be a generic implementation of that interface which is used by default for all objects. This can simply use `append()`. +* specific collectors may have alternative implementations of the interface where overrides are required. + +Particular overrides known at this point in time: + +* Longhorn and Ceph have a namespace config, if multiple collectors are specified with different namespaces then each should be collected +* clusterResources has an option namespaces config, these should be merged and deduplicated +* Two runPod collectors with the same name and different commands are unmergable. Only one should be run and an error logged. +* Two configMap collectors can be deduplicated and merged depending on the configurations +* Two logs collectors likely only need deduplication to prevent collecting the same thing twice + +## Limitations + +### Visibility +Right now Vendors have full control over what will be included in their support bundles. If we move to a model where collectors and analyzers are added at runtime which they didn’t specify, this could present an issue of trust for them. Vendors would still be able to redact as needed, but not sure if this alone is sufficient. + +Note that the current implementation for Vendors already merges the spec included in kots along with the Vendor's spec. + +We therefore need to be sure the Troubleshoot client allows for selection of what to run. While the default would be to run everything, a user should be able to just run specific SupportBundle specs based on labels, namespaces, etc. By doing this no functionality is lost if a vendor wants to run only their spec. + +### Airgap + +Some of the spec locations such as URL wouldn’t work here. There may be other considerations around airgap as well. + +Airgap installs are intentionally feature limited. Failure to retrieve new versions of specs should fail gracefully and proceed with the run. + +As a follow up feature to improve support bundle collection Troubleshoot could gain some abilities to help manage and upgrade SupportBundles found in a cluster in a fashion that allows the user to split the discovery, download, and upgrade so they could be run on different machines. + +An example implementation would be: + +* A command to find, deduplicate, and output a list of SupportBundles in the cluster. The list would include at least the upstream URL and version currently installed in the cluster. +* A command to take the output of the above command and check for updates, downloading all updated SupportBundles into a single .tgz file. +* A command to apply the above .tgz file to the cluster upgrading any existing SupportBundles with the newer versions. + +## Assumptions + +* all deployments are able to deploy CRDs to the Kubernetes cluster + +## Testing + +## Alternatives Considered + +### Use the existing URL functionality in Troubleshoot + +Proposal: Provide a custom URL for each install, that when called by Troubleshoot returns a spec composed of the ‘latest’ for every component spec specified in that installation, much like we do with https://kurl.sh/latest now. + +Pros: + +* No changes required to Troubleshoot +* Always use the latest specs for components other than the Vendor’s application + +Cons: + +* Would not work for airgap installations +* Vendors need to update their application in order to get new Vendor specs +* Replicated would need to provide a new API/web service to host the specs +* Discourages individual projects from maintaining Troubleshoot specs for that project + + +## Security Considerations + +Passing control of component specs to individual projects presents a possibility of reducing the amount of review a spec goes through for each update, allowing a spec provided by, say, a kURL addon to run collectors. The current implementation relies on the kots review process for default specs, plus the review of individual example specs. Depending on the final implementation, it is unlikely that this change itself increases any risk since the delivery of automated and default specs to Troubleshoot is maintained within kots, and/or Troubleshoot itself much as it is already. \ No newline at end of file From 45170304733ae9f5144e1d7ee5ff3b1a455cac03 Mon Sep 17 00:00:00 2001 From: Chris Sanders Date: Fri, 12 Aug 2022 08:52:32 -0500 Subject: [PATCH 2/2] Adjust some phrasing to use less Replciated specific terminology --- design/proposal-mergeable-spec.md | 47 ++++++++++++++----------------- 1 file changed, 21 insertions(+), 26 deletions(-) diff --git a/design/proposal-mergeable-spec.md b/design/proposal-mergeable-spec.md index 84f82549..b13641ea 100644 --- a/design/proposal-mergeable-spec.md +++ b/design/proposal-mergeable-spec.md @@ -18,10 +18,9 @@ This proposal initially adds the ability to supply multiple support bundle specs This will: -* Ensure support bundles generated by Vendor’s end customers are comprehensive and contain all needed information for a Replicated engineer to investigate -* Allow multiple specs from a Vendor, should they run multiple applications maintained by different teams -* Allow contribution of specific specs for each addon to be maintained by either kURL, or even upstream (e.g. in the Prometheus operator project sources). -* Empower Replicated and the open source community to continually update specs for components to prevent issues recurring across multiple customers/vendors. +* Ensure support bundles generated by users are comprehensive and contain all needed information for a project maintainer to action +* Allow multiple specs from different projects, should a cluster have multiple projects using Troubleshoot +* Software developers to continually update specs for components to easily identify known issues and reduce support noise. ## Non Goals @@ -30,13 +29,13 @@ This will: ## Background -When end users of Troubleshoot create a support bundle from the CLI, they use a single spec that is provided from: +When users of Troubleshoot create a support bundle from the CLI, they use a single spec that is provided from: -* the Vendor’s application, or -* http://kots.io, or -* some other single example spec, e.g. https://github.com/replicatedhq/troubleshoot-specs +* A secret installed in the cluster by an application such as [KOTS](kots.io) +* A url like +* some other single example spec, e.g. -When Replicated customers collect support bundles from the KOTS UI, the spec is a [merged](https://github.com/replicatedhq/kots/blob/main/pkg/supportbundle/spec.go#L426) combination of the following: +When KOTS collects a support bundles from the KOTS UI, the spec is a [merged](https://github.com/replicatedhq/kots/blob/main/pkg/supportbundle/spec.go#L426) combination of the following: * The [default spec](https://github.com/replicatedhq/kots/blob/main/pkg/supportbundle/defaultspec/spec.yaml) in the kots code * The spec provided by the Vendor’s application bundle @@ -48,7 +47,7 @@ Although we’re currently making strides in improving the Troubleshoot project When folks find a support issue that could have been identified by Troubleshoot, we ask them to write a new collector, and/or analyzer, for that information. However, if we do that, the new collector/analyzer is not easily available to end users. -Similar to the issues we see with vendors defining their own kURL specs, we often find that vendors omit key collectors and analyzers from their spec making the troubleshooting process inefficient when an engineer needs to request additional information after the creation of a support case. +Some of the useful features of Troubleshoot are actually implemented as part of KOTS, while both open source Troubleshoot should address this independently. ## High-Level Design @@ -56,13 +55,13 @@ Add CRD support to Troubleshoot: * Design a custom resource (CRD) that allows adding spec(s) to the Kubernetes cluster using `kubectl`. There is no need to extend this to use [API server aggregation](https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#api-server-aggregation). * Update Troubleshoot to allow consuming the first object found of the new custom resource, by default - i.e. if there are no specs provided on the CLI or entrypoint, use CRD -* Once merge is implemented, update Troubleshoot to consume and merge all the objects in that CRD. +* Once merge is implemented, update Troubleshoot to consume and merge all the instances of the CRD. * To minimize code changes in Troubleshoot, we could implement two CRDs, of `type: SupportBundle` and `type: Redactor`. However, there is no specific need to separate the two and it maybe advantageous to keep things simple, combining them. Allow multiple specs to be merged by Troubleshoot: -* create an interface for collectors, analyzers and redactors to merge -* each collector/analyzer/redactor can use a generic implementation of the interface for the merge, or can use a specific one for that task if there are 'special' rules for that particular collector. +* create an interface for collectors, analyzers, and redactors to merge +* each collector/analyzer/redactor can use a generic implementation of the interface for the merge, or can use a specific one for that task if that particular collector will benefit from a more intelligent merge. * the spec merge code in kots can, at this stage, be removed once kots ships CRDs for the Troubleshoot specs Spec sources: @@ -70,7 +69,7 @@ Spec sources: * Alter Troubleshoot to accept multiple yaml files on the CLI * Create a CRD containing a spec, have Troubleshoot search for spec CRDs and combine them all when run * maintain the URL compatibility as is -* maintain the configMap compatibilty as is +* maintain the configMap compatibility as is ## Detailed Design @@ -78,13 +77,12 @@ CRD: to be designed Merge: -* define a new interface that provides the merge functionality for collectors, analyzers and redactors +* define a new interface that provides the merge functionality for collectors, analyzers, and redactors * there should be a generic implementation of that interface which is used by default for all objects. This can simply use `append()`. * specific collectors may have alternative implementations of the interface where overrides are required. Particular overrides known at this point in time: -* Longhorn and Ceph have a namespace config, if multiple collectors are specified with different namespaces then each should be collected * clusterResources has an option namespaces config, these should be merged and deduplicated * Two runPod collectors with the same name and different commands are unmergable. Only one should be run and an error logged. * Two configMap collectors can be deduplicated and merged depending on the configurations @@ -93,11 +91,8 @@ Particular overrides known at this point in time: ## Limitations ### Visibility -Right now Vendors have full control over what will be included in their support bundles. If we move to a model where collectors and analyzers are added at runtime which they didn’t specify, this could present an issue of trust for them. Vendors would still be able to redact as needed, but not sure if this alone is sufficient. -Note that the current implementation for Vendors already merges the spec included in kots along with the Vendor's spec. - -We therefore need to be sure the Troubleshoot client allows for selection of what to run. While the default would be to run everything, a user should be able to just run specific SupportBundle specs based on labels, namespaces, etc. By doing this no functionality is lost if a vendor wants to run only their spec. +Users may collect more than intended with the new implementation if all available CRD objects are run. We therefore need to be sure the Troubleshoot client allows for selection of what to run. While the default would be to run everything, a user should be able to just run specific SupportBundle specs based on labels, namespaces, etc. By doing this no functionality is lost. ### Airgap @@ -105,7 +100,7 @@ Some of the spec locations such as URL wouldn’t work here. There may be other Airgap installs are intentionally feature limited. Failure to retrieve new versions of specs should fail gracefully and proceed with the run. -As a follow up feature to improve support bundle collection Troubleshoot could gain some abilities to help manage and upgrade SupportBundles found in a cluster in a fashion that allows the user to split the discovery, download, and upgrade so they could be run on different machines. +As a follow up feature to improve support bundle collection Troubleshoot could gain some abilities to help manage and upgrade SupportBundles found in a cluster in a fashion that allows the user to split the discovery, download, and upgrade so they could be run on different machines. An example implementation would be: @@ -123,7 +118,7 @@ An example implementation would be: ### Use the existing URL functionality in Troubleshoot -Proposal: Provide a custom URL for each install, that when called by Troubleshoot returns a spec composed of the ‘latest’ for every component spec specified in that installation, much like we do with https://kurl.sh/latest now. +Proposal: Provide a custom URL for each install, that when called by Troubleshoot returns a spec composed of the ‘latest’ for every component spec specified in that installation, much like we do with now. Pros: @@ -133,11 +128,11 @@ Pros: Cons: * Would not work for airgap installations -* Vendors need to update their application in order to get new Vendor specs -* Replicated would need to provide a new API/web service to host the specs +* Software developers need to update their application in order to get new specs from all dependencies +* Replicated would need to provide a new API/web service to host the specs, reducing the general community applicability of troubleshoot as a stand alone project * Discourages individual projects from maintaining Troubleshoot specs for that project - ## Security Considerations -Passing control of component specs to individual projects presents a possibility of reducing the amount of review a spec goes through for each update, allowing a spec provided by, say, a kURL addon to run collectors. The current implementation relies on the kots review process for default specs, plus the review of individual example specs. Depending on the final implementation, it is unlikely that this change itself increases any risk since the delivery of automated and default specs to Troubleshoot is maintained within kots, and/or Troubleshoot itself much as it is already. \ No newline at end of file +Passing control of component specs to individual projects presents a possibility of reducing the amount of review a spec goes through for each update, allowing a spec provided by, say, a kURL addon to run collectors. The current implementation relies on the kots review process for default specs, plus the review of individual example specs. Depending on the final implementation, it is unlikely that this change itself increases any risk since the delivery of automated and default specs to Troubleshoot is maintained within kots, and/or Troubleshoot itself much as it is already. +