From 2cc5e902ad5270665d5e32b1ed583da2b5d32caa Mon Sep 17 00:00:00 2001 From: Zack Brady Date: Fri, 20 Oct 2023 03:49:43 -0400 Subject: [PATCH 1/6] updated readme --- README.md | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index 5cba003..e474d8d 100644 --- a/README.md +++ b/README.md @@ -1,34 +1,35 @@ -# Hauler: Airgap Swiss Army Knife +# Rancher Government Hauler -> ⚠️ This project is still in active development and _not_ GA. While a lot of the core features are ready, we're still adding a _ton_, and we may break API compatibility as we change features in newer versions. +## Airgap Swiss Army Knife -`hauler` simplifies the airgap experience without forcing you to adopt a specific workflow for your infrastructure or application. +> ⚠️ This project is still in active development and *not* Generally Available (GA). Most of the core functionality and features are ready, but may have breaking changes. Please review the [Release Notes](https://github.com/rancherfederal/hauler/releases) for more information! -To accomplish this, it focuses strictly on two of the biggest airgap pain points: +`Rancher Government Hauler` simplifies the airgap experience without requiring users to adopt a specific workflow. **Hauler** simplifies the airgapping process, by representing assets (images, charts, files, etc...) as content and collections to allow users to easily fetch, store, package, and distribute these assets with declarative manifests or through the command line. -* content collection -* content distribution +`Hauler` does this by storing contents and collections as OCI Objects and Specifications and allowing users to serve contents and collections via an embedded registry and fileserver. Additionally, `Hauler` has the ability to inspect and verify image attestations and warn when attestations are available for a specific image. For more information, please review the **[Hauler Documentation](https://rancherfederal.github.io/hauler-docs)!** -As OCI registries have become ubiquitous nowadays for storing and distributing containers. Their success and widespread adoption has led many projects to expand beyond containers. +## Latest Release +* [Hauler v0.3.0](https://github.com/rancherfederal/hauler/releases/tag/v0.3.0) + * [Download - Source](https://github.com/rancherfederal/hauler/archive/refs/tags/v0.3.0.zip) + * [Download - Linux AMD64](https://github.com/rancherfederal/hauler/releases/download/v0.3.0/hauler_0.3.0_linux_amd64.tar.gz) + * [Download - Linux ARM64](https://github.com/rancherfederal/hauler/releases/download/v0.3.0/hauler_0.3.0_linux_arm64.tar.gz) -`hauler` capitalizes on this, and leverages the [`oci`](https://github.com/opencontainers) spec to be a simple, zero dependency tool to collect, transport, and distribute your artifacts. +## Installation -## Getting started - -See the [quickstart](docs/walkthrough.md#Quickstart) for a quick way to get started with some of `haulers` capabilities. - -For a guided example of all of `haulers` capabilities, check out the [guided example](docs/walkthrough.md#guided-examples). +```bash +curl -#OL https://github.com/rancherfederal/hauler/releases/download/v0.3.0/hauler_0.3.0_linux_amd64.tar.gz +tar -xf hauler_0.3.0_linux_amd64.tar.gz +mv hauler /usr/bin/hauler +``` ## Acknowledgements -`hauler` wouldn't be possible without the open source community, but there are a few dependent projects that stand out: - +`Hauler` wouldn't be possible without the open-source community, but there are a few projects that stand out: * [go-containerregistry](https://github.com/google/go-containerregistry) -* [oras](https://github.com/oras-project/oras) +* [oras cli](https://github.com/oras-project/oras) * [cosign](https://github.com/sigstore/cosign) ## Notices +**WARNING: Upcoming Deprecated Command(s)** -*** WARNING: Deprecated Command *** - -The `hauler download (dl)` and `hauler serve (not the store version)` commands are deprecated and will be removed in a future release of Hauler. \ No newline at end of file +`hauler download (dl)` and `hauler serve (not the store version)` commands are deprecated and will be removed in a future release. \ No newline at end of file From 99288f9b9d9065e7b7bdb4966a20b93764a7b8c6 Mon Sep 17 00:00:00 2001 From: Zack Brady Date: Fri, 20 Oct 2023 03:56:01 -0400 Subject: [PATCH 2/6] removed old docs --- docs/walkthrough.md | 236 -------------------------------------------- 1 file changed, 236 deletions(-) delete mode 100644 docs/walkthrough.md diff --git a/docs/walkthrough.md b/docs/walkthrough.md deleted file mode 100644 index b44a889..0000000 --- a/docs/walkthrough.md +++ /dev/null @@ -1,236 +0,0 @@ -# Walkthrough - -## Installation - -The latest version of `hauler` is available as statically compiled binaries for most combinations of operating systems and architectures on the GitHub [releases](https://github.com/rancherfederal/hauler/releases) page. - -## Quickstart - -The tl;dr for how to use `hauler` to fetch, transport, and distribute `content`: - -```bash -# fetch some content -hauler store add file "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl" -hauler store add chart longhorn --repo "https://charts.longhorn.io" -hauler store add image "rancher/cowsay" - -# transport the content -hauler store save - -# <-airgap the haul.tar.zst file generated-> - -# load the content -hauler store load - -# serve the content -hauler store serve -``` - -While the example above fits into a quickstart, it falls short of demonstrating all the capabilities `hauler` has to offer, including taking advantage of its fully declarative nature. Keep reading the [Guided Examples](#Guided-Examples) below for a more thorough walkthrough of `haulers` full capabilities. - -## Guided Examples - -Since `hauler`'s primary objective is to simplify the content collection/distribution airgap process, a lot of the design revolves around the typical airgap workflow: - -```bash -fetch -> save - | | -> validate/load -> distribute -``` - -This is accomplished as follows: - -```bash -# fetch content -hauler store add ... - -# compress and archive content -hauler store save - -# - -# validate/load content -hauler store load ... - -# distribute content -hauler store serve -``` - -At this point you're probably wondering: what is `content`? In `hauler` land, there are a few important terms given to important resources: - -* `artifact`: anything that can be represented as an [`oci artifact`](https://github.com/opencontainers/artifacts) -* `content`: built in "primitive" types of `artifacts` that `hauler` understands - -### Built in content - -As of today, `hauler` understands three types of `content`, one with a strong legacy of community support and consensus ([`image-spec`]()), one with a finalized spec and experimental support ([`chart-spec`]()), and one generic type created just for `hauler`. These `content` types are outlined below: - -__`files`__: - -Generic content that can be represented as a file, either sourced locally or remotely. - -```bash -# local file -hauler store add file path/to/local/file.txt - -# remote file -hauler store add file https://get.k3s.io -``` - -__`images`__: - -Any OCI compatible image can be fetched remotely. - -```bash -# "shorthand" image references -hauler store add image rancher/k3s:v1.22.2-k3s1 - -# fully qualified image references -hauler store add image ghcr.io/fluxcd/flux-cli@sha256:02aa820c3a9c57d67208afcfc4bce9661658c17d15940aea369da259d2b976dd -``` - -__`charts`__: - -Helm charts represented as OCI content. - -```bash -# add a helm chart (defaults to latest version) -hauler store add chart loki --repo "https://grafana.github.io/helm-charts" - -# add a specific version of a helm chart -hauler store add chart loki --repo "https://grafana.github.io/helm-charts" --version 2.8.1 - -# install directly from the oci content -HELM_EXPERIMENTAL_OCI=1 helm install loki oci://localhost:3000/library/loki --version 2.8.1 -``` - -> Note: `hauler` supports the currently experimental format of helm as OCI content, but can also be represented as the usual tarball if necessary - -### Content API - -While imperatively adding `content` to `hauler` is a simple way to get started, the recommended long term approach is to use the provided api that each `content` has, in conjunction with the `sync` command. - -```bash -# create a haul from declaratively defined content -hauler store sync -f testdata/contents.yaml -``` - -> For a commented view of the `contents` api, take a look at the `testdata` folder in the root of the project. - -The API for each type of built-in `content` allows you to easily and declaratively define all the `content` that exist within a `haul`, and ensures a more gitops compatible workflow for managing the lifecycle of your `hauls`. - -It's also easy to imperatively generate a declarative contents.yaml -* This will generate a list of all images running in your cluster, store it in an environment variable, then echo the environment variable to show the list. -```bash -export IMAGE_LIST=$(kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" |tr -s '[[:space:]]' '\n' |sort |uniq -c | cut -c 9-) -echo "$IMAGE_LIST" -``` - -* Example Output -```console -index.docker.io/rancher/hardened-etcd:v3.5.1-k3s1-build20220112 -index.docker.io/rancher/hardened-kubernetes:v1.23.5-rke2r1-build20220316 -index.docker.io/rancher/rke2-cloud-provider:v0.0.3-build20211118 -rancher/hardened-calico:v3.21.4-build20220228 -rancher/hardened-cluster-autoscaler:v1.8.5-build20211119 -rancher/hardened-coredns:v1.9.1-build20220318 -rancher/hardened-flannel:v0.17.0-build20220317 -rancher/hardened-k8s-metrics-server:v0.5.0-build20211119 -rancher/klipper-helm:v0.7.0-build20220315 -rancher/nginx-ingress-controller:nginx-1.0.2-hardened4 -``` - -* The generated list of images can then be used to easily generate a hauler config file by using the following logic: -```bash -temp=$(cat "$IMAGE_LIST" | sed 's/^/ - name: /') -# sed, stream editor, appends 2 spaces of indentation denotes -# an entry in a list, then adds name: to match expected syntax -cat << EOF | tee contents.yaml -apiVersion: content.hauler.cattle.io/v1alpha1 -kind: Images -metadata: - name: myimage -spec: - images: -$temp -EOF - -cat contents.yaml -``` - -* The result would look like this -```console -apiVersion: content.hauler.cattle.io/v1alpha1 -kind: Images -metadata: - name: myimage -spec: - images: - - name: index.docker.io/rancher/hardened-etcd:v3.5.1-k3s1-build20220112 - - name: index.docker.io/rancher/hardened-kubernetes:v1.23.5-rke2r1-build20220316 - - name: index.docker.io/rancher/rke2-cloud-provider:v0.0.3-build20211118 - - name: rancher/hardened-calico:v3.21.4-build20220228 - - name: rancher/hardened-cluster-autoscaler:v1.8.5-build20211119 - - name: rancher/hardened-coredns:v1.9.1-build20220318 - - name: rancher/hardened-flannel:v0.17.0-build20220317 - - name: rancher/hardened-k8s-metrics-server:v0.5.0-build20211119 - - name: rancher/klipper-helm:v0.7.0-build20220315 - - name: rancher/nginx-ingress-controller:nginx-1.0.2-hardened4 -``` - -### Collections - -Earlier we referred to `content` as "primitives". While the quotes justify the loose definition of that term, we call it that because they can be used to build groups of `content`, which we call `collections`. - -`collections` are groups of 1 or more `contents` that collectively represent something desirable. Just like `content`, there are a handful that are built in to `hauler`. - -Since `collections` usually contain more purposefully crafted `contents`, we restrict their use to the declarative commands (`sync`): - -```bash -# sync a collection -hauler store sync -f my-collection.yaml - -# sync sets of content/collection -hauler store sync -f collection.yaml -f content.yaml -``` - -__`thickcharts`__: - -Thick Charts represent the combination of `charts` and `images`. When storing a thick chart, the chart _and_ the charts dependent images will be fetched and stored by `hauler`. - -```yaml -# thick-chart.yaml -apiVersion: collection.hauler.cattle.io/v1alpha1 -kind: ThickCharts -metadata: - name: loki -spec: - charts: - - name: loki - repoURL: https://grafana.github.io/helm-charts -``` - -When syncing the collection above, `hauler` will identify the images the chart depends on and store those too - -> The method for identifying images is constantly changing, as of today, the chart is rendered and a configurable set of container defining json path's are processed. The most common paths are recognized by hauler, but this can be configured for the more niche CRDs out there. - -__`k3s`__: - -Combining `files` and `images`, full clusters can also be captured by `hauler` for further simplifying the already simple nature of `k3s`. - -```yaml -# k3s.yaml ---- -apiVersion: collection.hauler.cattle.io/v1alpha1 -kind: K3s -metadata: - name: k3s -spec: - version: stable -``` - -Using the collection above, the dependent files (`k3s` executable and `https://get.k3s.io` script) will be fetched, as well as all the dependent images. - -> We know not everyone uses the get.k3s.io script to provision k3s, in the future this may change, but until then you're welcome to mix and match the `collection` with any of your own additional `content` - -#### User defined `collections` - -Although `content` and `collections` can only be used when they are baked in to `hauler`, the goal is to allow these to be securely user-defined, allowing you to define your own desirable `collection` types, and leave the heavy lifting to `hauler`. Check out our [roadmap](../ROADMAP.md) and [milestones]() for more info on that. From 04c97b8a9778019a8b918c87b436329c501b1f48 Mon Sep 17 00:00:00 2001 From: Zack Brady Date: Fri, 20 Oct 2023 12:22:10 -0400 Subject: [PATCH 3/6] fixed typos --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index e474d8d..239b1a6 100644 --- a/README.md +++ b/README.md @@ -2,11 +2,11 @@ ## Airgap Swiss Army Knife -> ⚠️ This project is still in active development and *not* Generally Available (GA). Most of the core functionality and features are ready, but may have breaking changes. Please review the [Release Notes](https://github.com/rancherfederal/hauler/releases) for more information! +> ⚠️ This project is still in active development and *not* Generally Available (GA). Most of the core functionality and features are ready, but may have breaking changes. Please review the [Release Notes](https://github.com/rancherfederal/hauler/releases) for more information! `Rancher Government Hauler` simplifies the airgap experience without requiring users to adopt a specific workflow. **Hauler** simplifies the airgapping process, by representing assets (images, charts, files, etc...) as content and collections to allow users to easily fetch, store, package, and distribute these assets with declarative manifests or through the command line. -`Hauler` does this by storing contents and collections as OCI Objects and Specifications and allowing users to serve contents and collections via an embedded registry and fileserver. Additionally, `Hauler` has the ability to inspect and verify image attestations and warn when attestations are available for a specific image. For more information, please review the **[Hauler Documentation](https://rancherfederal.github.io/hauler-docs)!** +`Hauler` does this by storing contents and collections as OCI Objects and Specifications and allows users to serve contents and collections with an embedded registry and fileserver. Additionally, `Hauler` has the ability to inspect and verify image attestations and warn when attestations are available for a specific image. For more information, please review the **[Hauler Documentation](https://rancherfederal.github.io/hauler-docs)!** ## Latest Release * [Hauler v0.3.0](https://github.com/rancherfederal/hauler/releases/tag/v0.3.0) From 870f2ebda86d862869241a13301e6abf02590cf7 Mon Sep 17 00:00:00 2001 From: Zack Brady Date: Sat, 21 Oct 2023 02:37:42 -0400 Subject: [PATCH 4/6] last typo fixes --- README.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 239b1a6..9a3209b 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,9 @@ `Rancher Government Hauler` simplifies the airgap experience without requiring users to adopt a specific workflow. **Hauler** simplifies the airgapping process, by representing assets (images, charts, files, etc...) as content and collections to allow users to easily fetch, store, package, and distribute these assets with declarative manifests or through the command line. -`Hauler` does this by storing contents and collections as OCI Objects and Specifications and allows users to serve contents and collections with an embedded registry and fileserver. Additionally, `Hauler` has the ability to inspect and verify image attestations and warn when attestations are available for a specific image. For more information, please review the **[Hauler Documentation](https://rancherfederal.github.io/hauler-docs)!** +`Hauler` does this by storing contents and collections as OCI Artifacts and allows users to serve contents and collections with an embedded registry and fileserver. Additionally, `Hauler` has the ability to store and inspect various non-image OCI Artifacts. + +For more information, please review the **[Hauler Documentation](https://rancherfederal.github.io/hauler-docs)!** ## Latest Release * [Hauler v0.3.0](https://github.com/rancherfederal/hauler/releases/tag/v0.3.0) @@ -19,7 +21,7 @@ ```bash curl -#OL https://github.com/rancherfederal/hauler/releases/download/v0.3.0/hauler_0.3.0_linux_amd64.tar.gz tar -xf hauler_0.3.0_linux_amd64.tar.gz -mv hauler /usr/bin/hauler +sudo mv hauler /usr/bin/hauler ``` ## Acknowledgements @@ -30,6 +32,6 @@ mv hauler /usr/bin/hauler * [cosign](https://github.com/sigstore/cosign) ## Notices -**WARNING: Upcoming Deprecated Command(s)** +**WARNING - Upcoming Deprecated Command(s):** -`hauler download (dl)` and `hauler serve (not the store version)` commands are deprecated and will be removed in a future release. \ No newline at end of file +`hauler download (dl)` and `hauler serve (not the store version)` commands are deprecated and will be removed in a future release. From b616f54085afed1b111d6a56dc785e24b9cbb7e4 Mon Sep 17 00:00:00 2001 From: Zack Brady Date: Wed, 25 Oct 2023 17:03:35 -0400 Subject: [PATCH 5/6] updated readme for deprecated commands Co-authored-by: Jacob Blain Christen --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 9a3209b..226beac 100644 --- a/README.md +++ b/README.md @@ -34,4 +34,4 @@ sudo mv hauler /usr/bin/hauler ## Notices **WARNING - Upcoming Deprecated Command(s):** -`hauler download (dl)` and `hauler serve (not the store version)` commands are deprecated and will be removed in a future release. +`hauler download` (alternatively, `dl`) and `hauler serve` (_not_ `hauler store serve`) commands are deprecated and will be removed in a future release. From 865afb4a2d4ba33b7e0e2e96308d9d490b60919b Mon Sep 17 00:00:00 2001 From: Zack Brady Date: Thu, 26 Oct 2023 00:42:58 -0400 Subject: [PATCH 6/6] updated readme for extra info --- README.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/README.md b/README.md index 226beac..a9de3c8 100644 --- a/README.md +++ b/README.md @@ -6,16 +6,10 @@ `Rancher Government Hauler` simplifies the airgap experience without requiring users to adopt a specific workflow. **Hauler** simplifies the airgapping process, by representing assets (images, charts, files, etc...) as content and collections to allow users to easily fetch, store, package, and distribute these assets with declarative manifests or through the command line. -`Hauler` does this by storing contents and collections as OCI Artifacts and allows users to serve contents and collections with an embedded registry and fileserver. Additionally, `Hauler` has the ability to store and inspect various non-image OCI Artifacts. +`Hauler` does this by storing contents and collections as OCI Artifacts and allows users to serve contents and collections with an embedded registry and fileserver. Additionally, `Hauler` has the ability to store and inspect various non-image OCI Artifacts. For more information, please review the **[Hauler Documentation](https://rancherfederal.github.io/hauler-docs)!** -## Latest Release -* [Hauler v0.3.0](https://github.com/rancherfederal/hauler/releases/tag/v0.3.0) - * [Download - Source](https://github.com/rancherfederal/hauler/archive/refs/tags/v0.3.0.zip) - * [Download - Linux AMD64](https://github.com/rancherfederal/hauler/releases/download/v0.3.0/hauler_0.3.0_linux_amd64.tar.gz) - * [Download - Linux ARM64](https://github.com/rancherfederal/hauler/releases/download/v0.3.0/hauler_0.3.0_linux_arm64.tar.gz) - ## Installation ```bash