docs: add deploy scripts, skills, and pattern guides

Deploy scripts for webicached and webid (build, upload, restart). AGENTS.md with releases.conf reference and variant tagging docs. Installer archive pattern guide and version oddities reference.
2026-08-19 12:06:25 +00:00 · 2026-05-16 21:22:38 -06:00
parent 631147901a
commit af28ddb686
6 changed files with 505 additions and 0 deletions
@@ -3,6 +3,16 @@ install-*.sh
 install-*.bat
 install-*.ps1

+# Go build outputs (from go run/build in repo root)
+/classify
+/e2etest
+/fetchraw
+/inspect
+/uaparse
+/webicached
+/zigtest
+/distributables.csv
+
 # local config
 .env.*
 *.env
@@ -18,7 +28,13 @@ node_modules/
 *.bak
 *.bak.*

+# agent session files
+agents/
+
 # other
 .DS_Store
 desktop.ini
 .directory
+LIVE_cache
+/webid
+bin/
@@ -370,3 +370,123 @@ Commit messages: `feat(<pkg>): add installer`, `fix(<pkg>): update install.sh`,
  `arch`, or `ext` explicitly in `releases.js`.
 - **Goreleaser archives**: Typically contain a bare binary at the archive root
  (not nested in a directory). Use `mv ./cmd "$pkg_src_cmd"`.
+
+---
+
+## Go Cache Daemon
+
+The Go pipeline (`cmd/webicached`) replaces the Node.js release-fetching code.
+It reads `releases.conf` files, fetches upstream release metadata, classifies
+build assets, and writes to `~/.cache/webi/legacy/` in the format the Node.js server expects.
+
+### Canonical Vocabulary
+
+The classifier MUST use exactly these strings. They match the production API.
+
+**OS**: `macos` (NOT `darwin`), `linux`, `windows`, `freebsd`, `openbsd`,
+`netbsd`, `dragonfly`, `aix`, `illumos`, `plan9`, `solaris`, `posix_2017`
+
+**Arch** — exact equivalences:
+- `amd64` (NOT `x86_64`), `x86` (NOT `i386`/`i686`/`386`)
+- `arm64` (NOT `aarch64`)
+- `armv7l` (NOT `armv7`), `armv6l` (NOT `armv6`)
+- `mipsle` (NOT `mipsel`), `mips64le` (NOT `mips64el`)
+
+**Arch** — compatibility downcasts:
+- `armhf` → `armv7l`, `armv7a` → `armv7l`, `armel` → `arm`
+
+**Arch** — other: `arm`, `ppc64le`, `ppc64`, `loong64`, `riscv64`, `s390x`,
+`mips`, `mips64`
+
+**Libc**: `none` (never empty), `gnu`, `musl`, `msvc`
+
+**Ext**: `tar.gz`, `tar.xz`, `zip`, `exe`, `7z`, `pkg`, `msi`
+(no leading dot; `exe` for bare binaries)
+
+### releases.conf
+
+Each package directory contains a `releases.conf` that tells the daemon where
+to fetch releases. Format is `key = value`, one per line. `#` comments and
+blank lines are ignored.
+
+#### Source types (mutually exclusive — pick one)
+
+```ini
+# GitHub binary releases (most common)
+github_releases = sharkdp/bat
+
+# GitHub source tarballs (with optional git fallback)
+github_sources = bnnanet/serviceman
+git_url = https://github.com/bnnanet/serviceman.git
+
+# Git tag enumeration (vim plugins, shell scripts — git_url alone)
+git_url = https://github.com/tpope/vim-commentary.git
+
+# Gitea (full URL required, or short form + base_url)
+gitea_releases = https://git.rootprojects.org/root/pathman
+
+# GitLab (defaults to gitlab.com)
+gitlab_releases = owner/repo
+
+# HashiCorp releases API
+hashicorp_product = terraform
+
+# Custom source (servicemandist, nodedist, zigdist, etc.)
+source = nodedist
+url = https://nodejs.org/download/release
+```
+
+#### Filtering, versioning, and platform
+
+```ini
+tag_prefix = bun-                    # monorepo: strip prefix from version
+version_prefixes = jq-               # strip from version string (space-separated)
+asset_filter = MinGit                # filename must contain this substring
+exclude = busybox -src- -docs-       # skip assets containing these (space-separated)
+os = posix_2017                      # restrict ALL versions to this OS (blanket)
+alias_of = rg                        # mirrors another package's releases
+```
+
+#### Design rules
+
+- `os` is a blanket tag on ALL versions. Only use for packages that are always
+  POSIX-only. For version-dependent OS tagging, use a custom `TagVariants` in
+  `internal/releases/{pkg}/variants.go`.
+- `git_url` can be primary (gittag source when it's the only key) or secondary
+  fallback alongside `github_sources`/`gitea_sources`.
+- Full URL forms accepted for github/gitea/gitlab (e.g.
+  `github_releases = https://github.com/sharkdp/bat`).
+
+### Testing
+
+Test tools: `cmd/e2etest` (pipeline comparison), `cmd/comparecache` (cache diff),
+`cmd/inspect` (single-package debug). Run each with `--help` for usage.
+
+### Classifier vs Per-Package Tagger
+
+The general classifier (`internal/classify/`) handles patterns common across
+many projects. It MUST NOT contain one-off logic for a single package.
+
+Per-package taggers (`internal/releases/{pkg}/variants.go`) handle
+project-specific knowledge. Read the existing taggers for conventions.
+
+MUST: Derive arch/OS from concrete evidence — not blanket defaults.
+MUST: New general classifier patterns must apply to 2-3+ packages.
+
+### Deploying
+
+```sh
+./scripts/deploy-webicached.sh beta.webi.sh
+./scripts/deploy-webicached.sh next.webi.sh
+```
+
+First-time setup on a new host uses `serviceman`:
+
+```sh
+serviceman add --name webicached \
+  --workdir ~/srv/webid/installers/ -- \
+  ~/bin/webicached \
+    --envfile ~/srv/webid/.env.secret \
+    --conf ~/srv/webid/installers/ \
+    --raw ~/.cache/webi/raw
+```
@@ -0,0 +1,121 @@
+# Installer Archive Patterns
+
+Every package falls into one of these archive structure patterns. When writing
+or modifying an `install.sh`, identify the pattern first — it determines the
+extraction and installation strategy.
+
+## Pattern A: Bare Binary in Archive
+
+Archive contains the binary (and maybe LICENSE/README) at the top level.
+
+Examples: awless, caddy, cilium, curlie, dashmsg, deno, dotenv, dotenv-linter,
+ffuf, fzf, gitdeploy, gprox, grype, hugo, hugo-extended, k9s, keypairs, koji,
+lf, monorel, ots, runzip, sclient, sqlc, sqlpkg, sttr, terraform, uuidv7, xcaddy
+
+Install: extract, move binary to `~/.local/opt/{pkg}-{ver}/bin/{binary}`, symlink.
+
+## Pattern B: Subdirectory with Binary Only
+
+Archive contains a version-named directory wrapping the binary and docs.
+
+Examples: delta, hexyl, kubectx, kubens, shellcheck, trip, xsv
+
+Typical directory naming: `{tool}-{ver}-{triplet}/`
+
+Install: extract, find binary in subdirectory, move to opt, symlink.
+
+Special cases:
+- `pathman`: bare binary named with full release tag (needs rename)
+- `yq`: binary named with platform suffix `yq_linux_amd64` (needs rename)
+
+## Pattern C: Binary + Completions + Man Pages
+
+Archive includes shell completions and/or man pages alongside the binary.
+
+| Package | Completions Dir | Man Page |
+|---------|----------------|----------|
+| bat | `autocomplete/` | `bat.1` |
+| fd | `autocomplete/{fd.bash,.fish,_fd}` | `fd.1` |
+| goreleaser | `completions/{.bash,.fish,.zsh}` | `manpages/*.1.gz` |
+| lsd | `autocomplete/{lsd.bash-completion,.fish,_lsd}` | `lsd.1` |
+| rg | `complete/{rg.bash,.fish,_rg}` | `doc/rg.1` |
+| sd | `completions/{sd.bash,.fish,_sd}` | `sd.1` |
+| watchexec | `completions/{bash,fish,zsh}` | `watchexec.1` |
+| zoxide | `completions/{zoxide.bash,.fish,_zoxide}` | `man/man1/zoxide*.1` |
+
+Install: extract, install binary, install completions to standard dirs, install
+man pages. Completion naming varies: `autocomplete/`, `completions/`, `complete/`.
+
+## Pattern D: Binary + Libraries
+
+Complex packages that bundle shared libraries.
+
+| Package | Layout |
+|---------|--------|
+| ollama (Linux) | `bin/ollama` + `lib/ollama/{cuda_v12,cuda_v13,vulkan}/` |
+| pg/postgres/psql | `bin/psql` + `lib/{libpq,libz,...}.so` + `include/` |
+| sass | `dart-sass/sass` (wrapper) + `dart-sass/src/{dart,sass.snapshot}` |
+| syncthing | `syncthing-{triplet}-{ver}/syncthing` + `etc/{systemd,...}/` |
+| xz | `xz-{ver}-{triplet}/xz` + `xz-{ver}-{triplet}/unxz` |
+
+Install: extract entire directory tree into opt, symlink binary.
+
+## Pattern E: FHS-like Layout (bin/ + share/)
+
+Archive already follows standard layout.
+
+| Package | Layout |
+|---------|--------|
+| gh | `gh_{ver}_{os}_{arch}/bin/gh` + `share/man/man1/*.1` |
+| pandoc | `pandoc-{ver}/bin/{pandoc,...}` + `share/man/man1/*.1.gz` |
+
+Install: extract directly into opt (already correct layout).
+
+## Pattern G: Full SDK/Toolchain
+
+Self-contained toolchain with compiler, runtime, standard library.
+
+| Package | Layout |
+|---------|--------|
+| cmake | `cmake-{ver}-{os}-{arch}/bin/{cmake,ctest,...}` + `share/` + `man/` |
+| tinygo | `tinygo/bin/tinygo` + `tinygo/src/` + `tinygo/targets/` |
+| go | `go/bin/{go,gofmt}` + `go/src/` + `go/pkg/` |
+| zig | `zig-{os}-{arch}-{ver}/zig` + `lib/` |
+| flutter | `flutter/bin/flutter` + full SDK |
+| julia | `julia-{ver}/bin/julia` + full SDK |
+| node | `node-{ver}-{os}-{arch}/bin/{node,npm,npx}` + `lib/` |
+
+Install: extract entire tree into `~/.local/opt/{pkg}-{ver}/`, symlink `bin/*`.
+
+## Pattern H: .NET Runtime Bundle
+
+Flat archive with hundreds of DLLs.
+
+Example: pwsh — `pwsh` binary + `*.dll` + locale dirs
+
+Install: extract entire directory into opt, symlink primary binary.
+
+## Pattern I: Multi-Binary Distribution
+
+Archive contains multiple related binaries + libs.
+
+| Package | Layout |
+|---------|--------|
+| dashcore | `dashcore-{ver}/bin/{dashd,dash-cli,...}` + `lib/` + `share/man/` |
+| mutagen | `mutagen` + `mutagen-agents.tar.gz` (embedded agent archive) |
+
+Install: extract into opt, symlink primary binary.
+
+## Format Changes Over Time
+
+Most packages have stable formats. Notable structural changes:
+
+| Package | When | Change |
+|---------|------|--------|
+| sd | 2023 | zip → tar.gz, added completions + man page |
+| ollama | 2025-2026 | bare binary → no GitHub release → tar.zst with lib/ |
+| deno | 2020-2021 | .gz (gzipped binary) → .zip |
+| hugo | 2017-2018 | zip → tar.gz; 2024: macOS → .pkg only |
+| gh | 2024 | darwin: tar.gz → .pkg |
+| sclient | 2023 | tar.gz → tar.xz |
+| watchexec | 2019-2020 | tar.gz → tar.xz |
@@ -0,0 +1,74 @@
+# Version & Release Oddities
+
+Non-standard version formats and tag prefixes that affect parsing, sorting,
+and classification. The Go classifier and `internal/lexver` must handle all
+of these.
+
+## Non-Numeric Tag Prefixes
+
+| Package | Raw Tag | Cleaned | Transform |
+|---------|---------|---------|-----------|
+| lf | `r21` | `0.21.0` | `r` prefix → prepend `0.` |
+| bun | `bun-v1.0.0` | `1.0.0` | Strip `bun-` prefix |
+| jq | `jq-1.7` | `1.7` | Strip `jq-` prefix |
+| watchexec | `cli-v1.2.3` | `1.2.3` | Strip `cli-` prefix |
+| ffmpeg | `b6.0` | `6.0` | Strip `b` prefix |
+
+## Underscore-Delimited Tags
+
+| Package | Raw Tag | Cleaned | Transform |
+|---------|---------|---------|-----------|
+| postgres | `REL_17_0` | `17.0` | Strip `REL_`, replace `_` with `.` |
+| psql | `REL_17_0` | `17.0` | Same as postgres |
+
+## Platform Suffix in Version
+
+| Package | Raw Tag | Cleaned | Transform |
+|---------|---------|---------|-----------|
+| git (Windows) | `2.41.0.windows.1` | `2.41.0` | Strip `.windows.N` suffix |
+
+## 4-Part Versions
+
+| Package | Example | Notes |
+|---------|---------|-------|
+| chromedriver | `121.0.6120.0` | Google Chrome's versioning |
+| gpg | `2.2.19.0` | 4th segment is build metadata |
+
+## Date-Based Versions
+
+| Package | Notes |
+|---------|-------|
+| atomicparsley | Date-based version strings |
+
+## Complex Pre-Release Formats
+
+| Package | Example | Notes |
+|---------|---------|-------|
+| flutter | `2.3.0-16.0.pre` | Extra dots and numeric segments |
+| iterm2 | `iTerm2_3_5_0beta17` | Underscores, beta attached → `3.5.0-beta17` |
+
+## Channel Detection
+
+- Node.js: odd major = "current" not LTS (v15, v17, v19, v21, v23)
+- Go: `go` prefix stripped (`go1.23.6` → `1.23.6`)
+- Terraform: `-alpha`, `-beta`, `-rc` suffixes → beta channel
+
+## Directory Symlinks (Aliases)
+
+These are directory-level symlinks. They share all files (including
+releases.conf) with their target automatically.
+
+```
+msvc-runtime    → vcruntime
+msvcruntime     → vcruntime
+rust.vim        → vim-rust
+vc-redist       → vcruntime
+vc-runtime      → vcruntime
+vc_redist       → vcruntime
+vcredist        → vcruntime
+vcruntime140    → vcruntime
+vim-essential   → vim-essentials
+vim-mouse       → vim-gui
+vps-myip        → myip
+xcode-cli       → commandlinetools
+```
@@ -0,0 +1,63 @@
+#!/bin/sh
+set -e
+set -u
+
+# Build and deploy webicached to a target host
+
+g_host="${1:-beta.webi.sh}"
+g_bin="webicached"
+g_out="agents/tmp/${g_bin}"
+g_remote_bin="~/bin/${g_bin}"
+
+case "${g_host}" in
+	beta.webi.sh) g_remote_conf="~/srv/beta.webinstall.dev/installers/" ;;
+	next.webi.sh) g_remote_conf="~/srv/next.webinstall.dev/installers/" ;;
+	*) g_remote_conf="~/srv/webid/installers/" ;;
+esac
+
+fn_build() {
+	b_version="$(git describe --tags --always 2> /dev/null || echo '0.0.0-dev')"
+	b_commit="$(git rev-parse --short HEAD)"
+	b_date="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
+	b_ldflags="-X main.version=${b_version} -X main.commit=${b_commit} -X main.date=${b_date}"
+
+	printf 'Building %s %s %s (%s)...\n' "${g_bin}" "${b_version}" "${b_commit}" "${b_date}"
+	GOOS=linux GOARCH=amd64 GOAMD64=v2 go build -ldflags "${b_ldflags}" -o "${g_out}" ./cmd/webicached
+	printf 'Built: %s\n' "${g_out}"
+}
+
+fn_deploy() {
+	printf 'Stopping %s on %s...\n' "${g_bin}" "${g_host}"
+	ssh "${g_host}" "~/.local/bin/serviceman stop ${g_bin}" 2> /dev/null || true
+
+	printf 'Uploading binary...\n'
+	scp "${g_out}" "${g_host}:${g_remote_bin}"
+
+	printf 'Syncing releases.conf files...\n'
+	rsync -av \
+		--exclude='_cache' --exclude='.git' --exclude='agents' \
+		--exclude='bin' --exclude='cmd' --exclude='internal' \
+		--exclude='docs' --exclude='scripts' --exclude='node_modules' \
+		--include='*/' --include='releases.conf' --exclude='*' \
+		./ "${g_host}:${g_remote_conf}"
+
+	printf 'Starting %s...\n' "${g_bin}"
+	ssh "${g_host}" "~/.local/bin/serviceman start ${g_bin}"
+}
+
+fn_verify() {
+	printf 'Waiting 5s for startup...\n'
+	sleep 5
+
+	printf 'Checking version...\n'
+	ssh "${g_host}" "${g_remote_bin} -V"
+
+	printf 'Checking logs...\n'
+	ssh "${g_host}" "sudo journalctl -u ${g_bin} --no-pager -n 5"
+}
+
+fn_build
+fn_deploy
+fn_verify
+
+printf '\nDone. %s deployed to %s.\n' "${g_bin}" "${g_host}"
@@ -0,0 +1,111 @@
+---
+name: deploy-webicached
+description: Deploy webicached binary to beta.webi.sh. Use when building, uploading, or restarting the cache daemon. Covers cross-compile, conf sync, service management.
+---
+
+## One-step deploy
+
+```sh
+./scripts/deploy-webicached.sh beta.webi.sh
+```
+
+Builds with version ldflags, stops service, uploads, syncs conf, starts, verifies.
+
+## Manual steps (if needed)
+
+### Build
+
+```sh
+VERSION="$(git describe --tags --always)"
+COMMIT="$(git rev-parse --short HEAD)"
+DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
+GOOS=linux GOARCH=amd64 GOAMD64=v2 go build \
+  -ldflags "-X main.version=${VERSION} -X main.commit=${COMMIT} -X main.date=${DATE}" \
+  -o agents/tmp/webicached ./cmd/webicached
+```
+
+MUST: Build from the `ref-webi-go` worktree (or branch containing `cmd/webicached`).
+
+### Deploy
+
+```sh
+ssh beta.webi.sh "serviceman stop webicached"
+scp agents/tmp/webicached beta.webi.sh:~/bin/webicached
+```
+
+MUST: Stop service before scp — Linux refuses to overwrite a running binary.
+
+### Sync releases.conf
+
+```sh
+rsync -av --include='*/' --include='releases.conf' --exclude='*' \
+  ./ beta.webi.sh:~/srv/beta.webinstall.dev/installers/
+```
+
+MUST: Run from the worktree root. The server has no checkout of this branch — conf files must be synced explicitly.
+
+### Start
+
+```sh
+ssh beta.webi.sh "serviceman start webicached"
+```
+
+### Verify
+
+```sh
+ssh beta.webi.sh "sleep 5 && serviceman logs webicached"
+```
+
+Expected: "batch: N stale, refreshing 20" or "all packages fresh, sleeping 9s"
+
+## Smoke test
+
+```sh
+ssh beta.webi.sh "curl -sSf http://localhost:3080/api/releases/bat.json | head -c 100"
+ssh beta.webi.sh "curl -sSf -A 'curl/7.81.0 Linux x86_64' http://localhost:3080/api/installers/bat.sh | head -3"
+```
+
+Expected: JSON array with release objects; shell script with `PKG_NAME='bat'`.
+
+## Service management
+
+```sh
+serviceman status webicached
+serviceman restart webicached
+serviceman logs webicached
+```
+
+## Server layout
+
+| Path | Purpose |
+|------|---------|
+| `~/bin/webicached` | Binary |
+| `~/srv/beta.webinstall.dev/installers/` | Conf dir (releases.conf files) |
+| `~/.cache/webi/legacy/` | Cache output (fsstore, legacy JSON format) |
+| `~/.cache/webi/raw/` | Raw upstream API responses |
+| `~/srv/beta.webinstall.dev/.env.secret` | GITHUB_TOKEN |
+| `/etc/systemd/system/webicached.service` | Service unit (created by serviceman) |
+
+## Flags reference
+
+| Flag | Default | Purpose |
+|------|---------|---------|
+| `--conf` | `.` | Dir with `{pkg}/releases.conf` files |
+| `--legacy` | `~/.cache/webi/legacy` | Legacy cache output directory |
+| `--raw` | `~/.cache/webi/raw` | Raw upstream response cache |
+| `--token` | `$GITHUB_TOKEN` | GitHub API token |
+| `--interval` | `9s` | Delay between package fetches in a batch |
+| `--once` | false | Run once then exit |
+| `--eager` | false | Fetch all on startup (not staleness-based) |
+| `--shallow` | false | Only first page of releases |
+| `--no-fetch` | false | Classify from rawcache only |
+| `--page-delay` | `2s` | Delay between paginated API pages |
+
+## One-shot refresh (specific packages)
+
+```sh
+ssh beta.webi.sh ". ~/srv/beta.webinstall.dev/.env.secret && ~/bin/webicached \
+  --conf ~/srv/beta.webinstall.dev/installers/ \
+  --raw ~/.cache/webi/raw \
+  --once bat goreleaser"
+```