diff --git a/.gitignore b/.gitignore index 181d386..5f5aa3b 100644 --- a/.gitignore +++ b/.gitignore @@ -3,6 +3,16 @@ install-*.sh install-*.bat install-*.ps1 +# Go build outputs (from go run/build in repo root) +/classify +/e2etest +/fetchraw +/inspect +/uaparse +/webicached +/zigtest +/distributables.csv + # local config .env.* *.env @@ -18,7 +28,13 @@ node_modules/ *.bak *.bak.* +# agent session files +agents/ + # other .DS_Store desktop.ini .directory +LIVE_cache +/webid +bin/ diff --git a/AGENTS.md b/AGENTS.md index 787e855..e701723 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -370,3 +370,123 @@ Commit messages: `feat(): add installer`, `fix(): update install.sh`, `arch`, or `ext` explicitly in `releases.js`. - **Goreleaser archives**: Typically contain a bare binary at the archive root (not nested in a directory). Use `mv ./cmd "$pkg_src_cmd"`. + +--- + +## Go Cache Daemon + +The Go pipeline (`cmd/webicached`) replaces the Node.js release-fetching code. +It reads `releases.conf` files, fetches upstream release metadata, classifies +build assets, and writes to `~/.cache/webi/legacy/` in the format the Node.js server expects. + +### Canonical Vocabulary + +The classifier MUST use exactly these strings. They match the production API. + +**OS**: `macos` (NOT `darwin`), `linux`, `windows`, `freebsd`, `openbsd`, +`netbsd`, `dragonfly`, `aix`, `illumos`, `plan9`, `solaris`, `posix_2017` + +**Arch** — exact equivalences: +- `amd64` (NOT `x86_64`), `x86` (NOT `i386`/`i686`/`386`) +- `arm64` (NOT `aarch64`) +- `armv7l` (NOT `armv7`), `armv6l` (NOT `armv6`) +- `mipsle` (NOT `mipsel`), `mips64le` (NOT `mips64el`) + +**Arch** — compatibility downcasts: +- `armhf` → `armv7l`, `armv7a` → `armv7l`, `armel` → `arm` + +**Arch** — other: `arm`, `ppc64le`, `ppc64`, `loong64`, `riscv64`, `s390x`, +`mips`, `mips64` + +**Libc**: `none` (never empty), `gnu`, `musl`, `msvc` + +**Ext**: `tar.gz`, `tar.xz`, `zip`, `exe`, `7z`, `pkg`, `msi` +(no leading dot; `exe` for bare binaries) + +### releases.conf + +Each package directory contains a `releases.conf` that tells the daemon where +to fetch releases. Format is `key = value`, one per line. `#` comments and +blank lines are ignored. + +#### Source types (mutually exclusive — pick one) + +```ini +# GitHub binary releases (most common) +github_releases = sharkdp/bat + +# GitHub source tarballs (with optional git fallback) +github_sources = bnnanet/serviceman +git_url = https://github.com/bnnanet/serviceman.git + +# Git tag enumeration (vim plugins, shell scripts — git_url alone) +git_url = https://github.com/tpope/vim-commentary.git + +# Gitea (full URL required, or short form + base_url) +gitea_releases = https://git.rootprojects.org/root/pathman + +# GitLab (defaults to gitlab.com) +gitlab_releases = owner/repo + +# HashiCorp releases API +hashicorp_product = terraform + +# Custom source (servicemandist, nodedist, zigdist, etc.) +source = nodedist +url = https://nodejs.org/download/release +``` + +#### Filtering, versioning, and platform + +```ini +tag_prefix = bun- # monorepo: strip prefix from version +version_prefixes = jq- # strip from version string (space-separated) +asset_filter = MinGit # filename must contain this substring +exclude = busybox -src- -docs- # skip assets containing these (space-separated) +os = posix_2017 # restrict ALL versions to this OS (blanket) +alias_of = rg # mirrors another package's releases +``` + +#### Design rules + +- `os` is a blanket tag on ALL versions. Only use for packages that are always + POSIX-only. For version-dependent OS tagging, use a custom `TagVariants` in + `internal/releases/{pkg}/variants.go`. +- `git_url` can be primary (gittag source when it's the only key) or secondary + fallback alongside `github_sources`/`gitea_sources`. +- Full URL forms accepted for github/gitea/gitlab (e.g. + `github_releases = https://github.com/sharkdp/bat`). + +### Testing + +Test tools: `cmd/e2etest` (pipeline comparison), `cmd/comparecache` (cache diff), +`cmd/inspect` (single-package debug). Run each with `--help` for usage. + +### Classifier vs Per-Package Tagger + +The general classifier (`internal/classify/`) handles patterns common across +many projects. It MUST NOT contain one-off logic for a single package. + +Per-package taggers (`internal/releases/{pkg}/variants.go`) handle +project-specific knowledge. Read the existing taggers for conventions. + +MUST: Derive arch/OS from concrete evidence — not blanket defaults. +MUST: New general classifier patterns must apply to 2-3+ packages. + +### Deploying + +```sh +./scripts/deploy-webicached.sh beta.webi.sh +./scripts/deploy-webicached.sh next.webi.sh +``` + +First-time setup on a new host uses `serviceman`: + +```sh +serviceman add --name webicached \ + --workdir ~/srv/webid/installers/ -- \ + ~/bin/webicached \ + --envfile ~/srv/webid/.env.secret \ + --conf ~/srv/webid/installers/ \ + --raw ~/.cache/webi/raw +``` diff --git a/docs/installer-patterns.md b/docs/installer-patterns.md new file mode 100644 index 0000000..0657280 --- /dev/null +++ b/docs/installer-patterns.md @@ -0,0 +1,121 @@ +# Installer Archive Patterns + +Every package falls into one of these archive structure patterns. When writing +or modifying an `install.sh`, identify the pattern first — it determines the +extraction and installation strategy. + +## Pattern A: Bare Binary in Archive + +Archive contains the binary (and maybe LICENSE/README) at the top level. + +Examples: awless, caddy, cilium, curlie, dashmsg, deno, dotenv, dotenv-linter, +ffuf, fzf, gitdeploy, gprox, grype, hugo, hugo-extended, k9s, keypairs, koji, +lf, monorel, ots, runzip, sclient, sqlc, sqlpkg, sttr, terraform, uuidv7, xcaddy + +Install: extract, move binary to `~/.local/opt/{pkg}-{ver}/bin/{binary}`, symlink. + +## Pattern B: Subdirectory with Binary Only + +Archive contains a version-named directory wrapping the binary and docs. + +Examples: delta, hexyl, kubectx, kubens, shellcheck, trip, xsv + +Typical directory naming: `{tool}-{ver}-{triplet}/` + +Install: extract, find binary in subdirectory, move to opt, symlink. + +Special cases: +- `pathman`: bare binary named with full release tag (needs rename) +- `yq`: binary named with platform suffix `yq_linux_amd64` (needs rename) + +## Pattern C: Binary + Completions + Man Pages + +Archive includes shell completions and/or man pages alongside the binary. + +| Package | Completions Dir | Man Page | +|---------|----------------|----------| +| bat | `autocomplete/` | `bat.1` | +| fd | `autocomplete/{fd.bash,.fish,_fd}` | `fd.1` | +| goreleaser | `completions/{.bash,.fish,.zsh}` | `manpages/*.1.gz` | +| lsd | `autocomplete/{lsd.bash-completion,.fish,_lsd}` | `lsd.1` | +| rg | `complete/{rg.bash,.fish,_rg}` | `doc/rg.1` | +| sd | `completions/{sd.bash,.fish,_sd}` | `sd.1` | +| watchexec | `completions/{bash,fish,zsh}` | `watchexec.1` | +| zoxide | `completions/{zoxide.bash,.fish,_zoxide}` | `man/man1/zoxide*.1` | + +Install: extract, install binary, install completions to standard dirs, install +man pages. Completion naming varies: `autocomplete/`, `completions/`, `complete/`. + +## Pattern D: Binary + Libraries + +Complex packages that bundle shared libraries. + +| Package | Layout | +|---------|--------| +| ollama (Linux) | `bin/ollama` + `lib/ollama/{cuda_v12,cuda_v13,vulkan}/` | +| pg/postgres/psql | `bin/psql` + `lib/{libpq,libz,...}.so` + `include/` | +| sass | `dart-sass/sass` (wrapper) + `dart-sass/src/{dart,sass.snapshot}` | +| syncthing | `syncthing-{triplet}-{ver}/syncthing` + `etc/{systemd,...}/` | +| xz | `xz-{ver}-{triplet}/xz` + `xz-{ver}-{triplet}/unxz` | + +Install: extract entire directory tree into opt, symlink binary. + +## Pattern E: FHS-like Layout (bin/ + share/) + +Archive already follows standard layout. + +| Package | Layout | +|---------|--------| +| gh | `gh_{ver}_{os}_{arch}/bin/gh` + `share/man/man1/*.1` | +| pandoc | `pandoc-{ver}/bin/{pandoc,...}` + `share/man/man1/*.1.gz` | + +Install: extract directly into opt (already correct layout). + +## Pattern G: Full SDK/Toolchain + +Self-contained toolchain with compiler, runtime, standard library. + +| Package | Layout | +|---------|--------| +| cmake | `cmake-{ver}-{os}-{arch}/bin/{cmake,ctest,...}` + `share/` + `man/` | +| tinygo | `tinygo/bin/tinygo` + `tinygo/src/` + `tinygo/targets/` | +| go | `go/bin/{go,gofmt}` + `go/src/` + `go/pkg/` | +| zig | `zig-{os}-{arch}-{ver}/zig` + `lib/` | +| flutter | `flutter/bin/flutter` + full SDK | +| julia | `julia-{ver}/bin/julia` + full SDK | +| node | `node-{ver}-{os}-{arch}/bin/{node,npm,npx}` + `lib/` | + +Install: extract entire tree into `~/.local/opt/{pkg}-{ver}/`, symlink `bin/*`. + +## Pattern H: .NET Runtime Bundle + +Flat archive with hundreds of DLLs. + +Example: pwsh — `pwsh` binary + `*.dll` + locale dirs + +Install: extract entire directory into opt, symlink primary binary. + +## Pattern I: Multi-Binary Distribution + +Archive contains multiple related binaries + libs. + +| Package | Layout | +|---------|--------| +| dashcore | `dashcore-{ver}/bin/{dashd,dash-cli,...}` + `lib/` + `share/man/` | +| mutagen | `mutagen` + `mutagen-agents.tar.gz` (embedded agent archive) | + +Install: extract into opt, symlink primary binary. + +## Format Changes Over Time + +Most packages have stable formats. Notable structural changes: + +| Package | When | Change | +|---------|------|--------| +| sd | 2023 | zip → tar.gz, added completions + man page | +| ollama | 2025-2026 | bare binary → no GitHub release → tar.zst with lib/ | +| deno | 2020-2021 | .gz (gzipped binary) → .zip | +| hugo | 2017-2018 | zip → tar.gz; 2024: macOS → .pkg only | +| gh | 2024 | darwin: tar.gz → .pkg | +| sclient | 2023 | tar.gz → tar.xz | +| watchexec | 2019-2020 | tar.gz → tar.xz | diff --git a/docs/version-oddities.md b/docs/version-oddities.md new file mode 100644 index 0000000..402e61a --- /dev/null +++ b/docs/version-oddities.md @@ -0,0 +1,74 @@ +# Version & Release Oddities + +Non-standard version formats and tag prefixes that affect parsing, sorting, +and classification. The Go classifier and `internal/lexver` must handle all +of these. + +## Non-Numeric Tag Prefixes + +| Package | Raw Tag | Cleaned | Transform | +|---------|---------|---------|-----------| +| lf | `r21` | `0.21.0` | `r` prefix → prepend `0.` | +| bun | `bun-v1.0.0` | `1.0.0` | Strip `bun-` prefix | +| jq | `jq-1.7` | `1.7` | Strip `jq-` prefix | +| watchexec | `cli-v1.2.3` | `1.2.3` | Strip `cli-` prefix | +| ffmpeg | `b6.0` | `6.0` | Strip `b` prefix | + +## Underscore-Delimited Tags + +| Package | Raw Tag | Cleaned | Transform | +|---------|---------|---------|-----------| +| postgres | `REL_17_0` | `17.0` | Strip `REL_`, replace `_` with `.` | +| psql | `REL_17_0` | `17.0` | Same as postgres | + +## Platform Suffix in Version + +| Package | Raw Tag | Cleaned | Transform | +|---------|---------|---------|-----------| +| git (Windows) | `2.41.0.windows.1` | `2.41.0` | Strip `.windows.N` suffix | + +## 4-Part Versions + +| Package | Example | Notes | +|---------|---------|-------| +| chromedriver | `121.0.6120.0` | Google Chrome's versioning | +| gpg | `2.2.19.0` | 4th segment is build metadata | + +## Date-Based Versions + +| Package | Notes | +|---------|-------| +| atomicparsley | Date-based version strings | + +## Complex Pre-Release Formats + +| Package | Example | Notes | +|---------|---------|-------| +| flutter | `2.3.0-16.0.pre` | Extra dots and numeric segments | +| iterm2 | `iTerm2_3_5_0beta17` | Underscores, beta attached → `3.5.0-beta17` | + +## Channel Detection + +- Node.js: odd major = "current" not LTS (v15, v17, v19, v21, v23) +- Go: `go` prefix stripped (`go1.23.6` → `1.23.6`) +- Terraform: `-alpha`, `-beta`, `-rc` suffixes → beta channel + +## Directory Symlinks (Aliases) + +These are directory-level symlinks. They share all files (including +releases.conf) with their target automatically. + +``` +msvc-runtime → vcruntime +msvcruntime → vcruntime +rust.vim → vim-rust +vc-redist → vcruntime +vc-runtime → vcruntime +vc_redist → vcruntime +vcredist → vcruntime +vcruntime140 → vcruntime +vim-essential → vim-essentials +vim-mouse → vim-gui +vps-myip → myip +xcode-cli → commandlinetools +``` diff --git a/scripts/deploy-webicached.sh b/scripts/deploy-webicached.sh new file mode 100755 index 0000000..5f71b4a --- /dev/null +++ b/scripts/deploy-webicached.sh @@ -0,0 +1,63 @@ +#!/bin/sh +set -e +set -u + +# Build and deploy webicached to a target host + +g_host="${1:-beta.webi.sh}" +g_bin="webicached" +g_out="agents/tmp/${g_bin}" +g_remote_bin="~/bin/${g_bin}" + +case "${g_host}" in + beta.webi.sh) g_remote_conf="~/srv/beta.webinstall.dev/installers/" ;; + next.webi.sh) g_remote_conf="~/srv/next.webinstall.dev/installers/" ;; + *) g_remote_conf="~/srv/webid/installers/" ;; +esac + +fn_build() { + b_version="$(git describe --tags --always 2> /dev/null || echo '0.0.0-dev')" + b_commit="$(git rev-parse --short HEAD)" + b_date="$(date -u +%Y-%m-%dT%H:%M:%SZ)" + b_ldflags="-X main.version=${b_version} -X main.commit=${b_commit} -X main.date=${b_date}" + + printf 'Building %s %s %s (%s)...\n' "${g_bin}" "${b_version}" "${b_commit}" "${b_date}" + GOOS=linux GOARCH=amd64 GOAMD64=v2 go build -ldflags "${b_ldflags}" -o "${g_out}" ./cmd/webicached + printf 'Built: %s\n' "${g_out}" +} + +fn_deploy() { + printf 'Stopping %s on %s...\n' "${g_bin}" "${g_host}" + ssh "${g_host}" "~/.local/bin/serviceman stop ${g_bin}" 2> /dev/null || true + + printf 'Uploading binary...\n' + scp "${g_out}" "${g_host}:${g_remote_bin}" + + printf 'Syncing releases.conf files...\n' + rsync -av \ + --exclude='_cache' --exclude='.git' --exclude='agents' \ + --exclude='bin' --exclude='cmd' --exclude='internal' \ + --exclude='docs' --exclude='scripts' --exclude='node_modules' \ + --include='*/' --include='releases.conf' --exclude='*' \ + ./ "${g_host}:${g_remote_conf}" + + printf 'Starting %s...\n' "${g_bin}" + ssh "${g_host}" "~/.local/bin/serviceman start ${g_bin}" +} + +fn_verify() { + printf 'Waiting 5s for startup...\n' + sleep 5 + + printf 'Checking version...\n' + ssh "${g_host}" "${g_remote_bin} -V" + + printf 'Checking logs...\n' + ssh "${g_host}" "sudo journalctl -u ${g_bin} --no-pager -n 5" +} + +fn_build +fn_deploy +fn_verify + +printf '\nDone. %s deployed to %s.\n' "${g_bin}" "${g_host}" diff --git a/skills/deploy-webicached/SKILL.md b/skills/deploy-webicached/SKILL.md new file mode 100644 index 0000000..379979a --- /dev/null +++ b/skills/deploy-webicached/SKILL.md @@ -0,0 +1,111 @@ +--- +name: deploy-webicached +description: Deploy webicached binary to beta.webi.sh. Use when building, uploading, or restarting the cache daemon. Covers cross-compile, conf sync, service management. +--- + +## One-step deploy + +```sh +./scripts/deploy-webicached.sh beta.webi.sh +``` + +Builds with version ldflags, stops service, uploads, syncs conf, starts, verifies. + +## Manual steps (if needed) + +### Build + +```sh +VERSION="$(git describe --tags --always)" +COMMIT="$(git rev-parse --short HEAD)" +DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)" +GOOS=linux GOARCH=amd64 GOAMD64=v2 go build \ + -ldflags "-X main.version=${VERSION} -X main.commit=${COMMIT} -X main.date=${DATE}" \ + -o agents/tmp/webicached ./cmd/webicached +``` + +MUST: Build from the `ref-webi-go` worktree (or branch containing `cmd/webicached`). + +### Deploy + +```sh +ssh beta.webi.sh "serviceman stop webicached" +scp agents/tmp/webicached beta.webi.sh:~/bin/webicached +``` + +MUST: Stop service before scp — Linux refuses to overwrite a running binary. + +### Sync releases.conf + +```sh +rsync -av --include='*/' --include='releases.conf' --exclude='*' \ + ./ beta.webi.sh:~/srv/beta.webinstall.dev/installers/ +``` + +MUST: Run from the worktree root. The server has no checkout of this branch — conf files must be synced explicitly. + +### Start + +```sh +ssh beta.webi.sh "serviceman start webicached" +``` + +### Verify + +```sh +ssh beta.webi.sh "sleep 5 && serviceman logs webicached" +``` + +Expected: "batch: N stale, refreshing 20" or "all packages fresh, sleeping 9s" + +## Smoke test + +```sh +ssh beta.webi.sh "curl -sSf http://localhost:3080/api/releases/bat.json | head -c 100" +ssh beta.webi.sh "curl -sSf -A 'curl/7.81.0 Linux x86_64' http://localhost:3080/api/installers/bat.sh | head -3" +``` + +Expected: JSON array with release objects; shell script with `PKG_NAME='bat'`. + +## Service management + +```sh +serviceman status webicached +serviceman restart webicached +serviceman logs webicached +``` + +## Server layout + +| Path | Purpose | +|------|---------| +| `~/bin/webicached` | Binary | +| `~/srv/beta.webinstall.dev/installers/` | Conf dir (releases.conf files) | +| `~/.cache/webi/legacy/` | Cache output (fsstore, legacy JSON format) | +| `~/.cache/webi/raw/` | Raw upstream API responses | +| `~/srv/beta.webinstall.dev/.env.secret` | GITHUB_TOKEN | +| `/etc/systemd/system/webicached.service` | Service unit (created by serviceman) | + +## Flags reference + +| Flag | Default | Purpose | +|------|---------|---------| +| `--conf` | `.` | Dir with `{pkg}/releases.conf` files | +| `--legacy` | `~/.cache/webi/legacy` | Legacy cache output directory | +| `--raw` | `~/.cache/webi/raw` | Raw upstream response cache | +| `--token` | `$GITHUB_TOKEN` | GitHub API token | +| `--interval` | `9s` | Delay between package fetches in a batch | +| `--once` | false | Run once then exit | +| `--eager` | false | Fetch all on startup (not staleness-based) | +| `--shallow` | false | Only first page of releases | +| `--no-fetch` | false | Classify from rawcache only | +| `--page-delay` | `2s` | Delay between paginated API pages | + +## One-shot refresh (specific packages) + +```sh +ssh beta.webi.sh ". ~/srv/beta.webinstall.dev/.env.secret && ~/bin/webicached \ + --conf ~/srv/beta.webinstall.dev/installers/ \ + --raw ~/.cache/webi/raw \ + --once bat goreleaser" +```