Track year-by-year format changes for all packages. Identify structurally
significant changes (sd, ollama, caddy, deno, gh, hugo) vs cosmetic ones.
Most packages have stable formats — only ~11 have changes requiring
different install script eras.
Inspected archive contents of 60+ packages and categorized into patterns:
A) bare binary in archive (most common, ~28 packages)
B) subdirectory with binary only
C) subdirectory with completions/man pages (Rust tools)
D) complex with shared libraries (ollama, psql, sass, syncthing)
E) FHS-like layout with bin/ (gh, ollama)
F) renamed binary needing install-time rename (pathman, yq)
Document format evolution and install structure for sd (simple single-binary
with completions) and ollama (complex multi-lib GPU-accelerated server).
Track archive layouts, format changes across versions, and draft install
scripts targeting ~/.local/opt/<pkg>-<ver>/bin/.
When a GitHub release has no binary assets, fall back to tarball_url and
zipball_url. These are source distributions (platform-independent), marked
with extra=source.
- serviceman: 12 distributables (6 releases × tar.gz + zip)
- aliasman: 8 distributables (4 releases × tar.gz + zip)
- duckdns.sh: 6 distributables (3 releases × tar.gz + zip)
Total: 170,213 rows across 116 packages (no more zeros).
- .app.zip and .dmg formats now infer darwin OS when absent
- Filter .tgz (npm packages) and .d.ts (TypeScript defs) as meta-assets
- Reduces bun false positives by 64, deno by 294
- Add cmd/classify: reads raw cached releases and produces a CSV of all
distributables with sortable version columns (ver_major/minor/patch/pre)
- Export rawcache.ActivePath() for use by cmd/classify
- Add OS detection: openbsd, netbsd, dragonflybsd, plan9, mac→darwin
- Add arch detection: armv5, armhf→armv7, arm7→armv7, 386→x86,
32bit/64bit (no hyphen), universal→universal2, riscv64, loong64,
mipsle, mips64le
- Infer Linux from .deb/.rpm format when OS not in filename
- Add .deb and .rpm as recognized formats
- Normalize all per-source values to buildmeta vocabulary (x86_64, aarch64)
- Filter source archives and buildable-artifact meta-assets
- Add CAT-RULES.md tracking classifier learnings
- Add CATEGORIZED.md and LINKS.md for reference
Batch 1 tested: go, node, hugo, caddy, pathman (35,919 rows)
Vim plugins with gittag source:
- vim-airline, vim-airline-themes, vim-ale, vim-devicons, vim-go
- vim-nerdtree, vim-prettier, vim-rust, vim-sensible, vim-shfmt
- vim-syntastic
rust.vim is a directory symlink to vim-rust, so it shares the same
releases.conf automatically.
Alias confs (alias_of):
- postgresql → postgres
- postgresql-client, postgres-client → psql
- mariadb-server, mariadbd → mariadb
- gnupg → gpg, iterm → iterm2, ziglang → zig
- trippy → trip, powershell → pwsh
Fix: psql is its own package (postgres client), not an alias of
postgres (server). Both use the same GitHub repo
(bnnanet/postgresql-releases) but install different binaries.
New fetcher packages:
- chromedist: Chrome for Testing API (googlechromelabs.github.io)
- gpgdist: SourceForge RSS for GPG macOS
- mariadbdist: MariaDB downloads REST API
New releases.conf files for:
- GitHub: aliasman, awless, duckdns.sh, hugo-extended, kubens, rg, postgres
- gittag: vim-commentary, vim-zig
- gitea: pathman
- chromedist: chromedriver
- gpgdist: gpg
- mariadbdist: mariadb
- nodedist: node
Alias support (alias_of key):
- golang → go, dashd → dashcore, psql → postgres, zig.vim → vim-zig
- Aliases skip fetching and share cache with their target
Every package with a releases.js now has a releases.conf (except the
dead macos package). fetchraw dispatches to all 13 source types.
New fetcher packages:
- golang: golang.org/dl/?mode=json&include=all
- zigdist: ziglang.org/download/index.json
- flutterdist: Google Storage per-OS release indexes
- iterm2dist: scrapes iterm2.com/downloads.html
- hashicorp: releases.hashicorp.com/{product}/index.json
- juliadist: julialang-s3.julialang.org/bin/versions.json
Each follows the same iter.Seq2 pattern as the existing nodedist/github
fetchers. Added releases.conf files for all six packages and wired them
into cmd/fetchraw.
Fixed latest-version detection for sources that return unordered data
(hashicorp, zigdist, juliadist) by comparing all versions with lexver
instead of taking the first stable one found.
The discover() function now skips directories starting with _ (like
_example, _webi, _common) so infrastructure dirs aren't treated as
packages to fetch.
Discovers packages by globbing {confDir}/*/releases.conf. Adding a
new package is now just creating a conf file — no Go code changes.
Dispatches to the right fetcher based on source= (github, nodedist).
Simple key=value config per package declaring the fetch source and
its parameters. Greppable, no dependencies needed to parse.
grep 'source = github' */releases.conf
grep 'owner = therootcompany' */releases.conf
70 packages configured. installerconf package provides the reader.
fetchraw will be updated to read these instead of a hardcoded list.
Declarative key=value config files that specify the release source
(github or nodedist), owner/repo, and optional tag_prefix for
monorepo packages. These replace the per-package releases.js logic
for the Go rewrite.
- rawcache: add Merge() that skips unchanged releases, logs added/
changed events to an append-only JSONL audit log with SHA-256
- rawcache: drop .json extension from filenames — raw cache stores
opaque bytes (upstream may be JSON, CSV, XML, or bespoke)
- fetchraw: add all 68 GitHub packages, use Merge instead of Put
- fetchraw: log format shows +added ~changed =skipped
Put directly into the active slot instead of BeginRefresh. Existing
releases are skipped (Has check), new ones are added, _latest is
only updated if the candidate is newer. Safe to run repeatedly —
backports and delayed releases accumulate without losing history.
Fetches complete release histories from upstream APIs and stores
them in rawcache. Supports GitHub (with pagination, auth, monorepo
tag prefix filtering) and Node.js dist API (official + unofficial
as separate caches to avoid version collisions).
Tested with: node-official (834), node-unofficial (387),
hugo (365), caddy (134), monorel (3).
Reflect completed work (all fetchers, rawcache, classify, platlatest,
CompatArches), update repo layout to match actual packages, document
the fallback/compatibility design (classifier is 80/20 default,
per-installer config is the authority), add open questions for CPU
micro-arch detection and installer config format.
CompatArches returns what a given OS+arch can execute — OS-level
facts like Rosetta 2 (darwin arm64 runs x86_64), Windows ARM
emulation, and x86-64 micro-arch backward compat. Also adds
ArchUniversal1 (PPC+x86) and ArchUniversal2 (x86_64+ARM64).
Per-package/per-version overrides (libc compat, nonstandard naming)
remain the installer config's responsibility.
Arch and libc fallbacks are not universal — they depend on the OS,
the package, and even the version. ARM64 on macOS/Windows can run
x64 (Rosetta/emulation) but not on Linux. Musl can be static or
dynamically linked depending on the package version. Windows GNU
may or may not need mingw.
These rules belong in per-installer config, not in shared types.
platlatest stays as a simple fact store (triplet → version).
Resolution with fallbacks will be the caller's job.
Prefer latest version over best CPU match. An amd64v4 machine gets
v2.0.0 (baseline only) instead of v1.0.0 (which had a v4 build)
because recency beats specificity.
- buildmeta: add amd64v2/v3/v4 micro-levels, ArchFallbacks, LibcFallbacks
- classify: detect micro-arch levels, treat Windows "arm" as ARM64
- platlatest: add Resolve() that walks fallback chains picking newest
classify extracts OS, arch, libc, and format from release asset
filenames using regex pattern matching with priority ordering
(x86_64 before x86, arm64 before armv7, etc.).
platlatest tracks the newest release version per build target
(OS+arch+libc triplet) to handle the common case where Windows
or macOS releases lag behind Linux by several versions.
Stores one JSON file per release, named by tag. Supports:
- Incremental updates: atomic writes to the active slot
- Full refresh: write to standby slot, atomic symlink swap
- O(1) existence check and latest-tag lookup
For packages installed from auto-generated source tarballs rather
than uploaded binary assets (shell scripts, vim plugins, etc.).
Each delegates to its respective forge fetcher — the distinction
is organizational, signaling which fields the consumer should use.
Gitea's API is similar to GitHub's but not identical (different URL
prefix, limit vs per_page, token auth header). Give it its own types
and pagination logic rather than coupling through githubish.
GitLab's API differs from GitHub: different URL pattern
(/api/v4/projects/:id/releases), nested asset structure
(sources + links), page/per_page pagination with X-Total-Pages
header, and PRIVATE-TOKEN auth.
Some packages (shell scripts, vim plugins) use the auto-generated
source archives rather than uploaded binary assets. These URLs are
already in the API response — just needed to be deserialized.
gitea: thin wrapper over githubish that appends /api/v1 to the base URL.
gittag: clones/fetches a bare repo, lists version-like tags with
commit metadata, includes HEAD. For packages installed by cloning
(vim plugins, shell scripts) rather than downloading binaries.
githubish: generic fetcher for any GitHub-compatible API (GitHub,
Gitea, Forgejo). Paginates via Link headers, supports Bearer auth.
Returns raw API data with no transformation.
github: thin wrapper that sets the base URL to api.github.com.
nodedist: generic fetcher for any Node.js-style dist index.json API.
Returns raw API entries with no transformation or normalization.
Uses iter.Seq2 for a paginated interface consistent across sources.
node: calls nodedist twice — official builds and unofficial builds
(musl, loong64, etc.) — yielding one batch per source.
The user agent identifies itself through multiple signals — the
User-Agent header and query parameters (?os, ?arch). FromRequest
unifies both, with explicit query params taking precedence.
buildmeta: remove premature Release/PackageMeta structs and
ChannelNames slice — keep only the shared vocabulary types.
uadetect: replace regex-based matching with token-based matching.
Split UA on whitespace/slash/semicolon, match lowercase tokens.
Strip xnu kernel info for Rosetta. Single Parse() entry point.
httpclient: return plain *http.Client from New(). Make Do() and Get()
free functions. Only retry idempotent methods (GET/HEAD).
Original preserves the upstream tag as the releaser published it (e.g.
"REL_17_0"), while Raw holds Webi's normalized form ("17.0").
ExtraSort is an opaque string for package-specific ordering where Nums
alone can't capture sort order (e.g. flutter "2.3.0-16.0.pre"). Set by
release-fetcher code using zero-padded strings or whatever works for
that package.
Versions aren't always semver — chromedriver uses 4 parts, gpg uses 4
parts, atomicparsley uses dates. Replaced fixed Major/Minor/Patch/Build
fields with a Nums slice. Date is now time.Time for minute-level
precision.
Also adds ODDITIES.md cataloging non-standard version formats across
Webi packages for future reference.
Build is often a hash with no ordering meaning. When release dates are
known, they're a better tiebreaker for versions with the same
major.minor.patch. Date is only used when both sides have one.
Instead of encoding versions as padded strings (a JS-ism), parse into a
Version struct and compare fields directly. Pre-releases sort before
their corresponding stable release via channel comparison.
Each package doc now explains what problem it solves and why it exists,
with the public interface as the only "how" detail. Implementation
notes removed from doc comments.
webid still discovers new packages from storage without restart.
webicached knows its package set at startup and periodically refreshes
releases to both Postgres and filesystem.
Captures the full migration plan: architecture, API surface inventory,
storage double-buffer design, incremental migration phases, and key
design decisions. Not a straight port — redesigning internals while
preserving the public API contract.
Adds releases.js, install.sh, install.ps1, and README.md for monorel,
a Go monorepo release tool from therootcompany/golib. Filters monorepo
releases by tools/monorel/ prefix and auto-installs prerequisites
(git, gh, goreleaser).
