Replace direct string comparison with canonical equivalence checks so
naming convention differences (darwin/macos, x86_64/amd64, aarch64/arm64)
don't appear as false diffs. Now only real classification disagreements
surface:
- go: illumos/solaris→sunos mapping, arm ambiguity per OS
- sass: bare "arm" should be armv7, not armv6
- ffmpeg: Windows .gz ext classified as exe in prod
- terraform: alpha channel detected correctly by Go, missed by prod
- postgres: legacy EDB ext "tar" vs "tar.gz"
- pg/releases.conf: add asset_filter=postgres so pg only returns server
assets (which include the client), matching production releases.js
- classifypkg: add "pg" to postgres version normalizer switch case
- comparecache: compare os/arch/libc/ext/channel fields on shared assets,
distinguishing real disagreements (diff-*) from expected fill diffs
where Go classifies at write time but Node.js leaves fields empty
9 categories: universal2, solaris/illumos, armhf, armel, windows arm,
android, winx64, minor arch mismatches, sttr pkg misclassification.
Plus broad sweep failures and live-compare known diffs.
- Windows gnu (MinGW) builds are self-contained: classify as libc='none'
- Pad install.sh content to 8 spaces to match production template indent
- Use replaceMarkerLine for both bash and PS1 installer injection
Rust *-unknown-linux-musl builds are statically linked with zero
runtime libc dependency. Detect this pattern in classifyGitHub and
override libc from 'musl' to 'none'. Hard-musl packages (pwsh, bun,
node) use different filename patterns and keep libc='musl'.
Add PowerShell() function to render .ps1 installers by injecting
$Env: variables and splicing install.ps1 content. Wire it into
the webid server for .ps1 extension requests.
git_url is now a standalone field that can appear alongside any source
type. For githubsource packages, it adds a git clone entry per release
in addition to the tarball and zipball. Updated aliasman, duckdns.sh,
and serviceman configs.
Three distinct fetch/classify strategies:
- github: binary assets only, no source entries
- githubsource: tarball + zipball from GitHub releases API
- gittag: git clone + tag enumeration (existing)
GitHub binary packages (caddy, jq, shellcheck, etc.) no longer get
spurious .git and source tarball entries for old releases that had
no binary uploads. Source-installable packages (aliasman, duckdns.sh,
serviceman) now use github_source in releases.conf.
-sample N now randomly samples N assets from each package's diff list,
giving a representative view of classification differences instead of
showing only the first alphabetical entries. Implies -windowed -diffs
to filter out version-depth noise and focus on real bugs.
Tests were using separate source/owner/repo keys but the parser expects
github_repo=owner/repo, gitea_repo=owner/repo, etc. Fixed all test
configs to match. Also answered Issue 4 (darwin-universal) for other agent.
Usage: go run ./cmd/comparecache -sample 8 -diffs
Picks 8 random packages beyond any explicitly named ones, logs which
ones were sampled for reproducibility.
Production has two separate flows:
1. /{pkg} (curl-pipe bootstrap) — minimal script that sets WEBI_PKG,
WEBI_HOST, WEBI_CHECKSUM and downloads+runs webi
2. /api/installers/{pkg}.sh — full installer with resolved release
and embedded install.sh
Previously handleBootstrap served the full installer. Now:
- handleBootstrap: curl-pipe bootstrap (reads curl-pipe-bootstrap.tpl.sh)
- handleInstaller: full installer (/api/installers/{pkg}.sh)
Also:
- Export render.InjectVar for use by bootstrap handler
- Add webi.sh checksum calculation (SHA-1 first 8 chars)
- Add /api/installers/ route to mux and test server
The commaToTab byte replacement was fragile — URLs containing commas
would break. Now uses csv.Writer with Comma='\t' as the backend for
csvutil.Encoder, producing correct TSV output regardless of field content.
- Added TestV1ResolveJQ to verify jq resolves to binary, not git
- Changed upstream gap detection in resolve_cache_test to t.Skipf
(shellcheck/windows and xz/linux-arm64 don't have upstream builds)
- Updated ANSWERS.md with git assets investigation results
New API routes:
- GET /v1/releases/{pkg}.tab — list releases as TSV (with header)
- GET /v1/releases/{pkg}.json — list releases as JSON array
- GET /v1/resolve/{pkg}.tab — resolve best asset for platform (TSV)
- GET /v1/resolve/{pkg}.json — resolve best asset for platform (JSON)
Key design decisions:
- TSV as primary format via csvutil (easy for cut/grep/sort/agents)
- Go-native naming: darwin, x86_64, aarch64 (no legacy mapping)
- No quoted fields — spaces for lists within fields
- Always includes header row in TSV output
- Resolve endpoint returns single best match with triplet info
Query params: os, arch, libc, channel, version, lts, format, variant, limit
- JSON response returns bare array (not wrapped in {"releases": [...]})
- OS names mapped to Node.js conventions: darwin → macos
- Arch names mapped: x86_64 → amd64, aarch64 → arm64
- Version strings stripped of "v" prefix
- Extension stripped of "." prefix
- Empty libc defaults to "none"
- Tab format uses actual TSV (not comma-separated)
- Tab LTS field uses "lts" / "-" (not "true" / "false")
- Tab shows header row only with ?pretty=true
- Releases sorted newest-first by version (using lexver)
- Added comprehensive format tests and production comparison test
Renders package-install.tpl.sh with WEBI_* variable injection and
install.sh splicing. Bootstrap route at /{package}@{version} detects
UA, resolves best release, and returns rendered installer script.
Serves /api/releases/{pkg}@{version}.json and .tab matching the
Node.js format. Supports query params for os, arch, libc, channel,
formats, lts, limit. Handles selfhosted packages (install.sh only).
Pre-loads all cached packages on startup. Includes /api/debug for
UA detection and /api/health endpoint.
Triplet-based resolution with indexed lookup for fast matching.
Supports channel hierarchy (alpha > beta > rc > stable), LTS filtering,
variant selection, format preferences, and arch fallback via CompatArches.
All 13 unit tests and cache integration tests pass against real data
for 100+ packages.
Detailed instructions for the next step: making the Node.js server
read only from Go-generated _cache/ files, removing all upstream
API fetching from the Node.js code path.
Source type is now inferred from the primary key:
github_repo = owner/repo (was source=github + owner + repo)
git_url = https://... (was source=gittag + url)
gitea_repo = owner/repo (was source=gitea + owner + repo)
hashicorp_product = name (was source=hashicorp + product)
One-off dist sources (nodedist, zigdist, etc.) keep the explicit
source= key since they're already one-liners.
Parser still accepts the old format via the default fallback branch.
Tagless repos (only HEAD, no real version tags): rewrite HEAD version
to Node.js-compatible format (v2023.10.10-18.42.21) with full UTC
datetime.
Repos with real tags + HEAD: tag HEAD entries with "head" variant so
ExportLegacy filters them out (they shouldn't appear in legacy cache).
Hardcode the old 10.12, 10.13, 11.8, 12.3 releases from EnterpriseDB
that predate the bnnanet/postgresql-releases GitHub repo. Both postgres
and psql now match the live cache exactly.
- Default mode: classify all from rawcache on startup, then
fetch+refresh one package per tick (round-robin).
- --eager flag for the old behavior (fetch all on startup).
- Skip aliases and symlinked dirs — legacy cache doesn't create
entries for them (resolved at request time by the server).
- Add --page-delay (default 2s) to rate-limit paginated API requests.
- Add delayTransport wrapper on http.Client.
Symlinked directories (e.g. rust.vim → vim-rust) are now treated as
aliases instead of being independently fetched and classified. Creates
cache symlinks just like alias_of config entries.
Early PowerShell releases (pre-6.1) used Windows-version-specific
filenames (win10-win2016, win81-win2012r2) that the legacy cache
can't resolve. Tag them as variants so they're filtered from legacy
export but preserved for future Go resolver use.
The regex captured the beta/preview number but not the keyword itself,
so "3.0.0-preview" collapsed to "3.0.0". Also deduplicate by version
since the downloads page has duplicate links with different URL formats
(e.g. iTerm2-3_5_1beta1.zip and iTerm2-3_5_1_beta1.zip).
Flutter's API returns separate entries for universal (x64) and arm64
macOS builds under the same version/channel/os. The rawcache tag
was version-channel-os, so arm64 overwrote universal. Now extracts
arch from the archive path and appends it to the tag.
Re-fetched flutter: +218 entries recovered.
Packages with alias_of in releases.conf (e.g. dashd → dashcore,
golang → go) now get symlinked cache files so they resolve to the
same JSON as their target. 13 aliases total.
Added AliasOf as a proper field in installerconf.Conf, LinkAlias
method to fsstore, and alias handling in webicached's Run loop.
GitHub's prerelease boolean is often not set for rc/beta/alpha/dev/pre
releases. Add channelFromVersion() to detect these from the version
string as a fallback. Applied to github, gitea, gittag, and hashicorp
classifiers. Hashicorp's inline checks replaced with the shared helper.
-pre maps to beta (prerelease), -preview stays preview.
The cli- prefix is a watchexec-specific monorepo artifact, not a generic
config concern. Move it to internal/releases/watchexec/versions.go
alongside other per-package normalizers (git, lf).
