Upstream uses non-standard OS/arch names (x64, ia32, win32, arm) and
ships both bare binaries and .gz-compressed copies. classifyFFmpegDist
maps those to canonical names and keeps only bare binaries.
Also adds source-override logic to installerconf so that
github_releases + source = ffmpegdist works: GitHub is used for
fetching while the custom classifier handles classification.
Replaces the inline &http.Client{Timeout: 30s} with httpclient.New(),
which enforces TLS 1.2+, per-level timeouts, no HTTPS→HTTP redirect
downgrade, connection pooling, and automatic retry with backoff.
The delayTransport (page-delay flag) now wraps httpclient's transport
instead of http.DefaultTransport, preserving all security properties.
When a package has a git_url but uses a non-gittag source, the
supplementary git clone was always shallow. For packages never cloned
before, a shallow clone may miss older tags that clients need.
Now: check whether the _gittag raw cache is already populated. If it is,
reuse the shallow flag (fast refresh). If it is not, force a full clone
so all tags are available from the first fetch.
The --shallow flag (global) still overrides this so operators can cap
fetch depth when needed.
Rescans the conf directory at the start of each batch loop so new
{pkg}/releases.conf files dropped on disk are picked up without a restart.
Also runs a rescan after each individual package refresh mid-batch. If a
new conf is discovered, the inner loop breaks immediately so the outer
loop recomputes staleness — new packages have a zero timestamp and sort
to position 1, meaning they are fetched in the very next slot.
Deploy scripts for webicached and webid (build, upload, restart).
AGENTS.md with releases.conf reference and variant tagging docs.
Installer archive pattern guide and version oddities reference.
Rewrites the Node.js release classification pipeline in Go. webicached
fetches upstream releases (GitHub, Gitea, GitLab, HashiCorp, custom
sources), classifies assets by OS/arch/variant, and writes legacy-format
JSON caches compatible with the existing webinstall.dev API.
Git-clone packages emit git_tag and git_commit_hash from real repo
clones — no fabricated refs.
CYGWIN_NT-* and MINGW64_NT-* UAs (Git Bash / Cygwin on Windows) were
classified as linux, so Windows users got linux binaries or no match.
Three fixes:
- build-classifier v1.0.4: CYGWIN/MINGW → windows in termsToTarget
- ua-detect.js: same fix for the Node server's UA detection path
- builds-cacher.js: default hostTarget.libc to 'libc' when unset —
termsToTarget omits libc for plain UAs, causing triplets like
'linux-x86_64-undefined' that never matched cache entries
Stacked on the modifications PR. Now that no live code path references
the per-package fetchers, the shared HTTP/parsing helpers, the
in-process normalizer, or the example template, delete them. Pure
deletion — no behavior change.
- ~93 per-package <pkg>/releases.js fetcher modules.
- _common/{brew,fetcher,git-tag,gitea,github,github-source,
githubish,githubish-source}.js shared HTTP/parsing helpers.
- _webi/normalize.js in-process normalization layer (cache files
arrive normalized from webicached).
- _example/releases.js fetcher template for new packages.
The Go cache daemon (webicached) is now the sole producer of release
metadata; the Node process never makes an upstream request.
The Node server's read path now goes through ~/.cache/webi/legacy/ only
(see #1075). A handful of supporting tools and tests still carried
references to the obsolete upstream-fetcher modules and the old
year-month cache layout. Update them in place; the actual deletion of
the orphaned modules follows in #1076.
- _webi/classify-one.js — read from ~/.cache/webi/legacy/<pkg>.json
instead of ../_cache/<yearMonth>/<pkg>.json.
- _webi/builds-cacher-test.js — drop the bc.freshenRandomPackage(...)
call; the freshener was removed when fetching went away.
- _webi/builds.js — drop the //Releases: Releases stub comment.
- _webi/lint-builds.js — drop two now-unused require()s.
- _webi/test.js — adjust a single reference to the post-cleanup shape.
filterReleases unconditionally rejected libc=musl entries unless the
host was libc=musl, even when the caller never specified a libc in
the request. serve-releases.js defaults the libc parameter to 'libc'
(the catch-all glibc-host bucket the installer-side resolver uses),
so the website's release table and the WEBI_RELEASES probe were both
stripped of every musl entry that the cache actually contained — even
though the installer would happily consider those builds on a glibc
host (its waterfall is [none, gnu, musl, libc]).
Treat libc='libc' (and missing) as 'no preference' so the filter only
runs when the caller pinned a real libc (musl, gnu, msvc, etc.).
Specific-libc queries (?libc=musl, ?libc=gnu) still filter exactly as
before.
Lexver.parsePrefix now produces a true string-prefix of parseVersion
when the input has a release suffix (e.g. '1.0.0-beta',
'2025.11.15-15.42.45'). Unblocks pinned-version queries with a
non-trivial release suffix, including the 'webi zig.vim' alias chain
which redirects through 'vim-zig@2025.11.15-15.42.45' at install time.
See webinstall/webi-build-classifier#22.
Make _webi/builds-cacher.js and _webi/transform-releases.js read
exclusively from ~/.cache/webi/legacy/<name>.json and remove every code
path that fetched from upstream or wrote to disk. The Go cache daemon
(webicached) is now the sole writer; the Node server is a thin reader.
builds-cacher.js:
- Resolve cache files via Os.homedir() + '/.cache/webi/legacy/' instead
of the cacheDir argument. Drop the 'caches' constructor parameter.
- Remove getLatestBuilds / getLatestBuildsInner — they require()d
per-package releases.js modules, fetched upstream, and wrote
<yyyy-mm>/<name>.json + .updated.txt to disk.
- Remove the process.nextTick stale-refresh hook in _doGetPackages.
Cold reads return what's on disk; if the file is missing, return
empty meta instead of fetching.
- Remove freshenRandomPackage and its supporting state
(bc._staleNames, bc._freshenTimeout, bc._staleAge). The hourly
background freshener competed with webicached for the same files.
- In getProjectTypeByEntry, decide selfhosted vs valid by probing for
the cache file rather than require()-ing releases.js. Drop the
not_found / 'PROBLEM/SOLUTION/npm clean-install' diagnostic in
getProjectsByType — the cache-file probe replaces the module-load
failure mode.
transform-releases.js:
- Remove Releases.get and the _normalize import. Replace
getCachedReleases's fetch+race+stale-age machinery with a single
Fs.readFile of ~/.cache/webi/legacy/<pkg>.json.
- Drop the in-process version re-sort in createFormatsSorter; the
cache file is already version-sorted by webicached, so the sorter
only re-orders within the same version.
No callers' public signatures change. Every other file is untouched —
the per-package releases.js modules, _common/*.js fetchers, and
_webi/normalize.js still exist on disk but are no longer reachable
from the request path.
Pulls in webinstall/webi-build-classifier#21 (merged 2026-05-07,
SHA 574eff5) and the host-target x64/win32 fix from #20 (SHA 71c0768)
that landed alongside it.
#21 fixes `maybeInstallable` rejecting any package version ending in
`.1` whose download URL is a GitHub source-archive endpoint
(`/tarball/vX.Y.1` or `/zipball/vX.Y.1`). Without it, this PR's
`_enumerateTriplets` priority fix is undermined: even after picking
the correct posix_2017 triplet, the newest version (e.g. serviceman
v1.0.1) is silently dropped by the classifier and the resolver falls
back to v1.0.0.
Confirmed on next.webi.sh after deploying this branch with the bumped
submodule: `serviceman@stable.sh` now resolves to v1.0.1/zip on macOS
arm64 (was v1.0.0/zip with the pre-rebase pre-fix submodule).
In _enumerateTriplets, the order of `oses` and `arches` was
ANYOS/ANYARCH first, specific second. This caused findMatchingPackages
to pick the most-generic triplet (e.g. ANYOS-ANYARCH-none) before
trying specific OS triplets — and packages that have a wildcard git
fallback alongside per-platform binaries would resolve to the git
source instead of the binary, even when the client never asked for
git as an unpacker.
Reverse the order so specific platforms win:
- oses: hostTarget.os, posix_2017, posix_2024, ANYOS
- arches: arches.concat(['ANYARCH'])
Concrete example: serviceman has both posix_2017/*/tar.gz and
*/*/git in the cache. Pre-fix, findMatchingPackages picks
ANYOS-ANYARCH-none (containing only the .git entry). The .git gate
in getSortedFormats then correctly excludes git from format
candidates, but the chosen triplet has nothing else, so selectPackage
falls through to packages[0] = git entry. Post-fix,
findMatchingPackages picks posix_2017-ANYARCH-none first (containing
[tar.gz, zip]), and selectPackage returns tar.gz.
webi/webi.sh: detect unzstd/zstd alongside the existing git/unxz/
unzip/tar probes. Sends `?formats=...,zst` when zstd is available so
the server can pick a .tar.zst build only on hosts that can extract
it.
webi/webi-pwsh.ps1: replace the hardcoded `formats=zip,exe,tar,git`
TODO with real Get-Command probing for git, zstd, and 7z.
`.git` was pushed unconditionally into getSortedFormats's candidate
ext list, while sibling unpacker formats (.tar.xz, .tar.zst, .zip,
.7z) are gated on whether the caller's `formats` argument signals
the client has the corresponding tool.
Make `.git` consistent: only add it to the candidate list when
formats includes 'git'. The default WEBI_FORMATS ('tar,exe,zip,xz,
dmg') doesn't include git, so the change is a no-op for the
current default. Clients that want git-source packages installed
can pass `?formats=tar,exe,zip,xz,dmg,git` (or set the equivalent
in a future client-side probe).
For packages that have only a git-source asset (e.g. some vim
plugins), the existing fallback to `packages[0]` still returns the
git entry — behavior unchanged. The only observable change is for
packages where both a binary and a git fallback exist for the same
triplet: previously the git entry could win over the binary; now
it wins only when the client opts in.
When two HTTP requests arrived simultaneously for the same package on
a cold in-memory cache (bc._caches[name] === undefined), they would
both:
1. Enter getPackages, see no warm cache,
2. Read and parse the same _cache/{pkg}.json independently,
3. Both call transformAndUpdate, which re-runs _classify on every
build.
The first call populates bc._targetsByBuildIdCache as it classifies.
The second call then hits the cache shortcut at the top of _classify
and skips the projInfo.oses/arches/libcs/formats/triplets
accumulation block entirely. Its projInfo ends up with empty tracking
arrays (because the prior Object.assign(projInfo, meta) reset them),
and that poisoned projInfo gets written to bc._caches[name],
overwriting the first call's good cache.
After this, every subsequent installer request returns errPackage
because serve-installer.js checks projInfo.oses.includes(hostTarget.os)
— and projInfo.oses is now [].
Fix: a per-name in-flight promise. Concurrent callers for a cold
package share a single load. Calls for warm packages take the fast
path with no synchronization.
Reproduced reliably with Promise.all of 6 cold-cache calls for the
same package: 1/6 succeeded before the fix, 6/6 after. On staging at
HTTP concurrency=12, installer cand-only-errors went from 24-229
(cause-dependent) to 0.
Adds releases.js, install.sh, install.ps1, and README.md for monorel,
a Go monorepo release tool from therootcompany/golib. Filters monorepo
releases by tools/monorel/ prefix and auto-installs prerequisites
(git, gh, goreleaser).
