vim-ale/PROD_NOTES.md

Production Notes: Node.js Cache-Only Migration

Current State

The Node.js server no longer fetches from upstream APIs. It reads only from _cache/YYYY-MM/{pkg}.json files generated by the Go webicached daemon. All per-package releases.js files and _common/ fetcher libraries have been deleted — they are no longer needed.

Changes Made

• builds.js: Removed freshenRandomPackage() calls and background refresh
• builds-cacher.js: Removed getLatestBuilds(), stale re-fetch, and freshenRandomPackage(). Missing cache files return empty metadata.
• builds-cacher.js getProjectTypeByEntry(): Replaced require(releases.js) with cache file existence check. Packages are valid if they have a cache file, selfhosted if they don't.
• transform-releases.js: Legacy release API now reads from cache files instead of fetching upstream. Pre-classified fields are cleared before normalize() so the output matches the legacy format (e.g. darwin → macos, versions without v prefix).
• classify-one.js: Updated to read from cache instead of releases.js. Fixed hardcoded triplet key.
• Deleted 94 {pkg}/releases.js files and 8 _common/*.js files

Data Format Note

The Go cache uses different OS/arch names than the legacy normalize.js:

• Go: darwin → normalize.js: macos
• Go: amd64 → normalize.js: amd64 (same)
• Go: aarch64 → normalize.js: arm64

For the legacy /api/releases/ endpoint (transform-releases.js), we clear pre-classified fields and re-normalize from filenames to match production output.

For the installer endpoint (builds-cacher.js + serve-installer.js), the Go naming is used directly — the build-classifier handles its own mappings.

Known Intentional Differences from Production

The Go cache filters releases more aggressively than the old Node normalize.js:

• Excluded: .deb, .rpm, .sha256, .sig, .pem, .sbom, .txt files (non-installable metadata/package-manager assets)
• OS unknown: Some assets that normalize.js couldn't classify get os: "unknown". The Go cache either classifies them correctly or excludes them.
• These are improvements — the filtered results better reflect what webi can actually install.

Libc Taxonomy

Value	Meaning
none	Static build — no runtime libc dependency. Often built with musl but fully self-contained. Works everywhere.
musl	Requires musl C/C++ runtime at runtime (e.g. node-musl). NOT the same as a static musl build.
gnu	Requires glibc at runtime. Crashes on musl-only systems (Alpine).
libc	Host-reported UA value meaning "I have standard C library" (typically glibc). Not used in release metadata.

Resolved Issues

• WATERFALL libc vs gnu+musl (fixed in builds-cacher.js): Glibc hosts now try ['none', 'gnu', 'musl', 'libc']. gnu for glibc-linked builds, musl as fallback for static musl builds (e.g. rg v15+ dropped gnu x86_64).
• ANYOS/.git priority (fixed in builds-cacher.js): Triplet enumeration now puts specific OS before ANYOS and specific arch before ANYARCH.
• Version-first iteration (fixed in builds-cacher.js): findMatchingPackages now iterates versions newest-first then triplets, matching the Go resolver. Prevents selecting ancient versions from low-priority triplets.

Remaining Issues

• hugo darwin-universal: Build classifier rejects darwin-universal as x86_64 != universal2. Hugo v0.100+ only publishes universal macOS .pkg binaries, so darwin-aarch64 has no recent versions. Two fixes needed: (1) classifier maps universal → aarch64+x86_64, (2) .pkg added to accepted formats for macOS. Last .tar.gz darwin builds: v0.99.1.

Known Pre-existing Issues

• transform-releases.js self-test: The if (require.main === module) block calls module.exports({...}) but the module exports an object.
• armhf classification warnings: arm-unknown-linux-gnueabihf triggers "wrong arch" warnings in the build-classifier (armhf vs armv6).
• Go illumos/solaris warnings: Go's illumos and solaris builds trigger "wrong os" warnings (expected sunos).
• webinstall.dev serve-releases.js libc bug: Line 37 calls UaDetect.arch(req.query.libc) instead of UaDetect.libc(...). This means the libc query parameter always evaluates to 'error', effectively disabling libc filtering in the release API. (Bug is in the webinstall.dev repo, not webi-installers.)

Public API Endpoints

The HTTP routing is NOT in this repo — an external server calls into the Node modules. Here's the complete endpoint catalog:

1. Bootstrap / Installer Scripts

GET /{package}@{tag}
GET /{package}@{tag}.sh
GET /{package}@{tag}.ps1

Handler: serve-installer.js:serveInstaller(baseurl, ua, pkg, tag, ext, formats, libc)

Flow:

1. Parse User-Agent → build-classifier/host-targets.js:termsToTarget() → {os, arch, libc}
2. Resolve alias → builds-cacher.js:getProjectType() (symlink or README alias)
3. Load cache → builds-cacher.js:getPackages() → reads _cache/YYYY-MM/{pkg}.json
4. Classify → builds-cacher.js:transformAndUpdate() → triplets, versions, formats
5. Match → builds-cacher.js:findMatchingPackages() → filter by OS/arch/libc/version
6. Select → builds-cacher.js:selectPackage() → pick preferred format
7. Render → installers.js:renderBash() or renderPowerShell() with template vars

UA format (sent by webi bootstrap): {arch}/unknown {OS}/{version} {libc}

• e.g. aarch64/unknown Darwin/24.2.0 libc
• e.g. x86_64/unknown Linux/5.15.0 musl

Template variables injected: WEBI_VERSION, WEBI_PKG_URL, WEBI_PKG_FILE, WEBI_OS, WEBI_ARCH, WEBI_EXT, WEBI_CHANNEL, PKG_NAME

2. Release Metadata API (Legacy)

GET /api/releases/{package}.json
GET /api/releases/{package}@{version}.json
GET /api/releases/{package}.tab
GET /api/releases/{package}@{version}.tab

Handler: transform-releases.js:getReleases({pkg, ver, os, arch, libc, lts, channel, formats, limit})

Flow:

1. Read cache → _cache/YYYY-MM/{pkg}.json
2. Re-normalize → normalize.js (clears pre-classified fields, re-detects from filenames)
3. Filter → filterReleases() by query params
4. Sort → by version (descending), then format preference

Query params: os, arch, libc, lts, channel, formats, limit

Response (JSON): { oses, arches, libcs, formats, releases: [{name, version, lts, channel, date, os, arch, ext, download, libc}] }

Response (TSV): version \t lts \t channel \t date \t os \t arch \t ext \t - \t download

Key format details:

• OS: macos (not darwin), linux, windows
• Arch: arm64 (not aarch64), amd64, armv6l, armv7l, x86
• Versions: no v prefix (0.26.1 not v0.26.1)

3. Curl-pipe Bootstrap

GET /{package}@{tag} (with curl/wget User-Agent)

Handler: serve-installer.js:getPosixCurlPipeBootstrap({baseurl, pkg, ver}) or getPwshCurlPipeBootstrap({baseurl, pkg, ver, exename})

Sets env vars WEBI_PKG, WEBI_HOST, WEBI_CHECKSUM in the bootstrap template.

4. Package Metadata

GET /packages/{package}/README.md (or other assets)

Handler: packages.js:get(name) → reads README.md frontmatter via frontmarker.js

Response: { title, tagline, description, bash, windows }

5. Debug

GET /api/debug

Handler: ua-detect.js:request(req) — returns detected OS/arch/libc from UA.

Testing

Automated compatibility test

# Refresh golden data from live site
node _webi/test-api-compat.js --refresh   # (not yet implemented)

# Run comparison
node _webi/test-api-compat.js

Manual smoke tests

# builds-cacher: load packages from cache
node -e "let B = require('./_webi/builds.js'); B.init().then(...);"

# transform-releases: legacy API from cache
node -e "let R = require('./_webi/transform-releases.js'); R.getReleases({pkg:'bat', ...});"

# serve-installer: full end-to-end resolution
node -e "let I = require('./_webi/serve-installer.js'); I.helper({unameAgent:'aarch64/unknown Darwin/24.2.0 libc', ...});"

# classify-one: dev tool reads cache
node _webi/classify-one.js bat

Project Type Detection

builds-cacher.js:getProjectTypeByEntry() classifies packages:

Type	Meaning	Check
alias	Symlink or README has alias: x	symlink or README frontmatter
valid	Has cache file	_cache/YYYY-MM/{name}.json exists
selfhosted	No cache file	cache file missing
hidden	System dirs, _*, .* etc	naming convention
invalid	No README.md	file check

Package Counts

• 101 packages detected as valid (have cache files)
• 58 packages detected as selfhosted (no cache)
• 33 packages detected as alias