amd64[_-]?v2 matched syncthing filenames like 'amd64-v2.0.5' where '-v2' is
the start of the release version, not an arch micro-level. Changed to amd64_?v2
(underscore optional, dash excluded) which correctly matches:
- amd64v2 (no separator, original form)
- amd64_v2 (underscore, pathman form)
But NOT amd64-v2.0.5 (dash + version number, syncthing).
Correct the previous commit's two mistakes (3655ef3):
1. universal2/universal1: translate arch to x86_64 instead of dropping.
The Node classifier maps 'universal' in filename to x86_64, so the cache
must say x86_64 to match. The darwin WATERFALL (aarch64→[aarch64,x86_64])
ensures arm64 users still receive these builds as a fallback.
Previously dropping these caused cmake/hugo macOS to resolve to old versions.
2. solaris/illumos: keep as-is instead of dropping.
The build-classifier (triplet.js) recognizes solaris, illumos, and sunos as
distinct values and matches them correctly. ed5239a was right all along.
Also add minor arch translations:
- x86_64_v2/v3/v4 → x86_64: classifier doesn't recognize micro-arch suffixes
- mips64r6/mips64r6el → mips64: MIPS Release 6 variants
The Node build-classifier re-parses filenames independently and drops any
cache entry where its extraction doesn't match the pre-classified field.
New drops in ExportLegacy (with LegacyDropStats tracking):
- universal2/universal1 arch: classifier maps 'universal' in filename to
x86_64 and rejects entries with arch='universal2'
- solaris/illumos OS: Node never served these; classifier mismatches
are unfixable without changing the filename
- android OS: classifier maps android filenames to linux
ARM arch translations in legacyFieldBackport (filename-based):
- gnueabihf / armhf filename → 'armhf' (Go normalizes to armv6/armv7;
Node classifier preserves the original Debian/Rust naming)
- armel filename → 'armel' (Go normalizes to armv6)
- armv5 filename → 'armel' (Node tiered map: armv5 falls back to armel)
- armv7a filename → 'armv7a' (Go normalizes to armv7)
- armv7l / armv6l: no translation needed (both Go and Node say armv7/armv6)
LIVE_cache confirms: go.json uses 'illumos' and 'solaris' as distinct
values. No package in the live cache uses 'sunos'. The build-classifier
(triplet.js) also keeps all three distinct (illumos, solaris, sunos).
The previous sunos translation matched the old normalize.js path, not
the build-classifier path that the Go rewrite targets.
Covers all cases where canonical data can't be expressed in the legacy
Node.js cache format, with explicit verification:
- Variant builds dropped and counted (Node.js has no variant logic)
- Unknown formats dropped and counted (.apk, .AppImage, .deb, .rpm)
- Empty format passes through (bare binaries, git sources)
- solaris/illumos translated to sunos (Node.js only knows sunos)
- ffmpeg windows .gz overridden to .exe (per-package compat rule)
- ffmpeg translation not applied to other packages
- universal2 passes through as-is (Node.js WATERFALL handles it)
- Mixed: correct counts when drops and translations occur together
The Node classifier maps armv6l filenames → armv6, so the cache should
emit armv6 not arm. The previous backport predates the armv6l→armv6
normalizeGoArch fix and is no longer needed.
Expanding universal2 into aarch64+x86_64 entries creates filename/arch
mismatches: the filename still contains 'universal' so the Node classifier
re-parses it as universal2, then sees arch=aarch64 and flags a mismatch.
Keep universal2 as-is; the Node agent will add it to their WATERFALL.
Classifier fixes:
- Remove Windows arm→arm64 auto-promotion; packages like caddy/fzf/goreleaser
have genuine arm32 Windows builds (windows_armv6) that were wrongly promoted
- Add armel and gnueabihf as ARMv6 aliases (jq, caddy and others use these)
- Add winx64 to Windows OS pattern (MariaDB uses winx64 in filenames)
- Add ppc64el as ppc64le alias (Debian/Ubuntu naming, used by jq)
- Normalize armv6l → armv6 in normalizeGoArch (Go dist had armv6l filenames)
- Fix classifyGPGDist hardcoded "amd64" → buildmeta.ArchAMD64 ("x86_64")
Legacy export fixes:
- Map solaris/illumos → sunos globally (Node.js only knows "sunos")
- Expand universal2 → two entries (aarch64 + x86_64) so Hugo/cmake/gh/syncthing
work on both Apple Silicon and Intel Mac in the legacy resolver
- Remove double-application of legacyFieldBackport (toLegacy no longer calls it)
- Remove LegacyBackport from classifypkg and webicached; canonical values
now flow through storage untouched
- Add legacyFieldBackport() in storage/legacy.go, called only at export time
(go: armv6→arm, ffmpeg windows: .gz/.empty→.exe)
- ExportLegacy now takes pkg name and returns LegacyDropStats (variants + formats dropped)
- fsstore.Commit logs dropped assets so filtering is visible
- Add FormatAPK (.apk) and FormatAppImage (.AppImage) to buildmeta and classify
so those files are properly classified and then correctly dropped from legacy export
rather than passing through as empty-format
Implements storage.Store for PostgreSQL using pgx/v5.
Schema uses double-buffered generations per package — write into the
inactive gen, then atomically swap the active pointer on Commit. Readers
always see a complete consistent snapshot.
Write path: BeginRefresh → Put (staged in-memory) → Commit (CopyFrom + swap)
Read path: Load → reads active gen from webi_packages, fetches assets
Both webid and webicached now accept -pg=<dsn> to use pgstore instead
of fsstore. Schema is applied idempotently on startup.
Also:
- storage.Store interface gains ListPackages(ctx) — fsstore reads the
directory; pgstore queries webi_packages
- webid.loadAll() uses ListPackages instead of filepath.ReadDir
- Fixed .gitignore: /webid (root binary) was incorrectly matching cmd/webid/
Move legacy-specific field translations out of the core classifier into
LegacyBackport(), called by webicached before writing the JSON cache.
Core classifier now outputs canonical values:
- Go dist arm → armv6 (correct per GOARM default)
- ffmpeg Windows .gz → .gz (correct file extension)
LegacyBackport remaps for Node.js compat:
- Go dist armv6 → arm (production keeps raw API value)
- ffmpeg Windows .gz → exe (production releases.js override)
sass armv6→armv7 stays in classifier (Dart Sass genuinely targets ARMv7).
- sass: bare arm → armv7 (Dart Sass targets ARMv7, not v6)
- ffmpeg: Windows .gz → ext exe (gzipped bare executables)
- go: keep bare arm as-is from Go dist API (matches production)
Reduces comparecache diffs from 6 packages to 3 (iterm2 channel edge
cases, postgres legacy ext, terraform alpha detection — all understood).
The golang dist API provides structured os/arch fields. Using these
instead of filename-based classification fixes:
- illumos/solaris kept distinct (not merged to sunos)
- arm arch correctly mapped per GOARCH convention
- buildmeta: add OSIllumos and OSSolaris constants
- pg/releases.conf: add asset_filter=postgres so pg only returns server
assets (which include the client), matching production releases.js
- classifypkg: add "pg" to postgres version normalizer switch case
- comparecache: compare os/arch/libc/ext/channel fields on shared assets,
distinguishing real disagreements (diff-*) from expected fill diffs
where Go classifies at write time but Node.js leaves fields empty
- Windows gnu (MinGW) builds are self-contained: classify as libc='none'
- Pad install.sh content to 8 spaces to match production template indent
- Use replaceMarkerLine for both bash and PS1 installer injection
Rust *-unknown-linux-musl builds are statically linked with zero
runtime libc dependency. Detect this pattern in classifyGitHub and
override libc from 'musl' to 'none'. Hard-musl packages (pwsh, bun,
node) use different filename patterns and keep libc='musl'.
Add PowerShell() function to render .ps1 installers by injecting
$Env: variables and splicing install.ps1 content. Wire it into
the webid server for .ps1 extension requests.
git_url is now a standalone field that can appear alongside any source
type. For githubsource packages, it adds a git clone entry per release
in addition to the tarball and zipball. Updated aliasman, duckdns.sh,
and serviceman configs.
Three distinct fetch/classify strategies:
- github: binary assets only, no source entries
- githubsource: tarball + zipball from GitHub releases API
- gittag: git clone + tag enumeration (existing)
GitHub binary packages (caddy, jq, shellcheck, etc.) no longer get
spurious .git and source tarball entries for old releases that had
no binary uploads. Source-installable packages (aliasman, duckdns.sh,
serviceman) now use github_source in releases.conf.
Tests were using separate source/owner/repo keys but the parser expects
github_repo=owner/repo, gitea_repo=owner/repo, etc. Fixed all test
configs to match. Also answered Issue 4 (darwin-universal) for other agent.
Production has two separate flows:
1. /{pkg} (curl-pipe bootstrap) — minimal script that sets WEBI_PKG,
WEBI_HOST, WEBI_CHECKSUM and downloads+runs webi
2. /api/installers/{pkg}.sh — full installer with resolved release
and embedded install.sh
Previously handleBootstrap served the full installer. Now:
- handleBootstrap: curl-pipe bootstrap (reads curl-pipe-bootstrap.tpl.sh)
- handleInstaller: full installer (/api/installers/{pkg}.sh)
Also:
- Export render.InjectVar for use by bootstrap handler
- Add webi.sh checksum calculation (SHA-1 first 8 chars)
- Add /api/installers/ route to mux and test server
- Added TestV1ResolveJQ to verify jq resolves to binary, not git
- Changed upstream gap detection in resolve_cache_test to t.Skipf
(shellcheck/windows and xz/linux-arm64 don't have upstream builds)
- Updated ANSWERS.md with git assets investigation results
Renders package-install.tpl.sh with WEBI_* variable injection and
install.sh splicing. Bootstrap route at /{package}@{version} detects
UA, resolves best release, and returns rendered installer script.
Serves /api/releases/{pkg}@{version}.json and .tab matching the
Node.js format. Supports query params for os, arch, libc, channel,
formats, lts, limit. Handles selfhosted packages (install.sh only).
Pre-loads all cached packages on startup. Includes /api/debug for
UA detection and /api/health endpoint.
Triplet-based resolution with indexed lookup for fast matching.
Supports channel hierarchy (alpha > beta > rc > stable), LTS filtering,
variant selection, format preferences, and arch fallback via CompatArches.
All 13 unit tests and cache integration tests pass against real data
for 100+ packages.
Source type is now inferred from the primary key:
github_repo = owner/repo (was source=github + owner + repo)
git_url = https://... (was source=gittag + url)
gitea_repo = owner/repo (was source=gitea + owner + repo)
hashicorp_product = name (was source=hashicorp + product)
One-off dist sources (nodedist, zigdist, etc.) keep the explicit
source= key since they're already one-liners.
Parser still accepts the old format via the default fallback branch.
Tagless repos (only HEAD, no real version tags): rewrite HEAD version
to Node.js-compatible format (v2023.10.10-18.42.21) with full UTC
datetime.
Repos with real tags + HEAD: tag HEAD entries with "head" variant so
ExportLegacy filters them out (they shouldn't appear in legacy cache).
Hardcode the old 10.12, 10.13, 11.8, 12.3 releases from EnterpriseDB
that predate the bnnanet/postgresql-releases GitHub repo. Both postgres
and psql now match the live cache exactly.
Early PowerShell releases (pre-6.1) used Windows-version-specific
filenames (win10-win2016, win81-win2012r2) that the legacy cache
can't resolve. Tag them as variants so they're filtered from legacy
export but preserved for future Go resolver use.
The regex captured the beta/preview number but not the keyword itself,
so "3.0.0-preview" collapsed to "3.0.0". Also deduplicate by version
since the downloads page has duplicate links with different URL formats
(e.g. iTerm2-3_5_1beta1.zip and iTerm2-3_5_1_beta1.zip).
Packages with alias_of in releases.conf (e.g. dashd → dashcore,
golang → go) now get symlinked cache files so they resolve to the
same JSON as their target. 13 aliases total.
Added AliasOf as a proper field in installerconf.Conf, LinkAlias
method to fsstore, and alias handling in webicached's Run loop.
GitHub's prerelease boolean is often not set for rc/beta/alpha/dev/pre
releases. Add channelFromVersion() to detect these from the version
string as a fallback. Applied to github, gitea, gittag, and hashicorp
classifiers. Hashicorp's inline checks replaced with the shared helper.
-pre maps to beta (prerelease), -preview stays preview.
The cli- prefix is a watchexec-specific monorepo artifact, not a generic
config concern. Move it to internal/releases/watchexec/versions.go
alongside other per-package normalizers (git, lf).
- postgres/psql: add asset_filter to separate assets from shared repo
(bnnanet/postgresql-releases contains postgres-*, postgresql-*, psql-*)
- watchexec: change tag_prefix to version_prefixes so old plain-tagged
releases (v1.20.6+) aren't filtered out — only strip the cli- prefix
- classify: add .minisig, b3sums, dist-manifest.json to IsMetaAsset
filter to prevent checksum/signature files from leaking into cache
- ffmpeg: add version_prefix = b to strip 'b' from tags (b6.0 → 6.0)
- legacy.go: add .gz to legacyFormats for bare gzipped binaries
- iterm2: broaden regex to handle preview/beta variants, skip empty
versions
Match count: 75/106
Node.js pads Go versions like "1.10" to "1.10.0". Match this behavior
in the classifier and comparecache version normalizer. Also filter
-arm6. malformed arch and .src. source tarballs from comparison noise.
Match count: 73/106
- Add .app.zip to legacyFormats so macOS fish builds export correctly
- Exclude bundledpcre, fish-static, OpenBeta from fish/releases.conf
- Add fish Linux binaries to comparecache noise (Go improvement)
Match count: 72/106
Git for Windows uses tags like v2.53.0.windows.1. Node.js strips
".windows.1" and replaces ".windows.N" (N>1) with ".N".
Add NormalizeVersions to the git package and wire it into the classify
pipeline. Also add version normalization to comparecache so the
comparison uses canonical versions for both caches.
Remaining git diffs: data freshness (.windows.2 releases Go hasn't
fetched) and RC versions in Go that live doesn't have.
- Hugo: exclude Linux-64bit legacy filename alias
- Hugo-extended: exclude Linux-64bit legacy filename alias
- Gitea: exclude -src- and -docs- tarballs
- Pathman: exclude armv8 legacy alias
- UUID v7: exclude exotic architectures (thumb, armeb, loong, gnux32, risc)
- comparecache: filter bare executables and docs tarballs as noise,
apply noise filter to both live and Go sides
- legacy.go: add .tar.bz2 to legacyFormats
Match count: 69/106 (up from 58)
