Compare commits

...

4 Commits

Author SHA1 Message Date
AJ ONeal
f5a8a0cd5a feat(tools): add development and testing utilities (classify, comparecache, e2etest, fetchraw, inspect, uaparse) 2026-05-14 17:49:29 -06:00
AJ ONeal
1bc9e40bf6 feat(ffmpeg): custom ffmpegdist classifier for non-standard release names
eugeneware/ffmpeg-static uses non-standard filenames (x64, ia32, win32,
arm) and ships both bare binaries and .gz variants. The generic classifier
mishandles the arch mapping and the installer has no bare .gz handler.

Add ffmpegdist custom classifier that:
- Maps non-standard OS/arch names to canonical values
- Filters to bare binaries only (skips .gz, ffprobe, LICENSE, README)
- Strips 'b' version prefix from tags

Also fix installerconf to allow explicit 'source' to override the
inferred source when both are present (e.g. source=ffmpegdist +
github_releases for fetching).

Closes #947
2026-05-14 17:48:01 -06:00
AJ ONeal
7d9fc3387c docs: add deploy scripts, skills, and pattern guides
Deploy scripts for webicached and webid (build, upload, restart).
AGENTS.md with releases.conf reference and variant tagging docs.
Installer archive pattern guide and version oddities reference.
2026-05-14 17:48:01 -06:00
AJ ONeal
3cfd10f197 feat: add Go release cache daemon (webicached) and HTTP API server (webid)
Rewrites the Node.js release classification pipeline in Go. webicached
fetches upstream releases, classifies assets, and writes legacy-format
JSON caches. webid serves the HTTP API (releases, resolve, bootstrap
scripts). Git-clone packages emit git_tag and git_commit_hash from
real repo clones — no fabricated refs.
2026-05-14 17:47:23 -06:00
92 changed files with 18140 additions and 0 deletions

16
.gitignore vendored
View File

@@ -3,6 +3,16 @@ install-*.sh
install-*.bat
install-*.ps1
# Go build outputs (from go run/build in repo root)
/classify
/e2etest
/fetchraw
/inspect
/uaparse
/webicached
/zigtest
/distributables.csv
# local config
.env.*
*.env
@@ -18,7 +28,13 @@ node_modules/
*.bak
*.bak.*
# agent session files
agents/
# other
.DS_Store
desktop.ini
.directory
LIVE_cache
/webid
bin/

120
AGENTS.md
View File

@@ -370,3 +370,123 @@ Commit messages: `feat(<pkg>): add installer`, `fix(<pkg>): update install.sh`,
`arch`, or `ext` explicitly in `releases.js`.
- **Goreleaser archives**: Typically contain a bare binary at the archive root
(not nested in a directory). Use `mv ./cmd "$pkg_src_cmd"`.
---
## Go Cache Daemon
The Go pipeline (`cmd/webicached`) replaces the Node.js release-fetching code.
It reads `releases.conf` files, fetches upstream release metadata, classifies
build assets, and writes to `~/.cache/webi/legacy/` in the format the Node.js server expects.
### Canonical Vocabulary
The classifier MUST use exactly these strings. They match the production API.
**OS**: `macos` (NOT `darwin`), `linux`, `windows`, `freebsd`, `openbsd`,
`netbsd`, `dragonfly`, `aix`, `illumos`, `plan9`, `solaris`, `posix_2017`
**Arch** — exact equivalences:
- `amd64` (NOT `x86_64`), `x86` (NOT `i386`/`i686`/`386`)
- `arm64` (NOT `aarch64`)
- `armv7l` (NOT `armv7`), `armv6l` (NOT `armv6`)
- `mipsle` (NOT `mipsel`), `mips64le` (NOT `mips64el`)
**Arch** — compatibility downcasts:
- `armhf` → `armv7l`, `armv7a` → `armv7l`, `armel` → `arm`
**Arch** — other: `arm`, `ppc64le`, `ppc64`, `loong64`, `riscv64`, `s390x`,
`mips`, `mips64`
**Libc**: `none` (never empty), `gnu`, `musl`, `msvc`
**Ext**: `tar.gz`, `tar.xz`, `zip`, `exe`, `7z`, `pkg`, `msi`
(no leading dot; `exe` for bare binaries)
### releases.conf
Each package directory contains a `releases.conf` that tells the daemon where
to fetch releases. Format is `key = value`, one per line. `#` comments and
blank lines are ignored.
#### Source types (mutually exclusive — pick one)
```ini
# GitHub binary releases (most common)
github_releases = sharkdp/bat
# GitHub source tarballs (with optional git fallback)
github_sources = bnnanet/serviceman
git_url = https://github.com/bnnanet/serviceman.git
# Git tag enumeration (vim plugins, shell scripts — git_url alone)
git_url = https://github.com/tpope/vim-commentary.git
# Gitea (full URL required, or short form + base_url)
gitea_releases = https://git.rootprojects.org/root/pathman
# GitLab (defaults to gitlab.com)
gitlab_releases = owner/repo
# HashiCorp releases API
hashicorp_product = terraform
# Custom source (servicemandist, nodedist, zigdist, etc.)
source = nodedist
url = https://nodejs.org/download/release
```
#### Filtering, versioning, and platform
```ini
tag_prefix = bun- # monorepo: strip prefix from version
version_prefixes = jq- # strip from version string (space-separated)
asset_filter = MinGit # filename must contain this substring
exclude = busybox -src- -docs- # skip assets containing these (space-separated)
os = posix_2017 # restrict ALL versions to this OS (blanket)
alias_of = rg # mirrors another package's releases
```
#### Design rules
- `os` is a blanket tag on ALL versions. Only use for packages that are always
POSIX-only. For version-dependent OS tagging, use a custom `TagVariants` in
`internal/releases/{pkg}/variants.go`.
- `git_url` can be primary (gittag source when it's the only key) or secondary
fallback alongside `github_sources`/`gitea_sources`.
- Full URL forms accepted for github/gitea/gitlab (e.g.
`github_releases = https://github.com/sharkdp/bat`).
### Testing
Test tools: `cmd/e2etest` (pipeline comparison), `cmd/comparecache` (cache diff),
`cmd/inspect` (single-package debug). Run each with `--help` for usage.
### Classifier vs Per-Package Tagger
The general classifier (`internal/classify/`) handles patterns common across
many projects. It MUST NOT contain one-off logic for a single package.
Per-package taggers (`internal/releases/{pkg}/variants.go`) handle
project-specific knowledge. Read the existing taggers for conventions.
MUST: Derive arch/OS from concrete evidence — not blanket defaults.
MUST: New general classifier patterns must apply to 2-3+ packages.
### Deploying
```sh
./scripts/deploy-webicached.sh beta.webi.sh
./scripts/deploy-webicached.sh next.webi.sh
```
First-time setup on a new host uses `serviceman`:
```sh
serviceman add --name webicached \
--workdir ~/srv/webid/installers/ -- \
~/bin/webicached \
--envfile ~/srv/webid/.env.secret \
--conf ~/srv/webid/installers/ \
--raw ~/.cache/webi/raw
```

1293
cmd/classify/main.go Normal file

File diff suppressed because it is too large Load Diff

914
cmd/comparecache/main.go Normal file
View File

@@ -0,0 +1,914 @@
// Command comparecache compares Go-generated cache output against the
// Node.js LIVE_cache. It identifies categorical differences in asset
// selection — which filenames appear in one cache but not the other.
//
// The comparison is done at the filename level (not OS/arch/ext fields)
// because the Node.js cache leaves those empty (normalize.js fills them
// at serve time), while the Go pipeline classifies at write time.
//
// Usage:
//
// go run ./cmd/comparecache -live ./LIVE_cache -go ./_cache
// go run ./cmd/comparecache -live ./LIVE_cache -go ./_cache bat jq
// go run ./cmd/comparecache -live ./LIVE_cache -go ./_cache -summary
package main
import (
"encoding/json"
"flag"
"fmt"
"log"
"math/rand/v2"
"os"
"path/filepath"
"slices"
"sort"
"strings"
"time"
"github.com/webinstall/webi-installers/internal/classify"
"github.com/webinstall/webi-installers/internal/lexver"
)
type cacheEntry struct {
Releases []cacheRelease `json:"releases"`
}
type cacheRelease struct {
Name string `json:"name"`
Filename string `json:"_filename"` // Node.js uses _filename for some sources
Version string `json:"version"`
Download string `json:"download"`
Channel string `json:"channel"`
OS string `json:"os"`
Arch string `json:"arch"`
Libc string `json:"libc"`
Ext string `json:"ext"`
}
// fieldDiff records a field-level difference for an asset that exists
// in both caches (same filename) but has different classification.
type fieldDiff struct {
Filename string
Field string // "os", "arch", "libc", "ext", "channel"
Live string
Go string
BothSet bool // true when both live and go have non-empty values
}
type packageDiff struct {
Name string
LiveCount int
GoCount int
OnlyInLive []string // filenames only in Node.js cache
OnlyInGo []string // filenames only in Go cache
FieldDiffs []fieldDiff // classification differences on shared assets
VersionsLive []string // unique versions in live
VersionsGo []string // unique versions in go
GoMissing bool // true if Go didn't produce output for this package
LiveMissing bool // true if no live cache for this package
Categories []string // categorical difference labels
}
func main() {
liveDir := flag.String("live", "./LIVE_cache", "path to Node.js LIVE_cache directory")
goDir := flag.String("go", "./_cache", "path to Go cache directory")
summary := flag.Bool("summary", false, "only print summary, not per-package details")
diffsOnly := flag.Bool("diffs", false, "only show packages with asset differences (skip matches)")
latest := flag.Bool("latest", false, "only compare latest version in each cache")
windowed := flag.Bool("windowed", false, "limit Go versions to the Node.js version range (2nd to 2nd-to-last)")
sample := flag.Int("sample", 0, "for each package diff, show N randomly sampled assets (implies -windowed -diffs)")
flag.Parse()
filterPkgs := flag.Args()
// -sample implies -windowed and -diffs so we focus on real classification
// differences, not version-depth noise.
if *sample > 0 {
*windowed = true
*diffsOnly = true
}
totalStart := time.Now()
// Find the most recent month directory in each cache.
liveMonth := findLatestMonth(*liveDir)
goMonth := findLatestMonth(*goDir)
if liveMonth == "" {
log.Fatalf("no month directories found in %s", *liveDir)
}
livePath := filepath.Join(*liveDir, liveMonth)
goPath := ""
if goMonth != "" {
goPath = filepath.Join(*goDir, goMonth)
}
// Discover all packages across both caches.
discoverStart := time.Now()
allPkgs := discoverPackages(livePath, goPath)
if len(filterPkgs) > 0 {
nameSet := make(map[string]bool, len(filterPkgs))
for _, n := range filterPkgs {
nameSet[n] = true
}
var filtered []string
for _, p := range allPkgs {
if nameSet[p] {
filtered = append(filtered, p)
}
}
allPkgs = filtered
}
log.Printf("discovered %d packages in %s", len(allPkgs), time.Since(discoverStart))
compareStart := time.Now()
var diffs []packageDiff
for _, pkg := range allPkgs {
d := compare(livePath, goPath, pkg, *latest, *windowed)
categorize(&d)
diffs = append(diffs, d)
}
log.Printf("compared %d packages in %s", len(diffs), time.Since(compareStart))
if *summary {
printSummary(diffs)
} else {
printDetails(diffs, *diffsOnly, *sample)
}
log.Printf("total: %s", time.Since(totalStart))
}
func findLatestMonth(dir string) string {
entries, err := os.ReadDir(dir)
if err != nil {
return ""
}
var months []string
for _, e := range entries {
if e.IsDir() && len(e.Name()) == 7 && e.Name()[4] == '-' {
months = append(months, e.Name())
}
}
if len(months) == 0 {
return ""
}
sort.Strings(months)
return months[len(months)-1]
}
func discoverPackages(livePath, goPath string) []string {
seen := make(map[string]bool)
for _, dir := range []string{livePath, goPath} {
if dir == "" {
continue
}
entries, err := os.ReadDir(dir)
if err != nil {
continue
}
for _, e := range entries {
name := e.Name()
if strings.HasSuffix(name, ".json") && !strings.HasSuffix(name, ".updated.txt") {
pkg := strings.TrimSuffix(name, ".json")
seen[pkg] = true
}
}
}
var pkgs []string
for p := range seen {
pkgs = append(pkgs, p)
}
sort.Strings(pkgs)
return pkgs
}
func loadCache(dir, pkg string) *cacheEntry {
if dir == "" {
return nil
}
data, err := os.ReadFile(filepath.Join(dir, pkg+".json"))
if err != nil {
return nil
}
var entry cacheEntry
if err := json.Unmarshal(data, &entry); err != nil {
return nil
}
return &entry
}
// effectiveName returns the best available filename for a release entry.
// Node.js sometimes uses _filename (a path) instead of name.
func effectiveName(name, filename, download string) string {
if name != "" {
return name
}
if filename != "" {
// _filename may be a path like "stable/macos/flutter_macos_3.41.4.zip"
if i := strings.LastIndex(filename, "/"); i >= 0 {
return filename[i+1:]
}
return filename
}
// Last resort: basename of download URL.
if download != "" {
if i := strings.LastIndex(download, "/"); i >= 0 {
return download[i+1:]
}
}
return ""
}
// versionWindow returns the 2nd and 2nd-to-last versions from a sorted
// version list. This trims the edges where Node.js may have a newer fetch
// or Go may have deeper history, focusing on the overlapping middle.
func versionWindow(versions []string) (low, high string) {
if len(versions) <= 2 {
// Too few versions to window — use all.
if len(versions) > 0 {
return versions[0], versions[len(versions)-1]
}
return "", ""
}
// 2nd version (skip oldest) and 2nd-to-last (skip newest).
return versions[1], versions[len(versions)-2]
}
// filterVersionRange returns only the versions in sorted order that fall
// within [low, high] inclusive (by lexver comparison).
func filterVersionRange(vf map[string]map[string]bool, versions []string, low, high string) (map[string]bool, []string) {
lowV := lexver.Parse(low)
highV := lexver.Parse(high)
files := make(map[string]bool)
var kept []string
for _, v := range versions {
pv := lexver.Parse(v)
if lexver.Compare(pv, lowV) >= 0 && lexver.Compare(pv, highV) <= 0 {
kept = append(kept, v)
for f := range vf[v] {
files[f] = true
}
}
}
return files, kept
}
func compare(livePath, goPath, pkg string, latestOnly, windowed bool) packageDiff {
live := loadCache(livePath, pkg)
goCache := loadCache(goPath, pkg)
d := packageDiff{Name: pkg}
if live == nil {
d.LiveMissing = true
}
if goCache == nil {
d.GoMissing = true
}
if d.LiveMissing && d.GoMissing {
return d
}
normVersion := normalizeVersionFunc(pkg)
// Collect filenames by version. If filter is non-nil, skip filenames it rejects.
extractVersionFiles := func(ce *cacheEntry, filter func(string) bool) (map[string]map[string]bool, []string) {
vf := make(map[string]map[string]bool)
for _, r := range ce.Releases {
name := effectiveName(r.Name, r.Filename, r.Download)
if filter != nil && !filter(name) {
continue
}
ver := normVersion(r.Version)
if vf[ver] == nil {
vf[ver] = make(map[string]bool)
}
vf[ver][name] = true
}
var versions []string
for v := range vf {
versions = append(versions, v)
}
slices.SortFunc(versions, func(a, b string) int {
return lexver.Compare(lexver.Parse(a), lexver.Parse(b))
})
return vf, versions
}
notNoise := func(name string) bool { return !isLiveNoise(name) }
var liveFiles, goFiles map[string]bool
// Parse live cache.
var liveVF map[string]map[string]bool
var liveVersions []string
if live != nil {
liveVF, liveVersions = extractVersionFiles(live, notNoise)
d.VersionsLive = liveVersions
d.LiveCount = len(live.Releases)
}
// Parse Go cache.
var goVF map[string]map[string]bool
var goVersions []string
if goCache != nil {
goVF, goVersions = extractVersionFiles(goCache, notNoise)
d.VersionsGo = goVersions
d.GoCount = len(goCache.Releases)
}
// Determine which files to compare based on mode.
if latestOnly {
// Compare only the latest version from each cache.
if live != nil && len(liveVersions) > 0 {
liveFiles = liveVF[liveVersions[len(liveVersions)-1]]
}
if goCache != nil && len(goVersions) > 0 {
goFiles = goVF[goVersions[len(goVersions)-1]]
}
} else if windowed && live != nil && len(liveVersions) > 0 {
// Use the Node.js version range (2nd to 2nd-to-last) to establish
// the window. Include ALL Node.js versions in the window (so missing
// Go versions are visible), but exclude Go-only versions (those are
// just deeper history, not real gaps).
low, high := versionWindow(liveVersions)
lowV := lexver.Parse(low)
highV := lexver.Parse(high)
// Collect all live files in the window.
liveFiles = make(map[string]bool)
liveInWindow := make(map[string]bool)
for _, v := range liveVersions {
pv := lexver.Parse(v)
if lexver.Compare(pv, lowV) >= 0 && lexver.Compare(pv, highV) <= 0 {
liveInWindow[v] = true
for f := range liveVF[v] {
liveFiles[f] = true
}
}
}
// For Go, only include versions that Node.js also has in the window.
// Go-only versions are hidden (deeper history, not gaps).
goFiles = make(map[string]bool)
for _, v := range goVersions {
if !liveInWindow[v] {
continue
}
for f := range goVF[v] {
goFiles[f] = true
}
}
} else {
// Compare all versions — use pre-filtered version maps.
if live != nil {
liveFiles = make(map[string]bool)
for _, files := range liveVF {
for f := range files {
liveFiles[f] = true
}
}
}
if goCache != nil {
goFiles = make(map[string]bool)
for _, files := range goVF {
for f := range files {
goFiles[f] = true
}
}
}
}
if liveFiles == nil {
liveFiles = make(map[string]bool)
}
if goFiles == nil {
goFiles = make(map[string]bool)
}
for f := range liveFiles {
if !goFiles[f] {
d.OnlyInLive = append(d.OnlyInLive, f)
}
}
for f := range goFiles {
if !liveFiles[f] {
d.OnlyInGo = append(d.OnlyInGo, f)
}
}
sort.Strings(d.OnlyInLive)
sort.Strings(d.OnlyInGo)
// Field-level comparison on assets that exist in both caches.
// Build version+filename → fields maps from each cache.
if live != nil && goCache != nil {
type assetKey struct {
version string
filename string
}
liveByKey := make(map[assetKey]cacheRelease)
for _, r := range live.Releases {
name := effectiveName(r.Name, r.Filename, r.Download)
ver := normVersion(r.Version)
liveByKey[assetKey{ver, name}] = r
}
for _, r := range goCache.Releases {
name := effectiveName(r.Name, r.Filename, r.Download)
ver := normVersion(r.Version)
lr, ok := liveByKey[assetKey{ver, name}]
if !ok {
continue
}
// Compare classification fields.
// Use equivalence checks for os/arch/ext so naming
// convention differences don't mask real classification bugs.
for _, cmp := range []struct {
field string
live string
go_ string
equiv bool
}{
{"os", lr.OS, r.OS, equivOS(lr.OS, r.OS)},
{"arch", lr.Arch, r.Arch, equivArch(lr.Arch, r.Arch)},
{"libc", lr.Libc, r.Libc, lr.Libc == r.Libc},
{"ext", lr.Ext, r.Ext, equivExt(lr.Ext, r.Ext)},
{"channel", lr.Channel, r.Channel, lr.Channel == r.Channel},
} {
if cmp.equiv {
continue
}
d.FieldDiffs = append(d.FieldDiffs, fieldDiff{
Filename: name,
Field: cmp.field,
Live: cmp.live,
Go: cmp.go_,
BothSet: cmp.live != "" && cmp.go_ != "",
})
}
}
sort.Slice(d.FieldDiffs, func(i, j int) bool {
if d.FieldDiffs[i].Field != d.FieldDiffs[j].Field {
return d.FieldDiffs[i].Field < d.FieldDiffs[j].Field
}
return d.FieldDiffs[i].Filename < d.FieldDiffs[j].Filename
})
}
return d
}
// equivOS returns true if two OS values are equivalent across naming conventions.
func equivOS(a, b string) bool {
return a == b || canonicalOS(a) == canonicalOS(b)
}
func canonicalOS(s string) string {
switch strings.ToLower(s) {
case "darwin", "macos", "mac", "osx":
return "darwin"
case "win", "windows":
return "windows"
default:
return strings.ToLower(s)
}
}
// equivArch returns true if two arch values are equivalent.
func equivArch(a, b string) bool {
return a == b || canonicalArch(a) == canonicalArch(b)
}
func canonicalArch(s string) string {
switch strings.ToLower(s) {
case "x86_64", "amd64", "x64":
return "x86_64"
case "aarch64", "arm64":
return "aarch64"
case "armv7", "armv7l":
return "armv7"
case "armv6", "armv6l":
return "armv6"
case "x86", "i386", "i686", "386":
return "x86"
default:
return strings.ToLower(s)
}
}
// equivExt returns true if two extension values are equivalent.
func equivExt(a, b string) bool {
// Normalize: strip leading dot, handle common aliases.
return a == b || canonicalExt(a) == canonicalExt(b)
}
func canonicalExt(s string) string {
s = strings.TrimPrefix(s, ".")
switch s {
case "tgz":
return "tar.gz"
default:
return s
}
}
func categorize(d *packageDiff) {
if d.GoMissing {
d.Categories = append(d.Categories, "go-missing")
return
}
if d.LiveMissing {
d.Categories = append(d.Categories, "live-missing")
return
}
if len(d.OnlyInLive) == 0 && len(d.OnlyInGo) == 0 && len(d.FieldDiffs) == 0 {
d.Categories = append(d.Categories, "match")
return
}
if len(d.OnlyInLive) == 0 && len(d.OnlyInGo) == 0 && len(d.FieldDiffs) > 0 {
d.Categories = append(d.Categories, "fields-only")
}
// Check if differences are only version depth (Go has more history).
liveVersionSet := make(map[string]bool, len(d.VersionsLive))
for _, v := range d.VersionsLive {
liveVersionSet[v] = true
}
goVersionSet := make(map[string]bool, len(d.VersionsGo))
for _, v := range d.VersionsGo {
goVersionSet[v] = true
}
goExtraVersions := 0
for _, v := range d.VersionsGo {
if !liveVersionSet[v] {
goExtraVersions++
}
}
liveExtraVersions := 0
for _, v := range d.VersionsLive {
if !goVersionSet[v] {
liveExtraVersions++
}
}
if goExtraVersions > 0 {
d.Categories = append(d.Categories, fmt.Sprintf("go-extra-versions(%d)", goExtraVersions))
}
if liveExtraVersions > 0 {
d.Categories = append(d.Categories, fmt.Sprintf("live-extra-versions(%d)", liveExtraVersions))
}
// Check for meta-asset filtering differences.
metaOnlyInLive := 0
nonMetaOnlyInLive := 0
for _, f := range d.OnlyInLive {
if classify.IsMetaAsset(f) {
metaOnlyInLive++
} else {
nonMetaOnlyInLive++
}
}
metaOnlyInGo := 0
nonMetaOnlyInGo := 0
for _, f := range d.OnlyInGo {
if classify.IsMetaAsset(f) {
metaOnlyInGo++
} else {
nonMetaOnlyInGo++
}
}
if metaOnlyInLive > 0 {
d.Categories = append(d.Categories, fmt.Sprintf("live-has-meta(%d)", metaOnlyInLive))
}
if metaOnlyInGo > 0 {
d.Categories = append(d.Categories, fmt.Sprintf("go-has-meta(%d)", metaOnlyInGo))
}
// Check for source tarball differences.
srcOnlyInGo := 0
for _, f := range d.OnlyInGo {
if strings.HasSuffix(f, ".tar.gz") || strings.HasSuffix(f, ".zip") {
if strings.HasPrefix(f, "v") || strings.HasPrefix(f, "refs/") {
srcOnlyInGo++
}
}
}
if srcOnlyInGo > 0 {
d.Categories = append(d.Categories, fmt.Sprintf("go-has-source-tarballs(%d)", srcOnlyInGo))
}
if nonMetaOnlyInLive > 0 {
d.Categories = append(d.Categories, fmt.Sprintf("live-extra-assets(%d)", nonMetaOnlyInLive))
}
if nonMetaOnlyInGo > 0 {
d.Categories = append(d.Categories, fmt.Sprintf("go-extra-assets(%d)", nonMetaOnlyInGo))
}
// Count field diffs by field name, separating real disagreements
// from expected "live empty, Go classified" differences.
type fieldCount struct {
bothSet int // both caches have a value but they disagree
oneEmpty int // one side is empty (typically live — normalize.js fills at serve time)
}
fieldCounts := make(map[string]fieldCount)
for _, fd := range d.FieldDiffs {
fc := fieldCounts[fd.Field]
if fd.BothSet {
fc.bothSet++
} else {
fc.oneEmpty++
}
fieldCounts[fd.Field] = fc
}
for _, field := range []string{"os", "arch", "libc", "ext", "channel"} {
fc := fieldCounts[field]
if fc.bothSet > 0 {
d.Categories = append(d.Categories, fmt.Sprintf("diff-%s(%d)", field, fc.bothSet))
}
if fc.oneEmpty > 0 {
d.Categories = append(d.Categories, fmt.Sprintf("fill-%s(%d)", field, fc.oneEmpty))
}
}
}
// isLiveNoise returns true for filenames that the Node.js cache keeps
// but Go intentionally filters out. Pre-filtering these from the live
// side prevents them from appearing as live-extra-assets noise.
//
// This includes everything classify.IsMetaAsset catches plus formats
// that Go's legacy export strips (.deb, .rpm, etc.).
func isLiveNoise(name string) bool {
if classify.IsMetaAsset(name) {
return true
}
lower := strings.ToLower(name)
// Formats Go filters from legacy export but Node.js keeps.
for _, suffix := range []string{
".deb", ".rpm", ".gpg",
} {
if strings.HasSuffix(lower, suffix) {
return true
}
}
// Source tarballs (e.g. gitea-src-1.25.4.tar.gz, caddy_2.10.0_src.tar.gz, go1.26.1.src.tar.gz).
if strings.Contains(lower, "-src-") || strings.Contains(lower, "_src.") || strings.Contains(lower, ".src.") || strings.HasPrefix(lower, "src-") {
return true
}
// Docs tarballs (e.g. gitea-docs-1.22.3.tar.gz).
if strings.Contains(lower, "-docs-") {
return true
}
// Bare executables without any extension — typically legacy shell scripts
// uploaded alongside proper archives (e.g. kubectx, kubens).
if !strings.Contains(name, ".") {
return true
}
// GPU accelerator / hardware variants that Go tags as variants
// but Node.js keeps with special arch names.
for _, v := range []string{"-rocm", "-jetpack"} {
if strings.Contains(lower, v) {
return true
}
}
// Linux binaries for packages where Node.js only kept macOS .app.zip.
// Go correctly includes these as installable on Linux.
if strings.HasPrefix(lower, "fish-") && strings.Contains(lower, "-linux-") {
return true
}
return false
}
// normalizeVersionFunc returns a version normalizer for a given package.
// Most packages return the identity function. Some (like git) need
// version string normalization to match across Go and Node.js caches.
func normalizeVersionFunc(pkg string) func(string) string {
switch pkg {
case "git":
return func(v string) string {
// Git for Windows: v2.53.0.windows.1 → v2.53.0
// v2.53.0.windows.2 → v2.53.0.2
idx := strings.Index(v, ".windows.")
if idx < 0 {
return v
}
suffix := v[idx+len(".windows."):]
base := v[:idx]
if suffix == "1" {
return base
}
return base + "." + suffix
}
case "lf":
return func(v string) string {
// lf: r21 → 0.21.0
if strings.HasPrefix(v, "r") {
return "0." + v[1:] + ".0"
}
return v
}
case "bun":
return func(v string) string {
// bun: bun-v1.3.9 → v1.3.9
return strings.TrimPrefix(v, "bun-")
}
case "watchexec":
return func(v string) string {
// watchexec monorepo: cli-v1.20.5 → v1.20.5
return strings.TrimPrefix(v, "cli-")
}
case "go":
return func(v string) string {
// Go: go1.10 → 1.10.0 (pad to 3 parts)
v = strings.TrimPrefix(v, "go")
parts := strings.SplitN(v, ".", 3)
for len(parts) < 3 {
parts = append(parts, "0")
}
return strings.Join(parts, ".")
}
default:
return func(v string) string { return v }
}
}
func printSummary(diffs []packageDiff) {
// Count by category.
categoryCounts := make(map[string]int)
for _, d := range diffs {
for _, c := range d.Categories {
// Strip the count suffix for grouping.
base := c
if idx := strings.Index(c, "("); idx != -1 {
base = c[:idx]
}
categoryCounts[base]++
}
}
fmt.Println("=== COMPARISON SUMMARY ===")
fmt.Printf("Total packages: %d\n\n", len(diffs))
var cats []string
for c := range categoryCounts {
cats = append(cats, c)
}
sort.Strings(cats)
for _, c := range cats {
fmt.Printf(" %-30s %d\n", c, categoryCounts[c])
}
fmt.Println("\n=== PER-PACKAGE CATEGORIES ===")
for _, d := range diffs {
fmt.Printf("%-25s %s\n", d.Name, strings.Join(d.Categories, ", "))
}
}
func printDetails(diffs []packageDiff, diffsOnly bool, sampleN int) {
for _, d := range diffs {
if diffsOnly && len(d.OnlyInLive) == 0 && len(d.OnlyInGo) == 0 && len(d.FieldDiffs) == 0 {
continue
}
fmt.Printf("=== %s ===\n", d.Name)
fmt.Printf(" Categories: %s\n", strings.Join(d.Categories, ", "))
fmt.Printf(" Live: %d assets, %d versions | Go: %d assets, %d versions\n",
d.LiveCount, len(d.VersionsLive), d.GoCount, len(d.VersionsGo))
printAssetList("Only in LIVE", d.OnlyInLive, sampleN)
printAssetList("Only in Go", d.OnlyInGo, sampleN)
printFieldDiffs(d.FieldDiffs, sampleN)
fmt.Println()
}
}
// printFieldDiffs shows classification differences on shared assets.
// Shows "real" diffs (both sides non-empty) first, then "fill" diffs
// (one side empty) as a summary count only.
func printFieldDiffs(diffs []fieldDiff, sampleN int) {
if len(diffs) == 0 {
return
}
// Separate real disagreements from fill diffs.
var real, fill []fieldDiff
for _, fd := range diffs {
if fd.BothSet {
real = append(real, fd)
} else {
fill = append(fill, fd)
}
}
// Show real disagreements in detail.
if len(real) > 0 {
byField := make(map[string][]fieldDiff)
for _, fd := range real {
byField[fd.Field] = append(byField[fd.Field], fd)
}
for _, field := range []string{"os", "arch", "libc", "ext", "channel"} {
fds := byField[field]
if len(fds) == 0 {
continue
}
fmt.Printf(" DISAGREE %s (%d):\n", field, len(fds))
printFieldDiffItems(fds, sampleN)
}
}
// Summarize fill diffs (live empty, Go classified) as counts.
if len(fill) > 0 {
byField := make(map[string]int)
for _, fd := range fill {
byField[fd.Field]++
}
var parts []string
for _, field := range []string{"os", "arch", "libc", "ext", "channel"} {
if n := byField[field]; n > 0 {
parts = append(parts, fmt.Sprintf("%s(%d)", field, n))
}
}
if len(parts) > 0 {
fmt.Printf(" Go fills empty: %s\n", strings.Join(parts, ", "))
}
}
}
func printFieldDiffItems(fds []fieldDiff, sampleN int) {
items := fds
if sampleN > 0 && len(items) > sampleN {
sampled := make([]fieldDiff, len(items))
copy(sampled, items)
rand.Shuffle(len(sampled), func(i, j int) {
sampled[i], sampled[j] = sampled[j], sampled[i]
})
items = sampled[:sampleN]
sort.Slice(items, func(i, j int) bool {
return items[i].Filename < items[j].Filename
})
}
limit := 20
for i, fd := range items {
if sampleN == 0 && i >= limit {
fmt.Printf(" ... and %d more\n", len(fds)-limit)
break
}
fmt.Printf(" - %s: live=%q go=%q\n", fd.Filename, fd.Live, fd.Go)
}
if sampleN > 0 && len(fds) > sampleN {
fmt.Printf(" ... sampled %d of %d\n", sampleN, len(fds))
}
}
// printAssetList prints a list of asset filenames, optionally sampling N at
// random. When sampleN > 0 and the list is longer, it picks N random items
// so you can spot classification bugs across the full range instead of only
// seeing the first alphabetical entries.
func printAssetList(label string, items []string, sampleN int) {
if len(items) == 0 {
return
}
fmt.Printf(" %s (%d):\n", label, len(items))
if sampleN > 0 && len(items) > sampleN {
// Shuffle a copy, take first N, then sort for readable output.
sampled := make([]string, len(items))
copy(sampled, items)
rand.Shuffle(len(sampled), func(i, j int) {
sampled[i], sampled[j] = sampled[j], sampled[i]
})
picked := sampled[:sampleN]
sort.Strings(picked)
for _, f := range picked {
fmt.Printf(" - %s\n", f)
}
fmt.Printf(" ... sampled %d of %d (run again for different sample)\n", sampleN, len(items))
return
}
limit := 20
for i, f := range items {
if i >= limit {
fmt.Printf(" ... and %d more\n", len(items)-limit)
break
}
fmt.Printf(" - %s\n", f)
}
}

846
cmd/e2etest/main.go Normal file
View File

@@ -0,0 +1,846 @@
// Command e2etest runs the full release pipeline for selected packages
// and compares results against the live webi.sh API.
//
// It fetches from upstream, classifies assets, resolves the best match
// for a set of test queries, then fetches the same queries from the live
// API and reports any differences.
//
// Usage:
//
// go run ./cmd/e2etest
// go run ./cmd/e2etest -packages goreleaser,ollama,node
// go run ./cmd/e2etest -cache ./_cache/raw # reuse existing cache
package main
import (
"bufio"
"context"
"encoding/json"
"flag"
"fmt"
"io"
"log"
"net/http"
"os"
"path/filepath"
"strings"
"time"
"github.com/webinstall/webi-installers/internal/buildmeta"
"github.com/webinstall/webi-installers/internal/installerconf"
"github.com/webinstall/webi-installers/internal/lexver"
"github.com/webinstall/webi-installers/internal/rawcache"
"github.com/webinstall/webi-installers/internal/releases/github"
"github.com/webinstall/webi-installers/internal/releases/githubish"
"github.com/webinstall/webi-installers/internal/releases/nodedist"
"github.com/webinstall/webi-installers/internal/resolve"
)
// testCase is one query to resolve and compare against the live API.
type testCase struct {
Name string
Package string
OS buildmeta.OS
Arch buildmeta.Arch
Libc buildmeta.Libc
Formats []string
UA string // User-Agent for live API query
}
// liveResult holds parsed fields from the live webi API response.
type liveResult struct {
Version string
OS string
Arch string
Libc string
Ext string
PkgURL string
PkgFile string
Channel string
Stable string
Latest string
Oses string
Arches string
Libcs string
Formats string
}
// UA format from webi.sh bootstrap: "curl {uname -s}/{uname -r} {uname -m}/unknown {libc}"
// libc is "gnu", "musl", or "libc" (for darwin/other)
var cases = []testCase{
{
Name: "goreleaser/linux/x86_64", Package: "goreleaser",
OS: buildmeta.OSLinux, Arch: buildmeta.ArchAMD64, Libc: buildmeta.LibcGNU,
Formats: []string{".tar.gz", ".tar.xz", ".zip"},
UA: "curl Linux/6.6.123 x86_64/unknown gnu",
},
{
Name: "goreleaser/darwin/arm64", Package: "goreleaser",
OS: buildmeta.OSDarwin, Arch: buildmeta.ArchARM64, Libc: "",
Formats: []string{".tar.gz", ".tar.xz", ".zip"},
UA: "curl Darwin/25.2.0 arm64/unknown libc",
},
{
Name: "goreleaser/windows/x86_64", Package: "goreleaser",
OS: buildmeta.OSWindows, Arch: buildmeta.ArchAMD64, Libc: "",
Formats: []string{".zip", ".exe"},
UA: "PowerShell/7.0 Windows/10.0 x86_64/unknown msvc",
},
{
Name: "ollama/linux/x86_64", Package: "ollama",
OS: buildmeta.OSLinux, Arch: buildmeta.ArchAMD64, Libc: buildmeta.LibcGNU,
Formats: []string{".tar.gz", ".tar.xz", ".tar.zst", ".zip"},
UA: "curl Linux/6.6.123 x86_64/unknown gnu",
},
{
Name: "ollama/darwin/arm64", Package: "ollama",
OS: buildmeta.OSDarwin, Arch: buildmeta.ArchARM64, Libc: "",
Formats: []string{".tar.gz", ".tar.xz", ".tar.zst", ".zip", ".dmg"},
UA: "curl Darwin/25.2.0 arm64/unknown libc",
},
{
Name: "ollama/linux/arm64", Package: "ollama",
OS: buildmeta.OSLinux, Arch: buildmeta.ArchARM64, Libc: buildmeta.LibcGNU,
Formats: []string{".tar.gz", ".tar.xz", ".tar.zst", ".zip"},
UA: "curl Linux/6.6.123 aarch64/unknown gnu",
},
{
Name: "node/linux/x86_64", Package: "node",
OS: buildmeta.OSLinux, Arch: buildmeta.ArchAMD64, Libc: buildmeta.LibcGNU,
Formats: []string{".tar.xz", ".tar.gz", ".zip"},
UA: "curl Linux/6.6.123 x86_64/unknown gnu",
},
{
Name: "node/darwin/arm64", Package: "node",
OS: buildmeta.OSDarwin, Arch: buildmeta.ArchARM64, Libc: "",
Formats: []string{".tar.xz", ".tar.gz", ".zip"},
UA: "curl Darwin/25.2.0 arm64/unknown libc",
},
{
Name: "node/linux/arm64", Package: "node",
OS: buildmeta.OSLinux, Arch: buildmeta.ArchARM64, Libc: buildmeta.LibcGNU,
Formats: []string{".tar.xz", ".tar.gz", ".zip"},
UA: "curl Linux/6.6.123 aarch64/unknown gnu",
},
}
func main() {
cacheDir := flag.String("cache", "_cache/raw", "root directory for raw cache")
confDir := flag.String("conf", ".", "root directory containing {pkg}/releases.conf files")
token := flag.String("token", os.Getenv("GITHUB_TOKEN"), "GitHub API token")
skipFetch := flag.Bool("skip-fetch", false, "skip fetching, use existing cache")
skipLive := flag.Bool("skip-live", false, "skip live API comparison")
packages := flag.String("packages", "goreleaser,ollama,node", "comma-separated packages to test")
flag.Parse()
pkgList := strings.Split(*packages, ",")
pkgSet := make(map[string]bool, len(pkgList))
for _, p := range pkgList {
pkgSet[strings.TrimSpace(p)] = true
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
defer cancel()
client := &http.Client{Timeout: 30 * time.Second}
var auth *githubish.Auth
if *token != "" {
auth = &githubish.Auth{Token: *token}
}
// Step 1: Fetch raw releases.
if !*skipFetch {
log.Println("=== Step 1: Fetching releases ===")
for _, pkg := range pkgList {
if err := fetchPackage(ctx, client, *cacheDir, *confDir, pkg, auth); err != nil {
log.Fatalf("fetch %s: %v", pkg, err)
}
}
} else {
log.Println("=== Step 1: Skipping fetch (using cache) ===")
}
// Step 2: Classify releases.
log.Println("=== Step 2: Classifying releases ===")
allDists := make(map[string][]resolve.Dist)
for _, pkg := range pkgList {
conf, err := installerconf.Read(filepath.Join(*confDir, pkg, "releases.conf"))
if err != nil {
log.Fatalf("read conf %s: %v", pkg, err)
}
d, err := rawcache.Open(filepath.Join(*cacheDir, pkg))
if err != nil {
log.Fatalf("open cache %s: %v", pkg, err)
}
dists, err := classifyFromCache(pkg, conf, d)
if err != nil {
log.Fatalf("classify %s: %v", pkg, err)
}
allDists[pkg] = dists
log.Printf(" %s: %d distributables", pkg, len(dists))
// Show catalog.
cat := resolve.Survey(dists)
log.Printf(" oses=%v arches=%v libcs=%v formats=%v", cat.OSes, cat.Arches, cat.Libcs, cat.Formats)
log.Printf(" latest=%s stable=%s", cat.Latest, cat.Stable)
}
// Step 3: Resolve best match for each test case.
log.Println("=== Step 3: Resolving best matches ===")
type result struct {
tc testCase
match *resolve.Match
live *liveResult
}
var results []result
for _, tc := range cases {
if !pkgSet[tc.Package] {
continue
}
dists := allDists[tc.Package]
q := resolve.Query{
OS: tc.OS,
Arch: tc.Arch,
Libc: tc.Libc,
Formats: tc.Formats,
Channel: "stable",
}
m := resolve.Best(dists, q)
results = append(results, result{tc: tc, match: m})
}
// Step 4: Compare with live API.
if !*skipLive {
log.Println("=== Step 4: Comparing with live API ===")
for i := range results {
tc := results[i].tc
live, err := queryLiveAPI(client, tc)
if err != nil {
log.Printf(" %s: live API error: %v", tc.Name, err)
continue
}
results[i].live = live
}
}
// Step 5: Report.
log.Println("")
log.Println("=== Results ===")
log.Println("")
pass, fail, warn := 0, 0, 0
for _, r := range results {
tc := r.tc
m := r.match
live := r.live
if m == nil {
log.Printf("FAIL %s: no match found", tc.Name)
fail++
continue
}
log.Printf("--- %s ---", tc.Name)
log.Printf(" Go: version=%s file=%s ext=%s url=%s", m.Version, m.Filename, m.Format, m.Download)
if live != nil {
log.Printf(" Live: version=%s file=%s ext=%s url=%s", live.Version, live.PkgFile, live.Ext, live.PkgURL)
if live.Version == "0.0.0" {
log.Printf(" WARN: live API returned error (no match)")
warn++
} else if m.Version == live.Version && m.Filename == live.PkgFile {
log.Printf(" PASS: exact match")
pass++
} else if m.Version == live.Version && m.Download == live.PkgURL {
log.Printf(" PASS: same URL (filename display differs: go=%s live=%s)", m.Filename, live.PkgFile)
pass++
} else if m.Version == live.Version {
log.Printf(" WARN: same version, different file (go=%s live=%s)", m.Filename, live.PkgFile)
warn++
} else {
log.Printf(" DIFF: version mismatch (go=%s live=%s)", m.Version, live.Version)
fail++
}
} else {
log.Printf(" (no live comparison)")
pass++
}
}
log.Println("")
log.Printf("Summary: %d pass, %d fail, %d warn (live API errors)", pass, fail, warn)
if fail > 0 {
os.Exit(1)
}
}
// fetchPackage fetches raw releases for one package.
func fetchPackage(ctx context.Context, client *http.Client, cacheRoot, confDir, pkg string, auth *githubish.Auth) error {
conf, err := installerconf.Read(filepath.Join(confDir, pkg, "releases.conf"))
if err != nil {
return fmt.Errorf("read conf: %w", err)
}
source := conf.Source
log.Printf(" %s: source=%s", pkg, source)
switch source {
case "github":
return fetchGitHub(ctx, client, cacheRoot, pkg, conf, auth)
case "nodedist":
return fetchNodeDist(ctx, client, cacheRoot, pkg, conf)
default:
return fmt.Errorf("unsupported source %q (only github and nodedist for e2e test)", source)
}
}
func fetchGitHub(ctx context.Context, client *http.Client, cacheRoot, pkg string, conf *installerconf.Conf, auth *githubish.Auth) error {
owner := conf.Owner
repo := conf.Repo
tagPrefix := conf.TagPrefix
d, err := rawcache.Open(filepath.Join(cacheRoot, pkg))
if err != nil {
return err
}
var added, changed, skipped int
var latest string
for batch, err := range github.Fetch(ctx, client, owner, repo, auth) {
if err != nil {
return fmt.Errorf("github %s/%s: %w", owner, repo, err)
}
for _, rel := range batch {
if rel.Draft {
continue
}
tag := rel.TagName
if tagPrefix != "" {
if !strings.HasPrefix(tag, tagPrefix) {
continue
}
tag = strings.TrimPrefix(tag, tagPrefix)
}
data, err := json.Marshal(rel)
if err != nil {
return fmt.Errorf("marshal %s: %w", tag, err)
}
action, err := d.Merge(tag, data)
if err != nil {
return err
}
switch action {
case "added":
added++
case "changed":
changed++
default:
skipped++
}
if latest == "" && !rel.Prerelease {
latest = tag
}
}
}
if latest != "" {
current := d.Latest()
if current == "" || lexver.Compare(lexver.Parse(latest), lexver.Parse(current)) > 0 {
d.SetLatest(latest)
}
}
log.Printf(" +%d ~%d =%d latest=%s", added, changed, skipped, d.Latest())
return nil
}
func fetchNodeDist(ctx context.Context, client *http.Client, cacheRoot, pkg string, conf *installerconf.Conf) error {
baseURL := conf.BaseURL
d, err := rawcache.Open(filepath.Join(cacheRoot, pkg))
if err != nil {
return err
}
var added, changed, skipped int
var latest string
for batch, err := range nodedist.Fetch(ctx, client, baseURL) {
if err != nil {
return fmt.Errorf("nodedist: %w", err)
}
for _, entry := range batch {
tag := entry.Version
data, err := json.Marshal(entry)
if err != nil {
return fmt.Errorf("marshal %s: %w", tag, err)
}
action, err := d.Merge(tag, data)
if err != nil {
return err
}
switch action {
case "added":
added++
case "changed":
changed++
default:
skipped++
}
if latest == "" {
latest = tag
}
}
}
if latest != "" {
current := d.Latest()
if current == "" || lexver.Compare(lexver.Parse(latest), lexver.Parse(current)) > 0 {
d.SetLatest(latest)
}
}
log.Printf(" +%d ~%d =%d latest=%s", added, changed, skipped, d.Latest())
return nil
}
// classifyFromCache reads the raw cache and produces classified dists.
func classifyFromCache(pkg string, conf *installerconf.Conf, d *rawcache.Dir) ([]resolve.Dist, error) {
source := conf.Source
switch source {
case "github":
return classifyGitHub(pkg, conf, d)
case "nodedist":
return classifyNodeDist(pkg, conf, d)
default:
return nil, fmt.Errorf("unsupported source %q", source)
}
}
func classifyGitHub(pkg string, conf *installerconf.Conf, d *rawcache.Dir) ([]resolve.Dist, error) {
tagPrefix := conf.TagPrefix
releases, err := readAllReleases(d)
if err != nil {
return nil, err
}
var dists []resolve.Dist
for _, data := range releases {
var rel struct {
TagName string `json:"tag_name"`
Prerelease bool `json:"prerelease"`
Draft bool `json:"draft"`
PublishedAt string `json:"published_at"`
Assets []struct {
Name string `json:"name"`
BrowserDownloadURL string `json:"browser_download_url"`
Size int64 `json:"size"`
} `json:"assets"`
}
if err := json.Unmarshal(data, &rel); err != nil {
continue
}
if rel.Draft {
continue
}
version := rel.TagName
if tagPrefix != "" {
version = strings.TrimPrefix(version, tagPrefix)
}
// Strip leading "v" for version normalization.
version = strings.TrimPrefix(version, "v")
channel := "stable"
if rel.Prerelease {
channel = "beta"
}
date := ""
if len(rel.PublishedAt) >= 10 {
date = rel.PublishedAt[:10]
}
for _, asset := range rel.Assets {
if isMetaAsset(asset.Name) {
continue
}
r := classifyFilename(asset.Name)
extra := detectExtra(asset.Name)
dists = append(dists, resolve.Dist{
Package: pkg,
Version: version,
Channel: channel,
OS: r.os,
Arch: r.arch,
Libc: r.libc,
Format: r.format,
Download: asset.BrowserDownloadURL,
Filename: asset.Name,
Size: asset.Size,
Date: date,
Extra: extra,
})
}
}
return dists, nil
}
func classifyNodeDist(pkg string, conf *installerconf.Conf, d *rawcache.Dir) ([]resolve.Dist, error) {
baseURL := conf.BaseURL
releases, err := readAllReleases(d)
if err != nil {
return nil, err
}
var dists []resolve.Dist
for _, data := range releases {
var entry struct {
Version string `json:"version"`
Date string `json:"date"`
Files []string `json:"files"`
LTS json.RawMessage `json:"lts"`
Security bool `json:"security"`
}
if err := json.Unmarshal(data, &entry); err != nil {
continue
}
lts := string(entry.LTS) != "false" && string(entry.LTS) != ""
version := strings.TrimPrefix(entry.Version, "v")
// Webi treats even major versions as "stable" (LTS-eligible).
channel := "stable"
parts := strings.SplitN(version, ".", 2)
if len(parts) > 0 {
var major int
fmt.Sscanf(parts[0], "%d", &major)
if major%2 != 0 {
channel = "beta"
}
}
for _, file := range entry.Files {
if file == "src" || file == "headers" {
continue
}
fileDists := expandNodeFile(pkg, entry.Version, version, channel, entry.Date, lts, baseURL, file)
dists = append(dists, fileDists...)
}
}
return dists, nil
}
func expandNodeFile(pkg, rawVersion, version, channel, date string, lts bool, baseURL, file string) []resolve.Dist {
parts := strings.Split(file, "-")
if len(parts) < 2 {
return nil
}
osMap := map[string]string{
"osx": "darwin", "linux": "linux", "win": "windows",
"sunos": "sunos", "aix": "aix",
}
archMap := map[string]string{
"x64": "x86_64", "x86": "x86", "arm64": "aarch64",
"armv7l": "armv7", "armv6l": "armv6",
"ppc64": "ppc64", "ppc64le": "ppc64le", "s390x": "s390x",
"loong64": "loong64", "riscv64": "riscv64",
}
os_ := osMap[parts[0]]
arch := archMap[parts[1]]
if os_ == "" || arch == "" {
return nil
}
libc := ""
pkgType := ""
if len(parts) > 2 {
pkgType = parts[2]
}
var formats []string
switch pkgType {
case "musl":
libc = "musl"
formats = []string{".tar.gz", ".tar.xz"}
case "tar":
formats = []string{".tar.gz", ".tar.xz"}
case "zip":
formats = []string{".zip"}
case "7z":
formats = []string{".7z"}
case "pkg":
formats = []string{".pkg"}
case "msi":
formats = []string{".msi"}
case "exe":
formats = []string{".exe"}
case "":
formats = []string{".tar.gz", ".tar.xz"}
default:
return nil
}
if libc == "" && os_ == "linux" {
libc = "gnu"
}
osPart := parts[0]
if osPart == "osx" {
osPart = "darwin"
}
archPart := parts[1]
muslExtra := ""
if libc == "musl" {
muslExtra = "-musl"
}
var dists []resolve.Dist
for _, format := range formats {
var filename string
if format == ".msi" {
filename = fmt.Sprintf("node-%s-%s%s%s", rawVersion, archPart, muslExtra, format)
} else {
filename = fmt.Sprintf("node-%s-%s-%s%s%s", rawVersion, osPart, archPart, muslExtra, format)
}
dists = append(dists, resolve.Dist{
Package: pkg,
Version: version,
Channel: channel,
OS: os_,
Arch: arch,
Libc: libc,
Format: format,
Download: fmt.Sprintf("%s/%s/%s", baseURL, rawVersion, filename),
Filename: filename,
LTS: lts,
Date: date,
})
}
return dists
}
// queryLiveAPI queries the live webi.sh API and parses the response header.
func queryLiveAPI(client *http.Client, tc testCase) (*liveResult, error) {
// Build format string matching what the webi.sh bootstrap sends.
// Order: tar,exe,zip,xz,dmg,git (least to most favorable in bootstrap,
// but the API doesn't care about order).
fmtParam := "tar,exe,zip,xz,dmg"
url := fmt.Sprintf("https://webi.sh/api/installers/%s@stable.sh?formats=%s", tc.Package, fmtParam)
req, err := http.NewRequest("GET", url, nil)
if err != nil {
return nil, err
}
req.Header.Set("User-Agent", tc.UA)
resp, err := client.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
body, err := io.ReadAll(resp.Body)
if err != nil {
return nil, err
}
return parseLiveResponse(string(body)), nil
}
// parseLiveResponse extracts WEBI_* and PKG_* variables from the shell script.
func parseLiveResponse(body string) *liveResult {
vars := make(map[string]string)
scanner := bufio.NewScanner(strings.NewReader(body))
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
for _, prefix := range []string{"WEBI_", "PKG_"} {
if strings.HasPrefix(line, prefix) {
if eq := strings.IndexByte(line, '='); eq > 0 {
key := line[:eq]
val := line[eq+1:]
val = strings.Trim(val, "'\"")
vars[key] = val
}
}
}
}
return &liveResult{
Version: vars["WEBI_VERSION"],
OS: vars["WEBI_OS"],
Arch: vars["WEBI_ARCH"],
Libc: vars["WEBI_LIBC"],
Ext: vars["WEBI_EXT"],
PkgURL: vars["WEBI_PKG_URL"],
PkgFile: vars["WEBI_PKG_FILE"],
Channel: vars["WEBI_CHANNEL"],
Stable: vars["PKG_STABLE"],
Latest: vars["PKG_LATEST"],
Oses: vars["PKG_OSES"],
Arches: vars["PKG_ARCHES"],
Libcs: vars["PKG_LIBCS"],
Formats: vars["PKG_FORMATS"],
}
}
// readAllReleases reads all cached release files.
func readAllReleases(d *rawcache.Dir) (map[string][]byte, error) {
active, err := d.ActivePath()
if err != nil {
return nil, err
}
entries, err := os.ReadDir(active)
if err != nil {
return nil, err
}
result := make(map[string][]byte, len(entries))
for _, e := range entries {
if e.IsDir() || strings.HasPrefix(e.Name(), "_") {
continue
}
data, err := os.ReadFile(filepath.Join(active, e.Name()))
if err != nil {
return nil, err
}
result[e.Name()] = data
}
return result, nil
}
type classResult struct {
os, arch, libc, format string
}
func classifyFilename(name string) classResult {
// Use the classify package.
// Import it indirectly to avoid circular deps — inline the logic
// we need for the e2e test.
lower := strings.ToLower(name)
var r classResult
r.format = detectFormat(name)
// OS detection
switch {
case strings.Contains(lower, "linux"):
r.os = "linux"
case strings.Contains(lower, "darwin") || strings.Contains(lower, "macos") || strings.Contains(lower, "apple"):
r.os = "darwin"
case strings.Contains(lower, "windows") || strings.Contains(lower, "win64") || strings.Contains(lower, "win32"):
r.os = "windows"
case strings.HasSuffix(lower, ".dmg") || strings.HasSuffix(lower, ".app.zip"):
r.os = "darwin"
case strings.HasSuffix(lower, ".exe") || strings.HasSuffix(lower, ".msi"):
r.os = "windows"
case strings.Contains(lower, "freebsd"):
r.os = "freebsd"
}
// Arch detection
switch {
case strings.Contains(lower, "x86_64") || strings.Contains(lower, "amd64") || strings.Contains(lower, "x64"):
r.arch = "x86_64"
case strings.Contains(lower, "aarch64") || strings.Contains(lower, "arm64"):
r.arch = "aarch64"
case strings.Contains(lower, "armv7") || strings.Contains(lower, "armhf"):
r.arch = "armv7"
case strings.Contains(lower, "armv6"):
r.arch = "armv6"
case strings.Contains(lower, "i686") || strings.Contains(lower, "i386") || strings.Contains(lower, "x86") || strings.Contains(lower, "386"):
r.arch = "x86"
case strings.Contains(lower, "ppc64le") || strings.Contains(lower, "powerpc64le"):
r.arch = "ppc64le"
case strings.Contains(lower, "ppc64") || strings.Contains(lower, "powerpc64"):
r.arch = "ppc64"
case strings.Contains(lower, "riscv64"):
r.arch = "riscv64"
case strings.Contains(lower, "s390x"):
r.arch = "s390x"
case strings.Contains(lower, "loong64"):
r.arch = "loong64"
}
// Libc detection
switch {
case strings.Contains(lower, "musl"):
r.libc = "musl"
case strings.Contains(lower, "gnu"):
r.libc = "gnu"
case strings.Contains(lower, "msvc"):
r.libc = "msvc"
}
return r
}
func detectFormat(name string) string {
lower := strings.ToLower(name)
for _, ext := range []string{".tar.gz", ".tar.xz", ".tar.bz2", ".tar.zst", ".exe.xz", ".app.zip"} {
if strings.HasSuffix(lower, ext) {
return ext
}
}
// .tgz is a common alias for .tar.gz
if strings.HasSuffix(lower, ".tgz") {
return ".tar.gz"
}
return filepath.Ext(lower)
}
// detectExtra identifies GPU/vendor-specific variant suffixes in filenames
// like "ollama-linux-amd64-rocm.tar.zst" or "ollama-linux-arm64-jetpack5.tar.zst".
func detectExtra(name string) string {
lower := strings.ToLower(name)
for _, variant := range []string{
"-rocm", "-jetpack", "-cuda", "-vulkan", "-metal",
"-extended", "-static", "-debug", "-nightly",
} {
if strings.Contains(lower, variant) {
return strings.TrimPrefix(variant, "-")
}
}
return ""
}
func isMetaAsset(name string) bool {
lower := strings.ToLower(name)
for _, suffix := range []string{
".sha256", ".sha256sum", ".sha512", ".sha512sum",
".md5", ".md5sum", ".sig", ".asc", ".pem",
"checksums.txt", "sha256sums", "sha512sums",
".sbom", ".spdx", ".json.sig", ".sigstore",
".d.ts", ".pub",
} {
if strings.HasSuffix(lower, suffix) {
return true
}
}
for _, contains := range []string{
"checksums", "sha256sum", "sha512sum",
"buildable-artifact",
} {
if strings.Contains(lower, contains) {
return true
}
}
for _, exact := range []string{
"install.sh", "install.ps1", "compat.json",
} {
if lower == exact {
return true
}
}
return false
}

862
cmd/fetchraw/main.go Normal file
View File

@@ -0,0 +1,862 @@
// Command fetchraw fetches release histories from upstream APIs and
// merges them into rawcache. Safe to run repeatedly — unchanged releases
// are skipped, new/changed ones are recorded in the audit log.
//
// Reads releases.conf files from package directories to discover what
// to fetch. Adding a new package is just creating a conf file.
//
// Usage:
//
// go run ./cmd/fetchraw -cache ./_cache/raw
// go run ./cmd/fetchraw -cache ./_cache/raw hugo caddy
package main
import (
"context"
"encoding/json"
"flag"
"fmt"
"log"
"net/http"
"os"
"path/filepath"
"sort"
"strings"
"time"
"github.com/webinstall/webi-installers/internal/installerconf"
"github.com/webinstall/webi-installers/internal/lexver"
"github.com/webinstall/webi-installers/internal/rawcache"
"github.com/webinstall/webi-installers/internal/releases/chromedist"
"github.com/webinstall/webi-installers/internal/releases/flutterdist"
"github.com/webinstall/webi-installers/internal/releases/gitea"
"github.com/webinstall/webi-installers/internal/releases/github"
"github.com/webinstall/webi-installers/internal/releases/githubish"
"github.com/webinstall/webi-installers/internal/releases/gittag"
"github.com/webinstall/webi-installers/internal/releases/golang"
"github.com/webinstall/webi-installers/internal/releases/gpgdist"
"github.com/webinstall/webi-installers/internal/releases/hashicorp"
"github.com/webinstall/webi-installers/internal/releases/iterm2dist"
"github.com/webinstall/webi-installers/internal/releases/juliadist"
"github.com/webinstall/webi-installers/internal/releases/mariadbdist"
"github.com/webinstall/webi-installers/internal/releases/nodedist"
"github.com/webinstall/webi-installers/internal/releases/zigdist"
)
func main() {
cacheDir := flag.String("cache", "_cache/raw", "root directory for raw cache")
confDir := flag.String("conf", ".", "root directory containing {pkg}/releases.conf files")
token := flag.String("token", os.Getenv("GITHUB_TOKEN"), "GitHub API token")
flag.Parse()
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
defer cancel()
client := &http.Client{Timeout: 30 * time.Second}
var auth *githubish.Auth
if *token != "" {
auth = &githubish.Auth{Token: *token}
}
// Discover packages from releases.conf files.
packages, err := discover(*confDir)
if err != nil {
log.Fatalf("discover: %v", err)
}
// Filter to requested packages if args given.
args := flag.Args()
if len(args) > 0 {
nameSet := make(map[string]bool, len(args))
for _, a := range args {
nameSet[a] = true
}
var filtered []pkgConf
for _, p := range packages {
if nameSet[p.name] {
filtered = append(filtered, p)
}
}
packages = filtered
}
log.Printf("found %d packages", len(packages))
for _, pkg := range packages {
// Aliases share cache with their target — skip fetching.
if alias := pkg.conf.Extra["alias_of"]; alias != "" {
log.Printf(" %s: alias of %s, skipping", pkg.name, alias)
continue
}
log.Printf("fetching %s...", pkg.name)
var err error
switch pkg.conf.Source {
case "github":
err = fetchGitHub(ctx, client, *cacheDir, pkg.name, pkg.conf, auth)
case "nodedist":
err = fetchNodeDist(ctx, client, *cacheDir, pkg.name, pkg.conf)
case "golang":
err = fetchGolang(ctx, client, *cacheDir, pkg.name)
case "zigdist":
err = fetchZig(ctx, client, *cacheDir, pkg.name)
case "flutterdist":
err = fetchFlutter(ctx, client, *cacheDir, pkg.name)
case "iterm2dist":
err = fetchITerm2(ctx, client, *cacheDir, pkg.name)
case "hashicorp":
err = fetchHashiCorp(ctx, client, *cacheDir, pkg.name, pkg.conf)
case "juliadist":
err = fetchJulia(ctx, client, *cacheDir, pkg.name)
case "gittag":
err = fetchGitTag(ctx, *cacheDir, pkg.name, pkg.conf)
case "gitea":
err = fetchGitea(ctx, client, *cacheDir, pkg.name, pkg.conf)
case "chromedist":
err = fetchChrome(ctx, client, *cacheDir, pkg.name)
case "gpgdist":
err = fetchGPG(ctx, client, *cacheDir, pkg.name)
case "mariadbdist":
err = fetchMariaDB(ctx, client, *cacheDir, pkg.name)
default:
log.Printf(" %s: unknown source %q, skipping", pkg.name, pkg.conf.Source)
continue
}
if err != nil {
log.Printf(" ERROR: %s: %v", pkg.name, err)
}
}
}
type pkgConf struct {
name string
conf *installerconf.Conf
}
// discover finds all {dir}/*/releases.conf files and returns them sorted.
func discover(dir string) ([]pkgConf, error) {
pattern := filepath.Join(dir, "*", "releases.conf")
matches, err := filepath.Glob(pattern)
if err != nil {
return nil, err
}
var packages []pkgConf
for _, path := range matches {
name := filepath.Base(filepath.Dir(path))
// Skip infrastructure dirs (_example, _webi, _common, etc.)
if strings.HasPrefix(name, "_") {
continue
}
conf, err := installerconf.Read(path)
if err != nil {
log.Printf("warning: %s: %v", path, err)
continue
}
packages = append(packages, pkgConf{name: name, conf: conf})
}
sort.Slice(packages, func(i, j int) bool {
return packages[i].name < packages[j].name
})
return packages, nil
}
func fetchNodeDist(ctx context.Context, client *http.Client, cacheRoot, pkgName string, conf *installerconf.Conf) error {
baseURL := conf.BaseURL
if baseURL == "" {
return fmt.Errorf("missing url in releases.conf")
}
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
if err != nil {
return err
}
var added, changed, skipped int
var latest string
for batch, err := range nodedist.Fetch(ctx, client, baseURL) {
if err != nil {
return fmt.Errorf("%s fetch: %w", pkgName, err)
}
for _, entry := range batch {
tag := entry.Version
data, err := json.Marshal(entry)
if err != nil {
return fmt.Errorf("%s marshal %s: %w", pkgName, tag, err)
}
action, err := d.Merge(tag, data)
if err != nil {
return err
}
switch action {
case "added":
added++
case "changed":
changed++
default:
skipped++
}
if latest == "" {
latest = tag
}
}
}
if err := updateLatest(d, latest); err != nil {
return err
}
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
return nil
}
func fetchGitHub(ctx context.Context, client *http.Client, cacheRoot, pkgName string, conf *installerconf.Conf, auth *githubish.Auth) error {
owner := conf.Owner
repo := conf.Repo
tagPrefix := conf.TagPrefix
if owner == "" || repo == "" {
return fmt.Errorf("missing owner or repo in releases.conf")
}
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
if err != nil {
return err
}
var added, changed, skipped int
var latest string
for batch, err := range github.Fetch(ctx, client, owner, repo, auth) {
if err != nil {
return fmt.Errorf("github %s/%s: %w", owner, repo, err)
}
for _, rel := range batch {
if rel.Draft {
continue
}
tag := rel.TagName
if tagPrefix != "" {
if !strings.HasPrefix(tag, tagPrefix) {
continue
}
tag = strings.TrimPrefix(tag, tagPrefix)
}
data, err := json.Marshal(rel)
if err != nil {
return fmt.Errorf("marshal %s: %w", tag, err)
}
action, err := d.Merge(tag, data)
if err != nil {
return err
}
switch action {
case "added":
added++
case "changed":
changed++
default:
skipped++
}
if latest == "" && !rel.Prerelease {
latest = tag
}
}
}
if err := updateLatest(d, latest); err != nil {
return err
}
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
return nil
}
func updateLatest(d *rawcache.Dir, candidate string) error {
if candidate == "" {
return nil
}
current := d.Latest()
if current == "" || lexver.Compare(lexver.Parse(candidate), lexver.Parse(current)) > 0 {
return d.SetLatest(candidate)
}
return nil
}
func fetchGolang(ctx context.Context, client *http.Client, cacheRoot, pkgName string) error {
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
if err != nil {
return err
}
var added, changed, skipped int
var latest string
for batch, err := range golang.Fetch(ctx, client) {
if err != nil {
return fmt.Errorf("golang: %w", err)
}
for _, rel := range batch {
tag := rel.Version // "go1.24.1"
data, err := json.Marshal(rel)
if err != nil {
return fmt.Errorf("golang marshal %s: %w", tag, err)
}
action, err := d.Merge(tag, data)
if err != nil {
return err
}
switch action {
case "added":
added++
case "changed":
changed++
default:
skipped++
}
if latest == "" && rel.Stable {
latest = tag
}
}
}
if err := updateLatest(d, latest); err != nil {
return err
}
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
return nil
}
func fetchZig(ctx context.Context, client *http.Client, cacheRoot, pkgName string) error {
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
if err != nil {
return err
}
var added, changed, skipped int
var latest string
for batch, err := range zigdist.Fetch(ctx, client) {
if err != nil {
return fmt.Errorf("zigdist: %w", err)
}
for _, rel := range batch {
tag := rel.Version
data, err := json.Marshal(rel)
if err != nil {
return fmt.Errorf("zigdist marshal %s: %w", tag, err)
}
action, err := d.Merge(tag, data)
if err != nil {
return err
}
switch action {
case "added":
added++
case "changed":
changed++
default:
skipped++
}
// Stable versions have dots and no dev/pre markers.
isStable := strings.Contains(tag, ".") && !strings.ContainsAny(tag, "+-")
if isStable {
if latest == "" || lexver.Compare(lexver.Parse(tag), lexver.Parse(latest)) > 0 {
latest = tag
}
}
}
}
if err := updateLatest(d, latest); err != nil {
return err
}
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
return nil
}
func fetchFlutter(ctx context.Context, client *http.Client, cacheRoot, pkgName string) error {
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
if err != nil {
return err
}
var added, changed, skipped int
var latest string
for batch, err := range flutterdist.Fetch(ctx, client) {
if err != nil {
return fmt.Errorf("flutterdist: %w", err)
}
for _, rel := range batch {
// Use version+channel+os+arch as the tag. The arch is embedded
// in the archive path (e.g. flutter_macos_arm64_3.0.0-stable.zip
// vs flutter_macos_3.0.0-stable.zip for universal/x64).
arch := ""
base := filepath.Base(rel.Archive)
prefix := "flutter_" + rel.OS + "_"
if after, ok := strings.CutPrefix(base, prefix); ok {
if !strings.HasPrefix(after, rel.Version) {
// There's an arch segment between OS and version.
if idx := strings.Index(after, "_"); idx > 0 {
arch = after[:idx]
}
}
}
tag := fmt.Sprintf("%s-%s-%s", rel.Version, rel.Channel, rel.OS)
if arch != "" {
tag += "-" + arch
}
data, err := json.Marshal(rel)
if err != nil {
return fmt.Errorf("flutterdist marshal %s: %w", tag, err)
}
action, err := d.Merge(tag, data)
if err != nil {
return err
}
switch action {
case "added":
added++
case "changed":
changed++
default:
skipped++
}
if latest == "" && rel.Channel == "stable" {
latest = tag
}
}
}
if err := updateLatest(d, latest); err != nil {
return err
}
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
return nil
}
func fetchITerm2(ctx context.Context, client *http.Client, cacheRoot, pkgName string) error {
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
if err != nil {
return err
}
var added, changed, skipped int
var latest string
for batch, err := range iterm2dist.Fetch(ctx, client) {
if err != nil {
return fmt.Errorf("iterm2dist: %w", err)
}
for _, entry := range batch {
tag := entry.Version
if tag == "" {
continue
}
data, err := json.Marshal(entry)
if err != nil {
return fmt.Errorf("iterm2dist marshal %s: %w", tag, err)
}
action, err := d.Merge(tag, data)
if err != nil {
return err
}
switch action {
case "added":
added++
case "changed":
changed++
default:
skipped++
}
if latest == "" && entry.Channel == "stable" {
latest = tag
}
}
}
if err := updateLatest(d, latest); err != nil {
return err
}
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
return nil
}
func fetchHashiCorp(ctx context.Context, client *http.Client, cacheRoot, pkgName string, conf *installerconf.Conf) error {
product := conf.Extra["product"]
if product == "" {
return fmt.Errorf("missing product in releases.conf")
}
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
if err != nil {
return err
}
var added, changed, skipped int
var latest string
for idx, err := range hashicorp.Fetch(ctx, client, product) {
if err != nil {
return fmt.Errorf("hashicorp %s: %w", product, err)
}
for tag, ver := range idx.Versions {
data, err := json.Marshal(ver)
if err != nil {
return fmt.Errorf("hashicorp marshal %s: %w", tag, err)
}
action, err := d.Merge(tag, data)
if err != nil {
return err
}
switch action {
case "added":
added++
case "changed":
changed++
default:
skipped++
}
// Stable = no prerelease markers. Compare all to find highest.
isStable := !strings.ContainsAny(tag, "-+")
if isStable {
if latest == "" || lexver.Compare(lexver.Parse(tag), lexver.Parse(latest)) > 0 {
latest = tag
}
}
}
}
if err := updateLatest(d, latest); err != nil {
return err
}
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
return nil
}
func fetchJulia(ctx context.Context, client *http.Client, cacheRoot, pkgName string) error {
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
if err != nil {
return err
}
var added, changed, skipped int
var latest string
for batch, err := range juliadist.Fetch(ctx, client) {
if err != nil {
return fmt.Errorf("juliadist: %w", err)
}
for _, rel := range batch {
tag := rel.Version
data, err := json.Marshal(rel)
if err != nil {
return fmt.Errorf("juliadist marshal %s: %w", tag, err)
}
action, err := d.Merge(tag, data)
if err != nil {
return err
}
switch action {
case "added":
added++
case "changed":
changed++
default:
skipped++
}
if rel.Stable {
if latest == "" || lexver.Compare(lexver.Parse(tag), lexver.Parse(latest)) > 0 {
latest = tag
}
}
}
}
if err := updateLatest(d, latest); err != nil {
return err
}
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
return nil
}
func fetchGitTag(ctx context.Context, cacheRoot, pkgName string, conf *installerconf.Conf) error {
gitURL := conf.BaseURL
if gitURL == "" {
return fmt.Errorf("missing url in releases.conf")
}
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
if err != nil {
return err
}
repoDir := filepath.Join(cacheRoot, "_repos")
if err := os.MkdirAll(repoDir, 0o755); err != nil {
return err
}
var added, changed, skipped int
var latest string
for batch, err := range gittag.Fetch(ctx, gitURL, repoDir) {
if err != nil {
return fmt.Errorf("gittag %s: %w", pkgName, err)
}
for _, entry := range batch {
tag := entry.Version
if tag == "" {
tag = "HEAD-" + entry.CommitHash
}
data, err := json.Marshal(entry)
if err != nil {
return fmt.Errorf("gittag marshal %s: %w", tag, err)
}
action, err := d.Merge(tag, data)
if err != nil {
return err
}
switch action {
case "added":
added++
case "changed":
changed++
default:
skipped++
}
if entry.GitTag != "" && entry.GitTag != "HEAD" {
if latest == "" || lexver.Compare(lexver.Parse(tag), lexver.Parse(latest)) > 0 {
latest = tag
}
}
}
}
if err := updateLatest(d, latest); err != nil {
return err
}
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
return nil
}
func fetchGitea(ctx context.Context, client *http.Client, cacheRoot, pkgName string, conf *installerconf.Conf) error {
baseURL := conf.BaseURL
owner := conf.Owner
repo := conf.Repo
if baseURL == "" || owner == "" || repo == "" {
return fmt.Errorf("missing base_url, owner, or repo in releases.conf")
}
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
if err != nil {
return err
}
var added, changed, skipped int
var latest string
for batch, err := range gitea.Fetch(ctx, client, baseURL, owner, repo, nil) {
if err != nil {
return fmt.Errorf("gitea %s/%s: %w", owner, repo, err)
}
for _, rel := range batch {
if rel.Draft {
continue
}
tag := rel.TagName
data, err := json.Marshal(rel)
if err != nil {
return fmt.Errorf("gitea marshal %s: %w", tag, err)
}
action, err := d.Merge(tag, data)
if err != nil {
return err
}
switch action {
case "added":
added++
case "changed":
changed++
default:
skipped++
}
if latest == "" && !rel.Prerelease {
latest = tag
}
}
}
if err := updateLatest(d, latest); err != nil {
return err
}
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
return nil
}
func fetchChrome(ctx context.Context, client *http.Client, cacheRoot, pkgName string) error {
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
if err != nil {
return err
}
var added, changed, skipped int
var latest string
for batch, err := range chromedist.Fetch(ctx, client) {
if err != nil {
return fmt.Errorf("chromedist: %w", err)
}
for _, ver := range batch {
tag := ver.Version
data, err := json.Marshal(ver)
if err != nil {
return fmt.Errorf("chromedist marshal %s: %w", tag, err)
}
action, err := d.Merge(tag, data)
if err != nil {
return err
}
switch action {
case "added":
added++
case "changed":
changed++
default:
skipped++
}
if latest == "" || lexver.Compare(lexver.Parse(tag), lexver.Parse(latest)) > 0 {
latest = tag
}
}
}
if err := updateLatest(d, latest); err != nil {
return err
}
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
return nil
}
func fetchGPG(ctx context.Context, client *http.Client, cacheRoot, pkgName string) error {
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
if err != nil {
return err
}
var added, changed, skipped int
var latest string
for batch, err := range gpgdist.Fetch(ctx, client) {
if err != nil {
return fmt.Errorf("gpgdist: %w", err)
}
for _, entry := range batch {
tag := entry.Version
data, err := json.Marshal(entry)
if err != nil {
return fmt.Errorf("gpgdist marshal %s: %w", tag, err)
}
action, err := d.Merge(tag, data)
if err != nil {
return err
}
switch action {
case "added":
added++
case "changed":
changed++
default:
skipped++
}
if latest == "" || lexver.Compare(lexver.Parse(tag), lexver.Parse(latest)) > 0 {
latest = tag
}
}
}
if err := updateLatest(d, latest); err != nil {
return err
}
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
return nil
}
func fetchMariaDB(ctx context.Context, client *http.Client, cacheRoot, pkgName string) error {
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
if err != nil {
return err
}
var added, changed, skipped int
var latest string
for batch, err := range mariadbdist.Fetch(ctx, client) {
if err != nil {
return fmt.Errorf("mariadbdist: %w", err)
}
for _, rel := range batch {
tag := rel.ReleaseID
data, err := json.Marshal(rel)
if err != nil {
return fmt.Errorf("mariadbdist marshal %s: %w", tag, err)
}
action, err := d.Merge(tag, data)
if err != nil {
return err
}
switch action {
case "added":
added++
case "changed":
changed++
default:
skipped++
}
isStable := rel.MajorStatus == "Stable"
if isStable {
if latest == "" || lexver.Compare(lexver.Parse(tag), lexver.Parse(latest)) > 0 {
latest = tag
}
}
}
}
if err := updateLatest(d, latest); err != nil {
return err
}
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
return nil
}

625
cmd/inspect/main.go Normal file
View File

@@ -0,0 +1,625 @@
// Command inspect downloads release archives, unpacks them, and reports
// their internal structure. This helps discover how packages are laid out
// and whether the layout changes across versions.
//
// Usage:
//
// go run ./cmd/inspect -csv distributables.csv -cache ./_cache/downloads ollama sd
package main
import (
"context"
"encoding/csv"
"flag"
"fmt"
"io"
"log"
"mime"
"net/http"
"os"
"os/exec"
"path/filepath"
"sort"
"strings"
"time"
"github.com/webinstall/webi-installers/internal/httpclient"
)
// Row is one CSV row from distributables.csv.
type Row struct {
Package string
Version string
Channel string
Date string
OS string
Arch string
Libc string
Format string
Download string
Filename string
Extra string
}
// archiveFormats are the formats we download and unpack.
var archiveFormats = map[string]bool{
".tar.gz": true,
".tar.xz": true,
".tar.bz2": true,
".tar.zst": true,
".zip": true,
".dmg": true,
".gz": true,
".xz": true,
}
// inspectOSes are the OSes we inspect.
var inspectOSes = map[string]bool{
"linux": true,
"darwin": true,
"windows": true,
"": true, // source-only packages
}
// preferredArch picks one arch per OS to download.
func preferredArch(os_ string) string {
switch os_ {
case "darwin":
return "aarch64"
default:
return "x86_64"
}
}
func main() {
csvFile := flag.String("csv", "distributables.csv", "path to distributables CSV")
cacheDir := flag.String("cache", "_cache/downloads", "download cache directory")
flag.Parse()
packages := flag.Args()
if len(packages) == 0 {
log.Fatal("usage: inspect [-csv FILE] [-cache DIR] PACKAGE [PACKAGE...]")
}
rows, err := readCSV(*csvFile)
if err != nil {
log.Fatalf("read csv: %v", err)
}
client := httpclient.New()
// Override timeout for large downloads.
client.Timeout = 10 * time.Minute
for _, pkg := range packages {
log.Printf("=== %s ===", pkg)
if err := inspectPackage(client, rows, pkg, *cacheDir); err != nil {
log.Printf("ERROR: %s: %v", pkg, err)
}
}
}
func readCSV(path string) ([]Row, error) {
f, err := os.Open(path)
if err != nil {
return nil, err
}
defer f.Close()
r := csv.NewReader(f)
header, err := r.Read()
if err != nil {
return nil, err
}
// Build column index.
idx := make(map[string]int, len(header))
for i, col := range header {
idx[col] = i
}
var rows []Row
for {
record, err := r.Read()
if err == io.EOF {
break
}
if err != nil {
return nil, err
}
get := func(col string) string {
if i, ok := idx[col]; ok && i < len(record) {
return record[i]
}
return ""
}
rows = append(rows, Row{
Package: get("package"),
Version: get("version"),
Channel: get("channel"),
Date: get("date"),
OS: get("os"),
Arch: get("arch"),
Libc: get("libc"),
Format: get("format"),
Download: get("download"),
Filename: get("filename"),
Extra: get("extra"),
})
}
return rows, nil
}
func inspectPackage(client *http.Client, allRows []Row, pkg, cacheDir string) error {
// Filter rows for this package.
var pkgRows []Row
for _, r := range allRows {
if r.Package == pkg {
pkgRows = append(pkgRows, r)
}
}
if len(pkgRows) == 0 {
return fmt.Errorf("no rows found")
}
// Find latest stable version, fall back to any version.
versions := findVersionsByDate(pkgRows)
if len(versions) == 0 {
return fmt.Errorf("no versions found")
}
latestVer := versions[0]
log.Printf(" latest version: %s", latestVer)
// Check if latest has assets uploaded (more than just source tarballs).
latestRows := filterVersion(pkgRows, latestVer)
hasRealAssets := false
for _, r := range latestRows {
if r.Extra != "source" && archiveFormats[r.Format] {
hasRealAssets = true
break
}
}
// If latest looks empty, step back one version.
if !hasRealAssets && len(versions) > 1 {
latestVer = versions[1]
latestRows = filterVersion(pkgRows, latestVer)
log.Printf(" latest has no assets, using: %s", latestVer)
}
// Inspect the latest version.
if err := inspectVersion(client, pkg, latestVer, latestRows, cacheDir); err != nil {
return err
}
// Find versions roughly a year apart going back.
yearVersions := findYearlyVersions(pkgRows, latestVer)
for _, v := range yearVersions {
log.Printf(" --- checking %s ---", v)
vRows := filterVersion(pkgRows, v)
if err := inspectVersion(client, pkg, v, vRows, cacheDir); err != nil {
log.Printf(" ERROR: %v", err)
}
}
return nil
}
// findVersionsByDate returns versions sorted newest first, preferring stable.
func findVersionsByDate(rows []Row) []string {
type vInfo struct {
version string
date string
stable bool
}
seen := map[string]*vInfo{}
for _, r := range rows {
if _, ok := seen[r.Version]; !ok {
seen[r.Version] = &vInfo{
version: r.Version,
date: r.Date,
stable: r.Channel == "stable",
}
}
}
var vs []*vInfo
for _, v := range seen {
vs = append(vs, v)
}
// Sort: stable first, then by date descending, then version descending.
sort.Slice(vs, func(i, j int) bool {
if vs[i].stable != vs[j].stable {
return vs[i].stable
}
if vs[i].date != vs[j].date {
return vs[i].date > vs[j].date
}
return vs[i].version > vs[j].version
})
result := make([]string, len(vs))
for i, v := range vs {
result[i] = v.version
}
return result
}
// findYearlyVersions picks versions roughly a year apart before the given version.
func findYearlyVersions(rows []Row, latestVer string) []string {
// Find the date of latest.
var latestDate string
for _, r := range rows {
if r.Version == latestVer && r.Date != "" {
latestDate = r.Date
break
}
}
if latestDate == "" {
return nil
}
latestTime, err := time.Parse("2006-01-02", latestDate)
if err != nil {
return nil
}
// Collect all stable versions with dates.
type vd struct {
version string
date time.Time
}
seen := map[string]bool{}
var all []vd
for _, r := range rows {
if seen[r.Version] || r.Date == "" || r.Channel != "stable" {
continue
}
seen[r.Version] = true
t, err := time.Parse("2006-01-02", r.Date)
if err != nil {
continue
}
if t.Before(latestTime) {
all = append(all, vd{r.Version, t})
}
}
sort.Slice(all, func(i, j int) bool {
return all[i].date.After(all[j].date)
})
// Pick versions roughly a year apart.
var result []string
nextTarget := latestTime.AddDate(-1, 0, 0)
for _, v := range all {
if v.date.Before(nextTarget) || v.date.Equal(nextTarget) {
result = append(result, v.version)
nextTarget = v.date.AddDate(-1, 0, 0)
}
}
return result
}
func filterVersion(rows []Row, version string) []Row {
var result []Row
for _, r := range rows {
if r.Version == version {
result = append(result, r)
}
}
return result
}
// inspectVersion downloads and inspects archives for one version.
func inspectVersion(client *http.Client, pkg, version string, rows []Row, cacheDir string) error {
// Group by OS, pick one arch per OS, pick distinct formats.
type dlKey struct {
os_ string
format string
}
selected := map[dlKey]*Row{}
for i := range rows {
r := &rows[i]
if !inspectOSes[r.OS] {
continue
}
if !archiveFormats[r.Format] {
continue
}
key := dlKey{r.OS, r.Format}
existing := selected[key]
if existing == nil {
selected[key] = r
continue
}
// Prefer the preferred arch.
pref := preferredArch(r.OS)
if r.Arch == pref && existing.Arch != pref {
selected[key] = r
}
// Skip rocm/jetpack variants.
if strings.Contains(r.Filename, "rocm") || strings.Contains(r.Filename, "jetpack") {
if !strings.Contains(existing.Filename, "rocm") && !strings.Contains(existing.Filename, "jetpack") {
continue // keep existing non-special variant
}
}
}
if len(selected) == 0 {
log.Printf(" %s: no downloadable archives", version)
return nil
}
// Sort keys for deterministic output.
var keys []dlKey
for k := range selected {
keys = append(keys, k)
}
sort.Slice(keys, func(i, j int) bool {
if keys[i].os_ != keys[j].os_ {
return keys[i].os_ < keys[j].os_
}
return keys[i].format < keys[j].format
})
for _, key := range keys {
r := selected[key]
os_ := r.OS
if os_ == "" {
os_ = "any"
}
log.Printf(" [%s] %s %s → %s", version, os_, r.Format, r.Filename)
dlPath, err := download(client, r.Download, r.Filename, filepath.Join(cacheDir, pkg, version))
if err != nil {
log.Printf(" download error: %v", err)
continue
}
contents, err := unpackAndList(dlPath, r.Format)
if err != nil {
log.Printf(" unpack error: %v", err)
continue
}
printContents(contents)
}
return nil
}
// download fetches a URL to the cache dir. Returns the path to the cached file.
// Skips download if the file already exists.
func download(client *http.Client, url, hintFilename, dir string) (string, error) {
// Check if already cached by hint filename.
cached := filepath.Join(dir, hintFilename)
if _, err := os.Stat(cached); err == nil {
return cached, nil
}
if err := os.MkdirAll(dir, 0o755); err != nil {
return "", err
}
ctx := context.Background()
resp, err := httpclient.Get(ctx, client, url)
if err != nil {
return "", fmt.Errorf("GET %s: %w", url, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return "", fmt.Errorf("GET %s: %s", url, resp.Status)
}
// Determine filename from Content-Disposition or hint.
filename := hintFilename
if cd := resp.Header.Get("Content-Disposition"); cd != "" {
_, params, err := mime.ParseMediaType(cd)
if err == nil {
if fn, ok := params["filename"]; ok && fn != "" {
filename = fn
}
}
}
outPath := filepath.Join(dir, filename)
// Atomic write: temp file + rename.
tmp := outPath + ".tmp"
f, err := os.Create(tmp)
if err != nil {
return "", err
}
n, err := io.Copy(f, resp.Body)
if closeErr := f.Close(); closeErr != nil && err == nil {
err = closeErr
}
if err != nil {
os.Remove(tmp)
return "", fmt.Errorf("download %s: %w", url, err)
}
if err := os.Rename(tmp, outPath); err != nil {
os.Remove(tmp)
return "", err
}
log.Printf(" downloaded %s (%d bytes)", filename, n)
return outPath, nil
}
// FileEntry describes one file inside an archive.
type FileEntry struct {
Path string
Size int64
Mode os.FileMode
IsDir bool
IsExec bool
IsSymlink bool
LinkTarget string
}
// unpackAndList extracts an archive to a temp dir and lists contents.
func unpackAndList(archivePath, format string) ([]FileEntry, error) {
tmpDir, err := os.MkdirTemp("", "webi-inspect-*")
if err != nil {
return nil, err
}
defer os.RemoveAll(tmpDir)
switch format {
case ".tar.gz":
err = run("tar", "xzf", archivePath, "-C", tmpDir)
case ".tar.xz":
err = run("tar", "xJf", archivePath, "-C", tmpDir)
case ".tar.bz2":
err = run("tar", "xjf", archivePath, "-C", tmpDir)
case ".tar.zst":
err = run("tar", "--zstd", "-xf", archivePath, "-C", tmpDir)
case ".zip":
err = run("unzip", "-q", "-o", archivePath, "-d", tmpDir)
case ".dmg":
err = extractDMG(archivePath, tmpDir)
case ".gz":
// Single file gzip.
base := filepath.Base(archivePath)
base = strings.TrimSuffix(base, ".gz")
outPath := filepath.Join(tmpDir, base)
err = run("sh", "-c", fmt.Sprintf("gunzip -c %q > %q", archivePath, outPath))
case ".xz":
base := filepath.Base(archivePath)
base = strings.TrimSuffix(base, ".xz")
outPath := filepath.Join(tmpDir, base)
err = run("sh", "-c", fmt.Sprintf("xz -dc %q > %q", archivePath, outPath))
default:
return nil, fmt.Errorf("unsupported format: %s", format)
}
if err != nil {
return nil, fmt.Errorf("extract %s: %w", format, err)
}
return listDir(tmpDir, "")
}
func extractDMG(dmgPath, outDir string) error {
// Try 7z first (doesn't require mounting).
if _, err := exec.LookPath("7z"); err == nil {
return run("7z", "x", "-o"+outDir, dmgPath)
}
// Fall back to hdiutil mount + copy + unmount.
mountPoint, err := os.MkdirTemp("", "webi-dmg-*")
if err != nil {
return err
}
defer os.RemoveAll(mountPoint)
if err := run("hdiutil", "attach", dmgPath, "-mountpoint", mountPoint, "-nobrowse", "-quiet"); err != nil {
return fmt.Errorf("mount dmg: %w", err)
}
defer run("hdiutil", "detach", mountPoint, "-quiet")
// Copy contents.
return run("cp", "-R", mountPoint+"/.", outDir)
}
func run(name string, args ...string) error {
cmd := exec.Command(name, args...)
cmd.Stderr = os.Stderr
return cmd.Run()
}
func listDir(root, prefix string) ([]FileEntry, error) {
entries, err := os.ReadDir(filepath.Join(root, prefix))
if err != nil {
return nil, err
}
var result []FileEntry
for _, e := range entries {
relPath := filepath.Join(prefix, e.Name())
fullPath := filepath.Join(root, relPath)
info, err := e.Info()
if err != nil {
continue
}
entry := FileEntry{
Path: relPath,
Size: info.Size(),
Mode: info.Mode(),
IsDir: e.IsDir(),
}
if info.Mode()&os.ModeSymlink != 0 {
entry.IsSymlink = true
target, _ := os.Readlink(fullPath)
entry.LinkTarget = target
}
if !e.IsDir() && info.Mode()&0o111 != 0 {
entry.IsExec = true
}
result = append(result, entry)
if e.IsDir() {
sub, err := listDir(root, relPath)
if err != nil {
continue
}
result = append(result, sub...)
}
}
return result, nil
}
func printContents(entries []FileEntry) {
for _, e := range entries {
marker := " "
if e.IsExec {
marker = "* "
}
if e.IsDir {
marker = "d "
}
if e.IsSymlink {
marker = "→ "
}
size := ""
if !e.IsDir {
size = formatSize(e.Size)
}
line := fmt.Sprintf(" %s%-50s %8s", marker, e.Path, size)
if e.IsSymlink {
line += " → " + e.LinkTarget
}
log.Print(line)
}
}
func formatSize(n int64) string {
switch {
case n >= 1<<30:
return fmt.Sprintf("%.1fG", float64(n)/float64(1<<30))
case n >= 1<<20:
return fmt.Sprintf("%.1fM", float64(n)/float64(1<<20))
case n >= 1<<10:
return fmt.Sprintf("%.1fK", float64(n)/float64(1<<10))
default:
return fmt.Sprintf("%dB", n)
}
}

356
cmd/uaparse/main.go Normal file
View File

@@ -0,0 +1,356 @@
// Command uaparse analyzes User-Agent strings from webi.sh logs.
//
// It reads UA strings (one per line) from stdin or a file, parses each
// through uadetect, and produces summary output showing:
// - unique platform tuples (os, arch, libc) with counts
// - platform hints extracted from kernel version strings (cloud provider,
// container runtime, device info)
// - detection failures and malformed UAs
//
// Usage:
//
// uaparse < LIVE-UAS.txt
// uaparse LIVE-UAS.txt
// uaparse -json LIVE-UAS.txt
// uaparse -fixtures LIVE-UAS.txt # output Go test fixtures
package main
import (
"bufio"
"encoding/json"
"flag"
"fmt"
"os"
"regexp"
"sort"
"strings"
"github.com/webinstall/webi-installers/internal/uadetect"
)
// PlatformKey is the resolution-relevant tuple — everything else is noise
// for artifact selection.
type PlatformKey struct {
OS string `json:"os"`
Arch string `json:"arch"`
Libc string `json:"libc"`
}
func (k PlatformKey) String() string {
return fmt.Sprintf("%-10s %-10s %s", k.OS, k.Arch, k.Libc)
}
// PlatformEntry holds a unique platform and its metadata.
type PlatformEntry struct {
Key PlatformKey `json:"key"`
Count int `json:"count"`
Examples []string `json:"examples"` // up to 3 representative UAs
Hints []string `json:"hints"` // unique platform hints seen
}
// UAIssue records a malformed or undetectable UA.
type UAIssue struct {
Line int `json:"line"`
UA string `json:"ua"`
Reason string `json:"reason"`
}
// Hint is a platform detail extracted from the kernel version string.
type Hint struct {
Tag string // short label: "amzn", "azure", "gcp", "wsl", etc.
Count int
}
func main() {
jsonOut := flag.Bool("json", false, "output as JSON")
fixtures := flag.Bool("fixtures", false, "output Go test fixture table")
flag.Parse()
var scanner *bufio.Scanner
if flag.NArg() > 0 {
f, err := os.Open(flag.Arg(0))
if err != nil {
fmt.Fprintf(os.Stderr, "uaparse: %v\n", err)
os.Exit(1)
}
defer f.Close()
scanner = bufio.NewScanner(f)
} else {
scanner = bufio.NewScanner(os.Stdin)
}
platforms := make(map[PlatformKey]*PlatformEntry)
hints := make(map[string]int)
var issues []UAIssue
lineNum := 0
for scanner.Scan() {
lineNum++
ua := strings.TrimSpace(scanner.Text())
if ua == "" {
continue
}
// Detect corruption: truncated/double-pasted lines.
if isMalformed(ua) {
issues = append(issues, UAIssue{
Line: lineNum,
UA: ua,
Reason: "malformed (truncated or corrupted)",
})
continue
}
// Parse through uadetect.
result := uadetect.Parse(ua)
// Check for detection failures.
if result.OS == "" {
issues = append(issues, UAIssue{
Line: lineNum,
UA: ua,
Reason: "OS not detected",
})
}
if result.Arch == "" {
issues = append(issues, UAIssue{
Line: lineNum,
UA: ua,
Reason: "arch not detected",
})
}
key := PlatformKey{
OS: string(result.OS),
Arch: string(result.Arch),
Libc: string(result.Libc),
}
entry, ok := platforms[key]
if !ok {
entry = &PlatformEntry{Key: key}
platforms[key] = entry
}
entry.Count++
if len(entry.Examples) < 3 {
entry.Examples = append(entry.Examples, ua)
}
// Extract platform hints from kernel version.
for _, h := range extractHints(ua) {
if !containsStr(entry.Hints, h) {
entry.Hints = append(entry.Hints, h)
}
hints[h]++
}
}
if err := scanner.Err(); err != nil {
fmt.Fprintf(os.Stderr, "uaparse: read error: %v\n", err)
os.Exit(1)
}
// Sort platforms by count descending.
entries := make([]*PlatformEntry, 0, len(platforms))
for _, e := range platforms {
entries = append(entries, e)
}
sort.Slice(entries, func(i, j int) bool {
return entries[i].Count > entries[j].Count
})
if *jsonOut {
outputJSON(entries, issues, hints)
} else if *fixtures {
outputFixtures(entries)
} else {
outputTable(entries, issues, hints, lineNum)
}
}
func outputTable(entries []*PlatformEntry, issues []UAIssue, hints map[string]int, total int) {
fmt.Printf("=== UA Analysis: %d lines → %d unique platforms ===\n\n", total, len(entries))
fmt.Printf("%-10s %-10s %-6s %6s %s\n", "OS", "ARCH", "LIBC", "COUNT", "HINTS")
fmt.Println(strings.Repeat("-", 72))
for _, e := range entries {
hintStr := ""
if len(e.Hints) > 0 {
hintStr = strings.Join(e.Hints, ", ")
}
fmt.Printf("%-10s %-10s %-6s %6d %s\n",
displayOS(e.Key.OS), e.Key.Arch, displayLibc(e.Key.Libc),
e.Count, hintStr)
}
if len(hints) > 0 {
fmt.Printf("\n=== Platform Hints (environment signals from kernel strings) ===\n\n")
sortedHints := make([]Hint, 0, len(hints))
for tag, count := range hints {
sortedHints = append(sortedHints, Hint{tag, count})
}
sort.Slice(sortedHints, func(i, j int) bool {
return sortedHints[i].Count > sortedHints[j].Count
})
for _, h := range sortedHints {
fmt.Printf(" %-20s %d\n", h.Tag, h.Count)
}
}
if len(issues) > 0 {
fmt.Printf("\n=== Issues (%d) ===\n\n", len(issues))
for _, iss := range issues {
fmt.Printf(" line %d: %s\n %s\n", iss.Line, iss.Reason, iss.UA)
}
}
}
func outputJSON(entries []*PlatformEntry, issues []UAIssue, hints map[string]int) {
out := struct {
Platforms []*PlatformEntry `json:"platforms"`
Issues []UAIssue `json:"issues"`
Hints map[string]int `json:"hints"`
}{entries, issues, hints}
enc := json.NewEncoder(os.Stdout)
enc.SetIndent("", " ")
_ = enc.Encode(out)
}
func outputFixtures(entries []*PlatformEntry) {
fmt.Println("// Generated by cmd/uaparse from live UA data.")
fmt.Println("// Each entry represents a unique (os, arch, libc) platform seen in production.")
fmt.Println("var liveUAPlatforms = []struct {")
fmt.Println("\tua string")
fmt.Println("\tos buildmeta.OS")
fmt.Println("\tarch buildmeta.Arch")
fmt.Println("\tlibc buildmeta.Libc")
fmt.Println("}{")
for _, e := range entries {
if e.Key.OS == "" || e.Key.Arch == "" {
continue // skip undetectable
}
ua := e.Examples[0]
fmt.Printf("\t{%q, %s, %s, %s},\n",
ua, goConst("OS", e.Key.OS), goConst("Arch", e.Key.Arch), goConst("Libc", e.Key.Libc))
}
fmt.Println("}")
}
// isMalformed checks for genuinely corrupted UA strings (network truncation).
func isMalformed(ua string) bool {
// Extremely short (less than 10 chars) suggests truncation.
if len(ua) < 10 {
return true
}
return false
}
// extractHints finds environment signals in a UA string.
func extractHints(ua string) []string {
lower := strings.ToLower(ua)
var out []string
patterns := []struct {
substr string
tag string
}{
{"amzn", "amzn"}, // Amazon Linux
{"-azure", "azure"}, // Azure VM
{"-gcp", "gcp"}, // Google Cloud
{"-aws", "aws"}, // AWS kernel
{"-oracle", "oracle"}, // Oracle Cloud
{"el7", "rhel7"}, // RHEL/CentOS 7
{"el8", "rhel8"}, // RHEL/CentOS 8
{"el9", "rhel9"}, // RHEL/CentOS 9
{".fc", "fedora"}, // Fedora
{"+deb", "debian"}, // Debian
{"-generic", "ubuntu"}, // Ubuntu generic kernel
{"-pve", "proxmox"}, // Proxmox VE
{"linuxkit", "docker"}, // Docker Desktop / linuxkit
{"orbstack", "orbstack"},
{"microsoft-standard-wsl", "wsl"},
{"android", "android"},
{"+rpt-rpi", "rpi"}, // Raspberry Pi
{"cygwin", "cygwin"},
{"mingw", "mingw"},
{"msys", "msys"},
{"freebsd", "freebsd"},
{"-nvidia", "nvidia"},
{"gentoo", "gentoo"},
{"coreweave", "coreweave"},
}
for _, p := range patterns {
if strings.Contains(lower, p.substr) {
out = append(out, p.tag)
}
}
return out
}
// androidDeviceRe extracts device/build info from Android kernel strings.
var androidDeviceRe = regexp.MustCompile(`ab[A-Z0-9]+`)
func displayOS(os string) string {
if os == "" {
return "(none)"
}
return os
}
func displayLibc(libc string) string {
if libc == "" {
return "(none)"
}
return libc
}
func goConst(prefix, val string) string {
m := map[string]map[string]string{
"OS": {
"darwin": "buildmeta.OSDarwin",
"linux": "buildmeta.OSLinux",
"windows": "buildmeta.OSWindows",
"freebsd": "buildmeta.OSFreeBSD",
"android": "buildmeta.OSAndroid",
"": `""`,
},
"Arch": {
"aarch64": "buildmeta.ArchARM64",
"x86_64": "buildmeta.ArchAMD64",
"armv7": "buildmeta.ArchARMv7",
"armv6": "buildmeta.ArchARMv6",
"x86": "buildmeta.ArchX86",
"ppc64le": "buildmeta.ArchPPC64LE",
"ppc64": "buildmeta.ArchPPC64",
"s390x": "buildmeta.ArchS390X",
"riscv64": "buildmeta.ArchRISCV64",
"": `""`,
},
"Libc": {
"gnu": "buildmeta.LibcGNU",
"musl": "buildmeta.LibcMusl",
"msvc": "buildmeta.LibcMSVC",
"none": "buildmeta.LibcNone",
"": `""`,
},
}
if v, ok := m[prefix][val]; ok {
return v
}
return fmt.Sprintf("%q /* unmapped */", val)
}
func containsStr(ss []string, s string) bool {
for _, v := range ss {
if v == s {
return true
}
}
return false
}

899
cmd/webicached/main.go Normal file
View File

@@ -0,0 +1,899 @@
// Command webicached is the release cache daemon. It fetches releases
// from upstream sources, classifies build assets, and writes them to
// the _cache/ directory in the format the Node.js server expects.
//
// This is the Go replacement for the Node.js release-fetching pipeline.
// It reads releases.conf files to discover packages, fetches from the
// configured source, classifies assets, and writes to fsstore.
//
// Default mode: classify all from existing rawcache on startup, then
// fetch+refresh one package per tick (round-robin, 15m default).
//
// Usage:
//
// go run ./cmd/webicached # default: round-robin, one per tick
// go run ./cmd/webicached -eager # fetch all packages on startup
// go run ./cmd/webicached -once -no-fetch # classify from rawcache and exit
// go run ./cmd/webicached bat goreleaser # only these packages
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"log"
"math/rand/v2"
"net/http"
"os"
"path/filepath"
"sort"
"strings"
"time"
"github.com/joho/godotenv"
"github.com/webinstall/webi-installers/internal/classifypkg"
"github.com/webinstall/webi-installers/internal/installerconf"
"github.com/webinstall/webi-installers/internal/rawcache"
"github.com/webinstall/webi-installers/internal/releases/chromedist"
"github.com/webinstall/webi-installers/internal/releases/flutterdist"
"github.com/webinstall/webi-installers/internal/releases/gitea"
"github.com/webinstall/webi-installers/internal/releases/github"
"github.com/webinstall/webi-installers/internal/releases/githubish"
"github.com/webinstall/webi-installers/internal/releases/gittag"
"github.com/webinstall/webi-installers/internal/releases/golang"
"github.com/webinstall/webi-installers/internal/releases/gpgdist"
"github.com/webinstall/webi-installers/internal/releases/hashicorp"
"github.com/webinstall/webi-installers/internal/releases/iterm2dist"
"github.com/webinstall/webi-installers/internal/releases/juliadist"
"github.com/webinstall/webi-installers/internal/releases/mariadbdist"
"github.com/webinstall/webi-installers/internal/releases/nodedist"
"github.com/webinstall/webi-installers/internal/releases/servicemandist"
"github.com/webinstall/webi-installers/internal/releases/zigdist"
"github.com/webinstall/webi-installers/internal/storage"
"github.com/webinstall/webi-installers/internal/storage/fsstore"
"github.com/webinstall/webi-installers/internal/storage/pgstore"
)
var (
name = "webicached"
version = "0.0.0-dev"
commit = "0000000"
date = "0001-01-01"
licenseYear = "2024"
licenseOwner = "AJ ONeal"
licenseType = "MPL-2.0"
)
func printVersion(w io.Writer) {
b_ver := strings.TrimPrefix(version, "v")
_, _ = fmt.Fprintf(w, "%s v%s %s (%s)\n", name, b_ver, commit[:7], date)
_, _ = fmt.Fprintf(w, "Copyright (C) %s %s\n", licenseYear, licenseOwner)
_, _ = fmt.Fprintf(w, "Licensed under %s\n", licenseType)
}
type MainConfig struct {
envFile string
confDir string
cacheDir string
pgDSN string
rawDir string
token string
once bool
noFetch bool
shallow bool
eager bool
interval time.Duration
pageDelay time.Duration
}
// WebiCache holds the configuration for the cache daemon.
type WebiCache struct {
ConfDir string // root directory with {pkg}/releases.conf files
Store storage.Store // classified asset storage (fsstore or pgstore)
RawDir string // raw upstream response cache
Client *http.Client // HTTP client for upstream calls
Auth *githubish.Auth // GitHub API auth (optional)
Shallow bool // fetch only the first page of releases
NoFetch bool // skip fetching, classify from existing raw data only
PageDelay time.Duration // delay between paginated API requests
}
// delayTransport wraps an http.RoundTripper to add a delay between requests.
type delayTransport struct {
base http.RoundTripper
delay time.Duration
last time.Time
}
func (t *delayTransport) RoundTrip(req *http.Request) (*http.Response, error) {
if !t.last.IsZero() && t.delay > 0 {
if wait := t.delay - time.Since(t.last); wait > 0 {
time.Sleep(wait)
}
}
t.last = time.Now()
return t.base.RoundTrip(req)
}
func main() {
if len(os.Args) > 1 {
switch os.Args[1] {
case "-V", "-version", "--version", "version":
printVersion(os.Stdout)
os.Exit(0)
case "help", "-help", "--help":
printVersion(os.Stdout)
fmt.Fprintln(os.Stdout, "")
fs := flag.NewFlagSet(os.Args[0], flag.ContinueOnError)
fs.SetOutput(os.Stdout)
registerFlags(fs, &MainConfig{})
fs.Usage()
os.Exit(0)
}
}
cfg := MainConfig{}
fs := flag.NewFlagSet(os.Args[0], flag.ContinueOnError)
registerFlags(fs, &cfg)
if err := fs.Parse(os.Args[1:]); err != nil {
if errors.Is(err, flag.ErrHelp) {
os.Exit(0)
}
os.Exit(1)
}
cfg.cacheDir = expandHome(cfg.cacheDir)
cfg.rawDir = expandHome(cfg.rawDir)
if cfg.envFile != "" {
if err := godotenv.Load(cfg.envFile); err != nil {
log.Fatalf("envfile: %v", err)
}
}
if cfg.token == "" {
cfg.token = os.Getenv("GITHUB_TOKEN")
}
var store storage.Store
if cfg.pgDSN != "" {
pg, err := pgstore.New(context.Background(), cfg.pgDSN)
if err != nil {
log.Fatalf("pgstore: %v", err)
}
store = pg
} else {
fs, err := fsstore.New(cfg.cacheDir)
if err != nil {
log.Fatalf("fsstore: %v", err)
}
store = fs
}
var auth *githubish.Auth
if cfg.token != "" {
auth = &githubish.Auth{Token: cfg.token}
}
client := &http.Client{Timeout: 30 * time.Second}
if cfg.pageDelay > 0 {
client.Transport = &delayTransport{
base: http.DefaultTransport,
delay: cfg.pageDelay,
}
}
wc := &WebiCache{
ConfDir: cfg.confDir,
Store: store,
RawDir: cfg.rawDir,
Client: client,
Auth: auth,
Shallow: cfg.shallow,
NoFetch: cfg.noFetch,
PageDelay: cfg.pageDelay,
}
filterPkgs := fs.Args()
if cfg.eager {
wc.Run(filterPkgs)
if cfg.once {
return
}
} else if cfg.once {
wc.Run(filterPkgs)
return
} else {
saved := wc.NoFetch
wc.NoFetch = true
wc.Run(filterPkgs)
wc.NoFetch = saved
}
packages, err := discover(wc.ConfDir)
if err != nil {
log.Fatalf("discover: %v", err)
}
if len(filterPkgs) > 0 {
nameSet := make(map[string]bool, len(filterPkgs))
for _, a := range filterPkgs {
nameSet[a] = true
}
var filtered []pkgConf
for _, p := range packages {
if nameSet[p.name] {
filtered = append(filtered, p)
}
}
packages = filtered
}
var real []pkgConf
for _, pkg := range packages {
if pkg.conf.AliasOf == "" {
real = append(real, pkg)
}
}
log.Printf("refreshing %d packages, interval %s, batch size 20 (ctrl-c to stop)", len(real), cfg.interval)
for {
stale := wc.stalest(real)
if len(stale) == 0 {
log.Printf("all packages fresh, sleeping %s", cfg.interval)
time.Sleep(cfg.interval)
continue
}
batch := stale
if len(batch) > 20 {
batch = batch[:20]
}
rand.Shuffle(len(batch), func(i, j int) {
batch[i], batch[j] = batch[j], batch[i]
})
log.Printf("batch: %d stale, refreshing %d (most stale first)", len(stale), len(batch))
for _, pkg := range batch {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
if err := wc.refreshPackage(ctx, pkg); err != nil {
log.Printf(" ERROR %s: %v", pkg.name, err)
}
cancel()
time.Sleep(cfg.interval)
}
}
}
func registerFlags(fs *flag.FlagSet, cfg *MainConfig) {
fs.StringVar(&cfg.envFile, "envfile", "", "path to .env file to load before running")
fs.StringVar(&cfg.confDir, "conf", ".", "root directory containing {pkg}/releases.conf files")
fs.StringVar(&cfg.cacheDir, "legacy", "~/.cache/webi/legacy", "legacy cache directory (fsstore root)")
fs.StringVar(&cfg.pgDSN, "pg", "", "PostgreSQL DSN (enables pgstore; mutually exclusive with -legacy)")
fs.StringVar(&cfg.rawDir, "raw", "~/.cache/webi/raw", "raw cache directory for upstream responses")
fs.StringVar(&cfg.token, "token", "", "GitHub API token (or set $GITHUB_TOKEN)")
fs.BoolVar(&cfg.once, "once", false, "run once then exit (no periodic refresh)")
fs.BoolVar(&cfg.noFetch, "no-fetch", false, "skip fetching, classify from existing raw data only")
fs.BoolVar(&cfg.shallow, "shallow", false, "fetch only the first page of releases (latest)")
fs.BoolVar(&cfg.eager, "eager", false, "fetch all packages on startup (default: one per tick)")
fs.DurationVar(&cfg.interval, "interval", 9*time.Second, "delay between individual package fetches")
fs.DurationVar(&cfg.pageDelay, "page-delay", 2*time.Second, "delay between paginated API requests")
}
func expandHome(path string) string {
if !strings.HasPrefix(path, "~/") {
return path
}
home, err := os.UserHomeDir()
if err != nil {
return path
}
return filepath.Join(home, path[2:])
}
// stalest returns packages sorted by most stale first (oldest UpdatedAt).
// Packages with no cache entry or empty assets are considered most stale.
func (wc *WebiCache) stalest(packages []pkgConf) []pkgConf {
type stamped struct {
pkg pkgConf
updatedAt time.Time
}
var stale []stamped
ctx := context.Background()
for _, pkg := range packages {
data, err := wc.Store.Load(ctx, pkg.name)
var t time.Time
hasAssets := false
if err == nil && data != nil {
t = data.UpdatedAt
hasAssets = len(data.Assets) > 0
}
// Never fetched, or has no assets despite having a timestamp
// (e.g. classified from empty rawcache), or older than 10 minutes.
if t.IsZero() || !hasAssets || time.Since(t) > 10*time.Minute {
stale = append(stale, stamped{pkg: pkg, updatedAt: t})
}
}
sort.SliceStable(stale, func(i, j int) bool {
ti, tj := stale[i].updatedAt, stale[j].updatedAt
if ti.Equal(tj) {
return stale[i].pkg.name < stale[j].pkg.name
}
return ti.Before(tj)
})
result := make([]pkgConf, len(stale))
for i, s := range stale {
result[i] = s.pkg
}
return result
}
// Run discovers packages and refreshes each one.
func (wc *WebiCache) Run(filterPkgs []string) {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
defer cancel()
packages, err := discover(wc.ConfDir)
if err != nil {
log.Printf("discover: %v", err)
return
}
if len(filterPkgs) > 0 {
nameSet := make(map[string]bool, len(filterPkgs))
for _, a := range filterPkgs {
nameSet[a] = true
}
var filtered []pkgConf
for _, p := range packages {
if nameSet[p.name] {
filtered = append(filtered, p)
}
}
packages = filtered
}
var real []pkgConf
for _, pkg := range packages {
if pkg.conf.AliasOf != "" {
continue
}
real = append(real, pkg)
}
log.Printf("refreshing %d packages", len(real))
runStart := time.Now()
for _, pkg := range real {
if err := wc.refreshPackage(ctx, pkg); err != nil {
log.Printf(" ERROR %s: %v", pkg.name, err)
}
}
log.Printf("refreshed %d packages in %s", len(real), time.Since(runStart))
}
type pkgConf struct {
name string
conf *installerconf.Conf
}
func discover(dir string) ([]pkgConf, error) {
pattern := filepath.Join(dir, "*", "releases.conf")
matches, err := filepath.Glob(pattern)
if err != nil {
return nil, err
}
var packages []pkgConf
for _, path := range matches {
pkgDir := filepath.Dir(path)
name := filepath.Base(pkgDir)
if strings.HasPrefix(name, "_") {
continue
}
// If the package directory is a symlink, treat it as an alias
// of the symlink target (e.g. rust.vim → vim-rust).
fi, err := os.Lstat(filepath.Join(dir, name))
if err != nil {
log.Printf("warning: %s: %v", name, err)
continue
}
if fi.Mode()&os.ModeSymlink != 0 {
target, err := os.Readlink(filepath.Join(dir, name))
if err != nil {
log.Printf("warning: readlink %s: %v", name, err)
continue
}
packages = append(packages, pkgConf{
name: name,
conf: &installerconf.Conf{AliasOf: target},
})
continue
}
conf, err := installerconf.Read(path)
if err != nil {
log.Printf("warning: %s: %v", path, err)
continue
}
packages = append(packages, pkgConf{name: name, conf: conf})
}
sort.Slice(packages, func(i, j int) bool {
return packages[i].name < packages[j].name
})
return packages, nil
}
// refreshPackage does the full pipeline for one package:
// fetch raw → classify → write to fsstore.
func (wc *WebiCache) refreshPackage(ctx context.Context, pkg pkgConf) error {
pkgStart := time.Now()
name := pkg.name
conf := pkg.conf
// Step 1: Fetch raw upstream data to rawcache (unless -no-fetch).
if !wc.NoFetch {
shallow := wc.Shallow
if !shallow {
d, err := rawcache.Open(filepath.Join(wc.RawDir, name))
if err == nil && d.Populated() {
shallow = true
}
}
fetchStart := time.Now()
if err := wc.fetchRaw(ctx, pkg, shallow); err != nil {
return fmt.Errorf("fetch: %w", err)
}
log.Printf(" %s: fetch %s", name, time.Since(fetchStart))
}
// Step 2: Classify raw data into assets, tag variants, apply config.
classifyStart := time.Now()
d, err := rawcache.Open(filepath.Join(wc.RawDir, name))
if err != nil {
return fmt.Errorf("rawcache open: %w", err)
}
// Open supplementary gittag raw cache if available (for packages with
// git_url that use a non-gittag source type like servicemandist).
var gitTagDir *rawcache.Dir
if conf.GitURL != "" && conf.Source != "gittag" {
gd, gdErr := rawcache.Open(filepath.Join(wc.RawDir, "_gittag", name))
if gdErr == nil && gd.Populated() {
gitTagDir = gd
}
}
assets, err := classifypkg.Package(name, conf, d, gitTagDir)
if err != nil {
return fmt.Errorf("classify: %w", err)
}
classifyDur := time.Since(classifyStart)
// Step 3: Write to fsstore.
writeStart := time.Now()
tx, err := wc.Store.BeginRefresh(ctx, name)
if err != nil {
return fmt.Errorf("begin refresh: %w", err)
}
if err := tx.Put(assets); err != nil {
tx.Rollback()
return fmt.Errorf("put: %w", err)
}
if err := tx.Commit(ctx); err != nil {
return fmt.Errorf("commit: %w", err)
}
writeDur := time.Since(writeStart)
log.Printf(" %s: %d assets (classify %s, write %s, total %s)",
name, len(assets), classifyDur, writeDur, time.Since(pkgStart))
return nil
}
// --- Fetch raw ---
func (wc *WebiCache) fetchRaw(ctx context.Context, pkg pkgConf, shallow bool) error {
switch pkg.conf.Source {
case "github", "githubsource":
if err := wc.fetchGitHub(ctx, pkg.name, pkg.conf, shallow); err != nil {
return err
}
case "nodedist":
return wc.fetchNodeDist(ctx, pkg.name, pkg.conf)
case "gittag":
return wc.fetchGitTag(ctx, pkg.name, pkg.conf, shallow)
case "gitea":
return wc.fetchGitea(ctx, pkg.name, pkg.conf, shallow)
case "chromedist":
return fetchChromeDist(ctx, wc.Client, wc.RawDir, pkg.name)
case "flutterdist":
return fetchFlutterDist(ctx, wc.Client, wc.RawDir, pkg.name)
case "golang":
return fetchGolang(ctx, wc.Client, wc.RawDir, pkg.name)
case "gpgdist":
return fetchGPGDist(ctx, wc.Client, wc.RawDir, pkg.name)
case "hashicorp":
return fetchHashiCorp(ctx, wc.Client, wc.RawDir, pkg.name, pkg.conf)
case "iterm2dist":
return fetchITerm2Dist(ctx, wc.Client, wc.RawDir, pkg.name)
case "juliadist":
return fetchJuliaDist(ctx, wc.Client, wc.RawDir, pkg.name)
case "mariadbdist":
return fetchMariaDBDist(ctx, wc.Client, wc.RawDir, pkg.name)
case "servicemandist":
if err := servicemandist.Fetch(ctx, wc.Client, wc.RawDir, pkg.name, wc.Auth, shallow); err != nil {
return err
}
case "zigdist":
return fetchZigDist(ctx, wc.Client, wc.RawDir, pkg.name)
default:
log.Printf(" %s: source %q not yet supported, skipping", pkg.name, pkg.conf.Source)
return nil
}
// For non-gittag sources with a git_url, also clone the repo to get
// commit hashes. Git entries are classified from this data in
// refreshPackage, not from the main raw cache.
if pkg.conf.GitURL != "" && pkg.conf.Source != "gittag" {
gitShallow := shallow
if !wc.Shallow {
gd, gdErr := rawcache.Open(filepath.Join(wc.RawDir, "_gittag", pkg.name))
if gdErr == nil && !gd.Populated() {
gitShallow = false
}
}
if err := wc.fetchGitTagSupplementary(ctx, pkg.name, pkg.conf.GitURL, gitShallow); err != nil {
log.Printf(" %s: supplementary gittag fetch: %v", pkg.name, err)
}
}
return nil
}
// fetchGitTagSupplementary clones a git repo to get commit hashes for
// packages that use a non-gittag source type (servicemandist, githubsource)
// but also have a git_url for source installs.
func (wc *WebiCache) fetchGitTagSupplementary(ctx context.Context, pkgName, gitURL string, shallow bool) error {
d, err := rawcache.Open(filepath.Join(wc.RawDir, "_gittag", pkgName))
if err != nil {
return err
}
repoDir := filepath.Join(wc.RawDir, "_repos")
os.MkdirAll(repoDir, 0o755)
for batch, err := range gittag.Fetch(ctx, gitURL, repoDir) {
if err != nil {
return err
}
for _, entry := range batch {
tag := entry.Version
if tag == "" {
tag = "HEAD-" + entry.CommitHash
}
data, _ := json.Marshal(entry)
d.Merge(tag, data)
}
if shallow {
break
}
}
return nil
}
func (wc *WebiCache) fetchGitHub(ctx context.Context, pkgName string, conf *installerconf.Conf, shallow bool) error {
owner, repo := conf.Owner, conf.Repo
if owner == "" || repo == "" {
return fmt.Errorf("missing owner or repo")
}
d, err := rawcache.Open(filepath.Join(wc.RawDir, pkgName))
if err != nil {
return err
}
tagPrefix := conf.TagPrefix
for batch, err := range github.Fetch(ctx, wc.Client, owner, repo, wc.Auth) {
if err != nil {
return fmt.Errorf("github %s/%s: %w", owner, repo, err)
}
for _, rel := range batch {
if rel.Draft {
continue
}
tag := rel.TagName
if tagPrefix != "" && !strings.HasPrefix(tag, tagPrefix) {
continue
}
data, _ := json.Marshal(rel)
d.Merge(tag, data)
}
if shallow {
break
}
}
return nil
}
func (wc *WebiCache) fetchNodeDist(ctx context.Context, pkgName string, conf *installerconf.Conf) error {
baseURL := conf.BaseURL
if baseURL == "" {
return fmt.Errorf("missing url")
}
d, err := rawcache.Open(filepath.Join(wc.RawDir, pkgName))
if err != nil {
return err
}
// Fetch from primary URL. Tag with "official/" prefix so unofficial
// entries for the same version don't overwrite.
for batch, err := range nodedist.Fetch(ctx, wc.Client, baseURL) {
if err != nil {
return err
}
for _, entry := range batch {
data, _ := json.Marshal(entry)
d.Merge("official/"+entry.Version, data)
}
}
// Fetch from unofficial URL if configured (e.g. Node.js unofficial builds
// which add musl, riscv64, loong64 targets).
if unofficialURL := conf.Extra["unofficial_url"]; unofficialURL != "" {
for batch, err := range nodedist.Fetch(ctx, wc.Client, unofficialURL) {
if err != nil {
log.Printf("warning: %s unofficial fetch: %v", pkgName, err)
break
}
for _, entry := range batch {
data, _ := json.Marshal(entry)
d.Merge("unofficial/"+entry.Version, data)
}
}
}
return nil
}
func (wc *WebiCache) fetchGitTag(ctx context.Context, pkgName string, conf *installerconf.Conf, shallow bool) error {
gitURL := conf.BaseURL
if gitURL == "" {
return fmt.Errorf("missing url")
}
d, err := rawcache.Open(filepath.Join(wc.RawDir, pkgName))
if err != nil {
return err
}
repoDir := filepath.Join(wc.RawDir, "_repos")
os.MkdirAll(repoDir, 0o755)
for batch, err := range gittag.Fetch(ctx, gitURL, repoDir) {
if err != nil {
return err
}
for _, entry := range batch {
tag := entry.Version
if tag == "" {
tag = "HEAD-" + entry.CommitHash
}
data, _ := json.Marshal(entry)
d.Merge(tag, data)
}
if shallow {
break
}
}
return nil
}
func (wc *WebiCache) fetchGitea(ctx context.Context, pkgName string, conf *installerconf.Conf, shallow bool) error {
baseURL, owner, repo := conf.BaseURL, conf.Owner, conf.Repo
if baseURL == "" || owner == "" || repo == "" {
return fmt.Errorf("missing base_url, owner, or repo")
}
d, err := rawcache.Open(filepath.Join(wc.RawDir, pkgName))
if err != nil {
return err
}
for batch, err := range gitea.Fetch(ctx, wc.Client, baseURL, owner, repo, nil) {
if err != nil {
return err
}
for _, rel := range batch {
if rel.Draft {
continue
}
data, _ := json.Marshal(rel)
d.Merge(rel.TagName, data)
}
if shallow {
break
}
}
return nil
}
func fetchChromeDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
if err != nil {
return err
}
for batch, err := range chromedist.Fetch(ctx, client) {
if err != nil {
return fmt.Errorf("chromedist: %w", err)
}
for _, ver := range batch {
data, _ := json.Marshal(ver)
d.Merge(ver.Version, data)
}
}
return nil
}
func fetchFlutterDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
if err != nil {
return err
}
for batch, err := range flutterdist.Fetch(ctx, client) {
if err != nil {
return fmt.Errorf("flutterdist: %w", err)
}
for _, rel := range batch {
// Key by version+channel+os for uniqueness.
key := rel.Version + "-" + rel.Channel + "-" + rel.OS
data, _ := json.Marshal(rel)
d.Merge(key, data)
}
}
return nil
}
func fetchGolang(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
if err != nil {
return err
}
for batch, err := range golang.Fetch(ctx, client) {
if err != nil {
return fmt.Errorf("golang: %w", err)
}
for _, rel := range batch {
data, _ := json.Marshal(rel)
d.Merge(rel.Version, data)
}
}
return nil
}
func fetchGPGDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
if err != nil {
return err
}
for batch, err := range gpgdist.Fetch(ctx, client) {
if err != nil {
return fmt.Errorf("gpgdist: %w", err)
}
for _, entry := range batch {
data, _ := json.Marshal(entry)
d.Merge(entry.Version, data)
}
}
return nil
}
func fetchHashiCorp(ctx context.Context, client *http.Client, rawDir, pkgName string, conf *installerconf.Conf) error {
product := conf.Repo
if product == "" {
product = pkgName
}
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
if err != nil {
return err
}
for idx, err := range hashicorp.Fetch(ctx, client, product) {
if err != nil {
return fmt.Errorf("hashicorp %s: %w", product, err)
}
for ver, vdata := range idx.Versions {
data, _ := json.Marshal(vdata)
d.Merge(ver, data)
}
}
return nil
}
func fetchITerm2Dist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
if err != nil {
return err
}
for batch, err := range iterm2dist.Fetch(ctx, client) {
if err != nil {
return fmt.Errorf("iterm2dist: %w", err)
}
for _, entry := range batch {
key := entry.Version
if entry.Channel == "beta" {
key += "-beta"
}
data, _ := json.Marshal(entry)
d.Merge(key, data)
}
}
return nil
}
func fetchJuliaDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
if err != nil {
return err
}
for batch, err := range juliadist.Fetch(ctx, client) {
if err != nil {
return fmt.Errorf("juliadist: %w", err)
}
for _, rel := range batch {
data, _ := json.Marshal(rel)
d.Merge(rel.Version, data)
}
}
return nil
}
func fetchMariaDBDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
if err != nil {
return err
}
for batch, err := range mariadbdist.Fetch(ctx, client) {
if err != nil {
return fmt.Errorf("mariadbdist: %w", err)
}
for _, rel := range batch {
data, _ := json.Marshal(rel)
d.Merge(rel.ReleaseID, data)
}
}
return nil
}
func fetchZigDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
if err != nil {
return err
}
for batch, err := range zigdist.Fetch(ctx, client) {
if err != nil {
return fmt.Errorf("zigdist: %w", err)
}
for _, rel := range batch {
data, _ := json.Marshal(rel)
d.Merge(rel.Version, data)
}
}
return nil
}

121
docs/installer-patterns.md Normal file
View File

@@ -0,0 +1,121 @@
# Installer Archive Patterns
Every package falls into one of these archive structure patterns. When writing
or modifying an `install.sh`, identify the pattern first — it determines the
extraction and installation strategy.
## Pattern A: Bare Binary in Archive
Archive contains the binary (and maybe LICENSE/README) at the top level.
Examples: awless, caddy, cilium, curlie, dashmsg, deno, dotenv, dotenv-linter,
ffuf, fzf, gitdeploy, gprox, grype, hugo, hugo-extended, k9s, keypairs, koji,
lf, monorel, ots, runzip, sclient, sqlc, sqlpkg, sttr, terraform, uuidv7, xcaddy
Install: extract, move binary to `~/.local/opt/{pkg}-{ver}/bin/{binary}`, symlink.
## Pattern B: Subdirectory with Binary Only
Archive contains a version-named directory wrapping the binary and docs.
Examples: delta, hexyl, kubectx, kubens, shellcheck, trip, xsv
Typical directory naming: `{tool}-{ver}-{triplet}/`
Install: extract, find binary in subdirectory, move to opt, symlink.
Special cases:
- `pathman`: bare binary named with full release tag (needs rename)
- `yq`: binary named with platform suffix `yq_linux_amd64` (needs rename)
## Pattern C: Binary + Completions + Man Pages
Archive includes shell completions and/or man pages alongside the binary.
| Package | Completions Dir | Man Page |
|---------|----------------|----------|
| bat | `autocomplete/` | `bat.1` |
| fd | `autocomplete/{fd.bash,.fish,_fd}` | `fd.1` |
| goreleaser | `completions/{.bash,.fish,.zsh}` | `manpages/*.1.gz` |
| lsd | `autocomplete/{lsd.bash-completion,.fish,_lsd}` | `lsd.1` |
| rg | `complete/{rg.bash,.fish,_rg}` | `doc/rg.1` |
| sd | `completions/{sd.bash,.fish,_sd}` | `sd.1` |
| watchexec | `completions/{bash,fish,zsh}` | `watchexec.1` |
| zoxide | `completions/{zoxide.bash,.fish,_zoxide}` | `man/man1/zoxide*.1` |
Install: extract, install binary, install completions to standard dirs, install
man pages. Completion naming varies: `autocomplete/`, `completions/`, `complete/`.
## Pattern D: Binary + Libraries
Complex packages that bundle shared libraries.
| Package | Layout |
|---------|--------|
| ollama (Linux) | `bin/ollama` + `lib/ollama/{cuda_v12,cuda_v13,vulkan}/` |
| pg/postgres/psql | `bin/psql` + `lib/{libpq,libz,...}.so` + `include/` |
| sass | `dart-sass/sass` (wrapper) + `dart-sass/src/{dart,sass.snapshot}` |
| syncthing | `syncthing-{triplet}-{ver}/syncthing` + `etc/{systemd,...}/` |
| xz | `xz-{ver}-{triplet}/xz` + `xz-{ver}-{triplet}/unxz` |
Install: extract entire directory tree into opt, symlink binary.
## Pattern E: FHS-like Layout (bin/ + share/)
Archive already follows standard layout.
| Package | Layout |
|---------|--------|
| gh | `gh_{ver}_{os}_{arch}/bin/gh` + `share/man/man1/*.1` |
| pandoc | `pandoc-{ver}/bin/{pandoc,...}` + `share/man/man1/*.1.gz` |
Install: extract directly into opt (already correct layout).
## Pattern G: Full SDK/Toolchain
Self-contained toolchain with compiler, runtime, standard library.
| Package | Layout |
|---------|--------|
| cmake | `cmake-{ver}-{os}-{arch}/bin/{cmake,ctest,...}` + `share/` + `man/` |
| tinygo | `tinygo/bin/tinygo` + `tinygo/src/` + `tinygo/targets/` |
| go | `go/bin/{go,gofmt}` + `go/src/` + `go/pkg/` |
| zig | `zig-{os}-{arch}-{ver}/zig` + `lib/` |
| flutter | `flutter/bin/flutter` + full SDK |
| julia | `julia-{ver}/bin/julia` + full SDK |
| node | `node-{ver}-{os}-{arch}/bin/{node,npm,npx}` + `lib/` |
Install: extract entire tree into `~/.local/opt/{pkg}-{ver}/`, symlink `bin/*`.
## Pattern H: .NET Runtime Bundle
Flat archive with hundreds of DLLs.
Example: pwsh — `pwsh` binary + `*.dll` + locale dirs
Install: extract entire directory into opt, symlink primary binary.
## Pattern I: Multi-Binary Distribution
Archive contains multiple related binaries + libs.
| Package | Layout |
|---------|--------|
| dashcore | `dashcore-{ver}/bin/{dashd,dash-cli,...}` + `lib/` + `share/man/` |
| mutagen | `mutagen` + `mutagen-agents.tar.gz` (embedded agent archive) |
Install: extract into opt, symlink primary binary.
## Format Changes Over Time
Most packages have stable formats. Notable structural changes:
| Package | When | Change |
|---------|------|--------|
| sd | 2023 | zip → tar.gz, added completions + man page |
| ollama | 2025-2026 | bare binary → no GitHub release → tar.zst with lib/ |
| deno | 2020-2021 | .gz (gzipped binary) → .zip |
| hugo | 2017-2018 | zip → tar.gz; 2024: macOS → .pkg only |
| gh | 2024 | darwin: tar.gz → .pkg |
| sclient | 2023 | tar.gz → tar.xz |
| watchexec | 2019-2020 | tar.gz → tar.xz |

74
docs/version-oddities.md Normal file
View File

@@ -0,0 +1,74 @@
# Version & Release Oddities
Non-standard version formats and tag prefixes that affect parsing, sorting,
and classification. The Go classifier and `internal/lexver` must handle all
of these.
## Non-Numeric Tag Prefixes
| Package | Raw Tag | Cleaned | Transform |
|---------|---------|---------|-----------|
| lf | `r21` | `0.21.0` | `r` prefix → prepend `0.` |
| bun | `bun-v1.0.0` | `1.0.0` | Strip `bun-` prefix |
| jq | `jq-1.7` | `1.7` | Strip `jq-` prefix |
| watchexec | `cli-v1.2.3` | `1.2.3` | Strip `cli-` prefix |
| ffmpeg | `b6.0` | `6.0` | Strip `b` prefix |
## Underscore-Delimited Tags
| Package | Raw Tag | Cleaned | Transform |
|---------|---------|---------|-----------|
| postgres | `REL_17_0` | `17.0` | Strip `REL_`, replace `_` with `.` |
| psql | `REL_17_0` | `17.0` | Same as postgres |
## Platform Suffix in Version
| Package | Raw Tag | Cleaned | Transform |
|---------|---------|---------|-----------|
| git (Windows) | `2.41.0.windows.1` | `2.41.0` | Strip `.windows.N` suffix |
## 4-Part Versions
| Package | Example | Notes |
|---------|---------|-------|
| chromedriver | `121.0.6120.0` | Google Chrome's versioning |
| gpg | `2.2.19.0` | 4th segment is build metadata |
## Date-Based Versions
| Package | Notes |
|---------|-------|
| atomicparsley | Date-based version strings |
## Complex Pre-Release Formats
| Package | Example | Notes |
|---------|---------|-------|
| flutter | `2.3.0-16.0.pre` | Extra dots and numeric segments |
| iterm2 | `iTerm2_3_5_0beta17` | Underscores, beta attached → `3.5.0-beta17` |
## Channel Detection
- Node.js: odd major = "current" not LTS (v15, v17, v19, v21, v23)
- Go: `go` prefix stripped (`go1.23.6``1.23.6`)
- Terraform: `-alpha`, `-beta`, `-rc` suffixes → beta channel
## Directory Symlinks (Aliases)
These are directory-level symlinks. They share all files (including
releases.conf) with their target automatically.
```
msvc-runtime → vcruntime
msvcruntime → vcruntime
rust.vim → vim-rust
vc-redist → vcruntime
vc-runtime → vcruntime
vc_redist → vcruntime
vcredist → vcruntime
vcruntime140 → vcruntime
vim-essential → vim-essentials
vim-mouse → vim-gui
vps-myip → myip
xcode-cli → commandlinetools
```

15
go.mod Normal file
View File

@@ -0,0 +1,15 @@
module github.com/webinstall/webi-installers
go 1.26.1
require (
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/pgx/v5 v5.8.0 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/joho/godotenv v1.5.1 // indirect
github.com/jszwec/csvutil v1.10.0 // indirect
github.com/therootcompany/golib/http/middleware/v2 v2.0.1 // indirect
golang.org/x/sync v0.17.0 // indirect
golang.org/x/text v0.29.0 // indirect
)

25
go.sum Normal file
View File

@@ -0,0 +1,25 @@
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.8.0 h1:TYPDoleBBme0xGSAX3/+NujXXtpZn9HBONkQC7IEZSo=
github.com/jackc/pgx/v5 v5.8.0/go.mod h1:QVeDInX2m9VyzvNeiCJVjCkNFqzsNb43204HshNSZKw=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
github.com/jszwec/csvutil v1.10.0 h1:upMDUxhQKqZ5ZDCs/wy+8Kib8rZR8I8lOR34yJkdqhI=
github.com/jszwec/csvutil v1.10.0/go.mod h1:/E4ONrmGkwmWsk9ae9jpXnv9QT8pLHEPcCirMFhxG9I=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/therootcompany/golib/http/middleware/v2 v2.0.1 h1:VNKpHcwyEW7cMct7/eO4fyrxwIQk2ycb6juVXSPs2Sk=
github.com/therootcompany/golib/http/middleware/v2 v2.0.1/go.mod h1:g5gb9qBidw74nW6/mwIauTKMpOKchiN2l0gt5qzJ2aQ=
golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug=
golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk=
golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

View File

@@ -0,0 +1,168 @@
// Package buildmeta is the shared vocabulary for Webi's build targets.
//
// Every package that deals with OS, architecture, libc, archive format, or
// release channel imports these types instead of passing raw strings. This
// prevents typos like "darwn" from compiling and gives a single place to
// enumerate what Webi supports.
package buildmeta
// OS represents a target operating system.
type OS string
const (
OSAny OS = "ANYOS"
OSDarwin OS = "darwin"
OSLinux OS = "linux"
OSWindows OS = "windows"
OSFreeBSD OS = "freebsd"
OSOpenBSD OS = "openbsd"
OSNetBSD OS = "netbsd"
OSDragonFly OS = "dragonfly"
OSSunOS OS = "sunos"
OSIllumos OS = "illumos"
OSSolaris OS = "solaris"
OSAIX OS = "aix"
OSAndroid OS = "android"
OSPlan9 OS = "plan9"
// POSIX compatibility levels — used when a package is a shell script
// or otherwise OS-independent for POSIX systems.
OSPosix2017 OS = "posix_2017"
OSPosix2024 OS = "posix_2024"
)
// Arch represents a target CPU architecture.
type Arch string
const (
ArchAny Arch = "ANYARCH"
ArchAMD64 Arch = "x86_64" // baseline (v1)
ArchAMD64v2 Arch = "x86_64_v2" // +SSE4, +POPCNT, etc.
ArchAMD64v3 Arch = "x86_64_v3" // +AVX2, +BMI, etc.
ArchAMD64v4 Arch = "x86_64_v4" // +AVX-512
ArchARM64 Arch = "aarch64"
ArchARMv7 Arch = "armv7"
ArchARMv6 Arch = "armv6"
ArchARMv5 Arch = "armv5"
ArchX86 Arch = "x86"
ArchPPC64LE Arch = "ppc64le"
ArchPPC64 Arch = "ppc64"
ArchPPC Arch = "powerpc" // 32-bit PowerPC (unsupported by webi, used to prevent gnueabihf over-matching)
ArchRISCV64 Arch = "riscv64"
ArchS390X Arch = "s390x"
ArchLoong64 Arch = "loong64"
ArchMIPS64LE Arch = "mips64le"
ArchMIPS64 Arch = "mips64"
ArchMIPS64R6EL Arch = "mips64r6el"
ArchMIPS64R6 Arch = "mips64r6"
ArchMIPSLE Arch = "mipsle"
ArchMIPS Arch = "mips"
// Universal (fat) binary architectures for macOS.
ArchUniversal1 Arch = "universal1" // PPC + x86 (Rosetta 1 era)
ArchUniversal2 Arch = "universal2" // x86_64 + ARM64 (Rosetta 2 era)
)
// Libc represents the C library a binary is linked against.
type Libc string
const (
LibcNone Libc = "none" // statically linked or no libc dependency (Go, Zig, etc.)
LibcGNU Libc = "gnu" // requires glibc (most Linux distros)
LibcMusl Libc = "musl" // requires musl (Alpine, some Docker images)
LibcMSVC Libc = "msvc" // Microsoft Visual C++ runtime
)
// Format represents an archive or package format.
type Format string
const (
FormatTarGz Format = ".tar.gz"
FormatTarXz Format = ".tar.xz"
FormatTarZst Format = ".tar.zst"
FormatTarBz2 Format = ".tar.bz2"
FormatZip Format = ".zip"
FormatGz Format = ".gz"
FormatXz Format = ".xz"
FormatZst Format = ".zst"
FormatExe Format = ".exe"
FormatExeXz Format = ".exe.xz"
FormatMSI Format = ".msi"
FormatDMG Format = ".dmg"
FormatPkg Format = ".pkg"
FormatAppZip Format = ".app.zip"
Format7z Format = ".7z"
FormatDeb Format = ".deb"
FormatRPM Format = ".rpm"
FormatSnap Format = ".snap"
FormatAppx Format = ".appx"
FormatAPK Format = ".apk"
FormatAppImage Format = ".AppImage"
FormatSh Format = ".sh"
FormatGit Format = ".git"
)
// Channel represents a release stability channel.
type Channel string
const (
ChannelStable Channel = "stable"
ChannelLatest Channel = "latest"
ChannelRC Channel = "rc"
ChannelPreview Channel = "preview"
ChannelBeta Channel = "beta"
ChannelAlpha Channel = "alpha"
ChannelDev Channel = "dev"
)
// Target represents a fully resolved build target.
type Target struct {
OS OS
Arch Arch
Libc Libc
}
// Triplet returns the canonical "os-arch-libc" string.
func (t Target) Triplet() string {
return string(t.OS) + "-" + string(t.Arch) + "-" + string(t.Libc)
}
// CompatArches returns the architectures that the given OS+arch
// combination can execute, ordered from most specific to least.
// The input arch is always first.
//
// These are OS-level facts (hardware + translation layer), not
// package-specific. Per-package overrides belong in installer config.
func CompatArches(os OS, arch Arch) []Arch {
switch os {
case OSDarwin:
switch arch {
case ArchARM64:
// Rosetta 2: Apple Silicon runs x86_64 binaries.
return []Arch{ArchARM64, ArchUniversal2, ArchAMD64}
case ArchAMD64:
return []Arch{ArchAMD64, ArchUniversal2, ArchX86}
}
case OSWindows:
switch arch {
case ArchARM64:
// Windows on ARM emulates x86_64 and x86.
return []Arch{ArchARM64, ArchAMD64, ArchX86}
}
}
// Micro-architecture fallbacks (universal across all OSes).
switch arch {
case ArchAMD64v4:
return []Arch{ArchAMD64v4, ArchAMD64v3, ArchAMD64v2, ArchAMD64}
case ArchAMD64v3:
return []Arch{ArchAMD64v3, ArchAMD64v2, ArchAMD64}
case ArchAMD64v2:
return []Arch{ArchAMD64v2, ArchAMD64}
case ArchARMv7:
return []Arch{ArchARMv7, ArchARMv6}
}
return []Arch{arch}
}

View File

@@ -0,0 +1,283 @@
// Package classify extracts build targets from release asset filenames.
//
// Standard toolchains (goreleaser, cargo-dist, zig build) produce predictable
// filenames like "tool_0.1.0_linux_amd64.tar.gz" or
// "tool-0.1.0-x86_64-unknown-linux-musl.tar.gz". This package matches those
// patterns directly using regex, avoiding heuristic guessing.
//
// Detection order matters: architectures are checked longest-first to prevent
// "x86" from matching inside "x86_64", and OS checks use word boundaries.
package classify
import (
"path"
"regexp"
"strings"
"github.com/webinstall/webi-installers/internal/buildmeta"
)
// Result holds the classification of an asset filename.
type Result struct {
OS buildmeta.OS
Arch buildmeta.Arch
Libc buildmeta.Libc
Format buildmeta.Format
}
// Target returns the build target (OS + Arch + Libc).
func (r Result) Target() buildmeta.Target {
return buildmeta.Target{OS: r.OS, Arch: r.Arch, Libc: r.Libc}
}
// Filename classifies a release asset filename, returning the detected
// OS, architecture, libc, and archive format. Undetected fields are empty.
//
// OS is detected first because it can influence arch interpretation.
// For example, "windows-arm" in modern releases means ARM64, while
// bare "arm" on Linux historically means ARMv6.
func Filename(name string) Result {
lower := strings.ToLower(name)
os := detectOS(lower)
arch := detectArch(lower)
format := detectFormat(lower)
// .deb, .rpm, .snap are Linux-only package formats.
if os == "" && (format == buildmeta.FormatDeb || format == buildmeta.FormatRPM || format == buildmeta.FormatSnap) {
os = buildmeta.OSLinux
}
// .app.zip and .dmg are macOS-only formats.
if os == "" && (format == buildmeta.FormatAppZip || format == buildmeta.FormatDMG) {
os = buildmeta.OSDarwin
}
return Result{
OS: os,
Arch: arch,
Libc: detectLibc(lower),
Format: format,
}
}
// b is a boundary: start/end of string or a non-alphanumeric separator.
// Go's RE2 doesn't support \b, so we use this instead.
const b = `(?:^|[^a-zA-Z0-9])`
const bEnd = `(?:[^a-zA-Z0-9]|$)`
// --- OS detection ---
var osPatterns = []struct {
os buildmeta.OS
pattern *regexp.Regexp
}{
// macos[\d.]* matches versioned names like "macos10.10", "macos11", "macos12.0" (cmake naming).
{buildmeta.OSDarwin, regexp.MustCompile(`(?i)(?:` + b + `(?:darwin|macos[\d.]*|macosx[\d.]*|osx[\d.]*|os-x|apple)` + bEnd + `|` + b + `mac` + bEnd + `)`)},
// linux[\d.]* matches versioned names like "linux64", "linux32" (chromedriver/dashcore naming).
{buildmeta.OSLinux, regexp.MustCompile(`(?i)` + b + `linux[\d.]*` + bEnd)},
{buildmeta.OSWindows, regexp.MustCompile(`(?i)` + b + `(?:windows|win(?:32|64|x64|dows)?)` + bEnd + `|\.exe(?:\.xz)?$|\.msi$`)},
// freebsd\d* matches versioned names like "freebsd13", "freebsd14" (Gitea naming).
{buildmeta.OSFreeBSD, regexp.MustCompile(`(?i)` + b + `freebsd\d*` + bEnd)},
{buildmeta.OSOpenBSD, regexp.MustCompile(`(?i)` + b + `openbsd` + bEnd)},
{buildmeta.OSNetBSD, regexp.MustCompile(`(?i)` + b + `netbsd` + bEnd)},
{buildmeta.OSDragonFly, regexp.MustCompile(`(?i)` + b + `dragonfly(?:bsd)?` + bEnd)},
// solaris, illumos, and sunos are distinct OS values in the Node build-classifier.
// Keep them separate so the legacy cache matches what the classifier extracts.
{buildmeta.OSSolaris, regexp.MustCompile(`(?i)` + b + `solaris` + bEnd)},
{buildmeta.OSIllumos, regexp.MustCompile(`(?i)` + b + `illumos` + bEnd)},
{buildmeta.OSSunOS, regexp.MustCompile(`(?i)` + b + `sunos` + bEnd)},
{buildmeta.OSAIX, regexp.MustCompile(`(?i)` + b + `aix` + bEnd)},
{buildmeta.OSAndroid, regexp.MustCompile(`(?i)` + b + `android` + bEnd)},
{buildmeta.OSPlan9, regexp.MustCompile(`(?i)` + b + `plan9` + bEnd)},
}
func detectOS(lower string) buildmeta.OS {
for _, p := range osPatterns {
if p.pattern.MatchString(lower) {
return p.os
}
}
return ""
}
// --- Arch detection ---
// Order matters: check longer/more-specific patterns first.
var archPatterns = []struct {
arch buildmeta.Arch
pattern *regexp.Regexp
}{
// Universal/fat binaries before specific arches.
{buildmeta.ArchUniversal2, regexp.MustCompile(`(?i)` + b + `(?:universal2?|fat)` + bEnd)},
// amd64 micro-levels before baseline — "amd64v3" must not fall through to amd64.
// amd64_?vN: underscore optional but no dash — dash is ambiguous with version numbers
// (e.g. syncthing "amd64-v2.0.5" where v2 is the release version, not an arch level).
{buildmeta.ArchAMD64v4, regexp.MustCompile(`(?i)(?:x86[_-]64[_-]v4|amd64_?v4|v4-amd64)`)},
{buildmeta.ArchAMD64v3, regexp.MustCompile(`(?i)(?:x86[_-]64[_-]v3|amd64_?v3|v3-amd64)`)},
{buildmeta.ArchAMD64v2, regexp.MustCompile(`(?i)(?:x86[_-]64[_-]v2|amd64_?v2|v2-amd64)`)},
// amd64 baseline before x86 — "x86_64" must not match as x86.
{buildmeta.ArchAMD64, regexp.MustCompile(`(?i)(?:x86[_-]64|amd64|x64|win64)`)},
// arm64 before armv7/armv6 — "aarch64" must not match as arm.
{buildmeta.ArchARM64, regexp.MustCompile(`(?i)(?:aarch64|arm64|armv8)`)},
{buildmeta.ArchARMv7, regexp.MustCompile(`(?i)(?:armv7l?|arm-?v7|arm7|arm32|armhf)`)},
// armel and gnueabihf are ARMv6 soft/hard-float ABI names used in Debian and Rust triplets.
{buildmeta.ArchARMv6, regexp.MustCompile(`(?i)(?:armv6l?|arm-?v6|aarch32|armel|gnueabihf|` + b + `arm` + bEnd + `)`)},
{buildmeta.ArchARMv5, regexp.MustCompile(`(?i)(?:armv5)`)},
// powerpc64le/ppc64le before powerpc64/ppc64 before powerpc32.
// The longer powerpc* forms must come first to prevent shorter matches from
// winning. All powerpc entries must appear BEFORE ARM patterns — otherwise
// "powerpc-linux-gnueabihf" would match gnueabihf → ARMv6.
// ppc64el is an alternative spelling used in Debian/Ubuntu.
{buildmeta.ArchPPC64LE, regexp.MustCompile(`(?i)(?:powerpc64le|ppc64le|ppc64el)`)},
{buildmeta.ArchPPC64, regexp.MustCompile(`(?i)(?:powerpc64|ppc64)`)},
// powerpc (32-bit): webi does not serve powerpc32, but we must classify it
// here to prevent the gnueabihf suffix from matching the ARMv6 pattern.
{buildmeta.ArchPPC, regexp.MustCompile(`(?i)` + b + `powerpc` + bEnd)},
{buildmeta.ArchRISCV64, regexp.MustCompile(`(?i)riscv64`)},
{buildmeta.ArchS390X, regexp.MustCompile(`(?i)s390x`)},
{buildmeta.ArchLoong64, regexp.MustCompile(`(?i)loong(?:arch)?64`)},
// mips64r6 before mips64 — "mips64r6" contains "mips64" as a prefix.
{buildmeta.ArchMIPS64R6EL, regexp.MustCompile(`(?i)mips64r6e(?:l|le)`)},
{buildmeta.ArchMIPS64R6, regexp.MustCompile(`(?i)mips64r6`)},
{buildmeta.ArchMIPS64LE, regexp.MustCompile(`(?i)mips64(?:el|le)`)},
{buildmeta.ArchMIPS64, regexp.MustCompile(`(?i)mips64`)},
{buildmeta.ArchMIPSLE, regexp.MustCompile(`(?i)mips(?:el|le)`)},
{buildmeta.ArchMIPS, regexp.MustCompile(`(?i)` + b + `mips` + bEnd)},
// x86 last — must not steal x86_64.
{buildmeta.ArchX86, regexp.MustCompile(`(?i)(?:` + b + `x86` + bEnd + `|i[3-6]86|ia32|win32|` + b + `386` + bEnd + `)`)},
}
func detectArch(lower string) buildmeta.Arch {
for _, p := range archPatterns {
if p.pattern.MatchString(lower) {
return p.arch
}
}
return ""
}
// --- Libc detection ---
var (
reMusl = regexp.MustCompile(`(?i)` + b + `musl` + bEnd)
reGNU = regexp.MustCompile(`(?i)` + b + `(?:gnu|glibc)` + bEnd)
reMSVC = regexp.MustCompile(`(?i)` + b + `msvc` + bEnd)
reStatic = regexp.MustCompile(`(?i)` + b + `static` + bEnd)
)
func detectLibc(lower string) buildmeta.Libc {
switch {
case reMusl.MatchString(lower):
return buildmeta.LibcMusl
case reGNU.MatchString(lower):
return buildmeta.LibcGNU
case reMSVC.MatchString(lower):
return buildmeta.LibcMSVC
case reStatic.MatchString(lower):
return buildmeta.LibcNone
}
return ""
}
// --- Format detection ---
// formatSuffixes maps file extensions to formats, longest first.
var formatSuffixes = []struct {
suffix string
format buildmeta.Format
}{
{".tar.gz", buildmeta.FormatTarGz},
{".tar.xz", buildmeta.FormatTarXz},
{".tar.zst", buildmeta.FormatTarZst},
{".tar.bz2", buildmeta.FormatTarBz2},
{".exe.xz", buildmeta.FormatExeXz},
{".app.zip", buildmeta.FormatAppZip},
{".tgz", buildmeta.FormatTarGz},
{".zip", buildmeta.FormatZip},
{".gz", buildmeta.FormatGz},
{".xz", buildmeta.FormatXz},
{".zst", buildmeta.FormatZst},
{".7z", buildmeta.Format7z},
{".exe", buildmeta.FormatExe},
{".msi", buildmeta.FormatMSI},
{".dmg", buildmeta.FormatDMG},
{".deb", buildmeta.FormatDeb},
{".rpm", buildmeta.FormatRPM},
{".snap", buildmeta.FormatSnap},
{".appx", buildmeta.FormatAppx},
{".apk", buildmeta.FormatAPK},
{".AppImage", buildmeta.FormatAppImage},
{".pkg", buildmeta.FormatPkg},
}
func detectFormat(lower string) buildmeta.Format {
// Use the base name to avoid directory separators confusing suffix matching.
base := path.Base(lower)
for _, s := range formatSuffixes {
if strings.HasSuffix(base, s.suffix) {
return s.format
}
}
return ""
}
// IsMetaAsset returns true if the filename is a non-installable meta file
// (checksums, signatures, source tarballs, documentation, etc.).
func IsMetaAsset(name string) bool {
lower := strings.ToLower(name)
for _, suffix := range []string{
".txt",
".sha256",
".sha256sum",
".sha512",
".sha512sum",
".md5",
".md5sum",
".sig",
".asc",
".pem",
".sbom",
".spdx",
".json.sig",
".sigstore",
".minisig",
"_src.tar.gz",
"_src.tar.xz",
"_src.zip",
"-src.tar.gz",
".src.tar.gz",
"-src.tar.xz",
"-src.zip",
".d.ts",
".pub",
".bsdiff",
".flatpak",
} {
if strings.HasSuffix(lower, suffix) {
return true
}
}
for _, substr := range []string{
"checksums",
"sha256sum",
"sha512sum",
"buildable-artifact",
".LICENSE",
".README",
} {
if strings.Contains(lower, substr) {
return true
}
}
for _, exact := range []string{
"install.sh",
"install.ps1",
"compat.json",
"b3sums",
"dist-manifest.json",
} {
if lower == exact {
return true
}
}
return false
}

View File

@@ -0,0 +1,352 @@
package classify_test
import (
"testing"
"github.com/webinstall/webi-installers/internal/buildmeta"
"github.com/webinstall/webi-installers/internal/classify"
)
func TestFilename(t *testing.T) {
tests := []struct {
name string
input string
wantOS buildmeta.OS
arch buildmeta.Arch
libc buildmeta.Libc
format buildmeta.Format
}{
// Goreleaser-style
{
name: "goreleaser linux amd64 tar.gz",
input: "hugo_0.145.0_linux-amd64.tar.gz",
wantOS: buildmeta.OSLinux,
arch: buildmeta.ArchAMD64,
format: buildmeta.FormatTarGz,
},
{
name: "goreleaser darwin arm64 tar.gz",
input: "hugo_0.145.0_darwin-arm64.tar.gz",
wantOS: buildmeta.OSDarwin,
arch: buildmeta.ArchARM64,
format: buildmeta.FormatTarGz,
},
{
name: "goreleaser windows amd64 zip",
input: "hugo_0.145.0_windows-amd64.zip",
wantOS: buildmeta.OSWindows,
arch: buildmeta.ArchAMD64,
format: buildmeta.FormatZip,
},
{
name: "goreleaser freebsd",
input: "hugo_0.145.0_freebsd-amd64.tar.gz",
wantOS: buildmeta.OSFreeBSD,
arch: buildmeta.ArchAMD64,
format: buildmeta.FormatTarGz,
},
// Rust/cargo-dist style
{
name: "rust linux musl",
input: "ripgrep-14.1.1-x86_64-unknown-linux-musl.tar.gz",
wantOS: buildmeta.OSLinux,
arch: buildmeta.ArchAMD64,
libc: buildmeta.LibcMusl,
format: buildmeta.FormatTarGz,
},
{
name: "rust linux gnu",
input: "bat-v0.24.0-x86_64-unknown-linux-gnu.tar.gz",
wantOS: buildmeta.OSLinux,
arch: buildmeta.ArchAMD64,
libc: buildmeta.LibcGNU,
format: buildmeta.FormatTarGz,
},
{
name: "rust apple darwin",
input: "ripgrep-14.1.1-x86_64-apple-darwin.tar.gz",
wantOS: buildmeta.OSDarwin,
arch: buildmeta.ArchAMD64,
format: buildmeta.FormatTarGz,
},
{
name: "rust windows msvc",
input: "bat-v0.24.0-x86_64-pc-windows-msvc.zip",
wantOS: buildmeta.OSWindows,
arch: buildmeta.ArchAMD64,
libc: buildmeta.LibcMSVC,
format: buildmeta.FormatZip,
},
{
name: "rust aarch64 linux",
input: "ripgrep-14.1.1-aarch64-unknown-linux-gnu.tar.gz",
wantOS: buildmeta.OSLinux,
arch: buildmeta.ArchARM64,
libc: buildmeta.LibcGNU,
format: buildmeta.FormatTarGz,
},
// Zig-style
{
name: "zig linux x86_64",
input: "zig-linux-x86_64-0.14.0.tar.xz",
wantOS: buildmeta.OSLinux,
arch: buildmeta.ArchAMD64,
format: buildmeta.FormatTarXz,
},
{
name: "zig macos aarch64",
input: "zig-macos-aarch64-0.14.0.tar.xz",
wantOS: buildmeta.OSDarwin,
arch: buildmeta.ArchARM64,
format: buildmeta.FormatTarXz,
},
// Windows executables
{
name: "bare exe",
input: "jq-windows-amd64.exe",
wantOS: buildmeta.OSWindows,
arch: buildmeta.ArchAMD64,
format: buildmeta.FormatExe,
},
{
name: "msi installer",
input: "caddy_2.9.0_windows_amd64.msi",
wantOS: buildmeta.OSWindows,
arch: buildmeta.ArchAMD64,
format: buildmeta.FormatMSI,
},
// macOS formats
{
name: "dmg installer",
input: "MyApp-1.0.0-darwin-arm64.dmg",
wantOS: buildmeta.OSDarwin,
arch: buildmeta.ArchARM64,
format: buildmeta.FormatDMG,
},
// Arch priority: x86_64 must not match x86
{
name: "x86_64 not x86",
input: "tool-x86_64-linux.tar.gz",
wantOS: buildmeta.OSLinux,
arch: buildmeta.ArchAMD64,
format: buildmeta.FormatTarGz,
},
{
name: "actual x86",
input: "tool-x86-linux.tar.gz",
wantOS: buildmeta.OSLinux,
arch: buildmeta.ArchX86,
format: buildmeta.FormatTarGz,
},
{
name: "i386",
input: "tool-linux-i386.tar.gz",
wantOS: buildmeta.OSLinux,
arch: buildmeta.ArchX86,
format: buildmeta.FormatTarGz,
},
// Windows ARM: bare "arm" is armv6 (some tools ship genuine arm32 Windows builds).
// Explicit "arm64" is always aarch64 regardless of OS.
{
name: "windows bare arm stays armv6",
input: "tool-1.0.0-windows-arm.zip",
wantOS: buildmeta.OSWindows,
arch: buildmeta.ArchARMv6,
format: buildmeta.FormatZip,
},
{
name: "windows armv6 stays armv6",
input: "tool-2.0.0-windows-armv6.zip",
wantOS: buildmeta.OSWindows,
arch: buildmeta.ArchARMv6,
format: buildmeta.FormatZip,
},
{
name: "windows arm64 stays arm64",
input: "tool-1.0.0-windows-arm64.zip",
wantOS: buildmeta.OSWindows,
arch: buildmeta.ArchARM64,
format: buildmeta.FormatZip,
},
// armel and gnueabihf are ARMv6 ABI names
{
name: "armel is armv6",
input: "jq-linux-armel",
wantOS: buildmeta.OSLinux,
arch: buildmeta.ArchARMv6,
},
{
name: "gnueabihf is armv6",
input: "tool-arm-unknown-linux-gnueabihf.tar.gz",
wantOS: buildmeta.OSLinux,
arch: buildmeta.ArchARMv6,
format: buildmeta.FormatTarGz,
},
// winx64 is a Windows x86_64 naming used by MariaDB
{
name: "winx64 is windows x86_64",
input: "mariadb-11.4.5-winx64.zip",
wantOS: buildmeta.OSWindows,
arch: buildmeta.ArchAMD64,
format: buildmeta.FormatZip,
},
// win32/win64 naming used by chromedriver, dashcore, etc.
{
name: "win32 is windows x86",
input: "chromedriver-win32.zip",
wantOS: buildmeta.OSWindows,
arch: buildmeta.ArchX86,
format: buildmeta.FormatZip,
},
{
name: "win64 is windows amd64",
input: "dashcore-23.1.2-win64-setup.exe",
wantOS: buildmeta.OSWindows,
arch: buildmeta.ArchAMD64,
format: buildmeta.FormatExe,
},
// ppc64el is a Debian/Ubuntu alias for ppc64le
{
name: "ppc64el is ppc64le",
input: "jq-linux-ppc64el",
arch: buildmeta.ArchPPC64LE,
},
// amd64 micro-architecture levels
{
name: "amd64v2",
input: "tool-linux-amd64v2.tar.gz",
arch: buildmeta.ArchAMD64v2,
},
{
name: "amd64v3",
input: "tool-linux-x86_64_v3.tar.gz",
arch: buildmeta.ArchAMD64v3,
},
{
name: "amd64v4",
input: "tool-linux-amd64v4.tar.gz",
arch: buildmeta.ArchAMD64v4,
},
{
name: "amd64v3 not baseline",
input: "tool-1.0.0-linux-amd64v3.tar.gz",
wantOS: buildmeta.OSLinux,
arch: buildmeta.ArchAMD64v3,
format: buildmeta.FormatTarGz,
},
// ARM variants: arm64 must not match armv7/armv6
{
name: "aarch64 not armv7",
input: "tool-aarch64-linux.tar.gz",
arch: buildmeta.ArchARM64,
},
{
name: "armv7",
input: "tool-armv7l-linux.tar.gz",
arch: buildmeta.ArchARMv7,
},
{
name: "armv6",
input: "tool-armv6l-linux.tar.gz",
arch: buildmeta.ArchARMv6,
},
// ppc64le before ppc64
{
name: "ppc64le",
input: "tool-linux-ppc64le.tar.gz",
arch: buildmeta.ArchPPC64LE,
},
{
name: "ppc64",
input: "tool-linux-ppc64.tar.gz",
arch: buildmeta.ArchPPC64,
},
// Static linking
{
name: "static binary",
input: "tool-linux-amd64-static.tar.gz",
libc: buildmeta.LibcNone,
},
// .exe implies Windows
{
name: "exe implies windows",
input: "tool-amd64.exe",
wantOS: buildmeta.OSWindows,
arch: buildmeta.ArchAMD64,
format: buildmeta.FormatExe,
},
// Compound extensions
{
name: "tar.zst",
input: "tool-linux-amd64.tar.zst",
format: buildmeta.FormatTarZst,
},
{
name: "exe.xz",
input: "tool-windows-amd64.exe.xz",
format: buildmeta.FormatExeXz,
},
{
name: "app.zip",
input: "MyApp-1.0.0.app.zip",
format: buildmeta.FormatAppZip,
},
{
name: "tgz alias",
input: "tool-linux-amd64.tgz",
format: buildmeta.FormatTarGz,
},
// s390x, mips
{
name: "s390x",
input: "tool-linux-s390x.tar.gz",
arch: buildmeta.ArchS390X,
},
{
name: "mips64",
input: "tool-linux-mips64.tar.gz",
arch: buildmeta.ArchMIPS64,
},
// Unknown / no match
{
name: "checksum file",
input: "checksums.txt",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := classify.Filename(tt.input)
if tt.wantOS != "" && got.OS != tt.wantOS {
t.Errorf("OS = %q, want %q", got.OS, tt.wantOS)
}
if tt.arch != "" && got.Arch != tt.arch {
t.Errorf("Arch = %q, want %q", got.Arch, tt.arch)
}
if tt.libc != "" && got.Libc != tt.libc {
t.Errorf("Libc = %q, want %q", got.Libc, tt.libc)
}
if tt.format != "" && got.Format != tt.format {
t.Errorf("Format = %q, want %q", got.Format, tt.format)
}
})
}
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,154 @@
// Package httpclient provides a well-configured [http.Client] for upstream
// API calls. It exists because [http.DefaultClient] has no timeouts, no TLS
// minimum, and follows redirects from HTTPS to HTTP — none of which are
// acceptable for a server calling GitHub, Gitea, etc. on behalf of users.
//
// Use [New] to create a configured client. Use [Do] to execute a request
// with automatic retries for transient failures.
package httpclient
import (
"context"
"crypto/tls"
"errors"
"fmt"
"math/rand/v2"
"net"
"net/http"
"strconv"
"time"
)
const userAgent = "Webi/2.0 (+https://webinstall.dev)"
// New returns an [http.Client] with secure, production-ready defaults:
// TLS 1.2+, timeouts at every level, connection pooling, no HTTPS→HTTP
// redirect, and a Webi User-Agent.
func New() *http.Client {
return &http.Client{
Transport: &http.Transport{
DialContext: (&net.Dialer{
Timeout: 10 * time.Second,
KeepAlive: 30 * time.Second,
}).DialContext,
TLSClientConfig: &tls.Config{
MinVersion: tls.VersionTLS12,
},
TLSHandshakeTimeout: 10 * time.Second,
ResponseHeaderTimeout: 30 * time.Second,
MaxIdleConns: 100,
MaxIdleConnsPerHost: 10,
IdleConnTimeout: 90 * time.Second,
ExpectContinueTimeout: 1 * time.Second,
ForceAttemptHTTP2: true,
},
Timeout: 60 * time.Second,
CheckRedirect: checkRedirect,
}
}
// checkRedirect prevents HTTPS→HTTP downgrades and limits redirect depth.
func checkRedirect(req *http.Request, via []*http.Request) error {
if len(via) >= 10 {
return fmt.Errorf("stopped after %d redirects", len(via))
}
if len(via) > 0 && via[0].URL.Scheme == "https" && req.URL.Scheme == "http" {
return errors.New("refused redirect from https to http")
}
return nil
}
// Get performs a GET request with the Webi User-Agent header.
func Get(ctx context.Context, client *http.Client, url string) (*http.Response, error) {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, err
}
req.Header.Set("User-Agent", userAgent)
return client.Do(req)
}
// Do executes a request with automatic retries for transient errors (429,
// 502, 503, 504). Retries up to 3 times with exponential backoff and jitter.
// Respects Retry-After headers. Only retries GET and HEAD (idempotent).
//
// Sets the Webi User-Agent header if not already present.
func Do(ctx context.Context, client *http.Client, req *http.Request) (*http.Response, error) {
if req.Header.Get("User-Agent") == "" {
req.Header.Set("User-Agent", userAgent)
}
// Only retry idempotent methods.
idempotent := req.Method == http.MethodGet || req.Method == http.MethodHead
const maxRetries = 3
var resp *http.Response
var err error
for attempt := range maxRetries + 1 {
if attempt > 0 {
if !idempotent {
break
}
delay := backoff(attempt, resp)
timer := time.NewTimer(delay)
select {
case <-ctx.Done():
timer.Stop()
return nil, ctx.Err()
case <-timer.C:
}
if resp != nil {
resp.Body.Close()
}
}
resp, err = client.Do(req)
if err != nil {
if ctx.Err() != nil {
return nil, ctx.Err()
}
continue
}
if !isRetryable(resp.StatusCode) {
return resp, nil
}
}
if err != nil {
return nil, fmt.Errorf("after %d retries: %w", maxRetries, err)
}
return resp, nil
}
func isRetryable(status int) bool {
return status == http.StatusTooManyRequests ||
status == http.StatusBadGateway ||
status == http.StatusServiceUnavailable ||
status == http.StatusGatewayTimeout
}
// backoff returns a delay before the next retry. Respects Retry-After,
// otherwise uses exponential backoff with jitter.
func backoff(attempt int, resp *http.Response) time.Duration {
if resp != nil {
if ra := resp.Header.Get("Retry-After"); ra != "" {
if seconds, err := strconv.Atoi(ra); err == nil && seconds > 0 && seconds < 300 {
return time.Duration(seconds) * time.Second
}
}
}
// 1s, 2s, 4s base delays
base := time.Second << (attempt - 1)
if base > 30*time.Second {
base = 30 * time.Second
}
// Add jitter: 75% to 125% of base
jitter := float64(base) * (0.75 + 0.5*rand.Float64())
return time.Duration(jitter)
}

View File

@@ -0,0 +1,286 @@
// Package installerconf reads per-package releases.conf files.
//
// The format is simple key=value, one per line. Blank lines and lines
// starting with # are ignored. Keys and values are trimmed of whitespace.
// Multi-value keys are whitespace-delimited.
//
// The source type is inferred from the primary key:
//
// GitHub binary releases:
//
// github_releases = sharkdp/bat
// github_releases = https://github.com/sharkdp/bat
//
// GitHub source archives (for source-installable packages):
//
// github_sources = BeyondCodeBootcamp/aliasman
// git_url = https://github.com/BeyondCodeBootcamp/aliasman.git
//
// Gitea binary releases (self-hosted, requires full URL or base_url):
//
// gitea_releases = https://git.rootprojects.org/root/pathman
//
// GitLab binary releases (defaults to gitlab.com):
//
// gitlab_releases = owner/repo
// gitlab_releases = https://gitlab.example.com/owner/repo
//
// Git tag enumeration (vim plugins, etc.):
//
// git_url = https://github.com/tpope/vim-commentary.git
//
// HashiCorp releases:
//
// hashicorp_product = terraform
//
// Other sources (one-off scrapers):
//
// source = nodedist
// url = https://nodejs.org/download/release
//
// Complex packages that need custom logic beyond what the classifier
// auto-detects (e.g. ollama's universal binaries, ffmpeg's non-standard
// naming) should put that logic in Go code, not in the config.
// The variants key documents known build variants for human readers;
// actual variant detection logic lives in Go.
package installerconf
import (
"bufio"
"fmt"
"net/url"
"os"
"strings"
)
// Conf holds the parsed per-package release configuration.
type Conf struct {
// Source is the fetch source type: "github", "githubsource",
// "gitea", "giteasource", "gitlab", "gitlabsource",
// "gittag", "nodedist", etc.
Source string
// Owner is the repository owner (org or user).
Owner string
// Repo is the repository name.
Repo string
// BaseURL is a custom base URL for non-GitHub sources
// (e.g. a Gitea instance or nodedist index URL).
BaseURL string
// GitURL is the git clone URL for source-installable packages.
// Present alongside github_sources/gitea_sources to provide a
// git clone fallback in addition to release tarballs.
GitURL string
// TagPrefix filters releases in monorepos. Only tags starting with
// this prefix are included, and the prefix is stripped from the
// version string. Example: "tools/monorel/"
TagPrefix string
// VersionPrefixes are stripped from version/tag strings.
// Whitespace-delimited. Each release tag is checked against these
// in order; the first match is stripped. Projects may change tag
// conventions across versions (e.g. "jq-1.7.1" older, "1.8.0" later).
VersionPrefixes []string
// Exclude lists filename substrings to filter out.
// Whitespace-delimited. Assets whose name contains any of these
// are skipped entirely (not stored).
Exclude []string
// AssetFilter is a substring that asset filenames must contain.
// Used when multiple packages share a GitHub release (e.g.
// kubectx/kubens) to select only the relevant assets.
AssetFilter string
// Variants documents known build variant names for this package.
// Whitespace-delimited. This is a human-readable cue — actual
// variant detection logic lives in Go code per-package.
Variants []string
// OS restricts all assets to this OS value when set.
// Use "posix_2017" for POSIX-only shell packages that don't
// support Windows.
OS string
// AliasOf names another package that this one mirrors.
// When set, the package has no releases of its own — it shares
// the cache output of the named target (e.g. dashd → dashcore).
AliasOf string
// Extra holds any unrecognized keys for forward compatibility.
Extra map[string]string
}
// parseRepoRef parses a value that is either "owner/repo" or a full URL
// like "https://github.com/owner/repo". Returns baseURL, owner, repo.
// For short form, baseURL is empty (caller uses the default for the forge).
// For full URL form, baseURL is the scheme+host (e.g. "https://github.com").
func parseRepoRef(val, defaultBase string) (baseURL, owner, repo string) {
if strings.Contains(val, "://") {
u, err := url.Parse(val)
if err == nil {
baseURL = u.Scheme + "://" + u.Host
path := strings.Trim(u.Path, "/")
owner, repo, _ = strings.Cut(path, "/")
return baseURL, owner, repo
}
}
// Short form: "owner/repo"
owner, repo, _ = strings.Cut(val, "/")
return defaultBase, owner, repo
}
// Read parses a releases.conf file.
func Read(path string) (*Conf, error) {
f, err := os.Open(path)
if err != nil {
return nil, fmt.Errorf("installerconf: %w", err)
}
defer f.Close()
raw := make(map[string]string)
scanner := bufio.NewScanner(f)
for scanner.Scan() {
line := strings.TrimSpace(scanner.Text())
if line == "" || line[0] == '#' {
continue
}
key, val, ok := strings.Cut(line, "=")
if !ok {
continue
}
raw[strings.TrimSpace(key)] = strings.TrimSpace(val)
}
if err := scanner.Err(); err != nil {
return nil, fmt.Errorf("installerconf: read %s: %w", path, err)
}
c := &Conf{}
// Infer source from primary key, falling back to explicit "source".
// When both github_releases and source are set, parse the repo ref
// from github_releases but use the explicit source for classification.
switch {
// GitHub binary releases.
case raw["github_releases"] != "":
c.Source = "github"
c.BaseURL, c.Owner, c.Repo = parseRepoRef(raw["github_releases"], "https://github.com")
// GitHub source tarballs.
case raw["github_sources"] != "":
c.Source = "githubsource"
c.BaseURL, c.Owner, c.Repo = parseRepoRef(raw["github_sources"], "https://github.com")
// Gitea binary releases (self-hosted only — requires full URL or base_url).
case raw["gitea_releases"] != "":
c.Source = "gitea"
c.BaseURL, c.Owner, c.Repo = parseRepoRef(raw["gitea_releases"], raw["base_url"])
// Gitea source tarballs (self-hosted only).
case raw["gitea_sources"] != "":
c.Source = "giteasource"
c.BaseURL, c.Owner, c.Repo = parseRepoRef(raw["gitea_sources"], raw["base_url"])
// GitLab binary releases (defaults to gitlab.com).
case raw["gitlab_releases"] != "":
c.Source = "gitlab"
c.BaseURL, c.Owner, c.Repo = parseRepoRef(raw["gitlab_releases"], "https://gitlab.com")
// GitLab source tarballs (defaults to gitlab.com).
case raw["gitlab_sources"] != "":
c.Source = "gitlabsource"
c.BaseURL, c.Owner, c.Repo = parseRepoRef(raw["gitlab_sources"], "https://gitlab.com")
// Explicit source type (servicemandist, nodedist, zigdist, etc.).
// Must come before git_url so that "source = X" + "git_url = ..."
// uses X as the primary source, not gittag.
case raw["source"] != "":
c.Source = raw["source"]
c.BaseURL = raw["url"]
// Git tag enumeration (only when no explicit source is set).
case raw["git_url"] != "":
c.Source = "gittag"
c.BaseURL = raw["git_url"]
// HashiCorp.
case raw["hashicorp_product"] != "":
c.Source = "hashicorp"
c.Repo = raw["hashicorp_product"]
default:
}
// Explicit "source" overrides the inferred source when both are present.
// This lets packages like ffmpeg use github_releases for fetching but
// a custom classifier for classification.
if raw["source"] != "" && c.Source != "" {
c.Source = raw["source"]
}
// git_url can appear alongside any source type (e.g. github_sources)
// to provide a git clone fallback. When it's the only key, it's the
// primary source (gittag).
c.GitURL = raw["git_url"]
c.TagPrefix = raw["tag_prefix"]
if v := raw["version_prefixes"]; v != "" {
c.VersionPrefixes = strings.Fields(v)
} else if v := raw["version_prefix"]; v != "" {
c.VersionPrefixes = strings.Fields(v)
}
// Accept both "exclude" and "asset_exclude" (back-compat).
if v := raw["exclude"]; v != "" {
c.Exclude = strings.Fields(v)
} else if v := raw["asset_exclude"]; v != "" {
c.Exclude = strings.Fields(v)
}
c.AssetFilter = raw["asset_filter"]
c.OS = raw["os"]
c.AliasOf = raw["alias_of"]
if v := raw["variants"]; v != "" {
c.Variants = strings.Fields(v)
}
// Collect unrecognized keys.
known := map[string]bool{
"source": true,
"github_releases": true,
"github_sources": true,
"gitea_releases": true,
"gitea_sources": true,
"gitlab_releases": true,
"gitlab_sources": true,
"git_url": true,
"hashicorp_product": true,
"base_url": true,
"url": true,
"tag_prefix": true,
"version_prefix": true,
"version_prefixes": true,
"exclude": true,
"asset_exclude": true,
"asset_filter": true,
"os": true,
"variants": true,
"alias_of": true,
}
for k, v := range raw {
if !known[k] {
if c.Extra == nil {
c.Extra = make(map[string]string)
}
c.Extra[k] = v
}
}
return c, nil
}

View File

@@ -0,0 +1,217 @@
package installerconf_test
import (
"os"
"path/filepath"
"testing"
"github.com/webinstall/webi-installers/internal/installerconf"
)
func TestGitHubReleases(t *testing.T) {
c := confFromString(t, `
github_releases = sharkdp/bat
`)
assertEqual(t, "Source", c.Source, "github")
assertEqual(t, "Owner", c.Owner, "sharkdp")
assertEqual(t, "Repo", c.Repo, "bat")
assertEqual(t, "BaseURL", c.BaseURL, "https://github.com")
assertEqual(t, "TagPrefix", c.TagPrefix, "")
if len(c.VersionPrefixes) != 0 {
t.Errorf("VersionPrefixes = %v, want empty", c.VersionPrefixes)
}
if len(c.Exclude) != 0 {
t.Errorf("Exclude = %v, want empty", c.Exclude)
}
}
func TestGitHubReleasesFullURL(t *testing.T) {
c := confFromString(t, `
github_releases = https://github.com/sharkdp/bat
`)
assertEqual(t, "Source", c.Source, "github")
assertEqual(t, "BaseURL", c.BaseURL, "https://github.com")
assertEqual(t, "Owner", c.Owner, "sharkdp")
assertEqual(t, "Repo", c.Repo, "bat")
}
func TestGitHubSources(t *testing.T) {
c := confFromString(t, `
github_sources = BeyondCodeBootcamp/aliasman
git_url = https://github.com/BeyondCodeBootcamp/aliasman.git
`)
assertEqual(t, "Source", c.Source, "githubsource")
assertEqual(t, "Owner", c.Owner, "BeyondCodeBootcamp")
assertEqual(t, "Repo", c.Repo, "aliasman")
assertEqual(t, "GitURL", c.GitURL, "https://github.com/BeyondCodeBootcamp/aliasman.git")
}
func TestGitHubSourcesFullURL(t *testing.T) {
c := confFromString(t, `
github_sources = https://github.com/BeyondCodeBootcamp/aliasman
git_url = https://github.com/BeyondCodeBootcamp/aliasman.git
`)
assertEqual(t, "Source", c.Source, "githubsource")
assertEqual(t, "BaseURL", c.BaseURL, "https://github.com")
assertEqual(t, "Owner", c.Owner, "BeyondCodeBootcamp")
assertEqual(t, "Repo", c.Repo, "aliasman")
}
func TestVersionPrefixes(t *testing.T) {
c := confFromString(t, `
github_releases = jqlang/jq
version_prefixes = jq- cli-
`)
if len(c.VersionPrefixes) != 2 {
t.Fatalf("VersionPrefixes has %d items, want 2: %v", len(c.VersionPrefixes), c.VersionPrefixes)
}
assertEqual(t, "VersionPrefixes[0]", c.VersionPrefixes[0], "jq-")
assertEqual(t, "VersionPrefixes[1]", c.VersionPrefixes[1], "cli-")
}
func TestExclude(t *testing.T) {
c := confFromString(t, `
github_releases = gohugoio/hugo
exclude = _extended_ Linux-64bit
`)
if len(c.Exclude) != 2 {
t.Fatalf("Exclude has %d items, want 2: %v", len(c.Exclude), c.Exclude)
}
assertEqual(t, "Exclude[0]", c.Exclude[0], "_extended_")
assertEqual(t, "Exclude[1]", c.Exclude[1], "Linux-64bit")
}
func TestMonorepoTagPrefix(t *testing.T) {
c := confFromString(t, `
github_releases = therootcompany/golib
tag_prefix = tools/monorel/
`)
assertEqual(t, "TagPrefix", c.TagPrefix, "tools/monorel/")
}
func TestNodeDist(t *testing.T) {
c := confFromString(t, `
source = nodedist
url = https://nodejs.org/download/release
`)
assertEqual(t, "Source", c.Source, "nodedist")
assertEqual(t, "BaseURL", c.BaseURL, "https://nodejs.org/download/release")
}
func TestGiteaReleases(t *testing.T) {
c := confFromString(t, `
gitea_releases = https://git.rootprojects.org/root/pathman
`)
assertEqual(t, "Source", c.Source, "gitea")
assertEqual(t, "BaseURL", c.BaseURL, "https://git.rootprojects.org")
assertEqual(t, "Owner", c.Owner, "root")
assertEqual(t, "Repo", c.Repo, "pathman")
}
func TestGiteaReleasesWithBaseURL(t *testing.T) {
c := confFromString(t, `
gitea_releases = root/pathman
base_url = https://git.rootprojects.org
`)
assertEqual(t, "Source", c.Source, "gitea")
assertEqual(t, "BaseURL", c.BaseURL, "https://git.rootprojects.org")
assertEqual(t, "Owner", c.Owner, "root")
assertEqual(t, "Repo", c.Repo, "pathman")
}
func TestGitLabReleases(t *testing.T) {
c := confFromString(t, `
gitlab_releases = owner/repo
`)
assertEqual(t, "Source", c.Source, "gitlab")
assertEqual(t, "BaseURL", c.BaseURL, "https://gitlab.com")
assertEqual(t, "Owner", c.Owner, "owner")
assertEqual(t, "Repo", c.Repo, "repo")
}
func TestGitLabReleasesFullURL(t *testing.T) {
c := confFromString(t, `
gitlab_releases = https://gitlab.example.com/myorg/myrepo
`)
assertEqual(t, "Source", c.Source, "gitlab")
assertEqual(t, "BaseURL", c.BaseURL, "https://gitlab.example.com")
assertEqual(t, "Owner", c.Owner, "myorg")
assertEqual(t, "Repo", c.Repo, "myrepo")
}
func TestBlanksAndComments(t *testing.T) {
c := confFromString(t, `
# Hugo config
github_releases = foo/bar
# exclude line
exclude = extended
`)
assertEqual(t, "Source", c.Source, "github")
assertEqual(t, "Owner", c.Owner, "foo")
assertEqual(t, "Repo", c.Repo, "bar")
}
func TestExtraKeys(t *testing.T) {
c := confFromString(t, `
github_releases = foo/bar
custom_thing = hello
`)
if c.Extra == nil || c.Extra["custom_thing"] != "hello" {
t.Errorf("Extra[custom_thing] = %q, want hello", c.Extra["custom_thing"])
}
}
func TestAssetExcludeAlias(t *testing.T) {
c := confFromString(t, `
github_releases = gohugoio/hugo
asset_exclude = extended
`)
if len(c.Exclude) != 1 {
t.Fatalf("Exclude has %d items, want 1: %v", len(c.Exclude), c.Exclude)
}
assertEqual(t, "Exclude[0]", c.Exclude[0], "extended")
}
func TestVariants(t *testing.T) {
c := confFromString(t, `
github_releases = jmorganca/ollama
variants = rocm jetpack5 jetpack6
`)
if len(c.Variants) != 3 {
t.Fatalf("Variants has %d items, want 3: %v", len(c.Variants), c.Variants)
}
assertEqual(t, "Variants[0]", c.Variants[0], "rocm")
assertEqual(t, "Variants[1]", c.Variants[1], "jetpack5")
assertEqual(t, "Variants[2]", c.Variants[2], "jetpack6")
}
func TestEmptyExclude(t *testing.T) {
c := confFromString(t, "github_releases = foo/bar\n")
if c.Exclude != nil {
t.Errorf("Exclude = %v, want nil", c.Exclude)
}
}
// helpers
func confFromString(t *testing.T, content string) *installerconf.Conf {
t.Helper()
dir := t.TempDir()
path := filepath.Join(dir, "releases.conf")
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
t.Fatal(err)
}
c, err := installerconf.Read(path)
if err != nil {
t.Fatal(err)
}
return c
}
func assertEqual(t *testing.T, name, got, want string) {
t.Helper()
if got != want {
t.Errorf("%s = %q, want %q", name, got, want)
}
}

189
internal/lexver/lexver.go Normal file
View File

@@ -0,0 +1,189 @@
// Package lexver makes version strings comparable and sortable.
//
// Not all version strings are semver. Webi handles 4-part versions
// (chromedriver 121.0.6120.0), date-based versions (atomicparsley),
// and pre-releases with extra dots (flutter 2.3.0-16.0.pre). Lexver
// parses these into a struct with an arbitrary-depth numeric segment
// list and provides a comparison function for use with [slices.SortFunc].
//
// Pre-releases sort before their corresponding stable release:
//
// 1.0.0-alpha1 < 1.0.0-beta1 < 1.0.0-rc1 < 1.0.0
//
// When release dates are known, they break ties between versions with
// identical numeric segments.
package lexver
import (
"cmp"
"strconv"
"strings"
"time"
"unicode"
)
// Version is a parsed version with comparable fields.
type Version struct {
// Nums holds the dotted numeric segments in order.
// "1.20.3" → [1, 20, 3], "121.0.6120.0" → [121, 0, 6120, 0].
Nums []int
Channel string // "" for stable, or "alpha", "beta", "dev", "pre", "preview", "rc"
ChannelNum int // e.g. 2 in "rc2"
Date time.Time // release date/time, if known; breaks ties between same-numbered versions
Original string // version string exactly as the releaser published it (e.g. "REL_17_0", "r21")
Raw string // version string after Webi's normalization (e.g. "17.0", "0.21.0")
// ExtraSort is an optional opaque string for package-specific ordering.
// Set by release-fetcher code for packages where Nums alone can't capture
// the sort order (e.g. flutter's "2.3.0-16.0.pre"). Compared as a plain
// string, only consulted when Nums and Channel are equal.
ExtraSort string
}
// Parse breaks a version string into its components.
// Both Original and Raw are set to s; callers that normalize versions
// (e.g. "REL_17_0" → "17.0") should set Original to the upstream tag
// and pass the normalized string to Parse.
func Parse(s string) Version {
v := Version{Original: s, Raw: s}
s = strings.TrimLeft(s, "vV")
numStr, prerelease := splitAtPrerelease(s)
v.Nums = splitNums(numStr)
if prerelease != "" {
v.Channel, v.ChannelNum = splitChannel(prerelease)
}
return v
}
// Major returns the first numeric segment, or 0 if none.
func (v Version) Major() int { return v.num(0) }
// Minor returns the second numeric segment, or 0 if none.
func (v Version) Minor() int { return v.num(1) }
// Patch returns the third numeric segment, or 0 if none.
func (v Version) Patch() int { return v.num(2) }
func (v Version) num(i int) int {
if i < len(v.Nums) {
return v.Nums[i]
}
return 0
}
// IsStable reports whether this is a stable (non-pre-release) version.
func (v Version) IsStable() bool {
return v.Channel == ""
}
// Compare returns -1, 0, or 1 for ordering two versions.
// Stable releases sort after pre-releases of the same numeric version.
func Compare(a, b Version) int {
// Compare numeric segments pairwise, treating missing segments as 0.
n := max(len(a.Nums), len(b.Nums))
for i := range n {
an, bn := a.num(i), b.num(i)
if c := cmp.Compare(an, bn); c != 0 {
return c
}
}
// Break ties with release date when both are known.
if !a.Date.IsZero() && !b.Date.IsZero() {
if c := a.Date.Compare(b.Date); c != 0 {
return c
}
}
// ExtraSort: package-specific tiebreaker set by release-fetcher code.
if a.ExtraSort != "" && b.ExtraSort != "" {
if c := cmp.Compare(a.ExtraSort, b.ExtraSort); c != 0 {
return c
}
}
// Both stable → equal.
if a.Channel == "" && b.Channel == "" {
return 0
}
// Stable beats any pre-release.
if a.Channel == "" {
return 1
}
if b.Channel == "" {
return -1
}
// Both pre-release: alphabetical channel, then number.
if c := cmp.Compare(a.Channel, b.Channel); c != 0 {
return c
}
return cmp.Compare(a.ChannelNum, b.ChannelNum)
}
// HasPrefix reports whether v matches a partial version prefix.
// A prefix with Nums [1, 20] matches any version starting with 1.20
// (e.g. 1.20.0, 1.20.3, 1.20.3.1).
func (v Version) HasPrefix(prefix Version) bool {
for i, pn := range prefix.Nums {
if i >= len(v.Nums) || v.Nums[i] != pn {
return false
}
}
return true
}
// splitAtPrerelease splits "1.20.3-beta1" into ("1.20.3", "beta1").
// Also handles "1.2beta3" (no separator).
func splitAtPrerelease(s string) (string, string) {
for _, sep := range []byte{'-', '+'} {
if idx := strings.IndexByte(s, sep); idx >= 0 {
return s[:idx], s[idx+1:]
}
}
// "1.2beta3": letter following a digit
for i := 1; i < len(s); i++ {
if unicode.IsLetter(rune(s[i])) && unicode.IsDigit(rune(s[i-1])) {
return s[:i], s[i:]
}
}
return s, ""
}
// splitNums parses "1.20.3" into [1, 20, 3].
// Handles any number of dot-separated segments.
func splitNums(s string) []int {
var nums []int
for _, seg := range strings.Split(s, ".") {
n, err := strconv.Atoi(seg)
if err != nil {
break
}
nums = append(nums, n)
}
return nums
}
// splitChannel separates "beta1" into ("beta", 1) or "rc" into ("rc", 0).
func splitChannel(s string) (string, int) {
s = strings.ToLower(s)
s = strings.NewReplacer("-", "", ".", "", "_", "").Replace(s)
i := len(s)
for i > 0 && unicode.IsDigit(rune(s[i-1])) {
i--
}
name := s[:i]
num := 0
if i < len(s) {
num, _ = strconv.Atoi(s[i:])
}
return name, num
}

View File

@@ -0,0 +1,270 @@
package lexver_test
import (
"slices"
"testing"
"time"
"github.com/webinstall/webi-installers/internal/lexver"
)
func TestParse(t *testing.T) {
tests := []struct {
input string
nums []int
channel string
chanNum int
}{
// Standard semver
{"1.0.0", []int{1, 0, 0}, "", 0},
{"v1.2.3", []int{1, 2, 3}, "", 0},
{"1.20.156", []int{1, 20, 156}, "", 0},
// Partial
{"1.20", []int{1, 20}, "", 0},
{"1", []int{1}, "", 0},
// 4-part (chromedriver, gpg)
{"121.0.6120.0", []int{121, 0, 6120, 0}, "", 0},
{"2.2.19.0", []int{2, 2, 19, 0}, "", 0},
// Pre-release
{"1.0.0-beta1", []int{1, 0, 0}, "beta", 1},
{"1.0.0-rc2", []int{1, 0, 0}, "rc", 2},
{"2.0.0-alpha3", []int{2, 0, 0}, "alpha", 3},
{"1.0.0-dev", []int{1, 0, 0}, "dev", 0},
// No separator before channel
{"1.2beta3", []int{1, 2}, "beta", 3},
{"1.0rc1", []int{1, 0}, "rc", 1},
}
for _, tt := range tests {
t.Run(tt.input, func(t *testing.T) {
v := lexver.Parse(tt.input)
if !slices.Equal(v.Nums, tt.nums) {
t.Errorf("Parse(%q).Nums = %v, want %v", tt.input, v.Nums, tt.nums)
}
if v.Channel != tt.channel || v.ChannelNum != tt.chanNum {
t.Errorf("Parse(%q) channel = %q/%d, want %q/%d",
tt.input, v.Channel, v.ChannelNum, tt.channel, tt.chanNum)
}
})
}
}
func TestAccessors(t *testing.T) {
v := lexver.Parse("121.0.6120.0")
if v.Major() != 121 || v.Minor() != 0 || v.Patch() != 6120 {
t.Errorf("got %d.%d.%d, want 121.0.6120", v.Major(), v.Minor(), v.Patch())
}
short := lexver.Parse("1")
if short.Minor() != 0 || short.Patch() != 0 {
t.Error("missing segments should return 0")
}
}
func TestSortOrder(t *testing.T) {
// Must be in ascending order.
ordered := []string{
"0.1.0",
"1.0.0-alpha1",
"1.0.0-alpha2",
"1.0.0-beta1",
"1.0.0-rc1",
"1.0.0-rc2",
"1.0.0",
"1.0.1",
"1.1.0",
"1.2.0",
"1.20.0",
"2.0.0-beta1",
"2.0.0",
}
for i := 1; i < len(ordered); i++ {
a := lexver.Parse(ordered[i-1])
b := lexver.Parse(ordered[i])
if lexver.Compare(a, b) >= 0 {
t.Errorf("expected %q < %q", ordered[i-1], ordered[i])
}
}
}
func TestSortOrder4Part(t *testing.T) {
ordered := []string{
"121.0.6120.0",
"121.0.6120.1",
"121.0.6121.0",
"122.0.6100.0",
}
for i := 1; i < len(ordered); i++ {
a := lexver.Parse(ordered[i-1])
b := lexver.Parse(ordered[i])
if lexver.Compare(a, b) >= 0 {
t.Errorf("expected %q < %q", ordered[i-1], ordered[i])
}
}
}
func TestMismatchedDepth(t *testing.T) {
// "1.0" and "1.0.0" should be equal (trailing zeros).
a := lexver.Parse("1.0")
b := lexver.Parse("1.0.0")
if lexver.Compare(a, b) != 0 {
t.Error("1.0 and 1.0.0 should be equal")
}
// "1.0.0.1" should be greater than "1.0.0".
c := lexver.Parse("1.0.0.1")
d := lexver.Parse("1.0.0")
if lexver.Compare(c, d) <= 0 {
t.Error("1.0.0.1 should be greater than 1.0.0")
}
}
func TestSortFunc(t *testing.T) {
versions := []string{"1.0.0", "2.0.0-rc1", "1.20.3", "1.20.2", "1.19.5", "2.0.0"}
parsed := make([]lexver.Version, len(versions))
for i, s := range versions {
parsed[i] = lexver.Parse(s)
}
// Sort descending (newest first).
slices.SortFunc(parsed, func(a, b lexver.Version) int {
return lexver.Compare(b, a)
})
want := []string{"2.0.0", "2.0.0-rc1", "1.20.3", "1.20.2", "1.19.5", "1.0.0"}
for i, v := range parsed {
if v.Raw != want[i] {
t.Errorf("index %d: got %q, want %q", i, v.Raw, want[i])
}
}
}
func TestIsStable(t *testing.T) {
tests := []struct {
input string
want bool
}{
{"1.0.0", true},
{"121.0.6120.0", true},
{"1.0.0-beta1", false},
{"v2.0.0-dev", false},
}
for _, tt := range tests {
t.Run(tt.input, func(t *testing.T) {
v := lexver.Parse(tt.input)
if v.IsStable() != tt.want {
t.Errorf("Parse(%q).IsStable() = %v, want %v", tt.input, v.IsStable(), tt.want)
}
})
}
}
func TestDateTiebreaker(t *testing.T) {
a := lexver.Parse("1.0.0")
a.Date = time.Date(2024, 1, 15, 0, 0, 0, 0, time.UTC)
b := lexver.Parse("1.0.0")
b.Date = time.Date(2024, 6, 1, 14, 30, 0, 0, time.UTC)
if lexver.Compare(a, b) >= 0 {
t.Error("earlier date should sort before later date at same version")
}
// Without dates, same version is equal.
c := lexver.Parse("1.0.0")
d := lexver.Parse("1.0.0")
if lexver.Compare(c, d) != 0 {
t.Error("same version without dates should be equal")
}
// Date only matters when both have it.
e := lexver.Parse("1.0.0")
e.Date = time.Date(2024, 1, 15, 0, 0, 0, 0, time.UTC)
f := lexver.Parse("1.0.0")
if lexver.Compare(e, f) != 0 {
t.Error("date should be ignored when only one side has it")
}
}
func TestDateMinutePrecision(t *testing.T) {
a := lexver.Parse("1.0.0")
a.Date = time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
b := lexver.Parse("1.0.0")
b.Date = time.Date(2024, 1, 15, 10, 30, 0, 0, time.UTC)
if lexver.Compare(a, b) >= 0 {
t.Error("same date, later time should sort after")
}
}
func TestOriginal(t *testing.T) {
// Parse sets both Original and Raw to the input.
v := lexver.Parse("17.0")
if v.Original != "17.0" {
t.Errorf("Original = %q, want %q", v.Original, "17.0")
}
// Release fetcher would do:
// v := lexver.Parse("17.0")
// v.Original = "REL_17_0"
v.Original = "REL_17_0"
if v.Raw != "17.0" {
t.Errorf("Raw should remain %q after setting Original, got %q", "17.0", v.Raw)
}
}
func TestExtraSort(t *testing.T) {
// Flutter example: 2.3.0-16.0.pre and 2.3.0-16.1.pre
// Nums and Channel are the same; ExtraSort distinguishes them.
a := lexver.Parse("2.3.0-pre")
a.ExtraSort = "0016.0000"
b := lexver.Parse("2.3.0-pre")
b.ExtraSort = "0016.0001"
if lexver.Compare(a, b) >= 0 {
t.Error("ExtraSort 0016.0000 should sort before 0016.0001")
}
// ExtraSort ignored when only one side has it.
c := lexver.Parse("2.3.0-pre")
c.ExtraSort = "0016.0000"
d := lexver.Parse("2.3.0-pre")
if lexver.Compare(c, d) != 0 {
t.Error("ExtraSort should be ignored when only one side has it")
}
}
func TestHasPrefix(t *testing.T) {
v := lexver.Parse("1.20.3")
if !v.HasPrefix(lexver.Parse("1.20")) {
t.Error("1.20.3 should match prefix 1.20")
}
if !v.HasPrefix(lexver.Parse("1")) {
t.Error("1.20.3 should match prefix 1")
}
if v.HasPrefix(lexver.Parse("1.19")) {
t.Error("1.20.3 should not match prefix 1.19")
}
if v.HasPrefix(lexver.Parse("2")) {
t.Error("1.20.3 should not match prefix 2")
}
// 4-part prefix matching
v4 := lexver.Parse("121.0.6120.0")
if !v4.HasPrefix(lexver.Parse("121.0.6120")) {
t.Error("121.0.6120.0 should match prefix 121.0.6120")
}
if !v4.HasPrefix(lexver.Parse("121.0")) {
t.Error("121.0.6120.0 should match prefix 121.0")
}
}

View File

@@ -0,0 +1,104 @@
// Package platlatest tracks the newest release version per build target.
//
// After classification determines which OS/arch/libc targets a release
// covers, this package records the latest version for each target. This
// handles the common case where Windows or macOS releases lag behind
// Linux by several versions.
//
// Storage is a single JSON file per package:
//
// {
// "linux-x86_64-gnu": "v0.145.0",
// "darwin-aarch64-none": "v0.144.1",
// "windows-x86_64-msvc": "v0.143.0"
// }
package platlatest
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"sync"
"github.com/webinstall/webi-installers/internal/buildmeta"
)
// Index tracks the latest version for each build target of a package.
type Index struct {
mu sync.RWMutex
path string
m map[string]string // triplet → version
}
// Open loads or creates a per-platform latest index at the given path.
func Open(path string) (*Index, error) {
idx := &Index{
path: path,
m: make(map[string]string),
}
data, err := os.ReadFile(path)
if err != nil {
if os.IsNotExist(err) {
return idx, nil
}
return nil, fmt.Errorf("platlatest: read %s: %w", path, err)
}
if err := json.Unmarshal(data, &idx.m); err != nil {
return nil, fmt.Errorf("platlatest: parse %s: %w", path, err)
}
return idx, nil
}
// Get returns the latest version for a target, or "" if unknown.
func (idx *Index) Get(t buildmeta.Target) string {
idx.mu.RLock()
defer idx.mu.RUnlock()
return idx.m[t.Triplet()]
}
// Set records a version as the latest for a target. Does not persist
// to disk — call Save after all updates.
func (idx *Index) Set(t buildmeta.Target, version string) {
idx.mu.Lock()
defer idx.mu.Unlock()
idx.m[t.Triplet()] = version
}
// All returns a copy of the full triplet→version map.
func (idx *Index) All() map[string]string {
idx.mu.RLock()
defer idx.mu.RUnlock()
out := make(map[string]string, len(idx.m))
for k, v := range idx.m {
out[k] = v
}
return out
}
// Save persists the index to disk (atomic write).
func (idx *Index) Save() error {
idx.mu.RLock()
data, err := json.MarshalIndent(idx.m, "", " ")
idx.mu.RUnlock()
if err != nil {
return fmt.Errorf("platlatest: marshal: %w", err)
}
dir := filepath.Dir(idx.path)
if err := os.MkdirAll(dir, 0o755); err != nil {
return fmt.Errorf("platlatest: mkdir: %w", err)
}
tmp := idx.path + ".tmp"
if err := os.WriteFile(tmp, data, 0o644); err != nil {
return fmt.Errorf("platlatest: write %s: %w", tmp, err)
}
if err := os.Rename(tmp, idx.path); err != nil {
os.Remove(tmp)
return fmt.Errorf("platlatest: rename %s: %w", idx.path, err)
}
return nil
}

View File

@@ -0,0 +1,104 @@
package platlatest_test
import (
"path/filepath"
"testing"
"github.com/webinstall/webi-installers/internal/buildmeta"
"github.com/webinstall/webi-installers/internal/platlatest"
)
var (
linuxAMD64 = buildmeta.Target{
OS: buildmeta.OSLinux, Arch: buildmeta.ArchAMD64, Libc: buildmeta.LibcGNU,
}
darwinARM64 = buildmeta.Target{
OS: buildmeta.OSDarwin, Arch: buildmeta.ArchARM64, Libc: buildmeta.LibcNone,
}
windowsAMD64 = buildmeta.Target{
OS: buildmeta.OSWindows, Arch: buildmeta.ArchAMD64, Libc: buildmeta.LibcMSVC,
}
)
func TestSetAndGet(t *testing.T) {
p := filepath.Join(t.TempDir(), "latest.json")
idx, err := platlatest.Open(p)
if err != nil {
t.Fatal(err)
}
if got := idx.Get(linuxAMD64); got != "" {
t.Errorf("Get before Set = %q, want empty", got)
}
idx.Set(linuxAMD64, "v0.145.0")
idx.Set(darwinARM64, "v0.144.1")
idx.Set(windowsAMD64, "v0.143.0")
if got := idx.Get(linuxAMD64); got != "v0.145.0" {
t.Errorf("linux = %q, want v0.145.0", got)
}
if got := idx.Get(darwinARM64); got != "v0.144.1" {
t.Errorf("darwin = %q, want v0.144.1", got)
}
if got := idx.Get(windowsAMD64); got != "v0.143.0" {
t.Errorf("windows = %q, want v0.143.0", got)
}
}
func TestSaveAndReload(t *testing.T) {
p := filepath.Join(t.TempDir(), "latest.json")
idx1, err := platlatest.Open(p)
if err != nil {
t.Fatal(err)
}
idx1.Set(linuxAMD64, "v0.145.0")
idx1.Set(darwinARM64, "v0.144.1")
if err := idx1.Save(); err != nil {
t.Fatal(err)
}
// Reload from disk.
idx2, err := platlatest.Open(p)
if err != nil {
t.Fatal(err)
}
if got := idx2.Get(linuxAMD64); got != "v0.145.0" {
t.Errorf("after reload: linux = %q, want v0.145.0", got)
}
if got := idx2.Get(darwinARM64); got != "v0.144.1" {
t.Errorf("after reload: darwin = %q, want v0.144.1", got)
}
}
func TestAll(t *testing.T) {
p := filepath.Join(t.TempDir(), "latest.json")
idx, err := platlatest.Open(p)
if err != nil {
t.Fatal(err)
}
idx.Set(linuxAMD64, "v1.0.0")
idx.Set(darwinARM64, "v0.9.0")
all := idx.All()
if len(all) != 2 {
t.Fatalf("All() returned %d entries, want 2", len(all))
}
if all[linuxAMD64.Triplet()] != "v1.0.0" {
t.Error("missing linux entry")
}
}
func TestOpenNonexistent(t *testing.T) {
p := filepath.Join(t.TempDir(), "does-not-exist.json")
idx, err := platlatest.Open(p)
if err != nil {
t.Fatal(err)
}
// Should be empty, not nil.
if all := idx.All(); len(all) != 0 {
t.Errorf("new index should be empty, got %v", all)
}
}

View File

@@ -0,0 +1,63 @@
package rawcache
import (
"crypto/sha256"
"encoding/hex"
"encoding/json"
"fmt"
"os"
"path/filepath"
"time"
)
// LogEntry records one event in the append-only audit log.
type LogEntry struct {
Time time.Time `json:"time"`
Tag string `json:"tag"`
Action string `json:"action"` // "added", "changed", "removed"
SHA256 string `json:"sha256,omitempty"`
}
// AuditLog is an append-only JSONL file that tracks when releases appear,
// change, or disappear from upstream. One file per package, lives alongside
// the double-buffer slots.
type AuditLog struct {
path string
}
// openLog returns the audit log for a Dir.
func (d *Dir) openLog() *AuditLog {
return &AuditLog{path: filepath.Join(d.root, "audit.jsonl")}
}
// Append writes one log entry.
func (l *AuditLog) Append(entry LogEntry) error {
if entry.Time.IsZero() {
entry.Time = time.Now().UTC()
}
data, err := json.Marshal(entry)
if err != nil {
return fmt.Errorf("rawcache: marshal log entry: %w", err)
}
data = append(data, '\n')
f, err := os.OpenFile(l.path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
if err != nil {
return fmt.Errorf("rawcache: open audit log: %w", err)
}
_, writeErr := f.Write(data)
closeErr := f.Close()
if writeErr != nil {
return fmt.Errorf("rawcache: write audit log: %w", writeErr)
}
if closeErr != nil {
return fmt.Errorf("rawcache: close audit log: %w", closeErr)
}
return nil
}
// ContentHash returns the SHA-256 hex digest of data.
func ContentHash(data []byte) string {
h := sha256.Sum256(data)
return hex.EncodeToString(h[:])
}

View File

@@ -0,0 +1,265 @@
// Package rawcache stores raw upstream API responses on disk, one file per
// release, with double-buffered full refreshes.
//
// Directory layout:
//
// {root}/
// active → a symlink to the current slot
// a/ slot A
// _latest one-line file: newest tag
// v0.145.0.json
// v0.144.1.json
// ...
// b/ slot B (standby)
//
// Incremental updates write directly to the active slot. Each file write
// is atomic (temp file + rename). Full refreshes write to the standby slot,
// then atomically swap the symlink.
package rawcache
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
)
// Dir manages a raw release cache for one package.
type Dir struct {
root string // e.g. "_cache/raw/github/gohugoio/hugo"
}
// Open returns a Dir for the given root path. Creates the directory
// structure (slots + symlink) if it doesn't exist.
func Open(root string) (*Dir, error) {
d := &Dir{root: root}
slotA := filepath.Join(root, "a")
slotB := filepath.Join(root, "b")
active := filepath.Join(root, "active")
// Create both slots.
for _, slot := range []string{slotA, slotB} {
if err := os.MkdirAll(slot, 0o755); err != nil {
return nil, fmt.Errorf("rawcache: create slot: %w", err)
}
}
// Create the active symlink if it doesn't exist.
if _, err := os.Lstat(active); errors.Is(err, os.ErrNotExist) {
if err := os.Symlink("a", active); err != nil {
return nil, fmt.Errorf("rawcache: create active symlink: %w", err)
}
}
return d, nil
}
// ActivePath returns the absolute path of the currently active slot.
func (d *Dir) ActivePath() (string, error) {
target, err := os.Readlink(filepath.Join(d.root, "active"))
if err != nil {
return "", fmt.Errorf("rawcache: read active symlink: %w", err)
}
return filepath.Join(d.root, target), nil
}
// standbySlot returns the name of the inactive slot ("a" or "b").
func (d *Dir) standbySlot() (string, error) {
target, err := os.Readlink(filepath.Join(d.root, "active"))
if err != nil {
return "", fmt.Errorf("rawcache: read active symlink: %w", err)
}
if target == "a" {
return "b", nil
}
return "a", nil
}
// Populated returns true if the active slot contains at least one release file.
func (d *Dir) Populated() bool {
active, err := d.ActivePath()
if err != nil {
return false
}
entries, err := os.ReadDir(active)
if err != nil {
return false
}
for _, e := range entries {
if !e.IsDir() && !strings.HasPrefix(e.Name(), "_") {
return true
}
}
return false
}
// Has reports whether a release file exists in the active slot.
func (d *Dir) Has(tag string) bool {
active, err := d.ActivePath()
if err != nil {
return false
}
_, err = os.Stat(filepath.Join(active, tagToFilename(tag)))
return err == nil
}
// Latest returns the newest tag from the active slot.
// Returns "" if no latest marker exists.
func (d *Dir) Latest() string {
active, err := d.ActivePath()
if err != nil {
return ""
}
data, err := os.ReadFile(filepath.Join(active, "_latest"))
if err != nil {
return ""
}
return strings.TrimSpace(string(data))
}
// Read returns the raw cached data for a tag from the active slot.
func (d *Dir) Read(tag string) ([]byte, error) {
active, err := d.ActivePath()
if err != nil {
return nil, err
}
return os.ReadFile(filepath.Join(active, tagToFilename(tag)))
}
// Put writes a release file to the active slot. The write is atomic
// (temp file + rename).
func (d *Dir) Put(tag string, data []byte) error {
active, err := d.ActivePath()
if err != nil {
return err
}
return atomicWrite(filepath.Join(active, tagToFilename(tag)), data)
}
// Merge writes a release to the active slot if it's new or changed.
// Returns the action taken: "added", "changed", or "" (unchanged).
// Logs the event to the audit log when something happens.
func (d *Dir) Merge(tag string, data []byte) (string, error) {
log := d.openLog()
hash := ContentHash(data)
if d.Has(tag) {
existing, err := d.Read(tag)
if err != nil {
return "", err
}
if ContentHash(existing) == hash {
return "", nil // unchanged
}
if err := d.Put(tag, data); err != nil {
return "", err
}
log.Append(LogEntry{Tag: tag, Action: "changed", SHA256: hash})
return "changed", nil
}
if err := d.Put(tag, data); err != nil {
return "", err
}
log.Append(LogEntry{Tag: tag, Action: "added", SHA256: hash})
return "added", nil
}
// SetLatest updates the _latest marker in the active slot.
func (d *Dir) SetLatest(tag string) error {
active, err := d.ActivePath()
if err != nil {
return err
}
return atomicWrite(filepath.Join(active, "_latest"), []byte(tag+"\n"))
}
// BeginRefresh starts a full refresh. Clears the standby slot and returns
// a Refresh handle for writing to it. Call Commit to atomically swap, or
// Abort to discard.
func (d *Dir) BeginRefresh() (*Refresh, error) {
standby, err := d.standbySlot()
if err != nil {
return nil, err
}
standbyPath := filepath.Join(d.root, standby)
// Clear the standby slot.
entries, _ := os.ReadDir(standbyPath)
for _, e := range entries {
os.Remove(filepath.Join(standbyPath, e.Name()))
}
return &Refresh{
dir: d,
slot: standby,
slotDir: standbyPath,
}, nil
}
// Refresh writes releases to the standby slot during a full refresh.
type Refresh struct {
dir *Dir
slot string // "a" or "b"
slotDir string
}
// Put writes a release file to the standby slot.
func (r *Refresh) Put(tag string, data []byte) error {
return atomicWrite(filepath.Join(r.slotDir, tagToFilename(tag)), data)
}
// SetLatest updates the _latest marker in the standby slot.
func (r *Refresh) SetLatest(tag string) error {
return atomicWrite(filepath.Join(r.slotDir, "_latest"), []byte(tag+"\n"))
}
// Commit atomically swaps the active symlink to point to the standby slot.
func (r *Refresh) Commit() error {
active := filepath.Join(r.dir.root, "active")
tmp := active + ".tmp"
// Remove stale temp symlink if it exists.
os.Remove(tmp)
if err := os.Symlink(r.slot, tmp); err != nil {
return fmt.Errorf("rawcache: create temp symlink: %w", err)
}
if err := os.Rename(tmp, active); err != nil {
os.Remove(tmp)
return fmt.Errorf("rawcache: swap active symlink: %w", err)
}
return nil
}
// Abort discards the standby slot contents.
func (r *Refresh) Abort() {
entries, _ := os.ReadDir(r.slotDir)
for _, e := range entries {
os.Remove(filepath.Join(r.slotDir, e.Name()))
}
}
// tagToFilename converts a tag to a safe filename.
// Tags like "v0.145.0" become "v0.145.0". The raw cache stores opaque
// bytes — no extension is assumed because upstream responses may be
// JSON, CSV, XML, or bespoke formats.
func tagToFilename(tag string) string {
// Replace path separators in case a tag contains slashes.
return strings.ReplaceAll(tag, "/", "_")
}
// atomicWrite writes data to path via a temp file + rename.
func atomicWrite(path string, data []byte) error {
tmp := path + ".tmp"
if err := os.WriteFile(tmp, data, 0o644); err != nil {
return fmt.Errorf("rawcache: write %s: %w", tmp, err)
}
if err := os.Rename(tmp, path); err != nil {
os.Remove(tmp)
return fmt.Errorf("rawcache: rename %s: %w", path, err)
}
return nil
}

View File

@@ -0,0 +1,173 @@
package rawcache_test
import (
"os"
"path/filepath"
"testing"
"github.com/webinstall/webi-installers/internal/rawcache"
)
func TestOpenCreatesStructure(t *testing.T) {
root := filepath.Join(t.TempDir(), "pkg")
d, err := rawcache.Open(root)
if err != nil {
t.Fatal(err)
}
_ = d
// Verify structure exists.
for _, name := range []string{"a", "b"} {
info, err := os.Stat(filepath.Join(root, name))
if err != nil {
t.Fatalf("slot %s: %v", name, err)
}
if !info.IsDir() {
t.Fatalf("slot %s is not a directory", name)
}
}
target, err := os.Readlink(filepath.Join(root, "active"))
if err != nil {
t.Fatal(err)
}
if target != "a" {
t.Errorf("active symlink = %q, want %q", target, "a")
}
}
func TestPutAndRead(t *testing.T) {
d, err := rawcache.Open(filepath.Join(t.TempDir(), "pkg"))
if err != nil {
t.Fatal(err)
}
data := []byte(`{"tag_name":"v1.0.0"}`)
if err := d.Put("v1.0.0", data); err != nil {
t.Fatal(err)
}
if !d.Has("v1.0.0") {
t.Error("Has(v1.0.0) = false after Put")
}
if d.Has("v2.0.0") {
t.Error("Has(v2.0.0) = true, should be false")
}
got, err := d.Read("v1.0.0")
if err != nil {
t.Fatal(err)
}
if string(got) != string(data) {
t.Errorf("Read = %q, want %q", got, data)
}
}
func TestLatest(t *testing.T) {
d, err := rawcache.Open(filepath.Join(t.TempDir(), "pkg"))
if err != nil {
t.Fatal(err)
}
if latest := d.Latest(); latest != "" {
t.Errorf("Latest() = %q before any writes, want empty", latest)
}
if err := d.SetLatest("v1.0.0"); err != nil {
t.Fatal(err)
}
if latest := d.Latest(); latest != "v1.0.0" {
t.Errorf("Latest() = %q, want %q", latest, "v1.0.0")
}
}
func TestRefreshDoubleBuffer(t *testing.T) {
root := filepath.Join(t.TempDir(), "pkg")
d, err := rawcache.Open(root)
if err != nil {
t.Fatal(err)
}
// Write to active slot (A).
d.Put("v1.0.0", []byte(`{"old":true}`))
d.SetLatest("v1.0.0")
// Start a full refresh — writes to standby (B).
r, err := d.BeginRefresh()
if err != nil {
t.Fatal(err)
}
r.Put("v1.0.0", []byte(`{"new":true}`))
r.Put("v2.0.0", []byte(`{"tag_name":"v2.0.0"}`))
r.SetLatest("v2.0.0")
// Before commit, active still points to A.
if d.Latest() != "v1.0.0" {
t.Error("latest should still be v1.0.0 before commit")
}
old, _ := d.Read("v1.0.0")
if string(old) != `{"old":true}` {
t.Errorf("active slot should still have old data, got %q", old)
}
// Commit swaps to B.
if err := r.Commit(); err != nil {
t.Fatal(err)
}
if d.Latest() != "v2.0.0" {
t.Errorf("Latest() = %q after commit, want %q", d.Latest(), "v2.0.0")
}
if !d.Has("v2.0.0") {
t.Error("v2.0.0 should exist after commit")
}
updated, _ := d.Read("v1.0.0")
if string(updated) != `{"new":true}` {
t.Errorf("v1.0.0 should be updated after commit, got %q", updated)
}
}
func TestRefreshAbort(t *testing.T) {
root := filepath.Join(t.TempDir(), "pkg")
d, err := rawcache.Open(root)
if err != nil {
t.Fatal(err)
}
d.Put("v1.0.0", []byte(`original`))
d.SetLatest("v1.0.0")
r, err := d.BeginRefresh()
if err != nil {
t.Fatal(err)
}
r.Put("v99.0.0", []byte(`aborted`))
r.Abort()
// Active slot should be unchanged.
if d.Latest() != "v1.0.0" {
t.Error("latest should still be v1.0.0 after abort")
}
if d.Has("v99.0.0") {
t.Error("v99.0.0 should not exist after abort")
}
}
func TestOpenIdempotent(t *testing.T) {
root := filepath.Join(t.TempDir(), "pkg")
d1, err := rawcache.Open(root)
if err != nil {
t.Fatal(err)
}
d1.Put("v1.0.0", []byte(`data`))
// Opening again should not lose data.
d2, err := rawcache.Open(root)
if err != nil {
t.Fatal(err)
}
if !d2.Has("v1.0.0") {
t.Error("data lost after re-open")
}
}

View File

@@ -0,0 +1,50 @@
// Package atomicparsley provides OS/arch classification for AtomicParsley releases.
//
// AtomicParsley uses non-standard filenames with no platform terms
// (e.g. "AtomicParsleyLinux.zip", "AtomicParsleyMacOS.zip"). The generic
// filename classifier can't extract OS or arch from these — this package
// applies the same hardcoded mapping that the production releases.js uses.
package atomicparsleydist
import (
"strings"
"github.com/webinstall/webi-installers/internal/storage"
)
// TagVariants sets OS, arch, and libc for AtomicParsley assets based on
// filename keyword matching. Replicates atomicparsley/releases.js mappings:
// - Alpine → linux/x86_64/musl
// - Linux → linux/x86_64/gnu
// - MacOS → darwin/x86_64
// - WindowsX86 → windows/x86/msvc
// - Windows → windows/x86_64/msvc
func TagVariants(assets []storage.Asset) {
for i := range assets {
if assets[i].OS != "" {
continue // already classified
}
lower := strings.ToLower(assets[i].Filename)
switch {
case strings.Contains(lower, "alpine"):
assets[i].OS = "linux"
assets[i].Arch = "x86_64"
assets[i].Libc = "musl"
case strings.Contains(lower, "linux"):
assets[i].OS = "linux"
assets[i].Arch = "x86_64"
assets[i].Libc = "gnu"
case strings.Contains(lower, "macos"):
assets[i].OS = "darwin"
assets[i].Arch = "x86_64"
case strings.Contains(lower, "windowsx86"):
assets[i].OS = "windows"
assets[i].Arch = "x86"
assets[i].Libc = "msvc"
case strings.Contains(lower, "windows"):
assets[i].OS = "windows"
assets[i].Arch = "x86_64"
assets[i].Libc = "msvc"
}
}
}

View File

@@ -0,0 +1,39 @@
// Package bun provides variant tagging for Bun releases.
//
// Bun publishes -profile (debug) builds and uses a non-standard arch
// convention: the default x86_64 build targets x86_64_v3 (AVX2+),
// while -baseline targets plain x86_64.
package bundist
import (
"strings"
"github.com/webinstall/webi-installers/internal/storage"
)
// TagVariants tags bun-specific build variants and remaps arch fields.
//
// Bun's default x86_64 build requires AVX2 (x86_64_v3). The -baseline
// build targets plain x86_64. For legacy export, baseline is the one
// we serve (matching Node.js behavior), so non-baseline gets a variant
// tag. The -baseline suffix is stripped from Filename (but not Download)
// so the legacy server sees a clean name.
func TagVariants(assets []storage.Asset) {
for i := range assets {
lower := strings.ToLower(assets[i].Filename)
if strings.Contains(lower, "-profile") {
assets[i].Variants = append(assets[i].Variants, "profile")
}
if assets[i].Arch == "x86_64" {
if strings.Contains(lower, "-baseline") {
// Baseline is plain x86_64 — strip the suffix from
// Filename so the legacy server sees a clean name.
assets[i].Filename = strings.Replace(assets[i].Filename, "-baseline", "", 1)
} else {
// Non-baseline is v3 — tag as variant (excluded from legacy).
assets[i].Arch = "x86_64_v3"
assets[i].Variants = append(assets[i].Variants, "v3")
}
}
}
}

View File

@@ -0,0 +1,72 @@
// Package chromedist fetches Chrome for Testing release data.
//
// Google publishes a JSON index of known-good Chrome/ChromeDriver versions at:
//
// https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json
//
// Each version entry has per-platform download URLs for chrome, chromedriver,
// and chrome-headless-shell.
package chromedist
import (
"context"
"encoding/json"
"fmt"
"iter"
"net/http"
)
// Index is the top-level response.
type Index struct {
Timestamp string `json:"timestamp"`
Versions []Version `json:"versions"`
}
// Version is one Chrome for Testing version with its downloads.
type Version struct {
Version string `json:"version"` // "121.0.6120.0"
Revision string `json:"revision"` // "1222902"
Downloads map[string][]Download `json:"downloads"` // "chromedriver" → []Download
}
// Download is one platform-specific download URL.
type Download struct {
Platform string `json:"platform"` // "linux64", "mac-arm64", "mac-x64", "win32", "win64"
URL string `json:"url"`
}
// Fetch retrieves the Chrome for Testing release index.
//
// Yields one batch containing all versions.
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]Version, error] {
return func(yield func([]Version, error) bool) {
url := "https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json"
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
yield(nil, fmt.Errorf("chromedist: %w", err))
return
}
req.Header.Set("Accept", "application/json")
resp, err := client.Do(req)
if err != nil {
yield(nil, fmt.Errorf("chromedist: fetch: %w", err))
return
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
yield(nil, fmt.Errorf("chromedist: fetch: %s", resp.Status))
return
}
var idx Index
if err := json.NewDecoder(resp.Body).Decode(&idx); err != nil {
yield(nil, fmt.Errorf("chromedist: decode: %w", err))
return
}
yield(idx.Versions, nil)
}
}

View File

@@ -0,0 +1,60 @@
package cmakedist
import (
"strings"
"github.com/webinstall/webi-installers/internal/storage"
)
// TagVariants tags cmake-specific build variants for exclusion from legacy export.
//
// cmake ships many formats and platforms that webi can't serve:
//
// - .sh self-extracting installer scripts: webi uses the .tar.gz archives.
//
// - .tar.Z files (old UNIX compress format): format not recognized by webi.
//
// - Darwin64 builds (pre-3.6 macOS naming): ancient format, superseded by
// the macos-universal builds.
//
// - sunos-sparc64 builds: unsupported platform (sparc64 arch not recognized).
//
// - AIX/powerpc builds: unsupported platform.
//
// - IRIX builds: unsupported platform.
//
// Note: macos10.N versioned builds (cmake-*-macos10.10-universal.tar.gz) are
// NOT dropped. Go correctly classifies them as os="darwin". The Node production
// classifier has a gap and can't parse "macos10.10" → that is a known prod bug,
// not a Go correctness issue. NODER should treat these as expected differences.
func TagVariants(assets []storage.Asset) {
for i := range assets {
lower := strings.ToLower(assets[i].Filename)
// Self-extracting installer scripts — webi uses .tar.gz archives.
if strings.HasSuffix(lower, ".sh") {
assets[i].Variants = append(assets[i].Variants, "installer")
continue
}
// Old UNIX compress format (.tar.Z) — not supported by webi.
if strings.HasSuffix(lower, ".tar.z") {
assets[i].Variants = append(assets[i].Variants, "legacy-archive")
continue
}
// Darwin64 builds: pre-cmake-3.6 macOS naming, superseded by macos-universal.
if strings.Contains(lower, "darwin64") {
assets[i].Variants = append(assets[i].Variants, "legacy-mac")
continue
}
// Unsupported platforms.
if strings.Contains(lower, "sunos") ||
strings.Contains(lower, "-aix-") ||
strings.Contains(lower, "irix") {
assets[i].Variants = append(assets[i].Variants, "unsupported-platform")
continue
}
}
}

View File

@@ -0,0 +1,28 @@
// Package fish provides variant tagging for fish shell releases.
//
// Fish publishes .pkg macOS installers alongside the standard archives.
// It also includes a source tarball (fish-{version}.tar.xz) as an
// uploaded release asset — no OS or arch in the name, indistinguishable
// from binaries by content_type. We tag it explicitly as "source".
package fishdist
import "github.com/webinstall/webi-installers/internal/storage"
// TagVariants tags fish-specific build variants.
func TagVariants(assets []storage.Asset) {
for i := range assets {
if assets[i].Format == ".pkg" {
assets[i].Variants = append(assets[i].Variants, "installer")
}
// Source tarball: no OS or arch detected by the classifier.
if assets[i].OS == "" && assets[i].Arch == "" {
assets[i].Variants = append(assets[i].Variants, "source")
}
// fish-*.app.zip is a macOS universal binary. Fish's naming puts
// arch in Linux filenames (e.g. fish-*-aarch64.tar.xz) but not in
// macOS .app.zip. Tag as x86_64; darwin waterfall serves arm64.
if assets[i].OS == "darwin" && assets[i].Arch == "" && assets[i].Format == ".app.zip" {
assets[i].Arch = "x86_64"
}
}
}

View File

@@ -0,0 +1,94 @@
// Package flutterdist fetches Flutter release data from Google Storage.
//
// Flutter publishes per-OS release indexes:
//
// https://storage.googleapis.com/flutter_infra_release/releases/releases_macos.json
// https://storage.googleapis.com/flutter_infra_release/releases/releases_linux.json
// https://storage.googleapis.com/flutter_infra_release/releases/releases_windows.json
//
// Each response has a base_url and a releases array with version, channel,
// release_date, archive path, and sha256.
package flutterdist
import (
"context"
"encoding/json"
"fmt"
"iter"
"net/http"
)
// index is the top-level JSON structure for one OS endpoint.
type index struct {
BaseURL string `json:"base_url"`
Releases []Release `json:"releases"`
}
// Release is one Flutter release entry.
type Release struct {
Hash string `json:"hash"` // git commit hash
Channel string `json:"channel"` // "stable", "beta", "dev"
Version string `json:"version"` // "3.29.2"
ReleaseDate string `json:"release_date"` // "2025-03-13T00:14:34.044690Z"
Archive string `json:"archive"` // "stable/macos/flutter_macos_arm64_3.29.2-stable.zip"
SHA256 string `json:"sha256"`
// DownloadURL is the fully-qualified URL, assembled from base_url + archive.
// Not in the upstream JSON — set by Fetch.
DownloadURL string `json:"download_url"`
// OS is the platform this entry came from ("macos", "linux", "windows").
// Not in the upstream JSON — set by Fetch.
OS string `json:"os"`
}
var defaultOSes = []string{"macos", "linux", "windows"}
// Fetch retrieves Flutter releases for all platforms.
//
// Yields one batch per OS. The iterator interface exists so callers use
// the same pattern as paginated sources.
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]Release, error] {
return func(yield func([]Release, error) bool) {
for _, osName := range defaultOSes {
url := fmt.Sprintf(
"https://storage.googleapis.com/flutter_infra_release/releases/releases_%s.json",
osName,
)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
yield(nil, fmt.Errorf("flutterdist: %w", err))
return
}
req.Header.Set("Accept", "application/json")
resp, err := client.Do(req)
if err != nil {
yield(nil, fmt.Errorf("flutterdist: fetch %s: %w", osName, err))
return
}
var idx index
err = json.NewDecoder(resp.Body).Decode(&idx)
resp.Body.Close()
if err != nil {
yield(nil, fmt.Errorf("flutterdist: decode %s: %w", osName, err))
return
}
if resp.StatusCode != http.StatusOK {
yield(nil, fmt.Errorf("flutterdist: fetch %s: %s", osName, resp.Status))
return
}
for i := range idx.Releases {
idx.Releases[i].DownloadURL = idx.BaseURL + "/" + idx.Releases[i].Archive
idx.Releases[i].OS = osName
}
if !yield(idx.Releases, nil) {
return
}
}
}
}

View File

@@ -0,0 +1,16 @@
package flutterdist
import "github.com/webinstall/webi-installers/internal/storage"
// TagVariants handles flutter-specific arch defaults.
//
// Flutter's naming convention: flutter_{os}_{version} for x86_64 builds,
// flutter_{os}_arm64_{version} for arm64. The absence of an arch token
// means x86_64 — arm64 is always explicit.
func TagVariants(assets []storage.Asset) {
for i := range assets {
if assets[i].Arch == "" && assets[i].OS != "" {
assets[i].Arch = "x86_64"
}
}
}

View File

@@ -0,0 +1,52 @@
// Package git provides variant tagging for Git for Windows releases.
//
// Git for Windows publishes GUI installer .exe files (Git-*-bit.exe),
// self-extracting PortableGit archives, and .pdb debug symbol packages
// alongside the MinGit .zip that webi installs.
package gitdist
import (
"strings"
"github.com/webinstall/webi-installers/internal/storage"
)
// TagVariants tags git-specific build variants and fixes OS/arch classification.
// All git-for-windows releases are Windows-only, but MinGit filenames like
// "MinGit-2.33.0-64-bit.zip" have no "windows" indicator — force OS=windows.
// MinGit uses "64-bit"/"32-bit" for arch — a convention specific to this project
// that the general classifier intentionally does not handle.
func TagVariants(assets []storage.Asset) {
for i := range assets {
// All git-for-windows assets are Windows. Filenames like
// "MinGit-2.33.0-64-bit.zip" have no OS term; set it explicitly.
if assets[i].OS == "" {
assets[i].OS = "windows"
}
// MinGit uses "64-bit"→x86_64, "32-bit"→x86 naming.
// "arm64" is already handled by the general classifier.
if assets[i].Arch == "" {
lower := strings.ToLower(assets[i].Filename)
if strings.Contains(lower, "64-bit") {
assets[i].Arch = "x86_64"
} else if strings.Contains(lower, "32-bit") {
assets[i].Arch = "x86"
}
}
lower := strings.ToLower(assets[i].Filename)
if assets[i].Format == ".exe" {
assets[i].Variants = append(assets[i].Variants, "installer")
}
if strings.Contains(lower, "portablegit") {
assets[i].Variants = append(assets[i].Variants, "installer")
}
if strings.Contains(lower, "-pdb") || strings.Contains(lower, "pdbs-for-") {
assets[i].Variants = append(assets[i].Variants, "pdb")
}
if strings.Contains(lower, "-busybox") {
assets[i].Variants = append(assets[i].Variants, "busybox")
}
}
}

View File

@@ -0,0 +1,33 @@
package gitdist
import (
"strings"
"github.com/webinstall/webi-installers/internal/storage"
)
// NormalizeVersions strips the ".windows.N" suffix from Git for Windows
// version strings to match the upstream Git version scheme.
//
// Git for Windows tags are like "v2.53.0.windows.1" or "v2.53.0.windows.2".
// Node.js strips ".windows.1" entirely and replaces ".windows.N" (N>1)
// with ".N":
//
// v2.53.0.windows.1 → v2.53.0
// v2.53.0.windows.2 → v2.53.0.2
func NormalizeVersions(assets []storage.Asset) {
for i := range assets {
v := assets[i].Version
idx := strings.Index(v, ".windows.")
if idx < 0 {
continue
}
suffix := v[idx+len(".windows."):]
base := v[:idx]
if suffix == "1" {
assets[i].Version = base
} else {
assets[i].Version = base + "." + suffix
}
}
}

View File

@@ -0,0 +1,120 @@
// Package gitea fetches releases from a Gitea or Forgejo instance.
//
// Gitea's release API lives under:
//
// GET {baseurl}/api/v1/repos/{owner}/{repo}/releases
//
// The response shape is similar to GitHub's but not identical. This package
// handles pagination, authentication, and deserialization independently.
package gitea
import (
"context"
"encoding/json"
"fmt"
"iter"
"net/http"
"regexp"
"strings"
)
// Release is one release from the Gitea releases API.
type Release struct {
TagName string `json:"tag_name"`
Name string `json:"name"`
Prerelease bool `json:"prerelease"`
Draft bool `json:"draft"`
PublishedAt string `json:"published_at"` // "2023-11-05T06:38:05Z"
Assets []Asset `json:"assets"`
TarballURL string `json:"tarball_url"`
ZipballURL string `json:"zipball_url"`
}
// Asset is one downloadable file attached to a release.
type Asset struct {
Name string `json:"name"` // "pathman-v0.6.0-darwin-amd64.tar.gz"
BrowserDownloadURL string `json:"browser_download_url"` // full URL
Size int64 `json:"size"`
}
// Auth holds optional credentials for authenticated API access.
type Auth struct {
Token string // personal access token or API key
}
// Fetch retrieves releases from a Gitea instance, paginating automatically.
// Each yield is one page of releases.
//
// The baseURL should be the Gitea root (e.g. "https://git.rootprojects.org").
// The /api/v1 prefix is appended automatically.
func Fetch(ctx context.Context, client *http.Client, baseURL, owner, repo string, auth *Auth) iter.Seq2[[]Release, error] {
return func(yield func([]Release, error) bool) {
base := strings.TrimRight(baseURL, "/")
page := 1
for {
url := fmt.Sprintf("%s/api/v1/repos/%s/%s/releases?limit=50&page=%d",
base, owner, repo, page)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
yield(nil, fmt.Errorf("gitea: %w", err))
return
}
req.Header.Set("Accept", "application/json")
if auth != nil && auth.Token != "" {
req.Header.Set("Authorization", "token "+auth.Token)
}
resp, err := client.Do(req)
if err != nil {
yield(nil, fmt.Errorf("gitea: fetch %s: %w", url, err))
return
}
if resp.StatusCode != http.StatusOK {
resp.Body.Close()
yield(nil, fmt.Errorf("gitea: fetch %s: %s", url, resp.Status))
return
}
var releases []Release
err = json.NewDecoder(resp.Body).Decode(&releases)
resp.Body.Close()
if err != nil {
yield(nil, fmt.Errorf("gitea: decode %s: %w", url, err))
return
}
if !yield(releases, nil) {
return
}
// Gitea uses Link headers like GitHub for pagination.
if nextURL := nextPageURL(resp.Header.Get("Link")); nextURL != "" {
url = nextURL
page++ // not strictly needed since we follow the URL, but keeps logic clear
continue
}
// No next link — also stop if we got fewer results than requested.
if len(releases) < 50 {
return
}
page++
}
}
}
var reNextLink = regexp.MustCompile(`<([^>]+)>;\s*rel="next"`)
func nextPageURL(link string) string {
if link == "" {
return ""
}
m := reNextLink.FindStringSubmatch(link)
if m == nil {
return ""
}
return m[1]
}

View File

@@ -0,0 +1,107 @@
package gitea_test
import (
"context"
"net/http"
"net/http/httptest"
"testing"
"github.com/webinstall/webi-installers/internal/releases/gitea"
)
const testReleases = `[
{
"tag_name": "v0.6.0",
"name": "v0.6.0",
"prerelease": false,
"draft": false,
"published_at": "2023-11-05T06:38:05Z",
"tarball_url": "https://example.com/archive/v0.6.0.tar.gz",
"zipball_url": "https://example.com/archive/v0.6.0.zip",
"assets": [
{
"name": "tool-v0.6.0-linux-amd64.tar.gz",
"browser_download_url": "https://example.com/releases/download/v0.6.0/tool-v0.6.0-linux-amd64.tar.gz",
"size": 89215
}
]
}
]`
func TestFetch(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/api/v1/repos/root/tool/releases" {
t.Errorf("unexpected path: %s", r.URL.Path)
http.NotFound(w, r)
return
}
w.Write([]byte(testReleases))
}))
defer srv.Close()
ctx := context.Background()
var all []gitea.Release
for releases, err := range gitea.Fetch(ctx, srv.Client(), srv.URL, "root", "tool", nil) {
if err != nil {
t.Fatal(err)
}
all = append(all, releases...)
}
if len(all) != 1 {
t.Fatalf("got %d releases, want 1", len(all))
}
if all[0].TagName != "v0.6.0" {
t.Errorf("TagName = %q, want %q", all[0].TagName, "v0.6.0")
}
if len(all[0].Assets) != 1 {
t.Errorf("got %d assets, want 1", len(all[0].Assets))
}
if all[0].TarballURL == "" {
t.Error("TarballURL is empty")
}
}
func TestFetchAuth(t *testing.T) {
var gotAuth string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotAuth = r.Header.Get("Authorization")
w.Write([]byte("[]"))
}))
defer srv.Close()
ctx := context.Background()
auth := &gitea.Auth{Token: "abc123"}
for _, err := range gitea.Fetch(ctx, srv.Client(), srv.URL, "root", "tool", auth) {
if err != nil {
t.Fatal(err)
}
}
if gotAuth != "token abc123" {
t.Errorf("Authorization = %q, want %q", gotAuth, "token abc123")
}
}
func TestFetchLive(t *testing.T) {
if testing.Short() {
t.Skip("skipping network test in short mode")
}
ctx := context.Background()
client := &http.Client{}
var total int
for releases, err := range gitea.Fetch(ctx, client, "https://git.rootprojects.org", "root", "pathman", nil) {
if err != nil {
t.Fatal(err)
}
total += len(releases)
}
if total < 1 {
t.Errorf("got %d releases, expected at least 1", total)
}
t.Logf("fetched %d releases", total)
}

View File

@@ -0,0 +1,25 @@
// Package gitea provides variant tagging for Gitea releases.
//
// Gitea publishes "gogit" builds that use an alternative pure-Go Git
// backend instead of the default C Git library.
package gitea
import (
"strings"
"github.com/webinstall/webi-installers/internal/storage"
)
// TagVariants tags gitea-specific build variants.
//
// Files containing "-gogit-" in the filename are tagged with the "gogit"
// variant. These use a pure-Go Git backend rather than the default C Git
// library.
func TagVariants(assets []storage.Asset) {
for i := range assets {
lower := strings.ToLower(assets[i].Filename)
if strings.Contains(lower, "gogit") {
assets[i].Variants = append(assets[i].Variants, "gogit")
}
}
}

View File

@@ -0,0 +1,25 @@
// Package giteasrc fetches source archives from Gitea/Forgejo releases.
//
// Some packages are installed from the auto-generated source tarballs
// rather than uploaded binary assets. This package fetches releases and
// exposes the tarball/zipball URLs.
//
// Use [gitea] for packages that use uploaded binary assets.
package giteasrc
import (
"context"
"iter"
"net/http"
"github.com/webinstall/webi-installers/internal/releases/gitea"
)
// Fetch retrieves releases from a Gitea instance for the given owner/repo.
// Paginates automatically, yielding one batch per API page.
//
// Callers should use [gitea.Release.TarballURL] and
// [gitea.Release.ZipballURL] rather than the Assets list.
func Fetch(ctx context.Context, client *http.Client, baseURL, owner, repo string, auth *gitea.Auth) iter.Seq2[[]gitea.Release, error] {
return gitea.Fetch(ctx, client, baseURL, owner, repo, auth)
}

View File

@@ -0,0 +1,22 @@
// Package github fetches releases from the GitHub API.
//
// This is a thin wrapper around [githubish] that sets the base URL to
// https://api.github.com. Use [githubish] directly for Gitea, Forgejo,
// or other GitHub-compatible forges.
package github
import (
"context"
"iter"
"net/http"
"github.com/webinstall/webi-installers/internal/releases/githubish"
)
const baseURL = "https://api.github.com"
// Fetch retrieves releases from GitHub for the given owner/repo.
// Paginates automatically, yielding one batch per API page.
func Fetch(ctx context.Context, client *http.Client, owner, repo string, auth *githubish.Auth) iter.Seq2[[]githubish.Release, error] {
return githubish.Fetch(ctx, client, baseURL, owner, repo, auth)
}

View File

@@ -0,0 +1,112 @@
// Package githubish fetches releases from GitHub-compatible APIs.
//
// GitHub, Gitea, Forgejo, and other forges expose the same releases
// endpoint shape:
//
// GET /repos/{owner}/{repo}/releases
//
// This package handles pagination (Link headers), authentication, and
// deserialization. It does not transform or normalize the data.
package githubish
import (
"context"
"encoding/json"
"fmt"
"iter"
"net/http"
"regexp"
)
// Release is one release from a GitHub-compatible API.
// Fields mirror the upstream JSON — only the fields Webi cares about are
// included; the rest are silently dropped by the decoder.
type Release struct {
TagName string `json:"tag_name"`
Name string `json:"name"`
Prerelease bool `json:"prerelease"`
Draft bool `json:"draft"`
PublishedAt string `json:"published_at"` // "2025-10-22T13:00:26Z"
Assets []Asset `json:"assets"`
TarballURL string `json:"tarball_url"` // auto-generated source tarball
ZipballURL string `json:"zipball_url"` // auto-generated source zipball
}
// Asset is one downloadable file attached to a release.
type Asset struct {
Name string `json:"name"` // "ripgrep-15.1.0-x86_64-apple-darwin.tar.gz"
BrowserDownloadURL string `json:"browser_download_url"` // full URL
Size int64 `json:"size"`
ContentType string `json:"content_type"`
}
// Auth holds optional credentials for authenticated API access.
// Without auth, GitHub's public rate limit is 60 requests/hour.
type Auth struct {
Token string // personal access token or fine-grained token
}
// Fetch retrieves releases from a GitHub-compatible API, paginating
// automatically. Each yield is one page of releases.
//
// The baseURL should be the API root (e.g. "https://api.github.com").
// For Gitea: "https://gitea.example.com/api/v1".
func Fetch(ctx context.Context, client *http.Client, baseURL, owner, repo string, auth *Auth) iter.Seq2[[]Release, error] {
return func(yield func([]Release, error) bool) {
url := fmt.Sprintf("%s/repos/%s/%s/releases?per_page=100", baseURL, owner, repo)
for url != "" {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
yield(nil, fmt.Errorf("githubish: %w", err))
return
}
req.Header.Set("Accept", "application/json")
if auth != nil && auth.Token != "" {
req.Header.Set("Authorization", "Bearer "+auth.Token)
}
resp, err := client.Do(req)
if err != nil {
yield(nil, fmt.Errorf("githubish: fetch %s: %w", url, err))
return
}
if resp.StatusCode != http.StatusOK {
resp.Body.Close()
yield(nil, fmt.Errorf("githubish: fetch %s: %s", url, resp.Status))
return
}
var releases []Release
err = json.NewDecoder(resp.Body).Decode(&releases)
resp.Body.Close()
if err != nil {
yield(nil, fmt.Errorf("githubish: decode %s: %w", url, err))
return
}
if !yield(releases, nil) {
return
}
url = nextPageURL(resp.Header.Get("Link"))
}
}
}
// reNextLink matches `<URL>; rel="next"` in a Link header.
var reNextLink = regexp.MustCompile(`<([^>]+)>;\s*rel="next"`)
// nextPageURL extracts the "next" URL from a GitHub Link header.
// Returns "" if there is no next page.
func nextPageURL(link string) string {
if link == "" {
return ""
}
m := reNextLink.FindStringSubmatch(link)
if m == nil {
return ""
}
return m[1]
}

View File

@@ -0,0 +1,201 @@
package githubish_test
import (
"context"
"fmt"
"net/http"
"net/http/httptest"
"testing"
"github.com/webinstall/webi-installers/internal/releases/githubish"
)
const page1 = `[
{
"tag_name": "v2.0.0",
"name": "v2.0.0",
"prerelease": false,
"draft": false,
"published_at": "2025-06-01T12:00:00Z",
"assets": [
{
"name": "tool-v2.0.0-linux-amd64.tar.gz",
"browser_download_url": "https://example.com/tool-v2.0.0-linux-amd64.tar.gz",
"size": 5000000,
"content_type": "application/gzip"
}
]
}
]`
const page2 = `[
{
"tag_name": "v1.0.0",
"name": "v1.0.0",
"prerelease": false,
"draft": false,
"published_at": "2024-01-15T08:00:00Z",
"assets": [
{
"name": "tool-v1.0.0-linux-amd64.tar.gz",
"browser_download_url": "https://example.com/tool-v1.0.0-linux-amd64.tar.gz",
"size": 4000000,
"content_type": "application/gzip"
},
{
"name": "tool-v1.0.0-darwin-arm64.tar.gz",
"browser_download_url": "https://example.com/tool-v1.0.0-darwin-arm64.tar.gz",
"size": 4500000,
"content_type": "application/gzip"
}
]
}
]`
func TestFetchPagination(t *testing.T) {
var srvURL string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/repos/acme/tool/releases" {
t.Errorf("unexpected path: %s", r.URL.Path)
http.NotFound(w, r)
return
}
page := r.URL.Query().Get("page")
switch page {
case "", "1":
// Link header pointing to page 2
w.Header().Set("Link",
fmt.Sprintf(`<%s/repos/acme/tool/releases?per_page=100&page=2>; rel="next"`, srvURL))
w.Write([]byte(page1))
case "2":
// No Link header — last page
w.Write([]byte(page2))
default:
http.NotFound(w, r)
}
}))
defer srv.Close()
srvURL = srv.URL
ctx := context.Background()
var batches int
var allReleases []githubish.Release
for releases, err := range githubish.Fetch(ctx, srv.Client(), srv.URL, "acme", "tool", nil) {
if err != nil {
t.Fatalf("batch %d: %v", batches, err)
}
batches++
allReleases = append(allReleases, releases...)
}
if batches != 2 {
t.Errorf("got %d batches, want 2", batches)
}
if len(allReleases) != 2 {
t.Fatalf("got %d releases, want 2", len(allReleases))
}
// Page 1: v2.0.0
if allReleases[0].TagName != "v2.0.0" {
t.Errorf("release[0].TagName = %q, want %q", allReleases[0].TagName, "v2.0.0")
}
if len(allReleases[0].Assets) != 1 {
t.Errorf("release[0] has %d assets, want 1", len(allReleases[0].Assets))
}
// Page 2: v1.0.0
if allReleases[1].TagName != "v1.0.0" {
t.Errorf("release[1].TagName = %q, want %q", allReleases[1].TagName, "v1.0.0")
}
if len(allReleases[1].Assets) != 2 {
t.Errorf("release[1] has %d assets, want 2", len(allReleases[1].Assets))
}
}
func TestFetchPrerelease(t *testing.T) {
body := `[{"tag_name":"v1.0.0-rc1","name":"","prerelease":true,"draft":false,"published_at":"2025-01-01T00:00:00Z","assets":[]}]`
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte(body))
}))
defer srv.Close()
ctx := context.Background()
for releases, err := range githubish.Fetch(ctx, srv.Client(), srv.URL, "acme", "tool", nil) {
if err != nil {
t.Fatal(err)
}
if len(releases) != 1 {
t.Fatalf("got %d releases, want 1", len(releases))
}
if !releases[0].Prerelease {
t.Error("expected Prerelease = true")
}
if releases[0].TagName != "v1.0.0-rc1" {
t.Errorf("TagName = %q, want %q", releases[0].TagName, "v1.0.0-rc1")
}
}
}
func TestFetchAuth(t *testing.T) {
var gotAuth string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotAuth = r.Header.Get("Authorization")
w.Write([]byte("[]"))
}))
defer srv.Close()
ctx := context.Background()
auth := &githubish.Auth{Token: "ghp_test123"}
for _, err := range githubish.Fetch(ctx, srv.Client(), srv.URL, "acme", "tool", auth) {
if err != nil {
t.Fatal(err)
}
}
if gotAuth != "Bearer ghp_test123" {
t.Errorf("Authorization = %q, want %q", gotAuth, "Bearer ghp_test123")
}
}
func TestFetchHTTPError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Error(w, "not found", http.StatusNotFound)
}))
defer srv.Close()
ctx := context.Background()
for _, err := range githubish.Fetch(ctx, srv.Client(), srv.URL, "acme", "tool", nil) {
if err == nil {
t.Fatal("expected error for 404 response")
}
return
}
}
func TestFetchEarlyBreak(t *testing.T) {
var requests int
var srvURL string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests++
// Always advertise a next page
w.Header().Set("Link",
fmt.Sprintf(`<%s/repos/acme/tool/releases?per_page=100&page=%d>; rel="next"`, srvURL, requests+1))
w.Write([]byte(`[{"tag_name":"v1.0.0","name":"","prerelease":false,"draft":false,"published_at":"2025-01-01T00:00:00Z","assets":[]}]`))
}))
defer srv.Close()
srvURL = srv.URL
ctx := context.Background()
for _, err := range githubish.Fetch(ctx, srv.Client(), srv.URL, "acme", "tool", nil) {
if err != nil {
t.Fatal(err)
}
break // stop after first page
}
if requests != 1 {
t.Errorf("server received %d requests, want 1 (early break should stop pagination)", requests)
}
}

View File

@@ -0,0 +1,27 @@
// Package githubsrc fetches source archives from GitHub releases.
//
// Some packages (shell scripts, vim plugins) are installed from the
// auto-generated source tarballs rather than uploaded binary assets.
// This package fetches releases and exposes the tarball/zipball URLs.
//
// Use [github] for packages that use uploaded binary assets.
package githubsrc
import (
"context"
"iter"
"net/http"
"github.com/webinstall/webi-installers/internal/releases/githubish"
)
const baseURL = "https://api.github.com"
// Fetch retrieves releases from GitHub for the given owner/repo.
// Paginates automatically, yielding one batch per API page.
//
// Callers should use [githubish.Release.TarballURL] and
// [githubish.Release.ZipballURL] rather than the Assets list.
func Fetch(ctx context.Context, client *http.Client, owner, repo string, auth *githubish.Auth) iter.Seq2[[]githubish.Release, error] {
return githubish.Fetch(ctx, client, baseURL, owner, repo, auth)
}

View File

@@ -0,0 +1,122 @@
// Package gitlab fetches releases from a GitLab instance.
//
// GitLab's releases API differs from GitHub's in structure:
//
// GET /api/v4/projects/:id/releases
//
// Where :id is the URL-encoded project path (e.g. "group%2Frepo") or a
// numeric project ID. Assets are split into auto-generated source archives
// and manually attached links. Pagination uses page/per_page query params
// and X-Total-Pages response headers (not Link headers).
//
// This package handles pagination, authentication, and deserialization.
// It does not transform or normalize the data.
package gitlab
import (
"context"
"encoding/json"
"fmt"
"iter"
"net/http"
"net/url"
"strconv"
)
// Release is one release from the GitLab releases API.
type Release struct {
TagName string `json:"tag_name"`
Name string `json:"name"`
ReleasedAt string `json:"released_at"` // "2025-10-22T13:00:26Z"
Assets Assets `json:"assets"`
}
// Assets holds both auto-generated source archives and attached links.
type Assets struct {
Sources []Source `json:"sources"`
Links []Link `json:"links"`
}
// Source is an auto-generated source archive (tar.gz, zip, etc.).
type Source struct {
Format string `json:"format"` // "zip", "tar.gz", "tar.bz2", "tar"
URL string `json:"url"`
}
// Link is a file attached to a release (binary, package, etc.).
type Link struct {
ID int `json:"id"`
Name string `json:"name"`
URL string `json:"url"`
DirectAssetPath string `json:"direct_asset_path"`
LinkType string `json:"link_type"` // "other", "runbook", "image", "package"
}
// Auth holds optional credentials for authenticated API access.
type Auth struct {
Token string // personal access token or deploy token
}
// Fetch retrieves releases from a GitLab instance, paginating automatically.
// Each yield is one page of releases.
//
// The baseURL should be the GitLab root (e.g. "https://gitlab.com").
// The project is identified by its path (e.g. "group/repo") — it will be
// URL-encoded automatically.
func Fetch(ctx context.Context, client *http.Client, baseURL, project string, auth *Auth) iter.Seq2[[]Release, error] {
return func(yield func([]Release, error) bool) {
encodedProject := url.PathEscape(project)
page := 1
for {
reqURL := fmt.Sprintf("%s/api/v4/projects/%s/releases?per_page=100&page=%d",
baseURL, encodedProject, page)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, reqURL, nil)
if err != nil {
yield(nil, fmt.Errorf("gitlab: %w", err))
return
}
req.Header.Set("Accept", "application/json")
if auth != nil && auth.Token != "" {
req.Header.Set("PRIVATE-TOKEN", auth.Token)
}
resp, err := client.Do(req)
if err != nil {
yield(nil, fmt.Errorf("gitlab: fetch %s: %w", reqURL, err))
return
}
if resp.StatusCode != http.StatusOK {
resp.Body.Close()
yield(nil, fmt.Errorf("gitlab: fetch %s: %s", reqURL, resp.Status))
return
}
var releases []Release
err = json.NewDecoder(resp.Body).Decode(&releases)
resp.Body.Close()
if err != nil {
yield(nil, fmt.Errorf("gitlab: decode %s: %w", reqURL, err))
return
}
if !yield(releases, nil) {
return
}
// Check if there are more pages.
totalPages := 1
if tp := resp.Header.Get("X-Total-Pages"); tp != "" {
if n, err := strconv.Atoi(tp); err == nil {
totalPages = n
}
}
if page >= totalPages {
return
}
page++
}
}
}

View File

@@ -0,0 +1,182 @@
package gitlab_test
import (
"context"
"fmt"
"net/http"
"net/http/httptest"
"testing"
"github.com/webinstall/webi-installers/internal/releases/gitlab"
)
const page1 = `[
{
"tag_name": "v2.0.0",
"name": "v2.0.0",
"released_at": "2025-06-01T12:00:00Z",
"assets": {
"sources": [
{"format": "tar.gz", "url": "https://example.com/archive/v2.0.0.tar.gz"},
{"format": "zip", "url": "https://example.com/archive/v2.0.0.zip"}
],
"links": [
{
"id": 1,
"name": "tool-v2.0.0-linux-amd64.tar.gz",
"url": "https://example.com/tool-v2.0.0-linux-amd64.tar.gz",
"direct_asset_path": "/binaries/linux-amd64",
"link_type": "package"
}
]
}
}
]`
const page2 = `[
{
"tag_name": "v1.0.0",
"name": "v1.0.0",
"released_at": "2024-01-15T08:00:00Z",
"assets": {
"sources": [
{"format": "tar.gz", "url": "https://example.com/archive/v1.0.0.tar.gz"}
],
"links": []
}
}
]`
func TestFetchPagination(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Go's http server decodes %2F back to /, so check RawPath
// for the encoded form or Path for the decoded form.
wantRaw := "/api/v4/projects/group%2Ftool/releases"
wantDecoded := "/api/v4/projects/group/tool/releases"
if r.URL.RawPath != wantRaw && r.URL.Path != wantDecoded {
t.Errorf("unexpected path: raw=%q decoded=%q", r.URL.RawPath, r.URL.Path)
http.NotFound(w, r)
return
}
page := r.URL.Query().Get("page")
w.Header().Set("X-Total-Pages", "2")
switch page {
case "", "1":
w.Write([]byte(page1))
case "2":
w.Write([]byte(page2))
default:
http.NotFound(w, r)
}
}))
defer srv.Close()
ctx := context.Background()
var batches int
var allReleases []gitlab.Release
for releases, err := range gitlab.Fetch(ctx, srv.Client(), srv.URL, "group/tool", nil) {
if err != nil {
t.Fatalf("batch %d: %v", batches, err)
}
batches++
allReleases = append(allReleases, releases...)
}
if batches != 2 {
t.Errorf("got %d batches, want 2", batches)
}
if len(allReleases) != 2 {
t.Fatalf("got %d releases, want 2", len(allReleases))
}
// Page 1: v2.0.0
r1 := allReleases[0]
if r1.TagName != "v2.0.0" {
t.Errorf("release[0].TagName = %q, want %q", r1.TagName, "v2.0.0")
}
if len(r1.Assets.Sources) != 2 {
t.Errorf("release[0] has %d sources, want 2", len(r1.Assets.Sources))
}
if len(r1.Assets.Links) != 1 {
t.Errorf("release[0] has %d links, want 1", len(r1.Assets.Links))
}
if r1.Assets.Links[0].LinkType != "package" {
t.Errorf("release[0] link type = %q, want %q", r1.Assets.Links[0].LinkType, "package")
}
// Page 2: v1.0.0
r2 := allReleases[1]
if r2.TagName != "v1.0.0" {
t.Errorf("release[1].TagName = %q, want %q", r2.TagName, "v1.0.0")
}
if len(r2.Assets.Links) != 0 {
t.Errorf("release[1] has %d links, want 0", len(r2.Assets.Links))
}
}
func TestFetchAuth(t *testing.T) {
var gotAuth string
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gotAuth = r.Header.Get("PRIVATE-TOKEN")
w.Write([]byte("[]"))
}))
defer srv.Close()
ctx := context.Background()
auth := &gitlab.Auth{Token: "glpat-test123"}
for _, err := range gitlab.Fetch(ctx, srv.Client(), srv.URL, "group/tool", auth) {
if err != nil {
t.Fatal(err)
}
}
if gotAuth != "glpat-test123" {
t.Errorf("PRIVATE-TOKEN = %q, want %q", gotAuth, "glpat-test123")
}
}
func TestFetchSinglePage(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// No X-Total-Pages header — defaults to 1 page.
w.Write([]byte(page1))
}))
defer srv.Close()
ctx := context.Background()
var batches int
for _, err := range gitlab.Fetch(ctx, srv.Client(), srv.URL, "group/tool", nil) {
if err != nil {
t.Fatal(err)
}
batches++
}
if batches != 1 {
t.Errorf("got %d batches, want 1 (no X-Total-Pages means single page)", batches)
}
}
func TestFetchEarlyBreak(t *testing.T) {
var requests int
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
requests++
w.Header().Set("X-Total-Pages", "10")
w.Write([]byte(fmt.Sprintf(`[{"tag_name":"v%d.0.0","name":"","released_at":"2025-01-01T00:00:00Z","assets":{"sources":[],"links":[]}}]`, requests)))
}))
defer srv.Close()
ctx := context.Background()
for _, err := range gitlab.Fetch(ctx, srv.Client(), srv.URL, "group/tool", nil) {
if err != nil {
t.Fatal(err)
}
break // stop after first page
}
if requests != 1 {
t.Errorf("server received %d requests, want 1", requests)
}
}

View File

@@ -0,0 +1,25 @@
// Package gitlabsrc fetches source archives from GitLab releases.
//
// Some packages are installed from the auto-generated source archives
// rather than attached binary links. This package fetches releases and
// exposes the source archive URLs.
//
// Use [gitlab] for packages that use attached release links (binaries).
package gitlabsrc
import (
"context"
"iter"
"net/http"
"github.com/webinstall/webi-installers/internal/releases/gitlab"
)
// Fetch retrieves releases from a GitLab instance.
// Paginates automatically, yielding one batch per API page.
//
// Callers should use [gitlab.Release.Assets.Sources] rather than
// [gitlab.Release.Assets.Links].
func Fetch(ctx context.Context, client *http.Client, baseURL, project string, auth *gitlab.Auth) iter.Seq2[[]gitlab.Release, error] {
return gitlab.Fetch(ctx, client, baseURL, project, auth)
}

View File

@@ -0,0 +1,178 @@
// Package gittag fetches release information from git tags in a bare repo.
//
// Some packages (vim plugins, shell scripts) are installed by cloning a git
// repo rather than downloading a binary. For these, each tag is a "release"
// and the download URL is the repo's git URL.
//
// This package clones (or fetches) a bare repo to a local cache directory,
// lists version-like tags, and returns them with their commit metadata.
// HEAD is also included as a potential release.
package gittag
import (
"context"
"fmt"
"iter"
"os"
"os/exec"
"path/filepath"
"regexp"
"strings"
"crypto/rand"
"encoding/hex"
)
// Entry is one tag (or HEAD) from a git repo.
type Entry struct {
Version string // tag name or date-based version for HEAD
GitTag string // the ref that can be passed to `git clone --branch`
CommitHash string // abbreviated commit hash
Date string // ISO 8601 commit date (author date)
}
// reVersionTag matches tags that look like versions: v1, v1.2, 1.0.0-rc, etc.
var reVersionTag = regexp.MustCompile(`^v?\d+(\.\d+)`)
// Fetch clones or updates a bare repo, then yields its version-like tags
// and HEAD as entries. The repoDir is the parent directory where bare repos
// are cached.
//
// Yields one batch containing all tags plus HEAD.
func Fetch(ctx context.Context, gitURL, repoDir string) iter.Seq2[[]Entry, error] {
return func(yield func([]Entry, error) bool) {
repoName := filepath.Base(gitURL)
repoName = strings.TrimSuffix(repoName, ".git")
repoPath := filepath.Join(repoDir, repoName+".git")
if err := ensureRepo(ctx, repoPath, gitURL); err != nil {
yield(nil, fmt.Errorf("gittag: %w", err))
return
}
tags, err := listVersionTags(ctx, repoPath)
if err != nil {
yield(nil, fmt.Errorf("gittag: %w", err))
return
}
var entries []Entry
for _, tag := range tags {
info, err := commitInfo(ctx, repoPath, tag)
if err != nil {
yield(nil, fmt.Errorf("gittag: commit info for %q: %w", tag, err))
return
}
info.Version = tag
info.GitTag = tag
entries = append(entries, info)
}
// HEAD as an additional entry
head, err := commitInfo(ctx, repoPath, "HEAD")
if err != nil {
yield(nil, fmt.Errorf("gittag: commit info for HEAD: %w", err))
return
}
branch, err := headBranch(ctx, repoPath)
if err != nil {
yield(nil, fmt.Errorf("gittag: HEAD branch: %w", err))
return
}
head.GitTag = branch
// Version for HEAD is set by the caller (date-based, etc.)
entries = append(entries, head)
yield(entries, nil)
}
}
// ensureRepo clones the repo if it doesn't exist, or fetches if it does.
func ensureRepo(ctx context.Context, repoPath, gitURL string) error {
if _, err := os.Stat(repoPath); err == nil {
// Exists — fetch updates.
cmd := exec.CommandContext(ctx, "git", "--git-dir="+repoPath, "fetch")
cmd.Stderr = os.Stderr
return cmd.Run()
}
// Clone bare with tree filter (metadata only).
var b [8]byte
rand.Read(b[:])
id := hex.EncodeToString(b[:])
tmpPath := repoPath + "." + id + ".tmp"
cmd := exec.CommandContext(ctx, "git", "clone", "--bare", "--filter=tree:0", gitURL, tmpPath)
cmd.Stderr = os.Stderr
if err := cmd.Run(); err != nil {
os.RemoveAll(tmpPath)
return fmt.Errorf("clone %s: %w", gitURL, err)
}
// Atomic swap — if repoPath appeared in a race, keep it and discard ours.
if err := os.Rename(tmpPath, repoPath); err != nil {
os.RemoveAll(tmpPath)
// If rename failed because repoPath now exists, that's fine.
if _, statErr := os.Stat(repoPath); statErr == nil {
return nil
}
return err
}
return nil
}
// listVersionTags returns tags that look like version numbers, newest first.
func listVersionTags(ctx context.Context, repoPath string) ([]string, error) {
cmd := exec.CommandContext(ctx, "git", "--git-dir="+repoPath, "tag")
out, err := cmd.Output()
if err != nil {
return nil, fmt.Errorf("git tag: %w", err)
}
var tags []string
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
if line == "" {
continue
}
if reVersionTag.MatchString(line) {
tags = append(tags, line)
}
}
// Reverse so newest tags come first (git tag outputs alphabetically).
for i, j := 0, len(tags)-1; i < j; i, j = i+1, j-1 {
tags[i], tags[j] = tags[j], tags[i]
}
return tags, nil
}
// commitInfo returns the abbreviated hash and author date for a commitish.
func commitInfo(ctx context.Context, repoPath, commitish string) (Entry, error) {
cmd := exec.CommandContext(ctx, "git", "--git-dir="+repoPath,
"log", "-1", "--format=%h %ad", "--date=iso-strict", commitish)
out, err := cmd.Output()
if err != nil {
return Entry{}, fmt.Errorf("git log %s: %w", commitish, err)
}
parts := strings.Fields(strings.TrimSpace(string(out)))
if len(parts) < 2 {
return Entry{}, fmt.Errorf("unexpected git log output: %q", out)
}
return Entry{
CommitHash: parts[0],
Date: parts[1],
}, nil
}
// headBranch returns the symbolic ref for HEAD (e.g. "main", "master").
func headBranch(ctx context.Context, repoPath string) (string, error) {
cmd := exec.CommandContext(ctx, "git", "--git-dir="+repoPath,
"rev-parse", "--abbrev-ref", "HEAD")
out, err := cmd.Output()
if err != nil {
return "", fmt.Errorf("git rev-parse HEAD: %w", err)
}
return strings.TrimSpace(string(out)), nil
}

View File

@@ -0,0 +1,56 @@
package gittag_test
import (
"context"
"testing"
"github.com/webinstall/webi-installers/internal/releases/gittag"
)
func TestFetch(t *testing.T) {
if testing.Short() {
t.Skip("skipping network/git test in short mode")
}
ctx := context.Background()
repoDir := t.TempDir()
// vim-commentary has a small number of tags.
var entries []gittag.Entry
for batch, err := range gittag.Fetch(ctx, "https://github.com/tpope/vim-commentary.git", repoDir) {
if err != nil {
t.Fatalf("Fetch: %v", err)
}
entries = append(entries, batch...)
}
if len(entries) < 2 {
t.Fatalf("got %d entries, expected at least 2 (tags + HEAD)", len(entries))
}
// Last entry should be HEAD (no Version set by the fetcher).
head := entries[len(entries)-1]
if head.CommitHash == "" {
t.Error("HEAD entry has empty CommitHash")
}
if head.Date == "" {
t.Error("HEAD entry has empty Date")
}
if head.GitTag == "" {
t.Error("HEAD entry has empty GitTag (branch name)")
}
// At least one tag entry should have a version.
found := false
for _, e := range entries[:len(entries)-1] {
if e.Version != "" {
found = true
break
}
}
if !found {
t.Error("no tag entries have a Version set")
}
t.Logf("fetched %d entries (last is HEAD on %q)", len(entries), head.GitTag)
}

View File

@@ -0,0 +1,72 @@
// Package golang fetches Go release data from golang.org.
//
// The API returns all releases (including unstable) as a JSON array:
//
// https://golang.org/dl/?mode=json&include=all
//
// Each release has a version string like "go1.24.1" and a list of file
// objects with filename, os, arch, sha256, size, and kind.
package golang
import (
"context"
"encoding/json"
"fmt"
"iter"
"net/http"
)
// Release is one Go version from the download API.
type Release struct {
Version string `json:"version"` // "go1.24.1"
Stable bool `json:"stable"`
Files []File `json:"files"`
}
// File is one downloadable artifact within a release.
type File struct {
Filename string `json:"filename"` // "go1.24.1.linux-amd64.tar.gz"
OS string `json:"os"` // "linux", "darwin", "windows", ""
Arch string `json:"arch"` // "amd64", "arm64", "386", ""
Version string `json:"version"` // "go1.24.1"
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
Kind string `json:"kind"` // "archive", "installer", "source"
}
// Fetch retrieves the Go release index.
//
// Yields one batch containing all releases. The iterator interface exists
// so callers use the same pattern as paginated sources.
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]Release, error] {
return func(yield func([]Release, error) bool) {
url := "https://golang.org/dl/?mode=json&include=all"
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
yield(nil, fmt.Errorf("golang: %w", err))
return
}
req.Header.Set("Accept", "application/json")
resp, err := client.Do(req)
if err != nil {
yield(nil, fmt.Errorf("golang: fetch: %w", err))
return
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
yield(nil, fmt.Errorf("golang: fetch: %s", resp.Status))
return
}
var releases []Release
if err := json.NewDecoder(resp.Body).Decode(&releases); err != nil {
yield(nil, fmt.Errorf("golang: decode: %w", err))
return
}
yield(releases, nil)
}
}

View File

@@ -0,0 +1,70 @@
// Package gpgdist fetches GPG for macOS release data from SourceForge RSS.
//
// The gpgosx project publishes DMG installers on SourceForge. The RSS feed
// at https://sourceforge.net/projects/gpgosx/rss?path=/ lists download links
// for each version.
package gpgdist
import (
"context"
"fmt"
"io"
"iter"
"net/http"
"regexp"
)
// Entry is one GPG macOS release.
type Entry struct {
Version string `json:"version"` // "2.4.7"
URL string `json:"url"` // full SourceForge download URL
}
var linkRe = regexp.MustCompile(
`<link>(https://sourceforge\.net/projects/gpgosx/files/GnuPG-([\d.]+)\.dmg/download)</link>`,
)
// Fetch retrieves GPG macOS releases from the SourceForge RSS feed.
//
// Yields one batch containing all releases.
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]Entry, error] {
return func(yield func([]Entry, error) bool) {
url := "https://sourceforge.net/projects/gpgosx/rss?path=/"
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
yield(nil, fmt.Errorf("gpgdist: %w", err))
return
}
req.Header.Set("Accept", "application/rss+xml")
resp, err := client.Do(req)
if err != nil {
yield(nil, fmt.Errorf("gpgdist: fetch: %w", err))
return
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
yield(nil, fmt.Errorf("gpgdist: fetch: %s", resp.Status))
return
}
body, err := io.ReadAll(resp.Body)
if err != nil {
yield(nil, fmt.Errorf("gpgdist: read: %w", err))
return
}
matches := linkRe.FindAllStringSubmatch(string(body), -1)
var entries []Entry
for _, m := range matches {
entries = append(entries, Entry{
URL: m[1],
Version: m[2],
})
}
yield(entries, nil)
}
}

View File

@@ -0,0 +1,79 @@
// Package hashicorp fetches release data from the HashiCorp releases API.
//
// HashiCorp publishes release indexes at:
//
// https://releases.hashicorp.com/{product}/index.json
//
// The response is a JSON object with a "versions" key mapping version strings
// to objects containing build arrays with url, os, arch, and filename.
package hashicorp
import (
"context"
"encoding/json"
"fmt"
"iter"
"net/http"
)
// Index is the top-level response from the HashiCorp releases API.
type Index struct {
Versions map[string]Version `json:"versions"`
}
// Version is one release version with its builds.
type Version struct {
Name string `json:"name"` // "terraform"
Version string `json:"version"` // "1.12.0"
SHASUMS string `json:"shasums,omitempty"` // URL to SHA256SUMS file
SHASUMSSig string `json:"shasums_signature"` // URL to signature
Builds []Build `json:"builds"`
TimestampCreated string `json:"timestamp_created,omitempty"`
TimestampUpdated string `json:"timestamp_updated,omitempty"`
}
// Build is one downloadable artifact.
type Build struct {
Name string `json:"name"` // "terraform"
Version string `json:"version"` // "1.12.0"
OS string `json:"os"` // "linux", "darwin", "windows"
Arch string `json:"arch"` // "amd64", "arm64", "386"
Filename string `json:"filename"` // "terraform_1.12.0_linux_amd64.zip"
URL string `json:"url"` // full download URL
}
// Fetch retrieves the HashiCorp release index for a product.
//
// Yields one batch containing all versions.
func Fetch(ctx context.Context, client *http.Client, product string) iter.Seq2[*Index, error] {
return func(yield func(*Index, error) bool) {
url := fmt.Sprintf("https://releases.hashicorp.com/%s/index.json", product)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
yield(nil, fmt.Errorf("hashicorp: %w", err))
return
}
req.Header.Set("Accept", "application/json")
resp, err := client.Do(req)
if err != nil {
yield(nil, fmt.Errorf("hashicorp: fetch %s: %w", product, err))
return
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
yield(nil, fmt.Errorf("hashicorp: fetch %s: %s", product, resp.Status))
return
}
var idx Index
if err := json.NewDecoder(resp.Body).Decode(&idx); err != nil {
yield(nil, fmt.Errorf("hashicorp: decode %s: %w", product, err))
return
}
yield(&idx, nil)
}
}

View File

@@ -0,0 +1,105 @@
// Package iterm2dist fetches iTerm2 release URLs from the downloads page.
//
// iTerm2 doesn't have a structured API — releases are listed as links on:
//
// https://iterm2.com/downloads.html
//
// This package scrapes download links matching iTerm2-[34]*.zip from the
// HTML and returns them as structured entries.
package iterm2dist
import (
"context"
"fmt"
"io"
"iter"
"net/http"
"regexp"
"strings"
)
// Entry is one iTerm2 download link with extracted metadata.
type Entry struct {
Version string `json:"version"` // "3.5.13"
Channel string `json:"channel"` // "stable" or "beta"
URL string `json:"url"` // full download URL
}
var linkRe = regexp.MustCompile(`href="(https://iterm2\.com/downloads/[^"]*\.zip)"`)
var versionRe = regexp.MustCompile(`iTerm2[-_]v?(\d+(?:_\d+)*)(?:[-_]?(beta|preview)[-_]?(\d*))?\.zip`)
// Fetch retrieves iTerm2 releases by scraping the downloads page.
//
// Yields one batch containing all releases.
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]Entry, error] {
return func(yield func([]Entry, error) bool) {
url := "https://iterm2.com/downloads.html"
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
yield(nil, fmt.Errorf("iterm2dist: %w", err))
return
}
req.Header.Set("Accept", "text/html")
resp, err := client.Do(req)
if err != nil {
yield(nil, fmt.Errorf("iterm2dist: fetch: %w", err))
return
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
yield(nil, fmt.Errorf("iterm2dist: fetch: %s", resp.Status))
return
}
body, err := io.ReadAll(resp.Body)
if err != nil {
yield(nil, fmt.Errorf("iterm2dist: read: %w", err))
return
}
matches := linkRe.FindAllStringSubmatch(string(body), -1)
var entries []Entry
seen := make(map[string]bool)
for _, m := range matches {
link := m[1]
// Only include iTerm2 v3+ downloads.
if !strings.Contains(link, "iTerm2-3") && !strings.Contains(link, "iTerm2-4") {
continue
}
entry := Entry{URL: link}
// Determine channel from URL path.
if strings.Contains(link, "/stable/") {
entry.Channel = "stable"
} else {
entry.Channel = "beta"
}
// Extract version: iTerm2-3_5_13.zip → 3.5.13
vm := versionRe.FindStringSubmatch(link)
if vm != nil {
entry.Version = strings.ReplaceAll(vm[1], "_", ".")
// vm[2] = "beta" or "preview", vm[3] = optional number
if vm[2] != "" {
entry.Version += "-" + vm[2] + vm[3]
}
}
// The downloads page has duplicate links for some betas
// (e.g. iTerm2-3_5_1beta1.zip and iTerm2-3_5_1_beta1.zip).
// Keep the first URL encountered per version.
if seen[entry.Version] {
continue
}
seen[entry.Version] = true
entries = append(entries, entry)
}
yield(entries, nil)
}
}

View File

@@ -0,0 +1,89 @@
// Package juliadist fetches Julia release data from the Julia S3 API.
//
// Julia publishes a version index at:
//
// https://julialang-s3.julialang.org/bin/versions.json
//
// The response is a JSON object keyed by version string, where each value
// has a "files" array of downloadable artifacts with url, triplet, kind,
// arch, os, sha256, size, and extension fields.
package juliadist
import (
"context"
"encoding/json"
"fmt"
"iter"
"net/http"
)
// Release is one Julia version with its file artifacts.
type Release struct {
Version string `json:"version"` // set by us from the key
Stable bool `json:"stable"`
Files []File `json:"files"`
}
// File is one downloadable artifact.
type File struct {
URL string `json:"url"` // full download URL
Triplet string `json:"triplet"` // "aarch64-apple-darwin14"
Kind string `json:"kind"` // "archive" or "installer"
Arch string `json:"arch"` // "aarch64", "x86_64", "i686"
OS string `json:"os"` // "mac", "linux", "winnt"
SHA256 string `json:"sha256"`
Size int64 `json:"size"`
Version string `json:"version"` // same as release version
Extension string `json:"extension"` // "tar.gz", "dmg", "exe"
}
// rawRelease is the upstream JSON shape (stable as bool, files array).
type rawRelease struct {
Stable bool `json:"stable"`
Files []File `json:"files"`
}
// Fetch retrieves the Julia release index.
//
// Yields one batch containing all releases.
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]Release, error] {
return func(yield func([]Release, error) bool) {
url := "https://julialang-s3.julialang.org/bin/versions.json"
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
yield(nil, fmt.Errorf("juliadist: %w", err))
return
}
req.Header.Set("Accept", "application/json")
resp, err := client.Do(req)
if err != nil {
yield(nil, fmt.Errorf("juliadist: fetch: %w", err))
return
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
yield(nil, fmt.Errorf("juliadist: fetch: %s", resp.Status))
return
}
var raw map[string]rawRelease
if err := json.NewDecoder(resp.Body).Decode(&raw); err != nil {
yield(nil, fmt.Errorf("juliadist: decode: %w", err))
return
}
var releases []Release
for version, r := range raw {
releases = append(releases, Release{
Version: version,
Stable: r.Stable,
Files: r.Files,
})
}
yield(releases, nil)
}
}

View File

@@ -0,0 +1,23 @@
// Package lsd provides variant tagging for lsd (LSDeluxe) releases.
//
// lsd publishes .deb packages and windows-msvc builds alongside
// the standard archives.
package lsddist
import (
"strings"
"github.com/webinstall/webi-installers/internal/storage"
)
// TagVariants tags lsd-specific build variants.
func TagVariants(assets []storage.Asset) {
for i := range assets {
if assets[i].Format == ".deb" {
assets[i].Variants = append(assets[i].Variants, "deb")
}
if strings.Contains(strings.ToLower(assets[i].Filename), "-msvc") {
assets[i].Variants = append(assets[i].Variants, "msvc")
}
}
}

View File

@@ -0,0 +1,159 @@
// Package mariadbdist fetches MariaDB release data from the downloads API.
//
// MariaDB publishes release information via a REST API:
//
// https://downloads.mariadb.org/rest-api/mariadb/
// https://downloads.mariadb.org/rest-api/mariadb/{major.minor}/
//
// The first endpoint lists major release series; the second lists all point
// releases within a series, including download URLs per platform.
package mariadbdist
import (
"context"
"encoding/json"
"fmt"
"iter"
"net/http"
"regexp"
)
// MajorRelease describes one release series (e.g. "11.4").
type MajorRelease struct {
ReleaseID string `json:"release_id"` // "11.4"
ReleaseName string `json:"release_name"` // "MariaDB Server 11.4"
ReleaseStatus string `json:"release_status"` // "Stable", "RC", "Alpha"
ReleaseSupportType string `json:"release_support_type"` // "Long Term Support", etc.
}
// Release is one point release with its downloadable files.
type Release struct {
ReleaseID string `json:"release_id"` // "11.4.5"
ReleaseName string `json:"release_name"` // "MariaDB Server 11.4.5"
DateOfRelease string `json:"date_of_release"` // "2025-02-12"
ReleaseNotesURL string `json:"release_notes_url"` // URL
Files []File `json:"files"`
// MajorStatus is copied from the parent MajorRelease. Not in upstream JSON.
MajorStatus string `json:"major_status,omitempty"`
}
// File is one downloadable artifact within a release.
type File struct {
FileID int `json:"file_id"`
FileName string `json:"file_name"`
PackageType string `json:"package_type"` // "gzipped tar file", "ZIP file"
OS string `json:"os"` // "Linux", "Windows", or ""
CPU string `json:"cpu"` // "x86_64" or ""
Checksum Checksum `json:"checksum"`
FileDownloadURL string `json:"file_download_url"`
}
// Checksum holds hash digests for a file.
type Checksum struct {
SHA256 string `json:"sha256sum"`
}
type majorResp struct {
MajorReleases []MajorRelease `json:"major_releases"`
}
type releaseResp struct {
Releases map[string]Release `json:"releases"`
}
var reVersion = regexp.MustCompile(`^\d+\.\d+$`)
// Fetch retrieves all MariaDB releases across all major series.
//
// Yields one batch per major release series.
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]Release, error] {
return func(yield func([]Release, error) bool) {
// Step 1: list major release series.
majors, err := fetchMajors(ctx, client)
if err != nil {
yield(nil, err)
return
}
// Step 2: fetch point releases for each series.
for _, major := range majors {
if !reVersion.MatchString(major.ReleaseID) {
continue
}
releases, err := fetchReleases(ctx, client, major.ReleaseID)
if err != nil {
yield(nil, fmt.Errorf("mariadbdist: %s: %w", major.ReleaseID, err))
return
}
// Tag each release with the major status.
for i := range releases {
releases[i].MajorStatus = major.ReleaseStatus
}
if !yield(releases, nil) {
return
}
}
}
}
func fetchMajors(ctx context.Context, client *http.Client) ([]MajorRelease, error) {
url := "https://downloads.mariadb.org/rest-api/mariadb/"
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, fmt.Errorf("mariadbdist: %w", err)
}
req.Header.Set("Accept", "application/json")
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("mariadbdist: fetch majors: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("mariadbdist: fetch majors: %s", resp.Status)
}
var result majorResp
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
return nil, fmt.Errorf("mariadbdist: decode majors: %w", err)
}
return result.MajorReleases, nil
}
func fetchReleases(ctx context.Context, client *http.Client, majorID string) ([]Release, error) {
url := fmt.Sprintf("https://downloads.mariadb.org/rest-api/mariadb/%s", majorID)
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return nil, fmt.Errorf("mariadbdist: %w", err)
}
req.Header.Set("Accept", "application/json")
resp, err := client.Do(req)
if err != nil {
return nil, fmt.Errorf("mariadbdist: fetch %s: %w", majorID, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return nil, fmt.Errorf("mariadbdist: fetch %s: %s", majorID, resp.Status)
}
var result releaseResp
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
return nil, fmt.Errorf("mariadbdist: decode %s: %w", majorID, err)
}
var releases []Release
for _, r := range result.Releases {
releases = append(releases, r)
}
return releases, nil
}

View File

@@ -0,0 +1,39 @@
// Package node fetches Node.js releases from both official and unofficial
// build sources.
//
// Official builds cover the standard platforms (linux-x64, osx-arm64, win-x64,
// etc.). Unofficial builds add musl, loong64, and other targets that the
// official CI doesn't produce.
//
// Both sources use the same index format, served by [nodedist].
package nodedist
import (
"context"
"iter"
"net/http"
"github.com/webinstall/webi-installers/internal/releases/nodedist"
)
const (
officialURL = "https://nodejs.org/download/release"
unofficialURL = "https://unofficial-builds.nodejs.org/download/release"
)
// Fetch retrieves Node.js releases from both official and unofficial sources.
// Yields one batch per source (official first, then unofficial).
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]nodedist.Entry, error] {
return func(yield func([]nodedist.Entry, error) bool) {
for entries, err := range nodedist.Fetch(ctx, client, officialURL) {
if !yield(entries, err) {
return
}
}
for entries, err := range nodedist.Fetch(ctx, client, unofficialURL) {
if !yield(entries, err) {
return
}
}
}
}

View File

@@ -0,0 +1,36 @@
package nodedist_test
import (
"context"
"net/http"
"testing"
"github.com/webinstall/webi-installers/internal/releases/node"
)
func TestFetchCombinesSources(t *testing.T) {
if testing.Short() {
t.Skip("skipping network test in short mode")
}
ctx := context.Background()
client := &http.Client{}
var batches int
var total int
for entries, err := range nodedist.Fetch(ctx, client) {
if err != nil {
t.Fatalf("batch %d: %v", batches, err)
}
batches++
total += len(entries)
}
if batches != 2 {
t.Errorf("got %d batches, want 2 (official + unofficial)", batches)
}
if total < 100 {
t.Errorf("got %d total entries, expected at least 100", total)
}
t.Logf("fetched %d entries in %d batches", total, batches)
}

View File

@@ -0,0 +1,20 @@
package nodedist
import "github.com/webinstall/webi-installers/internal/storage"
// TagVariants tags node-specific build variants.
//
// The bare .exe is just node.exe without npm — too minimal to be useful.
// The .msi is a Windows GUI installer — webi uses the .zip instead.
// The .pkg is a macOS installer package — webi uses the .tar.gz instead.
// Both are tagged as "installer" so ExportLegacy drops them.
func TagVariants(assets []storage.Asset) {
for i := range assets {
switch assets[i].Format {
case ".exe":
assets[i].Variants = append(assets[i].Variants, "bare-exe")
case ".msi", ".pkg":
assets[i].Variants = append(assets[i].Variants, "installer")
}
}
}

View File

@@ -0,0 +1,108 @@
// Package nodedist fetches a Node.js-style distribution index.
//
// Node.js publishes a JSON index of all releases at:
//
// https://nodejs.org/download/release/index.json
//
// Unofficial builds (musl, etc.) use the same format at:
//
// https://unofficial-builds.nodejs.org/download/release/index.json
//
// This package fetches and deserializes that index. It does not classify,
// normalize, or transform the data — the caller gets what the API returns.
package nodedist
import (
"context"
"encoding/json"
"fmt"
"iter"
"net/http"
)
// Entry is one release from a Node.js distribution index.
// Fields mirror the upstream JSON schema.
type Entry struct {
Version string `json:"version"` // "v25.8.0"
Date string `json:"date"` // "2026-03-03"
Files []string `json:"files"` // ["linux-arm64", "osx-arm64-tar", ...]
NPM string `json:"npm"` // "11.11.0"
V8 string `json:"v8"` // "14.1.146.11"
UV string `json:"uv"` // "1.51.0"
Zlib string `json:"zlib"` // "1.3.1"
OpenSSL string `json:"openssl"` // "3.5.5"
Modules string `json:"modules"` // "141"
LTS LTS `json:"lts"` // false or "Jod"
Security bool `json:"security"` // true if security release
}
// LTS holds the long-term support status. The upstream API encodes this as
// either the boolean false or a codename string like "Jod" or "Iron".
// An empty string means the release is not LTS.
type LTS string
func (l *LTS) UnmarshalJSON(data []byte) error {
// false → ""
if string(data) == "false" {
*l = ""
return nil
}
// "Codename" → Codename
var s string
if err := json.Unmarshal(data, &s); err != nil {
return fmt.Errorf("nodedist: unexpected lts value: %s", data)
}
*l = LTS(s)
return nil
}
func (l LTS) MarshalJSON() ([]byte, error) {
if l == "" {
return []byte("false"), nil
}
return json.Marshal(string(l))
}
// Fetch retrieves the Node.js distribution index from baseURL.
//
// The iterator yields one batch per HTTP response. The Node.js index API
// returns all releases in a single response, so there will be exactly one
// yield. The iterator interface exists so that callers use the same pattern
// for paginated sources (like GitHub).
//
// Standard base URLs:
// - https://nodejs.org/download/release
// - https://unofficial-builds.nodejs.org/download/release
func Fetch(ctx context.Context, client *http.Client, baseURL string) iter.Seq2[[]Entry, error] {
return func(yield func([]Entry, error) bool) {
url := baseURL + "/index.json"
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
yield(nil, fmt.Errorf("nodedist: %w", err))
return
}
req.Header.Set("Accept", "application/json")
resp, err := client.Do(req)
if err != nil {
yield(nil, fmt.Errorf("nodedist: fetch %s: %w", url, err))
return
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
yield(nil, fmt.Errorf("nodedist: fetch %s: %s", url, resp.Status))
return
}
var entries []Entry
if err := json.NewDecoder(resp.Body).Decode(&entries); err != nil {
yield(nil, fmt.Errorf("nodedist: decode %s: %w", url, err))
return
}
yield(entries, nil)
}
}

View File

@@ -0,0 +1,143 @@
package nodedist_test
import (
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"testing"
"github.com/webinstall/webi-installers/internal/releases/nodedist"
)
// Minimal fixture from the real Node.js dist API.
const testIndex = `[
{
"version": "v22.14.0",
"date": "2025-02-11",
"files": ["linux-arm64", "linux-x64", "osx-arm64-tar", "win-x64-zip", "src", "headers"],
"npm": "10.9.2",
"v8": "12.4.254.21",
"uv": "1.49.2",
"zlib": "1.3.0.1-motley-82a6be0",
"openssl": "3.0.15+quic",
"modules": "127",
"lts": "Jod",
"security": false
},
{
"version": "v23.7.0",
"date": "2025-02-04",
"files": ["linux-arm64", "linux-x64", "osx-arm64-tar", "win-x64-zip"],
"npm": "10.9.2",
"v8": "13.2.152.16",
"uv": "1.49.2",
"zlib": "1.3.0.1-motley-82a6be0",
"openssl": "3.0.15+quic",
"modules": "131",
"lts": false,
"security": true
}
]`
func TestFetch(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path != "/index.json" {
t.Errorf("unexpected path: %s", r.URL.Path)
http.NotFound(w, r)
return
}
w.Header().Set("Content-Type", "application/json")
w.Write([]byte(testIndex))
}))
defer srv.Close()
ctx := context.Background()
var got []nodedist.Entry
for entries, err := range nodedist.Fetch(ctx, srv.Client(), srv.URL) {
if err != nil {
t.Fatalf("Fetch: %v", err)
}
got = append(got, entries...)
}
if len(got) != 2 {
t.Fatalf("got %d entries, want 2", len(got))
}
// First entry: LTS release
if got[0].Version != "v22.14.0" {
t.Errorf("entry[0].Version = %q, want %q", got[0].Version, "v22.14.0")
}
if got[0].Date != "2025-02-11" {
t.Errorf("entry[0].Date = %q, want %q", got[0].Date, "2025-02-11")
}
if got[0].LTS != "Jod" {
t.Errorf("entry[0].LTS = %q, want %q", got[0].LTS, "Jod")
}
if got[0].Security {
t.Error("entry[0].Security = true, want false")
}
if len(got[0].Files) != 6 {
t.Errorf("entry[0].Files len = %d, want 6", len(got[0].Files))
}
// Second entry: non-LTS, security release
if got[1].Version != "v23.7.0" {
t.Errorf("entry[1].Version = %q, want %q", got[1].Version, "v23.7.0")
}
if got[1].LTS != "" {
t.Errorf("entry[1].LTS = %q, want empty (non-LTS)", got[1].LTS)
}
if !got[1].Security {
t.Error("entry[1].Security = false, want true")
}
}
func TestFetchHTTPError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
http.Error(w, "rate limited", http.StatusTooManyRequests)
}))
defer srv.Close()
ctx := context.Background()
for _, err := range nodedist.Fetch(ctx, srv.Client(), srv.URL) {
if err == nil {
t.Fatal("expected error for 429 response")
}
return
}
}
func TestLTSMarshalRoundTrip(t *testing.T) {
// LTS codename
entry := nodedist.Entry{LTS: "Jod"}
data, err := json.Marshal(entry)
if err != nil {
t.Fatal(err)
}
var got nodedist.Entry
if err := json.Unmarshal(data, &got); err != nil {
t.Fatal(err)
}
if got.LTS != "Jod" {
t.Errorf("LTS roundtrip: got %q, want %q", got.LTS, "Jod")
}
// Non-LTS
entry2 := nodedist.Entry{LTS: ""}
data2, err := json.Marshal(entry2)
if err != nil {
t.Fatal(err)
}
var got2 nodedist.Entry
if err := json.Unmarshal(data2, &got2); err != nil {
t.Fatal(err)
}
if got2.LTS != "" {
t.Errorf("non-LTS roundtrip: got %q, want empty", got2.LTS)
}
}

View File

@@ -0,0 +1,32 @@
// Package ollama provides variant tagging for Ollama releases.
//
// Ollama publishes GPU accelerator builds: -rocm (AMD), -jetpack5
// and -jetpack6 (NVIDIA Jetson).
package ollamadist
import (
"strings"
"github.com/webinstall/webi-installers/internal/storage"
)
// TagVariants tags ollama-specific build variants.
func TagVariants(assets []storage.Asset) {
for i := range assets {
lower := strings.ToLower(assets[i].Filename)
for _, v := range []string{"rocm", "jetpack5", "jetpack6"} {
if strings.Contains(lower, "-"+v) {
assets[i].Variants = append(assets[i].Variants, v)
}
}
// Ollama-darwin.zip (capital O) is the macOS .app bundle.
// Installable by Go (extract .app), but not in legacy cache.
if strings.HasPrefix(assets[i].Filename, "Ollama-") {
assets[i].Variants = append(assets[i].Variants, "app")
}
// ollama-darwin is a universal2 fat binary (arm64 + amd64).
if assets[i].OS == "darwin" && assets[i].Arch == "" {
assets[i].Arch = "universal2"
}
}
}

View File

@@ -0,0 +1,80 @@
package postgres
import (
"strings"
"github.com/webinstall/webi-installers/internal/storage"
)
// NormalizeVersions strips the REL_ prefix and converts underscores to dots.
// GitHub tags are "REL_17_0" → version becomes "17.0".
func NormalizeVersions(assets []storage.Asset) {
for i := range assets {
v := strings.TrimPrefix(assets[i].Version, "REL_")
assets[i].Version = strings.ReplaceAll(v, "_", ".")
}
}
// LegacyReleases returns the old EnterpriseDB binary releases that predate
// the bnnanet/postgresql-releases GitHub repo.
func LegacyReleases() []storage.Asset {
edbURL := "https://get.enterprisedb.com/postgresql/"
return []storage.Asset{
{
Filename: "postgresql-10.12-1-linux-x64-binaries.tar.gz",
Version: "10.12",
Channel: "stable",
OS: "linux",
Arch: "x86_64",
Libc: "gnu",
Format: ".tar.gz",
Download: edbURL + "postgresql-10.12-1-linux-x64-binaries.tar.gz?ls=Crossover&type=Crossover",
},
{
Filename: "postgresql-10.12-1-linux-binaries.tar.gz",
Version: "10.12",
Channel: "stable",
OS: "linux",
Arch: "x86",
Libc: "gnu",
Format: ".tar.gz",
Download: edbURL + "postgresql-10.12-1-linux-binaries.tar.gz?ls=Crossover&type=Crossover",
},
{
Filename: "postgresql-10.12-1-osx-binaries.zip",
Version: "10.12",
Channel: "stable",
OS: "darwin",
Arch: "x86_64",
Format: ".zip",
Download: edbURL + "postgresql-10.12-1-osx-binaries.zip?ls=Crossover&type=Crossover",
},
{
Filename: "postgresql-10.13-1-osx-binaries.zip",
Version: "10.13",
Channel: "stable",
OS: "darwin",
Arch: "x86_64",
Format: ".zip",
Download: edbURL + "postgresql-10.13-1-osx-binaries.zip?ls=Crossover&type=Crossover",
},
{
Filename: "postgresql-11.8-1-osx-binaries.zip",
Version: "11.8",
Channel: "stable",
OS: "darwin",
Arch: "x86_64",
Format: ".zip",
Download: edbURL + "postgresql-11.8-1-osx-binaries.zip?ls=Crossover&type=Crossover",
},
{
Filename: "postgresql-12.3-1-osx-binaries.zip",
Version: "12.3",
Channel: "stable",
OS: "darwin",
Arch: "x86_64",
Format: ".zip",
Download: edbURL + "postgresql-12.3-1-osx-binaries.zip?ls=Crossover&type=Crossover",
},
}
}

View File

@@ -0,0 +1,37 @@
// Package pwsh provides variant tagging for PowerShell releases.
//
// PowerShell publishes .NET framework-dependent builds (-fxdependent)
// that are smaller but require a .NET runtime to be installed.
package pwshdist
import (
"regexp"
"strings"
"github.com/webinstall/webi-installers/internal/storage"
)
// winVersionRe matches Windows-version-specific filenames like
// "win10-win2016-x64" or "win81-x64" from early PowerShell releases.
var winVersionRe = regexp.MustCompile(`(?i)-win(?:7|8|81|10|2008|2012|2016)`)
// TagVariants tags pwsh-specific build variants.
//
// Early releases (pre-6.1) used Windows-version-specific filenames
// like "win10-win2016-x64" and "win81-win2012r2-x64". These can't
// be resolved by the legacy cache and are tagged as variants.
func TagVariants(assets []storage.Asset) {
for i := range assets {
lower := strings.ToLower(assets[i].Filename)
switch {
case strings.Contains(lower, "-fxdependentwindesktop"):
assets[i].Variants = append(assets[i].Variants, "fxdependentWinDesktop")
case strings.Contains(lower, "-fxdependent"):
assets[i].Variants = append(assets[i].Variants, "fxdependent")
case winVersionRe.MatchString(lower):
assets[i].Variants = append(assets[i].Variants, "win-version-specific")
case strings.HasSuffix(lower, ".appimage"):
assets[i].Variants = append(assets[i].Variants, "appimage")
}
}
}

View File

@@ -0,0 +1,19 @@
// Package sass provides variant tagging for Dart Sass releases.
//
// Dart Sass uses bare "arm" in filenames to mean ARMv7 (the Dart VM's
// minimum ARM target). The generic classifier maps bare "arm" to armv6,
// so we correct it here.
package sassdist
import (
"github.com/webinstall/webi-installers/internal/storage"
)
// TagVariants remaps bare arm → armv7 for Dart Sass assets.
func TagVariants(assets []storage.Asset) {
for i := range assets {
if assets[i].Arch == "armv6" {
assets[i].Arch = "armv7"
}
}
}

View File

@@ -0,0 +1,75 @@
// Package servicemandist fetches serviceman releases from two GitHub repos.
//
// serviceman moved from therootcompany/serviceman (binary cross-platform
// releases, ≤v0.8.x) to bnnanet/serviceman (source-only POSIX, v0.9.x+).
// Both repos must be fetched to provide the complete version history,
// including the only Windows binary at v0.8.0.
package servicemandist
import (
"context"
"encoding/json"
"fmt"
"log"
"net/http"
"path/filepath"
"github.com/webinstall/webi-installers/internal/rawcache"
"github.com/webinstall/webi-installers/internal/releases/github"
"github.com/webinstall/webi-installers/internal/releases/githubish"
)
const (
primaryOwner = "bnnanet"
primaryRepo = "serviceman"
legacyOwner = "therootcompany"
legacyRepo = "serviceman"
)
// Fetch retrieves serviceman releases from both GitHub repos and merges
// them into the raw cache. The primary repo (bnnanet) contains v0.9.x+;
// the legacy repo (therootcompany) contains ≤v0.8.x with Windows binaries.
func Fetch(ctx context.Context, client *http.Client, rawDir, pkgName string, auth *githubish.Auth, shallow bool) error {
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
if err != nil {
return err
}
// Primary: bnnanet/serviceman (v0.9.x+ source tarballs).
for batch, err := range github.Fetch(ctx, client, primaryOwner, primaryRepo, auth) {
if err != nil {
return fmt.Errorf("servicemandist: %s/%s: %w", primaryOwner, primaryRepo, err)
}
for _, rel := range batch {
if rel.Draft {
continue
}
data, _ := json.Marshal(rel)
d.Merge(primaryOwner+"/"+rel.TagName, data)
}
if shallow {
break
}
}
// Legacy: therootcompany/serviceman (≤v0.8.x binaries).
for batch, err := range github.Fetch(ctx, client, legacyOwner, legacyRepo, auth) {
if err != nil {
log.Printf("warning: servicemandist: %s/%s: %v", legacyOwner, legacyRepo, err)
break
}
for _, rel := range batch {
if rel.Draft {
continue
}
data, _ := json.Marshal(rel)
d.Merge(legacyOwner+"/"+rel.TagName, data)
}
if shallow {
break
}
}
return nil
}

View File

@@ -0,0 +1,16 @@
package servicemandist
import (
"github.com/webinstall/webi-installers/internal/storage"
)
// TagVariants marks all git-format entries as POSIX-only.
// serviceman's git clone installs a POSIX shell script — no Windows support.
// Binary releases (v0.8.x tar.gz/zip) already have per-platform OS set.
func TagVariants(assets []storage.Asset) {
for i := range assets {
if assets[i].Format == "git" && assets[i].OS == "" {
assets[i].OS = "posix_2017"
}
}
}

View File

@@ -0,0 +1,36 @@
// Package sttr provides variant tagging for sttr releases.
//
// sttr ships a darwin_all (universal macOS) archive alongside per-arch builds.
// These universal archives have no arch in the filename — Go classifies them as
// os="darwin", arch="" which the Node builds-cacher rejects with FORMAT CHANGE
// (Node's classifier extracts a different arch from "all"). Production Node
// also stores these as os="", arch="" (unroutable).
//
// .sbom.json files are software bill-of-materials metadata — not installable
// archives. They pass through the format filter (ext="") but should not be
// served.
package sttrdist
import (
"strings"
"github.com/webinstall/webi-installers/internal/storage"
)
// TagVariants tags sttr-specific build variants for exclusion from legacy export.
func TagVariants(assets []storage.Asset) {
for i := range assets {
lower := strings.ToLower(assets[i].Filename)
// darwin_all / Darwin_all: universal macOS archive with no arch info.
// Node's classifier extracts a different result → FORMAT CHANGE.
// Production LIVE_cache has these as os="", arch="" (unroutable).
if strings.Contains(lower, "darwin_all") {
assets[i].Variants = append(assets[i].Variants, "universal-all")
continue
}
// .sbom.json: software bill-of-materials, not an installable archive.
if strings.HasSuffix(lower, ".sbom.json") {
assets[i].Variants = append(assets[i].Variants, "metadata")
}
}
}

View File

@@ -0,0 +1,18 @@
// Package uuidv7 provides variant tagging for uuidv7 releases.
package uuidv7dist
import "github.com/webinstall/webi-installers/internal/storage"
// TagVariants tags uuidv7-specific build variants for exclusion from legacy export.
//
// uuidv7 ships powerpc (32-bit) and powerpc64 binaries alongside the common
// platforms. Webi does not serve powerpc targets, and production Node also
// classifies these as os="", arch="" (not routable). Tag them unsupported.
func TagVariants(assets []storage.Asset) {
for i := range assets {
switch assets[i].Arch {
case "powerpc", "ppc64", "ppc64le":
assets[i].Variants = append(assets[i].Variants, "unsupported-platform")
}
}
}

View File

@@ -0,0 +1,18 @@
// Package watchexec provides variant tagging and version normalization for watchexec.
package watchexecdist
import "github.com/webinstall/webi-installers/internal/storage"
// TagVariants tags watchexec-specific build variants for exclusion from legacy export.
//
// Watchexec ships powerpc64le binaries alongside the common platforms.
// Webi does not serve powerpc targets, and production Node also classifies
// these as os="", arch="" (not routable). Tag them unsupported.
func TagVariants(assets []storage.Asset) {
for i := range assets {
switch assets[i].Arch {
case "powerpc", "ppc64", "ppc64le":
assets[i].Variants = append(assets[i].Variants, "unsupported-platform")
}
}
}

View File

@@ -0,0 +1,18 @@
package watchexecdist
import (
"strings"
"github.com/webinstall/webi-installers/internal/storage"
)
// NormalizeVersions strips the "cli-" prefix from watchexec version strings.
//
// Watchexec transitioned to a monorepo with cli-prefixed tags (cli-v1.20.0)
// while older releases used plain tags (v1.20.6). Both are valid releases;
// the prefix is just a tag namespace, not part of the version.
func NormalizeVersions(assets []storage.Asset) {
for i := range assets {
assets[i].Version = strings.TrimPrefix(assets[i].Version, "cli-")
}
}

View File

@@ -0,0 +1,15 @@
// Package xcaddy provides variant tagging for xcaddy releases.
//
// xcaddy publishes .deb packages alongside the standard archives.
package xcaddydist
import "github.com/webinstall/webi-installers/internal/storage"
// TagVariants tags xcaddy-specific build variants.
func TagVariants(assets []storage.Asset) {
for i := range assets {
if assets[i].Format == ".deb" {
assets[i].Variants = append(assets[i].Variants, "deb")
}
}
}

View File

@@ -0,0 +1,16 @@
package xzdist
import "github.com/webinstall/webi-installers/internal/storage"
// TagVariants handles xz-specific arch defaults.
//
// therootcompany/xz-static names builds xz-{version}-{os}-{arch} for
// Linux/macOS but xz-{version}-windows.zip for Windows (only amd64
// shipped). The arch token is absent only for the Windows build.
func TagVariants(assets []storage.Asset) {
for i := range assets {
if assets[i].Arch == "" && assets[i].OS == "windows" {
assets[i].Arch = "x86_64"
}
}
}

View File

@@ -0,0 +1,131 @@
// Package zigdist fetches Zig release data from ziglang.org.
//
// The API is a single JSON object keyed by version or branch name:
//
// https://ziglang.org/download/index.json
//
// Each version key maps to an object containing "date", "notes", and
// platform keys like "x86_64-linux", "aarch64-macos", etc. Platform
// values have "tarball", "shasum", and "size" fields.
package zigdist
import (
"context"
"encoding/json"
"fmt"
"iter"
"net/http"
)
// Release is one Zig version with its per-platform builds.
type Release struct {
Version string `json:"version"` // set by us from the key or inner "version" field
Date string `json:"date"`
Notes string `json:"notes,omitempty"`
Platforms map[string]Platform `json:"platforms,omitempty"` // "x86_64-linux" → Platform
}
// Platform is one downloadable artifact for a specific arch-os combo.
type Platform struct {
Tarball string `json:"tarball"`
Shasum string `json:"shasum"`
Size json.Number `json:"size"` // upstream sends as string
}
// Fetch retrieves the Zig release index.
//
// Yields one batch containing all releases. The iterator interface exists
// so callers use the same pattern as paginated sources.
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]Release, error] {
return func(yield func([]Release, error) bool) {
url := "https://ziglang.org/download/index.json"
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
yield(nil, fmt.Errorf("zigdist: %w", err))
return
}
req.Header.Set("Accept", "application/json")
resp, err := client.Do(req)
if err != nil {
yield(nil, fmt.Errorf("zigdist: fetch: %w", err))
return
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
yield(nil, fmt.Errorf("zigdist: fetch: %s", resp.Status))
return
}
// The JSON is an object keyed by version/branch name.
var raw map[string]json.RawMessage
if err := json.NewDecoder(resp.Body).Decode(&raw); err != nil {
yield(nil, fmt.Errorf("zigdist: decode: %w", err))
return
}
var releases []Release
for ref, data := range raw {
rel, err := parseRelease(ref, data)
if err != nil {
yield(nil, fmt.Errorf("zigdist: parse %s: %w", ref, err))
return
}
releases = append(releases, rel)
}
yield(releases, nil)
}
}
// parseRelease extracts a Release from one version entry. The JSON mixes
// metadata fields ("date", "notes", "version", "src") with platform keys
// ("x86_64-linux", "aarch64-macos", etc.).
func parseRelease(ref string, data json.RawMessage) (Release, error) {
// First pass: grab known metadata fields.
var meta struct {
Version string `json:"version"`
Date string `json:"date"`
Notes string `json:"notes"`
}
if err := json.Unmarshal(data, &meta); err != nil {
return Release{}, err
}
version := meta.Version
if version == "" {
version = ref
}
// Second pass: grab all platform entries.
var all map[string]json.RawMessage
if err := json.Unmarshal(data, &all); err != nil {
return Release{}, err
}
platforms := make(map[string]Platform)
for key, val := range all {
// Skip metadata keys.
switch key {
case "version", "date", "notes", "src":
continue
}
var p Platform
if err := json.Unmarshal(val, &p); err != nil {
continue // not a platform object
}
if p.Tarball == "" {
continue // not a platform object
}
platforms[key] = p
}
return Release{
Version: version,
Date: meta.Date,
Notes: meta.Notes,
Platforms: platforms,
}, nil
}

264
internal/render/render.go Normal file
View File

@@ -0,0 +1,264 @@
// Package render generates installer scripts by injecting release
// metadata into the package-install template.
//
// The template uses shell-style variable markers:
//
// #WEBI_VERSION= → WEBI_VERSION='1.2.3'
// #export WEBI_PKG_URL= → export WEBI_PKG_URL='https://...'
//
// The package's install.sh is injected at the {{ installer }} marker.
package render
import (
"fmt"
"os"
"path/filepath"
"regexp"
"strings"
)
// Params holds all the values to inject into the installer template.
type Params struct {
// Host is the base URL of the webi server (e.g. "https://webinstall.dev").
Host string
// Checksum is the webi.sh bootstrap script checksum (first 8 hex chars of SHA-1).
Checksum string
// Package name (e.g. "bat", "node").
PkgName string
// Tag is the version selector from the URL (e.g. "20", "stable", "").
Tag string
// OS, Arch, Libc are the detected platform strings.
OS string
Arch string
Libc string
// Resolved release info.
Version string
Major string
Minor string
Patch string
Build string
GitTag string
GitBranch string
LTS string // "true" or "false"
Channel string
Ext string // archive extension (e.g. "tar.gz", "zip")
Formats string // comma-separated format list
// Download info.
PkgURL string // download URL
PkgFile string // filename
// Releases API URL for this request.
ReleasesURL string
// CSV line for WEBI_CSV.
CSV string
// Package catalog info.
PkgStable string
PkgLatest string
PkgOSes string // space-separated
PkgArches string // space-separated
PkgLibcs string // space-separated
PkgFormats string // space-separated
}
// Bash renders a complete bash installer script by injecting params
// into the template and splicing in the package's install.sh.
func Bash(tplPath, installersDir, pkgName string, p Params) (string, error) {
tpl, err := os.ReadFile(tplPath)
if err != nil {
return "", fmt.Errorf("render: read template: %w", err)
}
// Read the package's install.sh.
installPath := filepath.Join(installersDir, pkgName, "install.sh")
installSh, err := os.ReadFile(installPath)
if err != nil {
return "", fmt.Errorf("render: read %s/install.sh: %w", pkgName, err)
}
text := string(tpl)
// Inject environment variables.
vars := []struct {
name string
value string
}{
{"WEBI_CHECKSUM", p.Checksum},
{"WEBI_PKG", p.PkgName + "@" + p.Tag},
{"WEBI_HOST", p.Host},
{"WEBI_OS", p.OS},
{"WEBI_ARCH", p.Arch},
{"WEBI_LIBC", p.Libc},
{"WEBI_TAG", p.Tag},
{"WEBI_RELEASES", p.ReleasesURL},
{"WEBI_CSV", p.CSV},
{"WEBI_VERSION", p.Version},
{"WEBI_MAJOR", p.Major},
{"WEBI_MINOR", p.Minor},
{"WEBI_PATCH", p.Patch},
{"WEBI_BUILD", p.Build},
{"WEBI_GIT_BRANCH", p.GitBranch},
{"WEBI_GIT_TAG", p.GitTag},
{"WEBI_LTS", p.LTS},
{"WEBI_CHANNEL", p.Channel},
{"WEBI_EXT", p.Ext},
{"WEBI_FORMATS", p.Formats},
{"WEBI_PKG_URL", p.PkgURL},
{"WEBI_PKG_PATHNAME", p.PkgFile},
{"WEBI_PKG_FILE", p.PkgFile},
{"PKG_NAME", p.PkgName},
{"PKG_STABLE", p.PkgStable},
{"PKG_LATEST", p.PkgLatest},
{"PKG_OSES", p.PkgOSes},
{"PKG_ARCHES", p.PkgArches},
{"PKG_LIBCS", p.PkgLibcs},
{"PKG_FORMATS", p.PkgFormats},
}
for _, v := range vars {
text = InjectVar(text, v.name, v.value)
}
// Inject the installer script at the {{ installer }} marker.
// The marker sits inside __init_installer() at 8-space indent.
// Production pads every line of install.sh to match, and replaces
// the entire line (including leading whitespace).
padded := padScript(string(installSh), " ")
text = replaceMarkerLine(text, "{{ installer }}", padded)
return text, nil
}
// PowerShell renders a complete PowerShell installer script by injecting
// params into the template and splicing in the package's install.ps1.
func PowerShell(tplPath, installersDir, pkgName string, p Params) (string, error) {
tpl, err := os.ReadFile(tplPath)
if err != nil {
return "", fmt.Errorf("render: read template: %w", err)
}
installPath := filepath.Join(installersDir, pkgName, "install.ps1")
installPs1, err := os.ReadFile(installPath)
if err != nil {
return "", fmt.Errorf("render: read %s/install.ps1: %w", pkgName, err)
}
text := string(tpl)
vars := []struct {
name string
value string
}{
{"WEBI_PKG", p.PkgName + "@" + p.Tag},
{"WEBI_HOST", p.Host},
{"WEBI_VERSION", p.Version},
{"WEBI_GIT_TAG", p.GitTag},
{"WEBI_PKG_URL", p.PkgURL},
{"WEBI_PKG_FILE", p.PkgFile},
{"WEBI_PKG_PATHNAME", p.PkgFile},
{"PKG_NAME", p.PkgName},
}
for _, v := range vars {
text = InjectPSVar(text, v.name, v.value)
}
// PS1 marker is at column 0, no padding needed.
text = replaceMarkerLine(text, "{{ installer }}", string(installPs1))
return text, nil
}
// InjectPSVar replaces a PowerShell template variable line with its value.
// Matches lines like:
//
// #$Env:WEBI_VERSION = v12.16.2
// $Env:WEBI_HOST = 'https://webinstall.dev'
func InjectPSVar(text, name, value string) string {
p := getPSVarPattern(name)
return p.ReplaceAllString(text, "${1}$$Env:"+name+" = '"+sanitizePSValue(value)+"'")
}
var psVarPatterns = map[string]*regexp.Regexp{}
func getPSVarPattern(name string) *regexp.Regexp {
if p, ok := psVarPatterns[name]; ok {
return p
}
// Match: optional leading whitespace, optional #, $Env:NAME, =, rest of line
p := regexp.MustCompile(`(?m)^([ \t]*)#?\$Env:` + regexp.QuoteMeta(name) + `\s*=.*$`)
psVarPatterns[name] = p
return p
}
// sanitizePSValue escapes single quotes for PowerShell single-quoted strings.
// In PowerShell, single quotes inside single-quoted strings are doubled: ''
func sanitizePSValue(s string) string {
return strings.ReplaceAll(s, "'", "''")
}
// varPattern matches shell variable declarations in the template.
// Matches lines like:
//
// #WEBI_VERSION=
// #export WEBI_PKG_URL=
// #WEBI_OS=
var varPatterns = map[string]*regexp.Regexp{}
func getVarPattern(name string) *regexp.Regexp {
if p, ok := varPatterns[name]; ok {
return p
}
// Match: optional leading whitespace, optional #, optional export, the var name, =, rest of line
p := regexp.MustCompile(`(?m)^([ \t]*)#?([ \t]*)(export[ \t]+)?[ \t]*(` + regexp.QuoteMeta(name) + `)=.*$`)
varPatterns[name] = p
return p
}
// InjectVar replaces a template variable line with its value.
// It matches lines like:
//
// #WEBI_VERSION=
// #export WEBI_PKG_URL=
// export WEBI_HOST=
//
// and replaces them with the value in single quotes.
func InjectVar(text, name, value string) string {
p := getVarPattern(name)
return p.ReplaceAllString(text, "${1}${3}"+name+"='"+sanitizeShellValue(value)+"'")
}
// sanitizeShellValue ensures a value is safe to embed in single quotes.
// Single quotes in shell can't be escaped inside single quotes, so we
// close-quote, add escaped quote, re-open quote: 'foo'\''bar'
func sanitizeShellValue(s string) string {
return strings.ReplaceAll(s, "'", `'\''`)
}
// padScript prepends each line of a script with the given indent string.
// This matches production behavior where install.sh content is indented
// to align with the surrounding template code.
func padScript(script, indent string) string {
lines := strings.Split(script, "\n")
for i, line := range lines {
if line != "" {
lines[i] = indent + line
}
}
return strings.Join(lines, "\n")
}
// replaceMarkerLine replaces an entire line containing the marker
// (including any leading whitespace) with the replacement text.
// This matches production's regex: /\s*#?\s*{{ installer }}/
func replaceMarkerLine(text, marker, replacement string) string {
re := regexp.MustCompile(`(?m)^[ \t]*#?[ \t]*` + regexp.QuoteMeta(marker) + `[^\n]*`)
return re.ReplaceAllLiteralString(text, replacement)
}

View File

@@ -0,0 +1,90 @@
package render
import (
"strings"
"testing"
)
func TestInjectVar(t *testing.T) {
tests := []struct {
name string
input string
key string
value string
want string
}{
{
name: "commented var",
input: " #WEBI_VERSION=",
key: "WEBI_VERSION",
value: "1.2.3",
want: " WEBI_VERSION='1.2.3'",
},
{
name: "commented export var",
input: " #export WEBI_PKG_URL=",
key: "WEBI_PKG_URL",
value: "https://example.com/foo.tar.gz",
want: " export WEBI_PKG_URL='https://example.com/foo.tar.gz'",
},
{
name: "existing value replaced",
input: " export WEBI_HOST=",
key: "WEBI_HOST",
value: "https://webinstall.dev",
want: " export WEBI_HOST='https://webinstall.dev'",
},
{
name: "value with single quotes",
input: " #PKG_NAME=",
key: "PKG_NAME",
value: "it's-a-test",
want: " PKG_NAME='it'\\''s-a-test'",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := InjectVar(tt.input, tt.key, tt.value)
if strings.TrimSpace(got) != strings.TrimSpace(tt.want) {
t.Errorf("got %q\nwant %q", got, tt.want)
}
})
}
}
func TestInjectVarInTemplate(t *testing.T) {
tpl := `#!/bin/sh
__bootstrap_webi() {
#PKG_NAME=
#WEBI_OS=
#WEBI_ARCH=
#WEBI_VERSION=
export WEBI_HOST=
WEBI_PKG_DOWNLOAD=""
`
result := tpl
result = InjectVar(result, "PKG_NAME", "bat")
result = InjectVar(result, "WEBI_OS", "linux")
result = InjectVar(result, "WEBI_ARCH", "x86_64")
result = InjectVar(result, "WEBI_VERSION", "0.26.1")
result = InjectVar(result, "WEBI_HOST", "https://webinstall.dev")
if !strings.Contains(result, "PKG_NAME='bat'") {
t.Error("PKG_NAME not injected")
}
if !strings.Contains(result, "WEBI_OS='linux'") {
t.Error("WEBI_OS not injected")
}
if !strings.Contains(result, "WEBI_VERSION='0.26.1'") {
t.Error("WEBI_VERSION not injected")
}
if !strings.Contains(result, "export WEBI_HOST='https://webinstall.dev'") {
t.Error("WEBI_HOST not injected")
}
// Should not have #PKG_NAME= anymore.
if strings.Contains(result, "#PKG_NAME=") {
t.Error("#PKG_NAME= should have been replaced")
}
}

303
internal/resolve/resolve.go Normal file
View File

@@ -0,0 +1,303 @@
// Package resolve picks the best release for a given platform query.
//
// Given a set of classified distributables and a target query (OS, arch,
// libc, format preferences, version constraint), it returns the single
// best matching release — or nil if nothing matches.
package resolve
import (
"strings"
"github.com/webinstall/webi-installers/internal/buildmeta"
"github.com/webinstall/webi-installers/internal/lexver"
)
// Dist is one downloadable distributable — matches the CSV row from classify.
type Dist struct {
Package string
Version string
Channel string
OS string
Arch string
Libc string
Format string
Download string
Filename string
SHA256 string
Size int64
LTS bool
Date string
Extra string // extra version info for sorting
GitTag string // original git tag or branch — only for format="git"
GitCommitHash string // short commit hash — only for format="git"
Variants []string // build qualifiers: "installer", "rocm", "fxdependent", etc.
}
// Query describes what the caller wants.
type Query struct {
OS buildmeta.OS
Arch buildmeta.Arch
Libc buildmeta.Libc
Formats []string // acceptable formats (e.g. ".tar.gz", ".zip"), in preference order
Channel string // "stable" (default), "beta", etc.
Version string // version prefix constraint ("24", "24.14", ""), empty = latest
Variants []string // if non-empty, only match assets with these variants
}
// Match is the resolved release.
type Match struct {
Version string
OS string
Arch string
Libc string
Format string
Download string
Filename string
LTS bool
Date string
Channel string
}
// Best finds the single best release matching the query.
// Returns nil if nothing matches.
func Best(dists []Dist, q Query) *Match {
channel := q.Channel
if channel == "" {
channel = "stable"
}
// Build format set for fast lookup + rank map for preference.
formatRank := make(map[string]int, len(q.Formats))
for i, f := range q.Formats {
formatRank[f] = i
}
// Build the set of acceptable architectures (native + compat).
compatArches := buildmeta.CompatArches(q.OS, q.Arch)
archRank := make(map[string]int, len(compatArches))
for i, a := range compatArches {
archRank[string(a)] = i
}
// Parse version prefix for constraint matching.
var versionPrefix lexver.Version
hasVersionConstraint := q.Version != ""
if hasVersionConstraint {
versionPrefix = lexver.Parse(q.Version)
}
var best *candidate
for i := range dists {
d := &dists[i]
// Channel filter.
if channel == "stable" && d.Channel != "stable" && d.Channel != "" {
continue
}
// OS filter: exact match, POSIX fallback, or ANYOS.
if !osMatches(q.OS, d.OS) {
continue
}
// Arch filter (including compat arches).
// Empty arch, ANYARCH, or "*" means "universal/platform-agnostic" —
// accept it but rank it lower than an exact match.
aRank, archOK := archRank[d.Arch]
if !archOK && (d.Arch == "" || d.Arch == "*" || d.Arch == string(buildmeta.ArchAny)) {
// Universal binary — rank after all specific arches.
aRank = len(compatArches)
archOK = true
}
if !archOK {
continue
}
// Libc filter.
if !libcMatches(q.OS, q.Libc, d.Libc) {
continue
}
// Format filter.
// Empty format means bare binary — accept as last resort.
fRank, formatOK := formatRank[d.Format]
if !formatOK && d.Format == "" {
// Bare binary — rank after all explicit formats.
fRank = len(q.Formats)
formatOK = true
}
if !formatOK && len(q.Formats) > 0 {
continue
}
if !formatOK {
fRank = 999
}
// Version constraint.
ver := lexver.Parse(d.Version)
if hasVersionConstraint && !ver.HasPrefix(versionPrefix) {
continue
}
c := &candidate{
dist: d,
ver: ver,
archRank: aRank,
formatRank: fRank,
hasVariants: len(d.Variants) > 0,
}
if best == nil || c.betterThan(best) {
best = c
}
}
if best == nil {
return nil
}
d := best.dist
return &Match{
Version: d.Version,
OS: d.OS,
Arch: d.Arch,
Libc: d.Libc,
Format: d.Format,
Download: d.Download,
Filename: d.Filename,
LTS: d.LTS,
Date: d.Date,
Channel: d.Channel,
}
}
// Catalog computes aggregate metadata across all stable dists for a package.
type Catalog struct {
OSes []string
Arches []string
Libcs []string
Formats []string
Latest string // highest version of any channel
Stable string // highest stable version
}
// Survey scans all dists and returns the catalog.
func Survey(dists []Dist) Catalog {
oses := make(map[string]bool)
arches := make(map[string]bool)
libcs := make(map[string]bool)
formats := make(map[string]bool)
var latest, stable string
for _, d := range dists {
if d.OS != "" {
oses[d.OS] = true
}
if d.Arch != "" {
arches[d.Arch] = true
}
if d.Libc != "" {
libcs[d.Libc] = true
}
if d.Format != "" {
formats[d.Format] = true
}
v := lexver.Parse(d.Version)
if latest == "" || lexver.Compare(v, lexver.Parse(latest)) > 0 {
latest = d.Version
}
if d.Channel == "stable" || d.Channel == "" {
if stable == "" || lexver.Compare(v, lexver.Parse(stable)) > 0 {
stable = d.Version
}
}
}
return Catalog{
OSes: sortedKeys(oses),
Arches: sortedKeys(arches),
Libcs: sortedKeys(libcs),
Formats: sortedKeys(formats),
Latest: latest,
Stable: stable,
}
}
type candidate struct {
dist *Dist
ver lexver.Version
archRank int
formatRank int
hasVariants bool // true if dist has variant qualifiers (GPU, installer, etc.)
}
// betterThan returns true if c is a better match than other.
// Priority: version (higher) > base over variant > arch rank (lower=native) > format rank (lower=preferred).
func (c *candidate) betterThan(other *candidate) bool {
cmp := lexver.Compare(c.ver, other.ver)
if cmp != 0 {
return cmp > 0
}
// Prefer base build over variant builds (rocm, installer, etc.)
if c.hasVariants != other.hasVariants {
return !c.hasVariants
}
if c.archRank != other.archRank {
return c.archRank < other.archRank
}
return c.formatRank < other.formatRank
}
// osMatches checks whether a dist's OS is acceptable for the query.
// Matches exact OS, ANYOS (universal), and POSIX compatibility levels
// (posix_2017 matches any non-Windows OS).
func osMatches(want buildmeta.OS, have string) bool {
if have == string(want) {
return true
}
if have == string(buildmeta.OSAny) {
return true
}
// POSIX assets run on any non-Windows system.
if want != buildmeta.OSWindows {
if have == string(buildmeta.OSPosix2017) || have == string(buildmeta.OSPosix2024) {
return true
}
}
return false
}
// libcMatches checks whether a dist's libc is acceptable for the query.
func libcMatches(os buildmeta.OS, want buildmeta.Libc, have string) bool {
// Darwin and Windows don't use libc tagging — accept anything.
if os == buildmeta.OSDarwin || os == buildmeta.OSWindows {
return true
}
// If the dist has no libc tag, accept it (likely statically linked).
if have == "" || have == "none" || have == string(buildmeta.LibcNone) {
return true
}
// If the query has no libc preference, accept any.
if want == "" || want == buildmeta.LibcNone {
return true
}
return have == string(want)
}
func sortedKeys(m map[string]bool) []string {
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
// Simple insertion sort — these are tiny sets.
for i := 1; i < len(keys); i++ {
for j := i; j > 0 && strings.Compare(keys[j-1], keys[j]) > 0; j-- {
keys[j-1], keys[j] = keys[j], keys[j-1]
}
}
return keys
}

View File

@@ -0,0 +1,422 @@
package resolve_test
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/webinstall/webi-installers/internal/buildmeta"
"github.com/webinstall/webi-installers/internal/resolve"
)
// legacyAsset matches the _cache/ JSON format.
type legacyAsset struct {
Name string `json:"name"`
Version string `json:"version"`
LTS bool `json:"lts"`
Channel string `json:"channel"`
Date string `json:"date"`
OS string `json:"os"`
Arch string `json:"arch"`
Libc string `json:"libc"`
Ext string `json:"ext"`
Download string `json:"download"`
}
type legacyCache struct {
Releases []legacyAsset `json:"releases"`
}
func loadCacheDists(t *testing.T, pkg string) []resolve.Dist {
t.Helper()
cacheDir := filepath.Join("..", "..", "_cache", "2026-03")
path := filepath.Join(cacheDir, pkg+".json")
data, err := os.ReadFile(path)
if err != nil {
t.Skipf("no cache file for %s: %v", pkg, err)
}
var lc legacyCache
if err := json.Unmarshal(data, &lc); err != nil {
t.Fatalf("parse %s: %v", pkg, err)
}
dists := make([]resolve.Dist, len(lc.Releases))
for i, la := range lc.Releases {
// Reverse-translate legacy Node.js vocabulary to Go canonical names.
// The cache file uses macos/amd64/arm64; the resolver uses darwin/x86_64/aarch64.
osStr := la.OS
if osStr == "macos" {
osStr = "darwin"
}
archStr := la.Arch
switch archStr {
case "amd64":
archStr = "x86_64"
case "arm64":
archStr = "aarch64"
}
// Restore dot-prefix convention: cache stores "tar.gz", resolver needs ".tar.gz".
// "exe" with no dot in filename = bare binary (Format ""), otherwise ".exe".
format := la.Ext
switch {
case format == "exe" && !strings.Contains(la.Name, "."):
format = ""
case format != "":
format = "." + format
}
dists[i] = resolve.Dist{
Filename: la.Name,
Version: la.Version,
LTS: la.LTS,
Channel: la.Channel,
Date: la.Date,
OS: osStr,
Arch: archStr,
Libc: la.Libc, // "none" = buildmeta.LibcNone (statically linked)
Format: format,
Download: la.Download,
}
}
return dists
}
// platforms is the standard webi target matrix.
var platforms = []struct {
name string
os buildmeta.OS
arch buildmeta.Arch
formats []string
}{
{"darwin-arm64", buildmeta.OSDarwin, buildmeta.ArchARM64, []string{".tar.xz", ".tar.gz", ".zip"}},
{"darwin-amd64", buildmeta.OSDarwin, buildmeta.ArchAMD64, []string{".tar.xz", ".tar.gz", ".zip"}},
{"linux-amd64", buildmeta.OSLinux, buildmeta.ArchAMD64, []string{".tar.xz", ".exe.xz", ".tar.gz", ".xz", ".gz", ".zip"}},
{"linux-arm64", buildmeta.OSLinux, buildmeta.ArchARM64, []string{".tar.xz", ".exe.xz", ".tar.gz", ".xz", ".gz", ".zip"}},
{"linux-armv7", buildmeta.OSLinux, buildmeta.ArchARMv7, []string{".tar.xz", ".tar.gz", ".xz", ".gz", ".zip"}},
{"linux-armv6", buildmeta.OSLinux, buildmeta.ArchARMv6, []string{".tar.xz", ".tar.gz", ".xz", ".gz", ".zip"}},
{"windows-amd64", buildmeta.OSWindows, buildmeta.ArchAMD64, []string{".zip", ".tar.gz", ".exe", ".7z"}},
{"windows-arm64", buildmeta.OSWindows, buildmeta.ArchARM64, []string{".zip", ".tar.gz", ".exe", ".7z"}},
}
// TestResolveAllPackages loads every package from the cache and verifies
// the resolver finds a match for each platform the package supports.
func TestResolveAllPackages(t *testing.T) {
cacheDir := filepath.Join("..", "..", "_cache", "2026-03")
entries, err := os.ReadDir(cacheDir)
if err != nil {
t.Skipf("no cache dir: %v", err)
}
var pkgs []string
for _, e := range entries {
if strings.HasSuffix(e.Name(), ".json") {
pkgs = append(pkgs, strings.TrimSuffix(e.Name(), ".json"))
}
}
if len(pkgs) < 50 {
t.Fatalf("expected at least 50 packages, got %d", len(pkgs))
}
for _, pkg := range pkgs {
t.Run(pkg, func(t *testing.T) {
dists := loadCacheDists(t, pkg)
if len(dists) == 0 {
t.Skip("no releases")
}
// Determine which platforms this package supports.
cat := resolve.Survey(dists)
osSet := make(map[string]bool, len(cat.OSes))
for _, o := range cat.OSes {
osSet[o] = true
}
for _, plat := range platforms {
platOS := string(plat.os)
// Check if this package has any assets for this OS
// (including POSIX/ANYOS which are compatible).
supported := osSet[platOS] ||
osSet[string(buildmeta.OSAny)] ||
(platOS != "windows" && (osSet[string(buildmeta.OSPosix2017)] || osSet[string(buildmeta.OSPosix2024)]))
if !supported {
continue
}
t.Run(plat.name, func(t *testing.T) {
m := resolve.Best(dists, resolve.Query{
OS: plat.os,
Arch: plat.arch,
Formats: plat.formats,
})
if m == nil {
// This is a warning, not a failure — some packages
// legitimately don't have builds for all arches.
// But log it so we can spot unexpected gaps.
t.Logf("WARN: no match for %s on %s (has OSes: %v, Arches: %v)",
pkg, plat.name, cat.OSes, cat.Arches)
return
}
if m.Version == "" {
t.Error("matched but Version is empty")
}
if m.Download == "" {
t.Error("matched but Download is empty")
}
})
}
})
}
}
// Packages with known platform expectations. Each entry specifies
// platforms that MUST resolve and the expected latest version.
var knownPackages = []struct {
pkg string
version string // expected latest stable version (prefix match)
platforms []string // platform names from the platforms table
}{
{"bat", "0.26", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
{"caddy", "2.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "linux-armv7", "linux-armv6", "windows-amd64"}},
{"delta", "0.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
{"fd", "10.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "linux-armv7", "windows-amd64"}},
{"fzf", "0.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "linux-armv7", "windows-amd64"}},
{"gh", "2.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "linux-armv6", "windows-amd64"}},
{"rg", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
{"shellcheck", "0.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "linux-armv6", "windows-amd64"}},
{"shfmt", "3.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "linux-armv6", "windows-amd64"}},
{"xz", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
{"yq", "4.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "linux-armv6", "windows-amd64"}},
{"zoxide", "0.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "linux-armv7", "windows-amd64"}},
{"aliasman", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64"}},
{"comrak", "0.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "windows-amd64"}},
{"hugo", "0.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
{"node", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
{"terraform", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
{"zig", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
}
// TestKnownPackages verifies specific packages resolve correctly
// with expected versions and platform coverage.
func TestKnownPackages(t *testing.T) {
platMap := make(map[string]struct {
os buildmeta.OS
arch buildmeta.Arch
formats []string
})
for _, p := range platforms {
platMap[p.name] = struct {
os buildmeta.OS
arch buildmeta.Arch
formats []string
}{p.os, p.arch, p.formats}
}
for _, kp := range knownPackages {
t.Run(kp.pkg, func(t *testing.T) {
dists := loadCacheDists(t, kp.pkg)
for _, platName := range kp.platforms {
plat := platMap[platName]
t.Run(platName, func(t *testing.T) {
m := resolve.Best(dists, resolve.Query{
OS: plat.os,
Arch: plat.arch,
Formats: plat.formats,
})
if m == nil {
t.Skipf("no build available for %s on %s — upstream gap", kp.pkg, platName)
return
}
if kp.version != "" {
// Strip leading "v" for prefix comparison.
v := strings.TrimPrefix(m.Version, "v")
if !strings.HasPrefix(v, kp.version) {
t.Errorf("Version = %q, want prefix %q", m.Version, kp.version)
}
}
})
}
})
}
}
// TestResolveVersionConstraints tests version pinning across real packages.
func TestResolveVersionConstraints(t *testing.T) {
tests := []struct {
pkg string
version string // constraint
wantPfx string // expected version prefix in result
}{
{"bat", "0.25", "0.25"},
{"bat", "0.26", "0.26"},
{"gh", "2.40", "2.40"},
{"node", "20", "20."},
{"node", "22", "22."},
{"hugo", "0.121", "0.121"},
}
for _, tt := range tests {
name := fmt.Sprintf("%s@%s", tt.pkg, tt.version)
t.Run(name, func(t *testing.T) {
dists := loadCacheDists(t, tt.pkg)
m := resolve.Best(dists, resolve.Query{
OS: buildmeta.OSLinux,
Arch: buildmeta.ArchAMD64,
Formats: []string{".tar.xz", ".tar.gz", ".zip"},
Version: tt.version,
})
if m == nil {
t.Fatalf("no match for %s@%s", tt.pkg, tt.version)
}
v := strings.TrimPrefix(m.Version, "v")
if !strings.HasPrefix(v, tt.wantPfx) {
t.Errorf("Version = %q, want prefix %q", m.Version, tt.wantPfx)
}
})
}
}
// TestResolveArchFallbackReal tests arch fallback with real package data.
func TestResolveArchFallbackReal(t *testing.T) {
// awless only has amd64 builds — macOS ARM64 should fall back.
dists := loadCacheDists(t, "awless")
m := resolve.Best(dists, resolve.Query{
OS: buildmeta.OSDarwin,
Arch: buildmeta.ArchARM64,
Formats: []string{".tar.gz", ".zip"},
})
if m == nil {
t.Fatal("expected Rosetta 2 fallback for awless")
}
if m.Arch != "x86_64" {
t.Errorf("Arch = %q, want x86_64", m.Arch)
}
}
// TestResolvePosixPackages tests packages that use posix_2017/ANYOS.
func TestResolvePosixPackages(t *testing.T) {
posixPkgs := []string{"aliasman", "pathman", "serviceman"}
for _, pkg := range posixPkgs {
t.Run(pkg, func(t *testing.T) {
dists := loadCacheDists(t, pkg)
if len(dists) == 0 {
t.Skip("no releases")
}
// Should resolve on Linux.
m := resolve.Best(dists, resolve.Query{
OS: buildmeta.OSLinux,
Arch: buildmeta.ArchAMD64,
Formats: []string{".tar.xz", ".tar.gz", ".zip", ".xz", ".gz"},
})
if m == nil {
t.Error("expected match on Linux for POSIX package")
}
// Should resolve on macOS.
m = resolve.Best(dists, resolve.Query{
OS: buildmeta.OSDarwin,
Arch: buildmeta.ArchARM64,
Formats: []string{".tar.xz", ".tar.gz", ".zip"},
})
if m == nil {
t.Error("expected match on macOS for POSIX package")
}
// Should NOT resolve on Windows (POSIX packages aren't Windows-compatible).
m = resolve.Best(dists, resolve.Query{
OS: buildmeta.OSWindows,
Arch: buildmeta.ArchAMD64,
Formats: []string{".zip", ".tar.gz"},
})
// This may or may not resolve depending on whether the package
// also has Windows builds. Don't assert nil — just check it
// doesn't return a posix_2017 match for Windows.
if m != nil && (m.OS == "posix_2017" || m.OS == "posix_2024") {
t.Errorf("POSIX package should not match Windows, got OS=%q", m.OS)
}
})
}
}
// TestResolveLibcPreference tests libc selection.
// bat is a Rust project — its musl builds are static (libc='none').
// pwsh has hard musl dependencies (libc='musl').
func TestResolveLibcPreference(t *testing.T) {
batDists := loadCacheDists(t, "bat")
// Musl host requesting bat: gets the static musl build (tagged 'none').
m := resolve.Best(batDists, resolve.Query{
OS: buildmeta.OSLinux,
Arch: buildmeta.ArchAMD64,
Libc: buildmeta.LibcMusl,
Formats: []string{".tar.gz"},
})
if m == nil {
t.Fatal("expected match for musl host")
}
// Rust musl builds are static — tagged as 'none', not 'musl'.
if m.Libc != "none" {
t.Errorf("bat musl request: Libc = %q, want none (static musl)", m.Libc)
}
// Explicit gnu request.
m = resolve.Best(batDists, resolve.Query{
OS: buildmeta.OSLinux,
Arch: buildmeta.ArchAMD64,
Libc: buildmeta.LibcGNU,
Formats: []string{".tar.gz"},
})
if m == nil {
t.Fatal("expected gnu match")
}
if m.Libc != "gnu" {
t.Errorf("Libc = %q, want gnu", m.Libc)
}
// No preference — should still match (accepts any).
m = resolve.Best(batDists, resolve.Query{
OS: buildmeta.OSLinux,
Arch: buildmeta.ArchAMD64,
Formats: []string{".tar.gz"},
})
if m == nil {
t.Fatal("expected match with no libc preference")
}
// pwsh has hard musl builds (dynamically linked, requires musl runtime).
pwshDists := loadCacheDists(t, "pwsh")
m = resolve.Best(pwshDists, resolve.Query{
OS: buildmeta.OSLinux,
Arch: buildmeta.ArchAMD64,
Libc: buildmeta.LibcMusl,
Formats: []string{".tar.gz"},
})
if m == nil {
t.Fatal("expected pwsh musl match")
}
if m.Libc != "musl" {
t.Errorf("pwsh musl request: Libc = %q, want musl", m.Libc)
}
}
// TestResolveFormatFallback tests format preference cascading.
func TestResolveFormatFallback(t *testing.T) {
// Request .tar.xz first, fall back to .tar.gz.
dists := loadCacheDists(t, "bat")
m := resolve.Best(dists, resolve.Query{
OS: buildmeta.OSLinux,
Arch: buildmeta.ArchAMD64,
Formats: []string{".tar.xz", ".tar.gz", ".zip"},
})
if m == nil {
t.Fatal("expected match")
}
// bat only has .tar.gz — should fall back from .tar.xz.
if m.Format != ".tar.gz" {
t.Errorf("Format = %q, want .tar.gz (fallback from .tar.xz)", m.Format)
}
}

View File

@@ -0,0 +1,250 @@
package resolve
import (
"testing"
"github.com/webinstall/webi-installers/internal/buildmeta"
)
// bat-style dists: standard goreleaser output.
var batDists = []Dist{
{Version: "0.26.1", Channel: "stable", OS: "darwin", Arch: "aarch64", Format: ".tar.gz", Filename: "bat-v0.26.1-aarch64-apple-darwin.tar.gz"},
{Version: "0.26.1", Channel: "stable", OS: "darwin", Arch: "x86_64", Format: ".tar.gz", Filename: "bat-v0.26.1-x86_64-apple-darwin.tar.gz"},
{Version: "0.26.1", Channel: "stable", OS: "linux", Arch: "aarch64", Libc: "gnu", Format: ".tar.gz", Filename: "bat-v0.26.1-aarch64-unknown-linux-gnu.tar.gz"},
{Version: "0.26.1", Channel: "stable", OS: "linux", Arch: "aarch64", Libc: "musl", Format: ".tar.gz", Filename: "bat-v0.26.1-aarch64-unknown-linux-musl.tar.gz"},
{Version: "0.26.1", Channel: "stable", OS: "linux", Arch: "x86_64", Libc: "gnu", Format: ".tar.gz", Filename: "bat-v0.26.1-x86_64-unknown-linux-gnu.tar.gz"},
{Version: "0.26.1", Channel: "stable", OS: "linux", Arch: "x86_64", Libc: "musl", Format: ".tar.gz", Filename: "bat-v0.26.1-x86_64-unknown-linux-musl.tar.gz"},
{Version: "0.26.1", Channel: "stable", OS: "windows", Arch: "aarch64", Format: ".zip", Filename: "bat-v0.26.1-aarch64-pc-windows-msvc.zip"},
{Version: "0.26.1", Channel: "stable", OS: "windows", Arch: "x86_64", Libc: "gnu", Format: ".zip", Filename: "bat-v0.26.1-x86_64-pc-windows-gnu.zip"},
{Version: "0.26.1", Channel: "stable", OS: "windows", Arch: "x86_64", Libc: "msvc", Format: ".zip", Filename: "bat-v0.26.1-x86_64-pc-windows-msvc.zip"},
// Older version.
{Version: "0.25.0", Channel: "stable", OS: "darwin", Arch: "aarch64", Format: ".tar.gz", Filename: "bat-v0.25.0-aarch64-apple-darwin.tar.gz"},
{Version: "0.25.0", Channel: "stable", OS: "linux", Arch: "x86_64", Libc: "gnu", Format: ".tar.gz", Filename: "bat-v0.25.0-x86_64-unknown-linux-gnu.tar.gz"},
}
func TestBestExactMatch(t *testing.T) {
m := Best(batDists, Query{
OS: buildmeta.OSLinux,
Arch: buildmeta.ArchAMD64,
Formats: []string{".tar.gz"},
})
if m == nil {
t.Fatal("expected match")
}
if m.Version != "0.26.1" {
t.Errorf("Version = %q, want 0.26.1", m.Version)
}
if m.Filename != "bat-v0.26.1-x86_64-unknown-linux-gnu.tar.gz" {
t.Errorf("Filename = %q", m.Filename)
}
}
func TestBestVersionConstraint(t *testing.T) {
m := Best(batDists, Query{
OS: buildmeta.OSDarwin,
Arch: buildmeta.ArchARM64,
Formats: []string{".tar.gz"},
Version: "0.25",
})
if m == nil {
t.Fatal("expected match")
}
if m.Version != "0.25.0" {
t.Errorf("Version = %q, want 0.25.0", m.Version)
}
}
func TestBestArchFallback(t *testing.T) {
// macOS ARM64 should fall back to x86_64 via Rosetta 2
// when no ARM64 build exists.
dists := []Dist{
{Version: "1.0.0", Channel: "stable", OS: "darwin", Arch: "x86_64", Format: ".tar.gz", Filename: "tool-darwin-amd64.tar.gz"},
}
m := Best(dists, Query{
OS: buildmeta.OSDarwin,
Arch: buildmeta.ArchARM64,
Formats: []string{".tar.gz"},
})
if m == nil {
t.Fatal("expected match via Rosetta 2 fallback")
}
if m.Arch != "x86_64" {
t.Errorf("Arch = %q, want x86_64", m.Arch)
}
}
func TestBestPrefersNativeOverCompat(t *testing.T) {
// When both native and compat builds exist, prefer native.
dists := []Dist{
{Version: "1.0.0", Channel: "stable", OS: "darwin", Arch: "x86_64", Format: ".tar.gz", Filename: "tool-darwin-amd64.tar.gz"},
{Version: "1.0.0", Channel: "stable", OS: "darwin", Arch: "aarch64", Format: ".tar.gz", Filename: "tool-darwin-arm64.tar.gz"},
}
m := Best(dists, Query{
OS: buildmeta.OSDarwin,
Arch: buildmeta.ArchARM64,
Formats: []string{".tar.gz"},
})
if m == nil {
t.Fatal("expected match")
}
if m.Arch != "aarch64" {
t.Errorf("Arch = %q, want aarch64 (native)", m.Arch)
}
}
func TestBestFormatPreference(t *testing.T) {
dists := []Dist{
{Version: "1.0.0", Channel: "stable", OS: "linux", Arch: "x86_64", Format: ".zip", Filename: "tool.zip"},
{Version: "1.0.0", Channel: "stable", OS: "linux", Arch: "x86_64", Format: ".tar.gz", Filename: "tool.tar.gz"},
{Version: "1.0.0", Channel: "stable", OS: "linux", Arch: "x86_64", Format: ".tar.xz", Filename: "tool.tar.xz"},
}
m := Best(dists, Query{
OS: buildmeta.OSLinux,
Arch: buildmeta.ArchAMD64,
Formats: []string{".tar.xz", ".tar.gz", ".zip"},
})
if m == nil {
t.Fatal("expected match")
}
if m.Format != ".tar.xz" {
t.Errorf("Format = %q, want .tar.xz", m.Format)
}
}
func TestBestNoMatch(t *testing.T) {
m := Best(batDists, Query{
OS: buildmeta.OSFreeBSD,
Arch: buildmeta.ArchAMD64,
Formats: []string{".tar.gz"},
})
if m != nil {
t.Errorf("expected nil, got %+v", m)
}
}
func TestBestLibcMusl(t *testing.T) {
m := Best(batDists, Query{
OS: buildmeta.OSLinux,
Arch: buildmeta.ArchAMD64,
Libc: buildmeta.LibcMusl,
Formats: []string{".tar.gz"},
})
if m == nil {
t.Fatal("expected match")
}
if m.Libc != "musl" {
t.Errorf("Libc = %q, want musl", m.Libc)
}
}
func TestBestPrefersBaseOverVariant(t *testing.T) {
dists := []Dist{
{Version: "1.0.0", Channel: "stable", OS: "linux", Arch: "x86_64", Format: ".tar.gz", Filename: "tool.tar.gz"},
{Version: "1.0.0", Channel: "stable", OS: "linux", Arch: "x86_64", Format: ".tar.gz", Filename: "tool-rocm.tar.gz", Variants: []string{"rocm"}},
}
m := Best(dists, Query{
OS: buildmeta.OSLinux,
Arch: buildmeta.ArchAMD64,
Formats: []string{".tar.gz"},
})
if m == nil {
t.Fatal("expected match")
}
if m.Filename != "tool.tar.gz" {
t.Errorf("got variant build %q, want base", m.Filename)
}
}
func TestBestPosixFallback(t *testing.T) {
dists := []Dist{
{Version: "1.0.0", Channel: "stable", OS: "posix_2017", Format: ".tar.gz", Filename: "script.tar.gz"},
}
m := Best(dists, Query{
OS: buildmeta.OSLinux,
Arch: buildmeta.ArchAMD64,
Formats: []string{".tar.gz"},
})
if m == nil {
t.Fatal("expected match via POSIX fallback")
}
if m.OS != "posix_2017" {
t.Errorf("OS = %q, want posix_2017", m.OS)
}
}
func TestBestAnyOS(t *testing.T) {
dists := []Dist{
{Version: "1.0.0", Channel: "stable", OS: "ANYOS", Format: ".tar.gz", Filename: "tool.tar.gz"},
}
m := Best(dists, Query{
OS: buildmeta.OSWindows,
Arch: buildmeta.ArchAMD64,
Formats: []string{".tar.gz"},
})
if m == nil {
t.Fatal("expected match via ANYOS")
}
}
func TestBestAnyArch(t *testing.T) {
dists := []Dist{
{Version: "1.0.0", Channel: "stable", OS: "linux", Arch: "ANYARCH", Format: ".tar.gz", Filename: "tool.tar.gz"},
}
m := Best(dists, Query{
OS: buildmeta.OSLinux,
Arch: buildmeta.ArchARM64,
Formats: []string{".tar.gz"},
})
if m == nil {
t.Fatal("expected match via ANYARCH")
}
}
func TestBestWindowsArchFallback(t *testing.T) {
// Windows ARM64 should fall back to x86_64 via emulation.
dists := []Dist{
{Version: "1.0.0", Channel: "stable", OS: "windows", Arch: "x86_64", Format: ".zip", Filename: "tool-win64.zip"},
}
m := Best(dists, Query{
OS: buildmeta.OSWindows,
Arch: buildmeta.ArchARM64,
Formats: []string{".zip"},
})
if m == nil {
t.Fatal("expected match via Windows ARM64 emulation")
}
if m.Arch != "x86_64" {
t.Errorf("Arch = %q, want x86_64", m.Arch)
}
}
func TestBestMicroArchFallback(t *testing.T) {
// amd64v3 query should fall back to amd64 baseline.
dists := []Dist{
{Version: "1.0.0", Channel: "stable", OS: "linux", Arch: "x86_64", Format: ".tar.gz", Filename: "tool-amd64.tar.gz"},
}
m := Best(dists, Query{
OS: buildmeta.OSLinux,
Arch: buildmeta.ArchAMD64v3,
Formats: []string{".tar.gz"},
})
if m == nil {
t.Fatal("expected match via micro-arch fallback")
}
if m.Arch != "x86_64" {
t.Errorf("Arch = %q, want x86_64 (baseline)", m.Arch)
}
}
func TestSurvey(t *testing.T) {
cat := Survey(batDists)
if cat.Stable != "0.26.1" {
t.Errorf("Stable = %q, want 0.26.1", cat.Stable)
}
if cat.Latest != "0.26.1" {
t.Errorf("Latest = %q, want 0.26.1", cat.Latest)
}
if len(cat.OSes) != 3 {
t.Errorf("OSes = %v, want 3", cat.OSes)
}
}

View File

@@ -0,0 +1,416 @@
// Package resolver selects the best release asset for a given platform
// and version constraint.
//
// The resolver takes a package's full asset list and a request describing
// what the client needs (OS, arch, libc, version prefix, channel, format
// preferences). It returns the single best matching asset or an error.
//
// Resolution order:
// 1. Filter assets by channel (inclusive: @stable includes stable+lts)
// 2. Sort versions descending, filter by version prefix if given
// 3. For each candidate version, try compatible platform triplets
// (OS × CompatArches fallback × libc) in preference order
// 4. Among platform matches, pick the best format
// 5. Among format matches, prefer assets without build variants
package resolver
import (
"errors"
"slices"
"strings"
"github.com/webinstall/webi-installers/internal/buildmeta"
"github.com/webinstall/webi-installers/internal/lexver"
"github.com/webinstall/webi-installers/internal/storage"
)
// ErrNoMatch is returned when no asset matches the request.
var ErrNoMatch = errors.New("resolver: no matching asset")
// Request describes what the client is looking for.
type Request struct {
// OS is the target operating system (e.g. "linux", "darwin", "windows").
OS string
// Arch is the target architecture (e.g. "aarch64", "x86_64").
Arch string
// Libc is the preferred C library (e.g. "gnu", "musl", "msvc").
// Empty means no preference — the resolver tries all libc values.
Libc string
// Version is a version prefix constraint (e.g. "1.20", "1", "").
// Empty means latest. Exact versions like "1.20.3" also work.
Version string
// Channel selects the release stability level. Values:
// ""/"stable" — stable and LTS only (default)
// "lts" — LTS releases only
// "rc" — rc + stable + LTS
// "beta" — beta + rc + stable + LTS
// "alpha" — everything (alpha + beta + rc + stable + LTS)
// "pre" — alias for beta (package-specific meaning)
Channel string
// LTS when true selects only LTS-flagged releases.
LTS bool
// Formats lists acceptable archive formats in preference order.
// If empty, a default preference order is used.
Formats []string
// Variant selects a specific build variant (e.g. "rocm", "jetpack6").
// If empty, assets with variants are deprioritized.
Variant string
}
// Result holds the resolved asset and metadata about the match.
type Result struct {
// Asset is the selected download.
Asset storage.Asset
// Version is the matched version string.
Version string
// Triplet is the matched platform triplet (os-arch-libc).
Triplet string
}
// Resolve finds the best matching asset for the given request.
func Resolve(assets []storage.Asset, req Request) (Result, error) {
if len(assets) == 0 {
return Result{}, ErrNoMatch
}
// Parse the version prefix for filtering.
var versionPrefix lexver.Version
hasPrefix := req.Version != ""
if hasPrefix {
versionPrefix = lexver.Parse(req.Version)
}
// Build the channel filter.
channelOK := channelFilter(req.Channel, req.LTS)
// Parse and sort all unique versions descending.
type versionEntry struct {
parsed lexver.Version
raw string
}
seen := make(map[string]bool)
var versions []versionEntry
for _, a := range assets {
if seen[a.Version] {
continue
}
seen[a.Version] = true
v := lexver.Parse(a.Version)
v.Raw = a.Version
versions = append(versions, versionEntry{parsed: v, raw: a.Version})
}
slices.SortFunc(versions, func(a, b versionEntry) int {
return lexver.Compare(b.parsed, a.parsed) // descending
})
// Build platform fallback list: ordered (os, arch, libc) combinations.
triplets := enumerateTriplets(req.OS, req.Arch, req.Libc)
// Build format preference list.
formats := req.Formats
if len(formats) == 0 {
formats = defaultFormats(req.OS)
}
// Index assets by version+triplet for fast lookup.
// Assets with empty OS/Arch (like git repos) use "" keys.
type tripletKey struct {
version string
os string
arch string
libc string
}
index := make(map[tripletKey][]storage.Asset)
for _, a := range assets {
key := tripletKey{
version: a.Version,
os: a.OS,
arch: a.Arch,
libc: a.Libc,
}
index[key] = append(index[key], a)
}
// Walk versions in descending order.
for _, ve := range versions {
// Check version prefix.
if hasPrefix && !ve.parsed.HasPrefix(versionPrefix) {
continue
}
// Check channel.
if !channelOK(ve.parsed.Channel, ve.raw) {
continue
}
// Try each compatible triplet.
for _, tri := range triplets {
key := tripletKey{
version: ve.raw,
os: tri.os,
arch: tri.arch,
libc: tri.libc,
}
candidates := index[key]
if len(candidates) == 0 {
continue
}
// Pick the best asset from candidates.
best, ok := pickBest(candidates, formats, req.Variant, req.LTS)
if !ok {
continue
}
triplet := tri.os + "-" + tri.arch + "-" + tri.libc
return Result{
Asset: best,
Version: ve.raw,
Triplet: triplet,
}, nil
}
}
return Result{}, ErrNoMatch
}
// channelFilter returns a function that checks whether a given channel
// is acceptable for the requested channel level.
func channelFilter(requested string, ltsOnly bool) func(channel string, version string) bool {
if ltsOnly {
return func(_ string, _ string) bool {
// LTS filtering happens at the asset level, not version level.
// We let all versions through and filter by LTS flag later.
// Actually, LTS is per-asset, so we handle it in pickBest.
return true
}
}
requested = strings.ToLower(requested)
if requested == "" {
requested = "stable"
}
if requested == "pre" {
requested = "beta"
}
if requested == "latest" {
requested = "stable"
}
// channelRank maps channel names to a numeric rank.
// Higher rank = less stable. A request for rank N accepts
// everything at rank N or below.
rank := func(ch string) int {
ch = strings.ToLower(ch)
switch ch {
case "", "stable":
return 0
case "rc":
return 1
case "beta", "preview":
return 2
case "alpha", "dev":
return 3
default:
return 2 // unknown pre-release channels default to beta-level
}
}
maxRank := rank(requested)
return func(channel string, _ string) bool {
return rank(channel) <= maxRank
}
}
type platformTriple struct {
os string
arch string
libc string
}
// enumerateTriplets builds the ordered list of platform combinations to try.
// It uses CompatArches for arch fallback and tries multiple libc values.
func enumerateTriplets(osStr, archStr, libcStr string) []platformTriple {
// OS candidates: specific OS first, then POSIX compat, then any.
var oses []string
switch osStr {
case "windows":
oses = []string{"windows", "ANYOS", ""}
case "android":
oses = []string{"android", "linux", "posix_2024", "posix_2017", "ANYOS", ""}
case "":
oses = []string{"ANYOS", ""}
default:
oses = []string{osStr, "posix_2024", "posix_2017", "ANYOS", ""}
}
// Arch candidates: use CompatArches for fallback chain.
arches := buildmeta.CompatArches(buildmeta.OS(osStr), buildmeta.Arch(archStr))
var archStrs []string
for _, a := range arches {
archStrs = append(archStrs, string(a))
}
// Also try ANYARCH and empty (for platform-agnostic assets like git repos).
archStrs = append(archStrs, "ANYARCH", "")
// Libc candidates.
var libcs []string
if libcStr != "" {
libcs = []string{libcStr, "none", ""}
} else {
// No preference: try all common options.
switch osStr {
case "linux":
// none first (static, no deps), then gnu, musl, empty.
libcs = []string{"none", "gnu", "musl", ""}
case "windows":
// none first (no deps), msvc last (needs vcredist).
libcs = []string{"none", "msvc", ""}
default:
libcs = []string{"none", ""}
}
}
var triplets []platformTriple
for _, os := range oses {
for _, arch := range archStrs {
for _, libc := range libcs {
triplets = append(triplets, platformTriple{
os: os,
arch: arch,
libc: libc,
})
}
}
}
return triplets
}
// pickBest selects the best asset from a set of candidates for the same
// version and platform. Prefers the requested variant (or no-variant if
// none requested), then picks by format preference.
func pickBest(candidates []storage.Asset, formats []string, wantVariant string, ltsOnly bool) (storage.Asset, bool) {
// Filter by LTS if requested.
if ltsOnly {
var lts []storage.Asset
for _, a := range candidates {
if a.LTS {
lts = append(lts, a)
}
}
if len(lts) == 0 {
return storage.Asset{}, false
}
candidates = lts
}
// Separate into variant-matched and non-variant pools.
var preferred []storage.Asset
var fallback []storage.Asset
for _, a := range candidates {
if wantVariant != "" {
// User requested a specific variant.
if hasVariant(a.Variants, wantVariant) {
preferred = append(preferred, a)
} else if len(a.Variants) == 0 {
fallback = append(fallback, a)
}
} else {
// No variant requested: prefer no-variant assets.
if len(a.Variants) == 0 {
preferred = append(preferred, a)
} else {
fallback = append(fallback, a)
}
}
}
// Try preferred pool first, then fallback.
for _, pool := range [][]storage.Asset{preferred, fallback} {
if len(pool) == 0 {
continue
}
if best, ok := pickByFormat(pool, formats); ok {
return best, true
}
}
return storage.Asset{}, false
}
// pickByFormat selects the asset with the most preferred format.
func pickByFormat(assets []storage.Asset, formats []string) (storage.Asset, bool) {
for _, fmt := range formats {
for _, a := range assets {
if a.Format == fmt {
return a, true
}
}
}
// No format match — return the first asset as last resort.
if len(assets) > 0 {
return assets[0], true
}
return storage.Asset{}, false
}
func hasVariant(variants []string, want string) bool {
for _, v := range variants {
if v == want {
return true
}
}
return false
}
// defaultFormats returns the format preference order for an OS.
// zst is preferred as the modern standard, but availability varies.
func defaultFormats(os string) []string {
switch os {
case "windows":
return []string{
".tar.zst",
".tar.xz",
".zip",
".tar.gz",
".exe.xz",
".7z",
".exe",
".msi",
"git",
}
case "darwin":
return []string{
".tar.zst",
".tar.xz",
".zip",
".tar.gz",
".gz",
".app.zip",
".dmg",
".pkg",
"git",
}
default:
// Linux and other POSIX.
return []string{
".tar.zst",
".tar.xz",
".tar.gz",
".gz",
".zip",
".xz",
"git",
}
}
}

View File

@@ -0,0 +1,290 @@
package resolver_test
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/webinstall/webi-installers/internal/resolver"
"github.com/webinstall/webi-installers/internal/storage"
)
func loadAssets(t *testing.T, pkg string) []storage.Asset {
t.Helper()
cacheDir := filepath.Join("..", "..", "_cache", "2026-03")
path := filepath.Join(cacheDir, pkg+".json")
data, err := os.ReadFile(path)
if err != nil {
t.Skipf("no cache file for %s: %v", pkg, err)
}
var lc storage.LegacyCache
if err := json.Unmarshal(data, &lc); err != nil {
t.Fatalf("parse %s: %v", pkg, err)
}
pd := storage.ImportLegacy(lc)
return pd.Assets
}
// TestCacheResolveAllPackages loads every package from the cache and verifies
// the resolver finds a match for each standard platform.
func TestCacheResolveAllPackages(t *testing.T) {
cacheDir := filepath.Join("..", "..", "_cache", "2026-03")
entries, err := os.ReadDir(cacheDir)
if err != nil {
t.Skipf("no cache dir: %v", err)
}
var pkgs []string
for _, e := range entries {
if strings.HasSuffix(e.Name(), ".json") {
pkgs = append(pkgs, strings.TrimSuffix(e.Name(), ".json"))
}
}
if len(pkgs) < 50 {
t.Fatalf("expected at least 50 packages, got %d", len(pkgs))
}
platforms := []struct {
name string
os string
arch string
}{
{"darwin-arm64", "darwin", "aarch64"},
{"darwin-amd64", "darwin", "x86_64"},
{"linux-amd64", "linux", "x86_64"},
{"linux-arm64", "linux", "aarch64"},
{"windows-amd64", "windows", "x86_64"},
}
for _, pkg := range pkgs {
t.Run(pkg, func(t *testing.T) {
assets := loadAssets(t, pkg)
if len(assets) == 0 {
t.Skip("no releases")
}
// Determine which OSes this package has.
osSet := make(map[string]bool)
for _, a := range assets {
if a.OS != "" {
osSet[a.OS] = true
}
}
// Also check for platform-agnostic assets.
hasAgnostic := false
for _, a := range assets {
if a.OS == "" {
hasAgnostic = true
break
}
}
for _, plat := range platforms {
supported := osSet[plat.os] ||
osSet["ANYOS"] ||
hasAgnostic ||
(plat.os != "windows" && (osSet["posix_2017"] || osSet["posix_2024"]))
if !supported {
continue
}
t.Run(plat.name, func(t *testing.T) {
res, err := resolver.Resolve(assets, resolver.Request{
OS: plat.os,
Arch: plat.arch,
})
if err != nil {
// Not a test failure — some packages don't have
// all arch builds. Log for visibility.
t.Logf("WARN: no match for %s on %s (has OSes: %v)",
pkg, plat.name, sortedOSes(osSet))
return
}
if res.Version == "" {
t.Error("matched but Version is empty")
}
if res.Asset.Download == "" {
t.Error("matched but Download is empty")
}
})
}
})
}
}
// TestCacheKnownPackages verifies specific packages resolve correctly.
var knownPackages = []struct {
pkg string
version string // expected latest stable version prefix
platforms []string
}{
{"bat", "0.26", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
{"caddy", "2.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
{"delta", "0.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
{"fd", "10.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
{"fzf", "0.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
{"gh", "2.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
{"rg", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
{"node", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
{"terraform", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
{"zig", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
}
func TestCacheKnownPackages(t *testing.T) {
platMap := map[string]resolver.Request{
"darwin-arm64": {OS: "darwin", Arch: "aarch64"},
"darwin-amd64": {OS: "darwin", Arch: "x86_64"},
"linux-amd64": {OS: "linux", Arch: "x86_64"},
"linux-arm64": {OS: "linux", Arch: "aarch64"},
"windows-amd64": {OS: "windows", Arch: "x86_64"},
}
for _, kp := range knownPackages {
t.Run(kp.pkg, func(t *testing.T) {
assets := loadAssets(t, kp.pkg)
for _, platName := range kp.platforms {
req := platMap[platName]
t.Run(platName, func(t *testing.T) {
res, err := resolver.Resolve(assets, req)
if err != nil {
t.Fatalf("no match for %s on %s", kp.pkg, platName)
}
if kp.version != "" {
v := strings.TrimPrefix(res.Version, "v")
if !strings.HasPrefix(v, kp.version) {
t.Errorf("Version = %q, want prefix %q", res.Version, kp.version)
}
}
})
}
})
}
}
// TestCacheVersionConstraints tests version pinning with real data.
func TestCacheVersionConstraints(t *testing.T) {
tests := []struct {
pkg string
version string
wantPfx string
}{
{"bat", "0.25", "0.25"},
{"bat", "0.26", "0.26"},
{"gh", "2.40", "2.40"},
{"node", "20", "20."},
{"node", "22", "22."},
}
for _, tt := range tests {
t.Run(tt.pkg+"@"+tt.version, func(t *testing.T) {
assets := loadAssets(t, tt.pkg)
res, err := resolver.Resolve(assets, resolver.Request{
OS: "linux",
Arch: "x86_64",
Version: tt.version,
})
if err != nil {
t.Fatalf("no match for %s@%s", tt.pkg, tt.version)
}
v := strings.TrimPrefix(res.Version, "v")
if !strings.HasPrefix(v, tt.wantPfx) {
t.Errorf("Version = %q, want prefix %q", res.Version, tt.wantPfx)
}
})
}
}
// TestCacheArchFallback verifies Rosetta-style fallback with real data.
func TestCacheArchFallback(t *testing.T) {
// awless only has amd64 builds — macOS ARM64 should fall back.
assets := loadAssets(t, "awless")
res, err := resolver.Resolve(assets, resolver.Request{
OS: "darwin",
Arch: "aarch64",
})
if err != nil {
t.Fatal("expected Rosetta 2 fallback for awless")
}
if res.Asset.Arch != "x86_64" {
t.Errorf("Arch = %q, want x86_64", res.Asset.Arch)
}
}
// TestCacheGitPackages verifies git-only packages resolve on any platform.
func TestCacheGitPackages(t *testing.T) {
gitPkgs := []string{"vim-essentials", "vim-spell"}
for _, pkg := range gitPkgs {
t.Run(pkg, func(t *testing.T) {
assets := loadAssets(t, pkg)
if len(assets) == 0 {
t.Skip("no releases")
}
// Should work on any platform.
for _, plat := range []struct {
os, arch string
}{
{"linux", "x86_64"},
{"darwin", "aarch64"},
{"windows", "x86_64"},
} {
res, err := resolver.Resolve(assets, resolver.Request{
OS: plat.os,
Arch: plat.arch,
})
if err != nil {
t.Errorf("expected match on %s-%s", plat.os, plat.arch)
continue
}
if res.Asset.Format != "git" {
t.Errorf("format = %q, want git", res.Asset.Format)
}
}
})
}
}
// TestCacheLibcPreference tests explicit libc selection.
// bat is Rust — its musl builds are static (tagged 'none').
func TestCacheLibcPreference(t *testing.T) {
assets := loadAssets(t, "bat")
// Musl host requesting bat: gets static musl build (tagged 'none').
res, err := resolver.Resolve(assets, resolver.Request{
OS: "linux",
Arch: "x86_64",
Libc: "musl",
})
if err != nil {
t.Fatal("expected match for musl host")
}
if res.Asset.Libc != "none" {
t.Errorf("Libc = %q, want none (static musl)", res.Asset.Libc)
}
// Explicit gnu.
res, err = resolver.Resolve(assets, resolver.Request{
OS: "linux",
Arch: "x86_64",
Libc: "gnu",
})
if err != nil {
t.Fatal("expected gnu match")
}
if res.Asset.Libc != "gnu" {
t.Errorf("Libc = %q, want gnu", res.Asset.Libc)
}
}
func sortedOSes(m map[string]bool) []string {
var keys []string
for k := range m {
keys = append(keys, k)
}
return keys
}

View File

@@ -0,0 +1,397 @@
package resolver
import (
"testing"
"github.com/webinstall/webi-installers/internal/storage"
)
func TestResolveSimple(t *testing.T) {
assets := []storage.Asset{
{
Filename: "bat-v0.25.0-x86_64-unknown-linux-musl.tar.gz",
Version: "0.25.0",
Channel: "stable",
OS: "linux",
Arch: "x86_64",
Libc: "musl",
Format: ".tar.gz",
Download: "https://example.com/bat-0.25.0-linux-x86_64.tar.gz",
},
{
Filename: "bat-v0.26.0-x86_64-unknown-linux-musl.tar.gz",
Version: "0.26.0",
Channel: "stable",
OS: "linux",
Arch: "x86_64",
Libc: "musl",
Format: ".tar.gz",
Download: "https://example.com/bat-0.26.0-linux-x86_64.tar.gz",
},
{
Filename: "bat-v0.26.0-aarch64-unknown-linux-musl.tar.gz",
Version: "0.26.0",
Channel: "stable",
OS: "linux",
Arch: "aarch64",
Libc: "musl",
Format: ".tar.gz",
Download: "https://example.com/bat-0.26.0-linux-aarch64.tar.gz",
},
{
Filename: "bat-v0.26.0-x86_64-pc-windows-msvc.zip",
Version: "0.26.0",
Channel: "stable",
OS: "windows",
Arch: "x86_64",
Libc: "msvc",
Format: ".zip",
Download: "https://example.com/bat-0.26.0-windows-x86_64.zip",
},
{
Filename: "bat-v0.26.0-x86_64-apple-darwin.tar.gz",
Version: "0.26.0",
Channel: "stable",
OS: "darwin",
Arch: "x86_64",
Format: ".tar.gz",
Download: "https://example.com/bat-0.26.0-darwin-x86_64.tar.gz",
},
}
t.Run("latest linux x86_64", func(t *testing.T) {
res, err := Resolve(assets, Request{
OS: "linux",
Arch: "x86_64",
})
if err != nil {
t.Fatal(err)
}
if res.Version != "0.26.0" {
t.Errorf("version = %q, want 0.26.0", res.Version)
}
if res.Asset.OS != "linux" {
t.Errorf("os = %q, want linux", res.Asset.OS)
}
if res.Asset.Arch != "x86_64" {
t.Errorf("arch = %q, want x86_64", res.Asset.Arch)
}
})
t.Run("latest linux aarch64", func(t *testing.T) {
res, err := Resolve(assets, Request{
OS: "linux",
Arch: "aarch64",
})
if err != nil {
t.Fatal(err)
}
if res.Version != "0.26.0" {
t.Errorf("version = %q, want 0.26.0", res.Version)
}
if res.Asset.Arch != "aarch64" {
t.Errorf("arch = %q, want aarch64", res.Asset.Arch)
}
})
t.Run("version prefix 0.25", func(t *testing.T) {
res, err := Resolve(assets, Request{
OS: "linux",
Arch: "x86_64",
Version: "0.25",
})
if err != nil {
t.Fatal(err)
}
if res.Version != "0.25.0" {
t.Errorf("version = %q, want 0.25.0", res.Version)
}
})
t.Run("darwin arm64 falls back to x86_64", func(t *testing.T) {
res, err := Resolve(assets, Request{
OS: "darwin",
Arch: "aarch64",
})
if err != nil {
t.Fatal(err)
}
if res.Asset.Arch != "x86_64" {
t.Errorf("arch = %q, want x86_64 (Rosetta fallback)", res.Asset.Arch)
}
})
t.Run("no match returns error", func(t *testing.T) {
_, err := Resolve(assets, Request{
OS: "freebsd",
Arch: "x86_64",
})
if err != ErrNoMatch {
t.Errorf("err = %v, want ErrNoMatch", err)
}
})
t.Run("windows gets zip", func(t *testing.T) {
res, err := Resolve(assets, Request{
OS: "windows",
Arch: "x86_64",
})
if err != nil {
t.Fatal(err)
}
if res.Asset.Format != ".zip" {
t.Errorf("format = %q, want .zip", res.Asset.Format)
}
})
}
func TestResolveChannels(t *testing.T) {
assets := []storage.Asset{
{
Filename: "tool-v2.0.0-rc1-linux-x86_64.tar.gz",
Version: "2.0.0-rc1",
Channel: "rc",
OS: "linux",
Arch: "x86_64",
Format: ".tar.gz",
},
{
Filename: "tool-v1.5.0-linux-x86_64.tar.gz",
Version: "1.5.0",
Channel: "stable",
OS: "linux",
Arch: "x86_64",
Format: ".tar.gz",
},
{
Filename: "tool-v2.0.0-beta2-linux-x86_64.tar.gz",
Version: "2.0.0-beta2",
Channel: "beta",
OS: "linux",
Arch: "x86_64",
Format: ".tar.gz",
},
}
t.Run("stable skips rc and beta", func(t *testing.T) {
res, err := Resolve(assets, Request{
OS: "linux",
Arch: "x86_64",
})
if err != nil {
t.Fatal(err)
}
if res.Version != "1.5.0" {
t.Errorf("version = %q, want 1.5.0", res.Version)
}
})
t.Run("rc includes rc and stable", func(t *testing.T) {
res, err := Resolve(assets, Request{
OS: "linux",
Arch: "x86_64",
Channel: "rc",
})
if err != nil {
t.Fatal(err)
}
if res.Version != "2.0.0-rc1" {
t.Errorf("version = %q, want 2.0.0-rc1", res.Version)
}
})
t.Run("beta includes beta, rc, and stable", func(t *testing.T) {
res, err := Resolve(assets, Request{
OS: "linux",
Arch: "x86_64",
Channel: "beta",
})
if err != nil {
t.Fatal(err)
}
// beta2 sorts after rc1 for the same numeric version (2.0.0),
// but rc1 is more stable. However, the user asked for beta channel
// which includes everything — and beta sorts before rc alphabetically.
// With lexver: 2.0.0-rc1 > 2.0.0-beta2 (rc > beta alphabetically).
if res.Version != "2.0.0-rc1" {
t.Errorf("version = %q, want 2.0.0-rc1", res.Version)
}
})
}
func TestResolveVariants(t *testing.T) {
assets := []storage.Asset{
{
Filename: "ollama-linux-amd64.tgz",
Version: "0.6.0",
Channel: "stable",
OS: "linux",
Arch: "x86_64",
Format: ".tar.gz",
},
{
Filename: "ollama-linux-amd64-rocm.tgz",
Version: "0.6.0",
Channel: "stable",
OS: "linux",
Arch: "x86_64",
Format: ".tar.gz",
Variants: []string{"rocm"},
},
}
t.Run("no variant prefers plain", func(t *testing.T) {
res, err := Resolve(assets, Request{
OS: "linux",
Arch: "x86_64",
})
if err != nil {
t.Fatal(err)
}
if len(res.Asset.Variants) != 0 {
t.Errorf("variants = %v, want empty", res.Asset.Variants)
}
})
t.Run("explicit variant selects it", func(t *testing.T) {
res, err := Resolve(assets, Request{
OS: "linux",
Arch: "x86_64",
Variant: "rocm",
})
if err != nil {
t.Fatal(err)
}
if !hasVariant(res.Asset.Variants, "rocm") {
t.Errorf("variants = %v, want [rocm]", res.Asset.Variants)
}
})
}
func TestResolveFormatPreference(t *testing.T) {
assets := []storage.Asset{
{
Filename: "tool-v1.0.0-linux-x86_64.tar.gz",
Version: "1.0.0",
Channel: "stable",
OS: "linux",
Arch: "x86_64",
Format: ".tar.gz",
},
{
Filename: "tool-v1.0.0-linux-x86_64.tar.xz",
Version: "1.0.0",
Channel: "stable",
OS: "linux",
Arch: "x86_64",
Format: ".tar.xz",
},
{
Filename: "tool-v1.0.0-linux-x86_64.tar.zst",
Version: "1.0.0",
Channel: "stable",
OS: "linux",
Arch: "x86_64",
Format: ".tar.zst",
},
}
t.Run("default prefers zst", func(t *testing.T) {
res, err := Resolve(assets, Request{
OS: "linux",
Arch: "x86_64",
})
if err != nil {
t.Fatal(err)
}
if res.Asset.Format != ".tar.zst" {
t.Errorf("format = %q, want .tar.zst", res.Asset.Format)
}
})
t.Run("explicit format preference", func(t *testing.T) {
res, err := Resolve(assets, Request{
OS: "linux",
Arch: "x86_64",
Formats: []string{".tar.gz"},
})
if err != nil {
t.Fatal(err)
}
if res.Asset.Format != ".tar.gz" {
t.Errorf("format = %q, want .tar.gz", res.Asset.Format)
}
})
}
func TestResolveGitAssets(t *testing.T) {
assets := []storage.Asset{
{
Filename: "vim-commentary-v1.2",
Version: "1.2",
Channel: "stable",
Format: "git",
Download: "https://github.com/tpope/vim-commentary.git",
},
{
Filename: "vim-commentary-v1.1",
Version: "1.1",
Channel: "stable",
Format: "git",
Download: "https://github.com/tpope/vim-commentary.git",
},
}
t.Run("git assets match any platform", func(t *testing.T) {
res, err := Resolve(assets, Request{
OS: "linux",
Arch: "x86_64",
})
if err != nil {
t.Fatal(err)
}
if res.Version != "1.2" {
t.Errorf("version = %q, want 1.2", res.Version)
}
if res.Asset.Format != "git" {
t.Errorf("format = %q, want git", res.Asset.Format)
}
})
}
func TestResolveLTS(t *testing.T) {
assets := []storage.Asset{
{
Filename: "node-v22.0.0-linux-x64.tar.gz",
Version: "22.0.0",
Channel: "stable",
OS: "linux",
Arch: "x86_64",
Format: ".tar.gz",
LTS: false,
},
{
Filename: "node-v20.15.0-linux-x64.tar.gz",
Version: "20.15.0",
Channel: "stable",
OS: "linux",
Arch: "x86_64",
Format: ".tar.gz",
LTS: true,
},
}
t.Run("LTS selects older LTS version", func(t *testing.T) {
res, err := Resolve(assets, Request{
OS: "linux",
Arch: "x86_64",
LTS: true,
})
if err != nil {
t.Fatal(err)
}
if res.Version != "20.15.0" {
t.Errorf("version = %q, want 20.15.0", res.Version)
}
})
}

View File

@@ -0,0 +1,207 @@
// Package fsstore implements [storage.Store] on the local filesystem.
//
// Directory layout:
//
// {root}/
// {package}.json # asset list
// {package}.updated.txt # unix timestamp (seconds.millis)
//
// Write transactions build the new JSON in memory, then atomically
// rename into place so readers never see a partial file.
package fsstore
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"path/filepath"
"sort"
"strconv"
"strings"
"time"
"github.com/webinstall/webi-installers/internal/lexver"
"github.com/webinstall/webi-installers/internal/storage"
)
// Store is a filesystem-backed asset store.
type Store struct {
root string
}
// Root returns the store's root directory path.
func (s *Store) Root() string {
return s.root
}
// New creates a Store rooted at the given directory.
// The directory is created if it doesn't exist.
func New(root string) (*Store, error) {
if err := os.MkdirAll(root, 0o755); err != nil {
return nil, fmt.Errorf("fsstore: create root: %w", err)
}
return &Store{root: root}, nil
}
// ListPackages returns the names of all cached packages.
func (s *Store) ListPackages(_ context.Context) ([]string, error) {
dir := s.root
entries, err := os.ReadDir(dir)
if os.IsNotExist(err) {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("fsstore: list packages: %w", err)
}
var pkgs []string
for _, e := range entries {
if strings.HasSuffix(e.Name(), ".json") {
pkgs = append(pkgs, strings.TrimSuffix(e.Name(), ".json"))
}
}
return pkgs, nil
}
// Load reads a package's cached assets from disk.
// Returns nil (not an error) if the package is not cached.
func (s *Store) Load(_ context.Context, pkg string) (*storage.PackageData, error) {
jsonPath := filepath.Join(s.root, pkg+".json")
data, err := os.ReadFile(jsonPath)
if os.IsNotExist(err) {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("fsstore: read %s: %w", pkg, err)
}
// Decode via legacy format (Node.js compat: "releases", "name", "ext").
var lc storage.LegacyCache
if err := json.Unmarshal(data, &lc); err != nil {
return nil, fmt.Errorf("fsstore: decode %s: %w", pkg, err)
}
pd := storage.ImportLegacy(lc)
// Read the timestamp file.
tsPath := filepath.Join(s.root, pkg+".updated.txt")
if tsData, err := os.ReadFile(tsPath); err == nil {
pd.UpdatedAt = parseTimestamp(strings.TrimSpace(string(tsData)))
}
return &pd, nil
}
// BeginRefresh starts a write transaction for a package.
func (s *Store) BeginRefresh(_ context.Context, pkg string) (storage.RefreshTx, error) {
return &refreshTx{
store: s,
pkg: pkg,
}, nil
}
type refreshTx struct {
store *Store
pkg string
assets []storage.Asset
}
func (tx *refreshTx) Put(assets []storage.Asset) error {
tx.assets = append(tx.assets, assets...)
return nil
}
func (tx *refreshTx) Commit(_ context.Context) error {
now := time.Now()
dir := tx.store.root
// Sort assets: stable/lts first, then beta, then rc, then alpha;
// within each channel, newest version first.
// The Node.js resolver picks the first matching entry, so stable[0] = latest stable
// must come before beta of a higher version number.
sort.SliceStable(tx.assets, func(i, j int) bool {
ri, rj := channelRank(tx.assets[i].Channel), channelRank(tx.assets[j].Channel)
if ri != rj {
return ri < rj
}
return lexver.Compare(lexver.Parse(tx.assets[i].Version), lexver.Parse(tx.assets[j].Version)) > 0
})
// Encode via legacy format (Node.js compat: "releases", "name", "ext").
// ExportLegacy applies per-package field backports and drops assets that
// can't be expressed in the legacy format (variants, unsupported formats).
lc, drops := storage.ExportLegacy(tx.pkg, storage.PackageData{Assets: tx.assets})
if drops.Variants > 0 || drops.Formats > 0 {
log.Printf(" %s: legacy export dropped %d variant assets, %d unsupported-format assets",
tx.pkg, drops.Variants, drops.Formats)
}
data, err := json.MarshalIndent(lc, "", " ")
if err != nil {
return fmt.Errorf("fsstore: encode %s: %w", tx.pkg, err)
}
// Write JSON atomically via temp file + rename.
jsonPath := filepath.Join(dir, tx.pkg+".json")
if err := atomicWrite(jsonPath, data); err != nil {
return err
}
// Write timestamp file.
tsPath := filepath.Join(dir, tx.pkg+".updated.txt")
ts := fmt.Sprintf("%.3f", float64(now.UnixMilli())/1000.0)
if err := atomicWrite(tsPath, []byte(ts)); err != nil {
return err
}
tx.assets = nil
return nil
}
func (tx *refreshTx) Rollback() error {
tx.assets = nil
return nil
}
// atomicWrite writes data to path via a temp file + rename.
func atomicWrite(path string, data []byte) error {
tmp := path + ".tmp"
if err := os.WriteFile(tmp, data, 0o644); err != nil {
return fmt.Errorf("fsstore: write tmp: %w", err)
}
if err := os.Rename(tmp, path); err != nil {
os.Remove(tmp)
return fmt.Errorf("fsstore: rename: %w", err)
}
return nil
}
// channelRank returns a sort key for release channels so stable sorts first.
// Lower rank = sorted earlier (stable/lts before beta/rc/alpha).
func channelRank(channel string) int {
switch channel {
case "", "stable", "lts":
return 0
case "rc":
return 1
case "beta":
return 2
case "alpha":
return 3
default:
return 4
}
}
// parseTimestamp parses the "seconds.millis" format from .updated.txt files.
func parseTimestamp(s string) time.Time {
f, err := strconv.ParseFloat(s, 64)
if err != nil || f == 0 {
return time.Time{}
}
sec := int64(f)
nsec := int64((f - float64(sec)) * 1e9)
return time.Unix(sec, nsec)
}

View File

@@ -0,0 +1,138 @@
package fsstore_test
import (
"context"
"testing"
"github.com/webinstall/webi-installers/internal/storage"
"github.com/webinstall/webi-installers/internal/storage/fsstore"
)
func TestRoundTrip(t *testing.T) {
dir := t.TempDir()
s, err := fsstore.New(dir)
if err != nil {
t.Fatal(err)
}
ctx := context.Background()
// Initially empty.
pd, err := s.Load(ctx, "bat")
if err != nil {
t.Fatal(err)
}
if pd != nil {
t.Fatal("expected nil for uncached package")
}
// Write some assets.
tx, err := s.BeginRefresh(ctx, "bat")
if err != nil {
t.Fatal(err)
}
tx.Put([]storage.Asset{
{
Filename: "bat-v0.26.1-aarch64-apple-darwin.tar.gz",
Version: "0.26.1",
Channel: "stable",
Date: "2025-12-02",
OS: "darwin",
Arch: "aarch64",
Format: ".tar.gz",
Download: "https://github.com/sharkdp/bat/releases/download/v0.26.1/bat-v0.26.1-aarch64-apple-darwin.tar.gz",
},
{
Filename: "bat-v0.26.1-x86_64-unknown-linux-gnu.tar.gz",
Version: "0.26.1",
Channel: "stable",
Date: "2025-12-02",
OS: "linux",
Arch: "x86_64",
Libc: "gnu",
Format: ".tar.gz",
Download: "https://github.com/sharkdp/bat/releases/download/v0.26.1/bat-v0.26.1-x86_64-unknown-linux-gnu.tar.gz",
},
})
if err := tx.Commit(ctx); err != nil {
t.Fatal(err)
}
// Read back.
pd, err = s.Load(ctx, "bat")
if err != nil {
t.Fatal(err)
}
if pd == nil {
t.Fatal("expected data after write")
}
if len(pd.Assets) != 2 {
t.Fatalf("got %d assets, want 2", len(pd.Assets))
}
if pd.Assets[0].Filename != "bat-v0.26.1-aarch64-apple-darwin.tar.gz" {
t.Errorf("asset[0].Filename = %q", pd.Assets[0].Filename)
}
if pd.Assets[1].OS != "linux" {
t.Errorf("asset[1].OS = %q", pd.Assets[1].OS)
}
if pd.UpdatedAt.IsZero() {
t.Error("UpdatedAt should be set")
}
}
func TestRollback(t *testing.T) {
dir := t.TempDir()
s, err := fsstore.New(dir)
if err != nil {
t.Fatal(err)
}
ctx := context.Background()
tx, err := s.BeginRefresh(ctx, "bat")
if err != nil {
t.Fatal(err)
}
tx.Put([]storage.Asset{{Filename: "test", Version: "1.0"}})
tx.Rollback()
pd, err := s.Load(ctx, "bat")
if err != nil {
t.Fatal(err)
}
if pd != nil {
t.Fatal("expected nil after rollback")
}
}
func TestReadLegacyFormat(t *testing.T) {
dir := t.TempDir()
s, err := fsstore.New(dir)
if err != nil {
t.Fatal(err)
}
ctx := context.Background()
// Write assets and read back — the JSON uses "releases" key
// and "name"/"ext" field names for Node.js compat.
tx, _ := s.BeginRefresh(ctx, "aliasman")
tx.Put([]storage.Asset{
{
Filename: "BeyondCodeBootcamp-aliasman-v1.1.2-0-g0e5e1c1.tar.gz",
Version: "v1.1.2",
Channel: "stable",
Date: "2023-02-23",
OS: "posix_2017",
Arch: "*",
Format: "",
Download: "https://codeload.github.com/BeyondCodeBootcamp/aliasman/legacy.tar.gz/refs/tags/v1.1.2",
},
})
tx.Commit(ctx)
pd, err := s.Load(ctx, "aliasman")
if err != nil {
t.Fatal(err)
}
if pd.Assets[0].OS != "posix_2017" {
t.Errorf("OS = %q, want posix_2017", pd.Assets[0].OS)
}
}

444
internal/storage/legacy.go Normal file
View File

@@ -0,0 +1,444 @@
package storage
import (
"sort"
"strings"
)
// Legacy types for reading/writing the Node.js _cache/ JSON format.
//
// The Node.js server calls assets "releases" and uses "name" for the
// filename and "ext" for the format. These types preserve that wire
// format for backward compatibility during migration.
//
// Internal Go code uses [Asset] and [PackageData] directly.
// LegacyAsset matches the JSON shape the Node.js server writes and reads.
type LegacyAsset struct {
Name string `json:"name"`
Version string `json:"version"`
GitTag string `json:"git_tag,omitempty"`
GitCommitHash string `json:"git_commit_hash,omitempty"`
LTS bool `json:"lts"`
Channel string `json:"channel"`
Date string `json:"date"`
OS string `json:"os"`
Arch string `json:"arch"`
Libc string `json:"libc"`
Ext string `json:"ext"`
Download string `json:"download"`
}
// LegacyCache matches the top-level JSON shape in _cache/{pkg}.json.
type LegacyCache struct {
OSes []string `json:"oses,omitempty"`
Arches []string `json:"arches,omitempty"`
Libcs []string `json:"libcs,omitempty"`
Formats []string `json:"formats,omitempty"`
Releases []LegacyAsset `json:"releases"`
Download string `json:"download"`
}
// LegacyDropStats reports how many assets were excluded during ExportLegacy.
type LegacyDropStats struct {
Variants int // dropped: has build variant tags (e.g. rocm, installer, fxdependent)
Formats int // dropped: format not recognized by the Node.js server
Android int // dropped: android OS — classifier maps android filenames to linux
NoTarget int // dropped: no OS and no arch — unclassifiable source tarballs
}
// ToAsset converts a LegacyAsset to the internal Asset type.
// It reverses the key vocabulary translations applied by toLegacy so that
// the internal (Go canonical) representation is preserved.
func (la LegacyAsset) ToAsset() Asset {
// Reverse-translate legacy Node.js vocabulary to Go canonical names.
// toLegacy writes macos/amd64/arm64; internal code uses darwin/x86_64/aarch64.
// "none" libc is buildmeta.LibcNone — preserve it (don't collapse to "").
os := la.OS
switch os {
case "macos":
os = "darwin"
case "*":
os = ""
}
arch := la.Arch
switch arch {
case "amd64":
arch = "x86_64"
case "arm64":
arch = "aarch64"
case "*":
arch = ""
}
// Restore the dot-prefix convention used throughout internal Go code.
// The cache stores ext without a leading dot (e.g. "tar.gz", "zip", "exe"),
// but Asset.Format uses dotted strings (e.g. ".tar.gz", ".zip", ".exe").
// "exe" is ambiguous: bare binary (no .exe suffix) vs Windows .exe file.
// Disambiguate by checking whether the filename ends with ".exe".
format := la.Ext
switch {
case format == "exe" && !strings.HasSuffix(strings.ToLower(la.Name), ".exe"):
format = "" // bare binary — internal convention is empty string
case format != "":
format = "." + format // restore dot prefix for internal use
}
return Asset{
Filename: la.Name,
Version: la.Version,
LTS: la.LTS,
Channel: la.Channel,
Date: la.Date,
OS: os,
Arch: arch,
Libc: la.Libc,
Format: format,
Download: la.Download,
GitTag: la.GitTag,
GitCommitHash: la.GitCommitHash,
}
}
// toLegacy converts an Asset to the LegacyAsset wire format.
// Callers must have already applied legacyFieldBackport before calling this.
func (a Asset) toLegacy() LegacyAsset {
libc := a.Libc
if libc == "" {
libc = "none" // API expects "none" rather than empty string
}
// Strip leading dot: API expects "tar.gz" not ".tar.gz".
ext := strings.TrimPrefix(a.Format, ".")
// Bare binaries: API expects "exe". Internal convention is Format=""
// for bare binaries (no archive extension). By the time we reach
// toLegacy, source tarballs and git-clone entries have been filtered
// or tagged, so Format="" reliably means bare binary.
if ext == "" {
ext = "exe"
}
return LegacyAsset{
Name: a.Filename,
Version: strings.TrimPrefix(a.Version, "v"), // API expects no v-prefix
GitTag: a.GitTag,
GitCommitHash: a.GitCommitHash,
LTS: a.LTS,
Channel: a.Channel,
Date: a.Date,
OS: a.OS,
Arch: a.Arch,
Libc: libc,
Ext: ext,
Download: a.Download,
}
}
// legacyFieldBackport translates canonical classifier field values to the
// values the legacy Node.js resolver expects. This is called at export time
// only — the canonical values are preserved in Go-native storage (pgstore).
//
// The Node build-classifier re-parses each asset's download filename and drops
// any entry where the cache field doesn't match what it extracts from the name.
// These translations ensure the cache matches the classifier's extraction.
//
// Global OS translations:
// - sunos → solaris: Node's classifier maps "sunos" filenames to "solaris".
// LIVE_cache has "solaris" and "illumos" but never "sunos".
//
// Global arch translations (all packages):
// - universal2/universal1 → x86_64: classifier maps "universal" in filename
// to x86_64. The darwin WATERFALL falls back aarch64→x86_64, so arm64
// users still receive these builds.
// - x86_64_v2/v3/v4 → x86_64: AMD64 microarch levels not in LIVE_cache;
// fold to baseline x86_64.
// - mips64r6 → mips64: exotic MIPS64R6, not in LIVE_cache.
// - mips64r6el → mips64le: exotic MIPS64R6 little-endian, not in LIVE_cache.
// - ARM (filename-based): explicit armvN takes priority over ABI tags.
// Go normalizes these; see legacyARMArchFromFilename for filename extraction.
// Final ARM vocab mapping to LIVE_cache values:
// armv6→armv6l, armv7a→armv7l, armhf→armv7l, armel→arm.
// - powerpc (32-bit): not in LIVE_cache; entry is dropped.
//
// Note: mipsle and mips64le are kept as-is — LIVE_cache uses these exact values.
// Note: solaris and illumos are kept as-is — both exist in LIVE_cache.
//
// Package-specific rules replicate per-package overrides in production's releases.js:
// - ffmpeg: Windows .gz → .exe (prod releases.js: rel.ext = 'exe')
//
// Git-clone entries:
// - format="git" with empty OS/arch → os="*", arch="*"
// The legacy cache uses "*" for ANYOS/ANYARCH (builds-cacher LEGACY_OS_MAP['*']='ANYOS').
// vim plugins, aliasman, serviceman, and other POSIX packages use this format.
func legacyFieldBackport(pkg string, a Asset) Asset {
// Git-clone entries are ANYOS/ANYARCH — legacy cache uses "*" for these.
// This matches production LIVE_cache for vim-commentary, aliasman, etc.
if a.Format == "git" {
if a.OS == "" {
a.OS = "*"
}
if a.Arch == "" {
a.Arch = "*"
}
}
// sunos → solaris: Node's classifier maps "sunos" filenames to "solaris".
// LIVE_cache has "solaris" and "illumos" but never "sunos".
if a.OS == "sunos" {
a.OS = "solaris"
}
// darwin → macos: LIVE_cache pre-classified packages (go, node, zig, fish, etc.)
// use "macos". Julia is the sole exception — LIVE julia.json uses "darwin".
if a.OS == "darwin" && pkg != "julia" {
a.OS = "macos"
}
// Universal fat binaries: expandUniversal splits these into per-arch
// entries earlier in the pipeline. This is a safety fallback in case
// any universal entries reach the legacy export unexpectedly.
if a.Arch == "universal2" || a.Arch == "universal1" {
a.Arch = "x86_64"
}
// AMD64 microarch levels: not in LIVE_cache; fold to baseline x86_64.
switch a.Arch {
case "x86_64_v2", "x86_64_v3", "x86_64_v4":
a.Arch = "x86_64"
}
// x86_64 → amd64, aarch64 → arm64: LIVE_cache pre-classified packages use
// "amd64" and "arm64". Go's classifier uses "x86_64" and "aarch64".
// These come after universal2→x86_64 and x86_64_v*/→x86_64 so the chains work.
if a.Arch == "x86_64" {
a.Arch = "amd64"
}
if a.Arch == "aarch64" {
a.Arch = "arm64"
}
// MIPS variants not in LIVE_cache: fold to nearest supported value.
// mipsle and mips64le are kept as-is — LIVE_cache uses these exact spellings.
switch a.Arch {
case "mips64r6":
a.Arch = "mips64"
case "mips64r6el":
a.Arch = "mips64le"
}
// powerpc (32-bit): not in LIVE_cache; mark for drop by clearing both fields.
// Per-package taggers (uuidv7, watchexec) handle this via variant tags, but
// for any package without a tagger, clear here so the NoTarget filter drops it.
if a.Arch == "powerpc" {
a.OS = ""
a.Arch = ""
}
// ARM arch: the Node classifier re-parses filenames and expects the cache
// arch to match what it extracts. Go normalizes arch values; use filename
// heuristics to match what Node would extract.
switch a.Arch {
case "armv5", "armv6", "armv7":
if leg := legacyARMArchFromFilename(a.Filename); leg != "" {
a.Arch = leg
}
}
// Translate ARM arch values to LIVE_cache vocabulary.
// legacyARMArchFromFilename can produce armhf/armel/armv7a which aren't
// in LIVE_cache; also translate raw armv6/armv7 (when no filename override).
switch a.Arch {
case "armv6":
a.Arch = "armv6l"
case "armv7":
a.Arch = "armv7l"
case "armhf":
a.Arch = "armv7l"
case "armel":
a.Arch = "arm"
case "armv7a":
a.Arch = "armv7l"
}
switch pkg {
case "ffmpeg":
if a.OS == "windows" {
switch a.Format {
case ".gz", "":
a.Format = ".exe"
}
}
}
return a
}
// legacyARMArchFromFilename returns the arch string the Node build-classifier
// would extract from a filename for ARM-family builds. Returns "" when the
// Go canonical arch value already matches what the classifier would extract.
//
// The Node classifier's extraction rules differ from Go's normalization:
// - armv7a (explicit) → "armv7a" (not "armv7")
// - armv7 (explicit, e.g. "armv7-unknown-linux-gnueabihf") → "armv7"
// The explicit version number takes priority over the ABI suffix.
// - arm-5 / arm-7 (Gitea naming: "linux-arm-5", "linux-arm-7") → "armel" / "armv7"
// patternToTerms converts "arm-5" → "armv5" and "arm-7" → "armv7".
// - armv6hf (shellcheck naming) → "armhf" (tpm['armv6hf'] = ARMHF)
// - gnueabihf (Rust triplet, no explicit armvN) → "armhf"
// - armhf (Debian armhf) → "armhf"
// - armel (Debian soft-float ABI) → "armel" (not "armv6")
// - armv5 (explicit) → "armel" (Node tiered map: armv5 falls back to armel)
func legacyARMArchFromFilename(filename string) string {
lower := strings.ToLower(filename)
// armv7a before armv7 — "armv7a" contains "armv7" as a prefix.
if strings.Contains(lower, "armv7a") {
return "armv7a"
}
// Explicit armv7 in filename: takes priority over ABI suffix (gnueabihf).
// e.g. "armv7-unknown-linux-gnueabihf" → classifier extracts "armv7".
if strings.Contains(lower, "armv7") {
return "armv7"
}
// armv6hf (shellcheck naming): tpm['armv6hf'] = ARMHF → "armhf".
if strings.Contains(lower, "armv6hf") {
return "armhf"
}
// Gitea arm-N naming: "linux-arm-5" → patternToTerms → "armv5" → armel.
if strings.Contains(lower, "arm-5") {
return "armel"
}
// Gitea arm-N naming: "linux-arm-7" → patternToTerms → "armv7" → armv7.
if strings.Contains(lower, "arm-7") {
return "armv7"
}
// Rust gnueabihf triplet (no explicit armvN): classifier → "armhf".
if strings.Contains(lower, "gnueabihf") {
return "armhf"
}
// Debian armhf (hard-float ABI): classifier → "armhf".
if strings.Contains(lower, "armhf") {
return "armhf"
}
if strings.Contains(lower, "armel") {
return "armel"
}
if strings.Contains(lower, "armv5") {
return "armel"
}
return ""
}
// ImportLegacy converts a LegacyCache to PackageData.
func ImportLegacy(lc LegacyCache) PackageData {
assets := make([]Asset, len(lc.Releases))
for i, la := range lc.Releases {
assets[i] = la.ToAsset()
}
return PackageData{Assets: assets}
}
// legacyFormats is the set of formats the Node.js server recognizes.
// Assets with formats not in this set are filtered out of legacy exports.
var legacyFormats = map[string]bool{
".zip": true,
".tar.gz": true,
".tar.xz": true,
".tar.zst": true,
".tar.bz2": true,
".tar": true,
".xz": true,
".7z": true,
".pkg": true,
".msi": true,
".exe": true,
".exe.xz": true,
".dmg": true,
".app.zip": true,
".gz": true,
"git": true,
}
// ExportLegacy converts canonical PackageData to the LegacyCache wire format.
//
// The pkg name is used to apply per-package field translations (see legacyFieldBackport).
// Assets are excluded when:
// - Variants is non-empty (Node.js has no variant logic)
// - OS is android (classifier maps android filenames to linux)
// - OS and arch are both empty (unclassifiable source tarballs)
// - Format is non-empty and not in the Node.js recognized set
//
// Dropped counts are returned in LegacyDropStats for logging.
func ExportLegacy(pkg string, pd PackageData) (LegacyCache, LegacyDropStats) {
var releases []LegacyAsset
var stats LegacyDropStats
for _, a := range pd.Assets {
// Skip variant builds — Node.js doesn't have variant logic.
if len(a.Variants) > 0 {
stats.Variants++
continue
}
// Skip android — classifier maps android filenames to linux OS,
// which mismatches cache entries tagged android.
if a.OS == "android" {
stats.Android++
continue
}
// Skip entries with no OS and no arch, unless they're git-clone packages.
// Source tarballs (cmake, dashcore, bun npm) have format != "git".
// Git-clone packages (vim plugins, aliasman) legitimately have no OS/arch —
// legacyFieldBackport will translate them to os="*", arch="*".
if a.OS == "" && a.Arch == "" && a.Format != "git" {
stats.NoTarget++
continue
}
// Apply per-package and global legacy field translations.
a = legacyFieldBackport(pkg, a)
// Skip formats Node.js doesn't recognize.
if a.Format != "" && !legacyFormats[a.Format] {
stats.Formats++
continue
}
releases = append(releases, a.toLegacy())
}
if releases == nil {
releases = []LegacyAsset{}
}
// Build sorted summary arrays from the included releases.
// These let the API skip normalize.js vocabulary filtering entirely.
oSet := map[string]bool{}
aSet := map[string]bool{}
lSet := map[string]bool{}
fSet := map[string]bool{}
for _, r := range releases {
if r.OS != "" && r.OS != "*" {
oSet[r.OS] = true
}
if r.Arch != "" && r.Arch != "*" {
aSet[r.Arch] = true
}
if r.Libc != "" {
lSet[r.Libc] = true
}
if r.Ext != "" {
fSet[strings.TrimPrefix(r.Ext, ".")] = true
}
}
lc := LegacyCache{
OSes: sortedKeys(oSet),
Arches: sortedKeys(aSet),
Libcs: sortedKeys(lSet),
Formats: sortedKeys(fSet),
Releases: releases,
}
return lc, stats
}
// sortedKeys returns the keys of a string set in sorted order.
func sortedKeys(m map[string]bool) []string {
if len(m) == 0 {
return nil
}
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}

View File

@@ -0,0 +1,609 @@
package storage_test
import (
"encoding/json"
"testing"
"github.com/webinstall/webi-installers/internal/storage"
)
// TestDecodeLegacyJSON verifies we can parse the exact JSON format
// the Node.js server writes to _cache/.
func TestDecodeLegacyJSON(t *testing.T) {
// Real data from _cache/2026-03/aliasman.json.
raw := `{
"releases": [
{
"name": "BeyondCodeBootcamp-aliasman-v1.1.2-0-g0e5e1c1.tar.gz",
"version": "v1.1.2",
"lts": false,
"channel": "stable",
"date": "2023-02-23",
"os": "posix_2017",
"arch": "*",
"libc": "",
"ext": "",
"download": "https://codeload.github.com/BeyondCodeBootcamp/aliasman/legacy.tar.gz/refs/tags/v1.1.2"
},
{
"name": "BeyondCodeBootcamp-aliasman-v1.1.2-0-g0e5e1c1.zip",
"version": "v1.1.2",
"lts": false,
"channel": "stable",
"date": "2023-02-23",
"os": "posix_2017",
"arch": "*",
"libc": "",
"ext": "",
"download": "https://codeload.github.com/BeyondCodeBootcamp/aliasman/legacy.zip/refs/tags/v1.1.2"
}
],
"download": ""
}`
var lc storage.LegacyCache
if err := json.Unmarshal([]byte(raw), &lc); err != nil {
t.Fatal(err)
}
if len(lc.Releases) != 2 {
t.Fatalf("got %d releases, want 2", len(lc.Releases))
}
pd := storage.ImportLegacy(lc)
if len(pd.Assets) != 2 {
t.Fatalf("got %d assets, want 2", len(pd.Assets))
}
a := pd.Assets[0]
if a.Filename != "BeyondCodeBootcamp-aliasman-v1.1.2-0-g0e5e1c1.tar.gz" {
t.Errorf("Filename = %q", a.Filename)
}
if a.Version != "v1.1.2" {
t.Errorf("Version = %q", a.Version)
}
if a.OS != "posix_2017" {
t.Errorf("OS = %q", a.OS)
}
if a.Arch != "" {
t.Errorf("Arch = %q, want %q (wildcard '*' reversed to empty)", a.Arch, "")
}
if a.Download != "https://codeload.github.com/BeyondCodeBootcamp/aliasman/legacy.tar.gz/refs/tags/v1.1.2" {
t.Errorf("Download = %q", a.Download)
}
// Round-trip: export back to legacy and verify JSON shape.
lc2, _ := storage.ExportLegacy("aliasman", pd)
data, _ := json.MarshalIndent(lc2, "", " ")
var lc3 storage.LegacyCache
json.Unmarshal(data, &lc3)
if lc3.Releases[0].Name != a.Filename {
t.Errorf("round-trip Name = %q, want %q", lc3.Releases[0].Name, a.Filename)
}
// Legacy data has ext:"" for this tarball — broken cache entry.
// toLegacy normalizes Format="" to ext:"exe" (bare binary convention).
// In the real Go pipeline, aliasman would have Format=".tar.gz".
if lc3.Releases[0].Ext != "exe" {
t.Errorf("round-trip Ext = %q, want %q", lc3.Releases[0].Ext, "exe")
}
}
// TestExportLegacyDrops verifies that ExportLegacy correctly drops and counts
// assets that can't be represented in the Node.js legacy cache format.
func TestExportLegacyDrops(t *testing.T) {
t.Run("variant_builds_dropped", func(t *testing.T) {
// Assets with variant tags (rocm, installer, fxdependent, etc.) are
// dropped because Node.js has no variant-selection logic.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "ollama-linux-amd64-rocm.tgz", OS: "linux", Arch: "x86_64", Format: ".tar.gz", Variants: []string{"rocm"}},
{Filename: "ollama-linux-amd64.tgz", OS: "linux", Arch: "x86_64", Format: ".tar.gz"},
},
}
lc, stats := storage.ExportLegacy("ollama", pd)
if stats.Variants != 1 {
t.Errorf("Variants dropped = %d, want 1", stats.Variants)
}
if len(lc.Releases) != 1 {
t.Errorf("releases = %d, want 1 (baseline only)", len(lc.Releases))
}
if lc.Releases[0].Name != "ollama-linux-amd64.tgz" {
t.Errorf("kept wrong release: %q", lc.Releases[0].Name)
}
})
t.Run("android_dropped", func(t *testing.T) {
// Android entries are dropped: the classifier maps android filenames to
// linux OS and then rejects the cache entry that says android.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "fzf-0.57.0-android-arm64.tar.gz", OS: "android", Arch: "aarch64", Format: ".tar.gz"},
{Filename: "fzf-0.57.0-linux-arm64.tar.gz", OS: "linux", Arch: "aarch64", Format: ".tar.gz"},
},
}
lc, stats := storage.ExportLegacy("fzf", pd)
if stats.Android != 1 {
t.Errorf("Android dropped = %d, want 1", stats.Android)
}
if len(lc.Releases) != 1 {
t.Errorf("releases = %d, want 1 (linux only)", len(lc.Releases))
}
})
t.Run("unknown_formats_dropped", func(t *testing.T) {
// .AppImage, .deb, .rpm are not in the Node.js format set.
// Assets have Arch set (matching real classifier output for these formats).
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "tool.AppImage", OS: "linux", Arch: "x86_64", Format: ".AppImage"},
{Filename: "tool.deb", OS: "linux", Arch: "x86_64", Format: ".deb"},
{Filename: "tool.rpm", OS: "linux", Arch: "x86_64", Format: ".rpm"},
{Filename: "tool-linux-amd64.tar.gz", OS: "linux", Arch: "x86_64", Format: ".tar.gz"},
},
}
lc, stats := storage.ExportLegacy("tool", pd)
if stats.Formats != 3 {
t.Errorf("Formats dropped = %d, want 3", stats.Formats)
}
if len(lc.Releases) != 1 {
t.Errorf("releases = %d, want 1 (tar.gz only)", len(lc.Releases))
}
})
t.Run("empty_format_passes_through", func(t *testing.T) {
// Assets with empty format (e.g. bare binaries, git sources) pass through.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "jq-linux-amd64", OS: "linux", Arch: "x86_64", Format: ""},
},
}
lc, stats := storage.ExportLegacy("jq", pd)
if stats.Formats != 0 {
t.Errorf("Formats dropped = %d, want 0", stats.Formats)
}
if len(lc.Releases) != 1 {
t.Errorf("releases = %d, want 1", len(lc.Releases))
}
})
}
// TestExportLegacyTranslations verifies that legacyFieldBackport applies the
// correct field translations for Node.js compatibility.
func TestExportLegacyTranslations(t *testing.T) {
t.Run("universal2_translated_to_amd64", func(t *testing.T) {
// universal2 fat binaries: the Node classifier sees "universal" in the
// filename and maps it to x86_64. Cache must say amd64 (via universal2→x86_64→amd64
// chain) to match. The darwin WATERFALL (arm64 → [arm64, amd64]) means arm64
// users also receive these builds as a fallback.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "hugo_0.145.0_darwin-universal.tar.gz", OS: "darwin", Arch: "universal2", Format: ".tar.gz"},
{Filename: "hugo_0.145.0_darwin-arm64.tar.gz", OS: "darwin", Arch: "aarch64", Format: ".tar.gz"},
},
}
lc, stats := storage.ExportLegacy("hugo", pd)
if stats.Variants != 0 || stats.Formats != 0 || stats.Android != 0 {
t.Errorf("unexpected drops: %+v", stats)
}
if len(lc.Releases) != 2 {
t.Fatalf("releases = %d, want 2", len(lc.Releases))
}
var universal2Arch string
for _, r := range lc.Releases {
if r.Name == "hugo_0.145.0_darwin-universal.tar.gz" {
universal2Arch = r.Arch
}
}
if universal2Arch != "amd64" {
t.Errorf("universal2 arch in legacy = %q, want amd64 (universal2→x86_64→amd64)", universal2Arch)
}
})
t.Run("solaris_kept_as_is", func(t *testing.T) {
// Solaris/illumos/sunos are kept as-is. The build-classifier (triplet.js)
// recognizes all three as distinct values and matches them correctly.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "go1.20.1.solaris-amd64.tar.gz", OS: "solaris", Arch: "x86_64", Format: ".tar.gz"},
},
}
lc, stats := storage.ExportLegacy("go", pd)
if stats.Android != 0 || stats.Variants != 0 || stats.Formats != 0 {
t.Errorf("unexpected drops: %+v", stats)
}
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].OS != "solaris" {
t.Errorf("OS = %q, want solaris", lc.Releases[0].OS)
}
})
t.Run("illumos_kept_as_is", func(t *testing.T) {
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "go1.20.1.illumos-amd64.tar.gz", OS: "illumos", Arch: "x86_64", Format: ".tar.gz"},
},
}
lc, _ := storage.ExportLegacy("go", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].OS != "illumos" {
t.Errorf("OS = %q, want illumos", lc.Releases[0].OS)
}
})
t.Run("darwin_to_macos", func(t *testing.T) {
// All packages except julia translate darwin → macos.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "go1.20.1.darwin-amd64.tar.gz", OS: "darwin", Arch: "aarch64", Format: ".tar.gz"},
},
}
lc, _ := storage.ExportLegacy("go", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].OS != "macos" {
t.Errorf("OS = %q, want macos (darwin → macos)", lc.Releases[0].OS)
}
})
t.Run("julia_darwin_kept_as_is", func(t *testing.T) {
// julia is the sole exception: LIVE julia.json uses "darwin", not "macos".
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "julia-1.9.3-mac64.tar.gz", OS: "darwin", Arch: "aarch64", Format: ".tar.gz"},
},
}
lc, _ := storage.ExportLegacy("julia", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].OS != "darwin" {
t.Errorf("OS = %q, want darwin (julia exception — LIVE uses darwin)", lc.Releases[0].OS)
}
})
t.Run("x86_64_v2_to_amd64", func(t *testing.T) {
// Micro-arch levels (v2/v3/v4): fold to baseline x86_64, then x86_64→amd64.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "tool-linux-x86_64_v2.tar.gz", OS: "linux", Arch: "x86_64_v2", Format: ".tar.gz"},
},
}
lc, _ := storage.ExportLegacy("tool", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].Arch != "amd64" {
t.Errorf("arch = %q, want amd64 (x86_64_v2 → x86_64 → amd64)", lc.Releases[0].Arch)
}
})
t.Run("mips64r6_folded", func(t *testing.T) {
// mips64r6/mips64r6el: exotic variants not in LIVE_cache; fold to mips64/mips64le.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "tool-linux-mips64r6.tar.gz", OS: "linux", Arch: "mips64r6", Format: ".tar.gz"},
{Filename: "tool-linux-mips64r6el.tar.gz", OS: "linux", Arch: "mips64r6el", Format: ".tar.gz"},
},
}
lc, _ := storage.ExportLegacy("tool", pd)
if len(lc.Releases) != 2 {
t.Fatalf("releases = %d, want 2", len(lc.Releases))
}
if lc.Releases[0].Arch != "mips64" {
t.Errorf("arch = %q, want mips64 (mips64r6 → mips64)", lc.Releases[0].Arch)
}
if lc.Releases[1].Arch != "mips64le" {
t.Errorf("arch = %q, want mips64le (mips64r6el → mips64le)", lc.Releases[1].Arch)
}
})
t.Run("mipsle_unchanged", func(t *testing.T) {
// mipsle: LIVE_cache uses "mipsle" — keep as-is.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "caddy_linux_mipsle.tar.gz", OS: "linux", Arch: "mipsle", Format: ".tar.gz"},
},
}
lc, _ := storage.ExportLegacy("caddy", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].Arch != "mipsle" {
t.Errorf("arch = %q, want mipsle (LIVE_cache uses mipsle)", lc.Releases[0].Arch)
}
})
t.Run("mips64le_unchanged", func(t *testing.T) {
// mips64le: LIVE_cache uses "mips64le" — keep as-is.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "gitea-linux-mips64le.tar.gz", OS: "linux", Arch: "mips64le", Format: ".tar.gz"},
},
}
lc, _ := storage.ExportLegacy("gitea", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].Arch != "mips64le" {
t.Errorf("arch = %q, want mips64le (LIVE_cache uses mips64le)", lc.Releases[0].Arch)
}
})
t.Run("ffmpeg_windows_gz_to_exe", func(t *testing.T) {
// ffmpeg Windows releases are .gz archives containing a bare .exe.
// Production releases.js overrides ext to 'exe' for install compatibility.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "ffmpeg-7.0-windows-amd64.gz", OS: "windows", Arch: "x86_64", Format: ".gz"},
{Filename: "ffmpeg-7.0-linux-amd64.tar.gz", OS: "linux", Arch: "x86_64", Format: ".tar.gz"},
},
}
lc, _ := storage.ExportLegacy("ffmpeg", pd)
if len(lc.Releases) != 2 {
t.Fatalf("releases = %d, want 2", len(lc.Releases))
}
var windowsExt string
for _, r := range lc.Releases {
if r.OS == "windows" {
windowsExt = r.Ext
}
}
if windowsExt != "exe" {
t.Errorf("ffmpeg windows ext = %q, want exe", windowsExt)
}
})
t.Run("ffmpeg_translation_not_applied_to_other_packages", func(t *testing.T) {
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "othertool-windows-amd64.gz", OS: "windows", Arch: "x86_64", Format: ".gz"},
},
}
lc, _ := storage.ExportLegacy("othertool", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].Ext != "gz" {
t.Errorf("ext = %q, want gz (no translation outside ffmpeg)", lc.Releases[0].Ext)
}
})
// ARM arch translations: translate Go-canonical values to LIVE_cache vocabulary.
// LIVE_cache uses: armv6l, armv7l, armv7, arm (not armv6, armhf, armel, armv7a).
t.Run("arm_gnueabihf_to_armv7l", func(t *testing.T) {
// gnueabihf ABI suffix (no explicit armvN): filename → armhf → armv7l
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "bat-v0.9.0-arm-unknown-linux-gnueabihf.tar.gz", OS: "linux", Arch: "armv6", Format: ".tar.gz"},
},
}
lc, _ := storage.ExportLegacy("bat", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].Arch != "armv7l" {
t.Errorf("arch = %q, want armv7l (gnueabihf → armhf → armv7l)", lc.Releases[0].Arch)
}
})
t.Run("arm_armhf_to_armv7l", func(t *testing.T) {
// Debian armhf = ARMv7 hard-float; LIVE_cache uses armv7l for this.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "caddy_linux_armhf.tar.gz", OS: "linux", Arch: "armv7", Format: ".tar.gz"},
},
}
lc, _ := storage.ExportLegacy("caddy", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].Arch != "armv7l" {
t.Errorf("arch = %q, want armv7l (armhf → armv7l)", lc.Releases[0].Arch)
}
})
t.Run("arm_armel_to_arm", func(t *testing.T) {
// Debian armel = ARM soft-float; LIVE_cache uses "arm" for this.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "caddy_linux_armel.tar.gz", OS: "linux", Arch: "armv6", Format: ".tar.gz"},
},
}
lc, _ := storage.ExportLegacy("caddy", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].Arch != "arm" {
t.Errorf("arch = %q, want arm (armel → arm)", lc.Releases[0].Arch)
}
})
t.Run("arm_armv5_to_arm", func(t *testing.T) {
// armv5 → legacyARMArchFromFilename → "armel" → "arm"
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "caddy_linux_armv5.tar.gz", OS: "linux", Arch: "armv5", Format: ".tar.gz"},
},
}
lc, _ := storage.ExportLegacy("caddy", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].Arch != "arm" {
t.Errorf("arch = %q, want arm (armv5 → armel → arm)", lc.Releases[0].Arch)
}
})
t.Run("arm_armv7a_to_armv7l", func(t *testing.T) {
// armv7a (ARM application profile): LIVE_cache uses armv7l.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "tool-armv7a-linux.tar.gz", OS: "linux", Arch: "armv7", Format: ".tar.gz"},
},
}
lc, _ := storage.ExportLegacy("tool", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].Arch != "armv7l" {
t.Errorf("arch = %q, want armv7l (armv7a → armv7l)", lc.Releases[0].Arch)
}
})
t.Run("arm_armv7l_filename_to_armv7l", func(t *testing.T) {
// armv7l in filename: legacyARMArchFromFilename extracts "armv7" (armv7l contains armv7),
// then the canonical armv7→armv7l translation maps it to armv7l (the correct API vocab).
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "tool-armv7l-linux.tar.gz", OS: "linux", Arch: "armv7", Format: ".tar.gz"},
},
}
lc, _ := storage.ExportLegacy("tool", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].Arch != "armv7l" {
t.Errorf("arch = %q, want armv7l (armv7l filename → armv7 → armv7l)", lc.Releases[0].Arch)
}
})
t.Run("arm_armv6l_to_armv6l", func(t *testing.T) {
// armv6l in filename: legacyARMArchFromFilename returns "" (no armv7/armhf/etc match).
// armv6 (Go canonical) → armv6l (LIVE_cache vocabulary).
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "tool-armv6l-linux.tar.gz", OS: "linux", Arch: "armv6", Format: ".tar.gz"},
},
}
lc, _ := storage.ExportLegacy("tool", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].Arch != "armv6l" {
t.Errorf("arch = %q, want armv6l (armv6 → armv6l)", lc.Releases[0].Arch)
}
})
t.Run("arm_armv7_gnueabihf_to_armv7l", func(t *testing.T) {
// Files like "ripgrep-14.1.0-armv7-unknown-linux-gnueabihf.tar.gz":
// Go classifies as armv7; the "armv7" term in filename takes priority
// over the gnueabihf ABI suffix. legacyARMArchFromFilename returns "armv7",
// then the canonical armv7→armv7l translation produces armv7l.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "ripgrep-14.1.0-armv7-unknown-linux-gnueabihf.tar.gz", OS: "linux", Arch: "armv7", Format: ".tar.gz"},
},
}
lc, _ := storage.ExportLegacy("ripgrep", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].Arch != "armv7l" {
t.Errorf("arch = %q, want armv7l (armv7 in filename → armv7 → armv7l)", lc.Releases[0].Arch)
}
})
t.Run("arm_armv6hf_to_armhf", func(t *testing.T) {
// shellcheck uses "armv6hf" naming; classifier tpm['armv6hf'] = ARMHF → "armhf".
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "shellcheck-v0.9.0.linux.armv6hf.tar.xz", OS: "linux", Arch: "armv6", Format: ".tar.xz"},
},
}
lc, _ := storage.ExportLegacy("shellcheck", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].Arch != "armv7l" {
t.Errorf("arch = %q, want armv7l (armv6hf → armhf → armv7l)", lc.Releases[0].Arch)
}
})
t.Run("arm_gitea_arm5_to_armel", func(t *testing.T) {
// Gitea uses "arm-5" naming; patternToTerms converts to "armv5" → tpm → "armel".
// Go sees \barm\b → classifies as armv6. Legacy export must correct to armel.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "gitea-1.20.0-linux-arm-5", OS: "linux", Arch: "armv6", Format: ""},
},
}
lc, _ := storage.ExportLegacy("gitea", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].Arch != "arm" {
t.Errorf("arch = %q, want arm (arm-5 → armel → arm)", lc.Releases[0].Arch)
}
})
t.Run("arm_gitea_arm7_to_armv7l", func(t *testing.T) {
// Gitea uses "arm-7" naming; patternToTerms converts to "armv7" → tpm → "armv7".
// Go sees \barm\b → classifies as armv6. legacyARMArchFromFilename returns "armv7",
// then the canonical armv7→armv7l translation produces armv7l.
pd := storage.PackageData{
Assets: []storage.Asset{
{Filename: "gitea-1.20.0-linux-arm-7", OS: "linux", Arch: "armv6", Format: ""},
},
}
lc, _ := storage.ExportLegacy("gitea", pd)
if len(lc.Releases) != 1 {
t.Fatalf("releases = %d, want 1", len(lc.Releases))
}
if lc.Releases[0].Arch != "armv7l" {
t.Errorf("arch = %q, want armv7l (arm-7 → armv7 → armv7l)", lc.Releases[0].Arch)
}
})
}
// TestExportLegacyMixed verifies correct counting when multiple drop categories
// appear together in a single export call.
func TestExportLegacyMixed(t *testing.T) {
pd := storage.PackageData{
Assets: []storage.Asset{
// kept: baseline linux build
{Filename: "tool-linux-amd64.tar.gz", OS: "linux", Arch: "x86_64", Format: ".tar.gz"},
// dropped: variant build
{Filename: "tool-linux-amd64-rocm.tar.gz", OS: "linux", Arch: "x86_64", Format: ".tar.gz", Variants: []string{"rocm"}},
// dropped: android
{Filename: "tool-android-arm64.tar.gz", OS: "android", Arch: "aarch64", Format: ".tar.gz"},
// dropped: .AppImage format
{Filename: "tool.AppImage", OS: "linux", Arch: "x86_64", Format: ".AppImage"},
// kept (translated): universal2 → x86_64
{Filename: "tool-darwin-universal.tar.gz", OS: "darwin", Arch: "universal2", Format: ".tar.gz"},
// kept: solaris as-is
{Filename: "tool-solaris-amd64.tar.gz", OS: "solaris", Arch: "x86_64", Format: ".tar.gz"},
},
}
lc, stats := storage.ExportLegacy("tool", pd)
if stats.Variants != 1 {
t.Errorf("Variants = %d, want 1", stats.Variants)
}
if stats.Android != 1 {
t.Errorf("Android = %d, want 1", stats.Android)
}
if stats.Formats != 1 {
t.Errorf("Formats = %d, want 1", stats.Formats)
}
if len(lc.Releases) != 3 {
t.Errorf("releases = %d, want 3 (linux + macos/amd64 + solaris)", len(lc.Releases))
}
// Verify universal2 was translated to amd64 (via universal2→x86_64→amd64),
// and darwin was translated to macos.
var macosArch string
for _, r := range lc.Releases {
if r.OS == "macos" {
macosArch = r.Arch
}
}
if macosArch != "amd64" {
t.Errorf("macos arch = %q, want amd64 (universal2→x86_64→amd64, darwin→macos)", macosArch)
}
}

View File

@@ -0,0 +1,295 @@
// Package pgstore implements [storage.Store] on PostgreSQL.
//
// Schema uses double-buffering: two asset generations per package (0 and 1).
// The active generation pointer in webi_packages is updated atomically on
// Commit, so readers always see a complete consistent snapshot.
//
// Write path:
//
// BeginRefresh → clears inactive generation, returns tx
// Put → stages assets in-memory
// Commit → bulk-inserts assets (COPY), swaps generation pointer
//
// Read path:
//
// Load → reads active generation from webi_packages, fetches assets
//
// Connection string format: standard libpq / pgx DSN, e.g.:
//
// postgres://user:pass@host/dbname?sslmode=require
// host=localhost user=webi dbname=webi sslmode=disable
package pgstore
import (
"context"
"fmt"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/webinstall/webi-installers/internal/storage"
)
// Schema holds the DDL for creating the required tables.
// Run once on startup or deploy to ensure the schema exists.
const Schema = `
CREATE TABLE IF NOT EXISTS webi_packages (
name TEXT NOT NULL PRIMARY KEY,
active_gen SMALLINT NOT NULL DEFAULT 0,
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE IF NOT EXISTS webi_assets (
id BIGSERIAL PRIMARY KEY,
pkg TEXT NOT NULL,
gen SMALLINT NOT NULL,
filename TEXT NOT NULL DEFAULT '',
version TEXT NOT NULL DEFAULT '',
lts BOOLEAN NOT NULL DEFAULT FALSE,
channel TEXT NOT NULL DEFAULT '',
date TEXT NOT NULL DEFAULT '',
os TEXT NOT NULL DEFAULT '',
arch TEXT NOT NULL DEFAULT '',
libc TEXT NOT NULL DEFAULT '',
format TEXT NOT NULL DEFAULT '',
download TEXT NOT NULL DEFAULT '',
extra TEXT NOT NULL DEFAULT '',
variants TEXT[] NOT NULL DEFAULT '{}'
);
CREATE INDEX IF NOT EXISTS webi_assets_pkg_gen ON webi_assets (pkg, gen);
`
// Store is a PostgreSQL-backed asset store.
type Store struct {
pool *pgxpool.Pool
}
// New opens a connection pool to the given DSN and applies the schema.
// Returns an error if the connection or schema creation fails.
func New(ctx context.Context, dsn string) (*Store, error) {
cfg, err := pgxpool.ParseConfig(dsn)
if err != nil {
return nil, fmt.Errorf("pgstore: parse dsn: %w", err)
}
pool, err := pgxpool.NewWithConfig(ctx, cfg)
if err != nil {
return nil, fmt.Errorf("pgstore: connect: %w", err)
}
if err := applySchema(ctx, pool); err != nil {
pool.Close()
return nil, err
}
return &Store{pool: pool}, nil
}
// Close releases the connection pool.
func (s *Store) Close() {
s.pool.Close()
}
// ListPackages returns the names of all packages in the store.
func (s *Store) ListPackages(ctx context.Context) ([]string, error) {
rows, err := s.pool.Query(ctx,
`SELECT name FROM webi_packages ORDER BY name`,
)
if err != nil {
return nil, fmt.Errorf("pgstore: list packages: %w", err)
}
defer rows.Close()
var pkgs []string
for rows.Next() {
var name string
if err := rows.Scan(&name); err != nil {
return nil, fmt.Errorf("pgstore: scan package name: %w", err)
}
pkgs = append(pkgs, name)
}
return pkgs, rows.Err()
}
// Load returns all assets for a package using the active generation.
// Returns nil (not an error) if the package is not cached.
func (s *Store) Load(ctx context.Context, pkg string) (*storage.PackageData, error) {
// Fetch active generation and updated_at.
var gen int16
var updatedAt time.Time
err := s.pool.QueryRow(ctx,
`SELECT active_gen, updated_at FROM webi_packages WHERE name = $1`,
pkg,
).Scan(&gen, &updatedAt)
if err == pgx.ErrNoRows {
return nil, nil
}
if err != nil {
return nil, fmt.Errorf("pgstore: load %s: %w", pkg, err)
}
// Fetch all assets for this generation.
rows, err := s.pool.Query(ctx, `
SELECT filename, version, lts, channel, date,
os, arch, libc, format, download, extra, variants
FROM webi_assets
WHERE pkg = $1 AND gen = $2
ORDER BY id
`, pkg, gen)
if err != nil {
return nil, fmt.Errorf("pgstore: load assets %s: %w", pkg, err)
}
defer rows.Close()
var assets []storage.Asset
for rows.Next() {
var a storage.Asset
if err := rows.Scan(
&a.Filename, &a.Version, &a.LTS, &a.Channel, &a.Date,
&a.OS, &a.Arch, &a.Libc, &a.Format, &a.Download,
&a.Extra, &a.Variants,
); err != nil {
return nil, fmt.Errorf("pgstore: scan asset %s: %w", pkg, err)
}
assets = append(assets, a)
}
if err := rows.Err(); err != nil {
return nil, fmt.Errorf("pgstore: rows %s: %w", pkg, err)
}
return &storage.PackageData{
Assets: assets,
UpdatedAt: updatedAt,
}, nil
}
// BeginRefresh starts a write transaction for a package.
// It determines the inactive generation and clears it, ready for new data.
func (s *Store) BeginRefresh(ctx context.Context, pkg string) (storage.RefreshTx, error) {
// Determine which generation to write into (the inactive one).
var activeGen int16
err := s.pool.QueryRow(ctx,
`SELECT active_gen FROM webi_packages WHERE name = $1`,
pkg,
).Scan(&activeGen)
if err != nil && err != pgx.ErrNoRows {
return nil, fmt.Errorf("pgstore: begin refresh %s: %w", pkg, err)
}
// If package doesn't exist yet, activeGen defaults to 0 and we write to gen 1.
// If package exists, we write to the inactive generation (1 - activeGen).
var writeGen int16
if err == pgx.ErrNoRows {
writeGen = 1
} else {
writeGen = 1 - activeGen
}
// Clear the write generation so we start fresh.
if _, err := s.pool.Exec(ctx,
`DELETE FROM webi_assets WHERE pkg = $1 AND gen = $2`,
pkg, writeGen,
); err != nil {
return nil, fmt.Errorf("pgstore: clear gen %d for %s: %w", writeGen, pkg, err)
}
return &refreshTx{
pool: s.pool,
pkg: pkg,
gen: writeGen,
}, nil
}
// refreshTx is an in-progress write for one package.
type refreshTx struct {
pool *pgxpool.Pool
pkg string
gen int16
assets []storage.Asset
}
// Put stages assets for writing. May be called multiple times.
func (tx *refreshTx) Put(assets []storage.Asset) error {
tx.assets = append(tx.assets, assets...)
return nil
}
// Commit bulk-inserts all staged assets, then atomically swaps the
// active generation pointer in webi_packages.
func (tx *refreshTx) Commit(ctx context.Context) error {
if len(tx.assets) == 0 {
return tx.swapGeneration(ctx)
}
// Build rows for pgx.CopyFromRows.
rows := make([][]any, len(tx.assets))
for i, a := range tx.assets {
variants := a.Variants
if variants == nil {
variants = []string{}
}
rows[i] = []any{
tx.pkg,
tx.gen,
a.Filename,
a.Version,
a.LTS,
a.Channel,
a.Date,
a.OS,
a.Arch,
a.Libc,
a.Format,
a.Download,
a.Extra,
variants,
}
}
cols := []string{
"pkg", "gen",
"filename", "version", "lts", "channel", "date",
"os", "arch", "libc", "format", "download", "extra", "variants",
}
_, err := tx.pool.CopyFrom(ctx,
pgx.Identifier{"webi_assets"},
cols,
pgx.CopyFromRows(rows),
)
if err != nil {
return fmt.Errorf("pgstore: copy assets %s: %w", tx.pkg, err)
}
return tx.swapGeneration(ctx)
}
// swapGeneration atomically updates the active generation pointer.
func (tx *refreshTx) swapGeneration(ctx context.Context) error {
_, err := tx.pool.Exec(ctx, `
INSERT INTO webi_packages (name, active_gen, updated_at)
VALUES ($1, $2, now())
ON CONFLICT (name)
DO UPDATE SET active_gen = $2, updated_at = now()
`, tx.pkg, tx.gen)
if err != nil {
return fmt.Errorf("pgstore: swap gen %s: %w", tx.pkg, err)
}
tx.assets = nil
return nil
}
// Rollback discards all staged assets without writing anything.
func (tx *refreshTx) Rollback() error {
tx.assets = nil
return nil
}
// applySchema runs the schema DDL idempotently.
func applySchema(ctx context.Context, pool *pgxpool.Pool) error {
if _, err := pool.Exec(ctx, Schema); err != nil {
return fmt.Errorf("pgstore: apply schema: %w", err)
}
return nil
}

View File

@@ -0,0 +1,71 @@
// Package storage defines the interface for reading and writing
// classified release assets.
//
// webid reads assets through [Store]. webicached writes them through
// [RefreshTx], obtained from [Store.BeginRefresh].
//
// The two implementations are fsstore (filesystem JSON, compatible with
// the Node.js _cache/ format) and pgstore (PostgreSQL, future).
package storage
import (
"context"
"time"
)
// Asset is a single downloadable file — one entry in a release.
// A release like "bat v0.26.1" has many assets (one per platform/format).
//
// No JSON tags — serialization goes through [LegacyAsset] for Node.js
// compat, or through a future v2 format.
type Asset struct {
Filename string
Version string
LTS bool
Channel string
Date string
OS string
Arch string
Libc string
Format string
Download string
Extra string // extra version info for sorting (e.g. build metadata)
GitTag string // original git tag (e.g. "v1.2", "master") — only for format="git"
GitCommitHash string // short commit hash (e.g. "54c216e") — only for format="git"
Variants []string // build qualifiers: "installer", "rocm", "jetpack5", "fxdependent", etc.
}
// PackageData is the full set of assets for a package, plus metadata.
type PackageData struct {
Assets []Asset
UpdatedAt time.Time
}
// Store is the read/write interface for release asset storage.
type Store interface {
// ListPackages returns the names of all packages in the store.
ListPackages(ctx context.Context) ([]string, error)
// Load returns all assets for a package, or nil if the package
// is not cached. The returned data may be stale — check UpdatedAt.
Load(ctx context.Context, pkg string) (*PackageData, error)
// BeginRefresh starts a write transaction for a package.
// Write assets via [RefreshTx.Put], then call Commit to atomically
// replace the stored data. Call Rollback to discard.
BeginRefresh(ctx context.Context, pkg string) (RefreshTx, error)
}
// RefreshTx is a write transaction for replacing a package's assets.
type RefreshTx interface {
// Put stages assets to be written. May be called multiple times
// to append assets incrementally.
Put(assets []Asset) error
// Commit atomically replaces the package's stored assets with
// everything staged via Put.
Commit(ctx context.Context) error
// Rollback discards all staged data.
Rollback() error
}

View File

@@ -0,0 +1,247 @@
// Package uadetect identifies the requesting agent's OS, CPU architecture,
// and libc so the server can select the correct release artifact.
//
// An agent identifies itself through multiple signals:
// - The User-Agent header: Webi's bootstrap scripts send "$(uname -srm)",
// e.g. "Darwin 23.1.0 arm64". Browsers, curl, and PowerShell send their
// own UA strings.
// - Query parameters: ?os=linux&arch=arm64 are an explicit declaration
// that takes precedence over the header.
//
// Use [FromRequest] to detect from an HTTP request (preferred).
// Use [Parse] to detect from a raw UA string.
package uadetect
import (
"net/http"
"strings"
"github.com/webinstall/webi-installers/internal/buildmeta"
)
// Result holds the detected platform info from a User-Agent string.
type Result struct {
OS buildmeta.OS
Arch buildmeta.Arch
Libc buildmeta.Libc
}
// FromRequest detects the agent's platform from an HTTP request.
// Query parameters ?os and ?arch override the User-Agent header.
func FromRequest(r *http.Request) Result {
qOS := r.URL.Query().Get("os")
qArch := r.URL.Query().Get("arch")
var ua string
switch {
case qOS != "" && qArch != "":
ua = qOS + " " + qArch
case qOS != "":
ua = qOS
case qArch != "":
ua = qArch
default:
ua = r.Header.Get("User-Agent")
}
return Parse(ua)
}
// Parse extracts OS, arch, and libc from a User-Agent string.
func Parse(ua string) Result {
if ua == "-" {
return Result{}
}
tokens := tokenize(ua)
return Result{
OS: matchOS(tokens),
Arch: matchArch(tokens),
Libc: matchLibc(tokens),
}
}
// tokenize splits a User-Agent into lowercase tokens for matching.
// Splits on whitespace, '/', and ';', since UAs come in various forms:
//
// "Darwin 23.1.0 arm64" (uname -srm)
// "PowerShell/7.3.0" (PowerShell)
// "MS AMD64" (Windows shorthand)
// "Macintosh; Intel Mac OS X 10_15_7" (browser)
func tokenize(ua string) []string {
// Strip xnu kernel info that can mislead arch detection under Rosetta.
// "xnu-7195.60.75~1/RELEASE_ARM64_T8101" contains ARM64 even when
// running as x86_64. This only appears in verbose uname output.
if i := strings.Index(ua, "xnu-"); i >= 0 {
end := strings.IndexByte(ua[i:], ' ')
if end < 0 {
ua = ua[:i]
} else {
ua = ua[:i] + ua[i+end:]
}
}
return strings.FieldsFunc(strings.ToLower(ua), func(r rune) bool {
return r == ' ' || r == '/' || r == ';' || r == '\t'
})
}
// matchOS identifies the operating system from tokens.
// Order matters: Android before Linux, Linux before Windows (for WSL).
func matchOS(tokens []string) buildmeta.OS {
has := func(s string) bool {
for _, t := range tokens {
if strings.Contains(t, s) {
return true
}
}
return false
}
// Android must be checked before Linux.
if has("android") {
return buildmeta.OSAndroid
}
if has("darwin") || has("macos") || has("macintosh") || has("iphone") || has("ios") || has("ipad") {
return buildmeta.OSDarwin
}
// "mac" alone (not in "macintosh" which is already matched)
for _, t := range tokens {
if t == "mac" {
return buildmeta.OSDarwin
}
}
// FreeBSD before Linux (both are POSIX, but FreeBSD never reports "linux").
if has("freebsd") {
return buildmeta.OSFreeBSD
}
// Linux before Windows because WSL UAs contain both "linux" and "microsoft".
// But exclude Cygwin/Msys/MINGW which report Linux-like strings on Windows.
if has("linux") && !has("cygwin") && !has("msysgit") && !has("msys") && !has("mingw") {
return buildmeta.OSLinux
}
// Cygwin, Msys, and MINGW are Windows environments.
if has("windows") || has("win32") || has("microsoft") || has("powershell") ||
has("cygwin") || has("msys") || has("mingw") {
return buildmeta.OSWindows
}
for _, t := range tokens {
if t == "ms" || t == "win" {
return buildmeta.OSWindows
}
}
// Fallback: curl and wget imply a POSIX system, almost always Linux.
if has("curl") || has("wget") {
return buildmeta.OSLinux
}
return ""
}
// matchArch identifies the CPU architecture from tokens.
// More specific patterns are checked before less specific ones.
func matchArch(tokens []string) buildmeta.Arch {
has := func(s string) bool {
for _, t := range tokens {
if strings.Contains(t, s) {
return true
}
}
return false
}
exact := func(s string) bool {
for _, t := range tokens {
if t == s {
return true
}
}
return false
}
// ARM 64-bit (most specific first)
if has("aarch64") || has("arm64") || has("armv8") {
return buildmeta.ArchARM64
}
// ARM 32-bit variants
if has("armv7") || has("arm32") {
return buildmeta.ArchARMv7
}
if has("armv6") {
return buildmeta.ArchARMv6
}
// Bare "arm" without a version qualifier → armv6 (conservative).
if exact("arm") {
return buildmeta.ArchARMv6
}
// POWER (check before generic 64-bit)
if has("ppc64le") {
return buildmeta.ArchPPC64LE
}
if has("ppc64") {
return buildmeta.ArchPPC64
}
// s390x (IBM Z)
if has("s390x") {
return buildmeta.ArchS390X
}
// RISC-V
if has("riscv64") {
return buildmeta.ArchRISCV64
}
// MIPS (check before generic 64-bit)
if has("mips64") {
return buildmeta.ArchMIPS64
}
if has("mips") {
return buildmeta.ArchMIPS
}
// x86-64
if has("x86_64") || has("amd64") || exact("x64") {
return buildmeta.ArchAMD64
}
// x86 32-bit (after x86_64 to avoid false match)
if has("i386") || has("i686") || exact("x86") {
return buildmeta.ArchX86
}
return ""
}
// matchLibc identifies the C library from tokens.
func matchLibc(tokens []string) buildmeta.Libc {
has := func(s string) bool {
for _, t := range tokens {
if strings.Contains(t, s) {
return true
}
}
return false
}
if has("musl") {
return buildmeta.LibcMusl
}
// Don't match "microsoft" — it appears in WSL kernel version strings
// (e.g. "5.15.146.1-microsoft-standard-WSL2") and doesn't indicate MSVC.
if has("msvc") || has("windows") {
return buildmeta.LibcMSVC
}
if has("gnu") || has("glibc") || has("linux") {
return buildmeta.LibcGNU
}
return buildmeta.LibcNone
}

View File

@@ -0,0 +1,190 @@
package uadetect_test
import (
"net/http"
"testing"
"github.com/webinstall/webi-installers/internal/buildmeta"
"github.com/webinstall/webi-installers/internal/uadetect"
)
func TestOS(t *testing.T) {
tests := []struct {
ua string
want buildmeta.OS
}{
// uname -srm style
{"Darwin 23.1.0 arm64", buildmeta.OSDarwin},
{"Darwin 20.2.0 x86_64", buildmeta.OSDarwin},
{"Linux 6.1.0-18-amd64 x86_64", buildmeta.OSLinux},
{"Linux 5.15.0 aarch64", buildmeta.OSLinux},
// WSL: Linux, not Windows (contains "microsoft" in kernel release)
{"Linux 5.15.146.1-microsoft-standard-WSL2 x86_64", buildmeta.OSLinux},
// Windows
{"MS AMD64", buildmeta.OSWindows},
{"PowerShell/7.3.0", buildmeta.OSWindows},
{"Microsoft Windows 10.0.19045", buildmeta.OSWindows},
// Msys/MINGW/Cygwin → Windows
{"webi/curl x86_64/unknown Msys/MINGW64_NT-10.0-19045/3.5.7-463ebcdc.x86_64 libc", buildmeta.OSWindows},
{"webi/curl+wget x86_64/unknown Msys/MSYS_NT-10.0-26200/3.6.6-1cdd4371.x86_64 libc", buildmeta.OSWindows},
{"webi/curl x86_64/unknown Cygwin/CYGWIN_NT-10.0/2.10.0(0.325/5/3) libc", buildmeta.OSWindows},
// FreeBSD
{"webi/curl amd64/unknown FreeBSD/14.3-RELEASE-p8 libc", buildmeta.OSFreeBSD},
// Android before Linux
{"Android 13 aarch64", buildmeta.OSAndroid},
{"webi/curl aarch64/unknown Android/Linux/6.6.77-android15-8 libc", buildmeta.OSAndroid},
// WSL: Linux, not Windows (kernel contains "microsoft")
{"webi/curl+wget x86_64/unknown GNU/Linux/5.15.146.1-microsoft-standard-WSL2 libc", buildmeta.OSLinux},
// Browser-style
{"Macintosh; Intel Mac OS X 10_15_7", buildmeta.OSDarwin},
// Minimal agents → assume Linux
{"curl/8.1.2", buildmeta.OSLinux},
{"wget/1.21", buildmeta.OSLinux},
// Explicit unknown
{"-", ""},
}
for _, tt := range tests {
t.Run(tt.ua, func(t *testing.T) {
got := uadetect.Parse(tt.ua).OS
if got != tt.want {
t.Errorf("Parse(%q).OS = %q, want %q", tt.ua, got, tt.want)
}
})
}
}
func TestArch(t *testing.T) {
tests := []struct {
ua string
want buildmeta.Arch
}{
{"Darwin 23.1.0 arm64", buildmeta.ArchARM64},
{"Linux 6.1.0 aarch64", buildmeta.ArchARM64},
{"Linux 5.4.0 x86_64", buildmeta.ArchAMD64},
{"MS AMD64", buildmeta.ArchAMD64},
{"Linux 5.10.0 armv7l", buildmeta.ArchARMv7},
{"Linux 5.10.0 armv6l", buildmeta.ArchARMv6},
{"Linux 5.4.0 ppc64le", buildmeta.ArchPPC64LE},
{"webi/curl+wget s390x/unknown GNU/Linux/6.4.0-150700.53.6-default libc", buildmeta.ArchS390X},
// FreeBSD uses "amd64" not "x86_64"
{"webi/curl amd64/unknown FreeBSD/14.3-RELEASE-p8 libc", buildmeta.ArchAMD64},
// Rosetta: xnu kernel info says ARM64 but actual arch is x86_64
{"Darwin 20.2.0 Darwin Kernel Version 20.2.0; root:xnu-7195.60.75~1/RELEASE_ARM64_T8101 x86_64", buildmeta.ArchAMD64},
{"-", ""},
}
for _, tt := range tests {
t.Run(tt.ua, func(t *testing.T) {
got := uadetect.Parse(tt.ua).Arch
if got != tt.want {
t.Errorf("Parse(%q).Arch = %q, want %q", tt.ua, got, tt.want)
}
})
}
}
func TestLibc(t *testing.T) {
tests := []struct {
ua string
want buildmeta.Libc
}{
{"Linux 6.1.0 x86_64 musl", buildmeta.LibcMusl},
{"Linux 6.1.0 x86_64 gnu", buildmeta.LibcGNU},
{"Linux 6.1.0 x86_64 linux", buildmeta.LibcGNU},
{"MS AMD64 msvc", buildmeta.LibcMSVC},
{"Microsoft Windows", buildmeta.LibcMSVC},
{"Darwin 23.1.0 arm64", buildmeta.LibcNone},
// WSL: kernel version contains "microsoft" but libc is gnu, not msvc
{"webi/curl+wget x86_64/unknown GNU/Linux/5.15.146.1-microsoft-standard-WSL2 libc", buildmeta.LibcGNU},
{"-", ""},
}
for _, tt := range tests {
t.Run(tt.ua, func(t *testing.T) {
got := uadetect.Parse(tt.ua).Libc
if got != tt.want {
t.Errorf("Parse(%q).Libc = %q, want %q", tt.ua, got, tt.want)
}
})
}
}
func TestFromRequest(t *testing.T) {
tests := []struct {
name string
ua string // User-Agent header
query string // raw query string
wantOS buildmeta.OS
wantAr buildmeta.Arch
}{
{
name: "UA header only",
ua: "Darwin 23.1.0 arm64",
wantOS: buildmeta.OSDarwin,
wantAr: buildmeta.ArchARM64,
},
{
name: "query params override UA",
ua: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)",
query: "os=linux&arch=aarch64",
wantOS: buildmeta.OSLinux,
wantAr: buildmeta.ArchARM64,
},
{
name: "os param only",
ua: "curl/8.1.2",
query: "os=windows",
wantOS: buildmeta.OSWindows,
},
{
name: "arch param only",
ua: "curl/8.1.2",
query: "arch=arm64",
wantAr: buildmeta.ArchARM64,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
req, _ := http.NewRequest("GET", "http://example.com/api?"+tt.query, nil)
if tt.ua != "" {
req.Header.Set("User-Agent", tt.ua)
}
got := uadetect.FromRequest(req)
if tt.wantOS != "" && got.OS != tt.wantOS {
t.Errorf("OS = %q, want %q", got.OS, tt.wantOS)
}
if tt.wantAr != "" && got.Arch != tt.wantAr {
t.Errorf("Arch = %q, want %q", got.Arch, tt.wantAr)
}
})
}
}
func TestFullParse(t *testing.T) {
r := uadetect.Parse("Darwin 23.1.0 arm64")
if r.OS != buildmeta.OSDarwin {
t.Errorf("OS = %q, want %q", r.OS, buildmeta.OSDarwin)
}
if r.Arch != buildmeta.ArchARM64 {
t.Errorf("Arch = %q, want %q", r.Arch, buildmeta.ArchARM64)
}
if r.Libc != buildmeta.LibcNone {
t.Errorf("Libc = %q, want %q", r.Libc, buildmeta.LibcNone)
}
}

63
scripts/deploy-webicached.sh Executable file
View File

@@ -0,0 +1,63 @@
#!/bin/sh
set -e
set -u
# Build and deploy webicached to a target host
g_host="${1:-beta.webi.sh}"
g_bin="webicached"
g_out="agents/tmp/${g_bin}"
g_remote_bin="~/bin/${g_bin}"
case "${g_host}" in
beta.webi.sh) g_remote_conf="~/srv/beta.webinstall.dev/installers/" ;;
next.webi.sh) g_remote_conf="~/srv/next.webinstall.dev/installers/" ;;
*) g_remote_conf="~/srv/webid/installers/" ;;
esac
fn_build() {
b_version="$(git describe --tags --always 2> /dev/null || echo '0.0.0-dev')"
b_commit="$(git rev-parse --short HEAD)"
b_date="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
b_ldflags="-X main.version=${b_version} -X main.commit=${b_commit} -X main.date=${b_date}"
printf 'Building %s %s %s (%s)...\n' "${g_bin}" "${b_version}" "${b_commit}" "${b_date}"
GOOS=linux GOARCH=amd64 GOAMD64=v2 go build -ldflags "${b_ldflags}" -o "${g_out}" ./cmd/webicached
printf 'Built: %s\n' "${g_out}"
}
fn_deploy() {
printf 'Stopping %s on %s...\n' "${g_bin}" "${g_host}"
ssh "${g_host}" "~/.local/bin/serviceman stop ${g_bin}" 2> /dev/null || true
printf 'Uploading binary...\n'
scp "${g_out}" "${g_host}:${g_remote_bin}"
printf 'Syncing releases.conf files...\n'
rsync -av \
--exclude='_cache' --exclude='.git' --exclude='agents' \
--exclude='bin' --exclude='cmd' --exclude='internal' \
--exclude='docs' --exclude='scripts' --exclude='node_modules' \
--include='*/' --include='releases.conf' --exclude='*' \
./ "${g_host}:${g_remote_conf}"
printf 'Starting %s...\n' "${g_bin}"
ssh "${g_host}" "~/.local/bin/serviceman start ${g_bin}"
}
fn_verify() {
printf 'Waiting 5s for startup...\n'
sleep 5
printf 'Checking version...\n'
ssh "${g_host}" "${g_remote_bin} -V"
printf 'Checking logs...\n'
ssh "${g_host}" "sudo journalctl -u ${g_bin} --no-pager -n 5"
}
fn_build
fn_deploy
fn_verify
printf '\nDone. %s deployed to %s.\n' "${g_bin}" "${g_host}"

View File

@@ -0,0 +1,111 @@
---
name: deploy-webicached
description: Deploy webicached binary to beta.webi.sh. Use when building, uploading, or restarting the cache daemon. Covers cross-compile, conf sync, service management.
---
## One-step deploy
```sh
./scripts/deploy-webicached.sh beta.webi.sh
```
Builds with version ldflags, stops service, uploads, syncs conf, starts, verifies.
## Manual steps (if needed)
### Build
```sh
VERSION="$(git describe --tags --always)"
COMMIT="$(git rev-parse --short HEAD)"
DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
GOOS=linux GOARCH=amd64 GOAMD64=v2 go build \
-ldflags "-X main.version=${VERSION} -X main.commit=${COMMIT} -X main.date=${DATE}" \
-o agents/tmp/webicached ./cmd/webicached
```
MUST: Build from the `ref-webi-go` worktree (or branch containing `cmd/webicached`).
### Deploy
```sh
ssh beta.webi.sh "serviceman stop webicached"
scp agents/tmp/webicached beta.webi.sh:~/bin/webicached
```
MUST: Stop service before scp — Linux refuses to overwrite a running binary.
### Sync releases.conf
```sh
rsync -av --include='*/' --include='releases.conf' --exclude='*' \
./ beta.webi.sh:~/srv/beta.webinstall.dev/installers/
```
MUST: Run from the worktree root. The server has no checkout of this branch — conf files must be synced explicitly.
### Start
```sh
ssh beta.webi.sh "serviceman start webicached"
```
### Verify
```sh
ssh beta.webi.sh "sleep 5 && serviceman logs webicached"
```
Expected: "batch: N stale, refreshing 20" or "all packages fresh, sleeping 9s"
## Smoke test
```sh
ssh beta.webi.sh "curl -sSf http://localhost:3080/api/releases/bat.json | head -c 100"
ssh beta.webi.sh "curl -sSf -A 'curl/7.81.0 Linux x86_64' http://localhost:3080/api/installers/bat.sh | head -3"
```
Expected: JSON array with release objects; shell script with `PKG_NAME='bat'`.
## Service management
```sh
serviceman status webicached
serviceman restart webicached
serviceman logs webicached
```
## Server layout
| Path | Purpose |
|------|---------|
| `~/bin/webicached` | Binary |
| `~/srv/beta.webinstall.dev/installers/` | Conf dir (releases.conf files) |
| `~/.cache/webi/legacy/` | Cache output (fsstore, legacy JSON format) |
| `~/.cache/webi/raw/` | Raw upstream API responses |
| `~/srv/beta.webinstall.dev/.env.secret` | GITHUB_TOKEN |
| `/etc/systemd/system/webicached.service` | Service unit (created by serviceman) |
## Flags reference
| Flag | Default | Purpose |
|------|---------|---------|
| `--conf` | `.` | Dir with `{pkg}/releases.conf` files |
| `--legacy` | `~/.cache/webi/legacy` | Legacy cache output directory |
| `--raw` | `~/.cache/webi/raw` | Raw upstream response cache |
| `--token` | `$GITHUB_TOKEN` | GitHub API token |
| `--interval` | `9s` | Delay between package fetches in a batch |
| `--once` | false | Run once then exit |
| `--eager` | false | Fetch all on startup (not staleness-based) |
| `--shallow` | false | Only first page of releases |
| `--no-fetch` | false | Classify from rawcache only |
| `--page-delay` | `2s` | Delay between paginated API pages |
## One-shot refresh (specific packages)
```sh
ssh beta.webi.sh ". ~/srv/beta.webinstall.dev/.env.secret && ~/bin/webicached \
--conf ~/srv/beta.webinstall.dev/installers/ \
--raw ~/.cache/webi/raw \
--once bat goreleaser"
```