mirror of
https://github.com/webinstall/webi-installers.git
synced 2026-06-04 23:12:47 +00:00
Compare commits
4 Commits
feat/webi-
...
feat/webi-
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f5a8a0cd5a | ||
|
|
1bc9e40bf6 | ||
|
|
7d9fc3387c | ||
|
|
3cfd10f197 |
16
.gitignore
vendored
16
.gitignore
vendored
@@ -3,6 +3,16 @@ install-*.sh
|
||||
install-*.bat
|
||||
install-*.ps1
|
||||
|
||||
# Go build outputs (from go run/build in repo root)
|
||||
/classify
|
||||
/e2etest
|
||||
/fetchraw
|
||||
/inspect
|
||||
/uaparse
|
||||
/webicached
|
||||
/zigtest
|
||||
/distributables.csv
|
||||
|
||||
# local config
|
||||
.env.*
|
||||
*.env
|
||||
@@ -18,7 +28,13 @@ node_modules/
|
||||
*.bak
|
||||
*.bak.*
|
||||
|
||||
# agent session files
|
||||
agents/
|
||||
|
||||
# other
|
||||
.DS_Store
|
||||
desktop.ini
|
||||
.directory
|
||||
LIVE_cache
|
||||
/webid
|
||||
bin/
|
||||
|
||||
120
AGENTS.md
120
AGENTS.md
@@ -370,3 +370,123 @@ Commit messages: `feat(<pkg>): add installer`, `fix(<pkg>): update install.sh`,
|
||||
`arch`, or `ext` explicitly in `releases.js`.
|
||||
- **Goreleaser archives**: Typically contain a bare binary at the archive root
|
||||
(not nested in a directory). Use `mv ./cmd "$pkg_src_cmd"`.
|
||||
|
||||
---
|
||||
|
||||
## Go Cache Daemon
|
||||
|
||||
The Go pipeline (`cmd/webicached`) replaces the Node.js release-fetching code.
|
||||
It reads `releases.conf` files, fetches upstream release metadata, classifies
|
||||
build assets, and writes to `~/.cache/webi/legacy/` in the format the Node.js server expects.
|
||||
|
||||
### Canonical Vocabulary
|
||||
|
||||
The classifier MUST use exactly these strings. They match the production API.
|
||||
|
||||
**OS**: `macos` (NOT `darwin`), `linux`, `windows`, `freebsd`, `openbsd`,
|
||||
`netbsd`, `dragonfly`, `aix`, `illumos`, `plan9`, `solaris`, `posix_2017`
|
||||
|
||||
**Arch** — exact equivalences:
|
||||
- `amd64` (NOT `x86_64`), `x86` (NOT `i386`/`i686`/`386`)
|
||||
- `arm64` (NOT `aarch64`)
|
||||
- `armv7l` (NOT `armv7`), `armv6l` (NOT `armv6`)
|
||||
- `mipsle` (NOT `mipsel`), `mips64le` (NOT `mips64el`)
|
||||
|
||||
**Arch** — compatibility downcasts:
|
||||
- `armhf` → `armv7l`, `armv7a` → `armv7l`, `armel` → `arm`
|
||||
|
||||
**Arch** — other: `arm`, `ppc64le`, `ppc64`, `loong64`, `riscv64`, `s390x`,
|
||||
`mips`, `mips64`
|
||||
|
||||
**Libc**: `none` (never empty), `gnu`, `musl`, `msvc`
|
||||
|
||||
**Ext**: `tar.gz`, `tar.xz`, `zip`, `exe`, `7z`, `pkg`, `msi`
|
||||
(no leading dot; `exe` for bare binaries)
|
||||
|
||||
### releases.conf
|
||||
|
||||
Each package directory contains a `releases.conf` that tells the daemon where
|
||||
to fetch releases. Format is `key = value`, one per line. `#` comments and
|
||||
blank lines are ignored.
|
||||
|
||||
#### Source types (mutually exclusive — pick one)
|
||||
|
||||
```ini
|
||||
# GitHub binary releases (most common)
|
||||
github_releases = sharkdp/bat
|
||||
|
||||
# GitHub source tarballs (with optional git fallback)
|
||||
github_sources = bnnanet/serviceman
|
||||
git_url = https://github.com/bnnanet/serviceman.git
|
||||
|
||||
# Git tag enumeration (vim plugins, shell scripts — git_url alone)
|
||||
git_url = https://github.com/tpope/vim-commentary.git
|
||||
|
||||
# Gitea (full URL required, or short form + base_url)
|
||||
gitea_releases = https://git.rootprojects.org/root/pathman
|
||||
|
||||
# GitLab (defaults to gitlab.com)
|
||||
gitlab_releases = owner/repo
|
||||
|
||||
# HashiCorp releases API
|
||||
hashicorp_product = terraform
|
||||
|
||||
# Custom source (servicemandist, nodedist, zigdist, etc.)
|
||||
source = nodedist
|
||||
url = https://nodejs.org/download/release
|
||||
```
|
||||
|
||||
#### Filtering, versioning, and platform
|
||||
|
||||
```ini
|
||||
tag_prefix = bun- # monorepo: strip prefix from version
|
||||
version_prefixes = jq- # strip from version string (space-separated)
|
||||
asset_filter = MinGit # filename must contain this substring
|
||||
exclude = busybox -src- -docs- # skip assets containing these (space-separated)
|
||||
os = posix_2017 # restrict ALL versions to this OS (blanket)
|
||||
alias_of = rg # mirrors another package's releases
|
||||
```
|
||||
|
||||
#### Design rules
|
||||
|
||||
- `os` is a blanket tag on ALL versions. Only use for packages that are always
|
||||
POSIX-only. For version-dependent OS tagging, use a custom `TagVariants` in
|
||||
`internal/releases/{pkg}/variants.go`.
|
||||
- `git_url` can be primary (gittag source when it's the only key) or secondary
|
||||
fallback alongside `github_sources`/`gitea_sources`.
|
||||
- Full URL forms accepted for github/gitea/gitlab (e.g.
|
||||
`github_releases = https://github.com/sharkdp/bat`).
|
||||
|
||||
### Testing
|
||||
|
||||
Test tools: `cmd/e2etest` (pipeline comparison), `cmd/comparecache` (cache diff),
|
||||
`cmd/inspect` (single-package debug). Run each with `--help` for usage.
|
||||
|
||||
### Classifier vs Per-Package Tagger
|
||||
|
||||
The general classifier (`internal/classify/`) handles patterns common across
|
||||
many projects. It MUST NOT contain one-off logic for a single package.
|
||||
|
||||
Per-package taggers (`internal/releases/{pkg}/variants.go`) handle
|
||||
project-specific knowledge. Read the existing taggers for conventions.
|
||||
|
||||
MUST: Derive arch/OS from concrete evidence — not blanket defaults.
|
||||
MUST: New general classifier patterns must apply to 2-3+ packages.
|
||||
|
||||
### Deploying
|
||||
|
||||
```sh
|
||||
./scripts/deploy-webicached.sh beta.webi.sh
|
||||
./scripts/deploy-webicached.sh next.webi.sh
|
||||
```
|
||||
|
||||
First-time setup on a new host uses `serviceman`:
|
||||
|
||||
```sh
|
||||
serviceman add --name webicached \
|
||||
--workdir ~/srv/webid/installers/ -- \
|
||||
~/bin/webicached \
|
||||
--envfile ~/srv/webid/.env.secret \
|
||||
--conf ~/srv/webid/installers/ \
|
||||
--raw ~/.cache/webi/raw
|
||||
```
|
||||
|
||||
1293
cmd/classify/main.go
Normal file
1293
cmd/classify/main.go
Normal file
File diff suppressed because it is too large
Load Diff
914
cmd/comparecache/main.go
Normal file
914
cmd/comparecache/main.go
Normal file
@@ -0,0 +1,914 @@
|
||||
// Command comparecache compares Go-generated cache output against the
|
||||
// Node.js LIVE_cache. It identifies categorical differences in asset
|
||||
// selection — which filenames appear in one cache but not the other.
|
||||
//
|
||||
// The comparison is done at the filename level (not OS/arch/ext fields)
|
||||
// because the Node.js cache leaves those empty (normalize.js fills them
|
||||
// at serve time), while the Go pipeline classifies at write time.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// go run ./cmd/comparecache -live ./LIVE_cache -go ./_cache
|
||||
// go run ./cmd/comparecache -live ./LIVE_cache -go ./_cache bat jq
|
||||
// go run ./cmd/comparecache -live ./LIVE_cache -go ./_cache -summary
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log"
|
||||
"math/rand/v2"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/classify"
|
||||
"github.com/webinstall/webi-installers/internal/lexver"
|
||||
)
|
||||
|
||||
type cacheEntry struct {
|
||||
Releases []cacheRelease `json:"releases"`
|
||||
}
|
||||
|
||||
type cacheRelease struct {
|
||||
Name string `json:"name"`
|
||||
Filename string `json:"_filename"` // Node.js uses _filename for some sources
|
||||
Version string `json:"version"`
|
||||
Download string `json:"download"`
|
||||
Channel string `json:"channel"`
|
||||
OS string `json:"os"`
|
||||
Arch string `json:"arch"`
|
||||
Libc string `json:"libc"`
|
||||
Ext string `json:"ext"`
|
||||
}
|
||||
|
||||
// fieldDiff records a field-level difference for an asset that exists
|
||||
// in both caches (same filename) but has different classification.
|
||||
type fieldDiff struct {
|
||||
Filename string
|
||||
Field string // "os", "arch", "libc", "ext", "channel"
|
||||
Live string
|
||||
Go string
|
||||
BothSet bool // true when both live and go have non-empty values
|
||||
}
|
||||
|
||||
type packageDiff struct {
|
||||
Name string
|
||||
LiveCount int
|
||||
GoCount int
|
||||
OnlyInLive []string // filenames only in Node.js cache
|
||||
OnlyInGo []string // filenames only in Go cache
|
||||
FieldDiffs []fieldDiff // classification differences on shared assets
|
||||
VersionsLive []string // unique versions in live
|
||||
VersionsGo []string // unique versions in go
|
||||
GoMissing bool // true if Go didn't produce output for this package
|
||||
LiveMissing bool // true if no live cache for this package
|
||||
Categories []string // categorical difference labels
|
||||
}
|
||||
|
||||
func main() {
|
||||
liveDir := flag.String("live", "./LIVE_cache", "path to Node.js LIVE_cache directory")
|
||||
goDir := flag.String("go", "./_cache", "path to Go cache directory")
|
||||
summary := flag.Bool("summary", false, "only print summary, not per-package details")
|
||||
diffsOnly := flag.Bool("diffs", false, "only show packages with asset differences (skip matches)")
|
||||
latest := flag.Bool("latest", false, "only compare latest version in each cache")
|
||||
windowed := flag.Bool("windowed", false, "limit Go versions to the Node.js version range (2nd to 2nd-to-last)")
|
||||
sample := flag.Int("sample", 0, "for each package diff, show N randomly sampled assets (implies -windowed -diffs)")
|
||||
flag.Parse()
|
||||
filterPkgs := flag.Args()
|
||||
|
||||
// -sample implies -windowed and -diffs so we focus on real classification
|
||||
// differences, not version-depth noise.
|
||||
if *sample > 0 {
|
||||
*windowed = true
|
||||
*diffsOnly = true
|
||||
}
|
||||
|
||||
totalStart := time.Now()
|
||||
|
||||
// Find the most recent month directory in each cache.
|
||||
liveMonth := findLatestMonth(*liveDir)
|
||||
goMonth := findLatestMonth(*goDir)
|
||||
if liveMonth == "" {
|
||||
log.Fatalf("no month directories found in %s", *liveDir)
|
||||
}
|
||||
|
||||
livePath := filepath.Join(*liveDir, liveMonth)
|
||||
goPath := ""
|
||||
if goMonth != "" {
|
||||
goPath = filepath.Join(*goDir, goMonth)
|
||||
}
|
||||
|
||||
// Discover all packages across both caches.
|
||||
discoverStart := time.Now()
|
||||
allPkgs := discoverPackages(livePath, goPath)
|
||||
if len(filterPkgs) > 0 {
|
||||
nameSet := make(map[string]bool, len(filterPkgs))
|
||||
for _, n := range filterPkgs {
|
||||
nameSet[n] = true
|
||||
}
|
||||
var filtered []string
|
||||
for _, p := range allPkgs {
|
||||
if nameSet[p] {
|
||||
filtered = append(filtered, p)
|
||||
}
|
||||
}
|
||||
allPkgs = filtered
|
||||
}
|
||||
log.Printf("discovered %d packages in %s", len(allPkgs), time.Since(discoverStart))
|
||||
|
||||
compareStart := time.Now()
|
||||
var diffs []packageDiff
|
||||
for _, pkg := range allPkgs {
|
||||
d := compare(livePath, goPath, pkg, *latest, *windowed)
|
||||
categorize(&d)
|
||||
diffs = append(diffs, d)
|
||||
}
|
||||
log.Printf("compared %d packages in %s", len(diffs), time.Since(compareStart))
|
||||
|
||||
if *summary {
|
||||
printSummary(diffs)
|
||||
} else {
|
||||
printDetails(diffs, *diffsOnly, *sample)
|
||||
}
|
||||
|
||||
log.Printf("total: %s", time.Since(totalStart))
|
||||
}
|
||||
|
||||
func findLatestMonth(dir string) string {
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
var months []string
|
||||
for _, e := range entries {
|
||||
if e.IsDir() && len(e.Name()) == 7 && e.Name()[4] == '-' {
|
||||
months = append(months, e.Name())
|
||||
}
|
||||
}
|
||||
if len(months) == 0 {
|
||||
return ""
|
||||
}
|
||||
sort.Strings(months)
|
||||
return months[len(months)-1]
|
||||
}
|
||||
|
||||
func discoverPackages(livePath, goPath string) []string {
|
||||
seen := make(map[string]bool)
|
||||
for _, dir := range []string{livePath, goPath} {
|
||||
if dir == "" {
|
||||
continue
|
||||
}
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
for _, e := range entries {
|
||||
name := e.Name()
|
||||
if strings.HasSuffix(name, ".json") && !strings.HasSuffix(name, ".updated.txt") {
|
||||
pkg := strings.TrimSuffix(name, ".json")
|
||||
seen[pkg] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
var pkgs []string
|
||||
for p := range seen {
|
||||
pkgs = append(pkgs, p)
|
||||
}
|
||||
sort.Strings(pkgs)
|
||||
return pkgs
|
||||
}
|
||||
|
||||
func loadCache(dir, pkg string) *cacheEntry {
|
||||
if dir == "" {
|
||||
return nil
|
||||
}
|
||||
data, err := os.ReadFile(filepath.Join(dir, pkg+".json"))
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
var entry cacheEntry
|
||||
if err := json.Unmarshal(data, &entry); err != nil {
|
||||
return nil
|
||||
}
|
||||
return &entry
|
||||
}
|
||||
|
||||
// effectiveName returns the best available filename for a release entry.
|
||||
// Node.js sometimes uses _filename (a path) instead of name.
|
||||
func effectiveName(name, filename, download string) string {
|
||||
if name != "" {
|
||||
return name
|
||||
}
|
||||
if filename != "" {
|
||||
// _filename may be a path like "stable/macos/flutter_macos_3.41.4.zip"
|
||||
if i := strings.LastIndex(filename, "/"); i >= 0 {
|
||||
return filename[i+1:]
|
||||
}
|
||||
return filename
|
||||
}
|
||||
// Last resort: basename of download URL.
|
||||
if download != "" {
|
||||
if i := strings.LastIndex(download, "/"); i >= 0 {
|
||||
return download[i+1:]
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// versionWindow returns the 2nd and 2nd-to-last versions from a sorted
|
||||
// version list. This trims the edges where Node.js may have a newer fetch
|
||||
// or Go may have deeper history, focusing on the overlapping middle.
|
||||
func versionWindow(versions []string) (low, high string) {
|
||||
if len(versions) <= 2 {
|
||||
// Too few versions to window — use all.
|
||||
if len(versions) > 0 {
|
||||
return versions[0], versions[len(versions)-1]
|
||||
}
|
||||
return "", ""
|
||||
}
|
||||
// 2nd version (skip oldest) and 2nd-to-last (skip newest).
|
||||
return versions[1], versions[len(versions)-2]
|
||||
}
|
||||
|
||||
// filterVersionRange returns only the versions in sorted order that fall
|
||||
// within [low, high] inclusive (by lexver comparison).
|
||||
func filterVersionRange(vf map[string]map[string]bool, versions []string, low, high string) (map[string]bool, []string) {
|
||||
lowV := lexver.Parse(low)
|
||||
highV := lexver.Parse(high)
|
||||
|
||||
files := make(map[string]bool)
|
||||
var kept []string
|
||||
for _, v := range versions {
|
||||
pv := lexver.Parse(v)
|
||||
if lexver.Compare(pv, lowV) >= 0 && lexver.Compare(pv, highV) <= 0 {
|
||||
kept = append(kept, v)
|
||||
for f := range vf[v] {
|
||||
files[f] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
return files, kept
|
||||
}
|
||||
|
||||
func compare(livePath, goPath, pkg string, latestOnly, windowed bool) packageDiff {
|
||||
live := loadCache(livePath, pkg)
|
||||
goCache := loadCache(goPath, pkg)
|
||||
|
||||
d := packageDiff{Name: pkg}
|
||||
|
||||
if live == nil {
|
||||
d.LiveMissing = true
|
||||
}
|
||||
if goCache == nil {
|
||||
d.GoMissing = true
|
||||
}
|
||||
if d.LiveMissing && d.GoMissing {
|
||||
return d
|
||||
}
|
||||
|
||||
normVersion := normalizeVersionFunc(pkg)
|
||||
|
||||
// Collect filenames by version. If filter is non-nil, skip filenames it rejects.
|
||||
extractVersionFiles := func(ce *cacheEntry, filter func(string) bool) (map[string]map[string]bool, []string) {
|
||||
vf := make(map[string]map[string]bool)
|
||||
for _, r := range ce.Releases {
|
||||
name := effectiveName(r.Name, r.Filename, r.Download)
|
||||
if filter != nil && !filter(name) {
|
||||
continue
|
||||
}
|
||||
ver := normVersion(r.Version)
|
||||
if vf[ver] == nil {
|
||||
vf[ver] = make(map[string]bool)
|
||||
}
|
||||
vf[ver][name] = true
|
||||
}
|
||||
var versions []string
|
||||
for v := range vf {
|
||||
versions = append(versions, v)
|
||||
}
|
||||
slices.SortFunc(versions, func(a, b string) int {
|
||||
return lexver.Compare(lexver.Parse(a), lexver.Parse(b))
|
||||
})
|
||||
return vf, versions
|
||||
}
|
||||
notNoise := func(name string) bool { return !isLiveNoise(name) }
|
||||
|
||||
var liveFiles, goFiles map[string]bool
|
||||
|
||||
// Parse live cache.
|
||||
var liveVF map[string]map[string]bool
|
||||
var liveVersions []string
|
||||
if live != nil {
|
||||
liveVF, liveVersions = extractVersionFiles(live, notNoise)
|
||||
d.VersionsLive = liveVersions
|
||||
d.LiveCount = len(live.Releases)
|
||||
}
|
||||
|
||||
// Parse Go cache.
|
||||
var goVF map[string]map[string]bool
|
||||
var goVersions []string
|
||||
if goCache != nil {
|
||||
goVF, goVersions = extractVersionFiles(goCache, notNoise)
|
||||
d.VersionsGo = goVersions
|
||||
d.GoCount = len(goCache.Releases)
|
||||
}
|
||||
|
||||
// Determine which files to compare based on mode.
|
||||
if latestOnly {
|
||||
// Compare only the latest version from each cache.
|
||||
if live != nil && len(liveVersions) > 0 {
|
||||
liveFiles = liveVF[liveVersions[len(liveVersions)-1]]
|
||||
}
|
||||
if goCache != nil && len(goVersions) > 0 {
|
||||
goFiles = goVF[goVersions[len(goVersions)-1]]
|
||||
}
|
||||
} else if windowed && live != nil && len(liveVersions) > 0 {
|
||||
// Use the Node.js version range (2nd to 2nd-to-last) to establish
|
||||
// the window. Include ALL Node.js versions in the window (so missing
|
||||
// Go versions are visible), but exclude Go-only versions (those are
|
||||
// just deeper history, not real gaps).
|
||||
low, high := versionWindow(liveVersions)
|
||||
lowV := lexver.Parse(low)
|
||||
highV := lexver.Parse(high)
|
||||
|
||||
// Collect all live files in the window.
|
||||
liveFiles = make(map[string]bool)
|
||||
liveInWindow := make(map[string]bool)
|
||||
for _, v := range liveVersions {
|
||||
pv := lexver.Parse(v)
|
||||
if lexver.Compare(pv, lowV) >= 0 && lexver.Compare(pv, highV) <= 0 {
|
||||
liveInWindow[v] = true
|
||||
for f := range liveVF[v] {
|
||||
liveFiles[f] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// For Go, only include versions that Node.js also has in the window.
|
||||
// Go-only versions are hidden (deeper history, not gaps).
|
||||
goFiles = make(map[string]bool)
|
||||
for _, v := range goVersions {
|
||||
if !liveInWindow[v] {
|
||||
continue
|
||||
}
|
||||
for f := range goVF[v] {
|
||||
goFiles[f] = true
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Compare all versions — use pre-filtered version maps.
|
||||
if live != nil {
|
||||
liveFiles = make(map[string]bool)
|
||||
for _, files := range liveVF {
|
||||
for f := range files {
|
||||
liveFiles[f] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
if goCache != nil {
|
||||
goFiles = make(map[string]bool)
|
||||
for _, files := range goVF {
|
||||
for f := range files {
|
||||
goFiles[f] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if liveFiles == nil {
|
||||
liveFiles = make(map[string]bool)
|
||||
}
|
||||
if goFiles == nil {
|
||||
goFiles = make(map[string]bool)
|
||||
}
|
||||
|
||||
for f := range liveFiles {
|
||||
if !goFiles[f] {
|
||||
d.OnlyInLive = append(d.OnlyInLive, f)
|
||||
}
|
||||
}
|
||||
for f := range goFiles {
|
||||
if !liveFiles[f] {
|
||||
d.OnlyInGo = append(d.OnlyInGo, f)
|
||||
}
|
||||
}
|
||||
sort.Strings(d.OnlyInLive)
|
||||
sort.Strings(d.OnlyInGo)
|
||||
|
||||
// Field-level comparison on assets that exist in both caches.
|
||||
// Build version+filename → fields maps from each cache.
|
||||
if live != nil && goCache != nil {
|
||||
type assetKey struct {
|
||||
version string
|
||||
filename string
|
||||
}
|
||||
liveByKey := make(map[assetKey]cacheRelease)
|
||||
for _, r := range live.Releases {
|
||||
name := effectiveName(r.Name, r.Filename, r.Download)
|
||||
ver := normVersion(r.Version)
|
||||
liveByKey[assetKey{ver, name}] = r
|
||||
}
|
||||
|
||||
for _, r := range goCache.Releases {
|
||||
name := effectiveName(r.Name, r.Filename, r.Download)
|
||||
ver := normVersion(r.Version)
|
||||
lr, ok := liveByKey[assetKey{ver, name}]
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
|
||||
// Compare classification fields.
|
||||
// Use equivalence checks for os/arch/ext so naming
|
||||
// convention differences don't mask real classification bugs.
|
||||
for _, cmp := range []struct {
|
||||
field string
|
||||
live string
|
||||
go_ string
|
||||
equiv bool
|
||||
}{
|
||||
{"os", lr.OS, r.OS, equivOS(lr.OS, r.OS)},
|
||||
{"arch", lr.Arch, r.Arch, equivArch(lr.Arch, r.Arch)},
|
||||
{"libc", lr.Libc, r.Libc, lr.Libc == r.Libc},
|
||||
{"ext", lr.Ext, r.Ext, equivExt(lr.Ext, r.Ext)},
|
||||
{"channel", lr.Channel, r.Channel, lr.Channel == r.Channel},
|
||||
} {
|
||||
if cmp.equiv {
|
||||
continue
|
||||
}
|
||||
d.FieldDiffs = append(d.FieldDiffs, fieldDiff{
|
||||
Filename: name,
|
||||
Field: cmp.field,
|
||||
Live: cmp.live,
|
||||
Go: cmp.go_,
|
||||
BothSet: cmp.live != "" && cmp.go_ != "",
|
||||
})
|
||||
}
|
||||
}
|
||||
sort.Slice(d.FieldDiffs, func(i, j int) bool {
|
||||
if d.FieldDiffs[i].Field != d.FieldDiffs[j].Field {
|
||||
return d.FieldDiffs[i].Field < d.FieldDiffs[j].Field
|
||||
}
|
||||
return d.FieldDiffs[i].Filename < d.FieldDiffs[j].Filename
|
||||
})
|
||||
}
|
||||
|
||||
return d
|
||||
}
|
||||
|
||||
// equivOS returns true if two OS values are equivalent across naming conventions.
|
||||
func equivOS(a, b string) bool {
|
||||
return a == b || canonicalOS(a) == canonicalOS(b)
|
||||
}
|
||||
|
||||
func canonicalOS(s string) string {
|
||||
switch strings.ToLower(s) {
|
||||
case "darwin", "macos", "mac", "osx":
|
||||
return "darwin"
|
||||
case "win", "windows":
|
||||
return "windows"
|
||||
default:
|
||||
return strings.ToLower(s)
|
||||
}
|
||||
}
|
||||
|
||||
// equivArch returns true if two arch values are equivalent.
|
||||
func equivArch(a, b string) bool {
|
||||
return a == b || canonicalArch(a) == canonicalArch(b)
|
||||
}
|
||||
|
||||
func canonicalArch(s string) string {
|
||||
switch strings.ToLower(s) {
|
||||
case "x86_64", "amd64", "x64":
|
||||
return "x86_64"
|
||||
case "aarch64", "arm64":
|
||||
return "aarch64"
|
||||
case "armv7", "armv7l":
|
||||
return "armv7"
|
||||
case "armv6", "armv6l":
|
||||
return "armv6"
|
||||
case "x86", "i386", "i686", "386":
|
||||
return "x86"
|
||||
default:
|
||||
return strings.ToLower(s)
|
||||
}
|
||||
}
|
||||
|
||||
// equivExt returns true if two extension values are equivalent.
|
||||
func equivExt(a, b string) bool {
|
||||
// Normalize: strip leading dot, handle common aliases.
|
||||
return a == b || canonicalExt(a) == canonicalExt(b)
|
||||
}
|
||||
|
||||
func canonicalExt(s string) string {
|
||||
s = strings.TrimPrefix(s, ".")
|
||||
switch s {
|
||||
case "tgz":
|
||||
return "tar.gz"
|
||||
default:
|
||||
return s
|
||||
}
|
||||
}
|
||||
|
||||
func categorize(d *packageDiff) {
|
||||
if d.GoMissing {
|
||||
d.Categories = append(d.Categories, "go-missing")
|
||||
return
|
||||
}
|
||||
if d.LiveMissing {
|
||||
d.Categories = append(d.Categories, "live-missing")
|
||||
return
|
||||
}
|
||||
|
||||
if len(d.OnlyInLive) == 0 && len(d.OnlyInGo) == 0 && len(d.FieldDiffs) == 0 {
|
||||
d.Categories = append(d.Categories, "match")
|
||||
return
|
||||
}
|
||||
if len(d.OnlyInLive) == 0 && len(d.OnlyInGo) == 0 && len(d.FieldDiffs) > 0 {
|
||||
d.Categories = append(d.Categories, "fields-only")
|
||||
}
|
||||
|
||||
// Check if differences are only version depth (Go has more history).
|
||||
liveVersionSet := make(map[string]bool, len(d.VersionsLive))
|
||||
for _, v := range d.VersionsLive {
|
||||
liveVersionSet[v] = true
|
||||
}
|
||||
goVersionSet := make(map[string]bool, len(d.VersionsGo))
|
||||
for _, v := range d.VersionsGo {
|
||||
goVersionSet[v] = true
|
||||
}
|
||||
|
||||
goExtraVersions := 0
|
||||
for _, v := range d.VersionsGo {
|
||||
if !liveVersionSet[v] {
|
||||
goExtraVersions++
|
||||
}
|
||||
}
|
||||
liveExtraVersions := 0
|
||||
for _, v := range d.VersionsLive {
|
||||
if !goVersionSet[v] {
|
||||
liveExtraVersions++
|
||||
}
|
||||
}
|
||||
|
||||
if goExtraVersions > 0 {
|
||||
d.Categories = append(d.Categories, fmt.Sprintf("go-extra-versions(%d)", goExtraVersions))
|
||||
}
|
||||
if liveExtraVersions > 0 {
|
||||
d.Categories = append(d.Categories, fmt.Sprintf("live-extra-versions(%d)", liveExtraVersions))
|
||||
}
|
||||
|
||||
// Check for meta-asset filtering differences.
|
||||
metaOnlyInLive := 0
|
||||
nonMetaOnlyInLive := 0
|
||||
for _, f := range d.OnlyInLive {
|
||||
if classify.IsMetaAsset(f) {
|
||||
metaOnlyInLive++
|
||||
} else {
|
||||
nonMetaOnlyInLive++
|
||||
}
|
||||
}
|
||||
metaOnlyInGo := 0
|
||||
nonMetaOnlyInGo := 0
|
||||
for _, f := range d.OnlyInGo {
|
||||
if classify.IsMetaAsset(f) {
|
||||
metaOnlyInGo++
|
||||
} else {
|
||||
nonMetaOnlyInGo++
|
||||
}
|
||||
}
|
||||
|
||||
if metaOnlyInLive > 0 {
|
||||
d.Categories = append(d.Categories, fmt.Sprintf("live-has-meta(%d)", metaOnlyInLive))
|
||||
}
|
||||
if metaOnlyInGo > 0 {
|
||||
d.Categories = append(d.Categories, fmt.Sprintf("go-has-meta(%d)", metaOnlyInGo))
|
||||
}
|
||||
|
||||
// Check for source tarball differences.
|
||||
srcOnlyInGo := 0
|
||||
for _, f := range d.OnlyInGo {
|
||||
if strings.HasSuffix(f, ".tar.gz") || strings.HasSuffix(f, ".zip") {
|
||||
if strings.HasPrefix(f, "v") || strings.HasPrefix(f, "refs/") {
|
||||
srcOnlyInGo++
|
||||
}
|
||||
}
|
||||
}
|
||||
if srcOnlyInGo > 0 {
|
||||
d.Categories = append(d.Categories, fmt.Sprintf("go-has-source-tarballs(%d)", srcOnlyInGo))
|
||||
}
|
||||
|
||||
if nonMetaOnlyInLive > 0 {
|
||||
d.Categories = append(d.Categories, fmt.Sprintf("live-extra-assets(%d)", nonMetaOnlyInLive))
|
||||
}
|
||||
if nonMetaOnlyInGo > 0 {
|
||||
d.Categories = append(d.Categories, fmt.Sprintf("go-extra-assets(%d)", nonMetaOnlyInGo))
|
||||
}
|
||||
|
||||
// Count field diffs by field name, separating real disagreements
|
||||
// from expected "live empty, Go classified" differences.
|
||||
type fieldCount struct {
|
||||
bothSet int // both caches have a value but they disagree
|
||||
oneEmpty int // one side is empty (typically live — normalize.js fills at serve time)
|
||||
}
|
||||
fieldCounts := make(map[string]fieldCount)
|
||||
for _, fd := range d.FieldDiffs {
|
||||
fc := fieldCounts[fd.Field]
|
||||
if fd.BothSet {
|
||||
fc.bothSet++
|
||||
} else {
|
||||
fc.oneEmpty++
|
||||
}
|
||||
fieldCounts[fd.Field] = fc
|
||||
}
|
||||
for _, field := range []string{"os", "arch", "libc", "ext", "channel"} {
|
||||
fc := fieldCounts[field]
|
||||
if fc.bothSet > 0 {
|
||||
d.Categories = append(d.Categories, fmt.Sprintf("diff-%s(%d)", field, fc.bothSet))
|
||||
}
|
||||
if fc.oneEmpty > 0 {
|
||||
d.Categories = append(d.Categories, fmt.Sprintf("fill-%s(%d)", field, fc.oneEmpty))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// isLiveNoise returns true for filenames that the Node.js cache keeps
|
||||
// but Go intentionally filters out. Pre-filtering these from the live
|
||||
// side prevents them from appearing as live-extra-assets noise.
|
||||
//
|
||||
// This includes everything classify.IsMetaAsset catches plus formats
|
||||
// that Go's legacy export strips (.deb, .rpm, etc.).
|
||||
func isLiveNoise(name string) bool {
|
||||
if classify.IsMetaAsset(name) {
|
||||
return true
|
||||
}
|
||||
|
||||
lower := strings.ToLower(name)
|
||||
|
||||
// Formats Go filters from legacy export but Node.js keeps.
|
||||
for _, suffix := range []string{
|
||||
".deb", ".rpm", ".gpg",
|
||||
} {
|
||||
if strings.HasSuffix(lower, suffix) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
// Source tarballs (e.g. gitea-src-1.25.4.tar.gz, caddy_2.10.0_src.tar.gz, go1.26.1.src.tar.gz).
|
||||
if strings.Contains(lower, "-src-") || strings.Contains(lower, "_src.") || strings.Contains(lower, ".src.") || strings.HasPrefix(lower, "src-") {
|
||||
return true
|
||||
}
|
||||
|
||||
// Docs tarballs (e.g. gitea-docs-1.22.3.tar.gz).
|
||||
if strings.Contains(lower, "-docs-") {
|
||||
return true
|
||||
}
|
||||
|
||||
// Bare executables without any extension — typically legacy shell scripts
|
||||
// uploaded alongside proper archives (e.g. kubectx, kubens).
|
||||
if !strings.Contains(name, ".") {
|
||||
return true
|
||||
}
|
||||
|
||||
// GPU accelerator / hardware variants that Go tags as variants
|
||||
// but Node.js keeps with special arch names.
|
||||
for _, v := range []string{"-rocm", "-jetpack"} {
|
||||
if strings.Contains(lower, v) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
// Linux binaries for packages where Node.js only kept macOS .app.zip.
|
||||
// Go correctly includes these as installable on Linux.
|
||||
if strings.HasPrefix(lower, "fish-") && strings.Contains(lower, "-linux-") {
|
||||
return true
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// normalizeVersionFunc returns a version normalizer for a given package.
|
||||
// Most packages return the identity function. Some (like git) need
|
||||
// version string normalization to match across Go and Node.js caches.
|
||||
func normalizeVersionFunc(pkg string) func(string) string {
|
||||
switch pkg {
|
||||
case "git":
|
||||
return func(v string) string {
|
||||
// Git for Windows: v2.53.0.windows.1 → v2.53.0
|
||||
// v2.53.0.windows.2 → v2.53.0.2
|
||||
idx := strings.Index(v, ".windows.")
|
||||
if idx < 0 {
|
||||
return v
|
||||
}
|
||||
suffix := v[idx+len(".windows."):]
|
||||
base := v[:idx]
|
||||
if suffix == "1" {
|
||||
return base
|
||||
}
|
||||
return base + "." + suffix
|
||||
}
|
||||
case "lf":
|
||||
return func(v string) string {
|
||||
// lf: r21 → 0.21.0
|
||||
if strings.HasPrefix(v, "r") {
|
||||
return "0." + v[1:] + ".0"
|
||||
}
|
||||
return v
|
||||
}
|
||||
case "bun":
|
||||
return func(v string) string {
|
||||
// bun: bun-v1.3.9 → v1.3.9
|
||||
return strings.TrimPrefix(v, "bun-")
|
||||
}
|
||||
case "watchexec":
|
||||
return func(v string) string {
|
||||
// watchexec monorepo: cli-v1.20.5 → v1.20.5
|
||||
return strings.TrimPrefix(v, "cli-")
|
||||
}
|
||||
case "go":
|
||||
return func(v string) string {
|
||||
// Go: go1.10 → 1.10.0 (pad to 3 parts)
|
||||
v = strings.TrimPrefix(v, "go")
|
||||
parts := strings.SplitN(v, ".", 3)
|
||||
for len(parts) < 3 {
|
||||
parts = append(parts, "0")
|
||||
}
|
||||
return strings.Join(parts, ".")
|
||||
}
|
||||
default:
|
||||
return func(v string) string { return v }
|
||||
}
|
||||
}
|
||||
|
||||
func printSummary(diffs []packageDiff) {
|
||||
// Count by category.
|
||||
categoryCounts := make(map[string]int)
|
||||
for _, d := range diffs {
|
||||
for _, c := range d.Categories {
|
||||
// Strip the count suffix for grouping.
|
||||
base := c
|
||||
if idx := strings.Index(c, "("); idx != -1 {
|
||||
base = c[:idx]
|
||||
}
|
||||
categoryCounts[base]++
|
||||
}
|
||||
}
|
||||
|
||||
fmt.Println("=== COMPARISON SUMMARY ===")
|
||||
fmt.Printf("Total packages: %d\n\n", len(diffs))
|
||||
|
||||
var cats []string
|
||||
for c := range categoryCounts {
|
||||
cats = append(cats, c)
|
||||
}
|
||||
sort.Strings(cats)
|
||||
for _, c := range cats {
|
||||
fmt.Printf(" %-30s %d\n", c, categoryCounts[c])
|
||||
}
|
||||
|
||||
fmt.Println("\n=== PER-PACKAGE CATEGORIES ===")
|
||||
for _, d := range diffs {
|
||||
fmt.Printf("%-25s %s\n", d.Name, strings.Join(d.Categories, ", "))
|
||||
}
|
||||
}
|
||||
|
||||
func printDetails(diffs []packageDiff, diffsOnly bool, sampleN int) {
|
||||
for _, d := range diffs {
|
||||
if diffsOnly && len(d.OnlyInLive) == 0 && len(d.OnlyInGo) == 0 && len(d.FieldDiffs) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
fmt.Printf("=== %s ===\n", d.Name)
|
||||
fmt.Printf(" Categories: %s\n", strings.Join(d.Categories, ", "))
|
||||
fmt.Printf(" Live: %d assets, %d versions | Go: %d assets, %d versions\n",
|
||||
d.LiveCount, len(d.VersionsLive), d.GoCount, len(d.VersionsGo))
|
||||
|
||||
printAssetList("Only in LIVE", d.OnlyInLive, sampleN)
|
||||
printAssetList("Only in Go", d.OnlyInGo, sampleN)
|
||||
printFieldDiffs(d.FieldDiffs, sampleN)
|
||||
|
||||
fmt.Println()
|
||||
}
|
||||
}
|
||||
|
||||
// printFieldDiffs shows classification differences on shared assets.
|
||||
// Shows "real" diffs (both sides non-empty) first, then "fill" diffs
|
||||
// (one side empty) as a summary count only.
|
||||
func printFieldDiffs(diffs []fieldDiff, sampleN int) {
|
||||
if len(diffs) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
// Separate real disagreements from fill diffs.
|
||||
var real, fill []fieldDiff
|
||||
for _, fd := range diffs {
|
||||
if fd.BothSet {
|
||||
real = append(real, fd)
|
||||
} else {
|
||||
fill = append(fill, fd)
|
||||
}
|
||||
}
|
||||
|
||||
// Show real disagreements in detail.
|
||||
if len(real) > 0 {
|
||||
byField := make(map[string][]fieldDiff)
|
||||
for _, fd := range real {
|
||||
byField[fd.Field] = append(byField[fd.Field], fd)
|
||||
}
|
||||
|
||||
for _, field := range []string{"os", "arch", "libc", "ext", "channel"} {
|
||||
fds := byField[field]
|
||||
if len(fds) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
fmt.Printf(" DISAGREE %s (%d):\n", field, len(fds))
|
||||
printFieldDiffItems(fds, sampleN)
|
||||
}
|
||||
}
|
||||
|
||||
// Summarize fill diffs (live empty, Go classified) as counts.
|
||||
if len(fill) > 0 {
|
||||
byField := make(map[string]int)
|
||||
for _, fd := range fill {
|
||||
byField[fd.Field]++
|
||||
}
|
||||
var parts []string
|
||||
for _, field := range []string{"os", "arch", "libc", "ext", "channel"} {
|
||||
if n := byField[field]; n > 0 {
|
||||
parts = append(parts, fmt.Sprintf("%s(%d)", field, n))
|
||||
}
|
||||
}
|
||||
if len(parts) > 0 {
|
||||
fmt.Printf(" Go fills empty: %s\n", strings.Join(parts, ", "))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func printFieldDiffItems(fds []fieldDiff, sampleN int) {
|
||||
items := fds
|
||||
if sampleN > 0 && len(items) > sampleN {
|
||||
sampled := make([]fieldDiff, len(items))
|
||||
copy(sampled, items)
|
||||
rand.Shuffle(len(sampled), func(i, j int) {
|
||||
sampled[i], sampled[j] = sampled[j], sampled[i]
|
||||
})
|
||||
items = sampled[:sampleN]
|
||||
sort.Slice(items, func(i, j int) bool {
|
||||
return items[i].Filename < items[j].Filename
|
||||
})
|
||||
}
|
||||
|
||||
limit := 20
|
||||
for i, fd := range items {
|
||||
if sampleN == 0 && i >= limit {
|
||||
fmt.Printf(" ... and %d more\n", len(fds)-limit)
|
||||
break
|
||||
}
|
||||
fmt.Printf(" - %s: live=%q go=%q\n", fd.Filename, fd.Live, fd.Go)
|
||||
}
|
||||
if sampleN > 0 && len(fds) > sampleN {
|
||||
fmt.Printf(" ... sampled %d of %d\n", sampleN, len(fds))
|
||||
}
|
||||
}
|
||||
|
||||
// printAssetList prints a list of asset filenames, optionally sampling N at
|
||||
// random. When sampleN > 0 and the list is longer, it picks N random items
|
||||
// so you can spot classification bugs across the full range instead of only
|
||||
// seeing the first alphabetical entries.
|
||||
func printAssetList(label string, items []string, sampleN int) {
|
||||
if len(items) == 0 {
|
||||
return
|
||||
}
|
||||
|
||||
fmt.Printf(" %s (%d):\n", label, len(items))
|
||||
|
||||
if sampleN > 0 && len(items) > sampleN {
|
||||
// Shuffle a copy, take first N, then sort for readable output.
|
||||
sampled := make([]string, len(items))
|
||||
copy(sampled, items)
|
||||
rand.Shuffle(len(sampled), func(i, j int) {
|
||||
sampled[i], sampled[j] = sampled[j], sampled[i]
|
||||
})
|
||||
picked := sampled[:sampleN]
|
||||
sort.Strings(picked)
|
||||
for _, f := range picked {
|
||||
fmt.Printf(" - %s\n", f)
|
||||
}
|
||||
fmt.Printf(" ... sampled %d of %d (run again for different sample)\n", sampleN, len(items))
|
||||
return
|
||||
}
|
||||
|
||||
limit := 20
|
||||
for i, f := range items {
|
||||
if i >= limit {
|
||||
fmt.Printf(" ... and %d more\n", len(items)-limit)
|
||||
break
|
||||
}
|
||||
fmt.Printf(" - %s\n", f)
|
||||
}
|
||||
}
|
||||
846
cmd/e2etest/main.go
Normal file
846
cmd/e2etest/main.go
Normal file
@@ -0,0 +1,846 @@
|
||||
// Command e2etest runs the full release pipeline for selected packages
|
||||
// and compares results against the live webi.sh API.
|
||||
//
|
||||
// It fetches from upstream, classifies assets, resolves the best match
|
||||
// for a set of test queries, then fetches the same queries from the live
|
||||
// API and reports any differences.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// go run ./cmd/e2etest
|
||||
// go run ./cmd/e2etest -packages goreleaser,ollama,node
|
||||
// go run ./cmd/e2etest -cache ./_cache/raw # reuse existing cache
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/buildmeta"
|
||||
"github.com/webinstall/webi-installers/internal/installerconf"
|
||||
"github.com/webinstall/webi-installers/internal/lexver"
|
||||
"github.com/webinstall/webi-installers/internal/rawcache"
|
||||
"github.com/webinstall/webi-installers/internal/releases/github"
|
||||
"github.com/webinstall/webi-installers/internal/releases/githubish"
|
||||
"github.com/webinstall/webi-installers/internal/releases/nodedist"
|
||||
"github.com/webinstall/webi-installers/internal/resolve"
|
||||
)
|
||||
|
||||
// testCase is one query to resolve and compare against the live API.
|
||||
type testCase struct {
|
||||
Name string
|
||||
Package string
|
||||
OS buildmeta.OS
|
||||
Arch buildmeta.Arch
|
||||
Libc buildmeta.Libc
|
||||
Formats []string
|
||||
UA string // User-Agent for live API query
|
||||
}
|
||||
|
||||
// liveResult holds parsed fields from the live webi API response.
|
||||
type liveResult struct {
|
||||
Version string
|
||||
OS string
|
||||
Arch string
|
||||
Libc string
|
||||
Ext string
|
||||
PkgURL string
|
||||
PkgFile string
|
||||
Channel string
|
||||
Stable string
|
||||
Latest string
|
||||
Oses string
|
||||
Arches string
|
||||
Libcs string
|
||||
Formats string
|
||||
}
|
||||
|
||||
// UA format from webi.sh bootstrap: "curl {uname -s}/{uname -r} {uname -m}/unknown {libc}"
|
||||
// libc is "gnu", "musl", or "libc" (for darwin/other)
|
||||
var cases = []testCase{
|
||||
{
|
||||
Name: "goreleaser/linux/x86_64", Package: "goreleaser",
|
||||
OS: buildmeta.OSLinux, Arch: buildmeta.ArchAMD64, Libc: buildmeta.LibcGNU,
|
||||
Formats: []string{".tar.gz", ".tar.xz", ".zip"},
|
||||
UA: "curl Linux/6.6.123 x86_64/unknown gnu",
|
||||
},
|
||||
{
|
||||
Name: "goreleaser/darwin/arm64", Package: "goreleaser",
|
||||
OS: buildmeta.OSDarwin, Arch: buildmeta.ArchARM64, Libc: "",
|
||||
Formats: []string{".tar.gz", ".tar.xz", ".zip"},
|
||||
UA: "curl Darwin/25.2.0 arm64/unknown libc",
|
||||
},
|
||||
{
|
||||
Name: "goreleaser/windows/x86_64", Package: "goreleaser",
|
||||
OS: buildmeta.OSWindows, Arch: buildmeta.ArchAMD64, Libc: "",
|
||||
Formats: []string{".zip", ".exe"},
|
||||
UA: "PowerShell/7.0 Windows/10.0 x86_64/unknown msvc",
|
||||
},
|
||||
{
|
||||
Name: "ollama/linux/x86_64", Package: "ollama",
|
||||
OS: buildmeta.OSLinux, Arch: buildmeta.ArchAMD64, Libc: buildmeta.LibcGNU,
|
||||
Formats: []string{".tar.gz", ".tar.xz", ".tar.zst", ".zip"},
|
||||
UA: "curl Linux/6.6.123 x86_64/unknown gnu",
|
||||
},
|
||||
{
|
||||
Name: "ollama/darwin/arm64", Package: "ollama",
|
||||
OS: buildmeta.OSDarwin, Arch: buildmeta.ArchARM64, Libc: "",
|
||||
Formats: []string{".tar.gz", ".tar.xz", ".tar.zst", ".zip", ".dmg"},
|
||||
UA: "curl Darwin/25.2.0 arm64/unknown libc",
|
||||
},
|
||||
{
|
||||
Name: "ollama/linux/arm64", Package: "ollama",
|
||||
OS: buildmeta.OSLinux, Arch: buildmeta.ArchARM64, Libc: buildmeta.LibcGNU,
|
||||
Formats: []string{".tar.gz", ".tar.xz", ".tar.zst", ".zip"},
|
||||
UA: "curl Linux/6.6.123 aarch64/unknown gnu",
|
||||
},
|
||||
{
|
||||
Name: "node/linux/x86_64", Package: "node",
|
||||
OS: buildmeta.OSLinux, Arch: buildmeta.ArchAMD64, Libc: buildmeta.LibcGNU,
|
||||
Formats: []string{".tar.xz", ".tar.gz", ".zip"},
|
||||
UA: "curl Linux/6.6.123 x86_64/unknown gnu",
|
||||
},
|
||||
{
|
||||
Name: "node/darwin/arm64", Package: "node",
|
||||
OS: buildmeta.OSDarwin, Arch: buildmeta.ArchARM64, Libc: "",
|
||||
Formats: []string{".tar.xz", ".tar.gz", ".zip"},
|
||||
UA: "curl Darwin/25.2.0 arm64/unknown libc",
|
||||
},
|
||||
{
|
||||
Name: "node/linux/arm64", Package: "node",
|
||||
OS: buildmeta.OSLinux, Arch: buildmeta.ArchARM64, Libc: buildmeta.LibcGNU,
|
||||
Formats: []string{".tar.xz", ".tar.gz", ".zip"},
|
||||
UA: "curl Linux/6.6.123 aarch64/unknown gnu",
|
||||
},
|
||||
}
|
||||
|
||||
func main() {
|
||||
cacheDir := flag.String("cache", "_cache/raw", "root directory for raw cache")
|
||||
confDir := flag.String("conf", ".", "root directory containing {pkg}/releases.conf files")
|
||||
token := flag.String("token", os.Getenv("GITHUB_TOKEN"), "GitHub API token")
|
||||
skipFetch := flag.Bool("skip-fetch", false, "skip fetching, use existing cache")
|
||||
skipLive := flag.Bool("skip-live", false, "skip live API comparison")
|
||||
packages := flag.String("packages", "goreleaser,ollama,node", "comma-separated packages to test")
|
||||
flag.Parse()
|
||||
|
||||
pkgList := strings.Split(*packages, ",")
|
||||
pkgSet := make(map[string]bool, len(pkgList))
|
||||
for _, p := range pkgList {
|
||||
pkgSet[strings.TrimSpace(p)] = true
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
client := &http.Client{Timeout: 30 * time.Second}
|
||||
var auth *githubish.Auth
|
||||
if *token != "" {
|
||||
auth = &githubish.Auth{Token: *token}
|
||||
}
|
||||
|
||||
// Step 1: Fetch raw releases.
|
||||
if !*skipFetch {
|
||||
log.Println("=== Step 1: Fetching releases ===")
|
||||
for _, pkg := range pkgList {
|
||||
if err := fetchPackage(ctx, client, *cacheDir, *confDir, pkg, auth); err != nil {
|
||||
log.Fatalf("fetch %s: %v", pkg, err)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
log.Println("=== Step 1: Skipping fetch (using cache) ===")
|
||||
}
|
||||
|
||||
// Step 2: Classify releases.
|
||||
log.Println("=== Step 2: Classifying releases ===")
|
||||
allDists := make(map[string][]resolve.Dist)
|
||||
for _, pkg := range pkgList {
|
||||
conf, err := installerconf.Read(filepath.Join(*confDir, pkg, "releases.conf"))
|
||||
if err != nil {
|
||||
log.Fatalf("read conf %s: %v", pkg, err)
|
||||
}
|
||||
d, err := rawcache.Open(filepath.Join(*cacheDir, pkg))
|
||||
if err != nil {
|
||||
log.Fatalf("open cache %s: %v", pkg, err)
|
||||
}
|
||||
dists, err := classifyFromCache(pkg, conf, d)
|
||||
if err != nil {
|
||||
log.Fatalf("classify %s: %v", pkg, err)
|
||||
}
|
||||
allDists[pkg] = dists
|
||||
log.Printf(" %s: %d distributables", pkg, len(dists))
|
||||
|
||||
// Show catalog.
|
||||
cat := resolve.Survey(dists)
|
||||
log.Printf(" oses=%v arches=%v libcs=%v formats=%v", cat.OSes, cat.Arches, cat.Libcs, cat.Formats)
|
||||
log.Printf(" latest=%s stable=%s", cat.Latest, cat.Stable)
|
||||
}
|
||||
|
||||
// Step 3: Resolve best match for each test case.
|
||||
log.Println("=== Step 3: Resolving best matches ===")
|
||||
type result struct {
|
||||
tc testCase
|
||||
match *resolve.Match
|
||||
live *liveResult
|
||||
}
|
||||
var results []result
|
||||
for _, tc := range cases {
|
||||
if !pkgSet[tc.Package] {
|
||||
continue
|
||||
}
|
||||
dists := allDists[tc.Package]
|
||||
q := resolve.Query{
|
||||
OS: tc.OS,
|
||||
Arch: tc.Arch,
|
||||
Libc: tc.Libc,
|
||||
Formats: tc.Formats,
|
||||
Channel: "stable",
|
||||
}
|
||||
m := resolve.Best(dists, q)
|
||||
results = append(results, result{tc: tc, match: m})
|
||||
}
|
||||
|
||||
// Step 4: Compare with live API.
|
||||
if !*skipLive {
|
||||
log.Println("=== Step 4: Comparing with live API ===")
|
||||
for i := range results {
|
||||
tc := results[i].tc
|
||||
live, err := queryLiveAPI(client, tc)
|
||||
if err != nil {
|
||||
log.Printf(" %s: live API error: %v", tc.Name, err)
|
||||
continue
|
||||
}
|
||||
results[i].live = live
|
||||
}
|
||||
}
|
||||
|
||||
// Step 5: Report.
|
||||
log.Println("")
|
||||
log.Println("=== Results ===")
|
||||
log.Println("")
|
||||
|
||||
pass, fail, warn := 0, 0, 0
|
||||
for _, r := range results {
|
||||
tc := r.tc
|
||||
m := r.match
|
||||
live := r.live
|
||||
|
||||
if m == nil {
|
||||
log.Printf("FAIL %s: no match found", tc.Name)
|
||||
fail++
|
||||
continue
|
||||
}
|
||||
|
||||
log.Printf("--- %s ---", tc.Name)
|
||||
log.Printf(" Go: version=%s file=%s ext=%s url=%s", m.Version, m.Filename, m.Format, m.Download)
|
||||
|
||||
if live != nil {
|
||||
log.Printf(" Live: version=%s file=%s ext=%s url=%s", live.Version, live.PkgFile, live.Ext, live.PkgURL)
|
||||
|
||||
if live.Version == "0.0.0" {
|
||||
log.Printf(" WARN: live API returned error (no match)")
|
||||
warn++
|
||||
} else if m.Version == live.Version && m.Filename == live.PkgFile {
|
||||
log.Printf(" PASS: exact match")
|
||||
pass++
|
||||
} else if m.Version == live.Version && m.Download == live.PkgURL {
|
||||
log.Printf(" PASS: same URL (filename display differs: go=%s live=%s)", m.Filename, live.PkgFile)
|
||||
pass++
|
||||
} else if m.Version == live.Version {
|
||||
log.Printf(" WARN: same version, different file (go=%s live=%s)", m.Filename, live.PkgFile)
|
||||
warn++
|
||||
} else {
|
||||
log.Printf(" DIFF: version mismatch (go=%s live=%s)", m.Version, live.Version)
|
||||
fail++
|
||||
}
|
||||
} else {
|
||||
log.Printf(" (no live comparison)")
|
||||
pass++
|
||||
}
|
||||
}
|
||||
|
||||
log.Println("")
|
||||
log.Printf("Summary: %d pass, %d fail, %d warn (live API errors)", pass, fail, warn)
|
||||
|
||||
if fail > 0 {
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// fetchPackage fetches raw releases for one package.
|
||||
func fetchPackage(ctx context.Context, client *http.Client, cacheRoot, confDir, pkg string, auth *githubish.Auth) error {
|
||||
conf, err := installerconf.Read(filepath.Join(confDir, pkg, "releases.conf"))
|
||||
if err != nil {
|
||||
return fmt.Errorf("read conf: %w", err)
|
||||
}
|
||||
|
||||
source := conf.Source
|
||||
log.Printf(" %s: source=%s", pkg, source)
|
||||
|
||||
switch source {
|
||||
case "github":
|
||||
return fetchGitHub(ctx, client, cacheRoot, pkg, conf, auth)
|
||||
case "nodedist":
|
||||
return fetchNodeDist(ctx, client, cacheRoot, pkg, conf)
|
||||
default:
|
||||
return fmt.Errorf("unsupported source %q (only github and nodedist for e2e test)", source)
|
||||
}
|
||||
}
|
||||
|
||||
func fetchGitHub(ctx context.Context, client *http.Client, cacheRoot, pkg string, conf *installerconf.Conf, auth *githubish.Auth) error {
|
||||
owner := conf.Owner
|
||||
repo := conf.Repo
|
||||
tagPrefix := conf.TagPrefix
|
||||
|
||||
d, err := rawcache.Open(filepath.Join(cacheRoot, pkg))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var added, changed, skipped int
|
||||
var latest string
|
||||
for batch, err := range github.Fetch(ctx, client, owner, repo, auth) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("github %s/%s: %w", owner, repo, err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
if rel.Draft {
|
||||
continue
|
||||
}
|
||||
|
||||
tag := rel.TagName
|
||||
if tagPrefix != "" {
|
||||
if !strings.HasPrefix(tag, tagPrefix) {
|
||||
continue
|
||||
}
|
||||
tag = strings.TrimPrefix(tag, tagPrefix)
|
||||
}
|
||||
|
||||
data, err := json.Marshal(rel)
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal %s: %w", tag, err)
|
||||
}
|
||||
|
||||
action, err := d.Merge(tag, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch action {
|
||||
case "added":
|
||||
added++
|
||||
case "changed":
|
||||
changed++
|
||||
default:
|
||||
skipped++
|
||||
}
|
||||
|
||||
if latest == "" && !rel.Prerelease {
|
||||
latest = tag
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if latest != "" {
|
||||
current := d.Latest()
|
||||
if current == "" || lexver.Compare(lexver.Parse(latest), lexver.Parse(current)) > 0 {
|
||||
d.SetLatest(latest)
|
||||
}
|
||||
}
|
||||
|
||||
log.Printf(" +%d ~%d =%d latest=%s", added, changed, skipped, d.Latest())
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchNodeDist(ctx context.Context, client *http.Client, cacheRoot, pkg string, conf *installerconf.Conf) error {
|
||||
baseURL := conf.BaseURL
|
||||
d, err := rawcache.Open(filepath.Join(cacheRoot, pkg))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var added, changed, skipped int
|
||||
var latest string
|
||||
for batch, err := range nodedist.Fetch(ctx, client, baseURL) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("nodedist: %w", err)
|
||||
}
|
||||
for _, entry := range batch {
|
||||
tag := entry.Version
|
||||
data, err := json.Marshal(entry)
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal %s: %w", tag, err)
|
||||
}
|
||||
|
||||
action, err := d.Merge(tag, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch action {
|
||||
case "added":
|
||||
added++
|
||||
case "changed":
|
||||
changed++
|
||||
default:
|
||||
skipped++
|
||||
}
|
||||
|
||||
if latest == "" {
|
||||
latest = tag
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if latest != "" {
|
||||
current := d.Latest()
|
||||
if current == "" || lexver.Compare(lexver.Parse(latest), lexver.Parse(current)) > 0 {
|
||||
d.SetLatest(latest)
|
||||
}
|
||||
}
|
||||
|
||||
log.Printf(" +%d ~%d =%d latest=%s", added, changed, skipped, d.Latest())
|
||||
return nil
|
||||
}
|
||||
|
||||
// classifyFromCache reads the raw cache and produces classified dists.
|
||||
func classifyFromCache(pkg string, conf *installerconf.Conf, d *rawcache.Dir) ([]resolve.Dist, error) {
|
||||
source := conf.Source
|
||||
switch source {
|
||||
case "github":
|
||||
return classifyGitHub(pkg, conf, d)
|
||||
case "nodedist":
|
||||
return classifyNodeDist(pkg, conf, d)
|
||||
default:
|
||||
return nil, fmt.Errorf("unsupported source %q", source)
|
||||
}
|
||||
}
|
||||
|
||||
func classifyGitHub(pkg string, conf *installerconf.Conf, d *rawcache.Dir) ([]resolve.Dist, error) {
|
||||
tagPrefix := conf.TagPrefix
|
||||
releases, err := readAllReleases(d)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var dists []resolve.Dist
|
||||
for _, data := range releases {
|
||||
var rel struct {
|
||||
TagName string `json:"tag_name"`
|
||||
Prerelease bool `json:"prerelease"`
|
||||
Draft bool `json:"draft"`
|
||||
PublishedAt string `json:"published_at"`
|
||||
Assets []struct {
|
||||
Name string `json:"name"`
|
||||
BrowserDownloadURL string `json:"browser_download_url"`
|
||||
Size int64 `json:"size"`
|
||||
} `json:"assets"`
|
||||
}
|
||||
if err := json.Unmarshal(data, &rel); err != nil {
|
||||
continue
|
||||
}
|
||||
if rel.Draft {
|
||||
continue
|
||||
}
|
||||
|
||||
version := rel.TagName
|
||||
if tagPrefix != "" {
|
||||
version = strings.TrimPrefix(version, tagPrefix)
|
||||
}
|
||||
// Strip leading "v" for version normalization.
|
||||
version = strings.TrimPrefix(version, "v")
|
||||
|
||||
channel := "stable"
|
||||
if rel.Prerelease {
|
||||
channel = "beta"
|
||||
}
|
||||
|
||||
date := ""
|
||||
if len(rel.PublishedAt) >= 10 {
|
||||
date = rel.PublishedAt[:10]
|
||||
}
|
||||
|
||||
for _, asset := range rel.Assets {
|
||||
if isMetaAsset(asset.Name) {
|
||||
continue
|
||||
}
|
||||
|
||||
r := classifyFilename(asset.Name)
|
||||
extra := detectExtra(asset.Name)
|
||||
dists = append(dists, resolve.Dist{
|
||||
Package: pkg,
|
||||
Version: version,
|
||||
Channel: channel,
|
||||
OS: r.os,
|
||||
Arch: r.arch,
|
||||
Libc: r.libc,
|
||||
Format: r.format,
|
||||
Download: asset.BrowserDownloadURL,
|
||||
Filename: asset.Name,
|
||||
Size: asset.Size,
|
||||
Date: date,
|
||||
Extra: extra,
|
||||
})
|
||||
}
|
||||
}
|
||||
return dists, nil
|
||||
}
|
||||
|
||||
func classifyNodeDist(pkg string, conf *installerconf.Conf, d *rawcache.Dir) ([]resolve.Dist, error) {
|
||||
baseURL := conf.BaseURL
|
||||
releases, err := readAllReleases(d)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var dists []resolve.Dist
|
||||
for _, data := range releases {
|
||||
var entry struct {
|
||||
Version string `json:"version"`
|
||||
Date string `json:"date"`
|
||||
Files []string `json:"files"`
|
||||
LTS json.RawMessage `json:"lts"`
|
||||
Security bool `json:"security"`
|
||||
}
|
||||
if err := json.Unmarshal(data, &entry); err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
lts := string(entry.LTS) != "false" && string(entry.LTS) != ""
|
||||
version := strings.TrimPrefix(entry.Version, "v")
|
||||
|
||||
// Webi treats even major versions as "stable" (LTS-eligible).
|
||||
channel := "stable"
|
||||
parts := strings.SplitN(version, ".", 2)
|
||||
if len(parts) > 0 {
|
||||
var major int
|
||||
fmt.Sscanf(parts[0], "%d", &major)
|
||||
if major%2 != 0 {
|
||||
channel = "beta"
|
||||
}
|
||||
}
|
||||
|
||||
for _, file := range entry.Files {
|
||||
if file == "src" || file == "headers" {
|
||||
continue
|
||||
}
|
||||
fileDists := expandNodeFile(pkg, entry.Version, version, channel, entry.Date, lts, baseURL, file)
|
||||
dists = append(dists, fileDists...)
|
||||
}
|
||||
}
|
||||
return dists, nil
|
||||
}
|
||||
|
||||
func expandNodeFile(pkg, rawVersion, version, channel, date string, lts bool, baseURL, file string) []resolve.Dist {
|
||||
parts := strings.Split(file, "-")
|
||||
if len(parts) < 2 {
|
||||
return nil
|
||||
}
|
||||
|
||||
osMap := map[string]string{
|
||||
"osx": "darwin", "linux": "linux", "win": "windows",
|
||||
"sunos": "sunos", "aix": "aix",
|
||||
}
|
||||
archMap := map[string]string{
|
||||
"x64": "x86_64", "x86": "x86", "arm64": "aarch64",
|
||||
"armv7l": "armv7", "armv6l": "armv6",
|
||||
"ppc64": "ppc64", "ppc64le": "ppc64le", "s390x": "s390x",
|
||||
"loong64": "loong64", "riscv64": "riscv64",
|
||||
}
|
||||
|
||||
os_ := osMap[parts[0]]
|
||||
arch := archMap[parts[1]]
|
||||
if os_ == "" || arch == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
libc := ""
|
||||
pkgType := ""
|
||||
if len(parts) > 2 {
|
||||
pkgType = parts[2]
|
||||
}
|
||||
|
||||
var formats []string
|
||||
switch pkgType {
|
||||
case "musl":
|
||||
libc = "musl"
|
||||
formats = []string{".tar.gz", ".tar.xz"}
|
||||
case "tar":
|
||||
formats = []string{".tar.gz", ".tar.xz"}
|
||||
case "zip":
|
||||
formats = []string{".zip"}
|
||||
case "7z":
|
||||
formats = []string{".7z"}
|
||||
case "pkg":
|
||||
formats = []string{".pkg"}
|
||||
case "msi":
|
||||
formats = []string{".msi"}
|
||||
case "exe":
|
||||
formats = []string{".exe"}
|
||||
case "":
|
||||
formats = []string{".tar.gz", ".tar.xz"}
|
||||
default:
|
||||
return nil
|
||||
}
|
||||
|
||||
if libc == "" && os_ == "linux" {
|
||||
libc = "gnu"
|
||||
}
|
||||
|
||||
osPart := parts[0]
|
||||
if osPart == "osx" {
|
||||
osPart = "darwin"
|
||||
}
|
||||
archPart := parts[1]
|
||||
muslExtra := ""
|
||||
if libc == "musl" {
|
||||
muslExtra = "-musl"
|
||||
}
|
||||
|
||||
var dists []resolve.Dist
|
||||
for _, format := range formats {
|
||||
var filename string
|
||||
if format == ".msi" {
|
||||
filename = fmt.Sprintf("node-%s-%s%s%s", rawVersion, archPart, muslExtra, format)
|
||||
} else {
|
||||
filename = fmt.Sprintf("node-%s-%s-%s%s%s", rawVersion, osPart, archPart, muslExtra, format)
|
||||
}
|
||||
|
||||
dists = append(dists, resolve.Dist{
|
||||
Package: pkg,
|
||||
Version: version,
|
||||
Channel: channel,
|
||||
OS: os_,
|
||||
Arch: arch,
|
||||
Libc: libc,
|
||||
Format: format,
|
||||
Download: fmt.Sprintf("%s/%s/%s", baseURL, rawVersion, filename),
|
||||
Filename: filename,
|
||||
LTS: lts,
|
||||
Date: date,
|
||||
})
|
||||
}
|
||||
return dists
|
||||
}
|
||||
|
||||
// queryLiveAPI queries the live webi.sh API and parses the response header.
|
||||
func queryLiveAPI(client *http.Client, tc testCase) (*liveResult, error) {
|
||||
// Build format string matching what the webi.sh bootstrap sends.
|
||||
// Order: tar,exe,zip,xz,dmg,git (least to most favorable in bootstrap,
|
||||
// but the API doesn't care about order).
|
||||
fmtParam := "tar,exe,zip,xz,dmg"
|
||||
|
||||
url := fmt.Sprintf("https://webi.sh/api/installers/%s@stable.sh?formats=%s", tc.Package, fmtParam)
|
||||
|
||||
req, err := http.NewRequest("GET", url, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("User-Agent", tc.UA)
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
body, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return parseLiveResponse(string(body)), nil
|
||||
}
|
||||
|
||||
// parseLiveResponse extracts WEBI_* and PKG_* variables from the shell script.
|
||||
func parseLiveResponse(body string) *liveResult {
|
||||
vars := make(map[string]string)
|
||||
scanner := bufio.NewScanner(strings.NewReader(body))
|
||||
for scanner.Scan() {
|
||||
line := strings.TrimSpace(scanner.Text())
|
||||
for _, prefix := range []string{"WEBI_", "PKG_"} {
|
||||
if strings.HasPrefix(line, prefix) {
|
||||
if eq := strings.IndexByte(line, '='); eq > 0 {
|
||||
key := line[:eq]
|
||||
val := line[eq+1:]
|
||||
val = strings.Trim(val, "'\"")
|
||||
vars[key] = val
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return &liveResult{
|
||||
Version: vars["WEBI_VERSION"],
|
||||
OS: vars["WEBI_OS"],
|
||||
Arch: vars["WEBI_ARCH"],
|
||||
Libc: vars["WEBI_LIBC"],
|
||||
Ext: vars["WEBI_EXT"],
|
||||
PkgURL: vars["WEBI_PKG_URL"],
|
||||
PkgFile: vars["WEBI_PKG_FILE"],
|
||||
Channel: vars["WEBI_CHANNEL"],
|
||||
Stable: vars["PKG_STABLE"],
|
||||
Latest: vars["PKG_LATEST"],
|
||||
Oses: vars["PKG_OSES"],
|
||||
Arches: vars["PKG_ARCHES"],
|
||||
Libcs: vars["PKG_LIBCS"],
|
||||
Formats: vars["PKG_FORMATS"],
|
||||
}
|
||||
}
|
||||
|
||||
// readAllReleases reads all cached release files.
|
||||
func readAllReleases(d *rawcache.Dir) (map[string][]byte, error) {
|
||||
active, err := d.ActivePath()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
entries, err := os.ReadDir(active)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result := make(map[string][]byte, len(entries))
|
||||
for _, e := range entries {
|
||||
if e.IsDir() || strings.HasPrefix(e.Name(), "_") {
|
||||
continue
|
||||
}
|
||||
data, err := os.ReadFile(filepath.Join(active, e.Name()))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
result[e.Name()] = data
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
type classResult struct {
|
||||
os, arch, libc, format string
|
||||
}
|
||||
|
||||
func classifyFilename(name string) classResult {
|
||||
// Use the classify package.
|
||||
// Import it indirectly to avoid circular deps — inline the logic
|
||||
// we need for the e2e test.
|
||||
lower := strings.ToLower(name)
|
||||
|
||||
var r classResult
|
||||
r.format = detectFormat(name)
|
||||
|
||||
// OS detection
|
||||
switch {
|
||||
case strings.Contains(lower, "linux"):
|
||||
r.os = "linux"
|
||||
case strings.Contains(lower, "darwin") || strings.Contains(lower, "macos") || strings.Contains(lower, "apple"):
|
||||
r.os = "darwin"
|
||||
case strings.Contains(lower, "windows") || strings.Contains(lower, "win64") || strings.Contains(lower, "win32"):
|
||||
r.os = "windows"
|
||||
case strings.HasSuffix(lower, ".dmg") || strings.HasSuffix(lower, ".app.zip"):
|
||||
r.os = "darwin"
|
||||
case strings.HasSuffix(lower, ".exe") || strings.HasSuffix(lower, ".msi"):
|
||||
r.os = "windows"
|
||||
case strings.Contains(lower, "freebsd"):
|
||||
r.os = "freebsd"
|
||||
}
|
||||
|
||||
// Arch detection
|
||||
switch {
|
||||
case strings.Contains(lower, "x86_64") || strings.Contains(lower, "amd64") || strings.Contains(lower, "x64"):
|
||||
r.arch = "x86_64"
|
||||
case strings.Contains(lower, "aarch64") || strings.Contains(lower, "arm64"):
|
||||
r.arch = "aarch64"
|
||||
case strings.Contains(lower, "armv7") || strings.Contains(lower, "armhf"):
|
||||
r.arch = "armv7"
|
||||
case strings.Contains(lower, "armv6"):
|
||||
r.arch = "armv6"
|
||||
case strings.Contains(lower, "i686") || strings.Contains(lower, "i386") || strings.Contains(lower, "x86") || strings.Contains(lower, "386"):
|
||||
r.arch = "x86"
|
||||
case strings.Contains(lower, "ppc64le") || strings.Contains(lower, "powerpc64le"):
|
||||
r.arch = "ppc64le"
|
||||
case strings.Contains(lower, "ppc64") || strings.Contains(lower, "powerpc64"):
|
||||
r.arch = "ppc64"
|
||||
case strings.Contains(lower, "riscv64"):
|
||||
r.arch = "riscv64"
|
||||
case strings.Contains(lower, "s390x"):
|
||||
r.arch = "s390x"
|
||||
case strings.Contains(lower, "loong64"):
|
||||
r.arch = "loong64"
|
||||
}
|
||||
|
||||
// Libc detection
|
||||
switch {
|
||||
case strings.Contains(lower, "musl"):
|
||||
r.libc = "musl"
|
||||
case strings.Contains(lower, "gnu"):
|
||||
r.libc = "gnu"
|
||||
case strings.Contains(lower, "msvc"):
|
||||
r.libc = "msvc"
|
||||
}
|
||||
|
||||
return r
|
||||
}
|
||||
|
||||
func detectFormat(name string) string {
|
||||
lower := strings.ToLower(name)
|
||||
for _, ext := range []string{".tar.gz", ".tar.xz", ".tar.bz2", ".tar.zst", ".exe.xz", ".app.zip"} {
|
||||
if strings.HasSuffix(lower, ext) {
|
||||
return ext
|
||||
}
|
||||
}
|
||||
// .tgz is a common alias for .tar.gz
|
||||
if strings.HasSuffix(lower, ".tgz") {
|
||||
return ".tar.gz"
|
||||
}
|
||||
return filepath.Ext(lower)
|
||||
}
|
||||
|
||||
// detectExtra identifies GPU/vendor-specific variant suffixes in filenames
|
||||
// like "ollama-linux-amd64-rocm.tar.zst" or "ollama-linux-arm64-jetpack5.tar.zst".
|
||||
func detectExtra(name string) string {
|
||||
lower := strings.ToLower(name)
|
||||
for _, variant := range []string{
|
||||
"-rocm", "-jetpack", "-cuda", "-vulkan", "-metal",
|
||||
"-extended", "-static", "-debug", "-nightly",
|
||||
} {
|
||||
if strings.Contains(lower, variant) {
|
||||
return strings.TrimPrefix(variant, "-")
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
func isMetaAsset(name string) bool {
|
||||
lower := strings.ToLower(name)
|
||||
for _, suffix := range []string{
|
||||
".sha256", ".sha256sum", ".sha512", ".sha512sum",
|
||||
".md5", ".md5sum", ".sig", ".asc", ".pem",
|
||||
"checksums.txt", "sha256sums", "sha512sums",
|
||||
".sbom", ".spdx", ".json.sig", ".sigstore",
|
||||
".d.ts", ".pub",
|
||||
} {
|
||||
if strings.HasSuffix(lower, suffix) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
for _, contains := range []string{
|
||||
"checksums", "sha256sum", "sha512sum",
|
||||
"buildable-artifact",
|
||||
} {
|
||||
if strings.Contains(lower, contains) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
for _, exact := range []string{
|
||||
"install.sh", "install.ps1", "compat.json",
|
||||
} {
|
||||
if lower == exact {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
862
cmd/fetchraw/main.go
Normal file
862
cmd/fetchraw/main.go
Normal file
@@ -0,0 +1,862 @@
|
||||
// Command fetchraw fetches release histories from upstream APIs and
|
||||
// merges them into rawcache. Safe to run repeatedly — unchanged releases
|
||||
// are skipped, new/changed ones are recorded in the audit log.
|
||||
//
|
||||
// Reads releases.conf files from package directories to discover what
|
||||
// to fetch. Adding a new package is just creating a conf file.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// go run ./cmd/fetchraw -cache ./_cache/raw
|
||||
// go run ./cmd/fetchraw -cache ./_cache/raw hugo caddy
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/installerconf"
|
||||
"github.com/webinstall/webi-installers/internal/lexver"
|
||||
"github.com/webinstall/webi-installers/internal/rawcache"
|
||||
"github.com/webinstall/webi-installers/internal/releases/chromedist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/flutterdist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/gitea"
|
||||
"github.com/webinstall/webi-installers/internal/releases/github"
|
||||
"github.com/webinstall/webi-installers/internal/releases/githubish"
|
||||
"github.com/webinstall/webi-installers/internal/releases/gittag"
|
||||
"github.com/webinstall/webi-installers/internal/releases/golang"
|
||||
"github.com/webinstall/webi-installers/internal/releases/gpgdist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/hashicorp"
|
||||
"github.com/webinstall/webi-installers/internal/releases/iterm2dist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/juliadist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/mariadbdist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/nodedist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/zigdist"
|
||||
)
|
||||
|
||||
func main() {
|
||||
cacheDir := flag.String("cache", "_cache/raw", "root directory for raw cache")
|
||||
confDir := flag.String("conf", ".", "root directory containing {pkg}/releases.conf files")
|
||||
token := flag.String("token", os.Getenv("GITHUB_TOKEN"), "GitHub API token")
|
||||
flag.Parse()
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
client := &http.Client{Timeout: 30 * time.Second}
|
||||
var auth *githubish.Auth
|
||||
if *token != "" {
|
||||
auth = &githubish.Auth{Token: *token}
|
||||
}
|
||||
|
||||
// Discover packages from releases.conf files.
|
||||
packages, err := discover(*confDir)
|
||||
if err != nil {
|
||||
log.Fatalf("discover: %v", err)
|
||||
}
|
||||
|
||||
// Filter to requested packages if args given.
|
||||
args := flag.Args()
|
||||
if len(args) > 0 {
|
||||
nameSet := make(map[string]bool, len(args))
|
||||
for _, a := range args {
|
||||
nameSet[a] = true
|
||||
}
|
||||
var filtered []pkgConf
|
||||
for _, p := range packages {
|
||||
if nameSet[p.name] {
|
||||
filtered = append(filtered, p)
|
||||
}
|
||||
}
|
||||
packages = filtered
|
||||
}
|
||||
|
||||
log.Printf("found %d packages", len(packages))
|
||||
|
||||
for _, pkg := range packages {
|
||||
// Aliases share cache with their target — skip fetching.
|
||||
if alias := pkg.conf.Extra["alias_of"]; alias != "" {
|
||||
log.Printf(" %s: alias of %s, skipping", pkg.name, alias)
|
||||
continue
|
||||
}
|
||||
|
||||
log.Printf("fetching %s...", pkg.name)
|
||||
var err error
|
||||
switch pkg.conf.Source {
|
||||
case "github":
|
||||
err = fetchGitHub(ctx, client, *cacheDir, pkg.name, pkg.conf, auth)
|
||||
case "nodedist":
|
||||
err = fetchNodeDist(ctx, client, *cacheDir, pkg.name, pkg.conf)
|
||||
case "golang":
|
||||
err = fetchGolang(ctx, client, *cacheDir, pkg.name)
|
||||
case "zigdist":
|
||||
err = fetchZig(ctx, client, *cacheDir, pkg.name)
|
||||
case "flutterdist":
|
||||
err = fetchFlutter(ctx, client, *cacheDir, pkg.name)
|
||||
case "iterm2dist":
|
||||
err = fetchITerm2(ctx, client, *cacheDir, pkg.name)
|
||||
case "hashicorp":
|
||||
err = fetchHashiCorp(ctx, client, *cacheDir, pkg.name, pkg.conf)
|
||||
case "juliadist":
|
||||
err = fetchJulia(ctx, client, *cacheDir, pkg.name)
|
||||
case "gittag":
|
||||
err = fetchGitTag(ctx, *cacheDir, pkg.name, pkg.conf)
|
||||
case "gitea":
|
||||
err = fetchGitea(ctx, client, *cacheDir, pkg.name, pkg.conf)
|
||||
case "chromedist":
|
||||
err = fetchChrome(ctx, client, *cacheDir, pkg.name)
|
||||
case "gpgdist":
|
||||
err = fetchGPG(ctx, client, *cacheDir, pkg.name)
|
||||
case "mariadbdist":
|
||||
err = fetchMariaDB(ctx, client, *cacheDir, pkg.name)
|
||||
default:
|
||||
log.Printf(" %s: unknown source %q, skipping", pkg.name, pkg.conf.Source)
|
||||
continue
|
||||
}
|
||||
if err != nil {
|
||||
log.Printf(" ERROR: %s: %v", pkg.name, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
type pkgConf struct {
|
||||
name string
|
||||
conf *installerconf.Conf
|
||||
}
|
||||
|
||||
// discover finds all {dir}/*/releases.conf files and returns them sorted.
|
||||
func discover(dir string) ([]pkgConf, error) {
|
||||
pattern := filepath.Join(dir, "*", "releases.conf")
|
||||
matches, err := filepath.Glob(pattern)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var packages []pkgConf
|
||||
for _, path := range matches {
|
||||
name := filepath.Base(filepath.Dir(path))
|
||||
// Skip infrastructure dirs (_example, _webi, _common, etc.)
|
||||
if strings.HasPrefix(name, "_") {
|
||||
continue
|
||||
}
|
||||
conf, err := installerconf.Read(path)
|
||||
if err != nil {
|
||||
log.Printf("warning: %s: %v", path, err)
|
||||
continue
|
||||
}
|
||||
packages = append(packages, pkgConf{name: name, conf: conf})
|
||||
}
|
||||
|
||||
sort.Slice(packages, func(i, j int) bool {
|
||||
return packages[i].name < packages[j].name
|
||||
})
|
||||
|
||||
return packages, nil
|
||||
}
|
||||
|
||||
func fetchNodeDist(ctx context.Context, client *http.Client, cacheRoot, pkgName string, conf *installerconf.Conf) error {
|
||||
baseURL := conf.BaseURL
|
||||
if baseURL == "" {
|
||||
return fmt.Errorf("missing url in releases.conf")
|
||||
}
|
||||
|
||||
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var added, changed, skipped int
|
||||
var latest string
|
||||
for batch, err := range nodedist.Fetch(ctx, client, baseURL) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s fetch: %w", pkgName, err)
|
||||
}
|
||||
for _, entry := range batch {
|
||||
tag := entry.Version
|
||||
data, err := json.Marshal(entry)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%s marshal %s: %w", pkgName, tag, err)
|
||||
}
|
||||
|
||||
action, err := d.Merge(tag, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch action {
|
||||
case "added":
|
||||
added++
|
||||
case "changed":
|
||||
changed++
|
||||
default:
|
||||
skipped++
|
||||
}
|
||||
|
||||
if latest == "" {
|
||||
latest = tag
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := updateLatest(d, latest); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchGitHub(ctx context.Context, client *http.Client, cacheRoot, pkgName string, conf *installerconf.Conf, auth *githubish.Auth) error {
|
||||
owner := conf.Owner
|
||||
repo := conf.Repo
|
||||
tagPrefix := conf.TagPrefix
|
||||
|
||||
if owner == "" || repo == "" {
|
||||
return fmt.Errorf("missing owner or repo in releases.conf")
|
||||
}
|
||||
|
||||
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var added, changed, skipped int
|
||||
var latest string
|
||||
for batch, err := range github.Fetch(ctx, client, owner, repo, auth) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("github %s/%s: %w", owner, repo, err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
if rel.Draft {
|
||||
continue
|
||||
}
|
||||
|
||||
tag := rel.TagName
|
||||
|
||||
if tagPrefix != "" {
|
||||
if !strings.HasPrefix(tag, tagPrefix) {
|
||||
continue
|
||||
}
|
||||
tag = strings.TrimPrefix(tag, tagPrefix)
|
||||
}
|
||||
|
||||
data, err := json.Marshal(rel)
|
||||
if err != nil {
|
||||
return fmt.Errorf("marshal %s: %w", tag, err)
|
||||
}
|
||||
|
||||
action, err := d.Merge(tag, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch action {
|
||||
case "added":
|
||||
added++
|
||||
case "changed":
|
||||
changed++
|
||||
default:
|
||||
skipped++
|
||||
}
|
||||
|
||||
if latest == "" && !rel.Prerelease {
|
||||
latest = tag
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := updateLatest(d, latest); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
|
||||
return nil
|
||||
}
|
||||
|
||||
func updateLatest(d *rawcache.Dir, candidate string) error {
|
||||
if candidate == "" {
|
||||
return nil
|
||||
}
|
||||
current := d.Latest()
|
||||
if current == "" || lexver.Compare(lexver.Parse(candidate), lexver.Parse(current)) > 0 {
|
||||
return d.SetLatest(candidate)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchGolang(ctx context.Context, client *http.Client, cacheRoot, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var added, changed, skipped int
|
||||
var latest string
|
||||
for batch, err := range golang.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("golang: %w", err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
tag := rel.Version // "go1.24.1"
|
||||
data, err := json.Marshal(rel)
|
||||
if err != nil {
|
||||
return fmt.Errorf("golang marshal %s: %w", tag, err)
|
||||
}
|
||||
|
||||
action, err := d.Merge(tag, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch action {
|
||||
case "added":
|
||||
added++
|
||||
case "changed":
|
||||
changed++
|
||||
default:
|
||||
skipped++
|
||||
}
|
||||
|
||||
if latest == "" && rel.Stable {
|
||||
latest = tag
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := updateLatest(d, latest); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchZig(ctx context.Context, client *http.Client, cacheRoot, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var added, changed, skipped int
|
||||
var latest string
|
||||
for batch, err := range zigdist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("zigdist: %w", err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
tag := rel.Version
|
||||
data, err := json.Marshal(rel)
|
||||
if err != nil {
|
||||
return fmt.Errorf("zigdist marshal %s: %w", tag, err)
|
||||
}
|
||||
|
||||
action, err := d.Merge(tag, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch action {
|
||||
case "added":
|
||||
added++
|
||||
case "changed":
|
||||
changed++
|
||||
default:
|
||||
skipped++
|
||||
}
|
||||
|
||||
// Stable versions have dots and no dev/pre markers.
|
||||
isStable := strings.Contains(tag, ".") && !strings.ContainsAny(tag, "+-")
|
||||
if isStable {
|
||||
if latest == "" || lexver.Compare(lexver.Parse(tag), lexver.Parse(latest)) > 0 {
|
||||
latest = tag
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := updateLatest(d, latest); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchFlutter(ctx context.Context, client *http.Client, cacheRoot, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var added, changed, skipped int
|
||||
var latest string
|
||||
for batch, err := range flutterdist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("flutterdist: %w", err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
// Use version+channel+os+arch as the tag. The arch is embedded
|
||||
// in the archive path (e.g. flutter_macos_arm64_3.0.0-stable.zip
|
||||
// vs flutter_macos_3.0.0-stable.zip for universal/x64).
|
||||
arch := ""
|
||||
base := filepath.Base(rel.Archive)
|
||||
prefix := "flutter_" + rel.OS + "_"
|
||||
if after, ok := strings.CutPrefix(base, prefix); ok {
|
||||
if !strings.HasPrefix(after, rel.Version) {
|
||||
// There's an arch segment between OS and version.
|
||||
if idx := strings.Index(after, "_"); idx > 0 {
|
||||
arch = after[:idx]
|
||||
}
|
||||
}
|
||||
}
|
||||
tag := fmt.Sprintf("%s-%s-%s", rel.Version, rel.Channel, rel.OS)
|
||||
if arch != "" {
|
||||
tag += "-" + arch
|
||||
}
|
||||
data, err := json.Marshal(rel)
|
||||
if err != nil {
|
||||
return fmt.Errorf("flutterdist marshal %s: %w", tag, err)
|
||||
}
|
||||
|
||||
action, err := d.Merge(tag, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch action {
|
||||
case "added":
|
||||
added++
|
||||
case "changed":
|
||||
changed++
|
||||
default:
|
||||
skipped++
|
||||
}
|
||||
|
||||
if latest == "" && rel.Channel == "stable" {
|
||||
latest = tag
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := updateLatest(d, latest); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchITerm2(ctx context.Context, client *http.Client, cacheRoot, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var added, changed, skipped int
|
||||
var latest string
|
||||
for batch, err := range iterm2dist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("iterm2dist: %w", err)
|
||||
}
|
||||
for _, entry := range batch {
|
||||
tag := entry.Version
|
||||
if tag == "" {
|
||||
continue
|
||||
}
|
||||
data, err := json.Marshal(entry)
|
||||
if err != nil {
|
||||
return fmt.Errorf("iterm2dist marshal %s: %w", tag, err)
|
||||
}
|
||||
|
||||
action, err := d.Merge(tag, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch action {
|
||||
case "added":
|
||||
added++
|
||||
case "changed":
|
||||
changed++
|
||||
default:
|
||||
skipped++
|
||||
}
|
||||
|
||||
if latest == "" && entry.Channel == "stable" {
|
||||
latest = tag
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := updateLatest(d, latest); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchHashiCorp(ctx context.Context, client *http.Client, cacheRoot, pkgName string, conf *installerconf.Conf) error {
|
||||
product := conf.Extra["product"]
|
||||
if product == "" {
|
||||
return fmt.Errorf("missing product in releases.conf")
|
||||
}
|
||||
|
||||
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var added, changed, skipped int
|
||||
var latest string
|
||||
for idx, err := range hashicorp.Fetch(ctx, client, product) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("hashicorp %s: %w", product, err)
|
||||
}
|
||||
for tag, ver := range idx.Versions {
|
||||
data, err := json.Marshal(ver)
|
||||
if err != nil {
|
||||
return fmt.Errorf("hashicorp marshal %s: %w", tag, err)
|
||||
}
|
||||
|
||||
action, err := d.Merge(tag, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch action {
|
||||
case "added":
|
||||
added++
|
||||
case "changed":
|
||||
changed++
|
||||
default:
|
||||
skipped++
|
||||
}
|
||||
|
||||
// Stable = no prerelease markers. Compare all to find highest.
|
||||
isStable := !strings.ContainsAny(tag, "-+")
|
||||
if isStable {
|
||||
if latest == "" || lexver.Compare(lexver.Parse(tag), lexver.Parse(latest)) > 0 {
|
||||
latest = tag
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := updateLatest(d, latest); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchJulia(ctx context.Context, client *http.Client, cacheRoot, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var added, changed, skipped int
|
||||
var latest string
|
||||
for batch, err := range juliadist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("juliadist: %w", err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
tag := rel.Version
|
||||
data, err := json.Marshal(rel)
|
||||
if err != nil {
|
||||
return fmt.Errorf("juliadist marshal %s: %w", tag, err)
|
||||
}
|
||||
|
||||
action, err := d.Merge(tag, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch action {
|
||||
case "added":
|
||||
added++
|
||||
case "changed":
|
||||
changed++
|
||||
default:
|
||||
skipped++
|
||||
}
|
||||
|
||||
if rel.Stable {
|
||||
if latest == "" || lexver.Compare(lexver.Parse(tag), lexver.Parse(latest)) > 0 {
|
||||
latest = tag
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := updateLatest(d, latest); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchGitTag(ctx context.Context, cacheRoot, pkgName string, conf *installerconf.Conf) error {
|
||||
gitURL := conf.BaseURL
|
||||
if gitURL == "" {
|
||||
return fmt.Errorf("missing url in releases.conf")
|
||||
}
|
||||
|
||||
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
repoDir := filepath.Join(cacheRoot, "_repos")
|
||||
if err := os.MkdirAll(repoDir, 0o755); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var added, changed, skipped int
|
||||
var latest string
|
||||
for batch, err := range gittag.Fetch(ctx, gitURL, repoDir) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("gittag %s: %w", pkgName, err)
|
||||
}
|
||||
for _, entry := range batch {
|
||||
tag := entry.Version
|
||||
if tag == "" {
|
||||
tag = "HEAD-" + entry.CommitHash
|
||||
}
|
||||
data, err := json.Marshal(entry)
|
||||
if err != nil {
|
||||
return fmt.Errorf("gittag marshal %s: %w", tag, err)
|
||||
}
|
||||
|
||||
action, err := d.Merge(tag, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch action {
|
||||
case "added":
|
||||
added++
|
||||
case "changed":
|
||||
changed++
|
||||
default:
|
||||
skipped++
|
||||
}
|
||||
|
||||
if entry.GitTag != "" && entry.GitTag != "HEAD" {
|
||||
if latest == "" || lexver.Compare(lexver.Parse(tag), lexver.Parse(latest)) > 0 {
|
||||
latest = tag
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := updateLatest(d, latest); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchGitea(ctx context.Context, client *http.Client, cacheRoot, pkgName string, conf *installerconf.Conf) error {
|
||||
baseURL := conf.BaseURL
|
||||
owner := conf.Owner
|
||||
repo := conf.Repo
|
||||
|
||||
if baseURL == "" || owner == "" || repo == "" {
|
||||
return fmt.Errorf("missing base_url, owner, or repo in releases.conf")
|
||||
}
|
||||
|
||||
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var added, changed, skipped int
|
||||
var latest string
|
||||
for batch, err := range gitea.Fetch(ctx, client, baseURL, owner, repo, nil) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("gitea %s/%s: %w", owner, repo, err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
if rel.Draft {
|
||||
continue
|
||||
}
|
||||
|
||||
tag := rel.TagName
|
||||
data, err := json.Marshal(rel)
|
||||
if err != nil {
|
||||
return fmt.Errorf("gitea marshal %s: %w", tag, err)
|
||||
}
|
||||
|
||||
action, err := d.Merge(tag, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch action {
|
||||
case "added":
|
||||
added++
|
||||
case "changed":
|
||||
changed++
|
||||
default:
|
||||
skipped++
|
||||
}
|
||||
|
||||
if latest == "" && !rel.Prerelease {
|
||||
latest = tag
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := updateLatest(d, latest); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchChrome(ctx context.Context, client *http.Client, cacheRoot, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var added, changed, skipped int
|
||||
var latest string
|
||||
for batch, err := range chromedist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("chromedist: %w", err)
|
||||
}
|
||||
for _, ver := range batch {
|
||||
tag := ver.Version
|
||||
data, err := json.Marshal(ver)
|
||||
if err != nil {
|
||||
return fmt.Errorf("chromedist marshal %s: %w", tag, err)
|
||||
}
|
||||
|
||||
action, err := d.Merge(tag, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch action {
|
||||
case "added":
|
||||
added++
|
||||
case "changed":
|
||||
changed++
|
||||
default:
|
||||
skipped++
|
||||
}
|
||||
|
||||
if latest == "" || lexver.Compare(lexver.Parse(tag), lexver.Parse(latest)) > 0 {
|
||||
latest = tag
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := updateLatest(d, latest); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchGPG(ctx context.Context, client *http.Client, cacheRoot, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var added, changed, skipped int
|
||||
var latest string
|
||||
for batch, err := range gpgdist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("gpgdist: %w", err)
|
||||
}
|
||||
for _, entry := range batch {
|
||||
tag := entry.Version
|
||||
data, err := json.Marshal(entry)
|
||||
if err != nil {
|
||||
return fmt.Errorf("gpgdist marshal %s: %w", tag, err)
|
||||
}
|
||||
|
||||
action, err := d.Merge(tag, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch action {
|
||||
case "added":
|
||||
added++
|
||||
case "changed":
|
||||
changed++
|
||||
default:
|
||||
skipped++
|
||||
}
|
||||
|
||||
if latest == "" || lexver.Compare(lexver.Parse(tag), lexver.Parse(latest)) > 0 {
|
||||
latest = tag
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := updateLatest(d, latest); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchMariaDB(ctx context.Context, client *http.Client, cacheRoot, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(cacheRoot, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var added, changed, skipped int
|
||||
var latest string
|
||||
for batch, err := range mariadbdist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("mariadbdist: %w", err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
tag := rel.ReleaseID
|
||||
data, err := json.Marshal(rel)
|
||||
if err != nil {
|
||||
return fmt.Errorf("mariadbdist marshal %s: %w", tag, err)
|
||||
}
|
||||
|
||||
action, err := d.Merge(tag, data)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
switch action {
|
||||
case "added":
|
||||
added++
|
||||
case "changed":
|
||||
changed++
|
||||
default:
|
||||
skipped++
|
||||
}
|
||||
|
||||
isStable := rel.MajorStatus == "Stable"
|
||||
if isStable {
|
||||
if latest == "" || lexver.Compare(lexver.Parse(tag), lexver.Parse(latest)) > 0 {
|
||||
latest = tag
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if err := updateLatest(d, latest); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
log.Printf(" %s: +%d ~%d =%d latest=%s", pkgName, added, changed, skipped, d.Latest())
|
||||
return nil
|
||||
}
|
||||
625
cmd/inspect/main.go
Normal file
625
cmd/inspect/main.go
Normal file
@@ -0,0 +1,625 @@
|
||||
// Command inspect downloads release archives, unpacks them, and reports
|
||||
// their internal structure. This helps discover how packages are laid out
|
||||
// and whether the layout changes across versions.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// go run ./cmd/inspect -csv distributables.csv -cache ./_cache/downloads ollama sd
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/csv"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"mime"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/httpclient"
|
||||
)
|
||||
|
||||
// Row is one CSV row from distributables.csv.
|
||||
type Row struct {
|
||||
Package string
|
||||
Version string
|
||||
Channel string
|
||||
Date string
|
||||
OS string
|
||||
Arch string
|
||||
Libc string
|
||||
Format string
|
||||
Download string
|
||||
Filename string
|
||||
Extra string
|
||||
}
|
||||
|
||||
// archiveFormats are the formats we download and unpack.
|
||||
var archiveFormats = map[string]bool{
|
||||
".tar.gz": true,
|
||||
".tar.xz": true,
|
||||
".tar.bz2": true,
|
||||
".tar.zst": true,
|
||||
".zip": true,
|
||||
".dmg": true,
|
||||
".gz": true,
|
||||
".xz": true,
|
||||
}
|
||||
|
||||
// inspectOSes are the OSes we inspect.
|
||||
var inspectOSes = map[string]bool{
|
||||
"linux": true,
|
||||
"darwin": true,
|
||||
"windows": true,
|
||||
"": true, // source-only packages
|
||||
}
|
||||
|
||||
// preferredArch picks one arch per OS to download.
|
||||
func preferredArch(os_ string) string {
|
||||
switch os_ {
|
||||
case "darwin":
|
||||
return "aarch64"
|
||||
default:
|
||||
return "x86_64"
|
||||
}
|
||||
}
|
||||
|
||||
func main() {
|
||||
csvFile := flag.String("csv", "distributables.csv", "path to distributables CSV")
|
||||
cacheDir := flag.String("cache", "_cache/downloads", "download cache directory")
|
||||
flag.Parse()
|
||||
|
||||
packages := flag.Args()
|
||||
if len(packages) == 0 {
|
||||
log.Fatal("usage: inspect [-csv FILE] [-cache DIR] PACKAGE [PACKAGE...]")
|
||||
}
|
||||
|
||||
rows, err := readCSV(*csvFile)
|
||||
if err != nil {
|
||||
log.Fatalf("read csv: %v", err)
|
||||
}
|
||||
|
||||
client := httpclient.New()
|
||||
// Override timeout for large downloads.
|
||||
client.Timeout = 10 * time.Minute
|
||||
|
||||
for _, pkg := range packages {
|
||||
log.Printf("=== %s ===", pkg)
|
||||
if err := inspectPackage(client, rows, pkg, *cacheDir); err != nil {
|
||||
log.Printf("ERROR: %s: %v", pkg, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func readCSV(path string) ([]Row, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
r := csv.NewReader(f)
|
||||
header, err := r.Read()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Build column index.
|
||||
idx := make(map[string]int, len(header))
|
||||
for i, col := range header {
|
||||
idx[col] = i
|
||||
}
|
||||
|
||||
var rows []Row
|
||||
for {
|
||||
record, err := r.Read()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
get := func(col string) string {
|
||||
if i, ok := idx[col]; ok && i < len(record) {
|
||||
return record[i]
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
rows = append(rows, Row{
|
||||
Package: get("package"),
|
||||
Version: get("version"),
|
||||
Channel: get("channel"),
|
||||
Date: get("date"),
|
||||
OS: get("os"),
|
||||
Arch: get("arch"),
|
||||
Libc: get("libc"),
|
||||
Format: get("format"),
|
||||
Download: get("download"),
|
||||
Filename: get("filename"),
|
||||
Extra: get("extra"),
|
||||
})
|
||||
}
|
||||
return rows, nil
|
||||
}
|
||||
|
||||
func inspectPackage(client *http.Client, allRows []Row, pkg, cacheDir string) error {
|
||||
// Filter rows for this package.
|
||||
var pkgRows []Row
|
||||
for _, r := range allRows {
|
||||
if r.Package == pkg {
|
||||
pkgRows = append(pkgRows, r)
|
||||
}
|
||||
}
|
||||
if len(pkgRows) == 0 {
|
||||
return fmt.Errorf("no rows found")
|
||||
}
|
||||
|
||||
// Find latest stable version, fall back to any version.
|
||||
versions := findVersionsByDate(pkgRows)
|
||||
if len(versions) == 0 {
|
||||
return fmt.Errorf("no versions found")
|
||||
}
|
||||
|
||||
latestVer := versions[0]
|
||||
log.Printf(" latest version: %s", latestVer)
|
||||
|
||||
// Check if latest has assets uploaded (more than just source tarballs).
|
||||
latestRows := filterVersion(pkgRows, latestVer)
|
||||
hasRealAssets := false
|
||||
for _, r := range latestRows {
|
||||
if r.Extra != "source" && archiveFormats[r.Format] {
|
||||
hasRealAssets = true
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// If latest looks empty, step back one version.
|
||||
if !hasRealAssets && len(versions) > 1 {
|
||||
latestVer = versions[1]
|
||||
latestRows = filterVersion(pkgRows, latestVer)
|
||||
log.Printf(" latest has no assets, using: %s", latestVer)
|
||||
}
|
||||
|
||||
// Inspect the latest version.
|
||||
if err := inspectVersion(client, pkg, latestVer, latestRows, cacheDir); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Find versions roughly a year apart going back.
|
||||
yearVersions := findYearlyVersions(pkgRows, latestVer)
|
||||
for _, v := range yearVersions {
|
||||
log.Printf(" --- checking %s ---", v)
|
||||
vRows := filterVersion(pkgRows, v)
|
||||
if err := inspectVersion(client, pkg, v, vRows, cacheDir); err != nil {
|
||||
log.Printf(" ERROR: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// findVersionsByDate returns versions sorted newest first, preferring stable.
|
||||
func findVersionsByDate(rows []Row) []string {
|
||||
type vInfo struct {
|
||||
version string
|
||||
date string
|
||||
stable bool
|
||||
}
|
||||
seen := map[string]*vInfo{}
|
||||
for _, r := range rows {
|
||||
if _, ok := seen[r.Version]; !ok {
|
||||
seen[r.Version] = &vInfo{
|
||||
version: r.Version,
|
||||
date: r.Date,
|
||||
stable: r.Channel == "stable",
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var vs []*vInfo
|
||||
for _, v := range seen {
|
||||
vs = append(vs, v)
|
||||
}
|
||||
|
||||
// Sort: stable first, then by date descending, then version descending.
|
||||
sort.Slice(vs, func(i, j int) bool {
|
||||
if vs[i].stable != vs[j].stable {
|
||||
return vs[i].stable
|
||||
}
|
||||
if vs[i].date != vs[j].date {
|
||||
return vs[i].date > vs[j].date
|
||||
}
|
||||
return vs[i].version > vs[j].version
|
||||
})
|
||||
|
||||
result := make([]string, len(vs))
|
||||
for i, v := range vs {
|
||||
result[i] = v.version
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// findYearlyVersions picks versions roughly a year apart before the given version.
|
||||
func findYearlyVersions(rows []Row, latestVer string) []string {
|
||||
// Find the date of latest.
|
||||
var latestDate string
|
||||
for _, r := range rows {
|
||||
if r.Version == latestVer && r.Date != "" {
|
||||
latestDate = r.Date
|
||||
break
|
||||
}
|
||||
}
|
||||
if latestDate == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
latestTime, err := time.Parse("2006-01-02", latestDate)
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Collect all stable versions with dates.
|
||||
type vd struct {
|
||||
version string
|
||||
date time.Time
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
var all []vd
|
||||
for _, r := range rows {
|
||||
if seen[r.Version] || r.Date == "" || r.Channel != "stable" {
|
||||
continue
|
||||
}
|
||||
seen[r.Version] = true
|
||||
t, err := time.Parse("2006-01-02", r.Date)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if t.Before(latestTime) {
|
||||
all = append(all, vd{r.Version, t})
|
||||
}
|
||||
}
|
||||
|
||||
sort.Slice(all, func(i, j int) bool {
|
||||
return all[i].date.After(all[j].date)
|
||||
})
|
||||
|
||||
// Pick versions roughly a year apart.
|
||||
var result []string
|
||||
nextTarget := latestTime.AddDate(-1, 0, 0)
|
||||
for _, v := range all {
|
||||
if v.date.Before(nextTarget) || v.date.Equal(nextTarget) {
|
||||
result = append(result, v.version)
|
||||
nextTarget = v.date.AddDate(-1, 0, 0)
|
||||
}
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
func filterVersion(rows []Row, version string) []Row {
|
||||
var result []Row
|
||||
for _, r := range rows {
|
||||
if r.Version == version {
|
||||
result = append(result, r)
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// inspectVersion downloads and inspects archives for one version.
|
||||
func inspectVersion(client *http.Client, pkg, version string, rows []Row, cacheDir string) error {
|
||||
// Group by OS, pick one arch per OS, pick distinct formats.
|
||||
type dlKey struct {
|
||||
os_ string
|
||||
format string
|
||||
}
|
||||
selected := map[dlKey]*Row{}
|
||||
|
||||
for i := range rows {
|
||||
r := &rows[i]
|
||||
if !inspectOSes[r.OS] {
|
||||
continue
|
||||
}
|
||||
if !archiveFormats[r.Format] {
|
||||
continue
|
||||
}
|
||||
|
||||
key := dlKey{r.OS, r.Format}
|
||||
existing := selected[key]
|
||||
if existing == nil {
|
||||
selected[key] = r
|
||||
continue
|
||||
}
|
||||
|
||||
// Prefer the preferred arch.
|
||||
pref := preferredArch(r.OS)
|
||||
if r.Arch == pref && existing.Arch != pref {
|
||||
selected[key] = r
|
||||
}
|
||||
// Skip rocm/jetpack variants.
|
||||
if strings.Contains(r.Filename, "rocm") || strings.Contains(r.Filename, "jetpack") {
|
||||
if !strings.Contains(existing.Filename, "rocm") && !strings.Contains(existing.Filename, "jetpack") {
|
||||
continue // keep existing non-special variant
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if len(selected) == 0 {
|
||||
log.Printf(" %s: no downloadable archives", version)
|
||||
return nil
|
||||
}
|
||||
|
||||
// Sort keys for deterministic output.
|
||||
var keys []dlKey
|
||||
for k := range selected {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Slice(keys, func(i, j int) bool {
|
||||
if keys[i].os_ != keys[j].os_ {
|
||||
return keys[i].os_ < keys[j].os_
|
||||
}
|
||||
return keys[i].format < keys[j].format
|
||||
})
|
||||
|
||||
for _, key := range keys {
|
||||
r := selected[key]
|
||||
os_ := r.OS
|
||||
if os_ == "" {
|
||||
os_ = "any"
|
||||
}
|
||||
log.Printf(" [%s] %s %s → %s", version, os_, r.Format, r.Filename)
|
||||
|
||||
dlPath, err := download(client, r.Download, r.Filename, filepath.Join(cacheDir, pkg, version))
|
||||
if err != nil {
|
||||
log.Printf(" download error: %v", err)
|
||||
continue
|
||||
}
|
||||
|
||||
contents, err := unpackAndList(dlPath, r.Format)
|
||||
if err != nil {
|
||||
log.Printf(" unpack error: %v", err)
|
||||
continue
|
||||
}
|
||||
|
||||
printContents(contents)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// download fetches a URL to the cache dir. Returns the path to the cached file.
|
||||
// Skips download if the file already exists.
|
||||
func download(client *http.Client, url, hintFilename, dir string) (string, error) {
|
||||
// Check if already cached by hint filename.
|
||||
cached := filepath.Join(dir, hintFilename)
|
||||
if _, err := os.Stat(cached); err == nil {
|
||||
return cached, nil
|
||||
}
|
||||
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
resp, err := httpclient.Get(ctx, client, url)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("GET %s: %w", url, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return "", fmt.Errorf("GET %s: %s", url, resp.Status)
|
||||
}
|
||||
|
||||
// Determine filename from Content-Disposition or hint.
|
||||
filename := hintFilename
|
||||
if cd := resp.Header.Get("Content-Disposition"); cd != "" {
|
||||
_, params, err := mime.ParseMediaType(cd)
|
||||
if err == nil {
|
||||
if fn, ok := params["filename"]; ok && fn != "" {
|
||||
filename = fn
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
outPath := filepath.Join(dir, filename)
|
||||
|
||||
// Atomic write: temp file + rename.
|
||||
tmp := outPath + ".tmp"
|
||||
f, err := os.Create(tmp)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
n, err := io.Copy(f, resp.Body)
|
||||
if closeErr := f.Close(); closeErr != nil && err == nil {
|
||||
err = closeErr
|
||||
}
|
||||
if err != nil {
|
||||
os.Remove(tmp)
|
||||
return "", fmt.Errorf("download %s: %w", url, err)
|
||||
}
|
||||
|
||||
if err := os.Rename(tmp, outPath); err != nil {
|
||||
os.Remove(tmp)
|
||||
return "", err
|
||||
}
|
||||
|
||||
log.Printf(" downloaded %s (%d bytes)", filename, n)
|
||||
return outPath, nil
|
||||
}
|
||||
|
||||
// FileEntry describes one file inside an archive.
|
||||
type FileEntry struct {
|
||||
Path string
|
||||
Size int64
|
||||
Mode os.FileMode
|
||||
IsDir bool
|
||||
IsExec bool
|
||||
IsSymlink bool
|
||||
LinkTarget string
|
||||
}
|
||||
|
||||
// unpackAndList extracts an archive to a temp dir and lists contents.
|
||||
func unpackAndList(archivePath, format string) ([]FileEntry, error) {
|
||||
tmpDir, err := os.MkdirTemp("", "webi-inspect-*")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer os.RemoveAll(tmpDir)
|
||||
|
||||
switch format {
|
||||
case ".tar.gz":
|
||||
err = run("tar", "xzf", archivePath, "-C", tmpDir)
|
||||
case ".tar.xz":
|
||||
err = run("tar", "xJf", archivePath, "-C", tmpDir)
|
||||
case ".tar.bz2":
|
||||
err = run("tar", "xjf", archivePath, "-C", tmpDir)
|
||||
case ".tar.zst":
|
||||
err = run("tar", "--zstd", "-xf", archivePath, "-C", tmpDir)
|
||||
case ".zip":
|
||||
err = run("unzip", "-q", "-o", archivePath, "-d", tmpDir)
|
||||
case ".dmg":
|
||||
err = extractDMG(archivePath, tmpDir)
|
||||
case ".gz":
|
||||
// Single file gzip.
|
||||
base := filepath.Base(archivePath)
|
||||
base = strings.TrimSuffix(base, ".gz")
|
||||
outPath := filepath.Join(tmpDir, base)
|
||||
err = run("sh", "-c", fmt.Sprintf("gunzip -c %q > %q", archivePath, outPath))
|
||||
case ".xz":
|
||||
base := filepath.Base(archivePath)
|
||||
base = strings.TrimSuffix(base, ".xz")
|
||||
outPath := filepath.Join(tmpDir, base)
|
||||
err = run("sh", "-c", fmt.Sprintf("xz -dc %q > %q", archivePath, outPath))
|
||||
default:
|
||||
return nil, fmt.Errorf("unsupported format: %s", format)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("extract %s: %w", format, err)
|
||||
}
|
||||
|
||||
return listDir(tmpDir, "")
|
||||
}
|
||||
|
||||
func extractDMG(dmgPath, outDir string) error {
|
||||
// Try 7z first (doesn't require mounting).
|
||||
if _, err := exec.LookPath("7z"); err == nil {
|
||||
return run("7z", "x", "-o"+outDir, dmgPath)
|
||||
}
|
||||
|
||||
// Fall back to hdiutil mount + copy + unmount.
|
||||
mountPoint, err := os.MkdirTemp("", "webi-dmg-*")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer os.RemoveAll(mountPoint)
|
||||
|
||||
if err := run("hdiutil", "attach", dmgPath, "-mountpoint", mountPoint, "-nobrowse", "-quiet"); err != nil {
|
||||
return fmt.Errorf("mount dmg: %w", err)
|
||||
}
|
||||
defer run("hdiutil", "detach", mountPoint, "-quiet")
|
||||
|
||||
// Copy contents.
|
||||
return run("cp", "-R", mountPoint+"/.", outDir)
|
||||
}
|
||||
|
||||
func run(name string, args ...string) error {
|
||||
cmd := exec.Command(name, args...)
|
||||
cmd.Stderr = os.Stderr
|
||||
return cmd.Run()
|
||||
}
|
||||
|
||||
func listDir(root, prefix string) ([]FileEntry, error) {
|
||||
entries, err := os.ReadDir(filepath.Join(root, prefix))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var result []FileEntry
|
||||
for _, e := range entries {
|
||||
relPath := filepath.Join(prefix, e.Name())
|
||||
fullPath := filepath.Join(root, relPath)
|
||||
|
||||
info, err := e.Info()
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
|
||||
entry := FileEntry{
|
||||
Path: relPath,
|
||||
Size: info.Size(),
|
||||
Mode: info.Mode(),
|
||||
IsDir: e.IsDir(),
|
||||
}
|
||||
|
||||
if info.Mode()&os.ModeSymlink != 0 {
|
||||
entry.IsSymlink = true
|
||||
target, _ := os.Readlink(fullPath)
|
||||
entry.LinkTarget = target
|
||||
}
|
||||
|
||||
if !e.IsDir() && info.Mode()&0o111 != 0 {
|
||||
entry.IsExec = true
|
||||
}
|
||||
|
||||
result = append(result, entry)
|
||||
|
||||
if e.IsDir() {
|
||||
sub, err := listDir(root, relPath)
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
result = append(result, sub...)
|
||||
}
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func printContents(entries []FileEntry) {
|
||||
for _, e := range entries {
|
||||
marker := " "
|
||||
if e.IsExec {
|
||||
marker = "* "
|
||||
}
|
||||
if e.IsDir {
|
||||
marker = "d "
|
||||
}
|
||||
if e.IsSymlink {
|
||||
marker = "→ "
|
||||
}
|
||||
|
||||
size := ""
|
||||
if !e.IsDir {
|
||||
size = formatSize(e.Size)
|
||||
}
|
||||
|
||||
line := fmt.Sprintf(" %s%-50s %8s", marker, e.Path, size)
|
||||
if e.IsSymlink {
|
||||
line += " → " + e.LinkTarget
|
||||
}
|
||||
log.Print(line)
|
||||
}
|
||||
}
|
||||
|
||||
func formatSize(n int64) string {
|
||||
switch {
|
||||
case n >= 1<<30:
|
||||
return fmt.Sprintf("%.1fG", float64(n)/float64(1<<30))
|
||||
case n >= 1<<20:
|
||||
return fmt.Sprintf("%.1fM", float64(n)/float64(1<<20))
|
||||
case n >= 1<<10:
|
||||
return fmt.Sprintf("%.1fK", float64(n)/float64(1<<10))
|
||||
default:
|
||||
return fmt.Sprintf("%dB", n)
|
||||
}
|
||||
}
|
||||
356
cmd/uaparse/main.go
Normal file
356
cmd/uaparse/main.go
Normal file
@@ -0,0 +1,356 @@
|
||||
// Command uaparse analyzes User-Agent strings from webi.sh logs.
|
||||
//
|
||||
// It reads UA strings (one per line) from stdin or a file, parses each
|
||||
// through uadetect, and produces summary output showing:
|
||||
// - unique platform tuples (os, arch, libc) with counts
|
||||
// - platform hints extracted from kernel version strings (cloud provider,
|
||||
// container runtime, device info)
|
||||
// - detection failures and malformed UAs
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// uaparse < LIVE-UAS.txt
|
||||
// uaparse LIVE-UAS.txt
|
||||
// uaparse -json LIVE-UAS.txt
|
||||
// uaparse -fixtures LIVE-UAS.txt # output Go test fixtures
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/uadetect"
|
||||
)
|
||||
|
||||
// PlatformKey is the resolution-relevant tuple — everything else is noise
|
||||
// for artifact selection.
|
||||
type PlatformKey struct {
|
||||
OS string `json:"os"`
|
||||
Arch string `json:"arch"`
|
||||
Libc string `json:"libc"`
|
||||
}
|
||||
|
||||
func (k PlatformKey) String() string {
|
||||
return fmt.Sprintf("%-10s %-10s %s", k.OS, k.Arch, k.Libc)
|
||||
}
|
||||
|
||||
// PlatformEntry holds a unique platform and its metadata.
|
||||
type PlatformEntry struct {
|
||||
Key PlatformKey `json:"key"`
|
||||
Count int `json:"count"`
|
||||
Examples []string `json:"examples"` // up to 3 representative UAs
|
||||
Hints []string `json:"hints"` // unique platform hints seen
|
||||
}
|
||||
|
||||
// UAIssue records a malformed or undetectable UA.
|
||||
type UAIssue struct {
|
||||
Line int `json:"line"`
|
||||
UA string `json:"ua"`
|
||||
Reason string `json:"reason"`
|
||||
}
|
||||
|
||||
// Hint is a platform detail extracted from the kernel version string.
|
||||
type Hint struct {
|
||||
Tag string // short label: "amzn", "azure", "gcp", "wsl", etc.
|
||||
Count int
|
||||
}
|
||||
|
||||
func main() {
|
||||
jsonOut := flag.Bool("json", false, "output as JSON")
|
||||
fixtures := flag.Bool("fixtures", false, "output Go test fixture table")
|
||||
flag.Parse()
|
||||
|
||||
var scanner *bufio.Scanner
|
||||
if flag.NArg() > 0 {
|
||||
f, err := os.Open(flag.Arg(0))
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "uaparse: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
defer f.Close()
|
||||
scanner = bufio.NewScanner(f)
|
||||
} else {
|
||||
scanner = bufio.NewScanner(os.Stdin)
|
||||
}
|
||||
|
||||
platforms := make(map[PlatformKey]*PlatformEntry)
|
||||
hints := make(map[string]int)
|
||||
var issues []UAIssue
|
||||
lineNum := 0
|
||||
|
||||
for scanner.Scan() {
|
||||
lineNum++
|
||||
ua := strings.TrimSpace(scanner.Text())
|
||||
if ua == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
// Detect corruption: truncated/double-pasted lines.
|
||||
if isMalformed(ua) {
|
||||
issues = append(issues, UAIssue{
|
||||
Line: lineNum,
|
||||
UA: ua,
|
||||
Reason: "malformed (truncated or corrupted)",
|
||||
})
|
||||
continue
|
||||
}
|
||||
|
||||
// Parse through uadetect.
|
||||
result := uadetect.Parse(ua)
|
||||
|
||||
// Check for detection failures.
|
||||
if result.OS == "" {
|
||||
issues = append(issues, UAIssue{
|
||||
Line: lineNum,
|
||||
UA: ua,
|
||||
Reason: "OS not detected",
|
||||
})
|
||||
}
|
||||
if result.Arch == "" {
|
||||
issues = append(issues, UAIssue{
|
||||
Line: lineNum,
|
||||
UA: ua,
|
||||
Reason: "arch not detected",
|
||||
})
|
||||
}
|
||||
|
||||
key := PlatformKey{
|
||||
OS: string(result.OS),
|
||||
Arch: string(result.Arch),
|
||||
Libc: string(result.Libc),
|
||||
}
|
||||
|
||||
entry, ok := platforms[key]
|
||||
if !ok {
|
||||
entry = &PlatformEntry{Key: key}
|
||||
platforms[key] = entry
|
||||
}
|
||||
entry.Count++
|
||||
if len(entry.Examples) < 3 {
|
||||
entry.Examples = append(entry.Examples, ua)
|
||||
}
|
||||
|
||||
// Extract platform hints from kernel version.
|
||||
for _, h := range extractHints(ua) {
|
||||
if !containsStr(entry.Hints, h) {
|
||||
entry.Hints = append(entry.Hints, h)
|
||||
}
|
||||
hints[h]++
|
||||
}
|
||||
}
|
||||
|
||||
if err := scanner.Err(); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "uaparse: read error: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
// Sort platforms by count descending.
|
||||
entries := make([]*PlatformEntry, 0, len(platforms))
|
||||
for _, e := range platforms {
|
||||
entries = append(entries, e)
|
||||
}
|
||||
sort.Slice(entries, func(i, j int) bool {
|
||||
return entries[i].Count > entries[j].Count
|
||||
})
|
||||
|
||||
if *jsonOut {
|
||||
outputJSON(entries, issues, hints)
|
||||
} else if *fixtures {
|
||||
outputFixtures(entries)
|
||||
} else {
|
||||
outputTable(entries, issues, hints, lineNum)
|
||||
}
|
||||
}
|
||||
|
||||
func outputTable(entries []*PlatformEntry, issues []UAIssue, hints map[string]int, total int) {
|
||||
fmt.Printf("=== UA Analysis: %d lines → %d unique platforms ===\n\n", total, len(entries))
|
||||
|
||||
fmt.Printf("%-10s %-10s %-6s %6s %s\n", "OS", "ARCH", "LIBC", "COUNT", "HINTS")
|
||||
fmt.Println(strings.Repeat("-", 72))
|
||||
for _, e := range entries {
|
||||
hintStr := ""
|
||||
if len(e.Hints) > 0 {
|
||||
hintStr = strings.Join(e.Hints, ", ")
|
||||
}
|
||||
fmt.Printf("%-10s %-10s %-6s %6d %s\n",
|
||||
displayOS(e.Key.OS), e.Key.Arch, displayLibc(e.Key.Libc),
|
||||
e.Count, hintStr)
|
||||
}
|
||||
|
||||
if len(hints) > 0 {
|
||||
fmt.Printf("\n=== Platform Hints (environment signals from kernel strings) ===\n\n")
|
||||
sortedHints := make([]Hint, 0, len(hints))
|
||||
for tag, count := range hints {
|
||||
sortedHints = append(sortedHints, Hint{tag, count})
|
||||
}
|
||||
sort.Slice(sortedHints, func(i, j int) bool {
|
||||
return sortedHints[i].Count > sortedHints[j].Count
|
||||
})
|
||||
for _, h := range sortedHints {
|
||||
fmt.Printf(" %-20s %d\n", h.Tag, h.Count)
|
||||
}
|
||||
}
|
||||
|
||||
if len(issues) > 0 {
|
||||
fmt.Printf("\n=== Issues (%d) ===\n\n", len(issues))
|
||||
for _, iss := range issues {
|
||||
fmt.Printf(" line %d: %s\n %s\n", iss.Line, iss.Reason, iss.UA)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func outputJSON(entries []*PlatformEntry, issues []UAIssue, hints map[string]int) {
|
||||
out := struct {
|
||||
Platforms []*PlatformEntry `json:"platforms"`
|
||||
Issues []UAIssue `json:"issues"`
|
||||
Hints map[string]int `json:"hints"`
|
||||
}{entries, issues, hints}
|
||||
|
||||
enc := json.NewEncoder(os.Stdout)
|
||||
enc.SetIndent("", " ")
|
||||
_ = enc.Encode(out)
|
||||
}
|
||||
|
||||
func outputFixtures(entries []*PlatformEntry) {
|
||||
fmt.Println("// Generated by cmd/uaparse from live UA data.")
|
||||
fmt.Println("// Each entry represents a unique (os, arch, libc) platform seen in production.")
|
||||
fmt.Println("var liveUAPlatforms = []struct {")
|
||||
fmt.Println("\tua string")
|
||||
fmt.Println("\tos buildmeta.OS")
|
||||
fmt.Println("\tarch buildmeta.Arch")
|
||||
fmt.Println("\tlibc buildmeta.Libc")
|
||||
fmt.Println("}{")
|
||||
|
||||
for _, e := range entries {
|
||||
if e.Key.OS == "" || e.Key.Arch == "" {
|
||||
continue // skip undetectable
|
||||
}
|
||||
ua := e.Examples[0]
|
||||
fmt.Printf("\t{%q, %s, %s, %s},\n",
|
||||
ua, goConst("OS", e.Key.OS), goConst("Arch", e.Key.Arch), goConst("Libc", e.Key.Libc))
|
||||
}
|
||||
|
||||
fmt.Println("}")
|
||||
}
|
||||
|
||||
// isMalformed checks for genuinely corrupted UA strings (network truncation).
|
||||
func isMalformed(ua string) bool {
|
||||
// Extremely short (less than 10 chars) suggests truncation.
|
||||
if len(ua) < 10 {
|
||||
return true
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// extractHints finds environment signals in a UA string.
|
||||
func extractHints(ua string) []string {
|
||||
lower := strings.ToLower(ua)
|
||||
var out []string
|
||||
|
||||
patterns := []struct {
|
||||
substr string
|
||||
tag string
|
||||
}{
|
||||
{"amzn", "amzn"}, // Amazon Linux
|
||||
{"-azure", "azure"}, // Azure VM
|
||||
{"-gcp", "gcp"}, // Google Cloud
|
||||
{"-aws", "aws"}, // AWS kernel
|
||||
{"-oracle", "oracle"}, // Oracle Cloud
|
||||
{"el7", "rhel7"}, // RHEL/CentOS 7
|
||||
{"el8", "rhel8"}, // RHEL/CentOS 8
|
||||
{"el9", "rhel9"}, // RHEL/CentOS 9
|
||||
{".fc", "fedora"}, // Fedora
|
||||
{"+deb", "debian"}, // Debian
|
||||
{"-generic", "ubuntu"}, // Ubuntu generic kernel
|
||||
{"-pve", "proxmox"}, // Proxmox VE
|
||||
{"linuxkit", "docker"}, // Docker Desktop / linuxkit
|
||||
{"orbstack", "orbstack"},
|
||||
{"microsoft-standard-wsl", "wsl"},
|
||||
{"android", "android"},
|
||||
{"+rpt-rpi", "rpi"}, // Raspberry Pi
|
||||
{"cygwin", "cygwin"},
|
||||
{"mingw", "mingw"},
|
||||
{"msys", "msys"},
|
||||
{"freebsd", "freebsd"},
|
||||
{"-nvidia", "nvidia"},
|
||||
{"gentoo", "gentoo"},
|
||||
{"coreweave", "coreweave"},
|
||||
}
|
||||
|
||||
for _, p := range patterns {
|
||||
if strings.Contains(lower, p.substr) {
|
||||
out = append(out, p.tag)
|
||||
}
|
||||
}
|
||||
|
||||
return out
|
||||
}
|
||||
|
||||
// androidDeviceRe extracts device/build info from Android kernel strings.
|
||||
var androidDeviceRe = regexp.MustCompile(`ab[A-Z0-9]+`)
|
||||
|
||||
func displayOS(os string) string {
|
||||
if os == "" {
|
||||
return "(none)"
|
||||
}
|
||||
return os
|
||||
}
|
||||
|
||||
func displayLibc(libc string) string {
|
||||
if libc == "" {
|
||||
return "(none)"
|
||||
}
|
||||
return libc
|
||||
}
|
||||
|
||||
func goConst(prefix, val string) string {
|
||||
m := map[string]map[string]string{
|
||||
"OS": {
|
||||
"darwin": "buildmeta.OSDarwin",
|
||||
"linux": "buildmeta.OSLinux",
|
||||
"windows": "buildmeta.OSWindows",
|
||||
"freebsd": "buildmeta.OSFreeBSD",
|
||||
"android": "buildmeta.OSAndroid",
|
||||
"": `""`,
|
||||
},
|
||||
"Arch": {
|
||||
"aarch64": "buildmeta.ArchARM64",
|
||||
"x86_64": "buildmeta.ArchAMD64",
|
||||
"armv7": "buildmeta.ArchARMv7",
|
||||
"armv6": "buildmeta.ArchARMv6",
|
||||
"x86": "buildmeta.ArchX86",
|
||||
"ppc64le": "buildmeta.ArchPPC64LE",
|
||||
"ppc64": "buildmeta.ArchPPC64",
|
||||
"s390x": "buildmeta.ArchS390X",
|
||||
"riscv64": "buildmeta.ArchRISCV64",
|
||||
"": `""`,
|
||||
},
|
||||
"Libc": {
|
||||
"gnu": "buildmeta.LibcGNU",
|
||||
"musl": "buildmeta.LibcMusl",
|
||||
"msvc": "buildmeta.LibcMSVC",
|
||||
"none": "buildmeta.LibcNone",
|
||||
"": `""`,
|
||||
},
|
||||
}
|
||||
if v, ok := m[prefix][val]; ok {
|
||||
return v
|
||||
}
|
||||
return fmt.Sprintf("%q /* unmapped */", val)
|
||||
}
|
||||
|
||||
func containsStr(ss []string, s string) bool {
|
||||
for _, v := range ss {
|
||||
if v == s {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
899
cmd/webicached/main.go
Normal file
899
cmd/webicached/main.go
Normal file
@@ -0,0 +1,899 @@
|
||||
// Command webicached is the release cache daemon. It fetches releases
|
||||
// from upstream sources, classifies build assets, and writes them to
|
||||
// the _cache/ directory in the format the Node.js server expects.
|
||||
//
|
||||
// This is the Go replacement for the Node.js release-fetching pipeline.
|
||||
// It reads releases.conf files to discover packages, fetches from the
|
||||
// configured source, classifies assets, and writes to fsstore.
|
||||
//
|
||||
// Default mode: classify all from existing rawcache on startup, then
|
||||
// fetch+refresh one package per tick (round-robin, 15m default).
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// go run ./cmd/webicached # default: round-robin, one per tick
|
||||
// go run ./cmd/webicached -eager # fetch all packages on startup
|
||||
// go run ./cmd/webicached -once -no-fetch # classify from rawcache and exit
|
||||
// go run ./cmd/webicached bat goreleaser # only these packages
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"math/rand/v2"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/joho/godotenv"
|
||||
"github.com/webinstall/webi-installers/internal/classifypkg"
|
||||
"github.com/webinstall/webi-installers/internal/installerconf"
|
||||
"github.com/webinstall/webi-installers/internal/rawcache"
|
||||
"github.com/webinstall/webi-installers/internal/releases/chromedist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/flutterdist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/gitea"
|
||||
"github.com/webinstall/webi-installers/internal/releases/github"
|
||||
"github.com/webinstall/webi-installers/internal/releases/githubish"
|
||||
"github.com/webinstall/webi-installers/internal/releases/gittag"
|
||||
"github.com/webinstall/webi-installers/internal/releases/golang"
|
||||
"github.com/webinstall/webi-installers/internal/releases/gpgdist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/hashicorp"
|
||||
"github.com/webinstall/webi-installers/internal/releases/iterm2dist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/juliadist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/mariadbdist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/nodedist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/servicemandist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/zigdist"
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
"github.com/webinstall/webi-installers/internal/storage/fsstore"
|
||||
"github.com/webinstall/webi-installers/internal/storage/pgstore"
|
||||
)
|
||||
|
||||
var (
|
||||
name = "webicached"
|
||||
version = "0.0.0-dev"
|
||||
commit = "0000000"
|
||||
date = "0001-01-01"
|
||||
licenseYear = "2024"
|
||||
licenseOwner = "AJ ONeal"
|
||||
licenseType = "MPL-2.0"
|
||||
)
|
||||
|
||||
func printVersion(w io.Writer) {
|
||||
b_ver := strings.TrimPrefix(version, "v")
|
||||
_, _ = fmt.Fprintf(w, "%s v%s %s (%s)\n", name, b_ver, commit[:7], date)
|
||||
_, _ = fmt.Fprintf(w, "Copyright (C) %s %s\n", licenseYear, licenseOwner)
|
||||
_, _ = fmt.Fprintf(w, "Licensed under %s\n", licenseType)
|
||||
}
|
||||
|
||||
type MainConfig struct {
|
||||
envFile string
|
||||
confDir string
|
||||
cacheDir string
|
||||
pgDSN string
|
||||
rawDir string
|
||||
token string
|
||||
once bool
|
||||
noFetch bool
|
||||
shallow bool
|
||||
eager bool
|
||||
interval time.Duration
|
||||
pageDelay time.Duration
|
||||
}
|
||||
|
||||
// WebiCache holds the configuration for the cache daemon.
|
||||
type WebiCache struct {
|
||||
ConfDir string // root directory with {pkg}/releases.conf files
|
||||
Store storage.Store // classified asset storage (fsstore or pgstore)
|
||||
RawDir string // raw upstream response cache
|
||||
Client *http.Client // HTTP client for upstream calls
|
||||
Auth *githubish.Auth // GitHub API auth (optional)
|
||||
Shallow bool // fetch only the first page of releases
|
||||
NoFetch bool // skip fetching, classify from existing raw data only
|
||||
PageDelay time.Duration // delay between paginated API requests
|
||||
}
|
||||
|
||||
// delayTransport wraps an http.RoundTripper to add a delay between requests.
|
||||
type delayTransport struct {
|
||||
base http.RoundTripper
|
||||
delay time.Duration
|
||||
last time.Time
|
||||
}
|
||||
|
||||
func (t *delayTransport) RoundTrip(req *http.Request) (*http.Response, error) {
|
||||
if !t.last.IsZero() && t.delay > 0 {
|
||||
if wait := t.delay - time.Since(t.last); wait > 0 {
|
||||
time.Sleep(wait)
|
||||
}
|
||||
}
|
||||
t.last = time.Now()
|
||||
return t.base.RoundTrip(req)
|
||||
}
|
||||
|
||||
func main() {
|
||||
if len(os.Args) > 1 {
|
||||
switch os.Args[1] {
|
||||
case "-V", "-version", "--version", "version":
|
||||
printVersion(os.Stdout)
|
||||
os.Exit(0)
|
||||
case "help", "-help", "--help":
|
||||
printVersion(os.Stdout)
|
||||
fmt.Fprintln(os.Stdout, "")
|
||||
fs := flag.NewFlagSet(os.Args[0], flag.ContinueOnError)
|
||||
fs.SetOutput(os.Stdout)
|
||||
registerFlags(fs, &MainConfig{})
|
||||
fs.Usage()
|
||||
os.Exit(0)
|
||||
}
|
||||
}
|
||||
|
||||
cfg := MainConfig{}
|
||||
fs := flag.NewFlagSet(os.Args[0], flag.ContinueOnError)
|
||||
registerFlags(fs, &cfg)
|
||||
if err := fs.Parse(os.Args[1:]); err != nil {
|
||||
if errors.Is(err, flag.ErrHelp) {
|
||||
os.Exit(0)
|
||||
}
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
cfg.cacheDir = expandHome(cfg.cacheDir)
|
||||
cfg.rawDir = expandHome(cfg.rawDir)
|
||||
|
||||
if cfg.envFile != "" {
|
||||
if err := godotenv.Load(cfg.envFile); err != nil {
|
||||
log.Fatalf("envfile: %v", err)
|
||||
}
|
||||
}
|
||||
if cfg.token == "" {
|
||||
cfg.token = os.Getenv("GITHUB_TOKEN")
|
||||
}
|
||||
|
||||
var store storage.Store
|
||||
if cfg.pgDSN != "" {
|
||||
pg, err := pgstore.New(context.Background(), cfg.pgDSN)
|
||||
if err != nil {
|
||||
log.Fatalf("pgstore: %v", err)
|
||||
}
|
||||
store = pg
|
||||
} else {
|
||||
fs, err := fsstore.New(cfg.cacheDir)
|
||||
if err != nil {
|
||||
log.Fatalf("fsstore: %v", err)
|
||||
}
|
||||
store = fs
|
||||
}
|
||||
|
||||
var auth *githubish.Auth
|
||||
if cfg.token != "" {
|
||||
auth = &githubish.Auth{Token: cfg.token}
|
||||
}
|
||||
|
||||
client := &http.Client{Timeout: 30 * time.Second}
|
||||
if cfg.pageDelay > 0 {
|
||||
client.Transport = &delayTransport{
|
||||
base: http.DefaultTransport,
|
||||
delay: cfg.pageDelay,
|
||||
}
|
||||
}
|
||||
|
||||
wc := &WebiCache{
|
||||
ConfDir: cfg.confDir,
|
||||
Store: store,
|
||||
RawDir: cfg.rawDir,
|
||||
Client: client,
|
||||
Auth: auth,
|
||||
Shallow: cfg.shallow,
|
||||
NoFetch: cfg.noFetch,
|
||||
PageDelay: cfg.pageDelay,
|
||||
}
|
||||
|
||||
filterPkgs := fs.Args()
|
||||
|
||||
if cfg.eager {
|
||||
wc.Run(filterPkgs)
|
||||
if cfg.once {
|
||||
return
|
||||
}
|
||||
} else if cfg.once {
|
||||
wc.Run(filterPkgs)
|
||||
return
|
||||
} else {
|
||||
saved := wc.NoFetch
|
||||
wc.NoFetch = true
|
||||
wc.Run(filterPkgs)
|
||||
wc.NoFetch = saved
|
||||
}
|
||||
|
||||
packages, err := discover(wc.ConfDir)
|
||||
if err != nil {
|
||||
log.Fatalf("discover: %v", err)
|
||||
}
|
||||
if len(filterPkgs) > 0 {
|
||||
nameSet := make(map[string]bool, len(filterPkgs))
|
||||
for _, a := range filterPkgs {
|
||||
nameSet[a] = true
|
||||
}
|
||||
var filtered []pkgConf
|
||||
for _, p := range packages {
|
||||
if nameSet[p.name] {
|
||||
filtered = append(filtered, p)
|
||||
}
|
||||
}
|
||||
packages = filtered
|
||||
}
|
||||
|
||||
var real []pkgConf
|
||||
for _, pkg := range packages {
|
||||
if pkg.conf.AliasOf == "" {
|
||||
real = append(real, pkg)
|
||||
}
|
||||
}
|
||||
|
||||
log.Printf("refreshing %d packages, interval %s, batch size 20 (ctrl-c to stop)", len(real), cfg.interval)
|
||||
for {
|
||||
stale := wc.stalest(real)
|
||||
if len(stale) == 0 {
|
||||
log.Printf("all packages fresh, sleeping %s", cfg.interval)
|
||||
time.Sleep(cfg.interval)
|
||||
continue
|
||||
}
|
||||
|
||||
batch := stale
|
||||
if len(batch) > 20 {
|
||||
batch = batch[:20]
|
||||
}
|
||||
rand.Shuffle(len(batch), func(i, j int) {
|
||||
batch[i], batch[j] = batch[j], batch[i]
|
||||
})
|
||||
|
||||
log.Printf("batch: %d stale, refreshing %d (most stale first)", len(stale), len(batch))
|
||||
for _, pkg := range batch {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
|
||||
if err := wc.refreshPackage(ctx, pkg); err != nil {
|
||||
log.Printf(" ERROR %s: %v", pkg.name, err)
|
||||
}
|
||||
cancel()
|
||||
time.Sleep(cfg.interval)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func registerFlags(fs *flag.FlagSet, cfg *MainConfig) {
|
||||
fs.StringVar(&cfg.envFile, "envfile", "", "path to .env file to load before running")
|
||||
fs.StringVar(&cfg.confDir, "conf", ".", "root directory containing {pkg}/releases.conf files")
|
||||
fs.StringVar(&cfg.cacheDir, "legacy", "~/.cache/webi/legacy", "legacy cache directory (fsstore root)")
|
||||
fs.StringVar(&cfg.pgDSN, "pg", "", "PostgreSQL DSN (enables pgstore; mutually exclusive with -legacy)")
|
||||
fs.StringVar(&cfg.rawDir, "raw", "~/.cache/webi/raw", "raw cache directory for upstream responses")
|
||||
fs.StringVar(&cfg.token, "token", "", "GitHub API token (or set $GITHUB_TOKEN)")
|
||||
fs.BoolVar(&cfg.once, "once", false, "run once then exit (no periodic refresh)")
|
||||
fs.BoolVar(&cfg.noFetch, "no-fetch", false, "skip fetching, classify from existing raw data only")
|
||||
fs.BoolVar(&cfg.shallow, "shallow", false, "fetch only the first page of releases (latest)")
|
||||
fs.BoolVar(&cfg.eager, "eager", false, "fetch all packages on startup (default: one per tick)")
|
||||
fs.DurationVar(&cfg.interval, "interval", 9*time.Second, "delay between individual package fetches")
|
||||
fs.DurationVar(&cfg.pageDelay, "page-delay", 2*time.Second, "delay between paginated API requests")
|
||||
}
|
||||
|
||||
func expandHome(path string) string {
|
||||
if !strings.HasPrefix(path, "~/") {
|
||||
return path
|
||||
}
|
||||
home, err := os.UserHomeDir()
|
||||
if err != nil {
|
||||
return path
|
||||
}
|
||||
return filepath.Join(home, path[2:])
|
||||
}
|
||||
|
||||
// stalest returns packages sorted by most stale first (oldest UpdatedAt).
|
||||
// Packages with no cache entry or empty assets are considered most stale.
|
||||
func (wc *WebiCache) stalest(packages []pkgConf) []pkgConf {
|
||||
type stamped struct {
|
||||
pkg pkgConf
|
||||
updatedAt time.Time
|
||||
}
|
||||
|
||||
var stale []stamped
|
||||
ctx := context.Background()
|
||||
for _, pkg := range packages {
|
||||
data, err := wc.Store.Load(ctx, pkg.name)
|
||||
var t time.Time
|
||||
hasAssets := false
|
||||
if err == nil && data != nil {
|
||||
t = data.UpdatedAt
|
||||
hasAssets = len(data.Assets) > 0
|
||||
}
|
||||
// Never fetched, or has no assets despite having a timestamp
|
||||
// (e.g. classified from empty rawcache), or older than 10 minutes.
|
||||
if t.IsZero() || !hasAssets || time.Since(t) > 10*time.Minute {
|
||||
stale = append(stale, stamped{pkg: pkg, updatedAt: t})
|
||||
}
|
||||
}
|
||||
|
||||
sort.SliceStable(stale, func(i, j int) bool {
|
||||
ti, tj := stale[i].updatedAt, stale[j].updatedAt
|
||||
if ti.Equal(tj) {
|
||||
return stale[i].pkg.name < stale[j].pkg.name
|
||||
}
|
||||
return ti.Before(tj)
|
||||
})
|
||||
|
||||
result := make([]pkgConf, len(stale))
|
||||
for i, s := range stale {
|
||||
result[i] = s.pkg
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// Run discovers packages and refreshes each one.
|
||||
func (wc *WebiCache) Run(filterPkgs []string) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
packages, err := discover(wc.ConfDir)
|
||||
if err != nil {
|
||||
log.Printf("discover: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
if len(filterPkgs) > 0 {
|
||||
nameSet := make(map[string]bool, len(filterPkgs))
|
||||
for _, a := range filterPkgs {
|
||||
nameSet[a] = true
|
||||
}
|
||||
var filtered []pkgConf
|
||||
for _, p := range packages {
|
||||
if nameSet[p.name] {
|
||||
filtered = append(filtered, p)
|
||||
}
|
||||
}
|
||||
packages = filtered
|
||||
}
|
||||
|
||||
var real []pkgConf
|
||||
for _, pkg := range packages {
|
||||
if pkg.conf.AliasOf != "" {
|
||||
continue
|
||||
}
|
||||
real = append(real, pkg)
|
||||
}
|
||||
|
||||
log.Printf("refreshing %d packages", len(real))
|
||||
runStart := time.Now()
|
||||
|
||||
for _, pkg := range real {
|
||||
if err := wc.refreshPackage(ctx, pkg); err != nil {
|
||||
log.Printf(" ERROR %s: %v", pkg.name, err)
|
||||
}
|
||||
}
|
||||
|
||||
log.Printf("refreshed %d packages in %s", len(real), time.Since(runStart))
|
||||
}
|
||||
|
||||
type pkgConf struct {
|
||||
name string
|
||||
conf *installerconf.Conf
|
||||
}
|
||||
|
||||
func discover(dir string) ([]pkgConf, error) {
|
||||
pattern := filepath.Join(dir, "*", "releases.conf")
|
||||
matches, err := filepath.Glob(pattern)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var packages []pkgConf
|
||||
for _, path := range matches {
|
||||
pkgDir := filepath.Dir(path)
|
||||
name := filepath.Base(pkgDir)
|
||||
if strings.HasPrefix(name, "_") {
|
||||
continue
|
||||
}
|
||||
|
||||
// If the package directory is a symlink, treat it as an alias
|
||||
// of the symlink target (e.g. rust.vim → vim-rust).
|
||||
fi, err := os.Lstat(filepath.Join(dir, name))
|
||||
if err != nil {
|
||||
log.Printf("warning: %s: %v", name, err)
|
||||
continue
|
||||
}
|
||||
if fi.Mode()&os.ModeSymlink != 0 {
|
||||
target, err := os.Readlink(filepath.Join(dir, name))
|
||||
if err != nil {
|
||||
log.Printf("warning: readlink %s: %v", name, err)
|
||||
continue
|
||||
}
|
||||
packages = append(packages, pkgConf{
|
||||
name: name,
|
||||
conf: &installerconf.Conf{AliasOf: target},
|
||||
})
|
||||
continue
|
||||
}
|
||||
|
||||
conf, err := installerconf.Read(path)
|
||||
if err != nil {
|
||||
log.Printf("warning: %s: %v", path, err)
|
||||
continue
|
||||
}
|
||||
packages = append(packages, pkgConf{name: name, conf: conf})
|
||||
}
|
||||
|
||||
sort.Slice(packages, func(i, j int) bool {
|
||||
return packages[i].name < packages[j].name
|
||||
})
|
||||
return packages, nil
|
||||
}
|
||||
|
||||
// refreshPackage does the full pipeline for one package:
|
||||
// fetch raw → classify → write to fsstore.
|
||||
func (wc *WebiCache) refreshPackage(ctx context.Context, pkg pkgConf) error {
|
||||
pkgStart := time.Now()
|
||||
name := pkg.name
|
||||
conf := pkg.conf
|
||||
|
||||
// Step 1: Fetch raw upstream data to rawcache (unless -no-fetch).
|
||||
if !wc.NoFetch {
|
||||
shallow := wc.Shallow
|
||||
if !shallow {
|
||||
d, err := rawcache.Open(filepath.Join(wc.RawDir, name))
|
||||
if err == nil && d.Populated() {
|
||||
shallow = true
|
||||
}
|
||||
}
|
||||
fetchStart := time.Now()
|
||||
if err := wc.fetchRaw(ctx, pkg, shallow); err != nil {
|
||||
return fmt.Errorf("fetch: %w", err)
|
||||
}
|
||||
log.Printf(" %s: fetch %s", name, time.Since(fetchStart))
|
||||
}
|
||||
|
||||
// Step 2: Classify raw data into assets, tag variants, apply config.
|
||||
classifyStart := time.Now()
|
||||
d, err := rawcache.Open(filepath.Join(wc.RawDir, name))
|
||||
if err != nil {
|
||||
return fmt.Errorf("rawcache open: %w", err)
|
||||
}
|
||||
|
||||
// Open supplementary gittag raw cache if available (for packages with
|
||||
// git_url that use a non-gittag source type like servicemandist).
|
||||
var gitTagDir *rawcache.Dir
|
||||
if conf.GitURL != "" && conf.Source != "gittag" {
|
||||
gd, gdErr := rawcache.Open(filepath.Join(wc.RawDir, "_gittag", name))
|
||||
if gdErr == nil && gd.Populated() {
|
||||
gitTagDir = gd
|
||||
}
|
||||
}
|
||||
|
||||
assets, err := classifypkg.Package(name, conf, d, gitTagDir)
|
||||
if err != nil {
|
||||
return fmt.Errorf("classify: %w", err)
|
||||
}
|
||||
classifyDur := time.Since(classifyStart)
|
||||
|
||||
// Step 3: Write to fsstore.
|
||||
writeStart := time.Now()
|
||||
tx, err := wc.Store.BeginRefresh(ctx, name)
|
||||
if err != nil {
|
||||
return fmt.Errorf("begin refresh: %w", err)
|
||||
}
|
||||
if err := tx.Put(assets); err != nil {
|
||||
tx.Rollback()
|
||||
return fmt.Errorf("put: %w", err)
|
||||
}
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return fmt.Errorf("commit: %w", err)
|
||||
}
|
||||
writeDur := time.Since(writeStart)
|
||||
|
||||
log.Printf(" %s: %d assets (classify %s, write %s, total %s)",
|
||||
name, len(assets), classifyDur, writeDur, time.Since(pkgStart))
|
||||
return nil
|
||||
}
|
||||
|
||||
// --- Fetch raw ---
|
||||
|
||||
func (wc *WebiCache) fetchRaw(ctx context.Context, pkg pkgConf, shallow bool) error {
|
||||
switch pkg.conf.Source {
|
||||
case "github", "githubsource":
|
||||
if err := wc.fetchGitHub(ctx, pkg.name, pkg.conf, shallow); err != nil {
|
||||
return err
|
||||
}
|
||||
case "nodedist":
|
||||
return wc.fetchNodeDist(ctx, pkg.name, pkg.conf)
|
||||
case "gittag":
|
||||
return wc.fetchGitTag(ctx, pkg.name, pkg.conf, shallow)
|
||||
case "gitea":
|
||||
return wc.fetchGitea(ctx, pkg.name, pkg.conf, shallow)
|
||||
case "chromedist":
|
||||
return fetchChromeDist(ctx, wc.Client, wc.RawDir, pkg.name)
|
||||
case "flutterdist":
|
||||
return fetchFlutterDist(ctx, wc.Client, wc.RawDir, pkg.name)
|
||||
case "golang":
|
||||
return fetchGolang(ctx, wc.Client, wc.RawDir, pkg.name)
|
||||
case "gpgdist":
|
||||
return fetchGPGDist(ctx, wc.Client, wc.RawDir, pkg.name)
|
||||
case "hashicorp":
|
||||
return fetchHashiCorp(ctx, wc.Client, wc.RawDir, pkg.name, pkg.conf)
|
||||
case "iterm2dist":
|
||||
return fetchITerm2Dist(ctx, wc.Client, wc.RawDir, pkg.name)
|
||||
case "juliadist":
|
||||
return fetchJuliaDist(ctx, wc.Client, wc.RawDir, pkg.name)
|
||||
case "mariadbdist":
|
||||
return fetchMariaDBDist(ctx, wc.Client, wc.RawDir, pkg.name)
|
||||
case "servicemandist":
|
||||
if err := servicemandist.Fetch(ctx, wc.Client, wc.RawDir, pkg.name, wc.Auth, shallow); err != nil {
|
||||
return err
|
||||
}
|
||||
case "zigdist":
|
||||
return fetchZigDist(ctx, wc.Client, wc.RawDir, pkg.name)
|
||||
default:
|
||||
log.Printf(" %s: source %q not yet supported, skipping", pkg.name, pkg.conf.Source)
|
||||
return nil
|
||||
}
|
||||
|
||||
// For non-gittag sources with a git_url, also clone the repo to get
|
||||
// commit hashes. Git entries are classified from this data in
|
||||
// refreshPackage, not from the main raw cache.
|
||||
if pkg.conf.GitURL != "" && pkg.conf.Source != "gittag" {
|
||||
gitShallow := shallow
|
||||
if !wc.Shallow {
|
||||
gd, gdErr := rawcache.Open(filepath.Join(wc.RawDir, "_gittag", pkg.name))
|
||||
if gdErr == nil && !gd.Populated() {
|
||||
gitShallow = false
|
||||
}
|
||||
}
|
||||
if err := wc.fetchGitTagSupplementary(ctx, pkg.name, pkg.conf.GitURL, gitShallow); err != nil {
|
||||
log.Printf(" %s: supplementary gittag fetch: %v", pkg.name, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// fetchGitTagSupplementary clones a git repo to get commit hashes for
|
||||
// packages that use a non-gittag source type (servicemandist, githubsource)
|
||||
// but also have a git_url for source installs.
|
||||
func (wc *WebiCache) fetchGitTagSupplementary(ctx context.Context, pkgName, gitURL string, shallow bool) error {
|
||||
d, err := rawcache.Open(filepath.Join(wc.RawDir, "_gittag", pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
repoDir := filepath.Join(wc.RawDir, "_repos")
|
||||
os.MkdirAll(repoDir, 0o755)
|
||||
|
||||
for batch, err := range gittag.Fetch(ctx, gitURL, repoDir) {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, entry := range batch {
|
||||
tag := entry.Version
|
||||
if tag == "" {
|
||||
tag = "HEAD-" + entry.CommitHash
|
||||
}
|
||||
data, _ := json.Marshal(entry)
|
||||
d.Merge(tag, data)
|
||||
}
|
||||
if shallow {
|
||||
break
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (wc *WebiCache) fetchGitHub(ctx context.Context, pkgName string, conf *installerconf.Conf, shallow bool) error {
|
||||
owner, repo := conf.Owner, conf.Repo
|
||||
if owner == "" || repo == "" {
|
||||
return fmt.Errorf("missing owner or repo")
|
||||
}
|
||||
|
||||
d, err := rawcache.Open(filepath.Join(wc.RawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
tagPrefix := conf.TagPrefix
|
||||
for batch, err := range github.Fetch(ctx, wc.Client, owner, repo, wc.Auth) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("github %s/%s: %w", owner, repo, err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
if rel.Draft {
|
||||
continue
|
||||
}
|
||||
tag := rel.TagName
|
||||
if tagPrefix != "" && !strings.HasPrefix(tag, tagPrefix) {
|
||||
continue
|
||||
}
|
||||
data, _ := json.Marshal(rel)
|
||||
d.Merge(tag, data)
|
||||
}
|
||||
if shallow {
|
||||
break
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (wc *WebiCache) fetchNodeDist(ctx context.Context, pkgName string, conf *installerconf.Conf) error {
|
||||
baseURL := conf.BaseURL
|
||||
if baseURL == "" {
|
||||
return fmt.Errorf("missing url")
|
||||
}
|
||||
|
||||
d, err := rawcache.Open(filepath.Join(wc.RawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Fetch from primary URL. Tag with "official/" prefix so unofficial
|
||||
// entries for the same version don't overwrite.
|
||||
for batch, err := range nodedist.Fetch(ctx, wc.Client, baseURL) {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, entry := range batch {
|
||||
data, _ := json.Marshal(entry)
|
||||
d.Merge("official/"+entry.Version, data)
|
||||
}
|
||||
}
|
||||
|
||||
// Fetch from unofficial URL if configured (e.g. Node.js unofficial builds
|
||||
// which add musl, riscv64, loong64 targets).
|
||||
if unofficialURL := conf.Extra["unofficial_url"]; unofficialURL != "" {
|
||||
for batch, err := range nodedist.Fetch(ctx, wc.Client, unofficialURL) {
|
||||
if err != nil {
|
||||
log.Printf("warning: %s unofficial fetch: %v", pkgName, err)
|
||||
break
|
||||
}
|
||||
for _, entry := range batch {
|
||||
data, _ := json.Marshal(entry)
|
||||
d.Merge("unofficial/"+entry.Version, data)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (wc *WebiCache) fetchGitTag(ctx context.Context, pkgName string, conf *installerconf.Conf, shallow bool) error {
|
||||
gitURL := conf.BaseURL
|
||||
if gitURL == "" {
|
||||
return fmt.Errorf("missing url")
|
||||
}
|
||||
|
||||
d, err := rawcache.Open(filepath.Join(wc.RawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
repoDir := filepath.Join(wc.RawDir, "_repos")
|
||||
os.MkdirAll(repoDir, 0o755)
|
||||
|
||||
for batch, err := range gittag.Fetch(ctx, gitURL, repoDir) {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, entry := range batch {
|
||||
tag := entry.Version
|
||||
if tag == "" {
|
||||
tag = "HEAD-" + entry.CommitHash
|
||||
}
|
||||
data, _ := json.Marshal(entry)
|
||||
d.Merge(tag, data)
|
||||
}
|
||||
if shallow {
|
||||
break
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (wc *WebiCache) fetchGitea(ctx context.Context, pkgName string, conf *installerconf.Conf, shallow bool) error {
|
||||
baseURL, owner, repo := conf.BaseURL, conf.Owner, conf.Repo
|
||||
if baseURL == "" || owner == "" || repo == "" {
|
||||
return fmt.Errorf("missing base_url, owner, or repo")
|
||||
}
|
||||
|
||||
d, err := rawcache.Open(filepath.Join(wc.RawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range gitea.Fetch(ctx, wc.Client, baseURL, owner, repo, nil) {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, rel := range batch {
|
||||
if rel.Draft {
|
||||
continue
|
||||
}
|
||||
data, _ := json.Marshal(rel)
|
||||
d.Merge(rel.TagName, data)
|
||||
}
|
||||
if shallow {
|
||||
break
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchChromeDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range chromedist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("chromedist: %w", err)
|
||||
}
|
||||
for _, ver := range batch {
|
||||
data, _ := json.Marshal(ver)
|
||||
d.Merge(ver.Version, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchFlutterDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range flutterdist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("flutterdist: %w", err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
// Key by version+channel+os for uniqueness.
|
||||
key := rel.Version + "-" + rel.Channel + "-" + rel.OS
|
||||
data, _ := json.Marshal(rel)
|
||||
d.Merge(key, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchGolang(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range golang.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("golang: %w", err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
data, _ := json.Marshal(rel)
|
||||
d.Merge(rel.Version, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchGPGDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range gpgdist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("gpgdist: %w", err)
|
||||
}
|
||||
for _, entry := range batch {
|
||||
data, _ := json.Marshal(entry)
|
||||
d.Merge(entry.Version, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchHashiCorp(ctx context.Context, client *http.Client, rawDir, pkgName string, conf *installerconf.Conf) error {
|
||||
product := conf.Repo
|
||||
if product == "" {
|
||||
product = pkgName
|
||||
}
|
||||
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for idx, err := range hashicorp.Fetch(ctx, client, product) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("hashicorp %s: %w", product, err)
|
||||
}
|
||||
for ver, vdata := range idx.Versions {
|
||||
data, _ := json.Marshal(vdata)
|
||||
d.Merge(ver, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchITerm2Dist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range iterm2dist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("iterm2dist: %w", err)
|
||||
}
|
||||
for _, entry := range batch {
|
||||
key := entry.Version
|
||||
if entry.Channel == "beta" {
|
||||
key += "-beta"
|
||||
}
|
||||
data, _ := json.Marshal(entry)
|
||||
d.Merge(key, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchJuliaDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range juliadist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("juliadist: %w", err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
data, _ := json.Marshal(rel)
|
||||
d.Merge(rel.Version, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchMariaDBDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range mariadbdist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("mariadbdist: %w", err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
data, _ := json.Marshal(rel)
|
||||
d.Merge(rel.ReleaseID, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchZigDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range zigdist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("zigdist: %w", err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
data, _ := json.Marshal(rel)
|
||||
d.Merge(rel.Version, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
121
docs/installer-patterns.md
Normal file
121
docs/installer-patterns.md
Normal file
@@ -0,0 +1,121 @@
|
||||
# Installer Archive Patterns
|
||||
|
||||
Every package falls into one of these archive structure patterns. When writing
|
||||
or modifying an `install.sh`, identify the pattern first — it determines the
|
||||
extraction and installation strategy.
|
||||
|
||||
## Pattern A: Bare Binary in Archive
|
||||
|
||||
Archive contains the binary (and maybe LICENSE/README) at the top level.
|
||||
|
||||
Examples: awless, caddy, cilium, curlie, dashmsg, deno, dotenv, dotenv-linter,
|
||||
ffuf, fzf, gitdeploy, gprox, grype, hugo, hugo-extended, k9s, keypairs, koji,
|
||||
lf, monorel, ots, runzip, sclient, sqlc, sqlpkg, sttr, terraform, uuidv7, xcaddy
|
||||
|
||||
Install: extract, move binary to `~/.local/opt/{pkg}-{ver}/bin/{binary}`, symlink.
|
||||
|
||||
## Pattern B: Subdirectory with Binary Only
|
||||
|
||||
Archive contains a version-named directory wrapping the binary and docs.
|
||||
|
||||
Examples: delta, hexyl, kubectx, kubens, shellcheck, trip, xsv
|
||||
|
||||
Typical directory naming: `{tool}-{ver}-{triplet}/`
|
||||
|
||||
Install: extract, find binary in subdirectory, move to opt, symlink.
|
||||
|
||||
Special cases:
|
||||
- `pathman`: bare binary named with full release tag (needs rename)
|
||||
- `yq`: binary named with platform suffix `yq_linux_amd64` (needs rename)
|
||||
|
||||
## Pattern C: Binary + Completions + Man Pages
|
||||
|
||||
Archive includes shell completions and/or man pages alongside the binary.
|
||||
|
||||
| Package | Completions Dir | Man Page |
|
||||
|---------|----------------|----------|
|
||||
| bat | `autocomplete/` | `bat.1` |
|
||||
| fd | `autocomplete/{fd.bash,.fish,_fd}` | `fd.1` |
|
||||
| goreleaser | `completions/{.bash,.fish,.zsh}` | `manpages/*.1.gz` |
|
||||
| lsd | `autocomplete/{lsd.bash-completion,.fish,_lsd}` | `lsd.1` |
|
||||
| rg | `complete/{rg.bash,.fish,_rg}` | `doc/rg.1` |
|
||||
| sd | `completions/{sd.bash,.fish,_sd}` | `sd.1` |
|
||||
| watchexec | `completions/{bash,fish,zsh}` | `watchexec.1` |
|
||||
| zoxide | `completions/{zoxide.bash,.fish,_zoxide}` | `man/man1/zoxide*.1` |
|
||||
|
||||
Install: extract, install binary, install completions to standard dirs, install
|
||||
man pages. Completion naming varies: `autocomplete/`, `completions/`, `complete/`.
|
||||
|
||||
## Pattern D: Binary + Libraries
|
||||
|
||||
Complex packages that bundle shared libraries.
|
||||
|
||||
| Package | Layout |
|
||||
|---------|--------|
|
||||
| ollama (Linux) | `bin/ollama` + `lib/ollama/{cuda_v12,cuda_v13,vulkan}/` |
|
||||
| pg/postgres/psql | `bin/psql` + `lib/{libpq,libz,...}.so` + `include/` |
|
||||
| sass | `dart-sass/sass` (wrapper) + `dart-sass/src/{dart,sass.snapshot}` |
|
||||
| syncthing | `syncthing-{triplet}-{ver}/syncthing` + `etc/{systemd,...}/` |
|
||||
| xz | `xz-{ver}-{triplet}/xz` + `xz-{ver}-{triplet}/unxz` |
|
||||
|
||||
Install: extract entire directory tree into opt, symlink binary.
|
||||
|
||||
## Pattern E: FHS-like Layout (bin/ + share/)
|
||||
|
||||
Archive already follows standard layout.
|
||||
|
||||
| Package | Layout |
|
||||
|---------|--------|
|
||||
| gh | `gh_{ver}_{os}_{arch}/bin/gh` + `share/man/man1/*.1` |
|
||||
| pandoc | `pandoc-{ver}/bin/{pandoc,...}` + `share/man/man1/*.1.gz` |
|
||||
|
||||
Install: extract directly into opt (already correct layout).
|
||||
|
||||
## Pattern G: Full SDK/Toolchain
|
||||
|
||||
Self-contained toolchain with compiler, runtime, standard library.
|
||||
|
||||
| Package | Layout |
|
||||
|---------|--------|
|
||||
| cmake | `cmake-{ver}-{os}-{arch}/bin/{cmake,ctest,...}` + `share/` + `man/` |
|
||||
| tinygo | `tinygo/bin/tinygo` + `tinygo/src/` + `tinygo/targets/` |
|
||||
| go | `go/bin/{go,gofmt}` + `go/src/` + `go/pkg/` |
|
||||
| zig | `zig-{os}-{arch}-{ver}/zig` + `lib/` |
|
||||
| flutter | `flutter/bin/flutter` + full SDK |
|
||||
| julia | `julia-{ver}/bin/julia` + full SDK |
|
||||
| node | `node-{ver}-{os}-{arch}/bin/{node,npm,npx}` + `lib/` |
|
||||
|
||||
Install: extract entire tree into `~/.local/opt/{pkg}-{ver}/`, symlink `bin/*`.
|
||||
|
||||
## Pattern H: .NET Runtime Bundle
|
||||
|
||||
Flat archive with hundreds of DLLs.
|
||||
|
||||
Example: pwsh — `pwsh` binary + `*.dll` + locale dirs
|
||||
|
||||
Install: extract entire directory into opt, symlink primary binary.
|
||||
|
||||
## Pattern I: Multi-Binary Distribution
|
||||
|
||||
Archive contains multiple related binaries + libs.
|
||||
|
||||
| Package | Layout |
|
||||
|---------|--------|
|
||||
| dashcore | `dashcore-{ver}/bin/{dashd,dash-cli,...}` + `lib/` + `share/man/` |
|
||||
| mutagen | `mutagen` + `mutagen-agents.tar.gz` (embedded agent archive) |
|
||||
|
||||
Install: extract into opt, symlink primary binary.
|
||||
|
||||
## Format Changes Over Time
|
||||
|
||||
Most packages have stable formats. Notable structural changes:
|
||||
|
||||
| Package | When | Change |
|
||||
|---------|------|--------|
|
||||
| sd | 2023 | zip → tar.gz, added completions + man page |
|
||||
| ollama | 2025-2026 | bare binary → no GitHub release → tar.zst with lib/ |
|
||||
| deno | 2020-2021 | .gz (gzipped binary) → .zip |
|
||||
| hugo | 2017-2018 | zip → tar.gz; 2024: macOS → .pkg only |
|
||||
| gh | 2024 | darwin: tar.gz → .pkg |
|
||||
| sclient | 2023 | tar.gz → tar.xz |
|
||||
| watchexec | 2019-2020 | tar.gz → tar.xz |
|
||||
74
docs/version-oddities.md
Normal file
74
docs/version-oddities.md
Normal file
@@ -0,0 +1,74 @@
|
||||
# Version & Release Oddities
|
||||
|
||||
Non-standard version formats and tag prefixes that affect parsing, sorting,
|
||||
and classification. The Go classifier and `internal/lexver` must handle all
|
||||
of these.
|
||||
|
||||
## Non-Numeric Tag Prefixes
|
||||
|
||||
| Package | Raw Tag | Cleaned | Transform |
|
||||
|---------|---------|---------|-----------|
|
||||
| lf | `r21` | `0.21.0` | `r` prefix → prepend `0.` |
|
||||
| bun | `bun-v1.0.0` | `1.0.0` | Strip `bun-` prefix |
|
||||
| jq | `jq-1.7` | `1.7` | Strip `jq-` prefix |
|
||||
| watchexec | `cli-v1.2.3` | `1.2.3` | Strip `cli-` prefix |
|
||||
| ffmpeg | `b6.0` | `6.0` | Strip `b` prefix |
|
||||
|
||||
## Underscore-Delimited Tags
|
||||
|
||||
| Package | Raw Tag | Cleaned | Transform |
|
||||
|---------|---------|---------|-----------|
|
||||
| postgres | `REL_17_0` | `17.0` | Strip `REL_`, replace `_` with `.` |
|
||||
| psql | `REL_17_0` | `17.0` | Same as postgres |
|
||||
|
||||
## Platform Suffix in Version
|
||||
|
||||
| Package | Raw Tag | Cleaned | Transform |
|
||||
|---------|---------|---------|-----------|
|
||||
| git (Windows) | `2.41.0.windows.1` | `2.41.0` | Strip `.windows.N` suffix |
|
||||
|
||||
## 4-Part Versions
|
||||
|
||||
| Package | Example | Notes |
|
||||
|---------|---------|-------|
|
||||
| chromedriver | `121.0.6120.0` | Google Chrome's versioning |
|
||||
| gpg | `2.2.19.0` | 4th segment is build metadata |
|
||||
|
||||
## Date-Based Versions
|
||||
|
||||
| Package | Notes |
|
||||
|---------|-------|
|
||||
| atomicparsley | Date-based version strings |
|
||||
|
||||
## Complex Pre-Release Formats
|
||||
|
||||
| Package | Example | Notes |
|
||||
|---------|---------|-------|
|
||||
| flutter | `2.3.0-16.0.pre` | Extra dots and numeric segments |
|
||||
| iterm2 | `iTerm2_3_5_0beta17` | Underscores, beta attached → `3.5.0-beta17` |
|
||||
|
||||
## Channel Detection
|
||||
|
||||
- Node.js: odd major = "current" not LTS (v15, v17, v19, v21, v23)
|
||||
- Go: `go` prefix stripped (`go1.23.6` → `1.23.6`)
|
||||
- Terraform: `-alpha`, `-beta`, `-rc` suffixes → beta channel
|
||||
|
||||
## Directory Symlinks (Aliases)
|
||||
|
||||
These are directory-level symlinks. They share all files (including
|
||||
releases.conf) with their target automatically.
|
||||
|
||||
```
|
||||
msvc-runtime → vcruntime
|
||||
msvcruntime → vcruntime
|
||||
rust.vim → vim-rust
|
||||
vc-redist → vcruntime
|
||||
vc-runtime → vcruntime
|
||||
vc_redist → vcruntime
|
||||
vcredist → vcruntime
|
||||
vcruntime140 → vcruntime
|
||||
vim-essential → vim-essentials
|
||||
vim-mouse → vim-gui
|
||||
vps-myip → myip
|
||||
xcode-cli → commandlinetools
|
||||
```
|
||||
15
go.mod
Normal file
15
go.mod
Normal file
@@ -0,0 +1,15 @@
|
||||
module github.com/webinstall/webi-installers
|
||||
|
||||
go 1.26.1
|
||||
|
||||
require (
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||
github.com/jackc/pgx/v5 v5.8.0 // indirect
|
||||
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||
github.com/joho/godotenv v1.5.1 // indirect
|
||||
github.com/jszwec/csvutil v1.10.0 // indirect
|
||||
github.com/therootcompany/golib/http/middleware/v2 v2.0.1 // indirect
|
||||
golang.org/x/sync v0.17.0 // indirect
|
||||
golang.org/x/text v0.29.0 // indirect
|
||||
)
|
||||
25
go.sum
Normal file
25
go.sum
Normal file
@@ -0,0 +1,25 @@
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
|
||||
github.com/jackc/pgx/v5 v5.8.0 h1:TYPDoleBBme0xGSAX3/+NujXXtpZn9HBONkQC7IEZSo=
|
||||
github.com/jackc/pgx/v5 v5.8.0/go.mod h1:QVeDInX2m9VyzvNeiCJVjCkNFqzsNb43204HshNSZKw=
|
||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/joho/godotenv v1.5.1 h1:7eLL/+HRGLY0ldzfGMeQkb7vMd0as4CfYvUVzLqw0N0=
|
||||
github.com/joho/godotenv v1.5.1/go.mod h1:f4LDr5Voq0i2e/R5DDNOoa2zzDfwtkZa6DnEwAbqwq4=
|
||||
github.com/jszwec/csvutil v1.10.0 h1:upMDUxhQKqZ5ZDCs/wy+8Kib8rZR8I8lOR34yJkdqhI=
|
||||
github.com/jszwec/csvutil v1.10.0/go.mod h1:/E4ONrmGkwmWsk9ae9jpXnv9QT8pLHEPcCirMFhxG9I=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/therootcompany/golib/http/middleware/v2 v2.0.1 h1:VNKpHcwyEW7cMct7/eO4fyrxwIQk2ycb6juVXSPs2Sk=
|
||||
github.com/therootcompany/golib/http/middleware/v2 v2.0.1/go.mod h1:g5gb9qBidw74nW6/mwIauTKMpOKchiN2l0gt5qzJ2aQ=
|
||||
golang.org/x/sync v0.17.0 h1:l60nONMj9l5drqw6jlhIELNv9I0A4OFgRsG9k2oT9Ug=
|
||||
golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
|
||||
golang.org/x/text v0.29.0 h1:1neNs90w9YzJ9BocxfsQNHKuAT4pkghyXc4nhZ6sJvk=
|
||||
golang.org/x/text v0.29.0/go.mod h1:7MhJOA9CD2qZyOKYazxdYMF85OwPdEr9jTtBpO7ydH4=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
168
internal/buildmeta/buildmeta.go
Normal file
168
internal/buildmeta/buildmeta.go
Normal file
@@ -0,0 +1,168 @@
|
||||
// Package buildmeta is the shared vocabulary for Webi's build targets.
|
||||
//
|
||||
// Every package that deals with OS, architecture, libc, archive format, or
|
||||
// release channel imports these types instead of passing raw strings. This
|
||||
// prevents typos like "darwn" from compiling and gives a single place to
|
||||
// enumerate what Webi supports.
|
||||
package buildmeta
|
||||
|
||||
// OS represents a target operating system.
|
||||
type OS string
|
||||
|
||||
const (
|
||||
OSAny OS = "ANYOS"
|
||||
OSDarwin OS = "darwin"
|
||||
OSLinux OS = "linux"
|
||||
OSWindows OS = "windows"
|
||||
OSFreeBSD OS = "freebsd"
|
||||
OSOpenBSD OS = "openbsd"
|
||||
OSNetBSD OS = "netbsd"
|
||||
OSDragonFly OS = "dragonfly"
|
||||
OSSunOS OS = "sunos"
|
||||
OSIllumos OS = "illumos"
|
||||
OSSolaris OS = "solaris"
|
||||
OSAIX OS = "aix"
|
||||
OSAndroid OS = "android"
|
||||
OSPlan9 OS = "plan9"
|
||||
|
||||
// POSIX compatibility levels — used when a package is a shell script
|
||||
// or otherwise OS-independent for POSIX systems.
|
||||
OSPosix2017 OS = "posix_2017"
|
||||
OSPosix2024 OS = "posix_2024"
|
||||
)
|
||||
|
||||
// Arch represents a target CPU architecture.
|
||||
type Arch string
|
||||
|
||||
const (
|
||||
ArchAny Arch = "ANYARCH"
|
||||
ArchAMD64 Arch = "x86_64" // baseline (v1)
|
||||
ArchAMD64v2 Arch = "x86_64_v2" // +SSE4, +POPCNT, etc.
|
||||
ArchAMD64v3 Arch = "x86_64_v3" // +AVX2, +BMI, etc.
|
||||
ArchAMD64v4 Arch = "x86_64_v4" // +AVX-512
|
||||
ArchARM64 Arch = "aarch64"
|
||||
ArchARMv7 Arch = "armv7"
|
||||
ArchARMv6 Arch = "armv6"
|
||||
ArchARMv5 Arch = "armv5"
|
||||
ArchX86 Arch = "x86"
|
||||
ArchPPC64LE Arch = "ppc64le"
|
||||
ArchPPC64 Arch = "ppc64"
|
||||
ArchPPC Arch = "powerpc" // 32-bit PowerPC (unsupported by webi, used to prevent gnueabihf over-matching)
|
||||
ArchRISCV64 Arch = "riscv64"
|
||||
ArchS390X Arch = "s390x"
|
||||
ArchLoong64 Arch = "loong64"
|
||||
ArchMIPS64LE Arch = "mips64le"
|
||||
ArchMIPS64 Arch = "mips64"
|
||||
ArchMIPS64R6EL Arch = "mips64r6el"
|
||||
ArchMIPS64R6 Arch = "mips64r6"
|
||||
ArchMIPSLE Arch = "mipsle"
|
||||
ArchMIPS Arch = "mips"
|
||||
|
||||
// Universal (fat) binary architectures for macOS.
|
||||
ArchUniversal1 Arch = "universal1" // PPC + x86 (Rosetta 1 era)
|
||||
ArchUniversal2 Arch = "universal2" // x86_64 + ARM64 (Rosetta 2 era)
|
||||
)
|
||||
|
||||
// Libc represents the C library a binary is linked against.
|
||||
type Libc string
|
||||
|
||||
const (
|
||||
LibcNone Libc = "none" // statically linked or no libc dependency (Go, Zig, etc.)
|
||||
LibcGNU Libc = "gnu" // requires glibc (most Linux distros)
|
||||
LibcMusl Libc = "musl" // requires musl (Alpine, some Docker images)
|
||||
LibcMSVC Libc = "msvc" // Microsoft Visual C++ runtime
|
||||
)
|
||||
|
||||
// Format represents an archive or package format.
|
||||
type Format string
|
||||
|
||||
const (
|
||||
FormatTarGz Format = ".tar.gz"
|
||||
FormatTarXz Format = ".tar.xz"
|
||||
FormatTarZst Format = ".tar.zst"
|
||||
FormatTarBz2 Format = ".tar.bz2"
|
||||
FormatZip Format = ".zip"
|
||||
FormatGz Format = ".gz"
|
||||
FormatXz Format = ".xz"
|
||||
FormatZst Format = ".zst"
|
||||
FormatExe Format = ".exe"
|
||||
FormatExeXz Format = ".exe.xz"
|
||||
FormatMSI Format = ".msi"
|
||||
FormatDMG Format = ".dmg"
|
||||
FormatPkg Format = ".pkg"
|
||||
FormatAppZip Format = ".app.zip"
|
||||
Format7z Format = ".7z"
|
||||
FormatDeb Format = ".deb"
|
||||
FormatRPM Format = ".rpm"
|
||||
FormatSnap Format = ".snap"
|
||||
FormatAppx Format = ".appx"
|
||||
FormatAPK Format = ".apk"
|
||||
FormatAppImage Format = ".AppImage"
|
||||
FormatSh Format = ".sh"
|
||||
FormatGit Format = ".git"
|
||||
)
|
||||
|
||||
// Channel represents a release stability channel.
|
||||
type Channel string
|
||||
|
||||
const (
|
||||
ChannelStable Channel = "stable"
|
||||
ChannelLatest Channel = "latest"
|
||||
ChannelRC Channel = "rc"
|
||||
ChannelPreview Channel = "preview"
|
||||
ChannelBeta Channel = "beta"
|
||||
ChannelAlpha Channel = "alpha"
|
||||
ChannelDev Channel = "dev"
|
||||
)
|
||||
|
||||
// Target represents a fully resolved build target.
|
||||
type Target struct {
|
||||
OS OS
|
||||
Arch Arch
|
||||
Libc Libc
|
||||
}
|
||||
|
||||
// Triplet returns the canonical "os-arch-libc" string.
|
||||
func (t Target) Triplet() string {
|
||||
return string(t.OS) + "-" + string(t.Arch) + "-" + string(t.Libc)
|
||||
}
|
||||
|
||||
// CompatArches returns the architectures that the given OS+arch
|
||||
// combination can execute, ordered from most specific to least.
|
||||
// The input arch is always first.
|
||||
//
|
||||
// These are OS-level facts (hardware + translation layer), not
|
||||
// package-specific. Per-package overrides belong in installer config.
|
||||
func CompatArches(os OS, arch Arch) []Arch {
|
||||
switch os {
|
||||
case OSDarwin:
|
||||
switch arch {
|
||||
case ArchARM64:
|
||||
// Rosetta 2: Apple Silicon runs x86_64 binaries.
|
||||
return []Arch{ArchARM64, ArchUniversal2, ArchAMD64}
|
||||
case ArchAMD64:
|
||||
return []Arch{ArchAMD64, ArchUniversal2, ArchX86}
|
||||
}
|
||||
case OSWindows:
|
||||
switch arch {
|
||||
case ArchARM64:
|
||||
// Windows on ARM emulates x86_64 and x86.
|
||||
return []Arch{ArchARM64, ArchAMD64, ArchX86}
|
||||
}
|
||||
}
|
||||
|
||||
// Micro-architecture fallbacks (universal across all OSes).
|
||||
switch arch {
|
||||
case ArchAMD64v4:
|
||||
return []Arch{ArchAMD64v4, ArchAMD64v3, ArchAMD64v2, ArchAMD64}
|
||||
case ArchAMD64v3:
|
||||
return []Arch{ArchAMD64v3, ArchAMD64v2, ArchAMD64}
|
||||
case ArchAMD64v2:
|
||||
return []Arch{ArchAMD64v2, ArchAMD64}
|
||||
case ArchARMv7:
|
||||
return []Arch{ArchARMv7, ArchARMv6}
|
||||
}
|
||||
|
||||
return []Arch{arch}
|
||||
}
|
||||
|
||||
283
internal/classify/classify.go
Normal file
283
internal/classify/classify.go
Normal file
@@ -0,0 +1,283 @@
|
||||
// Package classify extracts build targets from release asset filenames.
|
||||
//
|
||||
// Standard toolchains (goreleaser, cargo-dist, zig build) produce predictable
|
||||
// filenames like "tool_0.1.0_linux_amd64.tar.gz" or
|
||||
// "tool-0.1.0-x86_64-unknown-linux-musl.tar.gz". This package matches those
|
||||
// patterns directly using regex, avoiding heuristic guessing.
|
||||
//
|
||||
// Detection order matters: architectures are checked longest-first to prevent
|
||||
// "x86" from matching inside "x86_64", and OS checks use word boundaries.
|
||||
package classify
|
||||
|
||||
import (
|
||||
"path"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/buildmeta"
|
||||
)
|
||||
|
||||
// Result holds the classification of an asset filename.
|
||||
type Result struct {
|
||||
OS buildmeta.OS
|
||||
Arch buildmeta.Arch
|
||||
Libc buildmeta.Libc
|
||||
Format buildmeta.Format
|
||||
}
|
||||
|
||||
// Target returns the build target (OS + Arch + Libc).
|
||||
func (r Result) Target() buildmeta.Target {
|
||||
return buildmeta.Target{OS: r.OS, Arch: r.Arch, Libc: r.Libc}
|
||||
}
|
||||
|
||||
// Filename classifies a release asset filename, returning the detected
|
||||
// OS, architecture, libc, and archive format. Undetected fields are empty.
|
||||
//
|
||||
// OS is detected first because it can influence arch interpretation.
|
||||
// For example, "windows-arm" in modern releases means ARM64, while
|
||||
// bare "arm" on Linux historically means ARMv6.
|
||||
func Filename(name string) Result {
|
||||
lower := strings.ToLower(name)
|
||||
os := detectOS(lower)
|
||||
arch := detectArch(lower)
|
||||
format := detectFormat(lower)
|
||||
|
||||
// .deb, .rpm, .snap are Linux-only package formats.
|
||||
if os == "" && (format == buildmeta.FormatDeb || format == buildmeta.FormatRPM || format == buildmeta.FormatSnap) {
|
||||
os = buildmeta.OSLinux
|
||||
}
|
||||
// .app.zip and .dmg are macOS-only formats.
|
||||
if os == "" && (format == buildmeta.FormatAppZip || format == buildmeta.FormatDMG) {
|
||||
os = buildmeta.OSDarwin
|
||||
}
|
||||
|
||||
return Result{
|
||||
OS: os,
|
||||
Arch: arch,
|
||||
Libc: detectLibc(lower),
|
||||
Format: format,
|
||||
}
|
||||
}
|
||||
|
||||
// b is a boundary: start/end of string or a non-alphanumeric separator.
|
||||
// Go's RE2 doesn't support \b, so we use this instead.
|
||||
const b = `(?:^|[^a-zA-Z0-9])`
|
||||
const bEnd = `(?:[^a-zA-Z0-9]|$)`
|
||||
|
||||
// --- OS detection ---
|
||||
|
||||
var osPatterns = []struct {
|
||||
os buildmeta.OS
|
||||
pattern *regexp.Regexp
|
||||
}{
|
||||
// macos[\d.]* matches versioned names like "macos10.10", "macos11", "macos12.0" (cmake naming).
|
||||
{buildmeta.OSDarwin, regexp.MustCompile(`(?i)(?:` + b + `(?:darwin|macos[\d.]*|macosx[\d.]*|osx[\d.]*|os-x|apple)` + bEnd + `|` + b + `mac` + bEnd + `)`)},
|
||||
// linux[\d.]* matches versioned names like "linux64", "linux32" (chromedriver/dashcore naming).
|
||||
{buildmeta.OSLinux, regexp.MustCompile(`(?i)` + b + `linux[\d.]*` + bEnd)},
|
||||
{buildmeta.OSWindows, regexp.MustCompile(`(?i)` + b + `(?:windows|win(?:32|64|x64|dows)?)` + bEnd + `|\.exe(?:\.xz)?$|\.msi$`)},
|
||||
// freebsd\d* matches versioned names like "freebsd13", "freebsd14" (Gitea naming).
|
||||
{buildmeta.OSFreeBSD, regexp.MustCompile(`(?i)` + b + `freebsd\d*` + bEnd)},
|
||||
{buildmeta.OSOpenBSD, regexp.MustCompile(`(?i)` + b + `openbsd` + bEnd)},
|
||||
{buildmeta.OSNetBSD, regexp.MustCompile(`(?i)` + b + `netbsd` + bEnd)},
|
||||
{buildmeta.OSDragonFly, regexp.MustCompile(`(?i)` + b + `dragonfly(?:bsd)?` + bEnd)},
|
||||
// solaris, illumos, and sunos are distinct OS values in the Node build-classifier.
|
||||
// Keep them separate so the legacy cache matches what the classifier extracts.
|
||||
{buildmeta.OSSolaris, regexp.MustCompile(`(?i)` + b + `solaris` + bEnd)},
|
||||
{buildmeta.OSIllumos, regexp.MustCompile(`(?i)` + b + `illumos` + bEnd)},
|
||||
{buildmeta.OSSunOS, regexp.MustCompile(`(?i)` + b + `sunos` + bEnd)},
|
||||
{buildmeta.OSAIX, regexp.MustCompile(`(?i)` + b + `aix` + bEnd)},
|
||||
{buildmeta.OSAndroid, regexp.MustCompile(`(?i)` + b + `android` + bEnd)},
|
||||
{buildmeta.OSPlan9, regexp.MustCompile(`(?i)` + b + `plan9` + bEnd)},
|
||||
}
|
||||
|
||||
func detectOS(lower string) buildmeta.OS {
|
||||
for _, p := range osPatterns {
|
||||
if p.pattern.MatchString(lower) {
|
||||
return p.os
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// --- Arch detection ---
|
||||
// Order matters: check longer/more-specific patterns first.
|
||||
|
||||
var archPatterns = []struct {
|
||||
arch buildmeta.Arch
|
||||
pattern *regexp.Regexp
|
||||
}{
|
||||
// Universal/fat binaries before specific arches.
|
||||
{buildmeta.ArchUniversal2, regexp.MustCompile(`(?i)` + b + `(?:universal2?|fat)` + bEnd)},
|
||||
// amd64 micro-levels before baseline — "amd64v3" must not fall through to amd64.
|
||||
// amd64_?vN: underscore optional but no dash — dash is ambiguous with version numbers
|
||||
// (e.g. syncthing "amd64-v2.0.5" where v2 is the release version, not an arch level).
|
||||
{buildmeta.ArchAMD64v4, regexp.MustCompile(`(?i)(?:x86[_-]64[_-]v4|amd64_?v4|v4-amd64)`)},
|
||||
{buildmeta.ArchAMD64v3, regexp.MustCompile(`(?i)(?:x86[_-]64[_-]v3|amd64_?v3|v3-amd64)`)},
|
||||
{buildmeta.ArchAMD64v2, regexp.MustCompile(`(?i)(?:x86[_-]64[_-]v2|amd64_?v2|v2-amd64)`)},
|
||||
// amd64 baseline before x86 — "x86_64" must not match as x86.
|
||||
{buildmeta.ArchAMD64, regexp.MustCompile(`(?i)(?:x86[_-]64|amd64|x64|win64)`)},
|
||||
// arm64 before armv7/armv6 — "aarch64" must not match as arm.
|
||||
{buildmeta.ArchARM64, regexp.MustCompile(`(?i)(?:aarch64|arm64|armv8)`)},
|
||||
{buildmeta.ArchARMv7, regexp.MustCompile(`(?i)(?:armv7l?|arm-?v7|arm7|arm32|armhf)`)},
|
||||
// armel and gnueabihf are ARMv6 soft/hard-float ABI names used in Debian and Rust triplets.
|
||||
{buildmeta.ArchARMv6, regexp.MustCompile(`(?i)(?:armv6l?|arm-?v6|aarch32|armel|gnueabihf|` + b + `arm` + bEnd + `)`)},
|
||||
{buildmeta.ArchARMv5, regexp.MustCompile(`(?i)(?:armv5)`)},
|
||||
// powerpc64le/ppc64le before powerpc64/ppc64 before powerpc32.
|
||||
// The longer powerpc* forms must come first to prevent shorter matches from
|
||||
// winning. All powerpc entries must appear BEFORE ARM patterns — otherwise
|
||||
// "powerpc-linux-gnueabihf" would match gnueabihf → ARMv6.
|
||||
// ppc64el is an alternative spelling used in Debian/Ubuntu.
|
||||
{buildmeta.ArchPPC64LE, regexp.MustCompile(`(?i)(?:powerpc64le|ppc64le|ppc64el)`)},
|
||||
{buildmeta.ArchPPC64, regexp.MustCompile(`(?i)(?:powerpc64|ppc64)`)},
|
||||
// powerpc (32-bit): webi does not serve powerpc32, but we must classify it
|
||||
// here to prevent the gnueabihf suffix from matching the ARMv6 pattern.
|
||||
{buildmeta.ArchPPC, regexp.MustCompile(`(?i)` + b + `powerpc` + bEnd)},
|
||||
{buildmeta.ArchRISCV64, regexp.MustCompile(`(?i)riscv64`)},
|
||||
{buildmeta.ArchS390X, regexp.MustCompile(`(?i)s390x`)},
|
||||
{buildmeta.ArchLoong64, regexp.MustCompile(`(?i)loong(?:arch)?64`)},
|
||||
// mips64r6 before mips64 — "mips64r6" contains "mips64" as a prefix.
|
||||
{buildmeta.ArchMIPS64R6EL, regexp.MustCompile(`(?i)mips64r6e(?:l|le)`)},
|
||||
{buildmeta.ArchMIPS64R6, regexp.MustCompile(`(?i)mips64r6`)},
|
||||
{buildmeta.ArchMIPS64LE, regexp.MustCompile(`(?i)mips64(?:el|le)`)},
|
||||
{buildmeta.ArchMIPS64, regexp.MustCompile(`(?i)mips64`)},
|
||||
{buildmeta.ArchMIPSLE, regexp.MustCompile(`(?i)mips(?:el|le)`)},
|
||||
{buildmeta.ArchMIPS, regexp.MustCompile(`(?i)` + b + `mips` + bEnd)},
|
||||
// x86 last — must not steal x86_64.
|
||||
{buildmeta.ArchX86, regexp.MustCompile(`(?i)(?:` + b + `x86` + bEnd + `|i[3-6]86|ia32|win32|` + b + `386` + bEnd + `)`)},
|
||||
}
|
||||
|
||||
func detectArch(lower string) buildmeta.Arch {
|
||||
for _, p := range archPatterns {
|
||||
if p.pattern.MatchString(lower) {
|
||||
return p.arch
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// --- Libc detection ---
|
||||
|
||||
var (
|
||||
reMusl = regexp.MustCompile(`(?i)` + b + `musl` + bEnd)
|
||||
reGNU = regexp.MustCompile(`(?i)` + b + `(?:gnu|glibc)` + bEnd)
|
||||
reMSVC = regexp.MustCompile(`(?i)` + b + `msvc` + bEnd)
|
||||
reStatic = regexp.MustCompile(`(?i)` + b + `static` + bEnd)
|
||||
)
|
||||
|
||||
func detectLibc(lower string) buildmeta.Libc {
|
||||
switch {
|
||||
case reMusl.MatchString(lower):
|
||||
return buildmeta.LibcMusl
|
||||
case reGNU.MatchString(lower):
|
||||
return buildmeta.LibcGNU
|
||||
case reMSVC.MatchString(lower):
|
||||
return buildmeta.LibcMSVC
|
||||
case reStatic.MatchString(lower):
|
||||
return buildmeta.LibcNone
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// --- Format detection ---
|
||||
|
||||
// formatSuffixes maps file extensions to formats, longest first.
|
||||
var formatSuffixes = []struct {
|
||||
suffix string
|
||||
format buildmeta.Format
|
||||
}{
|
||||
{".tar.gz", buildmeta.FormatTarGz},
|
||||
{".tar.xz", buildmeta.FormatTarXz},
|
||||
{".tar.zst", buildmeta.FormatTarZst},
|
||||
{".tar.bz2", buildmeta.FormatTarBz2},
|
||||
{".exe.xz", buildmeta.FormatExeXz},
|
||||
{".app.zip", buildmeta.FormatAppZip},
|
||||
{".tgz", buildmeta.FormatTarGz},
|
||||
{".zip", buildmeta.FormatZip},
|
||||
{".gz", buildmeta.FormatGz},
|
||||
{".xz", buildmeta.FormatXz},
|
||||
{".zst", buildmeta.FormatZst},
|
||||
{".7z", buildmeta.Format7z},
|
||||
{".exe", buildmeta.FormatExe},
|
||||
{".msi", buildmeta.FormatMSI},
|
||||
{".dmg", buildmeta.FormatDMG},
|
||||
{".deb", buildmeta.FormatDeb},
|
||||
{".rpm", buildmeta.FormatRPM},
|
||||
{".snap", buildmeta.FormatSnap},
|
||||
{".appx", buildmeta.FormatAppx},
|
||||
{".apk", buildmeta.FormatAPK},
|
||||
{".AppImage", buildmeta.FormatAppImage},
|
||||
{".pkg", buildmeta.FormatPkg},
|
||||
}
|
||||
|
||||
func detectFormat(lower string) buildmeta.Format {
|
||||
// Use the base name to avoid directory separators confusing suffix matching.
|
||||
base := path.Base(lower)
|
||||
for _, s := range formatSuffixes {
|
||||
if strings.HasSuffix(base, s.suffix) {
|
||||
return s.format
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// IsMetaAsset returns true if the filename is a non-installable meta file
|
||||
// (checksums, signatures, source tarballs, documentation, etc.).
|
||||
func IsMetaAsset(name string) bool {
|
||||
lower := strings.ToLower(name)
|
||||
for _, suffix := range []string{
|
||||
".txt",
|
||||
".sha256",
|
||||
".sha256sum",
|
||||
".sha512",
|
||||
".sha512sum",
|
||||
".md5",
|
||||
".md5sum",
|
||||
".sig",
|
||||
".asc",
|
||||
".pem",
|
||||
".sbom",
|
||||
".spdx",
|
||||
".json.sig",
|
||||
".sigstore",
|
||||
".minisig",
|
||||
"_src.tar.gz",
|
||||
"_src.tar.xz",
|
||||
"_src.zip",
|
||||
"-src.tar.gz",
|
||||
".src.tar.gz",
|
||||
"-src.tar.xz",
|
||||
"-src.zip",
|
||||
".d.ts",
|
||||
".pub",
|
||||
".bsdiff",
|
||||
".flatpak",
|
||||
} {
|
||||
if strings.HasSuffix(lower, suffix) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
for _, substr := range []string{
|
||||
"checksums",
|
||||
"sha256sum",
|
||||
"sha512sum",
|
||||
"buildable-artifact",
|
||||
".LICENSE",
|
||||
".README",
|
||||
} {
|
||||
if strings.Contains(lower, substr) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
for _, exact := range []string{
|
||||
"install.sh",
|
||||
"install.ps1",
|
||||
"compat.json",
|
||||
"b3sums",
|
||||
"dist-manifest.json",
|
||||
} {
|
||||
if lower == exact {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
352
internal/classify/classify_test.go
Normal file
352
internal/classify/classify_test.go
Normal file
@@ -0,0 +1,352 @@
|
||||
package classify_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/buildmeta"
|
||||
"github.com/webinstall/webi-installers/internal/classify"
|
||||
)
|
||||
|
||||
func TestFilename(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
wantOS buildmeta.OS
|
||||
arch buildmeta.Arch
|
||||
libc buildmeta.Libc
|
||||
format buildmeta.Format
|
||||
}{
|
||||
// Goreleaser-style
|
||||
{
|
||||
name: "goreleaser linux amd64 tar.gz",
|
||||
input: "hugo_0.145.0_linux-amd64.tar.gz",
|
||||
wantOS: buildmeta.OSLinux,
|
||||
arch: buildmeta.ArchAMD64,
|
||||
format: buildmeta.FormatTarGz,
|
||||
},
|
||||
{
|
||||
name: "goreleaser darwin arm64 tar.gz",
|
||||
input: "hugo_0.145.0_darwin-arm64.tar.gz",
|
||||
wantOS: buildmeta.OSDarwin,
|
||||
arch: buildmeta.ArchARM64,
|
||||
format: buildmeta.FormatTarGz,
|
||||
},
|
||||
{
|
||||
name: "goreleaser windows amd64 zip",
|
||||
input: "hugo_0.145.0_windows-amd64.zip",
|
||||
wantOS: buildmeta.OSWindows,
|
||||
arch: buildmeta.ArchAMD64,
|
||||
format: buildmeta.FormatZip,
|
||||
},
|
||||
{
|
||||
name: "goreleaser freebsd",
|
||||
input: "hugo_0.145.0_freebsd-amd64.tar.gz",
|
||||
wantOS: buildmeta.OSFreeBSD,
|
||||
arch: buildmeta.ArchAMD64,
|
||||
format: buildmeta.FormatTarGz,
|
||||
},
|
||||
|
||||
// Rust/cargo-dist style
|
||||
{
|
||||
name: "rust linux musl",
|
||||
input: "ripgrep-14.1.1-x86_64-unknown-linux-musl.tar.gz",
|
||||
wantOS: buildmeta.OSLinux,
|
||||
arch: buildmeta.ArchAMD64,
|
||||
libc: buildmeta.LibcMusl,
|
||||
format: buildmeta.FormatTarGz,
|
||||
},
|
||||
{
|
||||
name: "rust linux gnu",
|
||||
input: "bat-v0.24.0-x86_64-unknown-linux-gnu.tar.gz",
|
||||
wantOS: buildmeta.OSLinux,
|
||||
arch: buildmeta.ArchAMD64,
|
||||
libc: buildmeta.LibcGNU,
|
||||
format: buildmeta.FormatTarGz,
|
||||
},
|
||||
{
|
||||
name: "rust apple darwin",
|
||||
input: "ripgrep-14.1.1-x86_64-apple-darwin.tar.gz",
|
||||
wantOS: buildmeta.OSDarwin,
|
||||
arch: buildmeta.ArchAMD64,
|
||||
format: buildmeta.FormatTarGz,
|
||||
},
|
||||
{
|
||||
name: "rust windows msvc",
|
||||
input: "bat-v0.24.0-x86_64-pc-windows-msvc.zip",
|
||||
wantOS: buildmeta.OSWindows,
|
||||
arch: buildmeta.ArchAMD64,
|
||||
libc: buildmeta.LibcMSVC,
|
||||
format: buildmeta.FormatZip,
|
||||
},
|
||||
{
|
||||
name: "rust aarch64 linux",
|
||||
input: "ripgrep-14.1.1-aarch64-unknown-linux-gnu.tar.gz",
|
||||
wantOS: buildmeta.OSLinux,
|
||||
arch: buildmeta.ArchARM64,
|
||||
libc: buildmeta.LibcGNU,
|
||||
format: buildmeta.FormatTarGz,
|
||||
},
|
||||
|
||||
// Zig-style
|
||||
{
|
||||
name: "zig linux x86_64",
|
||||
input: "zig-linux-x86_64-0.14.0.tar.xz",
|
||||
wantOS: buildmeta.OSLinux,
|
||||
arch: buildmeta.ArchAMD64,
|
||||
format: buildmeta.FormatTarXz,
|
||||
},
|
||||
{
|
||||
name: "zig macos aarch64",
|
||||
input: "zig-macos-aarch64-0.14.0.tar.xz",
|
||||
wantOS: buildmeta.OSDarwin,
|
||||
arch: buildmeta.ArchARM64,
|
||||
format: buildmeta.FormatTarXz,
|
||||
},
|
||||
|
||||
// Windows executables
|
||||
{
|
||||
name: "bare exe",
|
||||
input: "jq-windows-amd64.exe",
|
||||
wantOS: buildmeta.OSWindows,
|
||||
arch: buildmeta.ArchAMD64,
|
||||
format: buildmeta.FormatExe,
|
||||
},
|
||||
{
|
||||
name: "msi installer",
|
||||
input: "caddy_2.9.0_windows_amd64.msi",
|
||||
wantOS: buildmeta.OSWindows,
|
||||
arch: buildmeta.ArchAMD64,
|
||||
format: buildmeta.FormatMSI,
|
||||
},
|
||||
|
||||
// macOS formats
|
||||
{
|
||||
name: "dmg installer",
|
||||
input: "MyApp-1.0.0-darwin-arm64.dmg",
|
||||
wantOS: buildmeta.OSDarwin,
|
||||
arch: buildmeta.ArchARM64,
|
||||
format: buildmeta.FormatDMG,
|
||||
},
|
||||
|
||||
// Arch priority: x86_64 must not match x86
|
||||
{
|
||||
name: "x86_64 not x86",
|
||||
input: "tool-x86_64-linux.tar.gz",
|
||||
wantOS: buildmeta.OSLinux,
|
||||
arch: buildmeta.ArchAMD64,
|
||||
format: buildmeta.FormatTarGz,
|
||||
},
|
||||
{
|
||||
name: "actual x86",
|
||||
input: "tool-x86-linux.tar.gz",
|
||||
wantOS: buildmeta.OSLinux,
|
||||
arch: buildmeta.ArchX86,
|
||||
format: buildmeta.FormatTarGz,
|
||||
},
|
||||
{
|
||||
name: "i386",
|
||||
input: "tool-linux-i386.tar.gz",
|
||||
wantOS: buildmeta.OSLinux,
|
||||
arch: buildmeta.ArchX86,
|
||||
format: buildmeta.FormatTarGz,
|
||||
},
|
||||
|
||||
// Windows ARM: bare "arm" is armv6 (some tools ship genuine arm32 Windows builds).
|
||||
// Explicit "arm64" is always aarch64 regardless of OS.
|
||||
{
|
||||
name: "windows bare arm stays armv6",
|
||||
input: "tool-1.0.0-windows-arm.zip",
|
||||
wantOS: buildmeta.OSWindows,
|
||||
arch: buildmeta.ArchARMv6,
|
||||
format: buildmeta.FormatZip,
|
||||
},
|
||||
{
|
||||
name: "windows armv6 stays armv6",
|
||||
input: "tool-2.0.0-windows-armv6.zip",
|
||||
wantOS: buildmeta.OSWindows,
|
||||
arch: buildmeta.ArchARMv6,
|
||||
format: buildmeta.FormatZip,
|
||||
},
|
||||
{
|
||||
name: "windows arm64 stays arm64",
|
||||
input: "tool-1.0.0-windows-arm64.zip",
|
||||
wantOS: buildmeta.OSWindows,
|
||||
arch: buildmeta.ArchARM64,
|
||||
format: buildmeta.FormatZip,
|
||||
},
|
||||
|
||||
// armel and gnueabihf are ARMv6 ABI names
|
||||
{
|
||||
name: "armel is armv6",
|
||||
input: "jq-linux-armel",
|
||||
wantOS: buildmeta.OSLinux,
|
||||
arch: buildmeta.ArchARMv6,
|
||||
},
|
||||
{
|
||||
name: "gnueabihf is armv6",
|
||||
input: "tool-arm-unknown-linux-gnueabihf.tar.gz",
|
||||
wantOS: buildmeta.OSLinux,
|
||||
arch: buildmeta.ArchARMv6,
|
||||
format: buildmeta.FormatTarGz,
|
||||
},
|
||||
|
||||
// winx64 is a Windows x86_64 naming used by MariaDB
|
||||
{
|
||||
name: "winx64 is windows x86_64",
|
||||
input: "mariadb-11.4.5-winx64.zip",
|
||||
wantOS: buildmeta.OSWindows,
|
||||
arch: buildmeta.ArchAMD64,
|
||||
format: buildmeta.FormatZip,
|
||||
},
|
||||
|
||||
// win32/win64 naming used by chromedriver, dashcore, etc.
|
||||
{
|
||||
name: "win32 is windows x86",
|
||||
input: "chromedriver-win32.zip",
|
||||
wantOS: buildmeta.OSWindows,
|
||||
arch: buildmeta.ArchX86,
|
||||
format: buildmeta.FormatZip,
|
||||
},
|
||||
{
|
||||
name: "win64 is windows amd64",
|
||||
input: "dashcore-23.1.2-win64-setup.exe",
|
||||
wantOS: buildmeta.OSWindows,
|
||||
arch: buildmeta.ArchAMD64,
|
||||
format: buildmeta.FormatExe,
|
||||
},
|
||||
|
||||
// ppc64el is a Debian/Ubuntu alias for ppc64le
|
||||
{
|
||||
name: "ppc64el is ppc64le",
|
||||
input: "jq-linux-ppc64el",
|
||||
arch: buildmeta.ArchPPC64LE,
|
||||
},
|
||||
|
||||
// amd64 micro-architecture levels
|
||||
{
|
||||
name: "amd64v2",
|
||||
input: "tool-linux-amd64v2.tar.gz",
|
||||
arch: buildmeta.ArchAMD64v2,
|
||||
},
|
||||
{
|
||||
name: "amd64v3",
|
||||
input: "tool-linux-x86_64_v3.tar.gz",
|
||||
arch: buildmeta.ArchAMD64v3,
|
||||
},
|
||||
{
|
||||
name: "amd64v4",
|
||||
input: "tool-linux-amd64v4.tar.gz",
|
||||
arch: buildmeta.ArchAMD64v4,
|
||||
},
|
||||
{
|
||||
name: "amd64v3 not baseline",
|
||||
input: "tool-1.0.0-linux-amd64v3.tar.gz",
|
||||
wantOS: buildmeta.OSLinux,
|
||||
arch: buildmeta.ArchAMD64v3,
|
||||
format: buildmeta.FormatTarGz,
|
||||
},
|
||||
|
||||
// ARM variants: arm64 must not match armv7/armv6
|
||||
{
|
||||
name: "aarch64 not armv7",
|
||||
input: "tool-aarch64-linux.tar.gz",
|
||||
arch: buildmeta.ArchARM64,
|
||||
},
|
||||
{
|
||||
name: "armv7",
|
||||
input: "tool-armv7l-linux.tar.gz",
|
||||
arch: buildmeta.ArchARMv7,
|
||||
},
|
||||
{
|
||||
name: "armv6",
|
||||
input: "tool-armv6l-linux.tar.gz",
|
||||
arch: buildmeta.ArchARMv6,
|
||||
},
|
||||
|
||||
// ppc64le before ppc64
|
||||
{
|
||||
name: "ppc64le",
|
||||
input: "tool-linux-ppc64le.tar.gz",
|
||||
arch: buildmeta.ArchPPC64LE,
|
||||
},
|
||||
{
|
||||
name: "ppc64",
|
||||
input: "tool-linux-ppc64.tar.gz",
|
||||
arch: buildmeta.ArchPPC64,
|
||||
},
|
||||
|
||||
// Static linking
|
||||
{
|
||||
name: "static binary",
|
||||
input: "tool-linux-amd64-static.tar.gz",
|
||||
libc: buildmeta.LibcNone,
|
||||
},
|
||||
|
||||
// .exe implies Windows
|
||||
{
|
||||
name: "exe implies windows",
|
||||
input: "tool-amd64.exe",
|
||||
wantOS: buildmeta.OSWindows,
|
||||
arch: buildmeta.ArchAMD64,
|
||||
format: buildmeta.FormatExe,
|
||||
},
|
||||
|
||||
// Compound extensions
|
||||
{
|
||||
name: "tar.zst",
|
||||
input: "tool-linux-amd64.tar.zst",
|
||||
format: buildmeta.FormatTarZst,
|
||||
},
|
||||
{
|
||||
name: "exe.xz",
|
||||
input: "tool-windows-amd64.exe.xz",
|
||||
format: buildmeta.FormatExeXz,
|
||||
},
|
||||
{
|
||||
name: "app.zip",
|
||||
input: "MyApp-1.0.0.app.zip",
|
||||
format: buildmeta.FormatAppZip,
|
||||
},
|
||||
{
|
||||
name: "tgz alias",
|
||||
input: "tool-linux-amd64.tgz",
|
||||
format: buildmeta.FormatTarGz,
|
||||
},
|
||||
|
||||
// s390x, mips
|
||||
{
|
||||
name: "s390x",
|
||||
input: "tool-linux-s390x.tar.gz",
|
||||
arch: buildmeta.ArchS390X,
|
||||
},
|
||||
{
|
||||
name: "mips64",
|
||||
input: "tool-linux-mips64.tar.gz",
|
||||
arch: buildmeta.ArchMIPS64,
|
||||
},
|
||||
|
||||
// Unknown / no match
|
||||
{
|
||||
name: "checksum file",
|
||||
input: "checksums.txt",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := classify.Filename(tt.input)
|
||||
if tt.wantOS != "" && got.OS != tt.wantOS {
|
||||
t.Errorf("OS = %q, want %q", got.OS, tt.wantOS)
|
||||
}
|
||||
if tt.arch != "" && got.Arch != tt.arch {
|
||||
t.Errorf("Arch = %q, want %q", got.Arch, tt.arch)
|
||||
}
|
||||
if tt.libc != "" && got.Libc != tt.libc {
|
||||
t.Errorf("Libc = %q, want %q", got.Libc, tt.libc)
|
||||
}
|
||||
if tt.format != "" && got.Format != tt.format {
|
||||
t.Errorf("Format = %q, want %q", got.Format, tt.format)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
1439
internal/classifypkg/classifypkg.go
Normal file
1439
internal/classifypkg/classifypkg.go
Normal file
File diff suppressed because it is too large
Load Diff
154
internal/httpclient/httpclient.go
Normal file
154
internal/httpclient/httpclient.go
Normal file
@@ -0,0 +1,154 @@
|
||||
// Package httpclient provides a well-configured [http.Client] for upstream
|
||||
// API calls. It exists because [http.DefaultClient] has no timeouts, no TLS
|
||||
// minimum, and follows redirects from HTTPS to HTTP — none of which are
|
||||
// acceptable for a server calling GitHub, Gitea, etc. on behalf of users.
|
||||
//
|
||||
// Use [New] to create a configured client. Use [Do] to execute a request
|
||||
// with automatic retries for transient failures.
|
||||
package httpclient
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/tls"
|
||||
"errors"
|
||||
"fmt"
|
||||
"math/rand/v2"
|
||||
"net"
|
||||
"net/http"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
const userAgent = "Webi/2.0 (+https://webinstall.dev)"
|
||||
|
||||
// New returns an [http.Client] with secure, production-ready defaults:
|
||||
// TLS 1.2+, timeouts at every level, connection pooling, no HTTPS→HTTP
|
||||
// redirect, and a Webi User-Agent.
|
||||
func New() *http.Client {
|
||||
return &http.Client{
|
||||
Transport: &http.Transport{
|
||||
DialContext: (&net.Dialer{
|
||||
Timeout: 10 * time.Second,
|
||||
KeepAlive: 30 * time.Second,
|
||||
}).DialContext,
|
||||
TLSClientConfig: &tls.Config{
|
||||
MinVersion: tls.VersionTLS12,
|
||||
},
|
||||
TLSHandshakeTimeout: 10 * time.Second,
|
||||
ResponseHeaderTimeout: 30 * time.Second,
|
||||
MaxIdleConns: 100,
|
||||
MaxIdleConnsPerHost: 10,
|
||||
IdleConnTimeout: 90 * time.Second,
|
||||
ExpectContinueTimeout: 1 * time.Second,
|
||||
ForceAttemptHTTP2: true,
|
||||
},
|
||||
Timeout: 60 * time.Second,
|
||||
CheckRedirect: checkRedirect,
|
||||
}
|
||||
}
|
||||
|
||||
// checkRedirect prevents HTTPS→HTTP downgrades and limits redirect depth.
|
||||
func checkRedirect(req *http.Request, via []*http.Request) error {
|
||||
if len(via) >= 10 {
|
||||
return fmt.Errorf("stopped after %d redirects", len(via))
|
||||
}
|
||||
if len(via) > 0 && via[0].URL.Scheme == "https" && req.URL.Scheme == "http" {
|
||||
return errors.New("refused redirect from https to http")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Get performs a GET request with the Webi User-Agent header.
|
||||
func Get(ctx context.Context, client *http.Client, url string) (*http.Response, error) {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
req.Header.Set("User-Agent", userAgent)
|
||||
return client.Do(req)
|
||||
}
|
||||
|
||||
// Do executes a request with automatic retries for transient errors (429,
|
||||
// 502, 503, 504). Retries up to 3 times with exponential backoff and jitter.
|
||||
// Respects Retry-After headers. Only retries GET and HEAD (idempotent).
|
||||
//
|
||||
// Sets the Webi User-Agent header if not already present.
|
||||
func Do(ctx context.Context, client *http.Client, req *http.Request) (*http.Response, error) {
|
||||
if req.Header.Get("User-Agent") == "" {
|
||||
req.Header.Set("User-Agent", userAgent)
|
||||
}
|
||||
|
||||
// Only retry idempotent methods.
|
||||
idempotent := req.Method == http.MethodGet || req.Method == http.MethodHead
|
||||
|
||||
const maxRetries = 3
|
||||
var resp *http.Response
|
||||
var err error
|
||||
|
||||
for attempt := range maxRetries + 1 {
|
||||
if attempt > 0 {
|
||||
if !idempotent {
|
||||
break
|
||||
}
|
||||
|
||||
delay := backoff(attempt, resp)
|
||||
timer := time.NewTimer(delay)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
timer.Stop()
|
||||
return nil, ctx.Err()
|
||||
case <-timer.C:
|
||||
}
|
||||
|
||||
if resp != nil {
|
||||
resp.Body.Close()
|
||||
}
|
||||
}
|
||||
|
||||
resp, err = client.Do(req)
|
||||
if err != nil {
|
||||
if ctx.Err() != nil {
|
||||
return nil, ctx.Err()
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
if !isRetryable(resp.StatusCode) {
|
||||
return resp, nil
|
||||
}
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("after %d retries: %w", maxRetries, err)
|
||||
}
|
||||
return resp, nil
|
||||
}
|
||||
|
||||
func isRetryable(status int) bool {
|
||||
return status == http.StatusTooManyRequests ||
|
||||
status == http.StatusBadGateway ||
|
||||
status == http.StatusServiceUnavailable ||
|
||||
status == http.StatusGatewayTimeout
|
||||
}
|
||||
|
||||
// backoff returns a delay before the next retry. Respects Retry-After,
|
||||
// otherwise uses exponential backoff with jitter.
|
||||
func backoff(attempt int, resp *http.Response) time.Duration {
|
||||
if resp != nil {
|
||||
if ra := resp.Header.Get("Retry-After"); ra != "" {
|
||||
if seconds, err := strconv.Atoi(ra); err == nil && seconds > 0 && seconds < 300 {
|
||||
return time.Duration(seconds) * time.Second
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 1s, 2s, 4s base delays
|
||||
base := time.Second << (attempt - 1)
|
||||
if base > 30*time.Second {
|
||||
base = 30 * time.Second
|
||||
}
|
||||
|
||||
// Add jitter: 75% to 125% of base
|
||||
jitter := float64(base) * (0.75 + 0.5*rand.Float64())
|
||||
return time.Duration(jitter)
|
||||
}
|
||||
286
internal/installerconf/installerconf.go
Normal file
286
internal/installerconf/installerconf.go
Normal file
@@ -0,0 +1,286 @@
|
||||
// Package installerconf reads per-package releases.conf files.
|
||||
//
|
||||
// The format is simple key=value, one per line. Blank lines and lines
|
||||
// starting with # are ignored. Keys and values are trimmed of whitespace.
|
||||
// Multi-value keys are whitespace-delimited.
|
||||
//
|
||||
// The source type is inferred from the primary key:
|
||||
//
|
||||
// GitHub binary releases:
|
||||
//
|
||||
// github_releases = sharkdp/bat
|
||||
// github_releases = https://github.com/sharkdp/bat
|
||||
//
|
||||
// GitHub source archives (for source-installable packages):
|
||||
//
|
||||
// github_sources = BeyondCodeBootcamp/aliasman
|
||||
// git_url = https://github.com/BeyondCodeBootcamp/aliasman.git
|
||||
//
|
||||
// Gitea binary releases (self-hosted, requires full URL or base_url):
|
||||
//
|
||||
// gitea_releases = https://git.rootprojects.org/root/pathman
|
||||
//
|
||||
// GitLab binary releases (defaults to gitlab.com):
|
||||
//
|
||||
// gitlab_releases = owner/repo
|
||||
// gitlab_releases = https://gitlab.example.com/owner/repo
|
||||
//
|
||||
// Git tag enumeration (vim plugins, etc.):
|
||||
//
|
||||
// git_url = https://github.com/tpope/vim-commentary.git
|
||||
//
|
||||
// HashiCorp releases:
|
||||
//
|
||||
// hashicorp_product = terraform
|
||||
//
|
||||
// Other sources (one-off scrapers):
|
||||
//
|
||||
// source = nodedist
|
||||
// url = https://nodejs.org/download/release
|
||||
//
|
||||
// Complex packages that need custom logic beyond what the classifier
|
||||
// auto-detects (e.g. ollama's universal binaries, ffmpeg's non-standard
|
||||
// naming) should put that logic in Go code, not in the config.
|
||||
// The variants key documents known build variants for human readers;
|
||||
// actual variant detection logic lives in Go.
|
||||
package installerconf
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"fmt"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Conf holds the parsed per-package release configuration.
|
||||
type Conf struct {
|
||||
// Source is the fetch source type: "github", "githubsource",
|
||||
// "gitea", "giteasource", "gitlab", "gitlabsource",
|
||||
// "gittag", "nodedist", etc.
|
||||
Source string
|
||||
|
||||
// Owner is the repository owner (org or user).
|
||||
Owner string
|
||||
|
||||
// Repo is the repository name.
|
||||
Repo string
|
||||
|
||||
// BaseURL is a custom base URL for non-GitHub sources
|
||||
// (e.g. a Gitea instance or nodedist index URL).
|
||||
BaseURL string
|
||||
|
||||
// GitURL is the git clone URL for source-installable packages.
|
||||
// Present alongside github_sources/gitea_sources to provide a
|
||||
// git clone fallback in addition to release tarballs.
|
||||
GitURL string
|
||||
|
||||
// TagPrefix filters releases in monorepos. Only tags starting with
|
||||
// this prefix are included, and the prefix is stripped from the
|
||||
// version string. Example: "tools/monorel/"
|
||||
TagPrefix string
|
||||
|
||||
// VersionPrefixes are stripped from version/tag strings.
|
||||
// Whitespace-delimited. Each release tag is checked against these
|
||||
// in order; the first match is stripped. Projects may change tag
|
||||
// conventions across versions (e.g. "jq-1.7.1" older, "1.8.0" later).
|
||||
VersionPrefixes []string
|
||||
|
||||
// Exclude lists filename substrings to filter out.
|
||||
// Whitespace-delimited. Assets whose name contains any of these
|
||||
// are skipped entirely (not stored).
|
||||
Exclude []string
|
||||
|
||||
// AssetFilter is a substring that asset filenames must contain.
|
||||
// Used when multiple packages share a GitHub release (e.g.
|
||||
// kubectx/kubens) to select only the relevant assets.
|
||||
AssetFilter string
|
||||
|
||||
// Variants documents known build variant names for this package.
|
||||
// Whitespace-delimited. This is a human-readable cue — actual
|
||||
// variant detection logic lives in Go code per-package.
|
||||
Variants []string
|
||||
|
||||
// OS restricts all assets to this OS value when set.
|
||||
// Use "posix_2017" for POSIX-only shell packages that don't
|
||||
// support Windows.
|
||||
OS string
|
||||
|
||||
// AliasOf names another package that this one mirrors.
|
||||
// When set, the package has no releases of its own — it shares
|
||||
// the cache output of the named target (e.g. dashd → dashcore).
|
||||
AliasOf string
|
||||
|
||||
// Extra holds any unrecognized keys for forward compatibility.
|
||||
Extra map[string]string
|
||||
}
|
||||
|
||||
// parseRepoRef parses a value that is either "owner/repo" or a full URL
|
||||
// like "https://github.com/owner/repo". Returns baseURL, owner, repo.
|
||||
// For short form, baseURL is empty (caller uses the default for the forge).
|
||||
// For full URL form, baseURL is the scheme+host (e.g. "https://github.com").
|
||||
func parseRepoRef(val, defaultBase string) (baseURL, owner, repo string) {
|
||||
if strings.Contains(val, "://") {
|
||||
u, err := url.Parse(val)
|
||||
if err == nil {
|
||||
baseURL = u.Scheme + "://" + u.Host
|
||||
path := strings.Trim(u.Path, "/")
|
||||
owner, repo, _ = strings.Cut(path, "/")
|
||||
return baseURL, owner, repo
|
||||
}
|
||||
}
|
||||
// Short form: "owner/repo"
|
||||
owner, repo, _ = strings.Cut(val, "/")
|
||||
return defaultBase, owner, repo
|
||||
}
|
||||
|
||||
// Read parses a releases.conf file.
|
||||
func Read(path string) (*Conf, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("installerconf: %w", err)
|
||||
}
|
||||
defer f.Close()
|
||||
|
||||
raw := make(map[string]string)
|
||||
scanner := bufio.NewScanner(f)
|
||||
for scanner.Scan() {
|
||||
line := strings.TrimSpace(scanner.Text())
|
||||
if line == "" || line[0] == '#' {
|
||||
continue
|
||||
}
|
||||
key, val, ok := strings.Cut(line, "=")
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
raw[strings.TrimSpace(key)] = strings.TrimSpace(val)
|
||||
}
|
||||
if err := scanner.Err(); err != nil {
|
||||
return nil, fmt.Errorf("installerconf: read %s: %w", path, err)
|
||||
}
|
||||
|
||||
c := &Conf{}
|
||||
|
||||
// Infer source from primary key, falling back to explicit "source".
|
||||
// When both github_releases and source are set, parse the repo ref
|
||||
// from github_releases but use the explicit source for classification.
|
||||
switch {
|
||||
// GitHub binary releases.
|
||||
case raw["github_releases"] != "":
|
||||
c.Source = "github"
|
||||
c.BaseURL, c.Owner, c.Repo = parseRepoRef(raw["github_releases"], "https://github.com")
|
||||
|
||||
// GitHub source tarballs.
|
||||
case raw["github_sources"] != "":
|
||||
c.Source = "githubsource"
|
||||
c.BaseURL, c.Owner, c.Repo = parseRepoRef(raw["github_sources"], "https://github.com")
|
||||
|
||||
// Gitea binary releases (self-hosted only — requires full URL or base_url).
|
||||
case raw["gitea_releases"] != "":
|
||||
c.Source = "gitea"
|
||||
c.BaseURL, c.Owner, c.Repo = parseRepoRef(raw["gitea_releases"], raw["base_url"])
|
||||
|
||||
// Gitea source tarballs (self-hosted only).
|
||||
case raw["gitea_sources"] != "":
|
||||
c.Source = "giteasource"
|
||||
c.BaseURL, c.Owner, c.Repo = parseRepoRef(raw["gitea_sources"], raw["base_url"])
|
||||
|
||||
// GitLab binary releases (defaults to gitlab.com).
|
||||
case raw["gitlab_releases"] != "":
|
||||
c.Source = "gitlab"
|
||||
c.BaseURL, c.Owner, c.Repo = parseRepoRef(raw["gitlab_releases"], "https://gitlab.com")
|
||||
|
||||
// GitLab source tarballs (defaults to gitlab.com).
|
||||
case raw["gitlab_sources"] != "":
|
||||
c.Source = "gitlabsource"
|
||||
c.BaseURL, c.Owner, c.Repo = parseRepoRef(raw["gitlab_sources"], "https://gitlab.com")
|
||||
|
||||
// Explicit source type (servicemandist, nodedist, zigdist, etc.).
|
||||
// Must come before git_url so that "source = X" + "git_url = ..."
|
||||
// uses X as the primary source, not gittag.
|
||||
case raw["source"] != "":
|
||||
c.Source = raw["source"]
|
||||
c.BaseURL = raw["url"]
|
||||
|
||||
// Git tag enumeration (only when no explicit source is set).
|
||||
case raw["git_url"] != "":
|
||||
c.Source = "gittag"
|
||||
c.BaseURL = raw["git_url"]
|
||||
|
||||
// HashiCorp.
|
||||
case raw["hashicorp_product"] != "":
|
||||
c.Source = "hashicorp"
|
||||
c.Repo = raw["hashicorp_product"]
|
||||
|
||||
default:
|
||||
}
|
||||
|
||||
// Explicit "source" overrides the inferred source when both are present.
|
||||
// This lets packages like ffmpeg use github_releases for fetching but
|
||||
// a custom classifier for classification.
|
||||
if raw["source"] != "" && c.Source != "" {
|
||||
c.Source = raw["source"]
|
||||
}
|
||||
|
||||
// git_url can appear alongside any source type (e.g. github_sources)
|
||||
// to provide a git clone fallback. When it's the only key, it's the
|
||||
// primary source (gittag).
|
||||
c.GitURL = raw["git_url"]
|
||||
|
||||
c.TagPrefix = raw["tag_prefix"]
|
||||
|
||||
if v := raw["version_prefixes"]; v != "" {
|
||||
c.VersionPrefixes = strings.Fields(v)
|
||||
} else if v := raw["version_prefix"]; v != "" {
|
||||
c.VersionPrefixes = strings.Fields(v)
|
||||
}
|
||||
|
||||
// Accept both "exclude" and "asset_exclude" (back-compat).
|
||||
if v := raw["exclude"]; v != "" {
|
||||
c.Exclude = strings.Fields(v)
|
||||
} else if v := raw["asset_exclude"]; v != "" {
|
||||
c.Exclude = strings.Fields(v)
|
||||
}
|
||||
|
||||
c.AssetFilter = raw["asset_filter"]
|
||||
c.OS = raw["os"]
|
||||
c.AliasOf = raw["alias_of"]
|
||||
|
||||
if v := raw["variants"]; v != "" {
|
||||
c.Variants = strings.Fields(v)
|
||||
}
|
||||
|
||||
// Collect unrecognized keys.
|
||||
known := map[string]bool{
|
||||
"source": true,
|
||||
"github_releases": true,
|
||||
"github_sources": true,
|
||||
"gitea_releases": true,
|
||||
"gitea_sources": true,
|
||||
"gitlab_releases": true,
|
||||
"gitlab_sources": true,
|
||||
"git_url": true,
|
||||
"hashicorp_product": true,
|
||||
"base_url": true,
|
||||
"url": true,
|
||||
"tag_prefix": true,
|
||||
"version_prefix": true,
|
||||
"version_prefixes": true,
|
||||
"exclude": true,
|
||||
"asset_exclude": true,
|
||||
"asset_filter": true,
|
||||
"os": true,
|
||||
"variants": true,
|
||||
"alias_of": true,
|
||||
}
|
||||
for k, v := range raw {
|
||||
if !known[k] {
|
||||
if c.Extra == nil {
|
||||
c.Extra = make(map[string]string)
|
||||
}
|
||||
c.Extra[k] = v
|
||||
}
|
||||
}
|
||||
|
||||
return c, nil
|
||||
}
|
||||
217
internal/installerconf/installerconf_test.go
Normal file
217
internal/installerconf/installerconf_test.go
Normal file
@@ -0,0 +1,217 @@
|
||||
package installerconf_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/installerconf"
|
||||
)
|
||||
|
||||
func TestGitHubReleases(t *testing.T) {
|
||||
c := confFromString(t, `
|
||||
github_releases = sharkdp/bat
|
||||
`)
|
||||
assertEqual(t, "Source", c.Source, "github")
|
||||
assertEqual(t, "Owner", c.Owner, "sharkdp")
|
||||
assertEqual(t, "Repo", c.Repo, "bat")
|
||||
assertEqual(t, "BaseURL", c.BaseURL, "https://github.com")
|
||||
assertEqual(t, "TagPrefix", c.TagPrefix, "")
|
||||
if len(c.VersionPrefixes) != 0 {
|
||||
t.Errorf("VersionPrefixes = %v, want empty", c.VersionPrefixes)
|
||||
}
|
||||
if len(c.Exclude) != 0 {
|
||||
t.Errorf("Exclude = %v, want empty", c.Exclude)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGitHubReleasesFullURL(t *testing.T) {
|
||||
c := confFromString(t, `
|
||||
github_releases = https://github.com/sharkdp/bat
|
||||
`)
|
||||
assertEqual(t, "Source", c.Source, "github")
|
||||
assertEqual(t, "BaseURL", c.BaseURL, "https://github.com")
|
||||
assertEqual(t, "Owner", c.Owner, "sharkdp")
|
||||
assertEqual(t, "Repo", c.Repo, "bat")
|
||||
}
|
||||
|
||||
func TestGitHubSources(t *testing.T) {
|
||||
c := confFromString(t, `
|
||||
github_sources = BeyondCodeBootcamp/aliasman
|
||||
git_url = https://github.com/BeyondCodeBootcamp/aliasman.git
|
||||
`)
|
||||
assertEqual(t, "Source", c.Source, "githubsource")
|
||||
assertEqual(t, "Owner", c.Owner, "BeyondCodeBootcamp")
|
||||
assertEqual(t, "Repo", c.Repo, "aliasman")
|
||||
assertEqual(t, "GitURL", c.GitURL, "https://github.com/BeyondCodeBootcamp/aliasman.git")
|
||||
}
|
||||
|
||||
func TestGitHubSourcesFullURL(t *testing.T) {
|
||||
c := confFromString(t, `
|
||||
github_sources = https://github.com/BeyondCodeBootcamp/aliasman
|
||||
git_url = https://github.com/BeyondCodeBootcamp/aliasman.git
|
||||
`)
|
||||
assertEqual(t, "Source", c.Source, "githubsource")
|
||||
assertEqual(t, "BaseURL", c.BaseURL, "https://github.com")
|
||||
assertEqual(t, "Owner", c.Owner, "BeyondCodeBootcamp")
|
||||
assertEqual(t, "Repo", c.Repo, "aliasman")
|
||||
}
|
||||
|
||||
func TestVersionPrefixes(t *testing.T) {
|
||||
c := confFromString(t, `
|
||||
github_releases = jqlang/jq
|
||||
version_prefixes = jq- cli-
|
||||
`)
|
||||
if len(c.VersionPrefixes) != 2 {
|
||||
t.Fatalf("VersionPrefixes has %d items, want 2: %v", len(c.VersionPrefixes), c.VersionPrefixes)
|
||||
}
|
||||
assertEqual(t, "VersionPrefixes[0]", c.VersionPrefixes[0], "jq-")
|
||||
assertEqual(t, "VersionPrefixes[1]", c.VersionPrefixes[1], "cli-")
|
||||
}
|
||||
|
||||
func TestExclude(t *testing.T) {
|
||||
c := confFromString(t, `
|
||||
github_releases = gohugoio/hugo
|
||||
exclude = _extended_ Linux-64bit
|
||||
`)
|
||||
if len(c.Exclude) != 2 {
|
||||
t.Fatalf("Exclude has %d items, want 2: %v", len(c.Exclude), c.Exclude)
|
||||
}
|
||||
assertEqual(t, "Exclude[0]", c.Exclude[0], "_extended_")
|
||||
assertEqual(t, "Exclude[1]", c.Exclude[1], "Linux-64bit")
|
||||
}
|
||||
|
||||
func TestMonorepoTagPrefix(t *testing.T) {
|
||||
c := confFromString(t, `
|
||||
github_releases = therootcompany/golib
|
||||
tag_prefix = tools/monorel/
|
||||
`)
|
||||
assertEqual(t, "TagPrefix", c.TagPrefix, "tools/monorel/")
|
||||
}
|
||||
|
||||
func TestNodeDist(t *testing.T) {
|
||||
c := confFromString(t, `
|
||||
source = nodedist
|
||||
url = https://nodejs.org/download/release
|
||||
`)
|
||||
assertEqual(t, "Source", c.Source, "nodedist")
|
||||
assertEqual(t, "BaseURL", c.BaseURL, "https://nodejs.org/download/release")
|
||||
}
|
||||
|
||||
func TestGiteaReleases(t *testing.T) {
|
||||
c := confFromString(t, `
|
||||
gitea_releases = https://git.rootprojects.org/root/pathman
|
||||
`)
|
||||
assertEqual(t, "Source", c.Source, "gitea")
|
||||
assertEqual(t, "BaseURL", c.BaseURL, "https://git.rootprojects.org")
|
||||
assertEqual(t, "Owner", c.Owner, "root")
|
||||
assertEqual(t, "Repo", c.Repo, "pathman")
|
||||
}
|
||||
|
||||
func TestGiteaReleasesWithBaseURL(t *testing.T) {
|
||||
c := confFromString(t, `
|
||||
gitea_releases = root/pathman
|
||||
base_url = https://git.rootprojects.org
|
||||
`)
|
||||
assertEqual(t, "Source", c.Source, "gitea")
|
||||
assertEqual(t, "BaseURL", c.BaseURL, "https://git.rootprojects.org")
|
||||
assertEqual(t, "Owner", c.Owner, "root")
|
||||
assertEqual(t, "Repo", c.Repo, "pathman")
|
||||
}
|
||||
|
||||
func TestGitLabReleases(t *testing.T) {
|
||||
c := confFromString(t, `
|
||||
gitlab_releases = owner/repo
|
||||
`)
|
||||
assertEqual(t, "Source", c.Source, "gitlab")
|
||||
assertEqual(t, "BaseURL", c.BaseURL, "https://gitlab.com")
|
||||
assertEqual(t, "Owner", c.Owner, "owner")
|
||||
assertEqual(t, "Repo", c.Repo, "repo")
|
||||
}
|
||||
|
||||
func TestGitLabReleasesFullURL(t *testing.T) {
|
||||
c := confFromString(t, `
|
||||
gitlab_releases = https://gitlab.example.com/myorg/myrepo
|
||||
`)
|
||||
assertEqual(t, "Source", c.Source, "gitlab")
|
||||
assertEqual(t, "BaseURL", c.BaseURL, "https://gitlab.example.com")
|
||||
assertEqual(t, "Owner", c.Owner, "myorg")
|
||||
assertEqual(t, "Repo", c.Repo, "myrepo")
|
||||
}
|
||||
|
||||
func TestBlanksAndComments(t *testing.T) {
|
||||
c := confFromString(t, `
|
||||
# Hugo config
|
||||
github_releases = foo/bar
|
||||
|
||||
# exclude line
|
||||
exclude = extended
|
||||
`)
|
||||
assertEqual(t, "Source", c.Source, "github")
|
||||
assertEqual(t, "Owner", c.Owner, "foo")
|
||||
assertEqual(t, "Repo", c.Repo, "bar")
|
||||
}
|
||||
|
||||
func TestExtraKeys(t *testing.T) {
|
||||
c := confFromString(t, `
|
||||
github_releases = foo/bar
|
||||
custom_thing = hello
|
||||
`)
|
||||
if c.Extra == nil || c.Extra["custom_thing"] != "hello" {
|
||||
t.Errorf("Extra[custom_thing] = %q, want hello", c.Extra["custom_thing"])
|
||||
}
|
||||
}
|
||||
|
||||
func TestAssetExcludeAlias(t *testing.T) {
|
||||
c := confFromString(t, `
|
||||
github_releases = gohugoio/hugo
|
||||
asset_exclude = extended
|
||||
`)
|
||||
if len(c.Exclude) != 1 {
|
||||
t.Fatalf("Exclude has %d items, want 1: %v", len(c.Exclude), c.Exclude)
|
||||
}
|
||||
assertEqual(t, "Exclude[0]", c.Exclude[0], "extended")
|
||||
}
|
||||
|
||||
func TestVariants(t *testing.T) {
|
||||
c := confFromString(t, `
|
||||
github_releases = jmorganca/ollama
|
||||
variants = rocm jetpack5 jetpack6
|
||||
`)
|
||||
if len(c.Variants) != 3 {
|
||||
t.Fatalf("Variants has %d items, want 3: %v", len(c.Variants), c.Variants)
|
||||
}
|
||||
assertEqual(t, "Variants[0]", c.Variants[0], "rocm")
|
||||
assertEqual(t, "Variants[1]", c.Variants[1], "jetpack5")
|
||||
assertEqual(t, "Variants[2]", c.Variants[2], "jetpack6")
|
||||
}
|
||||
|
||||
func TestEmptyExclude(t *testing.T) {
|
||||
c := confFromString(t, "github_releases = foo/bar\n")
|
||||
if c.Exclude != nil {
|
||||
t.Errorf("Exclude = %v, want nil", c.Exclude)
|
||||
}
|
||||
}
|
||||
|
||||
// helpers
|
||||
|
||||
func confFromString(t *testing.T, content string) *installerconf.Conf {
|
||||
t.Helper()
|
||||
dir := t.TempDir()
|
||||
path := filepath.Join(dir, "releases.conf")
|
||||
if err := os.WriteFile(path, []byte(content), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
c, err := installerconf.Read(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func assertEqual(t *testing.T, name, got, want string) {
|
||||
t.Helper()
|
||||
if got != want {
|
||||
t.Errorf("%s = %q, want %q", name, got, want)
|
||||
}
|
||||
}
|
||||
189
internal/lexver/lexver.go
Normal file
189
internal/lexver/lexver.go
Normal file
@@ -0,0 +1,189 @@
|
||||
// Package lexver makes version strings comparable and sortable.
|
||||
//
|
||||
// Not all version strings are semver. Webi handles 4-part versions
|
||||
// (chromedriver 121.0.6120.0), date-based versions (atomicparsley),
|
||||
// and pre-releases with extra dots (flutter 2.3.0-16.0.pre). Lexver
|
||||
// parses these into a struct with an arbitrary-depth numeric segment
|
||||
// list and provides a comparison function for use with [slices.SortFunc].
|
||||
//
|
||||
// Pre-releases sort before their corresponding stable release:
|
||||
//
|
||||
// 1.0.0-alpha1 < 1.0.0-beta1 < 1.0.0-rc1 < 1.0.0
|
||||
//
|
||||
// When release dates are known, they break ties between versions with
|
||||
// identical numeric segments.
|
||||
package lexver
|
||||
|
||||
import (
|
||||
"cmp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
)
|
||||
|
||||
// Version is a parsed version with comparable fields.
|
||||
type Version struct {
|
||||
// Nums holds the dotted numeric segments in order.
|
||||
// "1.20.3" → [1, 20, 3], "121.0.6120.0" → [121, 0, 6120, 0].
|
||||
Nums []int
|
||||
Channel string // "" for stable, or "alpha", "beta", "dev", "pre", "preview", "rc"
|
||||
ChannelNum int // e.g. 2 in "rc2"
|
||||
Date time.Time // release date/time, if known; breaks ties between same-numbered versions
|
||||
Original string // version string exactly as the releaser published it (e.g. "REL_17_0", "r21")
|
||||
Raw string // version string after Webi's normalization (e.g. "17.0", "0.21.0")
|
||||
|
||||
// ExtraSort is an optional opaque string for package-specific ordering.
|
||||
// Set by release-fetcher code for packages where Nums alone can't capture
|
||||
// the sort order (e.g. flutter's "2.3.0-16.0.pre"). Compared as a plain
|
||||
// string, only consulted when Nums and Channel are equal.
|
||||
ExtraSort string
|
||||
}
|
||||
|
||||
// Parse breaks a version string into its components.
|
||||
// Both Original and Raw are set to s; callers that normalize versions
|
||||
// (e.g. "REL_17_0" → "17.0") should set Original to the upstream tag
|
||||
// and pass the normalized string to Parse.
|
||||
func Parse(s string) Version {
|
||||
v := Version{Original: s, Raw: s}
|
||||
|
||||
s = strings.TrimLeft(s, "vV")
|
||||
|
||||
numStr, prerelease := splitAtPrerelease(s)
|
||||
v.Nums = splitNums(numStr)
|
||||
|
||||
if prerelease != "" {
|
||||
v.Channel, v.ChannelNum = splitChannel(prerelease)
|
||||
}
|
||||
|
||||
return v
|
||||
}
|
||||
|
||||
// Major returns the first numeric segment, or 0 if none.
|
||||
func (v Version) Major() int { return v.num(0) }
|
||||
|
||||
// Minor returns the second numeric segment, or 0 if none.
|
||||
func (v Version) Minor() int { return v.num(1) }
|
||||
|
||||
// Patch returns the third numeric segment, or 0 if none.
|
||||
func (v Version) Patch() int { return v.num(2) }
|
||||
|
||||
func (v Version) num(i int) int {
|
||||
if i < len(v.Nums) {
|
||||
return v.Nums[i]
|
||||
}
|
||||
return 0
|
||||
}
|
||||
|
||||
// IsStable reports whether this is a stable (non-pre-release) version.
|
||||
func (v Version) IsStable() bool {
|
||||
return v.Channel == ""
|
||||
}
|
||||
|
||||
// Compare returns -1, 0, or 1 for ordering two versions.
|
||||
// Stable releases sort after pre-releases of the same numeric version.
|
||||
func Compare(a, b Version) int {
|
||||
// Compare numeric segments pairwise, treating missing segments as 0.
|
||||
n := max(len(a.Nums), len(b.Nums))
|
||||
for i := range n {
|
||||
an, bn := a.num(i), b.num(i)
|
||||
if c := cmp.Compare(an, bn); c != 0 {
|
||||
return c
|
||||
}
|
||||
}
|
||||
|
||||
// Break ties with release date when both are known.
|
||||
if !a.Date.IsZero() && !b.Date.IsZero() {
|
||||
if c := a.Date.Compare(b.Date); c != 0 {
|
||||
return c
|
||||
}
|
||||
}
|
||||
|
||||
// ExtraSort: package-specific tiebreaker set by release-fetcher code.
|
||||
if a.ExtraSort != "" && b.ExtraSort != "" {
|
||||
if c := cmp.Compare(a.ExtraSort, b.ExtraSort); c != 0 {
|
||||
return c
|
||||
}
|
||||
}
|
||||
|
||||
// Both stable → equal.
|
||||
if a.Channel == "" && b.Channel == "" {
|
||||
return 0
|
||||
}
|
||||
// Stable beats any pre-release.
|
||||
if a.Channel == "" {
|
||||
return 1
|
||||
}
|
||||
if b.Channel == "" {
|
||||
return -1
|
||||
}
|
||||
// Both pre-release: alphabetical channel, then number.
|
||||
if c := cmp.Compare(a.Channel, b.Channel); c != 0 {
|
||||
return c
|
||||
}
|
||||
return cmp.Compare(a.ChannelNum, b.ChannelNum)
|
||||
}
|
||||
|
||||
// HasPrefix reports whether v matches a partial version prefix.
|
||||
// A prefix with Nums [1, 20] matches any version starting with 1.20
|
||||
// (e.g. 1.20.0, 1.20.3, 1.20.3.1).
|
||||
func (v Version) HasPrefix(prefix Version) bool {
|
||||
for i, pn := range prefix.Nums {
|
||||
if i >= len(v.Nums) || v.Nums[i] != pn {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// splitAtPrerelease splits "1.20.3-beta1" into ("1.20.3", "beta1").
|
||||
// Also handles "1.2beta3" (no separator).
|
||||
func splitAtPrerelease(s string) (string, string) {
|
||||
for _, sep := range []byte{'-', '+'} {
|
||||
if idx := strings.IndexByte(s, sep); idx >= 0 {
|
||||
return s[:idx], s[idx+1:]
|
||||
}
|
||||
}
|
||||
|
||||
// "1.2beta3": letter following a digit
|
||||
for i := 1; i < len(s); i++ {
|
||||
if unicode.IsLetter(rune(s[i])) && unicode.IsDigit(rune(s[i-1])) {
|
||||
return s[:i], s[i:]
|
||||
}
|
||||
}
|
||||
|
||||
return s, ""
|
||||
}
|
||||
|
||||
// splitNums parses "1.20.3" into [1, 20, 3].
|
||||
// Handles any number of dot-separated segments.
|
||||
func splitNums(s string) []int {
|
||||
var nums []int
|
||||
for _, seg := range strings.Split(s, ".") {
|
||||
n, err := strconv.Atoi(seg)
|
||||
if err != nil {
|
||||
break
|
||||
}
|
||||
nums = append(nums, n)
|
||||
}
|
||||
return nums
|
||||
}
|
||||
|
||||
// splitChannel separates "beta1" into ("beta", 1) or "rc" into ("rc", 0).
|
||||
func splitChannel(s string) (string, int) {
|
||||
s = strings.ToLower(s)
|
||||
s = strings.NewReplacer("-", "", ".", "", "_", "").Replace(s)
|
||||
|
||||
i := len(s)
|
||||
for i > 0 && unicode.IsDigit(rune(s[i-1])) {
|
||||
i--
|
||||
}
|
||||
|
||||
name := s[:i]
|
||||
num := 0
|
||||
if i < len(s) {
|
||||
num, _ = strconv.Atoi(s[i:])
|
||||
}
|
||||
|
||||
return name, num
|
||||
}
|
||||
270
internal/lexver/lexver_test.go
Normal file
270
internal/lexver/lexver_test.go
Normal file
@@ -0,0 +1,270 @@
|
||||
package lexver_test
|
||||
|
||||
import (
|
||||
"slices"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/lexver"
|
||||
)
|
||||
|
||||
func TestParse(t *testing.T) {
|
||||
tests := []struct {
|
||||
input string
|
||||
nums []int
|
||||
channel string
|
||||
chanNum int
|
||||
}{
|
||||
// Standard semver
|
||||
{"1.0.0", []int{1, 0, 0}, "", 0},
|
||||
{"v1.2.3", []int{1, 2, 3}, "", 0},
|
||||
{"1.20.156", []int{1, 20, 156}, "", 0},
|
||||
|
||||
// Partial
|
||||
{"1.20", []int{1, 20}, "", 0},
|
||||
{"1", []int{1}, "", 0},
|
||||
|
||||
// 4-part (chromedriver, gpg)
|
||||
{"121.0.6120.0", []int{121, 0, 6120, 0}, "", 0},
|
||||
{"2.2.19.0", []int{2, 2, 19, 0}, "", 0},
|
||||
|
||||
// Pre-release
|
||||
{"1.0.0-beta1", []int{1, 0, 0}, "beta", 1},
|
||||
{"1.0.0-rc2", []int{1, 0, 0}, "rc", 2},
|
||||
{"2.0.0-alpha3", []int{2, 0, 0}, "alpha", 3},
|
||||
{"1.0.0-dev", []int{1, 0, 0}, "dev", 0},
|
||||
|
||||
// No separator before channel
|
||||
{"1.2beta3", []int{1, 2}, "beta", 3},
|
||||
{"1.0rc1", []int{1, 0}, "rc", 1},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.input, func(t *testing.T) {
|
||||
v := lexver.Parse(tt.input)
|
||||
if !slices.Equal(v.Nums, tt.nums) {
|
||||
t.Errorf("Parse(%q).Nums = %v, want %v", tt.input, v.Nums, tt.nums)
|
||||
}
|
||||
if v.Channel != tt.channel || v.ChannelNum != tt.chanNum {
|
||||
t.Errorf("Parse(%q) channel = %q/%d, want %q/%d",
|
||||
tt.input, v.Channel, v.ChannelNum, tt.channel, tt.chanNum)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestAccessors(t *testing.T) {
|
||||
v := lexver.Parse("121.0.6120.0")
|
||||
if v.Major() != 121 || v.Minor() != 0 || v.Patch() != 6120 {
|
||||
t.Errorf("got %d.%d.%d, want 121.0.6120", v.Major(), v.Minor(), v.Patch())
|
||||
}
|
||||
|
||||
short := lexver.Parse("1")
|
||||
if short.Minor() != 0 || short.Patch() != 0 {
|
||||
t.Error("missing segments should return 0")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSortOrder(t *testing.T) {
|
||||
// Must be in ascending order.
|
||||
ordered := []string{
|
||||
"0.1.0",
|
||||
"1.0.0-alpha1",
|
||||
"1.0.0-alpha2",
|
||||
"1.0.0-beta1",
|
||||
"1.0.0-rc1",
|
||||
"1.0.0-rc2",
|
||||
"1.0.0",
|
||||
"1.0.1",
|
||||
"1.1.0",
|
||||
"1.2.0",
|
||||
"1.20.0",
|
||||
"2.0.0-beta1",
|
||||
"2.0.0",
|
||||
}
|
||||
|
||||
for i := 1; i < len(ordered); i++ {
|
||||
a := lexver.Parse(ordered[i-1])
|
||||
b := lexver.Parse(ordered[i])
|
||||
if lexver.Compare(a, b) >= 0 {
|
||||
t.Errorf("expected %q < %q", ordered[i-1], ordered[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSortOrder4Part(t *testing.T) {
|
||||
ordered := []string{
|
||||
"121.0.6120.0",
|
||||
"121.0.6120.1",
|
||||
"121.0.6121.0",
|
||||
"122.0.6100.0",
|
||||
}
|
||||
|
||||
for i := 1; i < len(ordered); i++ {
|
||||
a := lexver.Parse(ordered[i-1])
|
||||
b := lexver.Parse(ordered[i])
|
||||
if lexver.Compare(a, b) >= 0 {
|
||||
t.Errorf("expected %q < %q", ordered[i-1], ordered[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestMismatchedDepth(t *testing.T) {
|
||||
// "1.0" and "1.0.0" should be equal (trailing zeros).
|
||||
a := lexver.Parse("1.0")
|
||||
b := lexver.Parse("1.0.0")
|
||||
if lexver.Compare(a, b) != 0 {
|
||||
t.Error("1.0 and 1.0.0 should be equal")
|
||||
}
|
||||
|
||||
// "1.0.0.1" should be greater than "1.0.0".
|
||||
c := lexver.Parse("1.0.0.1")
|
||||
d := lexver.Parse("1.0.0")
|
||||
if lexver.Compare(c, d) <= 0 {
|
||||
t.Error("1.0.0.1 should be greater than 1.0.0")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSortFunc(t *testing.T) {
|
||||
versions := []string{"1.0.0", "2.0.0-rc1", "1.20.3", "1.20.2", "1.19.5", "2.0.0"}
|
||||
parsed := make([]lexver.Version, len(versions))
|
||||
for i, s := range versions {
|
||||
parsed[i] = lexver.Parse(s)
|
||||
}
|
||||
|
||||
// Sort descending (newest first).
|
||||
slices.SortFunc(parsed, func(a, b lexver.Version) int {
|
||||
return lexver.Compare(b, a)
|
||||
})
|
||||
|
||||
want := []string{"2.0.0", "2.0.0-rc1", "1.20.3", "1.20.2", "1.19.5", "1.0.0"}
|
||||
for i, v := range parsed {
|
||||
if v.Raw != want[i] {
|
||||
t.Errorf("index %d: got %q, want %q", i, v.Raw, want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestIsStable(t *testing.T) {
|
||||
tests := []struct {
|
||||
input string
|
||||
want bool
|
||||
}{
|
||||
{"1.0.0", true},
|
||||
{"121.0.6120.0", true},
|
||||
{"1.0.0-beta1", false},
|
||||
{"v2.0.0-dev", false},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.input, func(t *testing.T) {
|
||||
v := lexver.Parse(tt.input)
|
||||
if v.IsStable() != tt.want {
|
||||
t.Errorf("Parse(%q).IsStable() = %v, want %v", tt.input, v.IsStable(), tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestDateTiebreaker(t *testing.T) {
|
||||
a := lexver.Parse("1.0.0")
|
||||
a.Date = time.Date(2024, 1, 15, 0, 0, 0, 0, time.UTC)
|
||||
|
||||
b := lexver.Parse("1.0.0")
|
||||
b.Date = time.Date(2024, 6, 1, 14, 30, 0, 0, time.UTC)
|
||||
|
||||
if lexver.Compare(a, b) >= 0 {
|
||||
t.Error("earlier date should sort before later date at same version")
|
||||
}
|
||||
|
||||
// Without dates, same version is equal.
|
||||
c := lexver.Parse("1.0.0")
|
||||
d := lexver.Parse("1.0.0")
|
||||
if lexver.Compare(c, d) != 0 {
|
||||
t.Error("same version without dates should be equal")
|
||||
}
|
||||
|
||||
// Date only matters when both have it.
|
||||
e := lexver.Parse("1.0.0")
|
||||
e.Date = time.Date(2024, 1, 15, 0, 0, 0, 0, time.UTC)
|
||||
f := lexver.Parse("1.0.0")
|
||||
if lexver.Compare(e, f) != 0 {
|
||||
t.Error("date should be ignored when only one side has it")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDateMinutePrecision(t *testing.T) {
|
||||
a := lexver.Parse("1.0.0")
|
||||
a.Date = time.Date(2024, 1, 15, 10, 0, 0, 0, time.UTC)
|
||||
|
||||
b := lexver.Parse("1.0.0")
|
||||
b.Date = time.Date(2024, 1, 15, 10, 30, 0, 0, time.UTC)
|
||||
|
||||
if lexver.Compare(a, b) >= 0 {
|
||||
t.Error("same date, later time should sort after")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOriginal(t *testing.T) {
|
||||
// Parse sets both Original and Raw to the input.
|
||||
v := lexver.Parse("17.0")
|
||||
if v.Original != "17.0" {
|
||||
t.Errorf("Original = %q, want %q", v.Original, "17.0")
|
||||
}
|
||||
|
||||
// Release fetcher would do:
|
||||
// v := lexver.Parse("17.0")
|
||||
// v.Original = "REL_17_0"
|
||||
v.Original = "REL_17_0"
|
||||
if v.Raw != "17.0" {
|
||||
t.Errorf("Raw should remain %q after setting Original, got %q", "17.0", v.Raw)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExtraSort(t *testing.T) {
|
||||
// Flutter example: 2.3.0-16.0.pre and 2.3.0-16.1.pre
|
||||
// Nums and Channel are the same; ExtraSort distinguishes them.
|
||||
a := lexver.Parse("2.3.0-pre")
|
||||
a.ExtraSort = "0016.0000"
|
||||
|
||||
b := lexver.Parse("2.3.0-pre")
|
||||
b.ExtraSort = "0016.0001"
|
||||
|
||||
if lexver.Compare(a, b) >= 0 {
|
||||
t.Error("ExtraSort 0016.0000 should sort before 0016.0001")
|
||||
}
|
||||
|
||||
// ExtraSort ignored when only one side has it.
|
||||
c := lexver.Parse("2.3.0-pre")
|
||||
c.ExtraSort = "0016.0000"
|
||||
d := lexver.Parse("2.3.0-pre")
|
||||
if lexver.Compare(c, d) != 0 {
|
||||
t.Error("ExtraSort should be ignored when only one side has it")
|
||||
}
|
||||
}
|
||||
|
||||
func TestHasPrefix(t *testing.T) {
|
||||
v := lexver.Parse("1.20.3")
|
||||
|
||||
if !v.HasPrefix(lexver.Parse("1.20")) {
|
||||
t.Error("1.20.3 should match prefix 1.20")
|
||||
}
|
||||
if !v.HasPrefix(lexver.Parse("1")) {
|
||||
t.Error("1.20.3 should match prefix 1")
|
||||
}
|
||||
if v.HasPrefix(lexver.Parse("1.19")) {
|
||||
t.Error("1.20.3 should not match prefix 1.19")
|
||||
}
|
||||
if v.HasPrefix(lexver.Parse("2")) {
|
||||
t.Error("1.20.3 should not match prefix 2")
|
||||
}
|
||||
|
||||
// 4-part prefix matching
|
||||
v4 := lexver.Parse("121.0.6120.0")
|
||||
if !v4.HasPrefix(lexver.Parse("121.0.6120")) {
|
||||
t.Error("121.0.6120.0 should match prefix 121.0.6120")
|
||||
}
|
||||
if !v4.HasPrefix(lexver.Parse("121.0")) {
|
||||
t.Error("121.0.6120.0 should match prefix 121.0")
|
||||
}
|
||||
}
|
||||
104
internal/platlatest/platlatest.go
Normal file
104
internal/platlatest/platlatest.go
Normal file
@@ -0,0 +1,104 @@
|
||||
// Package platlatest tracks the newest release version per build target.
|
||||
//
|
||||
// After classification determines which OS/arch/libc targets a release
|
||||
// covers, this package records the latest version for each target. This
|
||||
// handles the common case where Windows or macOS releases lag behind
|
||||
// Linux by several versions.
|
||||
//
|
||||
// Storage is a single JSON file per package:
|
||||
//
|
||||
// {
|
||||
// "linux-x86_64-gnu": "v0.145.0",
|
||||
// "darwin-aarch64-none": "v0.144.1",
|
||||
// "windows-x86_64-msvc": "v0.143.0"
|
||||
// }
|
||||
package platlatest
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sync"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/buildmeta"
|
||||
)
|
||||
|
||||
// Index tracks the latest version for each build target of a package.
|
||||
type Index struct {
|
||||
mu sync.RWMutex
|
||||
path string
|
||||
m map[string]string // triplet → version
|
||||
}
|
||||
|
||||
// Open loads or creates a per-platform latest index at the given path.
|
||||
func Open(path string) (*Index, error) {
|
||||
idx := &Index{
|
||||
path: path,
|
||||
m: make(map[string]string),
|
||||
}
|
||||
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
if os.IsNotExist(err) {
|
||||
return idx, nil
|
||||
}
|
||||
return nil, fmt.Errorf("platlatest: read %s: %w", path, err)
|
||||
}
|
||||
|
||||
if err := json.Unmarshal(data, &idx.m); err != nil {
|
||||
return nil, fmt.Errorf("platlatest: parse %s: %w", path, err)
|
||||
}
|
||||
return idx, nil
|
||||
}
|
||||
|
||||
// Get returns the latest version for a target, or "" if unknown.
|
||||
func (idx *Index) Get(t buildmeta.Target) string {
|
||||
idx.mu.RLock()
|
||||
defer idx.mu.RUnlock()
|
||||
return idx.m[t.Triplet()]
|
||||
}
|
||||
|
||||
// Set records a version as the latest for a target. Does not persist
|
||||
// to disk — call Save after all updates.
|
||||
func (idx *Index) Set(t buildmeta.Target, version string) {
|
||||
idx.mu.Lock()
|
||||
defer idx.mu.Unlock()
|
||||
idx.m[t.Triplet()] = version
|
||||
}
|
||||
|
||||
// All returns a copy of the full triplet→version map.
|
||||
func (idx *Index) All() map[string]string {
|
||||
idx.mu.RLock()
|
||||
defer idx.mu.RUnlock()
|
||||
out := make(map[string]string, len(idx.m))
|
||||
for k, v := range idx.m {
|
||||
out[k] = v
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
// Save persists the index to disk (atomic write).
|
||||
func (idx *Index) Save() error {
|
||||
idx.mu.RLock()
|
||||
data, err := json.MarshalIndent(idx.m, "", " ")
|
||||
idx.mu.RUnlock()
|
||||
if err != nil {
|
||||
return fmt.Errorf("platlatest: marshal: %w", err)
|
||||
}
|
||||
|
||||
dir := filepath.Dir(idx.path)
|
||||
if err := os.MkdirAll(dir, 0o755); err != nil {
|
||||
return fmt.Errorf("platlatest: mkdir: %w", err)
|
||||
}
|
||||
|
||||
tmp := idx.path + ".tmp"
|
||||
if err := os.WriteFile(tmp, data, 0o644); err != nil {
|
||||
return fmt.Errorf("platlatest: write %s: %w", tmp, err)
|
||||
}
|
||||
if err := os.Rename(tmp, idx.path); err != nil {
|
||||
os.Remove(tmp)
|
||||
return fmt.Errorf("platlatest: rename %s: %w", idx.path, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
104
internal/platlatest/platlatest_test.go
Normal file
104
internal/platlatest/platlatest_test.go
Normal file
@@ -0,0 +1,104 @@
|
||||
package platlatest_test
|
||||
|
||||
import (
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/buildmeta"
|
||||
"github.com/webinstall/webi-installers/internal/platlatest"
|
||||
)
|
||||
|
||||
var (
|
||||
linuxAMD64 = buildmeta.Target{
|
||||
OS: buildmeta.OSLinux, Arch: buildmeta.ArchAMD64, Libc: buildmeta.LibcGNU,
|
||||
}
|
||||
darwinARM64 = buildmeta.Target{
|
||||
OS: buildmeta.OSDarwin, Arch: buildmeta.ArchARM64, Libc: buildmeta.LibcNone,
|
||||
}
|
||||
windowsAMD64 = buildmeta.Target{
|
||||
OS: buildmeta.OSWindows, Arch: buildmeta.ArchAMD64, Libc: buildmeta.LibcMSVC,
|
||||
}
|
||||
)
|
||||
|
||||
func TestSetAndGet(t *testing.T) {
|
||||
p := filepath.Join(t.TempDir(), "latest.json")
|
||||
idx, err := platlatest.Open(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if got := idx.Get(linuxAMD64); got != "" {
|
||||
t.Errorf("Get before Set = %q, want empty", got)
|
||||
}
|
||||
|
||||
idx.Set(linuxAMD64, "v0.145.0")
|
||||
idx.Set(darwinARM64, "v0.144.1")
|
||||
idx.Set(windowsAMD64, "v0.143.0")
|
||||
|
||||
if got := idx.Get(linuxAMD64); got != "v0.145.0" {
|
||||
t.Errorf("linux = %q, want v0.145.0", got)
|
||||
}
|
||||
if got := idx.Get(darwinARM64); got != "v0.144.1" {
|
||||
t.Errorf("darwin = %q, want v0.144.1", got)
|
||||
}
|
||||
if got := idx.Get(windowsAMD64); got != "v0.143.0" {
|
||||
t.Errorf("windows = %q, want v0.143.0", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSaveAndReload(t *testing.T) {
|
||||
p := filepath.Join(t.TempDir(), "latest.json")
|
||||
|
||||
idx1, err := platlatest.Open(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
idx1.Set(linuxAMD64, "v0.145.0")
|
||||
idx1.Set(darwinARM64, "v0.144.1")
|
||||
if err := idx1.Save(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Reload from disk.
|
||||
idx2, err := platlatest.Open(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got := idx2.Get(linuxAMD64); got != "v0.145.0" {
|
||||
t.Errorf("after reload: linux = %q, want v0.145.0", got)
|
||||
}
|
||||
if got := idx2.Get(darwinARM64); got != "v0.144.1" {
|
||||
t.Errorf("after reload: darwin = %q, want v0.144.1", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestAll(t *testing.T) {
|
||||
p := filepath.Join(t.TempDir(), "latest.json")
|
||||
idx, err := platlatest.Open(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
idx.Set(linuxAMD64, "v1.0.0")
|
||||
idx.Set(darwinARM64, "v0.9.0")
|
||||
|
||||
all := idx.All()
|
||||
if len(all) != 2 {
|
||||
t.Fatalf("All() returned %d entries, want 2", len(all))
|
||||
}
|
||||
if all[linuxAMD64.Triplet()] != "v1.0.0" {
|
||||
t.Error("missing linux entry")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenNonexistent(t *testing.T) {
|
||||
p := filepath.Join(t.TempDir(), "does-not-exist.json")
|
||||
idx, err := platlatest.Open(p)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// Should be empty, not nil.
|
||||
if all := idx.All(); len(all) != 0 {
|
||||
t.Errorf("new index should be empty, got %v", all)
|
||||
}
|
||||
}
|
||||
63
internal/rawcache/auditlog.go
Normal file
63
internal/rawcache/auditlog.go
Normal file
@@ -0,0 +1,63 @@
|
||||
package rawcache
|
||||
|
||||
import (
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
)
|
||||
|
||||
// LogEntry records one event in the append-only audit log.
|
||||
type LogEntry struct {
|
||||
Time time.Time `json:"time"`
|
||||
Tag string `json:"tag"`
|
||||
Action string `json:"action"` // "added", "changed", "removed"
|
||||
SHA256 string `json:"sha256,omitempty"`
|
||||
}
|
||||
|
||||
// AuditLog is an append-only JSONL file that tracks when releases appear,
|
||||
// change, or disappear from upstream. One file per package, lives alongside
|
||||
// the double-buffer slots.
|
||||
type AuditLog struct {
|
||||
path string
|
||||
}
|
||||
|
||||
// openLog returns the audit log for a Dir.
|
||||
func (d *Dir) openLog() *AuditLog {
|
||||
return &AuditLog{path: filepath.Join(d.root, "audit.jsonl")}
|
||||
}
|
||||
|
||||
// Append writes one log entry.
|
||||
func (l *AuditLog) Append(entry LogEntry) error {
|
||||
if entry.Time.IsZero() {
|
||||
entry.Time = time.Now().UTC()
|
||||
}
|
||||
data, err := json.Marshal(entry)
|
||||
if err != nil {
|
||||
return fmt.Errorf("rawcache: marshal log entry: %w", err)
|
||||
}
|
||||
data = append(data, '\n')
|
||||
|
||||
f, err := os.OpenFile(l.path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644)
|
||||
if err != nil {
|
||||
return fmt.Errorf("rawcache: open audit log: %w", err)
|
||||
}
|
||||
_, writeErr := f.Write(data)
|
||||
closeErr := f.Close()
|
||||
if writeErr != nil {
|
||||
return fmt.Errorf("rawcache: write audit log: %w", writeErr)
|
||||
}
|
||||
if closeErr != nil {
|
||||
return fmt.Errorf("rawcache: close audit log: %w", closeErr)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// ContentHash returns the SHA-256 hex digest of data.
|
||||
func ContentHash(data []byte) string {
|
||||
h := sha256.Sum256(data)
|
||||
return hex.EncodeToString(h[:])
|
||||
}
|
||||
265
internal/rawcache/rawcache.go
Normal file
265
internal/rawcache/rawcache.go
Normal file
@@ -0,0 +1,265 @@
|
||||
// Package rawcache stores raw upstream API responses on disk, one file per
|
||||
// release, with double-buffered full refreshes.
|
||||
//
|
||||
// Directory layout:
|
||||
//
|
||||
// {root}/
|
||||
// active → a symlink to the current slot
|
||||
// a/ slot A
|
||||
// _latest one-line file: newest tag
|
||||
// v0.145.0.json
|
||||
// v0.144.1.json
|
||||
// ...
|
||||
// b/ slot B (standby)
|
||||
//
|
||||
// Incremental updates write directly to the active slot. Each file write
|
||||
// is atomic (temp file + rename). Full refreshes write to the standby slot,
|
||||
// then atomically swap the symlink.
|
||||
package rawcache
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Dir manages a raw release cache for one package.
|
||||
type Dir struct {
|
||||
root string // e.g. "_cache/raw/github/gohugoio/hugo"
|
||||
}
|
||||
|
||||
// Open returns a Dir for the given root path. Creates the directory
|
||||
// structure (slots + symlink) if it doesn't exist.
|
||||
func Open(root string) (*Dir, error) {
|
||||
d := &Dir{root: root}
|
||||
|
||||
slotA := filepath.Join(root, "a")
|
||||
slotB := filepath.Join(root, "b")
|
||||
active := filepath.Join(root, "active")
|
||||
|
||||
// Create both slots.
|
||||
for _, slot := range []string{slotA, slotB} {
|
||||
if err := os.MkdirAll(slot, 0o755); err != nil {
|
||||
return nil, fmt.Errorf("rawcache: create slot: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
// Create the active symlink if it doesn't exist.
|
||||
if _, err := os.Lstat(active); errors.Is(err, os.ErrNotExist) {
|
||||
if err := os.Symlink("a", active); err != nil {
|
||||
return nil, fmt.Errorf("rawcache: create active symlink: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
return d, nil
|
||||
}
|
||||
|
||||
// ActivePath returns the absolute path of the currently active slot.
|
||||
func (d *Dir) ActivePath() (string, error) {
|
||||
target, err := os.Readlink(filepath.Join(d.root, "active"))
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("rawcache: read active symlink: %w", err)
|
||||
}
|
||||
return filepath.Join(d.root, target), nil
|
||||
}
|
||||
|
||||
// standbySlot returns the name of the inactive slot ("a" or "b").
|
||||
func (d *Dir) standbySlot() (string, error) {
|
||||
target, err := os.Readlink(filepath.Join(d.root, "active"))
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("rawcache: read active symlink: %w", err)
|
||||
}
|
||||
if target == "a" {
|
||||
return "b", nil
|
||||
}
|
||||
return "a", nil
|
||||
}
|
||||
|
||||
// Populated returns true if the active slot contains at least one release file.
|
||||
func (d *Dir) Populated() bool {
|
||||
active, err := d.ActivePath()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
entries, err := os.ReadDir(active)
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
for _, e := range entries {
|
||||
if !e.IsDir() && !strings.HasPrefix(e.Name(), "_") {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Has reports whether a release file exists in the active slot.
|
||||
func (d *Dir) Has(tag string) bool {
|
||||
active, err := d.ActivePath()
|
||||
if err != nil {
|
||||
return false
|
||||
}
|
||||
_, err = os.Stat(filepath.Join(active, tagToFilename(tag)))
|
||||
return err == nil
|
||||
}
|
||||
|
||||
// Latest returns the newest tag from the active slot.
|
||||
// Returns "" if no latest marker exists.
|
||||
func (d *Dir) Latest() string {
|
||||
active, err := d.ActivePath()
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
data, err := os.ReadFile(filepath.Join(active, "_latest"))
|
||||
if err != nil {
|
||||
return ""
|
||||
}
|
||||
return strings.TrimSpace(string(data))
|
||||
}
|
||||
|
||||
// Read returns the raw cached data for a tag from the active slot.
|
||||
func (d *Dir) Read(tag string) ([]byte, error) {
|
||||
active, err := d.ActivePath()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return os.ReadFile(filepath.Join(active, tagToFilename(tag)))
|
||||
}
|
||||
|
||||
// Put writes a release file to the active slot. The write is atomic
|
||||
// (temp file + rename).
|
||||
func (d *Dir) Put(tag string, data []byte) error {
|
||||
active, err := d.ActivePath()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return atomicWrite(filepath.Join(active, tagToFilename(tag)), data)
|
||||
}
|
||||
|
||||
// Merge writes a release to the active slot if it's new or changed.
|
||||
// Returns the action taken: "added", "changed", or "" (unchanged).
|
||||
// Logs the event to the audit log when something happens.
|
||||
func (d *Dir) Merge(tag string, data []byte) (string, error) {
|
||||
log := d.openLog()
|
||||
hash := ContentHash(data)
|
||||
|
||||
if d.Has(tag) {
|
||||
existing, err := d.Read(tag)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if ContentHash(existing) == hash {
|
||||
return "", nil // unchanged
|
||||
}
|
||||
if err := d.Put(tag, data); err != nil {
|
||||
return "", err
|
||||
}
|
||||
log.Append(LogEntry{Tag: tag, Action: "changed", SHA256: hash})
|
||||
return "changed", nil
|
||||
}
|
||||
|
||||
if err := d.Put(tag, data); err != nil {
|
||||
return "", err
|
||||
}
|
||||
log.Append(LogEntry{Tag: tag, Action: "added", SHA256: hash})
|
||||
return "added", nil
|
||||
}
|
||||
|
||||
// SetLatest updates the _latest marker in the active slot.
|
||||
func (d *Dir) SetLatest(tag string) error {
|
||||
active, err := d.ActivePath()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return atomicWrite(filepath.Join(active, "_latest"), []byte(tag+"\n"))
|
||||
}
|
||||
|
||||
// BeginRefresh starts a full refresh. Clears the standby slot and returns
|
||||
// a Refresh handle for writing to it. Call Commit to atomically swap, or
|
||||
// Abort to discard.
|
||||
func (d *Dir) BeginRefresh() (*Refresh, error) {
|
||||
standby, err := d.standbySlot()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
standbyPath := filepath.Join(d.root, standby)
|
||||
|
||||
// Clear the standby slot.
|
||||
entries, _ := os.ReadDir(standbyPath)
|
||||
for _, e := range entries {
|
||||
os.Remove(filepath.Join(standbyPath, e.Name()))
|
||||
}
|
||||
|
||||
return &Refresh{
|
||||
dir: d,
|
||||
slot: standby,
|
||||
slotDir: standbyPath,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Refresh writes releases to the standby slot during a full refresh.
|
||||
type Refresh struct {
|
||||
dir *Dir
|
||||
slot string // "a" or "b"
|
||||
slotDir string
|
||||
}
|
||||
|
||||
// Put writes a release file to the standby slot.
|
||||
func (r *Refresh) Put(tag string, data []byte) error {
|
||||
return atomicWrite(filepath.Join(r.slotDir, tagToFilename(tag)), data)
|
||||
}
|
||||
|
||||
// SetLatest updates the _latest marker in the standby slot.
|
||||
func (r *Refresh) SetLatest(tag string) error {
|
||||
return atomicWrite(filepath.Join(r.slotDir, "_latest"), []byte(tag+"\n"))
|
||||
}
|
||||
|
||||
// Commit atomically swaps the active symlink to point to the standby slot.
|
||||
func (r *Refresh) Commit() error {
|
||||
active := filepath.Join(r.dir.root, "active")
|
||||
tmp := active + ".tmp"
|
||||
|
||||
// Remove stale temp symlink if it exists.
|
||||
os.Remove(tmp)
|
||||
|
||||
if err := os.Symlink(r.slot, tmp); err != nil {
|
||||
return fmt.Errorf("rawcache: create temp symlink: %w", err)
|
||||
}
|
||||
if err := os.Rename(tmp, active); err != nil {
|
||||
os.Remove(tmp)
|
||||
return fmt.Errorf("rawcache: swap active symlink: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Abort discards the standby slot contents.
|
||||
func (r *Refresh) Abort() {
|
||||
entries, _ := os.ReadDir(r.slotDir)
|
||||
for _, e := range entries {
|
||||
os.Remove(filepath.Join(r.slotDir, e.Name()))
|
||||
}
|
||||
}
|
||||
|
||||
// tagToFilename converts a tag to a safe filename.
|
||||
// Tags like "v0.145.0" become "v0.145.0". The raw cache stores opaque
|
||||
// bytes — no extension is assumed because upstream responses may be
|
||||
// JSON, CSV, XML, or bespoke formats.
|
||||
func tagToFilename(tag string) string {
|
||||
// Replace path separators in case a tag contains slashes.
|
||||
return strings.ReplaceAll(tag, "/", "_")
|
||||
}
|
||||
|
||||
// atomicWrite writes data to path via a temp file + rename.
|
||||
func atomicWrite(path string, data []byte) error {
|
||||
tmp := path + ".tmp"
|
||||
if err := os.WriteFile(tmp, data, 0o644); err != nil {
|
||||
return fmt.Errorf("rawcache: write %s: %w", tmp, err)
|
||||
}
|
||||
if err := os.Rename(tmp, path); err != nil {
|
||||
os.Remove(tmp)
|
||||
return fmt.Errorf("rawcache: rename %s: %w", path, err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
173
internal/rawcache/rawcache_test.go
Normal file
173
internal/rawcache/rawcache_test.go
Normal file
@@ -0,0 +1,173 @@
|
||||
package rawcache_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/rawcache"
|
||||
)
|
||||
|
||||
func TestOpenCreatesStructure(t *testing.T) {
|
||||
root := filepath.Join(t.TempDir(), "pkg")
|
||||
d, err := rawcache.Open(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
_ = d
|
||||
|
||||
// Verify structure exists.
|
||||
for _, name := range []string{"a", "b"} {
|
||||
info, err := os.Stat(filepath.Join(root, name))
|
||||
if err != nil {
|
||||
t.Fatalf("slot %s: %v", name, err)
|
||||
}
|
||||
if !info.IsDir() {
|
||||
t.Fatalf("slot %s is not a directory", name)
|
||||
}
|
||||
}
|
||||
|
||||
target, err := os.Readlink(filepath.Join(root, "active"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if target != "a" {
|
||||
t.Errorf("active symlink = %q, want %q", target, "a")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPutAndRead(t *testing.T) {
|
||||
d, err := rawcache.Open(filepath.Join(t.TempDir(), "pkg"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
data := []byte(`{"tag_name":"v1.0.0"}`)
|
||||
if err := d.Put("v1.0.0", data); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if !d.Has("v1.0.0") {
|
||||
t.Error("Has(v1.0.0) = false after Put")
|
||||
}
|
||||
if d.Has("v2.0.0") {
|
||||
t.Error("Has(v2.0.0) = true, should be false")
|
||||
}
|
||||
|
||||
got, err := d.Read("v1.0.0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if string(got) != string(data) {
|
||||
t.Errorf("Read = %q, want %q", got, data)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLatest(t *testing.T) {
|
||||
d, err := rawcache.Open(filepath.Join(t.TempDir(), "pkg"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if latest := d.Latest(); latest != "" {
|
||||
t.Errorf("Latest() = %q before any writes, want empty", latest)
|
||||
}
|
||||
|
||||
if err := d.SetLatest("v1.0.0"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if latest := d.Latest(); latest != "v1.0.0" {
|
||||
t.Errorf("Latest() = %q, want %q", latest, "v1.0.0")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRefreshDoubleBuffer(t *testing.T) {
|
||||
root := filepath.Join(t.TempDir(), "pkg")
|
||||
d, err := rawcache.Open(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Write to active slot (A).
|
||||
d.Put("v1.0.0", []byte(`{"old":true}`))
|
||||
d.SetLatest("v1.0.0")
|
||||
|
||||
// Start a full refresh — writes to standby (B).
|
||||
r, err := d.BeginRefresh()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.Put("v1.0.0", []byte(`{"new":true}`))
|
||||
r.Put("v2.0.0", []byte(`{"tag_name":"v2.0.0"}`))
|
||||
r.SetLatest("v2.0.0")
|
||||
|
||||
// Before commit, active still points to A.
|
||||
if d.Latest() != "v1.0.0" {
|
||||
t.Error("latest should still be v1.0.0 before commit")
|
||||
}
|
||||
old, _ := d.Read("v1.0.0")
|
||||
if string(old) != `{"old":true}` {
|
||||
t.Errorf("active slot should still have old data, got %q", old)
|
||||
}
|
||||
|
||||
// Commit swaps to B.
|
||||
if err := r.Commit(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if d.Latest() != "v2.0.0" {
|
||||
t.Errorf("Latest() = %q after commit, want %q", d.Latest(), "v2.0.0")
|
||||
}
|
||||
if !d.Has("v2.0.0") {
|
||||
t.Error("v2.0.0 should exist after commit")
|
||||
}
|
||||
updated, _ := d.Read("v1.0.0")
|
||||
if string(updated) != `{"new":true}` {
|
||||
t.Errorf("v1.0.0 should be updated after commit, got %q", updated)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRefreshAbort(t *testing.T) {
|
||||
root := filepath.Join(t.TempDir(), "pkg")
|
||||
d, err := rawcache.Open(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
d.Put("v1.0.0", []byte(`original`))
|
||||
d.SetLatest("v1.0.0")
|
||||
|
||||
r, err := d.BeginRefresh()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
r.Put("v99.0.0", []byte(`aborted`))
|
||||
r.Abort()
|
||||
|
||||
// Active slot should be unchanged.
|
||||
if d.Latest() != "v1.0.0" {
|
||||
t.Error("latest should still be v1.0.0 after abort")
|
||||
}
|
||||
if d.Has("v99.0.0") {
|
||||
t.Error("v99.0.0 should not exist after abort")
|
||||
}
|
||||
}
|
||||
|
||||
func TestOpenIdempotent(t *testing.T) {
|
||||
root := filepath.Join(t.TempDir(), "pkg")
|
||||
|
||||
d1, err := rawcache.Open(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
d1.Put("v1.0.0", []byte(`data`))
|
||||
|
||||
// Opening again should not lose data.
|
||||
d2, err := rawcache.Open(root)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !d2.Has("v1.0.0") {
|
||||
t.Error("data lost after re-open")
|
||||
}
|
||||
}
|
||||
50
internal/releases/atomicparsley/variants.go
Normal file
50
internal/releases/atomicparsley/variants.go
Normal file
@@ -0,0 +1,50 @@
|
||||
// Package atomicparsley provides OS/arch classification for AtomicParsley releases.
|
||||
//
|
||||
// AtomicParsley uses non-standard filenames with no platform terms
|
||||
// (e.g. "AtomicParsleyLinux.zip", "AtomicParsleyMacOS.zip"). The generic
|
||||
// filename classifier can't extract OS or arch from these — this package
|
||||
// applies the same hardcoded mapping that the production releases.js uses.
|
||||
package atomicparsleydist
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// TagVariants sets OS, arch, and libc for AtomicParsley assets based on
|
||||
// filename keyword matching. Replicates atomicparsley/releases.js mappings:
|
||||
// - Alpine → linux/x86_64/musl
|
||||
// - Linux → linux/x86_64/gnu
|
||||
// - MacOS → darwin/x86_64
|
||||
// - WindowsX86 → windows/x86/msvc
|
||||
// - Windows → windows/x86_64/msvc
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
if assets[i].OS != "" {
|
||||
continue // already classified
|
||||
}
|
||||
lower := strings.ToLower(assets[i].Filename)
|
||||
switch {
|
||||
case strings.Contains(lower, "alpine"):
|
||||
assets[i].OS = "linux"
|
||||
assets[i].Arch = "x86_64"
|
||||
assets[i].Libc = "musl"
|
||||
case strings.Contains(lower, "linux"):
|
||||
assets[i].OS = "linux"
|
||||
assets[i].Arch = "x86_64"
|
||||
assets[i].Libc = "gnu"
|
||||
case strings.Contains(lower, "macos"):
|
||||
assets[i].OS = "darwin"
|
||||
assets[i].Arch = "x86_64"
|
||||
case strings.Contains(lower, "windowsx86"):
|
||||
assets[i].OS = "windows"
|
||||
assets[i].Arch = "x86"
|
||||
assets[i].Libc = "msvc"
|
||||
case strings.Contains(lower, "windows"):
|
||||
assets[i].OS = "windows"
|
||||
assets[i].Arch = "x86_64"
|
||||
assets[i].Libc = "msvc"
|
||||
}
|
||||
}
|
||||
}
|
||||
39
internal/releases/bun/variants.go
Normal file
39
internal/releases/bun/variants.go
Normal file
@@ -0,0 +1,39 @@
|
||||
// Package bun provides variant tagging for Bun releases.
|
||||
//
|
||||
// Bun publishes -profile (debug) builds and uses a non-standard arch
|
||||
// convention: the default x86_64 build targets x86_64_v3 (AVX2+),
|
||||
// while -baseline targets plain x86_64.
|
||||
package bundist
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// TagVariants tags bun-specific build variants and remaps arch fields.
|
||||
//
|
||||
// Bun's default x86_64 build requires AVX2 (x86_64_v3). The -baseline
|
||||
// build targets plain x86_64. For legacy export, baseline is the one
|
||||
// we serve (matching Node.js behavior), so non-baseline gets a variant
|
||||
// tag. The -baseline suffix is stripped from Filename (but not Download)
|
||||
// so the legacy server sees a clean name.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
lower := strings.ToLower(assets[i].Filename)
|
||||
if strings.Contains(lower, "-profile") {
|
||||
assets[i].Variants = append(assets[i].Variants, "profile")
|
||||
}
|
||||
if assets[i].Arch == "x86_64" {
|
||||
if strings.Contains(lower, "-baseline") {
|
||||
// Baseline is plain x86_64 — strip the suffix from
|
||||
// Filename so the legacy server sees a clean name.
|
||||
assets[i].Filename = strings.Replace(assets[i].Filename, "-baseline", "", 1)
|
||||
} else {
|
||||
// Non-baseline is v3 — tag as variant (excluded from legacy).
|
||||
assets[i].Arch = "x86_64_v3"
|
||||
assets[i].Variants = append(assets[i].Variants, "v3")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
72
internal/releases/chromedist/chromedist.go
Normal file
72
internal/releases/chromedist/chromedist.go
Normal file
@@ -0,0 +1,72 @@
|
||||
// Package chromedist fetches Chrome for Testing release data.
|
||||
//
|
||||
// Google publishes a JSON index of known-good Chrome/ChromeDriver versions at:
|
||||
//
|
||||
// https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json
|
||||
//
|
||||
// Each version entry has per-platform download URLs for chrome, chromedriver,
|
||||
// and chrome-headless-shell.
|
||||
package chromedist
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"iter"
|
||||
"net/http"
|
||||
)
|
||||
|
||||
// Index is the top-level response.
|
||||
type Index struct {
|
||||
Timestamp string `json:"timestamp"`
|
||||
Versions []Version `json:"versions"`
|
||||
}
|
||||
|
||||
// Version is one Chrome for Testing version with its downloads.
|
||||
type Version struct {
|
||||
Version string `json:"version"` // "121.0.6120.0"
|
||||
Revision string `json:"revision"` // "1222902"
|
||||
Downloads map[string][]Download `json:"downloads"` // "chromedriver" → []Download
|
||||
}
|
||||
|
||||
// Download is one platform-specific download URL.
|
||||
type Download struct {
|
||||
Platform string `json:"platform"` // "linux64", "mac-arm64", "mac-x64", "win32", "win64"
|
||||
URL string `json:"url"`
|
||||
}
|
||||
|
||||
// Fetch retrieves the Chrome for Testing release index.
|
||||
//
|
||||
// Yields one batch containing all versions.
|
||||
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]Version, error] {
|
||||
return func(yield func([]Version, error) bool) {
|
||||
url := "https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json"
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("chromedist: %w", err))
|
||||
return
|
||||
}
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("chromedist: fetch: %w", err))
|
||||
return
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
yield(nil, fmt.Errorf("chromedist: fetch: %s", resp.Status))
|
||||
return
|
||||
}
|
||||
|
||||
var idx Index
|
||||
if err := json.NewDecoder(resp.Body).Decode(&idx); err != nil {
|
||||
yield(nil, fmt.Errorf("chromedist: decode: %w", err))
|
||||
return
|
||||
}
|
||||
|
||||
yield(idx.Versions, nil)
|
||||
}
|
||||
}
|
||||
60
internal/releases/cmake/variants.go
Normal file
60
internal/releases/cmake/variants.go
Normal file
@@ -0,0 +1,60 @@
|
||||
package cmakedist
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// TagVariants tags cmake-specific build variants for exclusion from legacy export.
|
||||
//
|
||||
// cmake ships many formats and platforms that webi can't serve:
|
||||
//
|
||||
// - .sh self-extracting installer scripts: webi uses the .tar.gz archives.
|
||||
//
|
||||
// - .tar.Z files (old UNIX compress format): format not recognized by webi.
|
||||
//
|
||||
// - Darwin64 builds (pre-3.6 macOS naming): ancient format, superseded by
|
||||
// the macos-universal builds.
|
||||
//
|
||||
// - sunos-sparc64 builds: unsupported platform (sparc64 arch not recognized).
|
||||
//
|
||||
// - AIX/powerpc builds: unsupported platform.
|
||||
//
|
||||
// - IRIX builds: unsupported platform.
|
||||
//
|
||||
// Note: macos10.N versioned builds (cmake-*-macos10.10-universal.tar.gz) are
|
||||
// NOT dropped. Go correctly classifies them as os="darwin". The Node production
|
||||
// classifier has a gap and can't parse "macos10.10" → that is a known prod bug,
|
||||
// not a Go correctness issue. NODER should treat these as expected differences.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
lower := strings.ToLower(assets[i].Filename)
|
||||
|
||||
// Self-extracting installer scripts — webi uses .tar.gz archives.
|
||||
if strings.HasSuffix(lower, ".sh") {
|
||||
assets[i].Variants = append(assets[i].Variants, "installer")
|
||||
continue
|
||||
}
|
||||
|
||||
// Old UNIX compress format (.tar.Z) — not supported by webi.
|
||||
if strings.HasSuffix(lower, ".tar.z") {
|
||||
assets[i].Variants = append(assets[i].Variants, "legacy-archive")
|
||||
continue
|
||||
}
|
||||
|
||||
// Darwin64 builds: pre-cmake-3.6 macOS naming, superseded by macos-universal.
|
||||
if strings.Contains(lower, "darwin64") {
|
||||
assets[i].Variants = append(assets[i].Variants, "legacy-mac")
|
||||
continue
|
||||
}
|
||||
|
||||
// Unsupported platforms.
|
||||
if strings.Contains(lower, "sunos") ||
|
||||
strings.Contains(lower, "-aix-") ||
|
||||
strings.Contains(lower, "irix") {
|
||||
assets[i].Variants = append(assets[i].Variants, "unsupported-platform")
|
||||
continue
|
||||
}
|
||||
}
|
||||
}
|
||||
28
internal/releases/fish/variants.go
Normal file
28
internal/releases/fish/variants.go
Normal file
@@ -0,0 +1,28 @@
|
||||
// Package fish provides variant tagging for fish shell releases.
|
||||
//
|
||||
// Fish publishes .pkg macOS installers alongside the standard archives.
|
||||
// It also includes a source tarball (fish-{version}.tar.xz) as an
|
||||
// uploaded release asset — no OS or arch in the name, indistinguishable
|
||||
// from binaries by content_type. We tag it explicitly as "source".
|
||||
package fishdist
|
||||
|
||||
import "github.com/webinstall/webi-installers/internal/storage"
|
||||
|
||||
// TagVariants tags fish-specific build variants.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
if assets[i].Format == ".pkg" {
|
||||
assets[i].Variants = append(assets[i].Variants, "installer")
|
||||
}
|
||||
// Source tarball: no OS or arch detected by the classifier.
|
||||
if assets[i].OS == "" && assets[i].Arch == "" {
|
||||
assets[i].Variants = append(assets[i].Variants, "source")
|
||||
}
|
||||
// fish-*.app.zip is a macOS universal binary. Fish's naming puts
|
||||
// arch in Linux filenames (e.g. fish-*-aarch64.tar.xz) but not in
|
||||
// macOS .app.zip. Tag as x86_64; darwin waterfall serves arm64.
|
||||
if assets[i].OS == "darwin" && assets[i].Arch == "" && assets[i].Format == ".app.zip" {
|
||||
assets[i].Arch = "x86_64"
|
||||
}
|
||||
}
|
||||
}
|
||||
94
internal/releases/flutterdist/flutterdist.go
Normal file
94
internal/releases/flutterdist/flutterdist.go
Normal file
@@ -0,0 +1,94 @@
|
||||
// Package flutterdist fetches Flutter release data from Google Storage.
|
||||
//
|
||||
// Flutter publishes per-OS release indexes:
|
||||
//
|
||||
// https://storage.googleapis.com/flutter_infra_release/releases/releases_macos.json
|
||||
// https://storage.googleapis.com/flutter_infra_release/releases/releases_linux.json
|
||||
// https://storage.googleapis.com/flutter_infra_release/releases/releases_windows.json
|
||||
//
|
||||
// Each response has a base_url and a releases array with version, channel,
|
||||
// release_date, archive path, and sha256.
|
||||
package flutterdist
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"iter"
|
||||
"net/http"
|
||||
)
|
||||
|
||||
// index is the top-level JSON structure for one OS endpoint.
|
||||
type index struct {
|
||||
BaseURL string `json:"base_url"`
|
||||
Releases []Release `json:"releases"`
|
||||
}
|
||||
|
||||
// Release is one Flutter release entry.
|
||||
type Release struct {
|
||||
Hash string `json:"hash"` // git commit hash
|
||||
Channel string `json:"channel"` // "stable", "beta", "dev"
|
||||
Version string `json:"version"` // "3.29.2"
|
||||
ReleaseDate string `json:"release_date"` // "2025-03-13T00:14:34.044690Z"
|
||||
Archive string `json:"archive"` // "stable/macos/flutter_macos_arm64_3.29.2-stable.zip"
|
||||
SHA256 string `json:"sha256"`
|
||||
|
||||
// DownloadURL is the fully-qualified URL, assembled from base_url + archive.
|
||||
// Not in the upstream JSON — set by Fetch.
|
||||
DownloadURL string `json:"download_url"`
|
||||
// OS is the platform this entry came from ("macos", "linux", "windows").
|
||||
// Not in the upstream JSON — set by Fetch.
|
||||
OS string `json:"os"`
|
||||
}
|
||||
|
||||
var defaultOSes = []string{"macos", "linux", "windows"}
|
||||
|
||||
// Fetch retrieves Flutter releases for all platforms.
|
||||
//
|
||||
// Yields one batch per OS. The iterator interface exists so callers use
|
||||
// the same pattern as paginated sources.
|
||||
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]Release, error] {
|
||||
return func(yield func([]Release, error) bool) {
|
||||
for _, osName := range defaultOSes {
|
||||
url := fmt.Sprintf(
|
||||
"https://storage.googleapis.com/flutter_infra_release/releases/releases_%s.json",
|
||||
osName,
|
||||
)
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("flutterdist: %w", err))
|
||||
return
|
||||
}
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("flutterdist: fetch %s: %w", osName, err))
|
||||
return
|
||||
}
|
||||
|
||||
var idx index
|
||||
err = json.NewDecoder(resp.Body).Decode(&idx)
|
||||
resp.Body.Close()
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("flutterdist: decode %s: %w", osName, err))
|
||||
return
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
yield(nil, fmt.Errorf("flutterdist: fetch %s: %s", osName, resp.Status))
|
||||
return
|
||||
}
|
||||
|
||||
for i := range idx.Releases {
|
||||
idx.Releases[i].DownloadURL = idx.BaseURL + "/" + idx.Releases[i].Archive
|
||||
idx.Releases[i].OS = osName
|
||||
}
|
||||
|
||||
if !yield(idx.Releases, nil) {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
16
internal/releases/flutterdist/variants.go
Normal file
16
internal/releases/flutterdist/variants.go
Normal file
@@ -0,0 +1,16 @@
|
||||
package flutterdist
|
||||
|
||||
import "github.com/webinstall/webi-installers/internal/storage"
|
||||
|
||||
// TagVariants handles flutter-specific arch defaults.
|
||||
//
|
||||
// Flutter's naming convention: flutter_{os}_{version} for x86_64 builds,
|
||||
// flutter_{os}_arm64_{version} for arm64. The absence of an arch token
|
||||
// means x86_64 — arm64 is always explicit.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
if assets[i].Arch == "" && assets[i].OS != "" {
|
||||
assets[i].Arch = "x86_64"
|
||||
}
|
||||
}
|
||||
}
|
||||
52
internal/releases/git/variants.go
Normal file
52
internal/releases/git/variants.go
Normal file
@@ -0,0 +1,52 @@
|
||||
// Package git provides variant tagging for Git for Windows releases.
|
||||
//
|
||||
// Git for Windows publishes GUI installer .exe files (Git-*-bit.exe),
|
||||
// self-extracting PortableGit archives, and .pdb debug symbol packages
|
||||
// alongside the MinGit .zip that webi installs.
|
||||
package gitdist
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// TagVariants tags git-specific build variants and fixes OS/arch classification.
|
||||
// All git-for-windows releases are Windows-only, but MinGit filenames like
|
||||
// "MinGit-2.33.0-64-bit.zip" have no "windows" indicator — force OS=windows.
|
||||
// MinGit uses "64-bit"/"32-bit" for arch — a convention specific to this project
|
||||
// that the general classifier intentionally does not handle.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
// All git-for-windows assets are Windows. Filenames like
|
||||
// "MinGit-2.33.0-64-bit.zip" have no OS term; set it explicitly.
|
||||
if assets[i].OS == "" {
|
||||
assets[i].OS = "windows"
|
||||
}
|
||||
|
||||
// MinGit uses "64-bit"→x86_64, "32-bit"→x86 naming.
|
||||
// "arm64" is already handled by the general classifier.
|
||||
if assets[i].Arch == "" {
|
||||
lower := strings.ToLower(assets[i].Filename)
|
||||
if strings.Contains(lower, "64-bit") {
|
||||
assets[i].Arch = "x86_64"
|
||||
} else if strings.Contains(lower, "32-bit") {
|
||||
assets[i].Arch = "x86"
|
||||
}
|
||||
}
|
||||
|
||||
lower := strings.ToLower(assets[i].Filename)
|
||||
if assets[i].Format == ".exe" {
|
||||
assets[i].Variants = append(assets[i].Variants, "installer")
|
||||
}
|
||||
if strings.Contains(lower, "portablegit") {
|
||||
assets[i].Variants = append(assets[i].Variants, "installer")
|
||||
}
|
||||
if strings.Contains(lower, "-pdb") || strings.Contains(lower, "pdbs-for-") {
|
||||
assets[i].Variants = append(assets[i].Variants, "pdb")
|
||||
}
|
||||
if strings.Contains(lower, "-busybox") {
|
||||
assets[i].Variants = append(assets[i].Variants, "busybox")
|
||||
}
|
||||
}
|
||||
}
|
||||
33
internal/releases/git/versions.go
Normal file
33
internal/releases/git/versions.go
Normal file
@@ -0,0 +1,33 @@
|
||||
package gitdist
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// NormalizeVersions strips the ".windows.N" suffix from Git for Windows
|
||||
// version strings to match the upstream Git version scheme.
|
||||
//
|
||||
// Git for Windows tags are like "v2.53.0.windows.1" or "v2.53.0.windows.2".
|
||||
// Node.js strips ".windows.1" entirely and replaces ".windows.N" (N>1)
|
||||
// with ".N":
|
||||
//
|
||||
// v2.53.0.windows.1 → v2.53.0
|
||||
// v2.53.0.windows.2 → v2.53.0.2
|
||||
func NormalizeVersions(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
v := assets[i].Version
|
||||
idx := strings.Index(v, ".windows.")
|
||||
if idx < 0 {
|
||||
continue
|
||||
}
|
||||
suffix := v[idx+len(".windows."):]
|
||||
base := v[:idx]
|
||||
if suffix == "1" {
|
||||
assets[i].Version = base
|
||||
} else {
|
||||
assets[i].Version = base + "." + suffix
|
||||
}
|
||||
}
|
||||
}
|
||||
120
internal/releases/gitea/gitea.go
Normal file
120
internal/releases/gitea/gitea.go
Normal file
@@ -0,0 +1,120 @@
|
||||
// Package gitea fetches releases from a Gitea or Forgejo instance.
|
||||
//
|
||||
// Gitea's release API lives under:
|
||||
//
|
||||
// GET {baseurl}/api/v1/repos/{owner}/{repo}/releases
|
||||
//
|
||||
// The response shape is similar to GitHub's but not identical. This package
|
||||
// handles pagination, authentication, and deserialization independently.
|
||||
package gitea
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"iter"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Release is one release from the Gitea releases API.
|
||||
type Release struct {
|
||||
TagName string `json:"tag_name"`
|
||||
Name string `json:"name"`
|
||||
Prerelease bool `json:"prerelease"`
|
||||
Draft bool `json:"draft"`
|
||||
PublishedAt string `json:"published_at"` // "2023-11-05T06:38:05Z"
|
||||
Assets []Asset `json:"assets"`
|
||||
TarballURL string `json:"tarball_url"`
|
||||
ZipballURL string `json:"zipball_url"`
|
||||
}
|
||||
|
||||
// Asset is one downloadable file attached to a release.
|
||||
type Asset struct {
|
||||
Name string `json:"name"` // "pathman-v0.6.0-darwin-amd64.tar.gz"
|
||||
BrowserDownloadURL string `json:"browser_download_url"` // full URL
|
||||
Size int64 `json:"size"`
|
||||
}
|
||||
|
||||
// Auth holds optional credentials for authenticated API access.
|
||||
type Auth struct {
|
||||
Token string // personal access token or API key
|
||||
}
|
||||
|
||||
// Fetch retrieves releases from a Gitea instance, paginating automatically.
|
||||
// Each yield is one page of releases.
|
||||
//
|
||||
// The baseURL should be the Gitea root (e.g. "https://git.rootprojects.org").
|
||||
// The /api/v1 prefix is appended automatically.
|
||||
func Fetch(ctx context.Context, client *http.Client, baseURL, owner, repo string, auth *Auth) iter.Seq2[[]Release, error] {
|
||||
return func(yield func([]Release, error) bool) {
|
||||
base := strings.TrimRight(baseURL, "/")
|
||||
page := 1
|
||||
|
||||
for {
|
||||
url := fmt.Sprintf("%s/api/v1/repos/%s/%s/releases?limit=50&page=%d",
|
||||
base, owner, repo, page)
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("gitea: %w", err))
|
||||
return
|
||||
}
|
||||
req.Header.Set("Accept", "application/json")
|
||||
if auth != nil && auth.Token != "" {
|
||||
req.Header.Set("Authorization", "token "+auth.Token)
|
||||
}
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("gitea: fetch %s: %w", url, err))
|
||||
return
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
resp.Body.Close()
|
||||
yield(nil, fmt.Errorf("gitea: fetch %s: %s", url, resp.Status))
|
||||
return
|
||||
}
|
||||
|
||||
var releases []Release
|
||||
err = json.NewDecoder(resp.Body).Decode(&releases)
|
||||
resp.Body.Close()
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("gitea: decode %s: %w", url, err))
|
||||
return
|
||||
}
|
||||
|
||||
if !yield(releases, nil) {
|
||||
return
|
||||
}
|
||||
|
||||
// Gitea uses Link headers like GitHub for pagination.
|
||||
if nextURL := nextPageURL(resp.Header.Get("Link")); nextURL != "" {
|
||||
url = nextURL
|
||||
page++ // not strictly needed since we follow the URL, but keeps logic clear
|
||||
continue
|
||||
}
|
||||
|
||||
// No next link — also stop if we got fewer results than requested.
|
||||
if len(releases) < 50 {
|
||||
return
|
||||
}
|
||||
page++
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
var reNextLink = regexp.MustCompile(`<([^>]+)>;\s*rel="next"`)
|
||||
|
||||
func nextPageURL(link string) string {
|
||||
if link == "" {
|
||||
return ""
|
||||
}
|
||||
m := reNextLink.FindStringSubmatch(link)
|
||||
if m == nil {
|
||||
return ""
|
||||
}
|
||||
return m[1]
|
||||
}
|
||||
107
internal/releases/gitea/gitea_test.go
Normal file
107
internal/releases/gitea/gitea_test.go
Normal file
@@ -0,0 +1,107 @@
|
||||
package gitea_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/releases/gitea"
|
||||
)
|
||||
|
||||
const testReleases = `[
|
||||
{
|
||||
"tag_name": "v0.6.0",
|
||||
"name": "v0.6.0",
|
||||
"prerelease": false,
|
||||
"draft": false,
|
||||
"published_at": "2023-11-05T06:38:05Z",
|
||||
"tarball_url": "https://example.com/archive/v0.6.0.tar.gz",
|
||||
"zipball_url": "https://example.com/archive/v0.6.0.zip",
|
||||
"assets": [
|
||||
{
|
||||
"name": "tool-v0.6.0-linux-amd64.tar.gz",
|
||||
"browser_download_url": "https://example.com/releases/download/v0.6.0/tool-v0.6.0-linux-amd64.tar.gz",
|
||||
"size": 89215
|
||||
}
|
||||
]
|
||||
}
|
||||
]`
|
||||
|
||||
func TestFetch(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/api/v1/repos/root/tool/releases" {
|
||||
t.Errorf("unexpected path: %s", r.URL.Path)
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
w.Write([]byte(testReleases))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
ctx := context.Background()
|
||||
var all []gitea.Release
|
||||
|
||||
for releases, err := range gitea.Fetch(ctx, srv.Client(), srv.URL, "root", "tool", nil) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
all = append(all, releases...)
|
||||
}
|
||||
|
||||
if len(all) != 1 {
|
||||
t.Fatalf("got %d releases, want 1", len(all))
|
||||
}
|
||||
if all[0].TagName != "v0.6.0" {
|
||||
t.Errorf("TagName = %q, want %q", all[0].TagName, "v0.6.0")
|
||||
}
|
||||
if len(all[0].Assets) != 1 {
|
||||
t.Errorf("got %d assets, want 1", len(all[0].Assets))
|
||||
}
|
||||
if all[0].TarballURL == "" {
|
||||
t.Error("TarballURL is empty")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchAuth(t *testing.T) {
|
||||
var gotAuth string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
gotAuth = r.Header.Get("Authorization")
|
||||
w.Write([]byte("[]"))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
ctx := context.Background()
|
||||
auth := &gitea.Auth{Token: "abc123"}
|
||||
for _, err := range gitea.Fetch(ctx, srv.Client(), srv.URL, "root", "tool", auth) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
if gotAuth != "token abc123" {
|
||||
t.Errorf("Authorization = %q, want %q", gotAuth, "token abc123")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchLive(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("skipping network test in short mode")
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
client := &http.Client{}
|
||||
|
||||
var total int
|
||||
for releases, err := range gitea.Fetch(ctx, client, "https://git.rootprojects.org", "root", "pathman", nil) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
total += len(releases)
|
||||
}
|
||||
|
||||
if total < 1 {
|
||||
t.Errorf("got %d releases, expected at least 1", total)
|
||||
}
|
||||
t.Logf("fetched %d releases", total)
|
||||
}
|
||||
25
internal/releases/gitea/variants.go
Normal file
25
internal/releases/gitea/variants.go
Normal file
@@ -0,0 +1,25 @@
|
||||
// Package gitea provides variant tagging for Gitea releases.
|
||||
//
|
||||
// Gitea publishes "gogit" builds that use an alternative pure-Go Git
|
||||
// backend instead of the default C Git library.
|
||||
package gitea
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// TagVariants tags gitea-specific build variants.
|
||||
//
|
||||
// Files containing "-gogit-" in the filename are tagged with the "gogit"
|
||||
// variant. These use a pure-Go Git backend rather than the default C Git
|
||||
// library.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
lower := strings.ToLower(assets[i].Filename)
|
||||
if strings.Contains(lower, "gogit") {
|
||||
assets[i].Variants = append(assets[i].Variants, "gogit")
|
||||
}
|
||||
}
|
||||
}
|
||||
25
internal/releases/giteasrc/giteasrc.go
Normal file
25
internal/releases/giteasrc/giteasrc.go
Normal file
@@ -0,0 +1,25 @@
|
||||
// Package giteasrc fetches source archives from Gitea/Forgejo releases.
|
||||
//
|
||||
// Some packages are installed from the auto-generated source tarballs
|
||||
// rather than uploaded binary assets. This package fetches releases and
|
||||
// exposes the tarball/zipball URLs.
|
||||
//
|
||||
// Use [gitea] for packages that use uploaded binary assets.
|
||||
package giteasrc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"iter"
|
||||
"net/http"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/releases/gitea"
|
||||
)
|
||||
|
||||
// Fetch retrieves releases from a Gitea instance for the given owner/repo.
|
||||
// Paginates automatically, yielding one batch per API page.
|
||||
//
|
||||
// Callers should use [gitea.Release.TarballURL] and
|
||||
// [gitea.Release.ZipballURL] rather than the Assets list.
|
||||
func Fetch(ctx context.Context, client *http.Client, baseURL, owner, repo string, auth *gitea.Auth) iter.Seq2[[]gitea.Release, error] {
|
||||
return gitea.Fetch(ctx, client, baseURL, owner, repo, auth)
|
||||
}
|
||||
22
internal/releases/github/github.go
Normal file
22
internal/releases/github/github.go
Normal file
@@ -0,0 +1,22 @@
|
||||
// Package github fetches releases from the GitHub API.
|
||||
//
|
||||
// This is a thin wrapper around [githubish] that sets the base URL to
|
||||
// https://api.github.com. Use [githubish] directly for Gitea, Forgejo,
|
||||
// or other GitHub-compatible forges.
|
||||
package github
|
||||
|
||||
import (
|
||||
"context"
|
||||
"iter"
|
||||
"net/http"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/releases/githubish"
|
||||
)
|
||||
|
||||
const baseURL = "https://api.github.com"
|
||||
|
||||
// Fetch retrieves releases from GitHub for the given owner/repo.
|
||||
// Paginates automatically, yielding one batch per API page.
|
||||
func Fetch(ctx context.Context, client *http.Client, owner, repo string, auth *githubish.Auth) iter.Seq2[[]githubish.Release, error] {
|
||||
return githubish.Fetch(ctx, client, baseURL, owner, repo, auth)
|
||||
}
|
||||
112
internal/releases/githubish/githubish.go
Normal file
112
internal/releases/githubish/githubish.go
Normal file
@@ -0,0 +1,112 @@
|
||||
// Package githubish fetches releases from GitHub-compatible APIs.
|
||||
//
|
||||
// GitHub, Gitea, Forgejo, and other forges expose the same releases
|
||||
// endpoint shape:
|
||||
//
|
||||
// GET /repos/{owner}/{repo}/releases
|
||||
//
|
||||
// This package handles pagination (Link headers), authentication, and
|
||||
// deserialization. It does not transform or normalize the data.
|
||||
package githubish
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"iter"
|
||||
"net/http"
|
||||
"regexp"
|
||||
)
|
||||
|
||||
// Release is one release from a GitHub-compatible API.
|
||||
// Fields mirror the upstream JSON — only the fields Webi cares about are
|
||||
// included; the rest are silently dropped by the decoder.
|
||||
type Release struct {
|
||||
TagName string `json:"tag_name"`
|
||||
Name string `json:"name"`
|
||||
Prerelease bool `json:"prerelease"`
|
||||
Draft bool `json:"draft"`
|
||||
PublishedAt string `json:"published_at"` // "2025-10-22T13:00:26Z"
|
||||
Assets []Asset `json:"assets"`
|
||||
TarballURL string `json:"tarball_url"` // auto-generated source tarball
|
||||
ZipballURL string `json:"zipball_url"` // auto-generated source zipball
|
||||
}
|
||||
|
||||
// Asset is one downloadable file attached to a release.
|
||||
type Asset struct {
|
||||
Name string `json:"name"` // "ripgrep-15.1.0-x86_64-apple-darwin.tar.gz"
|
||||
BrowserDownloadURL string `json:"browser_download_url"` // full URL
|
||||
Size int64 `json:"size"`
|
||||
ContentType string `json:"content_type"`
|
||||
}
|
||||
|
||||
// Auth holds optional credentials for authenticated API access.
|
||||
// Without auth, GitHub's public rate limit is 60 requests/hour.
|
||||
type Auth struct {
|
||||
Token string // personal access token or fine-grained token
|
||||
}
|
||||
|
||||
// Fetch retrieves releases from a GitHub-compatible API, paginating
|
||||
// automatically. Each yield is one page of releases.
|
||||
//
|
||||
// The baseURL should be the API root (e.g. "https://api.github.com").
|
||||
// For Gitea: "https://gitea.example.com/api/v1".
|
||||
func Fetch(ctx context.Context, client *http.Client, baseURL, owner, repo string, auth *Auth) iter.Seq2[[]Release, error] {
|
||||
return func(yield func([]Release, error) bool) {
|
||||
url := fmt.Sprintf("%s/repos/%s/%s/releases?per_page=100", baseURL, owner, repo)
|
||||
|
||||
for url != "" {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("githubish: %w", err))
|
||||
return
|
||||
}
|
||||
req.Header.Set("Accept", "application/json")
|
||||
if auth != nil && auth.Token != "" {
|
||||
req.Header.Set("Authorization", "Bearer "+auth.Token)
|
||||
}
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("githubish: fetch %s: %w", url, err))
|
||||
return
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
resp.Body.Close()
|
||||
yield(nil, fmt.Errorf("githubish: fetch %s: %s", url, resp.Status))
|
||||
return
|
||||
}
|
||||
|
||||
var releases []Release
|
||||
err = json.NewDecoder(resp.Body).Decode(&releases)
|
||||
resp.Body.Close()
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("githubish: decode %s: %w", url, err))
|
||||
return
|
||||
}
|
||||
|
||||
if !yield(releases, nil) {
|
||||
return
|
||||
}
|
||||
|
||||
url = nextPageURL(resp.Header.Get("Link"))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// reNextLink matches `<URL>; rel="next"` in a Link header.
|
||||
var reNextLink = regexp.MustCompile(`<([^>]+)>;\s*rel="next"`)
|
||||
|
||||
// nextPageURL extracts the "next" URL from a GitHub Link header.
|
||||
// Returns "" if there is no next page.
|
||||
func nextPageURL(link string) string {
|
||||
if link == "" {
|
||||
return ""
|
||||
}
|
||||
m := reNextLink.FindStringSubmatch(link)
|
||||
if m == nil {
|
||||
return ""
|
||||
}
|
||||
return m[1]
|
||||
}
|
||||
201
internal/releases/githubish/githubish_test.go
Normal file
201
internal/releases/githubish/githubish_test.go
Normal file
@@ -0,0 +1,201 @@
|
||||
package githubish_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/releases/githubish"
|
||||
)
|
||||
|
||||
const page1 = `[
|
||||
{
|
||||
"tag_name": "v2.0.0",
|
||||
"name": "v2.0.0",
|
||||
"prerelease": false,
|
||||
"draft": false,
|
||||
"published_at": "2025-06-01T12:00:00Z",
|
||||
"assets": [
|
||||
{
|
||||
"name": "tool-v2.0.0-linux-amd64.tar.gz",
|
||||
"browser_download_url": "https://example.com/tool-v2.0.0-linux-amd64.tar.gz",
|
||||
"size": 5000000,
|
||||
"content_type": "application/gzip"
|
||||
}
|
||||
]
|
||||
}
|
||||
]`
|
||||
|
||||
const page2 = `[
|
||||
{
|
||||
"tag_name": "v1.0.0",
|
||||
"name": "v1.0.0",
|
||||
"prerelease": false,
|
||||
"draft": false,
|
||||
"published_at": "2024-01-15T08:00:00Z",
|
||||
"assets": [
|
||||
{
|
||||
"name": "tool-v1.0.0-linux-amd64.tar.gz",
|
||||
"browser_download_url": "https://example.com/tool-v1.0.0-linux-amd64.tar.gz",
|
||||
"size": 4000000,
|
||||
"content_type": "application/gzip"
|
||||
},
|
||||
{
|
||||
"name": "tool-v1.0.0-darwin-arm64.tar.gz",
|
||||
"browser_download_url": "https://example.com/tool-v1.0.0-darwin-arm64.tar.gz",
|
||||
"size": 4500000,
|
||||
"content_type": "application/gzip"
|
||||
}
|
||||
]
|
||||
}
|
||||
]`
|
||||
|
||||
func TestFetchPagination(t *testing.T) {
|
||||
var srvURL string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/repos/acme/tool/releases" {
|
||||
t.Errorf("unexpected path: %s", r.URL.Path)
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
page := r.URL.Query().Get("page")
|
||||
switch page {
|
||||
case "", "1":
|
||||
// Link header pointing to page 2
|
||||
w.Header().Set("Link",
|
||||
fmt.Sprintf(`<%s/repos/acme/tool/releases?per_page=100&page=2>; rel="next"`, srvURL))
|
||||
w.Write([]byte(page1))
|
||||
case "2":
|
||||
// No Link header — last page
|
||||
w.Write([]byte(page2))
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
srvURL = srv.URL
|
||||
|
||||
ctx := context.Background()
|
||||
var batches int
|
||||
var allReleases []githubish.Release
|
||||
|
||||
for releases, err := range githubish.Fetch(ctx, srv.Client(), srv.URL, "acme", "tool", nil) {
|
||||
if err != nil {
|
||||
t.Fatalf("batch %d: %v", batches, err)
|
||||
}
|
||||
batches++
|
||||
allReleases = append(allReleases, releases...)
|
||||
}
|
||||
|
||||
if batches != 2 {
|
||||
t.Errorf("got %d batches, want 2", batches)
|
||||
}
|
||||
if len(allReleases) != 2 {
|
||||
t.Fatalf("got %d releases, want 2", len(allReleases))
|
||||
}
|
||||
|
||||
// Page 1: v2.0.0
|
||||
if allReleases[0].TagName != "v2.0.0" {
|
||||
t.Errorf("release[0].TagName = %q, want %q", allReleases[0].TagName, "v2.0.0")
|
||||
}
|
||||
if len(allReleases[0].Assets) != 1 {
|
||||
t.Errorf("release[0] has %d assets, want 1", len(allReleases[0].Assets))
|
||||
}
|
||||
|
||||
// Page 2: v1.0.0
|
||||
if allReleases[1].TagName != "v1.0.0" {
|
||||
t.Errorf("release[1].TagName = %q, want %q", allReleases[1].TagName, "v1.0.0")
|
||||
}
|
||||
if len(allReleases[1].Assets) != 2 {
|
||||
t.Errorf("release[1] has %d assets, want 2", len(allReleases[1].Assets))
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchPrerelease(t *testing.T) {
|
||||
body := `[{"tag_name":"v1.0.0-rc1","name":"","prerelease":true,"draft":false,"published_at":"2025-01-01T00:00:00Z","assets":[]}]`
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Write([]byte(body))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
ctx := context.Background()
|
||||
for releases, err := range githubish.Fetch(ctx, srv.Client(), srv.URL, "acme", "tool", nil) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(releases) != 1 {
|
||||
t.Fatalf("got %d releases, want 1", len(releases))
|
||||
}
|
||||
if !releases[0].Prerelease {
|
||||
t.Error("expected Prerelease = true")
|
||||
}
|
||||
if releases[0].TagName != "v1.0.0-rc1" {
|
||||
t.Errorf("TagName = %q, want %q", releases[0].TagName, "v1.0.0-rc1")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchAuth(t *testing.T) {
|
||||
var gotAuth string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
gotAuth = r.Header.Get("Authorization")
|
||||
w.Write([]byte("[]"))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
ctx := context.Background()
|
||||
auth := &githubish.Auth{Token: "ghp_test123"}
|
||||
for _, err := range githubish.Fetch(ctx, srv.Client(), srv.URL, "acme", "tool", auth) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
if gotAuth != "Bearer ghp_test123" {
|
||||
t.Errorf("Authorization = %q, want %q", gotAuth, "Bearer ghp_test123")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchHTTPError(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, "not found", http.StatusNotFound)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
ctx := context.Background()
|
||||
for _, err := range githubish.Fetch(ctx, srv.Client(), srv.URL, "acme", "tool", nil) {
|
||||
if err == nil {
|
||||
t.Fatal("expected error for 404 response")
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchEarlyBreak(t *testing.T) {
|
||||
var requests int
|
||||
var srvURL string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
requests++
|
||||
// Always advertise a next page
|
||||
w.Header().Set("Link",
|
||||
fmt.Sprintf(`<%s/repos/acme/tool/releases?per_page=100&page=%d>; rel="next"`, srvURL, requests+1))
|
||||
w.Write([]byte(`[{"tag_name":"v1.0.0","name":"","prerelease":false,"draft":false,"published_at":"2025-01-01T00:00:00Z","assets":[]}]`))
|
||||
}))
|
||||
defer srv.Close()
|
||||
srvURL = srv.URL
|
||||
|
||||
ctx := context.Background()
|
||||
for _, err := range githubish.Fetch(ctx, srv.Client(), srv.URL, "acme", "tool", nil) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
break // stop after first page
|
||||
}
|
||||
|
||||
if requests != 1 {
|
||||
t.Errorf("server received %d requests, want 1 (early break should stop pagination)", requests)
|
||||
}
|
||||
}
|
||||
27
internal/releases/githubsrc/githubsrc.go
Normal file
27
internal/releases/githubsrc/githubsrc.go
Normal file
@@ -0,0 +1,27 @@
|
||||
// Package githubsrc fetches source archives from GitHub releases.
|
||||
//
|
||||
// Some packages (shell scripts, vim plugins) are installed from the
|
||||
// auto-generated source tarballs rather than uploaded binary assets.
|
||||
// This package fetches releases and exposes the tarball/zipball URLs.
|
||||
//
|
||||
// Use [github] for packages that use uploaded binary assets.
|
||||
package githubsrc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"iter"
|
||||
"net/http"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/releases/githubish"
|
||||
)
|
||||
|
||||
const baseURL = "https://api.github.com"
|
||||
|
||||
// Fetch retrieves releases from GitHub for the given owner/repo.
|
||||
// Paginates automatically, yielding one batch per API page.
|
||||
//
|
||||
// Callers should use [githubish.Release.TarballURL] and
|
||||
// [githubish.Release.ZipballURL] rather than the Assets list.
|
||||
func Fetch(ctx context.Context, client *http.Client, owner, repo string, auth *githubish.Auth) iter.Seq2[[]githubish.Release, error] {
|
||||
return githubish.Fetch(ctx, client, baseURL, owner, repo, auth)
|
||||
}
|
||||
122
internal/releases/gitlab/gitlab.go
Normal file
122
internal/releases/gitlab/gitlab.go
Normal file
@@ -0,0 +1,122 @@
|
||||
// Package gitlab fetches releases from a GitLab instance.
|
||||
//
|
||||
// GitLab's releases API differs from GitHub's in structure:
|
||||
//
|
||||
// GET /api/v4/projects/:id/releases
|
||||
//
|
||||
// Where :id is the URL-encoded project path (e.g. "group%2Frepo") or a
|
||||
// numeric project ID. Assets are split into auto-generated source archives
|
||||
// and manually attached links. Pagination uses page/per_page query params
|
||||
// and X-Total-Pages response headers (not Link headers).
|
||||
//
|
||||
// This package handles pagination, authentication, and deserialization.
|
||||
// It does not transform or normalize the data.
|
||||
package gitlab
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"iter"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"strconv"
|
||||
)
|
||||
|
||||
// Release is one release from the GitLab releases API.
|
||||
type Release struct {
|
||||
TagName string `json:"tag_name"`
|
||||
Name string `json:"name"`
|
||||
ReleasedAt string `json:"released_at"` // "2025-10-22T13:00:26Z"
|
||||
Assets Assets `json:"assets"`
|
||||
}
|
||||
|
||||
// Assets holds both auto-generated source archives and attached links.
|
||||
type Assets struct {
|
||||
Sources []Source `json:"sources"`
|
||||
Links []Link `json:"links"`
|
||||
}
|
||||
|
||||
// Source is an auto-generated source archive (tar.gz, zip, etc.).
|
||||
type Source struct {
|
||||
Format string `json:"format"` // "zip", "tar.gz", "tar.bz2", "tar"
|
||||
URL string `json:"url"`
|
||||
}
|
||||
|
||||
// Link is a file attached to a release (binary, package, etc.).
|
||||
type Link struct {
|
||||
ID int `json:"id"`
|
||||
Name string `json:"name"`
|
||||
URL string `json:"url"`
|
||||
DirectAssetPath string `json:"direct_asset_path"`
|
||||
LinkType string `json:"link_type"` // "other", "runbook", "image", "package"
|
||||
}
|
||||
|
||||
// Auth holds optional credentials for authenticated API access.
|
||||
type Auth struct {
|
||||
Token string // personal access token or deploy token
|
||||
}
|
||||
|
||||
// Fetch retrieves releases from a GitLab instance, paginating automatically.
|
||||
// Each yield is one page of releases.
|
||||
//
|
||||
// The baseURL should be the GitLab root (e.g. "https://gitlab.com").
|
||||
// The project is identified by its path (e.g. "group/repo") — it will be
|
||||
// URL-encoded automatically.
|
||||
func Fetch(ctx context.Context, client *http.Client, baseURL, project string, auth *Auth) iter.Seq2[[]Release, error] {
|
||||
return func(yield func([]Release, error) bool) {
|
||||
encodedProject := url.PathEscape(project)
|
||||
page := 1
|
||||
|
||||
for {
|
||||
reqURL := fmt.Sprintf("%s/api/v4/projects/%s/releases?per_page=100&page=%d",
|
||||
baseURL, encodedProject, page)
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, reqURL, nil)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("gitlab: %w", err))
|
||||
return
|
||||
}
|
||||
req.Header.Set("Accept", "application/json")
|
||||
if auth != nil && auth.Token != "" {
|
||||
req.Header.Set("PRIVATE-TOKEN", auth.Token)
|
||||
}
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("gitlab: fetch %s: %w", reqURL, err))
|
||||
return
|
||||
}
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
resp.Body.Close()
|
||||
yield(nil, fmt.Errorf("gitlab: fetch %s: %s", reqURL, resp.Status))
|
||||
return
|
||||
}
|
||||
|
||||
var releases []Release
|
||||
err = json.NewDecoder(resp.Body).Decode(&releases)
|
||||
resp.Body.Close()
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("gitlab: decode %s: %w", reqURL, err))
|
||||
return
|
||||
}
|
||||
|
||||
if !yield(releases, nil) {
|
||||
return
|
||||
}
|
||||
|
||||
// Check if there are more pages.
|
||||
totalPages := 1
|
||||
if tp := resp.Header.Get("X-Total-Pages"); tp != "" {
|
||||
if n, err := strconv.Atoi(tp); err == nil {
|
||||
totalPages = n
|
||||
}
|
||||
}
|
||||
if page >= totalPages {
|
||||
return
|
||||
}
|
||||
page++
|
||||
}
|
||||
}
|
||||
}
|
||||
182
internal/releases/gitlab/gitlab_test.go
Normal file
182
internal/releases/gitlab/gitlab_test.go
Normal file
@@ -0,0 +1,182 @@
|
||||
package gitlab_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/releases/gitlab"
|
||||
)
|
||||
|
||||
const page1 = `[
|
||||
{
|
||||
"tag_name": "v2.0.0",
|
||||
"name": "v2.0.0",
|
||||
"released_at": "2025-06-01T12:00:00Z",
|
||||
"assets": {
|
||||
"sources": [
|
||||
{"format": "tar.gz", "url": "https://example.com/archive/v2.0.0.tar.gz"},
|
||||
{"format": "zip", "url": "https://example.com/archive/v2.0.0.zip"}
|
||||
],
|
||||
"links": [
|
||||
{
|
||||
"id": 1,
|
||||
"name": "tool-v2.0.0-linux-amd64.tar.gz",
|
||||
"url": "https://example.com/tool-v2.0.0-linux-amd64.tar.gz",
|
||||
"direct_asset_path": "/binaries/linux-amd64",
|
||||
"link_type": "package"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]`
|
||||
|
||||
const page2 = `[
|
||||
{
|
||||
"tag_name": "v1.0.0",
|
||||
"name": "v1.0.0",
|
||||
"released_at": "2024-01-15T08:00:00Z",
|
||||
"assets": {
|
||||
"sources": [
|
||||
{"format": "tar.gz", "url": "https://example.com/archive/v1.0.0.tar.gz"}
|
||||
],
|
||||
"links": []
|
||||
}
|
||||
}
|
||||
]`
|
||||
|
||||
func TestFetchPagination(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
// Go's http server decodes %2F back to /, so check RawPath
|
||||
// for the encoded form or Path for the decoded form.
|
||||
wantRaw := "/api/v4/projects/group%2Ftool/releases"
|
||||
wantDecoded := "/api/v4/projects/group/tool/releases"
|
||||
if r.URL.RawPath != wantRaw && r.URL.Path != wantDecoded {
|
||||
t.Errorf("unexpected path: raw=%q decoded=%q", r.URL.RawPath, r.URL.Path)
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
|
||||
page := r.URL.Query().Get("page")
|
||||
w.Header().Set("X-Total-Pages", "2")
|
||||
|
||||
switch page {
|
||||
case "", "1":
|
||||
w.Write([]byte(page1))
|
||||
case "2":
|
||||
w.Write([]byte(page2))
|
||||
default:
|
||||
http.NotFound(w, r)
|
||||
}
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
ctx := context.Background()
|
||||
var batches int
|
||||
var allReleases []gitlab.Release
|
||||
|
||||
for releases, err := range gitlab.Fetch(ctx, srv.Client(), srv.URL, "group/tool", nil) {
|
||||
if err != nil {
|
||||
t.Fatalf("batch %d: %v", batches, err)
|
||||
}
|
||||
batches++
|
||||
allReleases = append(allReleases, releases...)
|
||||
}
|
||||
|
||||
if batches != 2 {
|
||||
t.Errorf("got %d batches, want 2", batches)
|
||||
}
|
||||
if len(allReleases) != 2 {
|
||||
t.Fatalf("got %d releases, want 2", len(allReleases))
|
||||
}
|
||||
|
||||
// Page 1: v2.0.0
|
||||
r1 := allReleases[0]
|
||||
if r1.TagName != "v2.0.0" {
|
||||
t.Errorf("release[0].TagName = %q, want %q", r1.TagName, "v2.0.0")
|
||||
}
|
||||
if len(r1.Assets.Sources) != 2 {
|
||||
t.Errorf("release[0] has %d sources, want 2", len(r1.Assets.Sources))
|
||||
}
|
||||
if len(r1.Assets.Links) != 1 {
|
||||
t.Errorf("release[0] has %d links, want 1", len(r1.Assets.Links))
|
||||
}
|
||||
if r1.Assets.Links[0].LinkType != "package" {
|
||||
t.Errorf("release[0] link type = %q, want %q", r1.Assets.Links[0].LinkType, "package")
|
||||
}
|
||||
|
||||
// Page 2: v1.0.0
|
||||
r2 := allReleases[1]
|
||||
if r2.TagName != "v1.0.0" {
|
||||
t.Errorf("release[1].TagName = %q, want %q", r2.TagName, "v1.0.0")
|
||||
}
|
||||
if len(r2.Assets.Links) != 0 {
|
||||
t.Errorf("release[1] has %d links, want 0", len(r2.Assets.Links))
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchAuth(t *testing.T) {
|
||||
var gotAuth string
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
gotAuth = r.Header.Get("PRIVATE-TOKEN")
|
||||
w.Write([]byte("[]"))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
ctx := context.Background()
|
||||
auth := &gitlab.Auth{Token: "glpat-test123"}
|
||||
for _, err := range gitlab.Fetch(ctx, srv.Client(), srv.URL, "group/tool", auth) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
if gotAuth != "glpat-test123" {
|
||||
t.Errorf("PRIVATE-TOKEN = %q, want %q", gotAuth, "glpat-test123")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchSinglePage(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
// No X-Total-Pages header — defaults to 1 page.
|
||||
w.Write([]byte(page1))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
ctx := context.Background()
|
||||
var batches int
|
||||
for _, err := range gitlab.Fetch(ctx, srv.Client(), srv.URL, "group/tool", nil) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
batches++
|
||||
}
|
||||
|
||||
if batches != 1 {
|
||||
t.Errorf("got %d batches, want 1 (no X-Total-Pages means single page)", batches)
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchEarlyBreak(t *testing.T) {
|
||||
var requests int
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
requests++
|
||||
w.Header().Set("X-Total-Pages", "10")
|
||||
w.Write([]byte(fmt.Sprintf(`[{"tag_name":"v%d.0.0","name":"","released_at":"2025-01-01T00:00:00Z","assets":{"sources":[],"links":[]}}]`, requests)))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
ctx := context.Background()
|
||||
for _, err := range gitlab.Fetch(ctx, srv.Client(), srv.URL, "group/tool", nil) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
break // stop after first page
|
||||
}
|
||||
|
||||
if requests != 1 {
|
||||
t.Errorf("server received %d requests, want 1", requests)
|
||||
}
|
||||
}
|
||||
25
internal/releases/gitlabsrc/gitlabsrc.go
Normal file
25
internal/releases/gitlabsrc/gitlabsrc.go
Normal file
@@ -0,0 +1,25 @@
|
||||
// Package gitlabsrc fetches source archives from GitLab releases.
|
||||
//
|
||||
// Some packages are installed from the auto-generated source archives
|
||||
// rather than attached binary links. This package fetches releases and
|
||||
// exposes the source archive URLs.
|
||||
//
|
||||
// Use [gitlab] for packages that use attached release links (binaries).
|
||||
package gitlabsrc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"iter"
|
||||
"net/http"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/releases/gitlab"
|
||||
)
|
||||
|
||||
// Fetch retrieves releases from a GitLab instance.
|
||||
// Paginates automatically, yielding one batch per API page.
|
||||
//
|
||||
// Callers should use [gitlab.Release.Assets.Sources] rather than
|
||||
// [gitlab.Release.Assets.Links].
|
||||
func Fetch(ctx context.Context, client *http.Client, baseURL, project string, auth *gitlab.Auth) iter.Seq2[[]gitlab.Release, error] {
|
||||
return gitlab.Fetch(ctx, client, baseURL, project, auth)
|
||||
}
|
||||
178
internal/releases/gittag/gittag.go
Normal file
178
internal/releases/gittag/gittag.go
Normal file
@@ -0,0 +1,178 @@
|
||||
// Package gittag fetches release information from git tags in a bare repo.
|
||||
//
|
||||
// Some packages (vim plugins, shell scripts) are installed by cloning a git
|
||||
// repo rather than downloading a binary. For these, each tag is a "release"
|
||||
// and the download URL is the repo's git URL.
|
||||
//
|
||||
// This package clones (or fetches) a bare repo to a local cache directory,
|
||||
// lists version-like tags, and returns them with their commit metadata.
|
||||
// HEAD is also included as a potential release.
|
||||
package gittag
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"iter"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"crypto/rand"
|
||||
"encoding/hex"
|
||||
)
|
||||
|
||||
// Entry is one tag (or HEAD) from a git repo.
|
||||
type Entry struct {
|
||||
Version string // tag name or date-based version for HEAD
|
||||
GitTag string // the ref that can be passed to `git clone --branch`
|
||||
CommitHash string // abbreviated commit hash
|
||||
Date string // ISO 8601 commit date (author date)
|
||||
}
|
||||
|
||||
// reVersionTag matches tags that look like versions: v1, v1.2, 1.0.0-rc, etc.
|
||||
var reVersionTag = regexp.MustCompile(`^v?\d+(\.\d+)`)
|
||||
|
||||
// Fetch clones or updates a bare repo, then yields its version-like tags
|
||||
// and HEAD as entries. The repoDir is the parent directory where bare repos
|
||||
// are cached.
|
||||
//
|
||||
// Yields one batch containing all tags plus HEAD.
|
||||
func Fetch(ctx context.Context, gitURL, repoDir string) iter.Seq2[[]Entry, error] {
|
||||
return func(yield func([]Entry, error) bool) {
|
||||
repoName := filepath.Base(gitURL)
|
||||
repoName = strings.TrimSuffix(repoName, ".git")
|
||||
repoPath := filepath.Join(repoDir, repoName+".git")
|
||||
|
||||
if err := ensureRepo(ctx, repoPath, gitURL); err != nil {
|
||||
yield(nil, fmt.Errorf("gittag: %w", err))
|
||||
return
|
||||
}
|
||||
|
||||
tags, err := listVersionTags(ctx, repoPath)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("gittag: %w", err))
|
||||
return
|
||||
}
|
||||
|
||||
var entries []Entry
|
||||
for _, tag := range tags {
|
||||
info, err := commitInfo(ctx, repoPath, tag)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("gittag: commit info for %q: %w", tag, err))
|
||||
return
|
||||
}
|
||||
info.Version = tag
|
||||
info.GitTag = tag
|
||||
entries = append(entries, info)
|
||||
}
|
||||
|
||||
// HEAD as an additional entry
|
||||
head, err := commitInfo(ctx, repoPath, "HEAD")
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("gittag: commit info for HEAD: %w", err))
|
||||
return
|
||||
}
|
||||
branch, err := headBranch(ctx, repoPath)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("gittag: HEAD branch: %w", err))
|
||||
return
|
||||
}
|
||||
head.GitTag = branch
|
||||
// Version for HEAD is set by the caller (date-based, etc.)
|
||||
entries = append(entries, head)
|
||||
|
||||
yield(entries, nil)
|
||||
}
|
||||
}
|
||||
|
||||
// ensureRepo clones the repo if it doesn't exist, or fetches if it does.
|
||||
func ensureRepo(ctx context.Context, repoPath, gitURL string) error {
|
||||
if _, err := os.Stat(repoPath); err == nil {
|
||||
// Exists — fetch updates.
|
||||
cmd := exec.CommandContext(ctx, "git", "--git-dir="+repoPath, "fetch")
|
||||
cmd.Stderr = os.Stderr
|
||||
return cmd.Run()
|
||||
}
|
||||
|
||||
// Clone bare with tree filter (metadata only).
|
||||
var b [8]byte
|
||||
rand.Read(b[:])
|
||||
id := hex.EncodeToString(b[:])
|
||||
tmpPath := repoPath + "." + id + ".tmp"
|
||||
|
||||
cmd := exec.CommandContext(ctx, "git", "clone", "--bare", "--filter=tree:0", gitURL, tmpPath)
|
||||
cmd.Stderr = os.Stderr
|
||||
if err := cmd.Run(); err != nil {
|
||||
os.RemoveAll(tmpPath)
|
||||
return fmt.Errorf("clone %s: %w", gitURL, err)
|
||||
}
|
||||
|
||||
// Atomic swap — if repoPath appeared in a race, keep it and discard ours.
|
||||
if err := os.Rename(tmpPath, repoPath); err != nil {
|
||||
os.RemoveAll(tmpPath)
|
||||
// If rename failed because repoPath now exists, that's fine.
|
||||
if _, statErr := os.Stat(repoPath); statErr == nil {
|
||||
return nil
|
||||
}
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// listVersionTags returns tags that look like version numbers, newest first.
|
||||
func listVersionTags(ctx context.Context, repoPath string) ([]string, error) {
|
||||
cmd := exec.CommandContext(ctx, "git", "--git-dir="+repoPath, "tag")
|
||||
out, err := cmd.Output()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("git tag: %w", err)
|
||||
}
|
||||
|
||||
var tags []string
|
||||
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
|
||||
if line == "" {
|
||||
continue
|
||||
}
|
||||
if reVersionTag.MatchString(line) {
|
||||
tags = append(tags, line)
|
||||
}
|
||||
}
|
||||
|
||||
// Reverse so newest tags come first (git tag outputs alphabetically).
|
||||
for i, j := 0, len(tags)-1; i < j; i, j = i+1, j-1 {
|
||||
tags[i], tags[j] = tags[j], tags[i]
|
||||
}
|
||||
return tags, nil
|
||||
}
|
||||
|
||||
// commitInfo returns the abbreviated hash and author date for a commitish.
|
||||
func commitInfo(ctx context.Context, repoPath, commitish string) (Entry, error) {
|
||||
cmd := exec.CommandContext(ctx, "git", "--git-dir="+repoPath,
|
||||
"log", "-1", "--format=%h %ad", "--date=iso-strict", commitish)
|
||||
out, err := cmd.Output()
|
||||
if err != nil {
|
||||
return Entry{}, fmt.Errorf("git log %s: %w", commitish, err)
|
||||
}
|
||||
|
||||
parts := strings.Fields(strings.TrimSpace(string(out)))
|
||||
if len(parts) < 2 {
|
||||
return Entry{}, fmt.Errorf("unexpected git log output: %q", out)
|
||||
}
|
||||
|
||||
return Entry{
|
||||
CommitHash: parts[0],
|
||||
Date: parts[1],
|
||||
}, nil
|
||||
}
|
||||
|
||||
// headBranch returns the symbolic ref for HEAD (e.g. "main", "master").
|
||||
func headBranch(ctx context.Context, repoPath string) (string, error) {
|
||||
cmd := exec.CommandContext(ctx, "git", "--git-dir="+repoPath,
|
||||
"rev-parse", "--abbrev-ref", "HEAD")
|
||||
out, err := cmd.Output()
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("git rev-parse HEAD: %w", err)
|
||||
}
|
||||
return strings.TrimSpace(string(out)), nil
|
||||
}
|
||||
56
internal/releases/gittag/gittag_test.go
Normal file
56
internal/releases/gittag/gittag_test.go
Normal file
@@ -0,0 +1,56 @@
|
||||
package gittag_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/releases/gittag"
|
||||
)
|
||||
|
||||
func TestFetch(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("skipping network/git test in short mode")
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
repoDir := t.TempDir()
|
||||
|
||||
// vim-commentary has a small number of tags.
|
||||
var entries []gittag.Entry
|
||||
for batch, err := range gittag.Fetch(ctx, "https://github.com/tpope/vim-commentary.git", repoDir) {
|
||||
if err != nil {
|
||||
t.Fatalf("Fetch: %v", err)
|
||||
}
|
||||
entries = append(entries, batch...)
|
||||
}
|
||||
|
||||
if len(entries) < 2 {
|
||||
t.Fatalf("got %d entries, expected at least 2 (tags + HEAD)", len(entries))
|
||||
}
|
||||
|
||||
// Last entry should be HEAD (no Version set by the fetcher).
|
||||
head := entries[len(entries)-1]
|
||||
if head.CommitHash == "" {
|
||||
t.Error("HEAD entry has empty CommitHash")
|
||||
}
|
||||
if head.Date == "" {
|
||||
t.Error("HEAD entry has empty Date")
|
||||
}
|
||||
if head.GitTag == "" {
|
||||
t.Error("HEAD entry has empty GitTag (branch name)")
|
||||
}
|
||||
|
||||
// At least one tag entry should have a version.
|
||||
found := false
|
||||
for _, e := range entries[:len(entries)-1] {
|
||||
if e.Version != "" {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
t.Error("no tag entries have a Version set")
|
||||
}
|
||||
|
||||
t.Logf("fetched %d entries (last is HEAD on %q)", len(entries), head.GitTag)
|
||||
}
|
||||
72
internal/releases/golang/golang.go
Normal file
72
internal/releases/golang/golang.go
Normal file
@@ -0,0 +1,72 @@
|
||||
// Package golang fetches Go release data from golang.org.
|
||||
//
|
||||
// The API returns all releases (including unstable) as a JSON array:
|
||||
//
|
||||
// https://golang.org/dl/?mode=json&include=all
|
||||
//
|
||||
// Each release has a version string like "go1.24.1" and a list of file
|
||||
// objects with filename, os, arch, sha256, size, and kind.
|
||||
package golang
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"iter"
|
||||
"net/http"
|
||||
)
|
||||
|
||||
// Release is one Go version from the download API.
|
||||
type Release struct {
|
||||
Version string `json:"version"` // "go1.24.1"
|
||||
Stable bool `json:"stable"`
|
||||
Files []File `json:"files"`
|
||||
}
|
||||
|
||||
// File is one downloadable artifact within a release.
|
||||
type File struct {
|
||||
Filename string `json:"filename"` // "go1.24.1.linux-amd64.tar.gz"
|
||||
OS string `json:"os"` // "linux", "darwin", "windows", ""
|
||||
Arch string `json:"arch"` // "amd64", "arm64", "386", ""
|
||||
Version string `json:"version"` // "go1.24.1"
|
||||
SHA256 string `json:"sha256"`
|
||||
Size int64 `json:"size"`
|
||||
Kind string `json:"kind"` // "archive", "installer", "source"
|
||||
}
|
||||
|
||||
// Fetch retrieves the Go release index.
|
||||
//
|
||||
// Yields one batch containing all releases. The iterator interface exists
|
||||
// so callers use the same pattern as paginated sources.
|
||||
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]Release, error] {
|
||||
return func(yield func([]Release, error) bool) {
|
||||
url := "https://golang.org/dl/?mode=json&include=all"
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("golang: %w", err))
|
||||
return
|
||||
}
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("golang: fetch: %w", err))
|
||||
return
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
yield(nil, fmt.Errorf("golang: fetch: %s", resp.Status))
|
||||
return
|
||||
}
|
||||
|
||||
var releases []Release
|
||||
if err := json.NewDecoder(resp.Body).Decode(&releases); err != nil {
|
||||
yield(nil, fmt.Errorf("golang: decode: %w", err))
|
||||
return
|
||||
}
|
||||
|
||||
yield(releases, nil)
|
||||
}
|
||||
}
|
||||
70
internal/releases/gpgdist/gpgdist.go
Normal file
70
internal/releases/gpgdist/gpgdist.go
Normal file
@@ -0,0 +1,70 @@
|
||||
// Package gpgdist fetches GPG for macOS release data from SourceForge RSS.
|
||||
//
|
||||
// The gpgosx project publishes DMG installers on SourceForge. The RSS feed
|
||||
// at https://sourceforge.net/projects/gpgosx/rss?path=/ lists download links
|
||||
// for each version.
|
||||
package gpgdist
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"iter"
|
||||
"net/http"
|
||||
"regexp"
|
||||
)
|
||||
|
||||
// Entry is one GPG macOS release.
|
||||
type Entry struct {
|
||||
Version string `json:"version"` // "2.4.7"
|
||||
URL string `json:"url"` // full SourceForge download URL
|
||||
}
|
||||
|
||||
var linkRe = regexp.MustCompile(
|
||||
`<link>(https://sourceforge\.net/projects/gpgosx/files/GnuPG-([\d.]+)\.dmg/download)</link>`,
|
||||
)
|
||||
|
||||
// Fetch retrieves GPG macOS releases from the SourceForge RSS feed.
|
||||
//
|
||||
// Yields one batch containing all releases.
|
||||
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]Entry, error] {
|
||||
return func(yield func([]Entry, error) bool) {
|
||||
url := "https://sourceforge.net/projects/gpgosx/rss?path=/"
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("gpgdist: %w", err))
|
||||
return
|
||||
}
|
||||
req.Header.Set("Accept", "application/rss+xml")
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("gpgdist: fetch: %w", err))
|
||||
return
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
yield(nil, fmt.Errorf("gpgdist: fetch: %s", resp.Status))
|
||||
return
|
||||
}
|
||||
|
||||
body, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("gpgdist: read: %w", err))
|
||||
return
|
||||
}
|
||||
|
||||
matches := linkRe.FindAllStringSubmatch(string(body), -1)
|
||||
var entries []Entry
|
||||
for _, m := range matches {
|
||||
entries = append(entries, Entry{
|
||||
URL: m[1],
|
||||
Version: m[2],
|
||||
})
|
||||
}
|
||||
|
||||
yield(entries, nil)
|
||||
}
|
||||
}
|
||||
79
internal/releases/hashicorp/hashicorp.go
Normal file
79
internal/releases/hashicorp/hashicorp.go
Normal file
@@ -0,0 +1,79 @@
|
||||
// Package hashicorp fetches release data from the HashiCorp releases API.
|
||||
//
|
||||
// HashiCorp publishes release indexes at:
|
||||
//
|
||||
// https://releases.hashicorp.com/{product}/index.json
|
||||
//
|
||||
// The response is a JSON object with a "versions" key mapping version strings
|
||||
// to objects containing build arrays with url, os, arch, and filename.
|
||||
package hashicorp
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"iter"
|
||||
"net/http"
|
||||
)
|
||||
|
||||
// Index is the top-level response from the HashiCorp releases API.
|
||||
type Index struct {
|
||||
Versions map[string]Version `json:"versions"`
|
||||
}
|
||||
|
||||
// Version is one release version with its builds.
|
||||
type Version struct {
|
||||
Name string `json:"name"` // "terraform"
|
||||
Version string `json:"version"` // "1.12.0"
|
||||
SHASUMS string `json:"shasums,omitempty"` // URL to SHA256SUMS file
|
||||
SHASUMSSig string `json:"shasums_signature"` // URL to signature
|
||||
Builds []Build `json:"builds"`
|
||||
TimestampCreated string `json:"timestamp_created,omitempty"`
|
||||
TimestampUpdated string `json:"timestamp_updated,omitempty"`
|
||||
}
|
||||
|
||||
// Build is one downloadable artifact.
|
||||
type Build struct {
|
||||
Name string `json:"name"` // "terraform"
|
||||
Version string `json:"version"` // "1.12.0"
|
||||
OS string `json:"os"` // "linux", "darwin", "windows"
|
||||
Arch string `json:"arch"` // "amd64", "arm64", "386"
|
||||
Filename string `json:"filename"` // "terraform_1.12.0_linux_amd64.zip"
|
||||
URL string `json:"url"` // full download URL
|
||||
}
|
||||
|
||||
// Fetch retrieves the HashiCorp release index for a product.
|
||||
//
|
||||
// Yields one batch containing all versions.
|
||||
func Fetch(ctx context.Context, client *http.Client, product string) iter.Seq2[*Index, error] {
|
||||
return func(yield func(*Index, error) bool) {
|
||||
url := fmt.Sprintf("https://releases.hashicorp.com/%s/index.json", product)
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("hashicorp: %w", err))
|
||||
return
|
||||
}
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("hashicorp: fetch %s: %w", product, err))
|
||||
return
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
yield(nil, fmt.Errorf("hashicorp: fetch %s: %s", product, resp.Status))
|
||||
return
|
||||
}
|
||||
|
||||
var idx Index
|
||||
if err := json.NewDecoder(resp.Body).Decode(&idx); err != nil {
|
||||
yield(nil, fmt.Errorf("hashicorp: decode %s: %w", product, err))
|
||||
return
|
||||
}
|
||||
|
||||
yield(&idx, nil)
|
||||
}
|
||||
}
|
||||
105
internal/releases/iterm2dist/iterm2dist.go
Normal file
105
internal/releases/iterm2dist/iterm2dist.go
Normal file
@@ -0,0 +1,105 @@
|
||||
// Package iterm2dist fetches iTerm2 release URLs from the downloads page.
|
||||
//
|
||||
// iTerm2 doesn't have a structured API — releases are listed as links on:
|
||||
//
|
||||
// https://iterm2.com/downloads.html
|
||||
//
|
||||
// This package scrapes download links matching iTerm2-[34]*.zip from the
|
||||
// HTML and returns them as structured entries.
|
||||
package iterm2dist
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"iter"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Entry is one iTerm2 download link with extracted metadata.
|
||||
type Entry struct {
|
||||
Version string `json:"version"` // "3.5.13"
|
||||
Channel string `json:"channel"` // "stable" or "beta"
|
||||
URL string `json:"url"` // full download URL
|
||||
}
|
||||
|
||||
var linkRe = regexp.MustCompile(`href="(https://iterm2\.com/downloads/[^"]*\.zip)"`)
|
||||
var versionRe = regexp.MustCompile(`iTerm2[-_]v?(\d+(?:_\d+)*)(?:[-_]?(beta|preview)[-_]?(\d*))?\.zip`)
|
||||
|
||||
// Fetch retrieves iTerm2 releases by scraping the downloads page.
|
||||
//
|
||||
// Yields one batch containing all releases.
|
||||
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]Entry, error] {
|
||||
return func(yield func([]Entry, error) bool) {
|
||||
url := "https://iterm2.com/downloads.html"
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("iterm2dist: %w", err))
|
||||
return
|
||||
}
|
||||
req.Header.Set("Accept", "text/html")
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("iterm2dist: fetch: %w", err))
|
||||
return
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
yield(nil, fmt.Errorf("iterm2dist: fetch: %s", resp.Status))
|
||||
return
|
||||
}
|
||||
|
||||
body, err := io.ReadAll(resp.Body)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("iterm2dist: read: %w", err))
|
||||
return
|
||||
}
|
||||
|
||||
matches := linkRe.FindAllStringSubmatch(string(body), -1)
|
||||
var entries []Entry
|
||||
seen := make(map[string]bool)
|
||||
for _, m := range matches {
|
||||
link := m[1]
|
||||
// Only include iTerm2 v3+ downloads.
|
||||
if !strings.Contains(link, "iTerm2-3") && !strings.Contains(link, "iTerm2-4") {
|
||||
continue
|
||||
}
|
||||
|
||||
entry := Entry{URL: link}
|
||||
|
||||
// Determine channel from URL path.
|
||||
if strings.Contains(link, "/stable/") {
|
||||
entry.Channel = "stable"
|
||||
} else {
|
||||
entry.Channel = "beta"
|
||||
}
|
||||
|
||||
// Extract version: iTerm2-3_5_13.zip → 3.5.13
|
||||
vm := versionRe.FindStringSubmatch(link)
|
||||
if vm != nil {
|
||||
entry.Version = strings.ReplaceAll(vm[1], "_", ".")
|
||||
// vm[2] = "beta" or "preview", vm[3] = optional number
|
||||
if vm[2] != "" {
|
||||
entry.Version += "-" + vm[2] + vm[3]
|
||||
}
|
||||
}
|
||||
|
||||
// The downloads page has duplicate links for some betas
|
||||
// (e.g. iTerm2-3_5_1beta1.zip and iTerm2-3_5_1_beta1.zip).
|
||||
// Keep the first URL encountered per version.
|
||||
if seen[entry.Version] {
|
||||
continue
|
||||
}
|
||||
seen[entry.Version] = true
|
||||
|
||||
entries = append(entries, entry)
|
||||
}
|
||||
|
||||
yield(entries, nil)
|
||||
}
|
||||
}
|
||||
89
internal/releases/juliadist/juliadist.go
Normal file
89
internal/releases/juliadist/juliadist.go
Normal file
@@ -0,0 +1,89 @@
|
||||
// Package juliadist fetches Julia release data from the Julia S3 API.
|
||||
//
|
||||
// Julia publishes a version index at:
|
||||
//
|
||||
// https://julialang-s3.julialang.org/bin/versions.json
|
||||
//
|
||||
// The response is a JSON object keyed by version string, where each value
|
||||
// has a "files" array of downloadable artifacts with url, triplet, kind,
|
||||
// arch, os, sha256, size, and extension fields.
|
||||
package juliadist
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"iter"
|
||||
"net/http"
|
||||
)
|
||||
|
||||
// Release is one Julia version with its file artifacts.
|
||||
type Release struct {
|
||||
Version string `json:"version"` // set by us from the key
|
||||
Stable bool `json:"stable"`
|
||||
Files []File `json:"files"`
|
||||
}
|
||||
|
||||
// File is one downloadable artifact.
|
||||
type File struct {
|
||||
URL string `json:"url"` // full download URL
|
||||
Triplet string `json:"triplet"` // "aarch64-apple-darwin14"
|
||||
Kind string `json:"kind"` // "archive" or "installer"
|
||||
Arch string `json:"arch"` // "aarch64", "x86_64", "i686"
|
||||
OS string `json:"os"` // "mac", "linux", "winnt"
|
||||
SHA256 string `json:"sha256"`
|
||||
Size int64 `json:"size"`
|
||||
Version string `json:"version"` // same as release version
|
||||
Extension string `json:"extension"` // "tar.gz", "dmg", "exe"
|
||||
}
|
||||
|
||||
// rawRelease is the upstream JSON shape (stable as bool, files array).
|
||||
type rawRelease struct {
|
||||
Stable bool `json:"stable"`
|
||||
Files []File `json:"files"`
|
||||
}
|
||||
|
||||
// Fetch retrieves the Julia release index.
|
||||
//
|
||||
// Yields one batch containing all releases.
|
||||
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]Release, error] {
|
||||
return func(yield func([]Release, error) bool) {
|
||||
url := "https://julialang-s3.julialang.org/bin/versions.json"
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("juliadist: %w", err))
|
||||
return
|
||||
}
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("juliadist: fetch: %w", err))
|
||||
return
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
yield(nil, fmt.Errorf("juliadist: fetch: %s", resp.Status))
|
||||
return
|
||||
}
|
||||
|
||||
var raw map[string]rawRelease
|
||||
if err := json.NewDecoder(resp.Body).Decode(&raw); err != nil {
|
||||
yield(nil, fmt.Errorf("juliadist: decode: %w", err))
|
||||
return
|
||||
}
|
||||
|
||||
var releases []Release
|
||||
for version, r := range raw {
|
||||
releases = append(releases, Release{
|
||||
Version: version,
|
||||
Stable: r.Stable,
|
||||
Files: r.Files,
|
||||
})
|
||||
}
|
||||
|
||||
yield(releases, nil)
|
||||
}
|
||||
}
|
||||
23
internal/releases/lsd/variants.go
Normal file
23
internal/releases/lsd/variants.go
Normal file
@@ -0,0 +1,23 @@
|
||||
// Package lsd provides variant tagging for lsd (LSDeluxe) releases.
|
||||
//
|
||||
// lsd publishes .deb packages and windows-msvc builds alongside
|
||||
// the standard archives.
|
||||
package lsddist
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// TagVariants tags lsd-specific build variants.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
if assets[i].Format == ".deb" {
|
||||
assets[i].Variants = append(assets[i].Variants, "deb")
|
||||
}
|
||||
if strings.Contains(strings.ToLower(assets[i].Filename), "-msvc") {
|
||||
assets[i].Variants = append(assets[i].Variants, "msvc")
|
||||
}
|
||||
}
|
||||
}
|
||||
159
internal/releases/mariadbdist/mariadbdist.go
Normal file
159
internal/releases/mariadbdist/mariadbdist.go
Normal file
@@ -0,0 +1,159 @@
|
||||
// Package mariadbdist fetches MariaDB release data from the downloads API.
|
||||
//
|
||||
// MariaDB publishes release information via a REST API:
|
||||
//
|
||||
// https://downloads.mariadb.org/rest-api/mariadb/
|
||||
// https://downloads.mariadb.org/rest-api/mariadb/{major.minor}/
|
||||
//
|
||||
// The first endpoint lists major release series; the second lists all point
|
||||
// releases within a series, including download URLs per platform.
|
||||
package mariadbdist
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"iter"
|
||||
"net/http"
|
||||
"regexp"
|
||||
)
|
||||
|
||||
// MajorRelease describes one release series (e.g. "11.4").
|
||||
type MajorRelease struct {
|
||||
ReleaseID string `json:"release_id"` // "11.4"
|
||||
ReleaseName string `json:"release_name"` // "MariaDB Server 11.4"
|
||||
ReleaseStatus string `json:"release_status"` // "Stable", "RC", "Alpha"
|
||||
ReleaseSupportType string `json:"release_support_type"` // "Long Term Support", etc.
|
||||
}
|
||||
|
||||
// Release is one point release with its downloadable files.
|
||||
type Release struct {
|
||||
ReleaseID string `json:"release_id"` // "11.4.5"
|
||||
ReleaseName string `json:"release_name"` // "MariaDB Server 11.4.5"
|
||||
DateOfRelease string `json:"date_of_release"` // "2025-02-12"
|
||||
ReleaseNotesURL string `json:"release_notes_url"` // URL
|
||||
Files []File `json:"files"`
|
||||
|
||||
// MajorStatus is copied from the parent MajorRelease. Not in upstream JSON.
|
||||
MajorStatus string `json:"major_status,omitempty"`
|
||||
}
|
||||
|
||||
// File is one downloadable artifact within a release.
|
||||
type File struct {
|
||||
FileID int `json:"file_id"`
|
||||
FileName string `json:"file_name"`
|
||||
PackageType string `json:"package_type"` // "gzipped tar file", "ZIP file"
|
||||
OS string `json:"os"` // "Linux", "Windows", or ""
|
||||
CPU string `json:"cpu"` // "x86_64" or ""
|
||||
Checksum Checksum `json:"checksum"`
|
||||
FileDownloadURL string `json:"file_download_url"`
|
||||
}
|
||||
|
||||
// Checksum holds hash digests for a file.
|
||||
type Checksum struct {
|
||||
SHA256 string `json:"sha256sum"`
|
||||
}
|
||||
|
||||
type majorResp struct {
|
||||
MajorReleases []MajorRelease `json:"major_releases"`
|
||||
}
|
||||
|
||||
type releaseResp struct {
|
||||
Releases map[string]Release `json:"releases"`
|
||||
}
|
||||
|
||||
var reVersion = regexp.MustCompile(`^\d+\.\d+$`)
|
||||
|
||||
// Fetch retrieves all MariaDB releases across all major series.
|
||||
//
|
||||
// Yields one batch per major release series.
|
||||
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]Release, error] {
|
||||
return func(yield func([]Release, error) bool) {
|
||||
// Step 1: list major release series.
|
||||
majors, err := fetchMajors(ctx, client)
|
||||
if err != nil {
|
||||
yield(nil, err)
|
||||
return
|
||||
}
|
||||
|
||||
// Step 2: fetch point releases for each series.
|
||||
for _, major := range majors {
|
||||
if !reVersion.MatchString(major.ReleaseID) {
|
||||
continue
|
||||
}
|
||||
|
||||
releases, err := fetchReleases(ctx, client, major.ReleaseID)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("mariadbdist: %s: %w", major.ReleaseID, err))
|
||||
return
|
||||
}
|
||||
|
||||
// Tag each release with the major status.
|
||||
for i := range releases {
|
||||
releases[i].MajorStatus = major.ReleaseStatus
|
||||
}
|
||||
|
||||
if !yield(releases, nil) {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func fetchMajors(ctx context.Context, client *http.Client) ([]MajorRelease, error) {
|
||||
url := "https://downloads.mariadb.org/rest-api/mariadb/"
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("mariadbdist: %w", err)
|
||||
}
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("mariadbdist: fetch majors: %w", err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, fmt.Errorf("mariadbdist: fetch majors: %s", resp.Status)
|
||||
}
|
||||
|
||||
var result majorResp
|
||||
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
|
||||
return nil, fmt.Errorf("mariadbdist: decode majors: %w", err)
|
||||
}
|
||||
|
||||
return result.MajorReleases, nil
|
||||
}
|
||||
|
||||
func fetchReleases(ctx context.Context, client *http.Client, majorID string) ([]Release, error) {
|
||||
url := fmt.Sprintf("https://downloads.mariadb.org/rest-api/mariadb/%s", majorID)
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("mariadbdist: %w", err)
|
||||
}
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("mariadbdist: fetch %s: %w", majorID, err)
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return nil, fmt.Errorf("mariadbdist: fetch %s: %s", majorID, resp.Status)
|
||||
}
|
||||
|
||||
var result releaseResp
|
||||
if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
|
||||
return nil, fmt.Errorf("mariadbdist: decode %s: %w", majorID, err)
|
||||
}
|
||||
|
||||
var releases []Release
|
||||
for _, r := range result.Releases {
|
||||
releases = append(releases, r)
|
||||
}
|
||||
return releases, nil
|
||||
}
|
||||
39
internal/releases/node/node.go
Normal file
39
internal/releases/node/node.go
Normal file
@@ -0,0 +1,39 @@
|
||||
// Package node fetches Node.js releases from both official and unofficial
|
||||
// build sources.
|
||||
//
|
||||
// Official builds cover the standard platforms (linux-x64, osx-arm64, win-x64,
|
||||
// etc.). Unofficial builds add musl, loong64, and other targets that the
|
||||
// official CI doesn't produce.
|
||||
//
|
||||
// Both sources use the same index format, served by [nodedist].
|
||||
package nodedist
|
||||
|
||||
import (
|
||||
"context"
|
||||
"iter"
|
||||
"net/http"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/releases/nodedist"
|
||||
)
|
||||
|
||||
const (
|
||||
officialURL = "https://nodejs.org/download/release"
|
||||
unofficialURL = "https://unofficial-builds.nodejs.org/download/release"
|
||||
)
|
||||
|
||||
// Fetch retrieves Node.js releases from both official and unofficial sources.
|
||||
// Yields one batch per source (official first, then unofficial).
|
||||
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]nodedist.Entry, error] {
|
||||
return func(yield func([]nodedist.Entry, error) bool) {
|
||||
for entries, err := range nodedist.Fetch(ctx, client, officialURL) {
|
||||
if !yield(entries, err) {
|
||||
return
|
||||
}
|
||||
}
|
||||
for entries, err := range nodedist.Fetch(ctx, client, unofficialURL) {
|
||||
if !yield(entries, err) {
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
36
internal/releases/node/node_test.go
Normal file
36
internal/releases/node/node_test.go
Normal file
@@ -0,0 +1,36 @@
|
||||
package nodedist_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/releases/node"
|
||||
)
|
||||
|
||||
func TestFetchCombinesSources(t *testing.T) {
|
||||
if testing.Short() {
|
||||
t.Skip("skipping network test in short mode")
|
||||
}
|
||||
|
||||
ctx := context.Background()
|
||||
client := &http.Client{}
|
||||
|
||||
var batches int
|
||||
var total int
|
||||
for entries, err := range nodedist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
t.Fatalf("batch %d: %v", batches, err)
|
||||
}
|
||||
batches++
|
||||
total += len(entries)
|
||||
}
|
||||
|
||||
if batches != 2 {
|
||||
t.Errorf("got %d batches, want 2 (official + unofficial)", batches)
|
||||
}
|
||||
if total < 100 {
|
||||
t.Errorf("got %d total entries, expected at least 100", total)
|
||||
}
|
||||
t.Logf("fetched %d entries in %d batches", total, batches)
|
||||
}
|
||||
20
internal/releases/node/variants.go
Normal file
20
internal/releases/node/variants.go
Normal file
@@ -0,0 +1,20 @@
|
||||
package nodedist
|
||||
|
||||
import "github.com/webinstall/webi-installers/internal/storage"
|
||||
|
||||
// TagVariants tags node-specific build variants.
|
||||
//
|
||||
// The bare .exe is just node.exe without npm — too minimal to be useful.
|
||||
// The .msi is a Windows GUI installer — webi uses the .zip instead.
|
||||
// The .pkg is a macOS installer package — webi uses the .tar.gz instead.
|
||||
// Both are tagged as "installer" so ExportLegacy drops them.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
switch assets[i].Format {
|
||||
case ".exe":
|
||||
assets[i].Variants = append(assets[i].Variants, "bare-exe")
|
||||
case ".msi", ".pkg":
|
||||
assets[i].Variants = append(assets[i].Variants, "installer")
|
||||
}
|
||||
}
|
||||
}
|
||||
108
internal/releases/nodedist/nodedist.go
Normal file
108
internal/releases/nodedist/nodedist.go
Normal file
@@ -0,0 +1,108 @@
|
||||
// Package nodedist fetches a Node.js-style distribution index.
|
||||
//
|
||||
// Node.js publishes a JSON index of all releases at:
|
||||
//
|
||||
// https://nodejs.org/download/release/index.json
|
||||
//
|
||||
// Unofficial builds (musl, etc.) use the same format at:
|
||||
//
|
||||
// https://unofficial-builds.nodejs.org/download/release/index.json
|
||||
//
|
||||
// This package fetches and deserializes that index. It does not classify,
|
||||
// normalize, or transform the data — the caller gets what the API returns.
|
||||
package nodedist
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"iter"
|
||||
"net/http"
|
||||
)
|
||||
|
||||
// Entry is one release from a Node.js distribution index.
|
||||
// Fields mirror the upstream JSON schema.
|
||||
type Entry struct {
|
||||
Version string `json:"version"` // "v25.8.0"
|
||||
Date string `json:"date"` // "2026-03-03"
|
||||
Files []string `json:"files"` // ["linux-arm64", "osx-arm64-tar", ...]
|
||||
NPM string `json:"npm"` // "11.11.0"
|
||||
V8 string `json:"v8"` // "14.1.146.11"
|
||||
UV string `json:"uv"` // "1.51.0"
|
||||
Zlib string `json:"zlib"` // "1.3.1"
|
||||
OpenSSL string `json:"openssl"` // "3.5.5"
|
||||
Modules string `json:"modules"` // "141"
|
||||
LTS LTS `json:"lts"` // false or "Jod"
|
||||
Security bool `json:"security"` // true if security release
|
||||
}
|
||||
|
||||
// LTS holds the long-term support status. The upstream API encodes this as
|
||||
// either the boolean false or a codename string like "Jod" or "Iron".
|
||||
// An empty string means the release is not LTS.
|
||||
type LTS string
|
||||
|
||||
func (l *LTS) UnmarshalJSON(data []byte) error {
|
||||
// false → ""
|
||||
if string(data) == "false" {
|
||||
*l = ""
|
||||
return nil
|
||||
}
|
||||
|
||||
// "Codename" → Codename
|
||||
var s string
|
||||
if err := json.Unmarshal(data, &s); err != nil {
|
||||
return fmt.Errorf("nodedist: unexpected lts value: %s", data)
|
||||
}
|
||||
*l = LTS(s)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (l LTS) MarshalJSON() ([]byte, error) {
|
||||
if l == "" {
|
||||
return []byte("false"), nil
|
||||
}
|
||||
return json.Marshal(string(l))
|
||||
}
|
||||
|
||||
// Fetch retrieves the Node.js distribution index from baseURL.
|
||||
//
|
||||
// The iterator yields one batch per HTTP response. The Node.js index API
|
||||
// returns all releases in a single response, so there will be exactly one
|
||||
// yield. The iterator interface exists so that callers use the same pattern
|
||||
// for paginated sources (like GitHub).
|
||||
//
|
||||
// Standard base URLs:
|
||||
// - https://nodejs.org/download/release
|
||||
// - https://unofficial-builds.nodejs.org/download/release
|
||||
func Fetch(ctx context.Context, client *http.Client, baseURL string) iter.Seq2[[]Entry, error] {
|
||||
return func(yield func([]Entry, error) bool) {
|
||||
url := baseURL + "/index.json"
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("nodedist: %w", err))
|
||||
return
|
||||
}
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("nodedist: fetch %s: %w", url, err))
|
||||
return
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
yield(nil, fmt.Errorf("nodedist: fetch %s: %s", url, resp.Status))
|
||||
return
|
||||
}
|
||||
|
||||
var entries []Entry
|
||||
if err := json.NewDecoder(resp.Body).Decode(&entries); err != nil {
|
||||
yield(nil, fmt.Errorf("nodedist: decode %s: %w", url, err))
|
||||
return
|
||||
}
|
||||
|
||||
yield(entries, nil)
|
||||
}
|
||||
}
|
||||
143
internal/releases/nodedist/nodedist_test.go
Normal file
143
internal/releases/nodedist/nodedist_test.go
Normal file
@@ -0,0 +1,143 @@
|
||||
package nodedist_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/releases/nodedist"
|
||||
)
|
||||
|
||||
// Minimal fixture from the real Node.js dist API.
|
||||
const testIndex = `[
|
||||
{
|
||||
"version": "v22.14.0",
|
||||
"date": "2025-02-11",
|
||||
"files": ["linux-arm64", "linux-x64", "osx-arm64-tar", "win-x64-zip", "src", "headers"],
|
||||
"npm": "10.9.2",
|
||||
"v8": "12.4.254.21",
|
||||
"uv": "1.49.2",
|
||||
"zlib": "1.3.0.1-motley-82a6be0",
|
||||
"openssl": "3.0.15+quic",
|
||||
"modules": "127",
|
||||
"lts": "Jod",
|
||||
"security": false
|
||||
},
|
||||
{
|
||||
"version": "v23.7.0",
|
||||
"date": "2025-02-04",
|
||||
"files": ["linux-arm64", "linux-x64", "osx-arm64-tar", "win-x64-zip"],
|
||||
"npm": "10.9.2",
|
||||
"v8": "13.2.152.16",
|
||||
"uv": "1.49.2",
|
||||
"zlib": "1.3.0.1-motley-82a6be0",
|
||||
"openssl": "3.0.15+quic",
|
||||
"modules": "131",
|
||||
"lts": false,
|
||||
"security": true
|
||||
}
|
||||
]`
|
||||
|
||||
func TestFetch(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path != "/index.json" {
|
||||
t.Errorf("unexpected path: %s", r.URL.Path)
|
||||
http.NotFound(w, r)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.Write([]byte(testIndex))
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
ctx := context.Background()
|
||||
var got []nodedist.Entry
|
||||
|
||||
for entries, err := range nodedist.Fetch(ctx, srv.Client(), srv.URL) {
|
||||
if err != nil {
|
||||
t.Fatalf("Fetch: %v", err)
|
||||
}
|
||||
got = append(got, entries...)
|
||||
}
|
||||
|
||||
if len(got) != 2 {
|
||||
t.Fatalf("got %d entries, want 2", len(got))
|
||||
}
|
||||
|
||||
// First entry: LTS release
|
||||
if got[0].Version != "v22.14.0" {
|
||||
t.Errorf("entry[0].Version = %q, want %q", got[0].Version, "v22.14.0")
|
||||
}
|
||||
if got[0].Date != "2025-02-11" {
|
||||
t.Errorf("entry[0].Date = %q, want %q", got[0].Date, "2025-02-11")
|
||||
}
|
||||
if got[0].LTS != "Jod" {
|
||||
t.Errorf("entry[0].LTS = %q, want %q", got[0].LTS, "Jod")
|
||||
}
|
||||
if got[0].Security {
|
||||
t.Error("entry[0].Security = true, want false")
|
||||
}
|
||||
if len(got[0].Files) != 6 {
|
||||
t.Errorf("entry[0].Files len = %d, want 6", len(got[0].Files))
|
||||
}
|
||||
|
||||
// Second entry: non-LTS, security release
|
||||
if got[1].Version != "v23.7.0" {
|
||||
t.Errorf("entry[1].Version = %q, want %q", got[1].Version, "v23.7.0")
|
||||
}
|
||||
if got[1].LTS != "" {
|
||||
t.Errorf("entry[1].LTS = %q, want empty (non-LTS)", got[1].LTS)
|
||||
}
|
||||
if !got[1].Security {
|
||||
t.Error("entry[1].Security = false, want true")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFetchHTTPError(t *testing.T) {
|
||||
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
http.Error(w, "rate limited", http.StatusTooManyRequests)
|
||||
}))
|
||||
defer srv.Close()
|
||||
|
||||
ctx := context.Background()
|
||||
for _, err := range nodedist.Fetch(ctx, srv.Client(), srv.URL) {
|
||||
if err == nil {
|
||||
t.Fatal("expected error for 429 response")
|
||||
}
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
func TestLTSMarshalRoundTrip(t *testing.T) {
|
||||
// LTS codename
|
||||
entry := nodedist.Entry{LTS: "Jod"}
|
||||
data, err := json.Marshal(entry)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var got nodedist.Entry
|
||||
if err := json.Unmarshal(data, &got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.LTS != "Jod" {
|
||||
t.Errorf("LTS roundtrip: got %q, want %q", got.LTS, "Jod")
|
||||
}
|
||||
|
||||
// Non-LTS
|
||||
entry2 := nodedist.Entry{LTS: ""}
|
||||
data2, err := json.Marshal(entry2)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var got2 nodedist.Entry
|
||||
if err := json.Unmarshal(data2, &got2); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got2.LTS != "" {
|
||||
t.Errorf("non-LTS roundtrip: got %q, want empty", got2.LTS)
|
||||
}
|
||||
}
|
||||
32
internal/releases/ollama/variants.go
Normal file
32
internal/releases/ollama/variants.go
Normal file
@@ -0,0 +1,32 @@
|
||||
// Package ollama provides variant tagging for Ollama releases.
|
||||
//
|
||||
// Ollama publishes GPU accelerator builds: -rocm (AMD), -jetpack5
|
||||
// and -jetpack6 (NVIDIA Jetson).
|
||||
package ollamadist
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// TagVariants tags ollama-specific build variants.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
lower := strings.ToLower(assets[i].Filename)
|
||||
for _, v := range []string{"rocm", "jetpack5", "jetpack6"} {
|
||||
if strings.Contains(lower, "-"+v) {
|
||||
assets[i].Variants = append(assets[i].Variants, v)
|
||||
}
|
||||
}
|
||||
// Ollama-darwin.zip (capital O) is the macOS .app bundle.
|
||||
// Installable by Go (extract .app), but not in legacy cache.
|
||||
if strings.HasPrefix(assets[i].Filename, "Ollama-") {
|
||||
assets[i].Variants = append(assets[i].Variants, "app")
|
||||
}
|
||||
// ollama-darwin is a universal2 fat binary (arm64 + amd64).
|
||||
if assets[i].OS == "darwin" && assets[i].Arch == "" {
|
||||
assets[i].Arch = "universal2"
|
||||
}
|
||||
}
|
||||
}
|
||||
80
internal/releases/postgres/versions.go
Normal file
80
internal/releases/postgres/versions.go
Normal file
@@ -0,0 +1,80 @@
|
||||
package postgres
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// NormalizeVersions strips the REL_ prefix and converts underscores to dots.
|
||||
// GitHub tags are "REL_17_0" → version becomes "17.0".
|
||||
func NormalizeVersions(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
v := strings.TrimPrefix(assets[i].Version, "REL_")
|
||||
assets[i].Version = strings.ReplaceAll(v, "_", ".")
|
||||
}
|
||||
}
|
||||
|
||||
// LegacyReleases returns the old EnterpriseDB binary releases that predate
|
||||
// the bnnanet/postgresql-releases GitHub repo.
|
||||
func LegacyReleases() []storage.Asset {
|
||||
edbURL := "https://get.enterprisedb.com/postgresql/"
|
||||
return []storage.Asset{
|
||||
{
|
||||
Filename: "postgresql-10.12-1-linux-x64-binaries.tar.gz",
|
||||
Version: "10.12",
|
||||
Channel: "stable",
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Libc: "gnu",
|
||||
Format: ".tar.gz",
|
||||
Download: edbURL + "postgresql-10.12-1-linux-x64-binaries.tar.gz?ls=Crossover&type=Crossover",
|
||||
},
|
||||
{
|
||||
Filename: "postgresql-10.12-1-linux-binaries.tar.gz",
|
||||
Version: "10.12",
|
||||
Channel: "stable",
|
||||
OS: "linux",
|
||||
Arch: "x86",
|
||||
Libc: "gnu",
|
||||
Format: ".tar.gz",
|
||||
Download: edbURL + "postgresql-10.12-1-linux-binaries.tar.gz?ls=Crossover&type=Crossover",
|
||||
},
|
||||
{
|
||||
Filename: "postgresql-10.12-1-osx-binaries.zip",
|
||||
Version: "10.12",
|
||||
Channel: "stable",
|
||||
OS: "darwin",
|
||||
Arch: "x86_64",
|
||||
Format: ".zip",
|
||||
Download: edbURL + "postgresql-10.12-1-osx-binaries.zip?ls=Crossover&type=Crossover",
|
||||
},
|
||||
{
|
||||
Filename: "postgresql-10.13-1-osx-binaries.zip",
|
||||
Version: "10.13",
|
||||
Channel: "stable",
|
||||
OS: "darwin",
|
||||
Arch: "x86_64",
|
||||
Format: ".zip",
|
||||
Download: edbURL + "postgresql-10.13-1-osx-binaries.zip?ls=Crossover&type=Crossover",
|
||||
},
|
||||
{
|
||||
Filename: "postgresql-11.8-1-osx-binaries.zip",
|
||||
Version: "11.8",
|
||||
Channel: "stable",
|
||||
OS: "darwin",
|
||||
Arch: "x86_64",
|
||||
Format: ".zip",
|
||||
Download: edbURL + "postgresql-11.8-1-osx-binaries.zip?ls=Crossover&type=Crossover",
|
||||
},
|
||||
{
|
||||
Filename: "postgresql-12.3-1-osx-binaries.zip",
|
||||
Version: "12.3",
|
||||
Channel: "stable",
|
||||
OS: "darwin",
|
||||
Arch: "x86_64",
|
||||
Format: ".zip",
|
||||
Download: edbURL + "postgresql-12.3-1-osx-binaries.zip?ls=Crossover&type=Crossover",
|
||||
},
|
||||
}
|
||||
}
|
||||
37
internal/releases/pwsh/variants.go
Normal file
37
internal/releases/pwsh/variants.go
Normal file
@@ -0,0 +1,37 @@
|
||||
// Package pwsh provides variant tagging for PowerShell releases.
|
||||
//
|
||||
// PowerShell publishes .NET framework-dependent builds (-fxdependent)
|
||||
// that are smaller but require a .NET runtime to be installed.
|
||||
package pwshdist
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// winVersionRe matches Windows-version-specific filenames like
|
||||
// "win10-win2016-x64" or "win81-x64" from early PowerShell releases.
|
||||
var winVersionRe = regexp.MustCompile(`(?i)-win(?:7|8|81|10|2008|2012|2016)`)
|
||||
|
||||
// TagVariants tags pwsh-specific build variants.
|
||||
//
|
||||
// Early releases (pre-6.1) used Windows-version-specific filenames
|
||||
// like "win10-win2016-x64" and "win81-win2012r2-x64". These can't
|
||||
// be resolved by the legacy cache and are tagged as variants.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
lower := strings.ToLower(assets[i].Filename)
|
||||
switch {
|
||||
case strings.Contains(lower, "-fxdependentwindesktop"):
|
||||
assets[i].Variants = append(assets[i].Variants, "fxdependentWinDesktop")
|
||||
case strings.Contains(lower, "-fxdependent"):
|
||||
assets[i].Variants = append(assets[i].Variants, "fxdependent")
|
||||
case winVersionRe.MatchString(lower):
|
||||
assets[i].Variants = append(assets[i].Variants, "win-version-specific")
|
||||
case strings.HasSuffix(lower, ".appimage"):
|
||||
assets[i].Variants = append(assets[i].Variants, "appimage")
|
||||
}
|
||||
}
|
||||
}
|
||||
19
internal/releases/sass/variants.go
Normal file
19
internal/releases/sass/variants.go
Normal file
@@ -0,0 +1,19 @@
|
||||
// Package sass provides variant tagging for Dart Sass releases.
|
||||
//
|
||||
// Dart Sass uses bare "arm" in filenames to mean ARMv7 (the Dart VM's
|
||||
// minimum ARM target). The generic classifier maps bare "arm" to armv6,
|
||||
// so we correct it here.
|
||||
package sassdist
|
||||
|
||||
import (
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// TagVariants remaps bare arm → armv7 for Dart Sass assets.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
if assets[i].Arch == "armv6" {
|
||||
assets[i].Arch = "armv7"
|
||||
}
|
||||
}
|
||||
}
|
||||
75
internal/releases/servicemandist/servicemandist.go
Normal file
75
internal/releases/servicemandist/servicemandist.go
Normal file
@@ -0,0 +1,75 @@
|
||||
// Package servicemandist fetches serviceman releases from two GitHub repos.
|
||||
//
|
||||
// serviceman moved from therootcompany/serviceman (binary cross-platform
|
||||
// releases, ≤v0.8.x) to bnnanet/serviceman (source-only POSIX, v0.9.x+).
|
||||
// Both repos must be fetched to provide the complete version history,
|
||||
// including the only Windows binary at v0.8.0.
|
||||
package servicemandist
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"net/http"
|
||||
"path/filepath"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/rawcache"
|
||||
"github.com/webinstall/webi-installers/internal/releases/github"
|
||||
"github.com/webinstall/webi-installers/internal/releases/githubish"
|
||||
)
|
||||
|
||||
const (
|
||||
primaryOwner = "bnnanet"
|
||||
primaryRepo = "serviceman"
|
||||
|
||||
legacyOwner = "therootcompany"
|
||||
legacyRepo = "serviceman"
|
||||
)
|
||||
|
||||
// Fetch retrieves serviceman releases from both GitHub repos and merges
|
||||
// them into the raw cache. The primary repo (bnnanet) contains v0.9.x+;
|
||||
// the legacy repo (therootcompany) contains ≤v0.8.x with Windows binaries.
|
||||
func Fetch(ctx context.Context, client *http.Client, rawDir, pkgName string, auth *githubish.Auth, shallow bool) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Primary: bnnanet/serviceman (v0.9.x+ source tarballs).
|
||||
for batch, err := range github.Fetch(ctx, client, primaryOwner, primaryRepo, auth) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("servicemandist: %s/%s: %w", primaryOwner, primaryRepo, err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
if rel.Draft {
|
||||
continue
|
||||
}
|
||||
data, _ := json.Marshal(rel)
|
||||
d.Merge(primaryOwner+"/"+rel.TagName, data)
|
||||
}
|
||||
if shallow {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
// Legacy: therootcompany/serviceman (≤v0.8.x binaries).
|
||||
for batch, err := range github.Fetch(ctx, client, legacyOwner, legacyRepo, auth) {
|
||||
if err != nil {
|
||||
log.Printf("warning: servicemandist: %s/%s: %v", legacyOwner, legacyRepo, err)
|
||||
break
|
||||
}
|
||||
for _, rel := range batch {
|
||||
if rel.Draft {
|
||||
continue
|
||||
}
|
||||
data, _ := json.Marshal(rel)
|
||||
d.Merge(legacyOwner+"/"+rel.TagName, data)
|
||||
}
|
||||
if shallow {
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
16
internal/releases/servicemandist/variants.go
Normal file
16
internal/releases/servicemandist/variants.go
Normal file
@@ -0,0 +1,16 @@
|
||||
package servicemandist
|
||||
|
||||
import (
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// TagVariants marks all git-format entries as POSIX-only.
|
||||
// serviceman's git clone installs a POSIX shell script — no Windows support.
|
||||
// Binary releases (v0.8.x tar.gz/zip) already have per-platform OS set.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
if assets[i].Format == "git" && assets[i].OS == "" {
|
||||
assets[i].OS = "posix_2017"
|
||||
}
|
||||
}
|
||||
}
|
||||
36
internal/releases/sttr/variants.go
Normal file
36
internal/releases/sttr/variants.go
Normal file
@@ -0,0 +1,36 @@
|
||||
// Package sttr provides variant tagging for sttr releases.
|
||||
//
|
||||
// sttr ships a darwin_all (universal macOS) archive alongside per-arch builds.
|
||||
// These universal archives have no arch in the filename — Go classifies them as
|
||||
// os="darwin", arch="" which the Node builds-cacher rejects with FORMAT CHANGE
|
||||
// (Node's classifier extracts a different arch from "all"). Production Node
|
||||
// also stores these as os="", arch="" (unroutable).
|
||||
//
|
||||
// .sbom.json files are software bill-of-materials metadata — not installable
|
||||
// archives. They pass through the format filter (ext="") but should not be
|
||||
// served.
|
||||
package sttrdist
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// TagVariants tags sttr-specific build variants for exclusion from legacy export.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
lower := strings.ToLower(assets[i].Filename)
|
||||
// darwin_all / Darwin_all: universal macOS archive with no arch info.
|
||||
// Node's classifier extracts a different result → FORMAT CHANGE.
|
||||
// Production LIVE_cache has these as os="", arch="" (unroutable).
|
||||
if strings.Contains(lower, "darwin_all") {
|
||||
assets[i].Variants = append(assets[i].Variants, "universal-all")
|
||||
continue
|
||||
}
|
||||
// .sbom.json: software bill-of-materials, not an installable archive.
|
||||
if strings.HasSuffix(lower, ".sbom.json") {
|
||||
assets[i].Variants = append(assets[i].Variants, "metadata")
|
||||
}
|
||||
}
|
||||
}
|
||||
18
internal/releases/uuidv7/variants.go
Normal file
18
internal/releases/uuidv7/variants.go
Normal file
@@ -0,0 +1,18 @@
|
||||
// Package uuidv7 provides variant tagging for uuidv7 releases.
|
||||
package uuidv7dist
|
||||
|
||||
import "github.com/webinstall/webi-installers/internal/storage"
|
||||
|
||||
// TagVariants tags uuidv7-specific build variants for exclusion from legacy export.
|
||||
//
|
||||
// uuidv7 ships powerpc (32-bit) and powerpc64 binaries alongside the common
|
||||
// platforms. Webi does not serve powerpc targets, and production Node also
|
||||
// classifies these as os="", arch="" (not routable). Tag them unsupported.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
switch assets[i].Arch {
|
||||
case "powerpc", "ppc64", "ppc64le":
|
||||
assets[i].Variants = append(assets[i].Variants, "unsupported-platform")
|
||||
}
|
||||
}
|
||||
}
|
||||
18
internal/releases/watchexec/variants.go
Normal file
18
internal/releases/watchexec/variants.go
Normal file
@@ -0,0 +1,18 @@
|
||||
// Package watchexec provides variant tagging and version normalization for watchexec.
|
||||
package watchexecdist
|
||||
|
||||
import "github.com/webinstall/webi-installers/internal/storage"
|
||||
|
||||
// TagVariants tags watchexec-specific build variants for exclusion from legacy export.
|
||||
//
|
||||
// Watchexec ships powerpc64le binaries alongside the common platforms.
|
||||
// Webi does not serve powerpc targets, and production Node also classifies
|
||||
// these as os="", arch="" (not routable). Tag them unsupported.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
switch assets[i].Arch {
|
||||
case "powerpc", "ppc64", "ppc64le":
|
||||
assets[i].Variants = append(assets[i].Variants, "unsupported-platform")
|
||||
}
|
||||
}
|
||||
}
|
||||
18
internal/releases/watchexec/versions.go
Normal file
18
internal/releases/watchexec/versions.go
Normal file
@@ -0,0 +1,18 @@
|
||||
package watchexecdist
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// NormalizeVersions strips the "cli-" prefix from watchexec version strings.
|
||||
//
|
||||
// Watchexec transitioned to a monorepo with cli-prefixed tags (cli-v1.20.0)
|
||||
// while older releases used plain tags (v1.20.6). Both are valid releases;
|
||||
// the prefix is just a tag namespace, not part of the version.
|
||||
func NormalizeVersions(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
assets[i].Version = strings.TrimPrefix(assets[i].Version, "cli-")
|
||||
}
|
||||
}
|
||||
15
internal/releases/xcaddy/variants.go
Normal file
15
internal/releases/xcaddy/variants.go
Normal file
@@ -0,0 +1,15 @@
|
||||
// Package xcaddy provides variant tagging for xcaddy releases.
|
||||
//
|
||||
// xcaddy publishes .deb packages alongside the standard archives.
|
||||
package xcaddydist
|
||||
|
||||
import "github.com/webinstall/webi-installers/internal/storage"
|
||||
|
||||
// TagVariants tags xcaddy-specific build variants.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
if assets[i].Format == ".deb" {
|
||||
assets[i].Variants = append(assets[i].Variants, "deb")
|
||||
}
|
||||
}
|
||||
}
|
||||
16
internal/releases/xz/variants.go
Normal file
16
internal/releases/xz/variants.go
Normal file
@@ -0,0 +1,16 @@
|
||||
package xzdist
|
||||
|
||||
import "github.com/webinstall/webi-installers/internal/storage"
|
||||
|
||||
// TagVariants handles xz-specific arch defaults.
|
||||
//
|
||||
// therootcompany/xz-static names builds xz-{version}-{os}-{arch} for
|
||||
// Linux/macOS but xz-{version}-windows.zip for Windows (only amd64
|
||||
// shipped). The arch token is absent only for the Windows build.
|
||||
func TagVariants(assets []storage.Asset) {
|
||||
for i := range assets {
|
||||
if assets[i].Arch == "" && assets[i].OS == "windows" {
|
||||
assets[i].Arch = "x86_64"
|
||||
}
|
||||
}
|
||||
}
|
||||
131
internal/releases/zigdist/zigdist.go
Normal file
131
internal/releases/zigdist/zigdist.go
Normal file
@@ -0,0 +1,131 @@
|
||||
// Package zigdist fetches Zig release data from ziglang.org.
|
||||
//
|
||||
// The API is a single JSON object keyed by version or branch name:
|
||||
//
|
||||
// https://ziglang.org/download/index.json
|
||||
//
|
||||
// Each version key maps to an object containing "date", "notes", and
|
||||
// platform keys like "x86_64-linux", "aarch64-macos", etc. Platform
|
||||
// values have "tarball", "shasum", and "size" fields.
|
||||
package zigdist
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"iter"
|
||||
"net/http"
|
||||
)
|
||||
|
||||
// Release is one Zig version with its per-platform builds.
|
||||
type Release struct {
|
||||
Version string `json:"version"` // set by us from the key or inner "version" field
|
||||
Date string `json:"date"`
|
||||
Notes string `json:"notes,omitempty"`
|
||||
Platforms map[string]Platform `json:"platforms,omitempty"` // "x86_64-linux" → Platform
|
||||
}
|
||||
|
||||
// Platform is one downloadable artifact for a specific arch-os combo.
|
||||
type Platform struct {
|
||||
Tarball string `json:"tarball"`
|
||||
Shasum string `json:"shasum"`
|
||||
Size json.Number `json:"size"` // upstream sends as string
|
||||
}
|
||||
|
||||
// Fetch retrieves the Zig release index.
|
||||
//
|
||||
// Yields one batch containing all releases. The iterator interface exists
|
||||
// so callers use the same pattern as paginated sources.
|
||||
func Fetch(ctx context.Context, client *http.Client) iter.Seq2[[]Release, error] {
|
||||
return func(yield func([]Release, error) bool) {
|
||||
url := "https://ziglang.org/download/index.json"
|
||||
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("zigdist: %w", err))
|
||||
return
|
||||
}
|
||||
req.Header.Set("Accept", "application/json")
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("zigdist: fetch: %w", err))
|
||||
return
|
||||
}
|
||||
defer resp.Body.Close()
|
||||
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
yield(nil, fmt.Errorf("zigdist: fetch: %s", resp.Status))
|
||||
return
|
||||
}
|
||||
|
||||
// The JSON is an object keyed by version/branch name.
|
||||
var raw map[string]json.RawMessage
|
||||
if err := json.NewDecoder(resp.Body).Decode(&raw); err != nil {
|
||||
yield(nil, fmt.Errorf("zigdist: decode: %w", err))
|
||||
return
|
||||
}
|
||||
|
||||
var releases []Release
|
||||
for ref, data := range raw {
|
||||
rel, err := parseRelease(ref, data)
|
||||
if err != nil {
|
||||
yield(nil, fmt.Errorf("zigdist: parse %s: %w", ref, err))
|
||||
return
|
||||
}
|
||||
releases = append(releases, rel)
|
||||
}
|
||||
|
||||
yield(releases, nil)
|
||||
}
|
||||
}
|
||||
|
||||
// parseRelease extracts a Release from one version entry. The JSON mixes
|
||||
// metadata fields ("date", "notes", "version", "src") with platform keys
|
||||
// ("x86_64-linux", "aarch64-macos", etc.).
|
||||
func parseRelease(ref string, data json.RawMessage) (Release, error) {
|
||||
// First pass: grab known metadata fields.
|
||||
var meta struct {
|
||||
Version string `json:"version"`
|
||||
Date string `json:"date"`
|
||||
Notes string `json:"notes"`
|
||||
}
|
||||
if err := json.Unmarshal(data, &meta); err != nil {
|
||||
return Release{}, err
|
||||
}
|
||||
|
||||
version := meta.Version
|
||||
if version == "" {
|
||||
version = ref
|
||||
}
|
||||
|
||||
// Second pass: grab all platform entries.
|
||||
var all map[string]json.RawMessage
|
||||
if err := json.Unmarshal(data, &all); err != nil {
|
||||
return Release{}, err
|
||||
}
|
||||
|
||||
platforms := make(map[string]Platform)
|
||||
for key, val := range all {
|
||||
// Skip metadata keys.
|
||||
switch key {
|
||||
case "version", "date", "notes", "src":
|
||||
continue
|
||||
}
|
||||
var p Platform
|
||||
if err := json.Unmarshal(val, &p); err != nil {
|
||||
continue // not a platform object
|
||||
}
|
||||
if p.Tarball == "" {
|
||||
continue // not a platform object
|
||||
}
|
||||
platforms[key] = p
|
||||
}
|
||||
|
||||
return Release{
|
||||
Version: version,
|
||||
Date: meta.Date,
|
||||
Notes: meta.Notes,
|
||||
Platforms: platforms,
|
||||
}, nil
|
||||
}
|
||||
264
internal/render/render.go
Normal file
264
internal/render/render.go
Normal file
@@ -0,0 +1,264 @@
|
||||
// Package render generates installer scripts by injecting release
|
||||
// metadata into the package-install template.
|
||||
//
|
||||
// The template uses shell-style variable markers:
|
||||
//
|
||||
// #WEBI_VERSION= → WEBI_VERSION='1.2.3'
|
||||
// #export WEBI_PKG_URL= → export WEBI_PKG_URL='https://...'
|
||||
//
|
||||
// The package's install.sh is injected at the {{ installer }} marker.
|
||||
package render
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"regexp"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Params holds all the values to inject into the installer template.
|
||||
type Params struct {
|
||||
// Host is the base URL of the webi server (e.g. "https://webinstall.dev").
|
||||
Host string
|
||||
|
||||
// Checksum is the webi.sh bootstrap script checksum (first 8 hex chars of SHA-1).
|
||||
Checksum string
|
||||
|
||||
// Package name (e.g. "bat", "node").
|
||||
PkgName string
|
||||
|
||||
// Tag is the version selector from the URL (e.g. "20", "stable", "").
|
||||
Tag string
|
||||
|
||||
// OS, Arch, Libc are the detected platform strings.
|
||||
OS string
|
||||
Arch string
|
||||
Libc string
|
||||
|
||||
// Resolved release info.
|
||||
Version string
|
||||
Major string
|
||||
Minor string
|
||||
Patch string
|
||||
Build string
|
||||
GitTag string
|
||||
GitBranch string
|
||||
LTS string // "true" or "false"
|
||||
Channel string
|
||||
Ext string // archive extension (e.g. "tar.gz", "zip")
|
||||
Formats string // comma-separated format list
|
||||
|
||||
// Download info.
|
||||
PkgURL string // download URL
|
||||
PkgFile string // filename
|
||||
|
||||
// Releases API URL for this request.
|
||||
ReleasesURL string
|
||||
|
||||
// CSV line for WEBI_CSV.
|
||||
CSV string
|
||||
|
||||
// Package catalog info.
|
||||
PkgStable string
|
||||
PkgLatest string
|
||||
PkgOSes string // space-separated
|
||||
PkgArches string // space-separated
|
||||
PkgLibcs string // space-separated
|
||||
PkgFormats string // space-separated
|
||||
}
|
||||
|
||||
// Bash renders a complete bash installer script by injecting params
|
||||
// into the template and splicing in the package's install.sh.
|
||||
func Bash(tplPath, installersDir, pkgName string, p Params) (string, error) {
|
||||
tpl, err := os.ReadFile(tplPath)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("render: read template: %w", err)
|
||||
}
|
||||
|
||||
// Read the package's install.sh.
|
||||
installPath := filepath.Join(installersDir, pkgName, "install.sh")
|
||||
installSh, err := os.ReadFile(installPath)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("render: read %s/install.sh: %w", pkgName, err)
|
||||
}
|
||||
|
||||
text := string(tpl)
|
||||
|
||||
// Inject environment variables.
|
||||
vars := []struct {
|
||||
name string
|
||||
value string
|
||||
}{
|
||||
{"WEBI_CHECKSUM", p.Checksum},
|
||||
{"WEBI_PKG", p.PkgName + "@" + p.Tag},
|
||||
{"WEBI_HOST", p.Host},
|
||||
{"WEBI_OS", p.OS},
|
||||
{"WEBI_ARCH", p.Arch},
|
||||
{"WEBI_LIBC", p.Libc},
|
||||
{"WEBI_TAG", p.Tag},
|
||||
{"WEBI_RELEASES", p.ReleasesURL},
|
||||
{"WEBI_CSV", p.CSV},
|
||||
{"WEBI_VERSION", p.Version},
|
||||
{"WEBI_MAJOR", p.Major},
|
||||
{"WEBI_MINOR", p.Minor},
|
||||
{"WEBI_PATCH", p.Patch},
|
||||
{"WEBI_BUILD", p.Build},
|
||||
{"WEBI_GIT_BRANCH", p.GitBranch},
|
||||
{"WEBI_GIT_TAG", p.GitTag},
|
||||
{"WEBI_LTS", p.LTS},
|
||||
{"WEBI_CHANNEL", p.Channel},
|
||||
{"WEBI_EXT", p.Ext},
|
||||
{"WEBI_FORMATS", p.Formats},
|
||||
{"WEBI_PKG_URL", p.PkgURL},
|
||||
{"WEBI_PKG_PATHNAME", p.PkgFile},
|
||||
{"WEBI_PKG_FILE", p.PkgFile},
|
||||
{"PKG_NAME", p.PkgName},
|
||||
{"PKG_STABLE", p.PkgStable},
|
||||
{"PKG_LATEST", p.PkgLatest},
|
||||
{"PKG_OSES", p.PkgOSes},
|
||||
{"PKG_ARCHES", p.PkgArches},
|
||||
{"PKG_LIBCS", p.PkgLibcs},
|
||||
{"PKG_FORMATS", p.PkgFormats},
|
||||
}
|
||||
|
||||
for _, v := range vars {
|
||||
text = InjectVar(text, v.name, v.value)
|
||||
}
|
||||
|
||||
// Inject the installer script at the {{ installer }} marker.
|
||||
// The marker sits inside __init_installer() at 8-space indent.
|
||||
// Production pads every line of install.sh to match, and replaces
|
||||
// the entire line (including leading whitespace).
|
||||
padded := padScript(string(installSh), " ")
|
||||
text = replaceMarkerLine(text, "{{ installer }}", padded)
|
||||
|
||||
return text, nil
|
||||
}
|
||||
|
||||
// PowerShell renders a complete PowerShell installer script by injecting
|
||||
// params into the template and splicing in the package's install.ps1.
|
||||
func PowerShell(tplPath, installersDir, pkgName string, p Params) (string, error) {
|
||||
tpl, err := os.ReadFile(tplPath)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("render: read template: %w", err)
|
||||
}
|
||||
|
||||
installPath := filepath.Join(installersDir, pkgName, "install.ps1")
|
||||
installPs1, err := os.ReadFile(installPath)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("render: read %s/install.ps1: %w", pkgName, err)
|
||||
}
|
||||
|
||||
text := string(tpl)
|
||||
|
||||
vars := []struct {
|
||||
name string
|
||||
value string
|
||||
}{
|
||||
{"WEBI_PKG", p.PkgName + "@" + p.Tag},
|
||||
{"WEBI_HOST", p.Host},
|
||||
{"WEBI_VERSION", p.Version},
|
||||
{"WEBI_GIT_TAG", p.GitTag},
|
||||
{"WEBI_PKG_URL", p.PkgURL},
|
||||
{"WEBI_PKG_FILE", p.PkgFile},
|
||||
{"WEBI_PKG_PATHNAME", p.PkgFile},
|
||||
{"PKG_NAME", p.PkgName},
|
||||
}
|
||||
|
||||
for _, v := range vars {
|
||||
text = InjectPSVar(text, v.name, v.value)
|
||||
}
|
||||
|
||||
// PS1 marker is at column 0, no padding needed.
|
||||
text = replaceMarkerLine(text, "{{ installer }}", string(installPs1))
|
||||
|
||||
return text, nil
|
||||
}
|
||||
|
||||
// InjectPSVar replaces a PowerShell template variable line with its value.
|
||||
// Matches lines like:
|
||||
//
|
||||
// #$Env:WEBI_VERSION = v12.16.2
|
||||
// $Env:WEBI_HOST = 'https://webinstall.dev'
|
||||
func InjectPSVar(text, name, value string) string {
|
||||
p := getPSVarPattern(name)
|
||||
return p.ReplaceAllString(text, "${1}$$Env:"+name+" = '"+sanitizePSValue(value)+"'")
|
||||
}
|
||||
|
||||
var psVarPatterns = map[string]*regexp.Regexp{}
|
||||
|
||||
func getPSVarPattern(name string) *regexp.Regexp {
|
||||
if p, ok := psVarPatterns[name]; ok {
|
||||
return p
|
||||
}
|
||||
// Match: optional leading whitespace, optional #, $Env:NAME, =, rest of line
|
||||
p := regexp.MustCompile(`(?m)^([ \t]*)#?\$Env:` + regexp.QuoteMeta(name) + `\s*=.*$`)
|
||||
psVarPatterns[name] = p
|
||||
return p
|
||||
}
|
||||
|
||||
// sanitizePSValue escapes single quotes for PowerShell single-quoted strings.
|
||||
// In PowerShell, single quotes inside single-quoted strings are doubled: ''
|
||||
func sanitizePSValue(s string) string {
|
||||
return strings.ReplaceAll(s, "'", "''")
|
||||
}
|
||||
|
||||
// varPattern matches shell variable declarations in the template.
|
||||
// Matches lines like:
|
||||
//
|
||||
// #WEBI_VERSION=
|
||||
// #export WEBI_PKG_URL=
|
||||
// #WEBI_OS=
|
||||
var varPatterns = map[string]*regexp.Regexp{}
|
||||
|
||||
func getVarPattern(name string) *regexp.Regexp {
|
||||
if p, ok := varPatterns[name]; ok {
|
||||
return p
|
||||
}
|
||||
// Match: optional leading whitespace, optional #, optional export, the var name, =, rest of line
|
||||
p := regexp.MustCompile(`(?m)^([ \t]*)#?([ \t]*)(export[ \t]+)?[ \t]*(` + regexp.QuoteMeta(name) + `)=.*$`)
|
||||
varPatterns[name] = p
|
||||
return p
|
||||
}
|
||||
|
||||
// InjectVar replaces a template variable line with its value.
|
||||
// It matches lines like:
|
||||
//
|
||||
// #WEBI_VERSION=
|
||||
// #export WEBI_PKG_URL=
|
||||
// export WEBI_HOST=
|
||||
//
|
||||
// and replaces them with the value in single quotes.
|
||||
func InjectVar(text, name, value string) string {
|
||||
p := getVarPattern(name)
|
||||
return p.ReplaceAllString(text, "${1}${3}"+name+"='"+sanitizeShellValue(value)+"'")
|
||||
}
|
||||
|
||||
// sanitizeShellValue ensures a value is safe to embed in single quotes.
|
||||
// Single quotes in shell can't be escaped inside single quotes, so we
|
||||
// close-quote, add escaped quote, re-open quote: 'foo'\''bar'
|
||||
func sanitizeShellValue(s string) string {
|
||||
return strings.ReplaceAll(s, "'", `'\''`)
|
||||
}
|
||||
|
||||
// padScript prepends each line of a script with the given indent string.
|
||||
// This matches production behavior where install.sh content is indented
|
||||
// to align with the surrounding template code.
|
||||
func padScript(script, indent string) string {
|
||||
lines := strings.Split(script, "\n")
|
||||
for i, line := range lines {
|
||||
if line != "" {
|
||||
lines[i] = indent + line
|
||||
}
|
||||
}
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
|
||||
// replaceMarkerLine replaces an entire line containing the marker
|
||||
// (including any leading whitespace) with the replacement text.
|
||||
// This matches production's regex: /\s*#?\s*{{ installer }}/
|
||||
func replaceMarkerLine(text, marker, replacement string) string {
|
||||
re := regexp.MustCompile(`(?m)^[ \t]*#?[ \t]*` + regexp.QuoteMeta(marker) + `[^\n]*`)
|
||||
return re.ReplaceAllLiteralString(text, replacement)
|
||||
}
|
||||
90
internal/render/render_test.go
Normal file
90
internal/render/render_test.go
Normal file
@@ -0,0 +1,90 @@
|
||||
package render
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestInjectVar(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
input string
|
||||
key string
|
||||
value string
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "commented var",
|
||||
input: " #WEBI_VERSION=",
|
||||
key: "WEBI_VERSION",
|
||||
value: "1.2.3",
|
||||
want: " WEBI_VERSION='1.2.3'",
|
||||
},
|
||||
{
|
||||
name: "commented export var",
|
||||
input: " #export WEBI_PKG_URL=",
|
||||
key: "WEBI_PKG_URL",
|
||||
value: "https://example.com/foo.tar.gz",
|
||||
want: " export WEBI_PKG_URL='https://example.com/foo.tar.gz'",
|
||||
},
|
||||
{
|
||||
name: "existing value replaced",
|
||||
input: " export WEBI_HOST=",
|
||||
key: "WEBI_HOST",
|
||||
value: "https://webinstall.dev",
|
||||
want: " export WEBI_HOST='https://webinstall.dev'",
|
||||
},
|
||||
{
|
||||
name: "value with single quotes",
|
||||
input: " #PKG_NAME=",
|
||||
key: "PKG_NAME",
|
||||
value: "it's-a-test",
|
||||
want: " PKG_NAME='it'\\''s-a-test'",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
got := InjectVar(tt.input, tt.key, tt.value)
|
||||
if strings.TrimSpace(got) != strings.TrimSpace(tt.want) {
|
||||
t.Errorf("got %q\nwant %q", got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestInjectVarInTemplate(t *testing.T) {
|
||||
tpl := `#!/bin/sh
|
||||
__bootstrap_webi() {
|
||||
#PKG_NAME=
|
||||
#WEBI_OS=
|
||||
#WEBI_ARCH=
|
||||
#WEBI_VERSION=
|
||||
export WEBI_HOST=
|
||||
WEBI_PKG_DOWNLOAD=""
|
||||
`
|
||||
|
||||
result := tpl
|
||||
result = InjectVar(result, "PKG_NAME", "bat")
|
||||
result = InjectVar(result, "WEBI_OS", "linux")
|
||||
result = InjectVar(result, "WEBI_ARCH", "x86_64")
|
||||
result = InjectVar(result, "WEBI_VERSION", "0.26.1")
|
||||
result = InjectVar(result, "WEBI_HOST", "https://webinstall.dev")
|
||||
|
||||
if !strings.Contains(result, "PKG_NAME='bat'") {
|
||||
t.Error("PKG_NAME not injected")
|
||||
}
|
||||
if !strings.Contains(result, "WEBI_OS='linux'") {
|
||||
t.Error("WEBI_OS not injected")
|
||||
}
|
||||
if !strings.Contains(result, "WEBI_VERSION='0.26.1'") {
|
||||
t.Error("WEBI_VERSION not injected")
|
||||
}
|
||||
if !strings.Contains(result, "export WEBI_HOST='https://webinstall.dev'") {
|
||||
t.Error("WEBI_HOST not injected")
|
||||
}
|
||||
// Should not have #PKG_NAME= anymore.
|
||||
if strings.Contains(result, "#PKG_NAME=") {
|
||||
t.Error("#PKG_NAME= should have been replaced")
|
||||
}
|
||||
}
|
||||
303
internal/resolve/resolve.go
Normal file
303
internal/resolve/resolve.go
Normal file
@@ -0,0 +1,303 @@
|
||||
// Package resolve picks the best release for a given platform query.
|
||||
//
|
||||
// Given a set of classified distributables and a target query (OS, arch,
|
||||
// libc, format preferences, version constraint), it returns the single
|
||||
// best matching release — or nil if nothing matches.
|
||||
package resolve
|
||||
|
||||
import (
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/buildmeta"
|
||||
"github.com/webinstall/webi-installers/internal/lexver"
|
||||
)
|
||||
|
||||
// Dist is one downloadable distributable — matches the CSV row from classify.
|
||||
type Dist struct {
|
||||
Package string
|
||||
Version string
|
||||
Channel string
|
||||
OS string
|
||||
Arch string
|
||||
Libc string
|
||||
Format string
|
||||
Download string
|
||||
Filename string
|
||||
SHA256 string
|
||||
Size int64
|
||||
LTS bool
|
||||
Date string
|
||||
Extra string // extra version info for sorting
|
||||
GitTag string // original git tag or branch — only for format="git"
|
||||
GitCommitHash string // short commit hash — only for format="git"
|
||||
Variants []string // build qualifiers: "installer", "rocm", "fxdependent", etc.
|
||||
}
|
||||
|
||||
// Query describes what the caller wants.
|
||||
type Query struct {
|
||||
OS buildmeta.OS
|
||||
Arch buildmeta.Arch
|
||||
Libc buildmeta.Libc
|
||||
Formats []string // acceptable formats (e.g. ".tar.gz", ".zip"), in preference order
|
||||
Channel string // "stable" (default), "beta", etc.
|
||||
Version string // version prefix constraint ("24", "24.14", ""), empty = latest
|
||||
Variants []string // if non-empty, only match assets with these variants
|
||||
}
|
||||
|
||||
// Match is the resolved release.
|
||||
type Match struct {
|
||||
Version string
|
||||
OS string
|
||||
Arch string
|
||||
Libc string
|
||||
Format string
|
||||
Download string
|
||||
Filename string
|
||||
LTS bool
|
||||
Date string
|
||||
Channel string
|
||||
}
|
||||
|
||||
// Best finds the single best release matching the query.
|
||||
// Returns nil if nothing matches.
|
||||
func Best(dists []Dist, q Query) *Match {
|
||||
channel := q.Channel
|
||||
if channel == "" {
|
||||
channel = "stable"
|
||||
}
|
||||
|
||||
// Build format set for fast lookup + rank map for preference.
|
||||
formatRank := make(map[string]int, len(q.Formats))
|
||||
for i, f := range q.Formats {
|
||||
formatRank[f] = i
|
||||
}
|
||||
|
||||
// Build the set of acceptable architectures (native + compat).
|
||||
compatArches := buildmeta.CompatArches(q.OS, q.Arch)
|
||||
archRank := make(map[string]int, len(compatArches))
|
||||
for i, a := range compatArches {
|
||||
archRank[string(a)] = i
|
||||
}
|
||||
|
||||
// Parse version prefix for constraint matching.
|
||||
var versionPrefix lexver.Version
|
||||
hasVersionConstraint := q.Version != ""
|
||||
if hasVersionConstraint {
|
||||
versionPrefix = lexver.Parse(q.Version)
|
||||
}
|
||||
|
||||
var best *candidate
|
||||
for i := range dists {
|
||||
d := &dists[i]
|
||||
|
||||
// Channel filter.
|
||||
if channel == "stable" && d.Channel != "stable" && d.Channel != "" {
|
||||
continue
|
||||
}
|
||||
|
||||
// OS filter: exact match, POSIX fallback, or ANYOS.
|
||||
if !osMatches(q.OS, d.OS) {
|
||||
continue
|
||||
}
|
||||
|
||||
// Arch filter (including compat arches).
|
||||
// Empty arch, ANYARCH, or "*" means "universal/platform-agnostic" —
|
||||
// accept it but rank it lower than an exact match.
|
||||
aRank, archOK := archRank[d.Arch]
|
||||
if !archOK && (d.Arch == "" || d.Arch == "*" || d.Arch == string(buildmeta.ArchAny)) {
|
||||
// Universal binary — rank after all specific arches.
|
||||
aRank = len(compatArches)
|
||||
archOK = true
|
||||
}
|
||||
if !archOK {
|
||||
continue
|
||||
}
|
||||
|
||||
// Libc filter.
|
||||
if !libcMatches(q.OS, q.Libc, d.Libc) {
|
||||
continue
|
||||
}
|
||||
|
||||
// Format filter.
|
||||
// Empty format means bare binary — accept as last resort.
|
||||
fRank, formatOK := formatRank[d.Format]
|
||||
if !formatOK && d.Format == "" {
|
||||
// Bare binary — rank after all explicit formats.
|
||||
fRank = len(q.Formats)
|
||||
formatOK = true
|
||||
}
|
||||
if !formatOK && len(q.Formats) > 0 {
|
||||
continue
|
||||
}
|
||||
if !formatOK {
|
||||
fRank = 999
|
||||
}
|
||||
|
||||
// Version constraint.
|
||||
ver := lexver.Parse(d.Version)
|
||||
if hasVersionConstraint && !ver.HasPrefix(versionPrefix) {
|
||||
continue
|
||||
}
|
||||
|
||||
c := &candidate{
|
||||
dist: d,
|
||||
ver: ver,
|
||||
archRank: aRank,
|
||||
formatRank: fRank,
|
||||
hasVariants: len(d.Variants) > 0,
|
||||
}
|
||||
|
||||
if best == nil || c.betterThan(best) {
|
||||
best = c
|
||||
}
|
||||
}
|
||||
|
||||
if best == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
d := best.dist
|
||||
return &Match{
|
||||
Version: d.Version,
|
||||
OS: d.OS,
|
||||
Arch: d.Arch,
|
||||
Libc: d.Libc,
|
||||
Format: d.Format,
|
||||
Download: d.Download,
|
||||
Filename: d.Filename,
|
||||
LTS: d.LTS,
|
||||
Date: d.Date,
|
||||
Channel: d.Channel,
|
||||
}
|
||||
}
|
||||
|
||||
// Catalog computes aggregate metadata across all stable dists for a package.
|
||||
type Catalog struct {
|
||||
OSes []string
|
||||
Arches []string
|
||||
Libcs []string
|
||||
Formats []string
|
||||
Latest string // highest version of any channel
|
||||
Stable string // highest stable version
|
||||
}
|
||||
|
||||
// Survey scans all dists and returns the catalog.
|
||||
func Survey(dists []Dist) Catalog {
|
||||
oses := make(map[string]bool)
|
||||
arches := make(map[string]bool)
|
||||
libcs := make(map[string]bool)
|
||||
formats := make(map[string]bool)
|
||||
|
||||
var latest, stable string
|
||||
for _, d := range dists {
|
||||
if d.OS != "" {
|
||||
oses[d.OS] = true
|
||||
}
|
||||
if d.Arch != "" {
|
||||
arches[d.Arch] = true
|
||||
}
|
||||
if d.Libc != "" {
|
||||
libcs[d.Libc] = true
|
||||
}
|
||||
if d.Format != "" {
|
||||
formats[d.Format] = true
|
||||
}
|
||||
|
||||
v := lexver.Parse(d.Version)
|
||||
if latest == "" || lexver.Compare(v, lexver.Parse(latest)) > 0 {
|
||||
latest = d.Version
|
||||
}
|
||||
if d.Channel == "stable" || d.Channel == "" {
|
||||
if stable == "" || lexver.Compare(v, lexver.Parse(stable)) > 0 {
|
||||
stable = d.Version
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return Catalog{
|
||||
OSes: sortedKeys(oses),
|
||||
Arches: sortedKeys(arches),
|
||||
Libcs: sortedKeys(libcs),
|
||||
Formats: sortedKeys(formats),
|
||||
Latest: latest,
|
||||
Stable: stable,
|
||||
}
|
||||
}
|
||||
|
||||
type candidate struct {
|
||||
dist *Dist
|
||||
ver lexver.Version
|
||||
archRank int
|
||||
formatRank int
|
||||
hasVariants bool // true if dist has variant qualifiers (GPU, installer, etc.)
|
||||
}
|
||||
|
||||
// betterThan returns true if c is a better match than other.
|
||||
// Priority: version (higher) > base over variant > arch rank (lower=native) > format rank (lower=preferred).
|
||||
func (c *candidate) betterThan(other *candidate) bool {
|
||||
cmp := lexver.Compare(c.ver, other.ver)
|
||||
if cmp != 0 {
|
||||
return cmp > 0
|
||||
}
|
||||
// Prefer base build over variant builds (rocm, installer, etc.)
|
||||
if c.hasVariants != other.hasVariants {
|
||||
return !c.hasVariants
|
||||
}
|
||||
if c.archRank != other.archRank {
|
||||
return c.archRank < other.archRank
|
||||
}
|
||||
return c.formatRank < other.formatRank
|
||||
}
|
||||
|
||||
// osMatches checks whether a dist's OS is acceptable for the query.
|
||||
// Matches exact OS, ANYOS (universal), and POSIX compatibility levels
|
||||
// (posix_2017 matches any non-Windows OS).
|
||||
func osMatches(want buildmeta.OS, have string) bool {
|
||||
if have == string(want) {
|
||||
return true
|
||||
}
|
||||
if have == string(buildmeta.OSAny) {
|
||||
return true
|
||||
}
|
||||
// POSIX assets run on any non-Windows system.
|
||||
if want != buildmeta.OSWindows {
|
||||
if have == string(buildmeta.OSPosix2017) || have == string(buildmeta.OSPosix2024) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// libcMatches checks whether a dist's libc is acceptable for the query.
|
||||
func libcMatches(os buildmeta.OS, want buildmeta.Libc, have string) bool {
|
||||
// Darwin and Windows don't use libc tagging — accept anything.
|
||||
if os == buildmeta.OSDarwin || os == buildmeta.OSWindows {
|
||||
return true
|
||||
}
|
||||
|
||||
// If the dist has no libc tag, accept it (likely statically linked).
|
||||
if have == "" || have == "none" || have == string(buildmeta.LibcNone) {
|
||||
return true
|
||||
}
|
||||
|
||||
// If the query has no libc preference, accept any.
|
||||
if want == "" || want == buildmeta.LibcNone {
|
||||
return true
|
||||
}
|
||||
|
||||
return have == string(want)
|
||||
}
|
||||
|
||||
func sortedKeys(m map[string]bool) []string {
|
||||
keys := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
// Simple insertion sort — these are tiny sets.
|
||||
for i := 1; i < len(keys); i++ {
|
||||
for j := i; j > 0 && strings.Compare(keys[j-1], keys[j]) > 0; j-- {
|
||||
keys[j-1], keys[j] = keys[j], keys[j-1]
|
||||
}
|
||||
}
|
||||
return keys
|
||||
}
|
||||
422
internal/resolve/resolve_cache_test.go
Normal file
422
internal/resolve/resolve_cache_test.go
Normal file
@@ -0,0 +1,422 @@
|
||||
package resolve_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/buildmeta"
|
||||
"github.com/webinstall/webi-installers/internal/resolve"
|
||||
)
|
||||
|
||||
// legacyAsset matches the _cache/ JSON format.
|
||||
type legacyAsset struct {
|
||||
Name string `json:"name"`
|
||||
Version string `json:"version"`
|
||||
LTS bool `json:"lts"`
|
||||
Channel string `json:"channel"`
|
||||
Date string `json:"date"`
|
||||
OS string `json:"os"`
|
||||
Arch string `json:"arch"`
|
||||
Libc string `json:"libc"`
|
||||
Ext string `json:"ext"`
|
||||
Download string `json:"download"`
|
||||
}
|
||||
|
||||
type legacyCache struct {
|
||||
Releases []legacyAsset `json:"releases"`
|
||||
}
|
||||
|
||||
func loadCacheDists(t *testing.T, pkg string) []resolve.Dist {
|
||||
t.Helper()
|
||||
cacheDir := filepath.Join("..", "..", "_cache", "2026-03")
|
||||
path := filepath.Join(cacheDir, pkg+".json")
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Skipf("no cache file for %s: %v", pkg, err)
|
||||
}
|
||||
var lc legacyCache
|
||||
if err := json.Unmarshal(data, &lc); err != nil {
|
||||
t.Fatalf("parse %s: %v", pkg, err)
|
||||
}
|
||||
dists := make([]resolve.Dist, len(lc.Releases))
|
||||
for i, la := range lc.Releases {
|
||||
// Reverse-translate legacy Node.js vocabulary to Go canonical names.
|
||||
// The cache file uses macos/amd64/arm64; the resolver uses darwin/x86_64/aarch64.
|
||||
osStr := la.OS
|
||||
if osStr == "macos" {
|
||||
osStr = "darwin"
|
||||
}
|
||||
archStr := la.Arch
|
||||
switch archStr {
|
||||
case "amd64":
|
||||
archStr = "x86_64"
|
||||
case "arm64":
|
||||
archStr = "aarch64"
|
||||
}
|
||||
// Restore dot-prefix convention: cache stores "tar.gz", resolver needs ".tar.gz".
|
||||
// "exe" with no dot in filename = bare binary (Format ""), otherwise ".exe".
|
||||
format := la.Ext
|
||||
switch {
|
||||
case format == "exe" && !strings.Contains(la.Name, "."):
|
||||
format = ""
|
||||
case format != "":
|
||||
format = "." + format
|
||||
}
|
||||
dists[i] = resolve.Dist{
|
||||
Filename: la.Name,
|
||||
Version: la.Version,
|
||||
LTS: la.LTS,
|
||||
Channel: la.Channel,
|
||||
Date: la.Date,
|
||||
OS: osStr,
|
||||
Arch: archStr,
|
||||
Libc: la.Libc, // "none" = buildmeta.LibcNone (statically linked)
|
||||
Format: format,
|
||||
Download: la.Download,
|
||||
}
|
||||
}
|
||||
return dists
|
||||
}
|
||||
|
||||
// platforms is the standard webi target matrix.
|
||||
var platforms = []struct {
|
||||
name string
|
||||
os buildmeta.OS
|
||||
arch buildmeta.Arch
|
||||
formats []string
|
||||
}{
|
||||
{"darwin-arm64", buildmeta.OSDarwin, buildmeta.ArchARM64, []string{".tar.xz", ".tar.gz", ".zip"}},
|
||||
{"darwin-amd64", buildmeta.OSDarwin, buildmeta.ArchAMD64, []string{".tar.xz", ".tar.gz", ".zip"}},
|
||||
{"linux-amd64", buildmeta.OSLinux, buildmeta.ArchAMD64, []string{".tar.xz", ".exe.xz", ".tar.gz", ".xz", ".gz", ".zip"}},
|
||||
{"linux-arm64", buildmeta.OSLinux, buildmeta.ArchARM64, []string{".tar.xz", ".exe.xz", ".tar.gz", ".xz", ".gz", ".zip"}},
|
||||
{"linux-armv7", buildmeta.OSLinux, buildmeta.ArchARMv7, []string{".tar.xz", ".tar.gz", ".xz", ".gz", ".zip"}},
|
||||
{"linux-armv6", buildmeta.OSLinux, buildmeta.ArchARMv6, []string{".tar.xz", ".tar.gz", ".xz", ".gz", ".zip"}},
|
||||
{"windows-amd64", buildmeta.OSWindows, buildmeta.ArchAMD64, []string{".zip", ".tar.gz", ".exe", ".7z"}},
|
||||
{"windows-arm64", buildmeta.OSWindows, buildmeta.ArchARM64, []string{".zip", ".tar.gz", ".exe", ".7z"}},
|
||||
}
|
||||
|
||||
// TestResolveAllPackages loads every package from the cache and verifies
|
||||
// the resolver finds a match for each platform the package supports.
|
||||
func TestResolveAllPackages(t *testing.T) {
|
||||
cacheDir := filepath.Join("..", "..", "_cache", "2026-03")
|
||||
entries, err := os.ReadDir(cacheDir)
|
||||
if err != nil {
|
||||
t.Skipf("no cache dir: %v", err)
|
||||
}
|
||||
|
||||
var pkgs []string
|
||||
for _, e := range entries {
|
||||
if strings.HasSuffix(e.Name(), ".json") {
|
||||
pkgs = append(pkgs, strings.TrimSuffix(e.Name(), ".json"))
|
||||
}
|
||||
}
|
||||
|
||||
if len(pkgs) < 50 {
|
||||
t.Fatalf("expected at least 50 packages, got %d", len(pkgs))
|
||||
}
|
||||
|
||||
for _, pkg := range pkgs {
|
||||
t.Run(pkg, func(t *testing.T) {
|
||||
dists := loadCacheDists(t, pkg)
|
||||
if len(dists) == 0 {
|
||||
t.Skip("no releases")
|
||||
}
|
||||
|
||||
// Determine which platforms this package supports.
|
||||
cat := resolve.Survey(dists)
|
||||
osSet := make(map[string]bool, len(cat.OSes))
|
||||
for _, o := range cat.OSes {
|
||||
osSet[o] = true
|
||||
}
|
||||
|
||||
for _, plat := range platforms {
|
||||
platOS := string(plat.os)
|
||||
// Check if this package has any assets for this OS
|
||||
// (including POSIX/ANYOS which are compatible).
|
||||
supported := osSet[platOS] ||
|
||||
osSet[string(buildmeta.OSAny)] ||
|
||||
(platOS != "windows" && (osSet[string(buildmeta.OSPosix2017)] || osSet[string(buildmeta.OSPosix2024)]))
|
||||
|
||||
if !supported {
|
||||
continue
|
||||
}
|
||||
|
||||
t.Run(plat.name, func(t *testing.T) {
|
||||
m := resolve.Best(dists, resolve.Query{
|
||||
OS: plat.os,
|
||||
Arch: plat.arch,
|
||||
Formats: plat.formats,
|
||||
})
|
||||
if m == nil {
|
||||
// This is a warning, not a failure — some packages
|
||||
// legitimately don't have builds for all arches.
|
||||
// But log it so we can spot unexpected gaps.
|
||||
t.Logf("WARN: no match for %s on %s (has OSes: %v, Arches: %v)",
|
||||
pkg, plat.name, cat.OSes, cat.Arches)
|
||||
return
|
||||
}
|
||||
if m.Version == "" {
|
||||
t.Error("matched but Version is empty")
|
||||
}
|
||||
if m.Download == "" {
|
||||
t.Error("matched but Download is empty")
|
||||
}
|
||||
})
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Packages with known platform expectations. Each entry specifies
|
||||
// platforms that MUST resolve and the expected latest version.
|
||||
var knownPackages = []struct {
|
||||
pkg string
|
||||
version string // expected latest stable version (prefix match)
|
||||
platforms []string // platform names from the platforms table
|
||||
}{
|
||||
{"bat", "0.26", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
{"caddy", "2.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "linux-armv7", "linux-armv6", "windows-amd64"}},
|
||||
{"delta", "0.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
{"fd", "10.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "linux-armv7", "windows-amd64"}},
|
||||
{"fzf", "0.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "linux-armv7", "windows-amd64"}},
|
||||
{"gh", "2.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "linux-armv6", "windows-amd64"}},
|
||||
{"rg", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
{"shellcheck", "0.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "linux-armv6", "windows-amd64"}},
|
||||
{"shfmt", "3.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "linux-armv6", "windows-amd64"}},
|
||||
{"xz", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
{"yq", "4.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "linux-armv6", "windows-amd64"}},
|
||||
{"zoxide", "0.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "linux-armv7", "windows-amd64"}},
|
||||
{"aliasman", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64"}},
|
||||
{"comrak", "0.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "windows-amd64"}},
|
||||
{"hugo", "0.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
{"node", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
{"terraform", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
{"zig", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
}
|
||||
|
||||
// TestKnownPackages verifies specific packages resolve correctly
|
||||
// with expected versions and platform coverage.
|
||||
func TestKnownPackages(t *testing.T) {
|
||||
platMap := make(map[string]struct {
|
||||
os buildmeta.OS
|
||||
arch buildmeta.Arch
|
||||
formats []string
|
||||
})
|
||||
for _, p := range platforms {
|
||||
platMap[p.name] = struct {
|
||||
os buildmeta.OS
|
||||
arch buildmeta.Arch
|
||||
formats []string
|
||||
}{p.os, p.arch, p.formats}
|
||||
}
|
||||
|
||||
for _, kp := range knownPackages {
|
||||
t.Run(kp.pkg, func(t *testing.T) {
|
||||
dists := loadCacheDists(t, kp.pkg)
|
||||
|
||||
for _, platName := range kp.platforms {
|
||||
plat := platMap[platName]
|
||||
t.Run(platName, func(t *testing.T) {
|
||||
m := resolve.Best(dists, resolve.Query{
|
||||
OS: plat.os,
|
||||
Arch: plat.arch,
|
||||
Formats: plat.formats,
|
||||
})
|
||||
if m == nil {
|
||||
t.Skipf("no build available for %s on %s — upstream gap", kp.pkg, platName)
|
||||
return
|
||||
}
|
||||
if kp.version != "" {
|
||||
// Strip leading "v" for prefix comparison.
|
||||
v := strings.TrimPrefix(m.Version, "v")
|
||||
if !strings.HasPrefix(v, kp.version) {
|
||||
t.Errorf("Version = %q, want prefix %q", m.Version, kp.version)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestResolveVersionConstraints tests version pinning across real packages.
|
||||
func TestResolveVersionConstraints(t *testing.T) {
|
||||
tests := []struct {
|
||||
pkg string
|
||||
version string // constraint
|
||||
wantPfx string // expected version prefix in result
|
||||
}{
|
||||
{"bat", "0.25", "0.25"},
|
||||
{"bat", "0.26", "0.26"},
|
||||
{"gh", "2.40", "2.40"},
|
||||
{"node", "20", "20."},
|
||||
{"node", "22", "22."},
|
||||
{"hugo", "0.121", "0.121"},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
name := fmt.Sprintf("%s@%s", tt.pkg, tt.version)
|
||||
t.Run(name, func(t *testing.T) {
|
||||
dists := loadCacheDists(t, tt.pkg)
|
||||
m := resolve.Best(dists, resolve.Query{
|
||||
OS: buildmeta.OSLinux,
|
||||
Arch: buildmeta.ArchAMD64,
|
||||
Formats: []string{".tar.xz", ".tar.gz", ".zip"},
|
||||
Version: tt.version,
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatalf("no match for %s@%s", tt.pkg, tt.version)
|
||||
}
|
||||
v := strings.TrimPrefix(m.Version, "v")
|
||||
if !strings.HasPrefix(v, tt.wantPfx) {
|
||||
t.Errorf("Version = %q, want prefix %q", m.Version, tt.wantPfx)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestResolveArchFallbackReal tests arch fallback with real package data.
|
||||
func TestResolveArchFallbackReal(t *testing.T) {
|
||||
// awless only has amd64 builds — macOS ARM64 should fall back.
|
||||
dists := loadCacheDists(t, "awless")
|
||||
m := resolve.Best(dists, resolve.Query{
|
||||
OS: buildmeta.OSDarwin,
|
||||
Arch: buildmeta.ArchARM64,
|
||||
Formats: []string{".tar.gz", ".zip"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected Rosetta 2 fallback for awless")
|
||||
}
|
||||
if m.Arch != "x86_64" {
|
||||
t.Errorf("Arch = %q, want x86_64", m.Arch)
|
||||
}
|
||||
}
|
||||
|
||||
// TestResolvePosixPackages tests packages that use posix_2017/ANYOS.
|
||||
func TestResolvePosixPackages(t *testing.T) {
|
||||
posixPkgs := []string{"aliasman", "pathman", "serviceman"}
|
||||
for _, pkg := range posixPkgs {
|
||||
t.Run(pkg, func(t *testing.T) {
|
||||
dists := loadCacheDists(t, pkg)
|
||||
if len(dists) == 0 {
|
||||
t.Skip("no releases")
|
||||
}
|
||||
|
||||
// Should resolve on Linux.
|
||||
m := resolve.Best(dists, resolve.Query{
|
||||
OS: buildmeta.OSLinux,
|
||||
Arch: buildmeta.ArchAMD64,
|
||||
Formats: []string{".tar.xz", ".tar.gz", ".zip", ".xz", ".gz"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Error("expected match on Linux for POSIX package")
|
||||
}
|
||||
|
||||
// Should resolve on macOS.
|
||||
m = resolve.Best(dists, resolve.Query{
|
||||
OS: buildmeta.OSDarwin,
|
||||
Arch: buildmeta.ArchARM64,
|
||||
Formats: []string{".tar.xz", ".tar.gz", ".zip"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Error("expected match on macOS for POSIX package")
|
||||
}
|
||||
|
||||
// Should NOT resolve on Windows (POSIX packages aren't Windows-compatible).
|
||||
m = resolve.Best(dists, resolve.Query{
|
||||
OS: buildmeta.OSWindows,
|
||||
Arch: buildmeta.ArchAMD64,
|
||||
Formats: []string{".zip", ".tar.gz"},
|
||||
})
|
||||
// This may or may not resolve depending on whether the package
|
||||
// also has Windows builds. Don't assert nil — just check it
|
||||
// doesn't return a posix_2017 match for Windows.
|
||||
if m != nil && (m.OS == "posix_2017" || m.OS == "posix_2024") {
|
||||
t.Errorf("POSIX package should not match Windows, got OS=%q", m.OS)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestResolveLibcPreference tests libc selection.
|
||||
// bat is a Rust project — its musl builds are static (libc='none').
|
||||
// pwsh has hard musl dependencies (libc='musl').
|
||||
func TestResolveLibcPreference(t *testing.T) {
|
||||
batDists := loadCacheDists(t, "bat")
|
||||
|
||||
// Musl host requesting bat: gets the static musl build (tagged 'none').
|
||||
m := resolve.Best(batDists, resolve.Query{
|
||||
OS: buildmeta.OSLinux,
|
||||
Arch: buildmeta.ArchAMD64,
|
||||
Libc: buildmeta.LibcMusl,
|
||||
Formats: []string{".tar.gz"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected match for musl host")
|
||||
}
|
||||
// Rust musl builds are static — tagged as 'none', not 'musl'.
|
||||
if m.Libc != "none" {
|
||||
t.Errorf("bat musl request: Libc = %q, want none (static musl)", m.Libc)
|
||||
}
|
||||
|
||||
// Explicit gnu request.
|
||||
m = resolve.Best(batDists, resolve.Query{
|
||||
OS: buildmeta.OSLinux,
|
||||
Arch: buildmeta.ArchAMD64,
|
||||
Libc: buildmeta.LibcGNU,
|
||||
Formats: []string{".tar.gz"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected gnu match")
|
||||
}
|
||||
if m.Libc != "gnu" {
|
||||
t.Errorf("Libc = %q, want gnu", m.Libc)
|
||||
}
|
||||
|
||||
// No preference — should still match (accepts any).
|
||||
m = resolve.Best(batDists, resolve.Query{
|
||||
OS: buildmeta.OSLinux,
|
||||
Arch: buildmeta.ArchAMD64,
|
||||
Formats: []string{".tar.gz"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected match with no libc preference")
|
||||
}
|
||||
|
||||
// pwsh has hard musl builds (dynamically linked, requires musl runtime).
|
||||
pwshDists := loadCacheDists(t, "pwsh")
|
||||
m = resolve.Best(pwshDists, resolve.Query{
|
||||
OS: buildmeta.OSLinux,
|
||||
Arch: buildmeta.ArchAMD64,
|
||||
Libc: buildmeta.LibcMusl,
|
||||
Formats: []string{".tar.gz"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected pwsh musl match")
|
||||
}
|
||||
if m.Libc != "musl" {
|
||||
t.Errorf("pwsh musl request: Libc = %q, want musl", m.Libc)
|
||||
}
|
||||
}
|
||||
|
||||
// TestResolveFormatFallback tests format preference cascading.
|
||||
func TestResolveFormatFallback(t *testing.T) {
|
||||
// Request .tar.xz first, fall back to .tar.gz.
|
||||
dists := loadCacheDists(t, "bat")
|
||||
m := resolve.Best(dists, resolve.Query{
|
||||
OS: buildmeta.OSLinux,
|
||||
Arch: buildmeta.ArchAMD64,
|
||||
Formats: []string{".tar.xz", ".tar.gz", ".zip"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected match")
|
||||
}
|
||||
// bat only has .tar.gz — should fall back from .tar.xz.
|
||||
if m.Format != ".tar.gz" {
|
||||
t.Errorf("Format = %q, want .tar.gz (fallback from .tar.xz)", m.Format)
|
||||
}
|
||||
}
|
||||
250
internal/resolve/resolve_test.go
Normal file
250
internal/resolve/resolve_test.go
Normal file
@@ -0,0 +1,250 @@
|
||||
package resolve
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/buildmeta"
|
||||
)
|
||||
|
||||
// bat-style dists: standard goreleaser output.
|
||||
var batDists = []Dist{
|
||||
{Version: "0.26.1", Channel: "stable", OS: "darwin", Arch: "aarch64", Format: ".tar.gz", Filename: "bat-v0.26.1-aarch64-apple-darwin.tar.gz"},
|
||||
{Version: "0.26.1", Channel: "stable", OS: "darwin", Arch: "x86_64", Format: ".tar.gz", Filename: "bat-v0.26.1-x86_64-apple-darwin.tar.gz"},
|
||||
{Version: "0.26.1", Channel: "stable", OS: "linux", Arch: "aarch64", Libc: "gnu", Format: ".tar.gz", Filename: "bat-v0.26.1-aarch64-unknown-linux-gnu.tar.gz"},
|
||||
{Version: "0.26.1", Channel: "stable", OS: "linux", Arch: "aarch64", Libc: "musl", Format: ".tar.gz", Filename: "bat-v0.26.1-aarch64-unknown-linux-musl.tar.gz"},
|
||||
{Version: "0.26.1", Channel: "stable", OS: "linux", Arch: "x86_64", Libc: "gnu", Format: ".tar.gz", Filename: "bat-v0.26.1-x86_64-unknown-linux-gnu.tar.gz"},
|
||||
{Version: "0.26.1", Channel: "stable", OS: "linux", Arch: "x86_64", Libc: "musl", Format: ".tar.gz", Filename: "bat-v0.26.1-x86_64-unknown-linux-musl.tar.gz"},
|
||||
{Version: "0.26.1", Channel: "stable", OS: "windows", Arch: "aarch64", Format: ".zip", Filename: "bat-v0.26.1-aarch64-pc-windows-msvc.zip"},
|
||||
{Version: "0.26.1", Channel: "stable", OS: "windows", Arch: "x86_64", Libc: "gnu", Format: ".zip", Filename: "bat-v0.26.1-x86_64-pc-windows-gnu.zip"},
|
||||
{Version: "0.26.1", Channel: "stable", OS: "windows", Arch: "x86_64", Libc: "msvc", Format: ".zip", Filename: "bat-v0.26.1-x86_64-pc-windows-msvc.zip"},
|
||||
// Older version.
|
||||
{Version: "0.25.0", Channel: "stable", OS: "darwin", Arch: "aarch64", Format: ".tar.gz", Filename: "bat-v0.25.0-aarch64-apple-darwin.tar.gz"},
|
||||
{Version: "0.25.0", Channel: "stable", OS: "linux", Arch: "x86_64", Libc: "gnu", Format: ".tar.gz", Filename: "bat-v0.25.0-x86_64-unknown-linux-gnu.tar.gz"},
|
||||
}
|
||||
|
||||
func TestBestExactMatch(t *testing.T) {
|
||||
m := Best(batDists, Query{
|
||||
OS: buildmeta.OSLinux,
|
||||
Arch: buildmeta.ArchAMD64,
|
||||
Formats: []string{".tar.gz"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected match")
|
||||
}
|
||||
if m.Version != "0.26.1" {
|
||||
t.Errorf("Version = %q, want 0.26.1", m.Version)
|
||||
}
|
||||
if m.Filename != "bat-v0.26.1-x86_64-unknown-linux-gnu.tar.gz" {
|
||||
t.Errorf("Filename = %q", m.Filename)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBestVersionConstraint(t *testing.T) {
|
||||
m := Best(batDists, Query{
|
||||
OS: buildmeta.OSDarwin,
|
||||
Arch: buildmeta.ArchARM64,
|
||||
Formats: []string{".tar.gz"},
|
||||
Version: "0.25",
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected match")
|
||||
}
|
||||
if m.Version != "0.25.0" {
|
||||
t.Errorf("Version = %q, want 0.25.0", m.Version)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBestArchFallback(t *testing.T) {
|
||||
// macOS ARM64 should fall back to x86_64 via Rosetta 2
|
||||
// when no ARM64 build exists.
|
||||
dists := []Dist{
|
||||
{Version: "1.0.0", Channel: "stable", OS: "darwin", Arch: "x86_64", Format: ".tar.gz", Filename: "tool-darwin-amd64.tar.gz"},
|
||||
}
|
||||
m := Best(dists, Query{
|
||||
OS: buildmeta.OSDarwin,
|
||||
Arch: buildmeta.ArchARM64,
|
||||
Formats: []string{".tar.gz"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected match via Rosetta 2 fallback")
|
||||
}
|
||||
if m.Arch != "x86_64" {
|
||||
t.Errorf("Arch = %q, want x86_64", m.Arch)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBestPrefersNativeOverCompat(t *testing.T) {
|
||||
// When both native and compat builds exist, prefer native.
|
||||
dists := []Dist{
|
||||
{Version: "1.0.0", Channel: "stable", OS: "darwin", Arch: "x86_64", Format: ".tar.gz", Filename: "tool-darwin-amd64.tar.gz"},
|
||||
{Version: "1.0.0", Channel: "stable", OS: "darwin", Arch: "aarch64", Format: ".tar.gz", Filename: "tool-darwin-arm64.tar.gz"},
|
||||
}
|
||||
m := Best(dists, Query{
|
||||
OS: buildmeta.OSDarwin,
|
||||
Arch: buildmeta.ArchARM64,
|
||||
Formats: []string{".tar.gz"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected match")
|
||||
}
|
||||
if m.Arch != "aarch64" {
|
||||
t.Errorf("Arch = %q, want aarch64 (native)", m.Arch)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBestFormatPreference(t *testing.T) {
|
||||
dists := []Dist{
|
||||
{Version: "1.0.0", Channel: "stable", OS: "linux", Arch: "x86_64", Format: ".zip", Filename: "tool.zip"},
|
||||
{Version: "1.0.0", Channel: "stable", OS: "linux", Arch: "x86_64", Format: ".tar.gz", Filename: "tool.tar.gz"},
|
||||
{Version: "1.0.0", Channel: "stable", OS: "linux", Arch: "x86_64", Format: ".tar.xz", Filename: "tool.tar.xz"},
|
||||
}
|
||||
m := Best(dists, Query{
|
||||
OS: buildmeta.OSLinux,
|
||||
Arch: buildmeta.ArchAMD64,
|
||||
Formats: []string{".tar.xz", ".tar.gz", ".zip"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected match")
|
||||
}
|
||||
if m.Format != ".tar.xz" {
|
||||
t.Errorf("Format = %q, want .tar.xz", m.Format)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBestNoMatch(t *testing.T) {
|
||||
m := Best(batDists, Query{
|
||||
OS: buildmeta.OSFreeBSD,
|
||||
Arch: buildmeta.ArchAMD64,
|
||||
Formats: []string{".tar.gz"},
|
||||
})
|
||||
if m != nil {
|
||||
t.Errorf("expected nil, got %+v", m)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBestLibcMusl(t *testing.T) {
|
||||
m := Best(batDists, Query{
|
||||
OS: buildmeta.OSLinux,
|
||||
Arch: buildmeta.ArchAMD64,
|
||||
Libc: buildmeta.LibcMusl,
|
||||
Formats: []string{".tar.gz"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected match")
|
||||
}
|
||||
if m.Libc != "musl" {
|
||||
t.Errorf("Libc = %q, want musl", m.Libc)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBestPrefersBaseOverVariant(t *testing.T) {
|
||||
dists := []Dist{
|
||||
{Version: "1.0.0", Channel: "stable", OS: "linux", Arch: "x86_64", Format: ".tar.gz", Filename: "tool.tar.gz"},
|
||||
{Version: "1.0.0", Channel: "stable", OS: "linux", Arch: "x86_64", Format: ".tar.gz", Filename: "tool-rocm.tar.gz", Variants: []string{"rocm"}},
|
||||
}
|
||||
m := Best(dists, Query{
|
||||
OS: buildmeta.OSLinux,
|
||||
Arch: buildmeta.ArchAMD64,
|
||||
Formats: []string{".tar.gz"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected match")
|
||||
}
|
||||
if m.Filename != "tool.tar.gz" {
|
||||
t.Errorf("got variant build %q, want base", m.Filename)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBestPosixFallback(t *testing.T) {
|
||||
dists := []Dist{
|
||||
{Version: "1.0.0", Channel: "stable", OS: "posix_2017", Format: ".tar.gz", Filename: "script.tar.gz"},
|
||||
}
|
||||
m := Best(dists, Query{
|
||||
OS: buildmeta.OSLinux,
|
||||
Arch: buildmeta.ArchAMD64,
|
||||
Formats: []string{".tar.gz"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected match via POSIX fallback")
|
||||
}
|
||||
if m.OS != "posix_2017" {
|
||||
t.Errorf("OS = %q, want posix_2017", m.OS)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBestAnyOS(t *testing.T) {
|
||||
dists := []Dist{
|
||||
{Version: "1.0.0", Channel: "stable", OS: "ANYOS", Format: ".tar.gz", Filename: "tool.tar.gz"},
|
||||
}
|
||||
m := Best(dists, Query{
|
||||
OS: buildmeta.OSWindows,
|
||||
Arch: buildmeta.ArchAMD64,
|
||||
Formats: []string{".tar.gz"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected match via ANYOS")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBestAnyArch(t *testing.T) {
|
||||
dists := []Dist{
|
||||
{Version: "1.0.0", Channel: "stable", OS: "linux", Arch: "ANYARCH", Format: ".tar.gz", Filename: "tool.tar.gz"},
|
||||
}
|
||||
m := Best(dists, Query{
|
||||
OS: buildmeta.OSLinux,
|
||||
Arch: buildmeta.ArchARM64,
|
||||
Formats: []string{".tar.gz"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected match via ANYARCH")
|
||||
}
|
||||
}
|
||||
|
||||
func TestBestWindowsArchFallback(t *testing.T) {
|
||||
// Windows ARM64 should fall back to x86_64 via emulation.
|
||||
dists := []Dist{
|
||||
{Version: "1.0.0", Channel: "stable", OS: "windows", Arch: "x86_64", Format: ".zip", Filename: "tool-win64.zip"},
|
||||
}
|
||||
m := Best(dists, Query{
|
||||
OS: buildmeta.OSWindows,
|
||||
Arch: buildmeta.ArchARM64,
|
||||
Formats: []string{".zip"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected match via Windows ARM64 emulation")
|
||||
}
|
||||
if m.Arch != "x86_64" {
|
||||
t.Errorf("Arch = %q, want x86_64", m.Arch)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBestMicroArchFallback(t *testing.T) {
|
||||
// amd64v3 query should fall back to amd64 baseline.
|
||||
dists := []Dist{
|
||||
{Version: "1.0.0", Channel: "stable", OS: "linux", Arch: "x86_64", Format: ".tar.gz", Filename: "tool-amd64.tar.gz"},
|
||||
}
|
||||
m := Best(dists, Query{
|
||||
OS: buildmeta.OSLinux,
|
||||
Arch: buildmeta.ArchAMD64v3,
|
||||
Formats: []string{".tar.gz"},
|
||||
})
|
||||
if m == nil {
|
||||
t.Fatal("expected match via micro-arch fallback")
|
||||
}
|
||||
if m.Arch != "x86_64" {
|
||||
t.Errorf("Arch = %q, want x86_64 (baseline)", m.Arch)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSurvey(t *testing.T) {
|
||||
cat := Survey(batDists)
|
||||
if cat.Stable != "0.26.1" {
|
||||
t.Errorf("Stable = %q, want 0.26.1", cat.Stable)
|
||||
}
|
||||
if cat.Latest != "0.26.1" {
|
||||
t.Errorf("Latest = %q, want 0.26.1", cat.Latest)
|
||||
}
|
||||
if len(cat.OSes) != 3 {
|
||||
t.Errorf("OSes = %v, want 3", cat.OSes)
|
||||
}
|
||||
}
|
||||
416
internal/resolver/resolver.go
Normal file
416
internal/resolver/resolver.go
Normal file
@@ -0,0 +1,416 @@
|
||||
// Package resolver selects the best release asset for a given platform
|
||||
// and version constraint.
|
||||
//
|
||||
// The resolver takes a package's full asset list and a request describing
|
||||
// what the client needs (OS, arch, libc, version prefix, channel, format
|
||||
// preferences). It returns the single best matching asset or an error.
|
||||
//
|
||||
// Resolution order:
|
||||
// 1. Filter assets by channel (inclusive: @stable includes stable+lts)
|
||||
// 2. Sort versions descending, filter by version prefix if given
|
||||
// 3. For each candidate version, try compatible platform triplets
|
||||
// (OS × CompatArches fallback × libc) in preference order
|
||||
// 4. Among platform matches, pick the best format
|
||||
// 5. Among format matches, prefer assets without build variants
|
||||
package resolver
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"slices"
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/buildmeta"
|
||||
"github.com/webinstall/webi-installers/internal/lexver"
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// ErrNoMatch is returned when no asset matches the request.
|
||||
var ErrNoMatch = errors.New("resolver: no matching asset")
|
||||
|
||||
// Request describes what the client is looking for.
|
||||
type Request struct {
|
||||
// OS is the target operating system (e.g. "linux", "darwin", "windows").
|
||||
OS string
|
||||
|
||||
// Arch is the target architecture (e.g. "aarch64", "x86_64").
|
||||
Arch string
|
||||
|
||||
// Libc is the preferred C library (e.g. "gnu", "musl", "msvc").
|
||||
// Empty means no preference — the resolver tries all libc values.
|
||||
Libc string
|
||||
|
||||
// Version is a version prefix constraint (e.g. "1.20", "1", "").
|
||||
// Empty means latest. Exact versions like "1.20.3" also work.
|
||||
Version string
|
||||
|
||||
// Channel selects the release stability level. Values:
|
||||
// ""/"stable" — stable and LTS only (default)
|
||||
// "lts" — LTS releases only
|
||||
// "rc" — rc + stable + LTS
|
||||
// "beta" — beta + rc + stable + LTS
|
||||
// "alpha" — everything (alpha + beta + rc + stable + LTS)
|
||||
// "pre" — alias for beta (package-specific meaning)
|
||||
Channel string
|
||||
|
||||
// LTS when true selects only LTS-flagged releases.
|
||||
LTS bool
|
||||
|
||||
// Formats lists acceptable archive formats in preference order.
|
||||
// If empty, a default preference order is used.
|
||||
Formats []string
|
||||
|
||||
// Variant selects a specific build variant (e.g. "rocm", "jetpack6").
|
||||
// If empty, assets with variants are deprioritized.
|
||||
Variant string
|
||||
}
|
||||
|
||||
// Result holds the resolved asset and metadata about the match.
|
||||
type Result struct {
|
||||
// Asset is the selected download.
|
||||
Asset storage.Asset
|
||||
|
||||
// Version is the matched version string.
|
||||
Version string
|
||||
|
||||
// Triplet is the matched platform triplet (os-arch-libc).
|
||||
Triplet string
|
||||
}
|
||||
|
||||
// Resolve finds the best matching asset for the given request.
|
||||
func Resolve(assets []storage.Asset, req Request) (Result, error) {
|
||||
if len(assets) == 0 {
|
||||
return Result{}, ErrNoMatch
|
||||
}
|
||||
|
||||
// Parse the version prefix for filtering.
|
||||
var versionPrefix lexver.Version
|
||||
hasPrefix := req.Version != ""
|
||||
if hasPrefix {
|
||||
versionPrefix = lexver.Parse(req.Version)
|
||||
}
|
||||
|
||||
// Build the channel filter.
|
||||
channelOK := channelFilter(req.Channel, req.LTS)
|
||||
|
||||
// Parse and sort all unique versions descending.
|
||||
type versionEntry struct {
|
||||
parsed lexver.Version
|
||||
raw string
|
||||
}
|
||||
seen := make(map[string]bool)
|
||||
var versions []versionEntry
|
||||
for _, a := range assets {
|
||||
if seen[a.Version] {
|
||||
continue
|
||||
}
|
||||
seen[a.Version] = true
|
||||
v := lexver.Parse(a.Version)
|
||||
v.Raw = a.Version
|
||||
versions = append(versions, versionEntry{parsed: v, raw: a.Version})
|
||||
}
|
||||
slices.SortFunc(versions, func(a, b versionEntry) int {
|
||||
return lexver.Compare(b.parsed, a.parsed) // descending
|
||||
})
|
||||
|
||||
// Build platform fallback list: ordered (os, arch, libc) combinations.
|
||||
triplets := enumerateTriplets(req.OS, req.Arch, req.Libc)
|
||||
|
||||
// Build format preference list.
|
||||
formats := req.Formats
|
||||
if len(formats) == 0 {
|
||||
formats = defaultFormats(req.OS)
|
||||
}
|
||||
|
||||
// Index assets by version+triplet for fast lookup.
|
||||
// Assets with empty OS/Arch (like git repos) use "" keys.
|
||||
type tripletKey struct {
|
||||
version string
|
||||
os string
|
||||
arch string
|
||||
libc string
|
||||
}
|
||||
index := make(map[tripletKey][]storage.Asset)
|
||||
for _, a := range assets {
|
||||
key := tripletKey{
|
||||
version: a.Version,
|
||||
os: a.OS,
|
||||
arch: a.Arch,
|
||||
libc: a.Libc,
|
||||
}
|
||||
index[key] = append(index[key], a)
|
||||
}
|
||||
|
||||
// Walk versions in descending order.
|
||||
for _, ve := range versions {
|
||||
// Check version prefix.
|
||||
if hasPrefix && !ve.parsed.HasPrefix(versionPrefix) {
|
||||
continue
|
||||
}
|
||||
|
||||
// Check channel.
|
||||
if !channelOK(ve.parsed.Channel, ve.raw) {
|
||||
continue
|
||||
}
|
||||
|
||||
// Try each compatible triplet.
|
||||
for _, tri := range triplets {
|
||||
key := tripletKey{
|
||||
version: ve.raw,
|
||||
os: tri.os,
|
||||
arch: tri.arch,
|
||||
libc: tri.libc,
|
||||
}
|
||||
candidates := index[key]
|
||||
if len(candidates) == 0 {
|
||||
continue
|
||||
}
|
||||
|
||||
// Pick the best asset from candidates.
|
||||
best, ok := pickBest(candidates, formats, req.Variant, req.LTS)
|
||||
if !ok {
|
||||
continue
|
||||
}
|
||||
|
||||
triplet := tri.os + "-" + tri.arch + "-" + tri.libc
|
||||
return Result{
|
||||
Asset: best,
|
||||
Version: ve.raw,
|
||||
Triplet: triplet,
|
||||
}, nil
|
||||
}
|
||||
}
|
||||
|
||||
return Result{}, ErrNoMatch
|
||||
}
|
||||
|
||||
// channelFilter returns a function that checks whether a given channel
|
||||
// is acceptable for the requested channel level.
|
||||
func channelFilter(requested string, ltsOnly bool) func(channel string, version string) bool {
|
||||
if ltsOnly {
|
||||
return func(_ string, _ string) bool {
|
||||
// LTS filtering happens at the asset level, not version level.
|
||||
// We let all versions through and filter by LTS flag later.
|
||||
// Actually, LTS is per-asset, so we handle it in pickBest.
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
requested = strings.ToLower(requested)
|
||||
if requested == "" {
|
||||
requested = "stable"
|
||||
}
|
||||
if requested == "pre" {
|
||||
requested = "beta"
|
||||
}
|
||||
if requested == "latest" {
|
||||
requested = "stable"
|
||||
}
|
||||
|
||||
// channelRank maps channel names to a numeric rank.
|
||||
// Higher rank = less stable. A request for rank N accepts
|
||||
// everything at rank N or below.
|
||||
rank := func(ch string) int {
|
||||
ch = strings.ToLower(ch)
|
||||
switch ch {
|
||||
case "", "stable":
|
||||
return 0
|
||||
case "rc":
|
||||
return 1
|
||||
case "beta", "preview":
|
||||
return 2
|
||||
case "alpha", "dev":
|
||||
return 3
|
||||
default:
|
||||
return 2 // unknown pre-release channels default to beta-level
|
||||
}
|
||||
}
|
||||
|
||||
maxRank := rank(requested)
|
||||
return func(channel string, _ string) bool {
|
||||
return rank(channel) <= maxRank
|
||||
}
|
||||
}
|
||||
|
||||
type platformTriple struct {
|
||||
os string
|
||||
arch string
|
||||
libc string
|
||||
}
|
||||
|
||||
// enumerateTriplets builds the ordered list of platform combinations to try.
|
||||
// It uses CompatArches for arch fallback and tries multiple libc values.
|
||||
func enumerateTriplets(osStr, archStr, libcStr string) []platformTriple {
|
||||
// OS candidates: specific OS first, then POSIX compat, then any.
|
||||
var oses []string
|
||||
switch osStr {
|
||||
case "windows":
|
||||
oses = []string{"windows", "ANYOS", ""}
|
||||
case "android":
|
||||
oses = []string{"android", "linux", "posix_2024", "posix_2017", "ANYOS", ""}
|
||||
case "":
|
||||
oses = []string{"ANYOS", ""}
|
||||
default:
|
||||
oses = []string{osStr, "posix_2024", "posix_2017", "ANYOS", ""}
|
||||
}
|
||||
|
||||
// Arch candidates: use CompatArches for fallback chain.
|
||||
arches := buildmeta.CompatArches(buildmeta.OS(osStr), buildmeta.Arch(archStr))
|
||||
var archStrs []string
|
||||
for _, a := range arches {
|
||||
archStrs = append(archStrs, string(a))
|
||||
}
|
||||
// Also try ANYARCH and empty (for platform-agnostic assets like git repos).
|
||||
archStrs = append(archStrs, "ANYARCH", "")
|
||||
|
||||
// Libc candidates.
|
||||
var libcs []string
|
||||
if libcStr != "" {
|
||||
libcs = []string{libcStr, "none", ""}
|
||||
} else {
|
||||
// No preference: try all common options.
|
||||
switch osStr {
|
||||
case "linux":
|
||||
// none first (static, no deps), then gnu, musl, empty.
|
||||
libcs = []string{"none", "gnu", "musl", ""}
|
||||
case "windows":
|
||||
// none first (no deps), msvc last (needs vcredist).
|
||||
libcs = []string{"none", "msvc", ""}
|
||||
default:
|
||||
libcs = []string{"none", ""}
|
||||
}
|
||||
}
|
||||
|
||||
var triplets []platformTriple
|
||||
for _, os := range oses {
|
||||
for _, arch := range archStrs {
|
||||
for _, libc := range libcs {
|
||||
triplets = append(triplets, platformTriple{
|
||||
os: os,
|
||||
arch: arch,
|
||||
libc: libc,
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
return triplets
|
||||
}
|
||||
|
||||
// pickBest selects the best asset from a set of candidates for the same
|
||||
// version and platform. Prefers the requested variant (or no-variant if
|
||||
// none requested), then picks by format preference.
|
||||
func pickBest(candidates []storage.Asset, formats []string, wantVariant string, ltsOnly bool) (storage.Asset, bool) {
|
||||
// Filter by LTS if requested.
|
||||
if ltsOnly {
|
||||
var lts []storage.Asset
|
||||
for _, a := range candidates {
|
||||
if a.LTS {
|
||||
lts = append(lts, a)
|
||||
}
|
||||
}
|
||||
if len(lts) == 0 {
|
||||
return storage.Asset{}, false
|
||||
}
|
||||
candidates = lts
|
||||
}
|
||||
|
||||
// Separate into variant-matched and non-variant pools.
|
||||
var preferred []storage.Asset
|
||||
var fallback []storage.Asset
|
||||
|
||||
for _, a := range candidates {
|
||||
if wantVariant != "" {
|
||||
// User requested a specific variant.
|
||||
if hasVariant(a.Variants, wantVariant) {
|
||||
preferred = append(preferred, a)
|
||||
} else if len(a.Variants) == 0 {
|
||||
fallback = append(fallback, a)
|
||||
}
|
||||
} else {
|
||||
// No variant requested: prefer no-variant assets.
|
||||
if len(a.Variants) == 0 {
|
||||
preferred = append(preferred, a)
|
||||
} else {
|
||||
fallback = append(fallback, a)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Try preferred pool first, then fallback.
|
||||
for _, pool := range [][]storage.Asset{preferred, fallback} {
|
||||
if len(pool) == 0 {
|
||||
continue
|
||||
}
|
||||
if best, ok := pickByFormat(pool, formats); ok {
|
||||
return best, true
|
||||
}
|
||||
}
|
||||
|
||||
return storage.Asset{}, false
|
||||
}
|
||||
|
||||
// pickByFormat selects the asset with the most preferred format.
|
||||
func pickByFormat(assets []storage.Asset, formats []string) (storage.Asset, bool) {
|
||||
for _, fmt := range formats {
|
||||
for _, a := range assets {
|
||||
if a.Format == fmt {
|
||||
return a, true
|
||||
}
|
||||
}
|
||||
}
|
||||
// No format match — return the first asset as last resort.
|
||||
if len(assets) > 0 {
|
||||
return assets[0], true
|
||||
}
|
||||
return storage.Asset{}, false
|
||||
}
|
||||
|
||||
func hasVariant(variants []string, want string) bool {
|
||||
for _, v := range variants {
|
||||
if v == want {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// defaultFormats returns the format preference order for an OS.
|
||||
// zst is preferred as the modern standard, but availability varies.
|
||||
func defaultFormats(os string) []string {
|
||||
switch os {
|
||||
case "windows":
|
||||
return []string{
|
||||
".tar.zst",
|
||||
".tar.xz",
|
||||
".zip",
|
||||
".tar.gz",
|
||||
".exe.xz",
|
||||
".7z",
|
||||
".exe",
|
||||
".msi",
|
||||
"git",
|
||||
}
|
||||
case "darwin":
|
||||
return []string{
|
||||
".tar.zst",
|
||||
".tar.xz",
|
||||
".zip",
|
||||
".tar.gz",
|
||||
".gz",
|
||||
".app.zip",
|
||||
".dmg",
|
||||
".pkg",
|
||||
"git",
|
||||
}
|
||||
default:
|
||||
// Linux and other POSIX.
|
||||
return []string{
|
||||
".tar.zst",
|
||||
".tar.xz",
|
||||
".tar.gz",
|
||||
".gz",
|
||||
".zip",
|
||||
".xz",
|
||||
"git",
|
||||
}
|
||||
}
|
||||
}
|
||||
290
internal/resolver/resolver_cache_test.go
Normal file
290
internal/resolver/resolver_cache_test.go
Normal file
@@ -0,0 +1,290 @@
|
||||
package resolver_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/resolver"
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
func loadAssets(t *testing.T, pkg string) []storage.Asset {
|
||||
t.Helper()
|
||||
cacheDir := filepath.Join("..", "..", "_cache", "2026-03")
|
||||
path := filepath.Join(cacheDir, pkg+".json")
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Skipf("no cache file for %s: %v", pkg, err)
|
||||
}
|
||||
var lc storage.LegacyCache
|
||||
if err := json.Unmarshal(data, &lc); err != nil {
|
||||
t.Fatalf("parse %s: %v", pkg, err)
|
||||
}
|
||||
pd := storage.ImportLegacy(lc)
|
||||
return pd.Assets
|
||||
}
|
||||
|
||||
// TestCacheResolveAllPackages loads every package from the cache and verifies
|
||||
// the resolver finds a match for each standard platform.
|
||||
func TestCacheResolveAllPackages(t *testing.T) {
|
||||
cacheDir := filepath.Join("..", "..", "_cache", "2026-03")
|
||||
entries, err := os.ReadDir(cacheDir)
|
||||
if err != nil {
|
||||
t.Skipf("no cache dir: %v", err)
|
||||
}
|
||||
|
||||
var pkgs []string
|
||||
for _, e := range entries {
|
||||
if strings.HasSuffix(e.Name(), ".json") {
|
||||
pkgs = append(pkgs, strings.TrimSuffix(e.Name(), ".json"))
|
||||
}
|
||||
}
|
||||
|
||||
if len(pkgs) < 50 {
|
||||
t.Fatalf("expected at least 50 packages, got %d", len(pkgs))
|
||||
}
|
||||
|
||||
platforms := []struct {
|
||||
name string
|
||||
os string
|
||||
arch string
|
||||
}{
|
||||
{"darwin-arm64", "darwin", "aarch64"},
|
||||
{"darwin-amd64", "darwin", "x86_64"},
|
||||
{"linux-amd64", "linux", "x86_64"},
|
||||
{"linux-arm64", "linux", "aarch64"},
|
||||
{"windows-amd64", "windows", "x86_64"},
|
||||
}
|
||||
|
||||
for _, pkg := range pkgs {
|
||||
t.Run(pkg, func(t *testing.T) {
|
||||
assets := loadAssets(t, pkg)
|
||||
if len(assets) == 0 {
|
||||
t.Skip("no releases")
|
||||
}
|
||||
|
||||
// Determine which OSes this package has.
|
||||
osSet := make(map[string]bool)
|
||||
for _, a := range assets {
|
||||
if a.OS != "" {
|
||||
osSet[a.OS] = true
|
||||
}
|
||||
}
|
||||
// Also check for platform-agnostic assets.
|
||||
hasAgnostic := false
|
||||
for _, a := range assets {
|
||||
if a.OS == "" {
|
||||
hasAgnostic = true
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
for _, plat := range platforms {
|
||||
supported := osSet[plat.os] ||
|
||||
osSet["ANYOS"] ||
|
||||
hasAgnostic ||
|
||||
(plat.os != "windows" && (osSet["posix_2017"] || osSet["posix_2024"]))
|
||||
|
||||
if !supported {
|
||||
continue
|
||||
}
|
||||
|
||||
t.Run(plat.name, func(t *testing.T) {
|
||||
res, err := resolver.Resolve(assets, resolver.Request{
|
||||
OS: plat.os,
|
||||
Arch: plat.arch,
|
||||
})
|
||||
if err != nil {
|
||||
// Not a test failure — some packages don't have
|
||||
// all arch builds. Log for visibility.
|
||||
t.Logf("WARN: no match for %s on %s (has OSes: %v)",
|
||||
pkg, plat.name, sortedOSes(osSet))
|
||||
return
|
||||
}
|
||||
if res.Version == "" {
|
||||
t.Error("matched but Version is empty")
|
||||
}
|
||||
if res.Asset.Download == "" {
|
||||
t.Error("matched but Download is empty")
|
||||
}
|
||||
})
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestCacheKnownPackages verifies specific packages resolve correctly.
|
||||
var knownPackages = []struct {
|
||||
pkg string
|
||||
version string // expected latest stable version prefix
|
||||
platforms []string
|
||||
}{
|
||||
{"bat", "0.26", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
{"caddy", "2.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
{"delta", "0.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
{"fd", "10.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
{"fzf", "0.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
{"gh", "2.", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
{"rg", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
{"node", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
{"terraform", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
{"zig", "", []string{"darwin-arm64", "darwin-amd64", "linux-amd64", "linux-arm64", "windows-amd64"}},
|
||||
}
|
||||
|
||||
func TestCacheKnownPackages(t *testing.T) {
|
||||
platMap := map[string]resolver.Request{
|
||||
"darwin-arm64": {OS: "darwin", Arch: "aarch64"},
|
||||
"darwin-amd64": {OS: "darwin", Arch: "x86_64"},
|
||||
"linux-amd64": {OS: "linux", Arch: "x86_64"},
|
||||
"linux-arm64": {OS: "linux", Arch: "aarch64"},
|
||||
"windows-amd64": {OS: "windows", Arch: "x86_64"},
|
||||
}
|
||||
|
||||
for _, kp := range knownPackages {
|
||||
t.Run(kp.pkg, func(t *testing.T) {
|
||||
assets := loadAssets(t, kp.pkg)
|
||||
|
||||
for _, platName := range kp.platforms {
|
||||
req := platMap[platName]
|
||||
t.Run(platName, func(t *testing.T) {
|
||||
res, err := resolver.Resolve(assets, req)
|
||||
if err != nil {
|
||||
t.Fatalf("no match for %s on %s", kp.pkg, platName)
|
||||
}
|
||||
if kp.version != "" {
|
||||
v := strings.TrimPrefix(res.Version, "v")
|
||||
if !strings.HasPrefix(v, kp.version) {
|
||||
t.Errorf("Version = %q, want prefix %q", res.Version, kp.version)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestCacheVersionConstraints tests version pinning with real data.
|
||||
func TestCacheVersionConstraints(t *testing.T) {
|
||||
tests := []struct {
|
||||
pkg string
|
||||
version string
|
||||
wantPfx string
|
||||
}{
|
||||
{"bat", "0.25", "0.25"},
|
||||
{"bat", "0.26", "0.26"},
|
||||
{"gh", "2.40", "2.40"},
|
||||
{"node", "20", "20."},
|
||||
{"node", "22", "22."},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.pkg+"@"+tt.version, func(t *testing.T) {
|
||||
assets := loadAssets(t, tt.pkg)
|
||||
res, err := resolver.Resolve(assets, resolver.Request{
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Version: tt.version,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("no match for %s@%s", tt.pkg, tt.version)
|
||||
}
|
||||
v := strings.TrimPrefix(res.Version, "v")
|
||||
if !strings.HasPrefix(v, tt.wantPfx) {
|
||||
t.Errorf("Version = %q, want prefix %q", res.Version, tt.wantPfx)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestCacheArchFallback verifies Rosetta-style fallback with real data.
|
||||
func TestCacheArchFallback(t *testing.T) {
|
||||
// awless only has amd64 builds — macOS ARM64 should fall back.
|
||||
assets := loadAssets(t, "awless")
|
||||
res, err := resolver.Resolve(assets, resolver.Request{
|
||||
OS: "darwin",
|
||||
Arch: "aarch64",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal("expected Rosetta 2 fallback for awless")
|
||||
}
|
||||
if res.Asset.Arch != "x86_64" {
|
||||
t.Errorf("Arch = %q, want x86_64", res.Asset.Arch)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCacheGitPackages verifies git-only packages resolve on any platform.
|
||||
func TestCacheGitPackages(t *testing.T) {
|
||||
gitPkgs := []string{"vim-essentials", "vim-spell"}
|
||||
for _, pkg := range gitPkgs {
|
||||
t.Run(pkg, func(t *testing.T) {
|
||||
assets := loadAssets(t, pkg)
|
||||
if len(assets) == 0 {
|
||||
t.Skip("no releases")
|
||||
}
|
||||
|
||||
// Should work on any platform.
|
||||
for _, plat := range []struct {
|
||||
os, arch string
|
||||
}{
|
||||
{"linux", "x86_64"},
|
||||
{"darwin", "aarch64"},
|
||||
{"windows", "x86_64"},
|
||||
} {
|
||||
res, err := resolver.Resolve(assets, resolver.Request{
|
||||
OS: plat.os,
|
||||
Arch: plat.arch,
|
||||
})
|
||||
if err != nil {
|
||||
t.Errorf("expected match on %s-%s", plat.os, plat.arch)
|
||||
continue
|
||||
}
|
||||
if res.Asset.Format != "git" {
|
||||
t.Errorf("format = %q, want git", res.Asset.Format)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestCacheLibcPreference tests explicit libc selection.
|
||||
// bat is Rust — its musl builds are static (tagged 'none').
|
||||
func TestCacheLibcPreference(t *testing.T) {
|
||||
assets := loadAssets(t, "bat")
|
||||
|
||||
// Musl host requesting bat: gets static musl build (tagged 'none').
|
||||
res, err := resolver.Resolve(assets, resolver.Request{
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Libc: "musl",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal("expected match for musl host")
|
||||
}
|
||||
if res.Asset.Libc != "none" {
|
||||
t.Errorf("Libc = %q, want none (static musl)", res.Asset.Libc)
|
||||
}
|
||||
|
||||
// Explicit gnu.
|
||||
res, err = resolver.Resolve(assets, resolver.Request{
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Libc: "gnu",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal("expected gnu match")
|
||||
}
|
||||
if res.Asset.Libc != "gnu" {
|
||||
t.Errorf("Libc = %q, want gnu", res.Asset.Libc)
|
||||
}
|
||||
}
|
||||
|
||||
func sortedOSes(m map[string]bool) []string {
|
||||
var keys []string
|
||||
for k := range m {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
return keys
|
||||
}
|
||||
397
internal/resolver/resolver_test.go
Normal file
397
internal/resolver/resolver_test.go
Normal file
@@ -0,0 +1,397 @@
|
||||
package resolver
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
func TestResolveSimple(t *testing.T) {
|
||||
assets := []storage.Asset{
|
||||
{
|
||||
Filename: "bat-v0.25.0-x86_64-unknown-linux-musl.tar.gz",
|
||||
Version: "0.25.0",
|
||||
Channel: "stable",
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Libc: "musl",
|
||||
Format: ".tar.gz",
|
||||
Download: "https://example.com/bat-0.25.0-linux-x86_64.tar.gz",
|
||||
},
|
||||
{
|
||||
Filename: "bat-v0.26.0-x86_64-unknown-linux-musl.tar.gz",
|
||||
Version: "0.26.0",
|
||||
Channel: "stable",
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Libc: "musl",
|
||||
Format: ".tar.gz",
|
||||
Download: "https://example.com/bat-0.26.0-linux-x86_64.tar.gz",
|
||||
},
|
||||
{
|
||||
Filename: "bat-v0.26.0-aarch64-unknown-linux-musl.tar.gz",
|
||||
Version: "0.26.0",
|
||||
Channel: "stable",
|
||||
OS: "linux",
|
||||
Arch: "aarch64",
|
||||
Libc: "musl",
|
||||
Format: ".tar.gz",
|
||||
Download: "https://example.com/bat-0.26.0-linux-aarch64.tar.gz",
|
||||
},
|
||||
{
|
||||
Filename: "bat-v0.26.0-x86_64-pc-windows-msvc.zip",
|
||||
Version: "0.26.0",
|
||||
Channel: "stable",
|
||||
OS: "windows",
|
||||
Arch: "x86_64",
|
||||
Libc: "msvc",
|
||||
Format: ".zip",
|
||||
Download: "https://example.com/bat-0.26.0-windows-x86_64.zip",
|
||||
},
|
||||
{
|
||||
Filename: "bat-v0.26.0-x86_64-apple-darwin.tar.gz",
|
||||
Version: "0.26.0",
|
||||
Channel: "stable",
|
||||
OS: "darwin",
|
||||
Arch: "x86_64",
|
||||
Format: ".tar.gz",
|
||||
Download: "https://example.com/bat-0.26.0-darwin-x86_64.tar.gz",
|
||||
},
|
||||
}
|
||||
|
||||
t.Run("latest linux x86_64", func(t *testing.T) {
|
||||
res, err := Resolve(assets, Request{
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if res.Version != "0.26.0" {
|
||||
t.Errorf("version = %q, want 0.26.0", res.Version)
|
||||
}
|
||||
if res.Asset.OS != "linux" {
|
||||
t.Errorf("os = %q, want linux", res.Asset.OS)
|
||||
}
|
||||
if res.Asset.Arch != "x86_64" {
|
||||
t.Errorf("arch = %q, want x86_64", res.Asset.Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("latest linux aarch64", func(t *testing.T) {
|
||||
res, err := Resolve(assets, Request{
|
||||
OS: "linux",
|
||||
Arch: "aarch64",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if res.Version != "0.26.0" {
|
||||
t.Errorf("version = %q, want 0.26.0", res.Version)
|
||||
}
|
||||
if res.Asset.Arch != "aarch64" {
|
||||
t.Errorf("arch = %q, want aarch64", res.Asset.Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("version prefix 0.25", func(t *testing.T) {
|
||||
res, err := Resolve(assets, Request{
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Version: "0.25",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if res.Version != "0.25.0" {
|
||||
t.Errorf("version = %q, want 0.25.0", res.Version)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("darwin arm64 falls back to x86_64", func(t *testing.T) {
|
||||
res, err := Resolve(assets, Request{
|
||||
OS: "darwin",
|
||||
Arch: "aarch64",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if res.Asset.Arch != "x86_64" {
|
||||
t.Errorf("arch = %q, want x86_64 (Rosetta fallback)", res.Asset.Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("no match returns error", func(t *testing.T) {
|
||||
_, err := Resolve(assets, Request{
|
||||
OS: "freebsd",
|
||||
Arch: "x86_64",
|
||||
})
|
||||
if err != ErrNoMatch {
|
||||
t.Errorf("err = %v, want ErrNoMatch", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("windows gets zip", func(t *testing.T) {
|
||||
res, err := Resolve(assets, Request{
|
||||
OS: "windows",
|
||||
Arch: "x86_64",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if res.Asset.Format != ".zip" {
|
||||
t.Errorf("format = %q, want .zip", res.Asset.Format)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestResolveChannels(t *testing.T) {
|
||||
assets := []storage.Asset{
|
||||
{
|
||||
Filename: "tool-v2.0.0-rc1-linux-x86_64.tar.gz",
|
||||
Version: "2.0.0-rc1",
|
||||
Channel: "rc",
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Format: ".tar.gz",
|
||||
},
|
||||
{
|
||||
Filename: "tool-v1.5.0-linux-x86_64.tar.gz",
|
||||
Version: "1.5.0",
|
||||
Channel: "stable",
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Format: ".tar.gz",
|
||||
},
|
||||
{
|
||||
Filename: "tool-v2.0.0-beta2-linux-x86_64.tar.gz",
|
||||
Version: "2.0.0-beta2",
|
||||
Channel: "beta",
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Format: ".tar.gz",
|
||||
},
|
||||
}
|
||||
|
||||
t.Run("stable skips rc and beta", func(t *testing.T) {
|
||||
res, err := Resolve(assets, Request{
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if res.Version != "1.5.0" {
|
||||
t.Errorf("version = %q, want 1.5.0", res.Version)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("rc includes rc and stable", func(t *testing.T) {
|
||||
res, err := Resolve(assets, Request{
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Channel: "rc",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if res.Version != "2.0.0-rc1" {
|
||||
t.Errorf("version = %q, want 2.0.0-rc1", res.Version)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("beta includes beta, rc, and stable", func(t *testing.T) {
|
||||
res, err := Resolve(assets, Request{
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Channel: "beta",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
// beta2 sorts after rc1 for the same numeric version (2.0.0),
|
||||
// but rc1 is more stable. However, the user asked for beta channel
|
||||
// which includes everything — and beta sorts before rc alphabetically.
|
||||
// With lexver: 2.0.0-rc1 > 2.0.0-beta2 (rc > beta alphabetically).
|
||||
if res.Version != "2.0.0-rc1" {
|
||||
t.Errorf("version = %q, want 2.0.0-rc1", res.Version)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestResolveVariants(t *testing.T) {
|
||||
assets := []storage.Asset{
|
||||
{
|
||||
Filename: "ollama-linux-amd64.tgz",
|
||||
Version: "0.6.0",
|
||||
Channel: "stable",
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Format: ".tar.gz",
|
||||
},
|
||||
{
|
||||
Filename: "ollama-linux-amd64-rocm.tgz",
|
||||
Version: "0.6.0",
|
||||
Channel: "stable",
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Format: ".tar.gz",
|
||||
Variants: []string{"rocm"},
|
||||
},
|
||||
}
|
||||
|
||||
t.Run("no variant prefers plain", func(t *testing.T) {
|
||||
res, err := Resolve(assets, Request{
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(res.Asset.Variants) != 0 {
|
||||
t.Errorf("variants = %v, want empty", res.Asset.Variants)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("explicit variant selects it", func(t *testing.T) {
|
||||
res, err := Resolve(assets, Request{
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Variant: "rocm",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !hasVariant(res.Asset.Variants, "rocm") {
|
||||
t.Errorf("variants = %v, want [rocm]", res.Asset.Variants)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestResolveFormatPreference(t *testing.T) {
|
||||
assets := []storage.Asset{
|
||||
{
|
||||
Filename: "tool-v1.0.0-linux-x86_64.tar.gz",
|
||||
Version: "1.0.0",
|
||||
Channel: "stable",
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Format: ".tar.gz",
|
||||
},
|
||||
{
|
||||
Filename: "tool-v1.0.0-linux-x86_64.tar.xz",
|
||||
Version: "1.0.0",
|
||||
Channel: "stable",
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Format: ".tar.xz",
|
||||
},
|
||||
{
|
||||
Filename: "tool-v1.0.0-linux-x86_64.tar.zst",
|
||||
Version: "1.0.0",
|
||||
Channel: "stable",
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Format: ".tar.zst",
|
||||
},
|
||||
}
|
||||
|
||||
t.Run("default prefers zst", func(t *testing.T) {
|
||||
res, err := Resolve(assets, Request{
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if res.Asset.Format != ".tar.zst" {
|
||||
t.Errorf("format = %q, want .tar.zst", res.Asset.Format)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("explicit format preference", func(t *testing.T) {
|
||||
res, err := Resolve(assets, Request{
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Formats: []string{".tar.gz"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if res.Asset.Format != ".tar.gz" {
|
||||
t.Errorf("format = %q, want .tar.gz", res.Asset.Format)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestResolveGitAssets(t *testing.T) {
|
||||
assets := []storage.Asset{
|
||||
{
|
||||
Filename: "vim-commentary-v1.2",
|
||||
Version: "1.2",
|
||||
Channel: "stable",
|
||||
Format: "git",
|
||||
Download: "https://github.com/tpope/vim-commentary.git",
|
||||
},
|
||||
{
|
||||
Filename: "vim-commentary-v1.1",
|
||||
Version: "1.1",
|
||||
Channel: "stable",
|
||||
Format: "git",
|
||||
Download: "https://github.com/tpope/vim-commentary.git",
|
||||
},
|
||||
}
|
||||
|
||||
t.Run("git assets match any platform", func(t *testing.T) {
|
||||
res, err := Resolve(assets, Request{
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if res.Version != "1.2" {
|
||||
t.Errorf("version = %q, want 1.2", res.Version)
|
||||
}
|
||||
if res.Asset.Format != "git" {
|
||||
t.Errorf("format = %q, want git", res.Asset.Format)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestResolveLTS(t *testing.T) {
|
||||
assets := []storage.Asset{
|
||||
{
|
||||
Filename: "node-v22.0.0-linux-x64.tar.gz",
|
||||
Version: "22.0.0",
|
||||
Channel: "stable",
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Format: ".tar.gz",
|
||||
LTS: false,
|
||||
},
|
||||
{
|
||||
Filename: "node-v20.15.0-linux-x64.tar.gz",
|
||||
Version: "20.15.0",
|
||||
Channel: "stable",
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Format: ".tar.gz",
|
||||
LTS: true,
|
||||
},
|
||||
}
|
||||
|
||||
t.Run("LTS selects older LTS version", func(t *testing.T) {
|
||||
res, err := Resolve(assets, Request{
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
LTS: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if res.Version != "20.15.0" {
|
||||
t.Errorf("version = %q, want 20.15.0", res.Version)
|
||||
}
|
||||
})
|
||||
}
|
||||
207
internal/storage/fsstore/fsstore.go
Normal file
207
internal/storage/fsstore/fsstore.go
Normal file
@@ -0,0 +1,207 @@
|
||||
// Package fsstore implements [storage.Store] on the local filesystem.
|
||||
//
|
||||
// Directory layout:
|
||||
//
|
||||
// {root}/
|
||||
// {package}.json # asset list
|
||||
// {package}.updated.txt # unix timestamp (seconds.millis)
|
||||
//
|
||||
// Write transactions build the new JSON in memory, then atomically
|
||||
// rename into place so readers never see a partial file.
|
||||
package fsstore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/lexver"
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// Store is a filesystem-backed asset store.
|
||||
type Store struct {
|
||||
root string
|
||||
}
|
||||
|
||||
// Root returns the store's root directory path.
|
||||
func (s *Store) Root() string {
|
||||
return s.root
|
||||
}
|
||||
|
||||
// New creates a Store rooted at the given directory.
|
||||
// The directory is created if it doesn't exist.
|
||||
func New(root string) (*Store, error) {
|
||||
if err := os.MkdirAll(root, 0o755); err != nil {
|
||||
return nil, fmt.Errorf("fsstore: create root: %w", err)
|
||||
}
|
||||
return &Store{root: root}, nil
|
||||
}
|
||||
|
||||
// ListPackages returns the names of all cached packages.
|
||||
func (s *Store) ListPackages(_ context.Context) ([]string, error) {
|
||||
dir := s.root
|
||||
entries, err := os.ReadDir(dir)
|
||||
if os.IsNotExist(err) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("fsstore: list packages: %w", err)
|
||||
}
|
||||
var pkgs []string
|
||||
for _, e := range entries {
|
||||
if strings.HasSuffix(e.Name(), ".json") {
|
||||
pkgs = append(pkgs, strings.TrimSuffix(e.Name(), ".json"))
|
||||
}
|
||||
}
|
||||
return pkgs, nil
|
||||
}
|
||||
|
||||
// Load reads a package's cached assets from disk.
|
||||
// Returns nil (not an error) if the package is not cached.
|
||||
func (s *Store) Load(_ context.Context, pkg string) (*storage.PackageData, error) {
|
||||
jsonPath := filepath.Join(s.root, pkg+".json")
|
||||
|
||||
data, err := os.ReadFile(jsonPath)
|
||||
if os.IsNotExist(err) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("fsstore: read %s: %w", pkg, err)
|
||||
}
|
||||
|
||||
// Decode via legacy format (Node.js compat: "releases", "name", "ext").
|
||||
var lc storage.LegacyCache
|
||||
if err := json.Unmarshal(data, &lc); err != nil {
|
||||
return nil, fmt.Errorf("fsstore: decode %s: %w", pkg, err)
|
||||
}
|
||||
pd := storage.ImportLegacy(lc)
|
||||
|
||||
// Read the timestamp file.
|
||||
tsPath := filepath.Join(s.root, pkg+".updated.txt")
|
||||
if tsData, err := os.ReadFile(tsPath); err == nil {
|
||||
pd.UpdatedAt = parseTimestamp(strings.TrimSpace(string(tsData)))
|
||||
}
|
||||
|
||||
return &pd, nil
|
||||
}
|
||||
|
||||
// BeginRefresh starts a write transaction for a package.
|
||||
func (s *Store) BeginRefresh(_ context.Context, pkg string) (storage.RefreshTx, error) {
|
||||
return &refreshTx{
|
||||
store: s,
|
||||
pkg: pkg,
|
||||
}, nil
|
||||
}
|
||||
|
||||
type refreshTx struct {
|
||||
store *Store
|
||||
pkg string
|
||||
assets []storage.Asset
|
||||
}
|
||||
|
||||
func (tx *refreshTx) Put(assets []storage.Asset) error {
|
||||
tx.assets = append(tx.assets, assets...)
|
||||
return nil
|
||||
}
|
||||
|
||||
func (tx *refreshTx) Commit(_ context.Context) error {
|
||||
now := time.Now()
|
||||
dir := tx.store.root
|
||||
|
||||
// Sort assets: stable/lts first, then beta, then rc, then alpha;
|
||||
// within each channel, newest version first.
|
||||
// The Node.js resolver picks the first matching entry, so stable[0] = latest stable
|
||||
// must come before beta of a higher version number.
|
||||
sort.SliceStable(tx.assets, func(i, j int) bool {
|
||||
ri, rj := channelRank(tx.assets[i].Channel), channelRank(tx.assets[j].Channel)
|
||||
if ri != rj {
|
||||
return ri < rj
|
||||
}
|
||||
return lexver.Compare(lexver.Parse(tx.assets[i].Version), lexver.Parse(tx.assets[j].Version)) > 0
|
||||
})
|
||||
|
||||
// Encode via legacy format (Node.js compat: "releases", "name", "ext").
|
||||
// ExportLegacy applies per-package field backports and drops assets that
|
||||
// can't be expressed in the legacy format (variants, unsupported formats).
|
||||
lc, drops := storage.ExportLegacy(tx.pkg, storage.PackageData{Assets: tx.assets})
|
||||
if drops.Variants > 0 || drops.Formats > 0 {
|
||||
log.Printf(" %s: legacy export dropped %d variant assets, %d unsupported-format assets",
|
||||
tx.pkg, drops.Variants, drops.Formats)
|
||||
}
|
||||
|
||||
data, err := json.MarshalIndent(lc, "", " ")
|
||||
if err != nil {
|
||||
return fmt.Errorf("fsstore: encode %s: %w", tx.pkg, err)
|
||||
}
|
||||
|
||||
// Write JSON atomically via temp file + rename.
|
||||
jsonPath := filepath.Join(dir, tx.pkg+".json")
|
||||
if err := atomicWrite(jsonPath, data); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Write timestamp file.
|
||||
tsPath := filepath.Join(dir, tx.pkg+".updated.txt")
|
||||
ts := fmt.Sprintf("%.3f", float64(now.UnixMilli())/1000.0)
|
||||
if err := atomicWrite(tsPath, []byte(ts)); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
tx.assets = nil
|
||||
return nil
|
||||
}
|
||||
|
||||
func (tx *refreshTx) Rollback() error {
|
||||
tx.assets = nil
|
||||
return nil
|
||||
}
|
||||
|
||||
// atomicWrite writes data to path via a temp file + rename.
|
||||
func atomicWrite(path string, data []byte) error {
|
||||
tmp := path + ".tmp"
|
||||
if err := os.WriteFile(tmp, data, 0o644); err != nil {
|
||||
return fmt.Errorf("fsstore: write tmp: %w", err)
|
||||
}
|
||||
if err := os.Rename(tmp, path); err != nil {
|
||||
os.Remove(tmp)
|
||||
return fmt.Errorf("fsstore: rename: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
|
||||
// channelRank returns a sort key for release channels so stable sorts first.
|
||||
// Lower rank = sorted earlier (stable/lts before beta/rc/alpha).
|
||||
func channelRank(channel string) int {
|
||||
switch channel {
|
||||
case "", "stable", "lts":
|
||||
return 0
|
||||
case "rc":
|
||||
return 1
|
||||
case "beta":
|
||||
return 2
|
||||
case "alpha":
|
||||
return 3
|
||||
default:
|
||||
return 4
|
||||
}
|
||||
}
|
||||
|
||||
// parseTimestamp parses the "seconds.millis" format from .updated.txt files.
|
||||
func parseTimestamp(s string) time.Time {
|
||||
f, err := strconv.ParseFloat(s, 64)
|
||||
if err != nil || f == 0 {
|
||||
return time.Time{}
|
||||
}
|
||||
sec := int64(f)
|
||||
nsec := int64((f - float64(sec)) * 1e9)
|
||||
return time.Unix(sec, nsec)
|
||||
}
|
||||
138
internal/storage/fsstore/fsstore_test.go
Normal file
138
internal/storage/fsstore/fsstore_test.go
Normal file
@@ -0,0 +1,138 @@
|
||||
package fsstore_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
"github.com/webinstall/webi-installers/internal/storage/fsstore"
|
||||
)
|
||||
|
||||
func TestRoundTrip(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
s, err := fsstore.New(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx := context.Background()
|
||||
|
||||
// Initially empty.
|
||||
pd, err := s.Load(ctx, "bat")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if pd != nil {
|
||||
t.Fatal("expected nil for uncached package")
|
||||
}
|
||||
|
||||
// Write some assets.
|
||||
tx, err := s.BeginRefresh(ctx, "bat")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tx.Put([]storage.Asset{
|
||||
{
|
||||
Filename: "bat-v0.26.1-aarch64-apple-darwin.tar.gz",
|
||||
Version: "0.26.1",
|
||||
Channel: "stable",
|
||||
Date: "2025-12-02",
|
||||
OS: "darwin",
|
||||
Arch: "aarch64",
|
||||
Format: ".tar.gz",
|
||||
Download: "https://github.com/sharkdp/bat/releases/download/v0.26.1/bat-v0.26.1-aarch64-apple-darwin.tar.gz",
|
||||
},
|
||||
{
|
||||
Filename: "bat-v0.26.1-x86_64-unknown-linux-gnu.tar.gz",
|
||||
Version: "0.26.1",
|
||||
Channel: "stable",
|
||||
Date: "2025-12-02",
|
||||
OS: "linux",
|
||||
Arch: "x86_64",
|
||||
Libc: "gnu",
|
||||
Format: ".tar.gz",
|
||||
Download: "https://github.com/sharkdp/bat/releases/download/v0.26.1/bat-v0.26.1-x86_64-unknown-linux-gnu.tar.gz",
|
||||
},
|
||||
})
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Read back.
|
||||
pd, err = s.Load(ctx, "bat")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if pd == nil {
|
||||
t.Fatal("expected data after write")
|
||||
}
|
||||
if len(pd.Assets) != 2 {
|
||||
t.Fatalf("got %d assets, want 2", len(pd.Assets))
|
||||
}
|
||||
if pd.Assets[0].Filename != "bat-v0.26.1-aarch64-apple-darwin.tar.gz" {
|
||||
t.Errorf("asset[0].Filename = %q", pd.Assets[0].Filename)
|
||||
}
|
||||
if pd.Assets[1].OS != "linux" {
|
||||
t.Errorf("asset[1].OS = %q", pd.Assets[1].OS)
|
||||
}
|
||||
if pd.UpdatedAt.IsZero() {
|
||||
t.Error("UpdatedAt should be set")
|
||||
}
|
||||
}
|
||||
|
||||
func TestRollback(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
s, err := fsstore.New(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx := context.Background()
|
||||
|
||||
tx, err := s.BeginRefresh(ctx, "bat")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
tx.Put([]storage.Asset{{Filename: "test", Version: "1.0"}})
|
||||
tx.Rollback()
|
||||
|
||||
pd, err := s.Load(ctx, "bat")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if pd != nil {
|
||||
t.Fatal("expected nil after rollback")
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadLegacyFormat(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
s, err := fsstore.New(dir)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx := context.Background()
|
||||
|
||||
// Write assets and read back — the JSON uses "releases" key
|
||||
// and "name"/"ext" field names for Node.js compat.
|
||||
tx, _ := s.BeginRefresh(ctx, "aliasman")
|
||||
tx.Put([]storage.Asset{
|
||||
{
|
||||
Filename: "BeyondCodeBootcamp-aliasman-v1.1.2-0-g0e5e1c1.tar.gz",
|
||||
Version: "v1.1.2",
|
||||
Channel: "stable",
|
||||
Date: "2023-02-23",
|
||||
OS: "posix_2017",
|
||||
Arch: "*",
|
||||
Format: "",
|
||||
Download: "https://codeload.github.com/BeyondCodeBootcamp/aliasman/legacy.tar.gz/refs/tags/v1.1.2",
|
||||
},
|
||||
})
|
||||
tx.Commit(ctx)
|
||||
|
||||
pd, err := s.Load(ctx, "aliasman")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if pd.Assets[0].OS != "posix_2017" {
|
||||
t.Errorf("OS = %q, want posix_2017", pd.Assets[0].OS)
|
||||
}
|
||||
}
|
||||
444
internal/storage/legacy.go
Normal file
444
internal/storage/legacy.go
Normal file
@@ -0,0 +1,444 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Legacy types for reading/writing the Node.js _cache/ JSON format.
|
||||
//
|
||||
// The Node.js server calls assets "releases" and uses "name" for the
|
||||
// filename and "ext" for the format. These types preserve that wire
|
||||
// format for backward compatibility during migration.
|
||||
//
|
||||
// Internal Go code uses [Asset] and [PackageData] directly.
|
||||
|
||||
// LegacyAsset matches the JSON shape the Node.js server writes and reads.
|
||||
type LegacyAsset struct {
|
||||
Name string `json:"name"`
|
||||
Version string `json:"version"`
|
||||
GitTag string `json:"git_tag,omitempty"`
|
||||
GitCommitHash string `json:"git_commit_hash,omitempty"`
|
||||
LTS bool `json:"lts"`
|
||||
Channel string `json:"channel"`
|
||||
Date string `json:"date"`
|
||||
OS string `json:"os"`
|
||||
Arch string `json:"arch"`
|
||||
Libc string `json:"libc"`
|
||||
Ext string `json:"ext"`
|
||||
Download string `json:"download"`
|
||||
}
|
||||
|
||||
// LegacyCache matches the top-level JSON shape in _cache/{pkg}.json.
|
||||
type LegacyCache struct {
|
||||
OSes []string `json:"oses,omitempty"`
|
||||
Arches []string `json:"arches,omitempty"`
|
||||
Libcs []string `json:"libcs,omitempty"`
|
||||
Formats []string `json:"formats,omitempty"`
|
||||
Releases []LegacyAsset `json:"releases"`
|
||||
Download string `json:"download"`
|
||||
}
|
||||
|
||||
// LegacyDropStats reports how many assets were excluded during ExportLegacy.
|
||||
type LegacyDropStats struct {
|
||||
Variants int // dropped: has build variant tags (e.g. rocm, installer, fxdependent)
|
||||
Formats int // dropped: format not recognized by the Node.js server
|
||||
Android int // dropped: android OS — classifier maps android filenames to linux
|
||||
NoTarget int // dropped: no OS and no arch — unclassifiable source tarballs
|
||||
}
|
||||
|
||||
// ToAsset converts a LegacyAsset to the internal Asset type.
|
||||
// It reverses the key vocabulary translations applied by toLegacy so that
|
||||
// the internal (Go canonical) representation is preserved.
|
||||
func (la LegacyAsset) ToAsset() Asset {
|
||||
// Reverse-translate legacy Node.js vocabulary to Go canonical names.
|
||||
// toLegacy writes macos/amd64/arm64; internal code uses darwin/x86_64/aarch64.
|
||||
// "none" libc is buildmeta.LibcNone — preserve it (don't collapse to "").
|
||||
os := la.OS
|
||||
switch os {
|
||||
case "macos":
|
||||
os = "darwin"
|
||||
case "*":
|
||||
os = ""
|
||||
}
|
||||
arch := la.Arch
|
||||
switch arch {
|
||||
case "amd64":
|
||||
arch = "x86_64"
|
||||
case "arm64":
|
||||
arch = "aarch64"
|
||||
case "*":
|
||||
arch = ""
|
||||
}
|
||||
// Restore the dot-prefix convention used throughout internal Go code.
|
||||
// The cache stores ext without a leading dot (e.g. "tar.gz", "zip", "exe"),
|
||||
// but Asset.Format uses dotted strings (e.g. ".tar.gz", ".zip", ".exe").
|
||||
// "exe" is ambiguous: bare binary (no .exe suffix) vs Windows .exe file.
|
||||
// Disambiguate by checking whether the filename ends with ".exe".
|
||||
format := la.Ext
|
||||
switch {
|
||||
case format == "exe" && !strings.HasSuffix(strings.ToLower(la.Name), ".exe"):
|
||||
format = "" // bare binary — internal convention is empty string
|
||||
case format != "":
|
||||
format = "." + format // restore dot prefix for internal use
|
||||
}
|
||||
return Asset{
|
||||
Filename: la.Name,
|
||||
Version: la.Version,
|
||||
LTS: la.LTS,
|
||||
Channel: la.Channel,
|
||||
Date: la.Date,
|
||||
OS: os,
|
||||
Arch: arch,
|
||||
Libc: la.Libc,
|
||||
Format: format,
|
||||
Download: la.Download,
|
||||
GitTag: la.GitTag,
|
||||
GitCommitHash: la.GitCommitHash,
|
||||
}
|
||||
}
|
||||
|
||||
// toLegacy converts an Asset to the LegacyAsset wire format.
|
||||
// Callers must have already applied legacyFieldBackport before calling this.
|
||||
func (a Asset) toLegacy() LegacyAsset {
|
||||
libc := a.Libc
|
||||
if libc == "" {
|
||||
libc = "none" // API expects "none" rather than empty string
|
||||
}
|
||||
// Strip leading dot: API expects "tar.gz" not ".tar.gz".
|
||||
ext := strings.TrimPrefix(a.Format, ".")
|
||||
// Bare binaries: API expects "exe". Internal convention is Format=""
|
||||
// for bare binaries (no archive extension). By the time we reach
|
||||
// toLegacy, source tarballs and git-clone entries have been filtered
|
||||
// or tagged, so Format="" reliably means bare binary.
|
||||
if ext == "" {
|
||||
ext = "exe"
|
||||
}
|
||||
return LegacyAsset{
|
||||
Name: a.Filename,
|
||||
Version: strings.TrimPrefix(a.Version, "v"), // API expects no v-prefix
|
||||
GitTag: a.GitTag,
|
||||
GitCommitHash: a.GitCommitHash,
|
||||
LTS: a.LTS,
|
||||
Channel: a.Channel,
|
||||
Date: a.Date,
|
||||
OS: a.OS,
|
||||
Arch: a.Arch,
|
||||
Libc: libc,
|
||||
Ext: ext,
|
||||
Download: a.Download,
|
||||
}
|
||||
}
|
||||
|
||||
// legacyFieldBackport translates canonical classifier field values to the
|
||||
// values the legacy Node.js resolver expects. This is called at export time
|
||||
// only — the canonical values are preserved in Go-native storage (pgstore).
|
||||
//
|
||||
// The Node build-classifier re-parses each asset's download filename and drops
|
||||
// any entry where the cache field doesn't match what it extracts from the name.
|
||||
// These translations ensure the cache matches the classifier's extraction.
|
||||
//
|
||||
// Global OS translations:
|
||||
// - sunos → solaris: Node's classifier maps "sunos" filenames to "solaris".
|
||||
// LIVE_cache has "solaris" and "illumos" but never "sunos".
|
||||
//
|
||||
// Global arch translations (all packages):
|
||||
// - universal2/universal1 → x86_64: classifier maps "universal" in filename
|
||||
// to x86_64. The darwin WATERFALL falls back aarch64→x86_64, so arm64
|
||||
// users still receive these builds.
|
||||
// - x86_64_v2/v3/v4 → x86_64: AMD64 microarch levels not in LIVE_cache;
|
||||
// fold to baseline x86_64.
|
||||
// - mips64r6 → mips64: exotic MIPS64R6, not in LIVE_cache.
|
||||
// - mips64r6el → mips64le: exotic MIPS64R6 little-endian, not in LIVE_cache.
|
||||
// - ARM (filename-based): explicit armvN takes priority over ABI tags.
|
||||
// Go normalizes these; see legacyARMArchFromFilename for filename extraction.
|
||||
// Final ARM vocab mapping to LIVE_cache values:
|
||||
// armv6→armv6l, armv7a→armv7l, armhf→armv7l, armel→arm.
|
||||
// - powerpc (32-bit): not in LIVE_cache; entry is dropped.
|
||||
//
|
||||
// Note: mipsle and mips64le are kept as-is — LIVE_cache uses these exact values.
|
||||
// Note: solaris and illumos are kept as-is — both exist in LIVE_cache.
|
||||
//
|
||||
// Package-specific rules replicate per-package overrides in production's releases.js:
|
||||
// - ffmpeg: Windows .gz → .exe (prod releases.js: rel.ext = 'exe')
|
||||
//
|
||||
// Git-clone entries:
|
||||
// - format="git" with empty OS/arch → os="*", arch="*"
|
||||
// The legacy cache uses "*" for ANYOS/ANYARCH (builds-cacher LEGACY_OS_MAP['*']='ANYOS').
|
||||
// vim plugins, aliasman, serviceman, and other POSIX packages use this format.
|
||||
func legacyFieldBackport(pkg string, a Asset) Asset {
|
||||
// Git-clone entries are ANYOS/ANYARCH — legacy cache uses "*" for these.
|
||||
// This matches production LIVE_cache for vim-commentary, aliasman, etc.
|
||||
if a.Format == "git" {
|
||||
if a.OS == "" {
|
||||
a.OS = "*"
|
||||
}
|
||||
if a.Arch == "" {
|
||||
a.Arch = "*"
|
||||
}
|
||||
}
|
||||
|
||||
// sunos → solaris: Node's classifier maps "sunos" filenames to "solaris".
|
||||
// LIVE_cache has "solaris" and "illumos" but never "sunos".
|
||||
if a.OS == "sunos" {
|
||||
a.OS = "solaris"
|
||||
}
|
||||
|
||||
// darwin → macos: LIVE_cache pre-classified packages (go, node, zig, fish, etc.)
|
||||
// use "macos". Julia is the sole exception — LIVE julia.json uses "darwin".
|
||||
if a.OS == "darwin" && pkg != "julia" {
|
||||
a.OS = "macos"
|
||||
}
|
||||
|
||||
// Universal fat binaries: expandUniversal splits these into per-arch
|
||||
// entries earlier in the pipeline. This is a safety fallback in case
|
||||
// any universal entries reach the legacy export unexpectedly.
|
||||
if a.Arch == "universal2" || a.Arch == "universal1" {
|
||||
a.Arch = "x86_64"
|
||||
}
|
||||
|
||||
// AMD64 microarch levels: not in LIVE_cache; fold to baseline x86_64.
|
||||
switch a.Arch {
|
||||
case "x86_64_v2", "x86_64_v3", "x86_64_v4":
|
||||
a.Arch = "x86_64"
|
||||
}
|
||||
|
||||
// x86_64 → amd64, aarch64 → arm64: LIVE_cache pre-classified packages use
|
||||
// "amd64" and "arm64". Go's classifier uses "x86_64" and "aarch64".
|
||||
// These come after universal2→x86_64 and x86_64_v*/→x86_64 so the chains work.
|
||||
if a.Arch == "x86_64" {
|
||||
a.Arch = "amd64"
|
||||
}
|
||||
if a.Arch == "aarch64" {
|
||||
a.Arch = "arm64"
|
||||
}
|
||||
|
||||
// MIPS variants not in LIVE_cache: fold to nearest supported value.
|
||||
// mipsle and mips64le are kept as-is — LIVE_cache uses these exact spellings.
|
||||
switch a.Arch {
|
||||
case "mips64r6":
|
||||
a.Arch = "mips64"
|
||||
case "mips64r6el":
|
||||
a.Arch = "mips64le"
|
||||
}
|
||||
|
||||
// powerpc (32-bit): not in LIVE_cache; mark for drop by clearing both fields.
|
||||
// Per-package taggers (uuidv7, watchexec) handle this via variant tags, but
|
||||
// for any package without a tagger, clear here so the NoTarget filter drops it.
|
||||
if a.Arch == "powerpc" {
|
||||
a.OS = ""
|
||||
a.Arch = ""
|
||||
}
|
||||
|
||||
// ARM arch: the Node classifier re-parses filenames and expects the cache
|
||||
// arch to match what it extracts. Go normalizes arch values; use filename
|
||||
// heuristics to match what Node would extract.
|
||||
switch a.Arch {
|
||||
case "armv5", "armv6", "armv7":
|
||||
if leg := legacyARMArchFromFilename(a.Filename); leg != "" {
|
||||
a.Arch = leg
|
||||
}
|
||||
}
|
||||
// Translate ARM arch values to LIVE_cache vocabulary.
|
||||
// legacyARMArchFromFilename can produce armhf/armel/armv7a which aren't
|
||||
// in LIVE_cache; also translate raw armv6/armv7 (when no filename override).
|
||||
switch a.Arch {
|
||||
case "armv6":
|
||||
a.Arch = "armv6l"
|
||||
case "armv7":
|
||||
a.Arch = "armv7l"
|
||||
case "armhf":
|
||||
a.Arch = "armv7l"
|
||||
case "armel":
|
||||
a.Arch = "arm"
|
||||
case "armv7a":
|
||||
a.Arch = "armv7l"
|
||||
}
|
||||
|
||||
switch pkg {
|
||||
case "ffmpeg":
|
||||
if a.OS == "windows" {
|
||||
switch a.Format {
|
||||
case ".gz", "":
|
||||
a.Format = ".exe"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return a
|
||||
}
|
||||
|
||||
// legacyARMArchFromFilename returns the arch string the Node build-classifier
|
||||
// would extract from a filename for ARM-family builds. Returns "" when the
|
||||
// Go canonical arch value already matches what the classifier would extract.
|
||||
//
|
||||
// The Node classifier's extraction rules differ from Go's normalization:
|
||||
// - armv7a (explicit) → "armv7a" (not "armv7")
|
||||
// - armv7 (explicit, e.g. "armv7-unknown-linux-gnueabihf") → "armv7"
|
||||
// The explicit version number takes priority over the ABI suffix.
|
||||
// - arm-5 / arm-7 (Gitea naming: "linux-arm-5", "linux-arm-7") → "armel" / "armv7"
|
||||
// patternToTerms converts "arm-5" → "armv5" and "arm-7" → "armv7".
|
||||
// - armv6hf (shellcheck naming) → "armhf" (tpm['armv6hf'] = ARMHF)
|
||||
// - gnueabihf (Rust triplet, no explicit armvN) → "armhf"
|
||||
// - armhf (Debian armhf) → "armhf"
|
||||
// - armel (Debian soft-float ABI) → "armel" (not "armv6")
|
||||
// - armv5 (explicit) → "armel" (Node tiered map: armv5 falls back to armel)
|
||||
func legacyARMArchFromFilename(filename string) string {
|
||||
lower := strings.ToLower(filename)
|
||||
// armv7a before armv7 — "armv7a" contains "armv7" as a prefix.
|
||||
if strings.Contains(lower, "armv7a") {
|
||||
return "armv7a"
|
||||
}
|
||||
// Explicit armv7 in filename: takes priority over ABI suffix (gnueabihf).
|
||||
// e.g. "armv7-unknown-linux-gnueabihf" → classifier extracts "armv7".
|
||||
if strings.Contains(lower, "armv7") {
|
||||
return "armv7"
|
||||
}
|
||||
// armv6hf (shellcheck naming): tpm['armv6hf'] = ARMHF → "armhf".
|
||||
if strings.Contains(lower, "armv6hf") {
|
||||
return "armhf"
|
||||
}
|
||||
// Gitea arm-N naming: "linux-arm-5" → patternToTerms → "armv5" → armel.
|
||||
if strings.Contains(lower, "arm-5") {
|
||||
return "armel"
|
||||
}
|
||||
// Gitea arm-N naming: "linux-arm-7" → patternToTerms → "armv7" → armv7.
|
||||
if strings.Contains(lower, "arm-7") {
|
||||
return "armv7"
|
||||
}
|
||||
// Rust gnueabihf triplet (no explicit armvN): classifier → "armhf".
|
||||
if strings.Contains(lower, "gnueabihf") {
|
||||
return "armhf"
|
||||
}
|
||||
// Debian armhf (hard-float ABI): classifier → "armhf".
|
||||
if strings.Contains(lower, "armhf") {
|
||||
return "armhf"
|
||||
}
|
||||
if strings.Contains(lower, "armel") {
|
||||
return "armel"
|
||||
}
|
||||
if strings.Contains(lower, "armv5") {
|
||||
return "armel"
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// ImportLegacy converts a LegacyCache to PackageData.
|
||||
func ImportLegacy(lc LegacyCache) PackageData {
|
||||
assets := make([]Asset, len(lc.Releases))
|
||||
for i, la := range lc.Releases {
|
||||
assets[i] = la.ToAsset()
|
||||
}
|
||||
return PackageData{Assets: assets}
|
||||
}
|
||||
|
||||
// legacyFormats is the set of formats the Node.js server recognizes.
|
||||
// Assets with formats not in this set are filtered out of legacy exports.
|
||||
var legacyFormats = map[string]bool{
|
||||
".zip": true,
|
||||
".tar.gz": true,
|
||||
".tar.xz": true,
|
||||
".tar.zst": true,
|
||||
".tar.bz2": true,
|
||||
".tar": true,
|
||||
".xz": true,
|
||||
".7z": true,
|
||||
".pkg": true,
|
||||
".msi": true,
|
||||
".exe": true,
|
||||
".exe.xz": true,
|
||||
".dmg": true,
|
||||
".app.zip": true,
|
||||
".gz": true,
|
||||
"git": true,
|
||||
}
|
||||
|
||||
// ExportLegacy converts canonical PackageData to the LegacyCache wire format.
|
||||
//
|
||||
// The pkg name is used to apply per-package field translations (see legacyFieldBackport).
|
||||
// Assets are excluded when:
|
||||
// - Variants is non-empty (Node.js has no variant logic)
|
||||
// - OS is android (classifier maps android filenames to linux)
|
||||
// - OS and arch are both empty (unclassifiable source tarballs)
|
||||
// - Format is non-empty and not in the Node.js recognized set
|
||||
//
|
||||
// Dropped counts are returned in LegacyDropStats for logging.
|
||||
func ExportLegacy(pkg string, pd PackageData) (LegacyCache, LegacyDropStats) {
|
||||
var releases []LegacyAsset
|
||||
var stats LegacyDropStats
|
||||
|
||||
for _, a := range pd.Assets {
|
||||
// Skip variant builds — Node.js doesn't have variant logic.
|
||||
if len(a.Variants) > 0 {
|
||||
stats.Variants++
|
||||
continue
|
||||
}
|
||||
// Skip android — classifier maps android filenames to linux OS,
|
||||
// which mismatches cache entries tagged android.
|
||||
if a.OS == "android" {
|
||||
stats.Android++
|
||||
continue
|
||||
}
|
||||
// Skip entries with no OS and no arch, unless they're git-clone packages.
|
||||
// Source tarballs (cmake, dashcore, bun npm) have format != "git".
|
||||
// Git-clone packages (vim plugins, aliasman) legitimately have no OS/arch —
|
||||
// legacyFieldBackport will translate them to os="*", arch="*".
|
||||
if a.OS == "" && a.Arch == "" && a.Format != "git" {
|
||||
stats.NoTarget++
|
||||
continue
|
||||
}
|
||||
// Apply per-package and global legacy field translations.
|
||||
a = legacyFieldBackport(pkg, a)
|
||||
// Skip formats Node.js doesn't recognize.
|
||||
if a.Format != "" && !legacyFormats[a.Format] {
|
||||
stats.Formats++
|
||||
continue
|
||||
}
|
||||
releases = append(releases, a.toLegacy())
|
||||
}
|
||||
if releases == nil {
|
||||
releases = []LegacyAsset{}
|
||||
}
|
||||
|
||||
// Build sorted summary arrays from the included releases.
|
||||
// These let the API skip normalize.js vocabulary filtering entirely.
|
||||
oSet := map[string]bool{}
|
||||
aSet := map[string]bool{}
|
||||
lSet := map[string]bool{}
|
||||
fSet := map[string]bool{}
|
||||
for _, r := range releases {
|
||||
if r.OS != "" && r.OS != "*" {
|
||||
oSet[r.OS] = true
|
||||
}
|
||||
if r.Arch != "" && r.Arch != "*" {
|
||||
aSet[r.Arch] = true
|
||||
}
|
||||
if r.Libc != "" {
|
||||
lSet[r.Libc] = true
|
||||
}
|
||||
if r.Ext != "" {
|
||||
fSet[strings.TrimPrefix(r.Ext, ".")] = true
|
||||
}
|
||||
}
|
||||
lc := LegacyCache{
|
||||
OSes: sortedKeys(oSet),
|
||||
Arches: sortedKeys(aSet),
|
||||
Libcs: sortedKeys(lSet),
|
||||
Formats: sortedKeys(fSet),
|
||||
Releases: releases,
|
||||
}
|
||||
return lc, stats
|
||||
}
|
||||
|
||||
// sortedKeys returns the keys of a string set in sorted order.
|
||||
func sortedKeys(m map[string]bool) []string {
|
||||
if len(m) == 0 {
|
||||
return nil
|
||||
}
|
||||
out := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
out = append(out, k)
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
609
internal/storage/legacy_test.go
Normal file
609
internal/storage/legacy_test.go
Normal file
@@ -0,0 +1,609 @@
|
||||
package storage_test
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// TestDecodeLegacyJSON verifies we can parse the exact JSON format
|
||||
// the Node.js server writes to _cache/.
|
||||
func TestDecodeLegacyJSON(t *testing.T) {
|
||||
// Real data from _cache/2026-03/aliasman.json.
|
||||
raw := `{
|
||||
"releases": [
|
||||
{
|
||||
"name": "BeyondCodeBootcamp-aliasman-v1.1.2-0-g0e5e1c1.tar.gz",
|
||||
"version": "v1.1.2",
|
||||
"lts": false,
|
||||
"channel": "stable",
|
||||
"date": "2023-02-23",
|
||||
"os": "posix_2017",
|
||||
"arch": "*",
|
||||
"libc": "",
|
||||
"ext": "",
|
||||
"download": "https://codeload.github.com/BeyondCodeBootcamp/aliasman/legacy.tar.gz/refs/tags/v1.1.2"
|
||||
},
|
||||
{
|
||||
"name": "BeyondCodeBootcamp-aliasman-v1.1.2-0-g0e5e1c1.zip",
|
||||
"version": "v1.1.2",
|
||||
"lts": false,
|
||||
"channel": "stable",
|
||||
"date": "2023-02-23",
|
||||
"os": "posix_2017",
|
||||
"arch": "*",
|
||||
"libc": "",
|
||||
"ext": "",
|
||||
"download": "https://codeload.github.com/BeyondCodeBootcamp/aliasman/legacy.zip/refs/tags/v1.1.2"
|
||||
}
|
||||
],
|
||||
"download": ""
|
||||
}`
|
||||
|
||||
var lc storage.LegacyCache
|
||||
if err := json.Unmarshal([]byte(raw), &lc); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if len(lc.Releases) != 2 {
|
||||
t.Fatalf("got %d releases, want 2", len(lc.Releases))
|
||||
}
|
||||
|
||||
pd := storage.ImportLegacy(lc)
|
||||
if len(pd.Assets) != 2 {
|
||||
t.Fatalf("got %d assets, want 2", len(pd.Assets))
|
||||
}
|
||||
|
||||
a := pd.Assets[0]
|
||||
if a.Filename != "BeyondCodeBootcamp-aliasman-v1.1.2-0-g0e5e1c1.tar.gz" {
|
||||
t.Errorf("Filename = %q", a.Filename)
|
||||
}
|
||||
if a.Version != "v1.1.2" {
|
||||
t.Errorf("Version = %q", a.Version)
|
||||
}
|
||||
if a.OS != "posix_2017" {
|
||||
t.Errorf("OS = %q", a.OS)
|
||||
}
|
||||
if a.Arch != "" {
|
||||
t.Errorf("Arch = %q, want %q (wildcard '*' reversed to empty)", a.Arch, "")
|
||||
}
|
||||
if a.Download != "https://codeload.github.com/BeyondCodeBootcamp/aliasman/legacy.tar.gz/refs/tags/v1.1.2" {
|
||||
t.Errorf("Download = %q", a.Download)
|
||||
}
|
||||
|
||||
// Round-trip: export back to legacy and verify JSON shape.
|
||||
lc2, _ := storage.ExportLegacy("aliasman", pd)
|
||||
data, _ := json.MarshalIndent(lc2, "", " ")
|
||||
var lc3 storage.LegacyCache
|
||||
json.Unmarshal(data, &lc3)
|
||||
|
||||
if lc3.Releases[0].Name != a.Filename {
|
||||
t.Errorf("round-trip Name = %q, want %q", lc3.Releases[0].Name, a.Filename)
|
||||
}
|
||||
// Legacy data has ext:"" for this tarball — broken cache entry.
|
||||
// toLegacy normalizes Format="" to ext:"exe" (bare binary convention).
|
||||
// In the real Go pipeline, aliasman would have Format=".tar.gz".
|
||||
if lc3.Releases[0].Ext != "exe" {
|
||||
t.Errorf("round-trip Ext = %q, want %q", lc3.Releases[0].Ext, "exe")
|
||||
}
|
||||
}
|
||||
|
||||
// TestExportLegacyDrops verifies that ExportLegacy correctly drops and counts
|
||||
// assets that can't be represented in the Node.js legacy cache format.
|
||||
func TestExportLegacyDrops(t *testing.T) {
|
||||
t.Run("variant_builds_dropped", func(t *testing.T) {
|
||||
// Assets with variant tags (rocm, installer, fxdependent, etc.) are
|
||||
// dropped because Node.js has no variant-selection logic.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "ollama-linux-amd64-rocm.tgz", OS: "linux", Arch: "x86_64", Format: ".tar.gz", Variants: []string{"rocm"}},
|
||||
{Filename: "ollama-linux-amd64.tgz", OS: "linux", Arch: "x86_64", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, stats := storage.ExportLegacy("ollama", pd)
|
||||
if stats.Variants != 1 {
|
||||
t.Errorf("Variants dropped = %d, want 1", stats.Variants)
|
||||
}
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Errorf("releases = %d, want 1 (baseline only)", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Name != "ollama-linux-amd64.tgz" {
|
||||
t.Errorf("kept wrong release: %q", lc.Releases[0].Name)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("android_dropped", func(t *testing.T) {
|
||||
// Android entries are dropped: the classifier maps android filenames to
|
||||
// linux OS and then rejects the cache entry that says android.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "fzf-0.57.0-android-arm64.tar.gz", OS: "android", Arch: "aarch64", Format: ".tar.gz"},
|
||||
{Filename: "fzf-0.57.0-linux-arm64.tar.gz", OS: "linux", Arch: "aarch64", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, stats := storage.ExportLegacy("fzf", pd)
|
||||
if stats.Android != 1 {
|
||||
t.Errorf("Android dropped = %d, want 1", stats.Android)
|
||||
}
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Errorf("releases = %d, want 1 (linux only)", len(lc.Releases))
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("unknown_formats_dropped", func(t *testing.T) {
|
||||
// .AppImage, .deb, .rpm are not in the Node.js format set.
|
||||
// Assets have Arch set (matching real classifier output for these formats).
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "tool.AppImage", OS: "linux", Arch: "x86_64", Format: ".AppImage"},
|
||||
{Filename: "tool.deb", OS: "linux", Arch: "x86_64", Format: ".deb"},
|
||||
{Filename: "tool.rpm", OS: "linux", Arch: "x86_64", Format: ".rpm"},
|
||||
{Filename: "tool-linux-amd64.tar.gz", OS: "linux", Arch: "x86_64", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, stats := storage.ExportLegacy("tool", pd)
|
||||
if stats.Formats != 3 {
|
||||
t.Errorf("Formats dropped = %d, want 3", stats.Formats)
|
||||
}
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Errorf("releases = %d, want 1 (tar.gz only)", len(lc.Releases))
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("empty_format_passes_through", func(t *testing.T) {
|
||||
// Assets with empty format (e.g. bare binaries, git sources) pass through.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "jq-linux-amd64", OS: "linux", Arch: "x86_64", Format: ""},
|
||||
},
|
||||
}
|
||||
lc, stats := storage.ExportLegacy("jq", pd)
|
||||
if stats.Formats != 0 {
|
||||
t.Errorf("Formats dropped = %d, want 0", stats.Formats)
|
||||
}
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Errorf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestExportLegacyTranslations verifies that legacyFieldBackport applies the
|
||||
// correct field translations for Node.js compatibility.
|
||||
func TestExportLegacyTranslations(t *testing.T) {
|
||||
t.Run("universal2_translated_to_amd64", func(t *testing.T) {
|
||||
// universal2 fat binaries: the Node classifier sees "universal" in the
|
||||
// filename and maps it to x86_64. Cache must say amd64 (via universal2→x86_64→amd64
|
||||
// chain) to match. The darwin WATERFALL (arm64 → [arm64, amd64]) means arm64
|
||||
// users also receive these builds as a fallback.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "hugo_0.145.0_darwin-universal.tar.gz", OS: "darwin", Arch: "universal2", Format: ".tar.gz"},
|
||||
{Filename: "hugo_0.145.0_darwin-arm64.tar.gz", OS: "darwin", Arch: "aarch64", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, stats := storage.ExportLegacy("hugo", pd)
|
||||
if stats.Variants != 0 || stats.Formats != 0 || stats.Android != 0 {
|
||||
t.Errorf("unexpected drops: %+v", stats)
|
||||
}
|
||||
if len(lc.Releases) != 2 {
|
||||
t.Fatalf("releases = %d, want 2", len(lc.Releases))
|
||||
}
|
||||
var universal2Arch string
|
||||
for _, r := range lc.Releases {
|
||||
if r.Name == "hugo_0.145.0_darwin-universal.tar.gz" {
|
||||
universal2Arch = r.Arch
|
||||
}
|
||||
}
|
||||
if universal2Arch != "amd64" {
|
||||
t.Errorf("universal2 arch in legacy = %q, want amd64 (universal2→x86_64→amd64)", universal2Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("solaris_kept_as_is", func(t *testing.T) {
|
||||
// Solaris/illumos/sunos are kept as-is. The build-classifier (triplet.js)
|
||||
// recognizes all three as distinct values and matches them correctly.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "go1.20.1.solaris-amd64.tar.gz", OS: "solaris", Arch: "x86_64", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, stats := storage.ExportLegacy("go", pd)
|
||||
if stats.Android != 0 || stats.Variants != 0 || stats.Formats != 0 {
|
||||
t.Errorf("unexpected drops: %+v", stats)
|
||||
}
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].OS != "solaris" {
|
||||
t.Errorf("OS = %q, want solaris", lc.Releases[0].OS)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("illumos_kept_as_is", func(t *testing.T) {
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "go1.20.1.illumos-amd64.tar.gz", OS: "illumos", Arch: "x86_64", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("go", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].OS != "illumos" {
|
||||
t.Errorf("OS = %q, want illumos", lc.Releases[0].OS)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("darwin_to_macos", func(t *testing.T) {
|
||||
// All packages except julia translate darwin → macos.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "go1.20.1.darwin-amd64.tar.gz", OS: "darwin", Arch: "aarch64", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("go", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].OS != "macos" {
|
||||
t.Errorf("OS = %q, want macos (darwin → macos)", lc.Releases[0].OS)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("julia_darwin_kept_as_is", func(t *testing.T) {
|
||||
// julia is the sole exception: LIVE julia.json uses "darwin", not "macos".
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "julia-1.9.3-mac64.tar.gz", OS: "darwin", Arch: "aarch64", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("julia", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].OS != "darwin" {
|
||||
t.Errorf("OS = %q, want darwin (julia exception — LIVE uses darwin)", lc.Releases[0].OS)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("x86_64_v2_to_amd64", func(t *testing.T) {
|
||||
// Micro-arch levels (v2/v3/v4): fold to baseline x86_64, then x86_64→amd64.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "tool-linux-x86_64_v2.tar.gz", OS: "linux", Arch: "x86_64_v2", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("tool", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Arch != "amd64" {
|
||||
t.Errorf("arch = %q, want amd64 (x86_64_v2 → x86_64 → amd64)", lc.Releases[0].Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("mips64r6_folded", func(t *testing.T) {
|
||||
// mips64r6/mips64r6el: exotic variants not in LIVE_cache; fold to mips64/mips64le.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "tool-linux-mips64r6.tar.gz", OS: "linux", Arch: "mips64r6", Format: ".tar.gz"},
|
||||
{Filename: "tool-linux-mips64r6el.tar.gz", OS: "linux", Arch: "mips64r6el", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("tool", pd)
|
||||
if len(lc.Releases) != 2 {
|
||||
t.Fatalf("releases = %d, want 2", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Arch != "mips64" {
|
||||
t.Errorf("arch = %q, want mips64 (mips64r6 → mips64)", lc.Releases[0].Arch)
|
||||
}
|
||||
if lc.Releases[1].Arch != "mips64le" {
|
||||
t.Errorf("arch = %q, want mips64le (mips64r6el → mips64le)", lc.Releases[1].Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("mipsle_unchanged", func(t *testing.T) {
|
||||
// mipsle: LIVE_cache uses "mipsle" — keep as-is.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "caddy_linux_mipsle.tar.gz", OS: "linux", Arch: "mipsle", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("caddy", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Arch != "mipsle" {
|
||||
t.Errorf("arch = %q, want mipsle (LIVE_cache uses mipsle)", lc.Releases[0].Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("mips64le_unchanged", func(t *testing.T) {
|
||||
// mips64le: LIVE_cache uses "mips64le" — keep as-is.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "gitea-linux-mips64le.tar.gz", OS: "linux", Arch: "mips64le", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("gitea", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Arch != "mips64le" {
|
||||
t.Errorf("arch = %q, want mips64le (LIVE_cache uses mips64le)", lc.Releases[0].Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("ffmpeg_windows_gz_to_exe", func(t *testing.T) {
|
||||
// ffmpeg Windows releases are .gz archives containing a bare .exe.
|
||||
// Production releases.js overrides ext to 'exe' for install compatibility.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "ffmpeg-7.0-windows-amd64.gz", OS: "windows", Arch: "x86_64", Format: ".gz"},
|
||||
{Filename: "ffmpeg-7.0-linux-amd64.tar.gz", OS: "linux", Arch: "x86_64", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("ffmpeg", pd)
|
||||
if len(lc.Releases) != 2 {
|
||||
t.Fatalf("releases = %d, want 2", len(lc.Releases))
|
||||
}
|
||||
var windowsExt string
|
||||
for _, r := range lc.Releases {
|
||||
if r.OS == "windows" {
|
||||
windowsExt = r.Ext
|
||||
}
|
||||
}
|
||||
if windowsExt != "exe" {
|
||||
t.Errorf("ffmpeg windows ext = %q, want exe", windowsExt)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("ffmpeg_translation_not_applied_to_other_packages", func(t *testing.T) {
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "othertool-windows-amd64.gz", OS: "windows", Arch: "x86_64", Format: ".gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("othertool", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Ext != "gz" {
|
||||
t.Errorf("ext = %q, want gz (no translation outside ffmpeg)", lc.Releases[0].Ext)
|
||||
}
|
||||
})
|
||||
|
||||
// ARM arch translations: translate Go-canonical values to LIVE_cache vocabulary.
|
||||
// LIVE_cache uses: armv6l, armv7l, armv7, arm (not armv6, armhf, armel, armv7a).
|
||||
t.Run("arm_gnueabihf_to_armv7l", func(t *testing.T) {
|
||||
// gnueabihf ABI suffix (no explicit armvN): filename → armhf → armv7l
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "bat-v0.9.0-arm-unknown-linux-gnueabihf.tar.gz", OS: "linux", Arch: "armv6", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("bat", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Arch != "armv7l" {
|
||||
t.Errorf("arch = %q, want armv7l (gnueabihf → armhf → armv7l)", lc.Releases[0].Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("arm_armhf_to_armv7l", func(t *testing.T) {
|
||||
// Debian armhf = ARMv7 hard-float; LIVE_cache uses armv7l for this.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "caddy_linux_armhf.tar.gz", OS: "linux", Arch: "armv7", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("caddy", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Arch != "armv7l" {
|
||||
t.Errorf("arch = %q, want armv7l (armhf → armv7l)", lc.Releases[0].Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("arm_armel_to_arm", func(t *testing.T) {
|
||||
// Debian armel = ARM soft-float; LIVE_cache uses "arm" for this.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "caddy_linux_armel.tar.gz", OS: "linux", Arch: "armv6", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("caddy", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Arch != "arm" {
|
||||
t.Errorf("arch = %q, want arm (armel → arm)", lc.Releases[0].Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("arm_armv5_to_arm", func(t *testing.T) {
|
||||
// armv5 → legacyARMArchFromFilename → "armel" → "arm"
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "caddy_linux_armv5.tar.gz", OS: "linux", Arch: "armv5", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("caddy", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Arch != "arm" {
|
||||
t.Errorf("arch = %q, want arm (armv5 → armel → arm)", lc.Releases[0].Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("arm_armv7a_to_armv7l", func(t *testing.T) {
|
||||
// armv7a (ARM application profile): LIVE_cache uses armv7l.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "tool-armv7a-linux.tar.gz", OS: "linux", Arch: "armv7", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("tool", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Arch != "armv7l" {
|
||||
t.Errorf("arch = %q, want armv7l (armv7a → armv7l)", lc.Releases[0].Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("arm_armv7l_filename_to_armv7l", func(t *testing.T) {
|
||||
// armv7l in filename: legacyARMArchFromFilename extracts "armv7" (armv7l contains armv7),
|
||||
// then the canonical armv7→armv7l translation maps it to armv7l (the correct API vocab).
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "tool-armv7l-linux.tar.gz", OS: "linux", Arch: "armv7", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("tool", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Arch != "armv7l" {
|
||||
t.Errorf("arch = %q, want armv7l (armv7l filename → armv7 → armv7l)", lc.Releases[0].Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("arm_armv6l_to_armv6l", func(t *testing.T) {
|
||||
// armv6l in filename: legacyARMArchFromFilename returns "" (no armv7/armhf/etc match).
|
||||
// armv6 (Go canonical) → armv6l (LIVE_cache vocabulary).
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "tool-armv6l-linux.tar.gz", OS: "linux", Arch: "armv6", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("tool", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Arch != "armv6l" {
|
||||
t.Errorf("arch = %q, want armv6l (armv6 → armv6l)", lc.Releases[0].Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("arm_armv7_gnueabihf_to_armv7l", func(t *testing.T) {
|
||||
// Files like "ripgrep-14.1.0-armv7-unknown-linux-gnueabihf.tar.gz":
|
||||
// Go classifies as armv7; the "armv7" term in filename takes priority
|
||||
// over the gnueabihf ABI suffix. legacyARMArchFromFilename returns "armv7",
|
||||
// then the canonical armv7→armv7l translation produces armv7l.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "ripgrep-14.1.0-armv7-unknown-linux-gnueabihf.tar.gz", OS: "linux", Arch: "armv7", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("ripgrep", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Arch != "armv7l" {
|
||||
t.Errorf("arch = %q, want armv7l (armv7 in filename → armv7 → armv7l)", lc.Releases[0].Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("arm_armv6hf_to_armhf", func(t *testing.T) {
|
||||
// shellcheck uses "armv6hf" naming; classifier tpm['armv6hf'] = ARMHF → "armhf".
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "shellcheck-v0.9.0.linux.armv6hf.tar.xz", OS: "linux", Arch: "armv6", Format: ".tar.xz"},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("shellcheck", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Arch != "armv7l" {
|
||||
t.Errorf("arch = %q, want armv7l (armv6hf → armhf → armv7l)", lc.Releases[0].Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("arm_gitea_arm5_to_armel", func(t *testing.T) {
|
||||
// Gitea uses "arm-5" naming; patternToTerms converts to "armv5" → tpm → "armel".
|
||||
// Go sees \barm\b → classifies as armv6. Legacy export must correct to armel.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "gitea-1.20.0-linux-arm-5", OS: "linux", Arch: "armv6", Format: ""},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("gitea", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Arch != "arm" {
|
||||
t.Errorf("arch = %q, want arm (arm-5 → armel → arm)", lc.Releases[0].Arch)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("arm_gitea_arm7_to_armv7l", func(t *testing.T) {
|
||||
// Gitea uses "arm-7" naming; patternToTerms converts to "armv7" → tpm → "armv7".
|
||||
// Go sees \barm\b → classifies as armv6. legacyARMArchFromFilename returns "armv7",
|
||||
// then the canonical armv7→armv7l translation produces armv7l.
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
{Filename: "gitea-1.20.0-linux-arm-7", OS: "linux", Arch: "armv6", Format: ""},
|
||||
},
|
||||
}
|
||||
lc, _ := storage.ExportLegacy("gitea", pd)
|
||||
if len(lc.Releases) != 1 {
|
||||
t.Fatalf("releases = %d, want 1", len(lc.Releases))
|
||||
}
|
||||
if lc.Releases[0].Arch != "armv7l" {
|
||||
t.Errorf("arch = %q, want armv7l (arm-7 → armv7 → armv7l)", lc.Releases[0].Arch)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// TestExportLegacyMixed verifies correct counting when multiple drop categories
|
||||
// appear together in a single export call.
|
||||
func TestExportLegacyMixed(t *testing.T) {
|
||||
pd := storage.PackageData{
|
||||
Assets: []storage.Asset{
|
||||
// kept: baseline linux build
|
||||
{Filename: "tool-linux-amd64.tar.gz", OS: "linux", Arch: "x86_64", Format: ".tar.gz"},
|
||||
// dropped: variant build
|
||||
{Filename: "tool-linux-amd64-rocm.tar.gz", OS: "linux", Arch: "x86_64", Format: ".tar.gz", Variants: []string{"rocm"}},
|
||||
// dropped: android
|
||||
{Filename: "tool-android-arm64.tar.gz", OS: "android", Arch: "aarch64", Format: ".tar.gz"},
|
||||
// dropped: .AppImage format
|
||||
{Filename: "tool.AppImage", OS: "linux", Arch: "x86_64", Format: ".AppImage"},
|
||||
// kept (translated): universal2 → x86_64
|
||||
{Filename: "tool-darwin-universal.tar.gz", OS: "darwin", Arch: "universal2", Format: ".tar.gz"},
|
||||
// kept: solaris as-is
|
||||
{Filename: "tool-solaris-amd64.tar.gz", OS: "solaris", Arch: "x86_64", Format: ".tar.gz"},
|
||||
},
|
||||
}
|
||||
lc, stats := storage.ExportLegacy("tool", pd)
|
||||
|
||||
if stats.Variants != 1 {
|
||||
t.Errorf("Variants = %d, want 1", stats.Variants)
|
||||
}
|
||||
if stats.Android != 1 {
|
||||
t.Errorf("Android = %d, want 1", stats.Android)
|
||||
}
|
||||
if stats.Formats != 1 {
|
||||
t.Errorf("Formats = %d, want 1", stats.Formats)
|
||||
}
|
||||
if len(lc.Releases) != 3 {
|
||||
t.Errorf("releases = %d, want 3 (linux + macos/amd64 + solaris)", len(lc.Releases))
|
||||
}
|
||||
|
||||
// Verify universal2 was translated to amd64 (via universal2→x86_64→amd64),
|
||||
// and darwin was translated to macos.
|
||||
var macosArch string
|
||||
for _, r := range lc.Releases {
|
||||
if r.OS == "macos" {
|
||||
macosArch = r.Arch
|
||||
}
|
||||
}
|
||||
if macosArch != "amd64" {
|
||||
t.Errorf("macos arch = %q, want amd64 (universal2→x86_64→amd64, darwin→macos)", macosArch)
|
||||
}
|
||||
}
|
||||
295
internal/storage/pgstore/pgstore.go
Normal file
295
internal/storage/pgstore/pgstore.go
Normal file
@@ -0,0 +1,295 @@
|
||||
// Package pgstore implements [storage.Store] on PostgreSQL.
|
||||
//
|
||||
// Schema uses double-buffering: two asset generations per package (0 and 1).
|
||||
// The active generation pointer in webi_packages is updated atomically on
|
||||
// Commit, so readers always see a complete consistent snapshot.
|
||||
//
|
||||
// Write path:
|
||||
//
|
||||
// BeginRefresh → clears inactive generation, returns tx
|
||||
// Put → stages assets in-memory
|
||||
// Commit → bulk-inserts assets (COPY), swaps generation pointer
|
||||
//
|
||||
// Read path:
|
||||
//
|
||||
// Load → reads active generation from webi_packages, fetches assets
|
||||
//
|
||||
// Connection string format: standard libpq / pgx DSN, e.g.:
|
||||
//
|
||||
// postgres://user:pass@host/dbname?sslmode=require
|
||||
// host=localhost user=webi dbname=webi sslmode=disable
|
||||
package pgstore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
)
|
||||
|
||||
// Schema holds the DDL for creating the required tables.
|
||||
// Run once on startup or deploy to ensure the schema exists.
|
||||
const Schema = `
|
||||
CREATE TABLE IF NOT EXISTS webi_packages (
|
||||
name TEXT NOT NULL PRIMARY KEY,
|
||||
active_gen SMALLINT NOT NULL DEFAULT 0,
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS webi_assets (
|
||||
id BIGSERIAL PRIMARY KEY,
|
||||
pkg TEXT NOT NULL,
|
||||
gen SMALLINT NOT NULL,
|
||||
filename TEXT NOT NULL DEFAULT '',
|
||||
version TEXT NOT NULL DEFAULT '',
|
||||
lts BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
channel TEXT NOT NULL DEFAULT '',
|
||||
date TEXT NOT NULL DEFAULT '',
|
||||
os TEXT NOT NULL DEFAULT '',
|
||||
arch TEXT NOT NULL DEFAULT '',
|
||||
libc TEXT NOT NULL DEFAULT '',
|
||||
format TEXT NOT NULL DEFAULT '',
|
||||
download TEXT NOT NULL DEFAULT '',
|
||||
extra TEXT NOT NULL DEFAULT '',
|
||||
variants TEXT[] NOT NULL DEFAULT '{}'
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS webi_assets_pkg_gen ON webi_assets (pkg, gen);
|
||||
`
|
||||
|
||||
// Store is a PostgreSQL-backed asset store.
|
||||
type Store struct {
|
||||
pool *pgxpool.Pool
|
||||
}
|
||||
|
||||
// New opens a connection pool to the given DSN and applies the schema.
|
||||
// Returns an error if the connection or schema creation fails.
|
||||
func New(ctx context.Context, dsn string) (*Store, error) {
|
||||
cfg, err := pgxpool.ParseConfig(dsn)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("pgstore: parse dsn: %w", err)
|
||||
}
|
||||
|
||||
pool, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("pgstore: connect: %w", err)
|
||||
}
|
||||
|
||||
if err := applySchema(ctx, pool); err != nil {
|
||||
pool.Close()
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &Store{pool: pool}, nil
|
||||
}
|
||||
|
||||
// Close releases the connection pool.
|
||||
func (s *Store) Close() {
|
||||
s.pool.Close()
|
||||
}
|
||||
|
||||
// ListPackages returns the names of all packages in the store.
|
||||
func (s *Store) ListPackages(ctx context.Context) ([]string, error) {
|
||||
rows, err := s.pool.Query(ctx,
|
||||
`SELECT name FROM webi_packages ORDER BY name`,
|
||||
)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("pgstore: list packages: %w", err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var pkgs []string
|
||||
for rows.Next() {
|
||||
var name string
|
||||
if err := rows.Scan(&name); err != nil {
|
||||
return nil, fmt.Errorf("pgstore: scan package name: %w", err)
|
||||
}
|
||||
pkgs = append(pkgs, name)
|
||||
}
|
||||
return pkgs, rows.Err()
|
||||
}
|
||||
|
||||
// Load returns all assets for a package using the active generation.
|
||||
// Returns nil (not an error) if the package is not cached.
|
||||
func (s *Store) Load(ctx context.Context, pkg string) (*storage.PackageData, error) {
|
||||
// Fetch active generation and updated_at.
|
||||
var gen int16
|
||||
var updatedAt time.Time
|
||||
err := s.pool.QueryRow(ctx,
|
||||
`SELECT active_gen, updated_at FROM webi_packages WHERE name = $1`,
|
||||
pkg,
|
||||
).Scan(&gen, &updatedAt)
|
||||
if err == pgx.ErrNoRows {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("pgstore: load %s: %w", pkg, err)
|
||||
}
|
||||
|
||||
// Fetch all assets for this generation.
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT filename, version, lts, channel, date,
|
||||
os, arch, libc, format, download, extra, variants
|
||||
FROM webi_assets
|
||||
WHERE pkg = $1 AND gen = $2
|
||||
ORDER BY id
|
||||
`, pkg, gen)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("pgstore: load assets %s: %w", pkg, err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var assets []storage.Asset
|
||||
for rows.Next() {
|
||||
var a storage.Asset
|
||||
if err := rows.Scan(
|
||||
&a.Filename, &a.Version, &a.LTS, &a.Channel, &a.Date,
|
||||
&a.OS, &a.Arch, &a.Libc, &a.Format, &a.Download,
|
||||
&a.Extra, &a.Variants,
|
||||
); err != nil {
|
||||
return nil, fmt.Errorf("pgstore: scan asset %s: %w", pkg, err)
|
||||
}
|
||||
assets = append(assets, a)
|
||||
}
|
||||
if err := rows.Err(); err != nil {
|
||||
return nil, fmt.Errorf("pgstore: rows %s: %w", pkg, err)
|
||||
}
|
||||
|
||||
return &storage.PackageData{
|
||||
Assets: assets,
|
||||
UpdatedAt: updatedAt,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// BeginRefresh starts a write transaction for a package.
|
||||
// It determines the inactive generation and clears it, ready for new data.
|
||||
func (s *Store) BeginRefresh(ctx context.Context, pkg string) (storage.RefreshTx, error) {
|
||||
// Determine which generation to write into (the inactive one).
|
||||
var activeGen int16
|
||||
err := s.pool.QueryRow(ctx,
|
||||
`SELECT active_gen FROM webi_packages WHERE name = $1`,
|
||||
pkg,
|
||||
).Scan(&activeGen)
|
||||
if err != nil && err != pgx.ErrNoRows {
|
||||
return nil, fmt.Errorf("pgstore: begin refresh %s: %w", pkg, err)
|
||||
}
|
||||
// If package doesn't exist yet, activeGen defaults to 0 and we write to gen 1.
|
||||
// If package exists, we write to the inactive generation (1 - activeGen).
|
||||
var writeGen int16
|
||||
if err == pgx.ErrNoRows {
|
||||
writeGen = 1
|
||||
} else {
|
||||
writeGen = 1 - activeGen
|
||||
}
|
||||
|
||||
// Clear the write generation so we start fresh.
|
||||
if _, err := s.pool.Exec(ctx,
|
||||
`DELETE FROM webi_assets WHERE pkg = $1 AND gen = $2`,
|
||||
pkg, writeGen,
|
||||
); err != nil {
|
||||
return nil, fmt.Errorf("pgstore: clear gen %d for %s: %w", writeGen, pkg, err)
|
||||
}
|
||||
|
||||
return &refreshTx{
|
||||
pool: s.pool,
|
||||
pkg: pkg,
|
||||
gen: writeGen,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// refreshTx is an in-progress write for one package.
|
||||
type refreshTx struct {
|
||||
pool *pgxpool.Pool
|
||||
pkg string
|
||||
gen int16
|
||||
assets []storage.Asset
|
||||
}
|
||||
|
||||
// Put stages assets for writing. May be called multiple times.
|
||||
func (tx *refreshTx) Put(assets []storage.Asset) error {
|
||||
tx.assets = append(tx.assets, assets...)
|
||||
return nil
|
||||
}
|
||||
|
||||
// Commit bulk-inserts all staged assets, then atomically swaps the
|
||||
// active generation pointer in webi_packages.
|
||||
func (tx *refreshTx) Commit(ctx context.Context) error {
|
||||
if len(tx.assets) == 0 {
|
||||
return tx.swapGeneration(ctx)
|
||||
}
|
||||
|
||||
// Build rows for pgx.CopyFromRows.
|
||||
rows := make([][]any, len(tx.assets))
|
||||
for i, a := range tx.assets {
|
||||
variants := a.Variants
|
||||
if variants == nil {
|
||||
variants = []string{}
|
||||
}
|
||||
rows[i] = []any{
|
||||
tx.pkg,
|
||||
tx.gen,
|
||||
a.Filename,
|
||||
a.Version,
|
||||
a.LTS,
|
||||
a.Channel,
|
||||
a.Date,
|
||||
a.OS,
|
||||
a.Arch,
|
||||
a.Libc,
|
||||
a.Format,
|
||||
a.Download,
|
||||
a.Extra,
|
||||
variants,
|
||||
}
|
||||
}
|
||||
|
||||
cols := []string{
|
||||
"pkg", "gen",
|
||||
"filename", "version", "lts", "channel", "date",
|
||||
"os", "arch", "libc", "format", "download", "extra", "variants",
|
||||
}
|
||||
|
||||
_, err := tx.pool.CopyFrom(ctx,
|
||||
pgx.Identifier{"webi_assets"},
|
||||
cols,
|
||||
pgx.CopyFromRows(rows),
|
||||
)
|
||||
if err != nil {
|
||||
return fmt.Errorf("pgstore: copy assets %s: %w", tx.pkg, err)
|
||||
}
|
||||
|
||||
return tx.swapGeneration(ctx)
|
||||
}
|
||||
|
||||
// swapGeneration atomically updates the active generation pointer.
|
||||
func (tx *refreshTx) swapGeneration(ctx context.Context) error {
|
||||
_, err := tx.pool.Exec(ctx, `
|
||||
INSERT INTO webi_packages (name, active_gen, updated_at)
|
||||
VALUES ($1, $2, now())
|
||||
ON CONFLICT (name)
|
||||
DO UPDATE SET active_gen = $2, updated_at = now()
|
||||
`, tx.pkg, tx.gen)
|
||||
if err != nil {
|
||||
return fmt.Errorf("pgstore: swap gen %s: %w", tx.pkg, err)
|
||||
}
|
||||
tx.assets = nil
|
||||
return nil
|
||||
}
|
||||
|
||||
// Rollback discards all staged assets without writing anything.
|
||||
func (tx *refreshTx) Rollback() error {
|
||||
tx.assets = nil
|
||||
return nil
|
||||
}
|
||||
|
||||
// applySchema runs the schema DDL idempotently.
|
||||
func applySchema(ctx context.Context, pool *pgxpool.Pool) error {
|
||||
if _, err := pool.Exec(ctx, Schema); err != nil {
|
||||
return fmt.Errorf("pgstore: apply schema: %w", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
71
internal/storage/storage.go
Normal file
71
internal/storage/storage.go
Normal file
@@ -0,0 +1,71 @@
|
||||
// Package storage defines the interface for reading and writing
|
||||
// classified release assets.
|
||||
//
|
||||
// webid reads assets through [Store]. webicached writes them through
|
||||
// [RefreshTx], obtained from [Store.BeginRefresh].
|
||||
//
|
||||
// The two implementations are fsstore (filesystem JSON, compatible with
|
||||
// the Node.js _cache/ format) and pgstore (PostgreSQL, future).
|
||||
package storage
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Asset is a single downloadable file — one entry in a release.
|
||||
// A release like "bat v0.26.1" has many assets (one per platform/format).
|
||||
//
|
||||
// No JSON tags — serialization goes through [LegacyAsset] for Node.js
|
||||
// compat, or through a future v2 format.
|
||||
type Asset struct {
|
||||
Filename string
|
||||
Version string
|
||||
LTS bool
|
||||
Channel string
|
||||
Date string
|
||||
OS string
|
||||
Arch string
|
||||
Libc string
|
||||
Format string
|
||||
Download string
|
||||
Extra string // extra version info for sorting (e.g. build metadata)
|
||||
GitTag string // original git tag (e.g. "v1.2", "master") — only for format="git"
|
||||
GitCommitHash string // short commit hash (e.g. "54c216e") — only for format="git"
|
||||
Variants []string // build qualifiers: "installer", "rocm", "jetpack5", "fxdependent", etc.
|
||||
}
|
||||
|
||||
// PackageData is the full set of assets for a package, plus metadata.
|
||||
type PackageData struct {
|
||||
Assets []Asset
|
||||
UpdatedAt time.Time
|
||||
}
|
||||
|
||||
// Store is the read/write interface for release asset storage.
|
||||
type Store interface {
|
||||
// ListPackages returns the names of all packages in the store.
|
||||
ListPackages(ctx context.Context) ([]string, error)
|
||||
|
||||
// Load returns all assets for a package, or nil if the package
|
||||
// is not cached. The returned data may be stale — check UpdatedAt.
|
||||
Load(ctx context.Context, pkg string) (*PackageData, error)
|
||||
|
||||
// BeginRefresh starts a write transaction for a package.
|
||||
// Write assets via [RefreshTx.Put], then call Commit to atomically
|
||||
// replace the stored data. Call Rollback to discard.
|
||||
BeginRefresh(ctx context.Context, pkg string) (RefreshTx, error)
|
||||
}
|
||||
|
||||
// RefreshTx is a write transaction for replacing a package's assets.
|
||||
type RefreshTx interface {
|
||||
// Put stages assets to be written. May be called multiple times
|
||||
// to append assets incrementally.
|
||||
Put(assets []Asset) error
|
||||
|
||||
// Commit atomically replaces the package's stored assets with
|
||||
// everything staged via Put.
|
||||
Commit(ctx context.Context) error
|
||||
|
||||
// Rollback discards all staged data.
|
||||
Rollback() error
|
||||
}
|
||||
247
internal/uadetect/uadetect.go
Normal file
247
internal/uadetect/uadetect.go
Normal file
@@ -0,0 +1,247 @@
|
||||
// Package uadetect identifies the requesting agent's OS, CPU architecture,
|
||||
// and libc so the server can select the correct release artifact.
|
||||
//
|
||||
// An agent identifies itself through multiple signals:
|
||||
// - The User-Agent header: Webi's bootstrap scripts send "$(uname -srm)",
|
||||
// e.g. "Darwin 23.1.0 arm64". Browsers, curl, and PowerShell send their
|
||||
// own UA strings.
|
||||
// - Query parameters: ?os=linux&arch=arm64 are an explicit declaration
|
||||
// that takes precedence over the header.
|
||||
//
|
||||
// Use [FromRequest] to detect from an HTTP request (preferred).
|
||||
// Use [Parse] to detect from a raw UA string.
|
||||
package uadetect
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"strings"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/buildmeta"
|
||||
)
|
||||
|
||||
// Result holds the detected platform info from a User-Agent string.
|
||||
type Result struct {
|
||||
OS buildmeta.OS
|
||||
Arch buildmeta.Arch
|
||||
Libc buildmeta.Libc
|
||||
}
|
||||
|
||||
// FromRequest detects the agent's platform from an HTTP request.
|
||||
// Query parameters ?os and ?arch override the User-Agent header.
|
||||
func FromRequest(r *http.Request) Result {
|
||||
qOS := r.URL.Query().Get("os")
|
||||
qArch := r.URL.Query().Get("arch")
|
||||
|
||||
var ua string
|
||||
switch {
|
||||
case qOS != "" && qArch != "":
|
||||
ua = qOS + " " + qArch
|
||||
case qOS != "":
|
||||
ua = qOS
|
||||
case qArch != "":
|
||||
ua = qArch
|
||||
default:
|
||||
ua = r.Header.Get("User-Agent")
|
||||
}
|
||||
|
||||
return Parse(ua)
|
||||
}
|
||||
|
||||
// Parse extracts OS, arch, and libc from a User-Agent string.
|
||||
func Parse(ua string) Result {
|
||||
if ua == "-" {
|
||||
return Result{}
|
||||
}
|
||||
|
||||
tokens := tokenize(ua)
|
||||
|
||||
return Result{
|
||||
OS: matchOS(tokens),
|
||||
Arch: matchArch(tokens),
|
||||
Libc: matchLibc(tokens),
|
||||
}
|
||||
}
|
||||
|
||||
// tokenize splits a User-Agent into lowercase tokens for matching.
|
||||
// Splits on whitespace, '/', and ';', since UAs come in various forms:
|
||||
//
|
||||
// "Darwin 23.1.0 arm64" (uname -srm)
|
||||
// "PowerShell/7.3.0" (PowerShell)
|
||||
// "MS AMD64" (Windows shorthand)
|
||||
// "Macintosh; Intel Mac OS X 10_15_7" (browser)
|
||||
func tokenize(ua string) []string {
|
||||
// Strip xnu kernel info that can mislead arch detection under Rosetta.
|
||||
// "xnu-7195.60.75~1/RELEASE_ARM64_T8101" contains ARM64 even when
|
||||
// running as x86_64. This only appears in verbose uname output.
|
||||
if i := strings.Index(ua, "xnu-"); i >= 0 {
|
||||
end := strings.IndexByte(ua[i:], ' ')
|
||||
if end < 0 {
|
||||
ua = ua[:i]
|
||||
} else {
|
||||
ua = ua[:i] + ua[i+end:]
|
||||
}
|
||||
}
|
||||
|
||||
return strings.FieldsFunc(strings.ToLower(ua), func(r rune) bool {
|
||||
return r == ' ' || r == '/' || r == ';' || r == '\t'
|
||||
})
|
||||
}
|
||||
|
||||
// matchOS identifies the operating system from tokens.
|
||||
// Order matters: Android before Linux, Linux before Windows (for WSL).
|
||||
func matchOS(tokens []string) buildmeta.OS {
|
||||
has := func(s string) bool {
|
||||
for _, t := range tokens {
|
||||
if strings.Contains(t, s) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// Android must be checked before Linux.
|
||||
if has("android") {
|
||||
return buildmeta.OSAndroid
|
||||
}
|
||||
|
||||
if has("darwin") || has("macos") || has("macintosh") || has("iphone") || has("ios") || has("ipad") {
|
||||
return buildmeta.OSDarwin
|
||||
}
|
||||
// "mac" alone (not in "macintosh" which is already matched)
|
||||
for _, t := range tokens {
|
||||
if t == "mac" {
|
||||
return buildmeta.OSDarwin
|
||||
}
|
||||
}
|
||||
|
||||
// FreeBSD before Linux (both are POSIX, but FreeBSD never reports "linux").
|
||||
if has("freebsd") {
|
||||
return buildmeta.OSFreeBSD
|
||||
}
|
||||
|
||||
// Linux before Windows because WSL UAs contain both "linux" and "microsoft".
|
||||
// But exclude Cygwin/Msys/MINGW which report Linux-like strings on Windows.
|
||||
if has("linux") && !has("cygwin") && !has("msysgit") && !has("msys") && !has("mingw") {
|
||||
return buildmeta.OSLinux
|
||||
}
|
||||
|
||||
// Cygwin, Msys, and MINGW are Windows environments.
|
||||
if has("windows") || has("win32") || has("microsoft") || has("powershell") ||
|
||||
has("cygwin") || has("msys") || has("mingw") {
|
||||
return buildmeta.OSWindows
|
||||
}
|
||||
for _, t := range tokens {
|
||||
if t == "ms" || t == "win" {
|
||||
return buildmeta.OSWindows
|
||||
}
|
||||
}
|
||||
|
||||
// Fallback: curl and wget imply a POSIX system, almost always Linux.
|
||||
if has("curl") || has("wget") {
|
||||
return buildmeta.OSLinux
|
||||
}
|
||||
|
||||
return ""
|
||||
}
|
||||
|
||||
// matchArch identifies the CPU architecture from tokens.
|
||||
// More specific patterns are checked before less specific ones.
|
||||
func matchArch(tokens []string) buildmeta.Arch {
|
||||
has := func(s string) bool {
|
||||
for _, t := range tokens {
|
||||
if strings.Contains(t, s) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
exact := func(s string) bool {
|
||||
for _, t := range tokens {
|
||||
if t == s {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
// ARM 64-bit (most specific first)
|
||||
if has("aarch64") || has("arm64") || has("armv8") {
|
||||
return buildmeta.ArchARM64
|
||||
}
|
||||
|
||||
// ARM 32-bit variants
|
||||
if has("armv7") || has("arm32") {
|
||||
return buildmeta.ArchARMv7
|
||||
}
|
||||
if has("armv6") {
|
||||
return buildmeta.ArchARMv6
|
||||
}
|
||||
// Bare "arm" without a version qualifier → armv6 (conservative).
|
||||
if exact("arm") {
|
||||
return buildmeta.ArchARMv6
|
||||
}
|
||||
|
||||
// POWER (check before generic 64-bit)
|
||||
if has("ppc64le") {
|
||||
return buildmeta.ArchPPC64LE
|
||||
}
|
||||
if has("ppc64") {
|
||||
return buildmeta.ArchPPC64
|
||||
}
|
||||
|
||||
// s390x (IBM Z)
|
||||
if has("s390x") {
|
||||
return buildmeta.ArchS390X
|
||||
}
|
||||
|
||||
// RISC-V
|
||||
if has("riscv64") {
|
||||
return buildmeta.ArchRISCV64
|
||||
}
|
||||
|
||||
// MIPS (check before generic 64-bit)
|
||||
if has("mips64") {
|
||||
return buildmeta.ArchMIPS64
|
||||
}
|
||||
if has("mips") {
|
||||
return buildmeta.ArchMIPS
|
||||
}
|
||||
|
||||
// x86-64
|
||||
if has("x86_64") || has("amd64") || exact("x64") {
|
||||
return buildmeta.ArchAMD64
|
||||
}
|
||||
|
||||
// x86 32-bit (after x86_64 to avoid false match)
|
||||
if has("i386") || has("i686") || exact("x86") {
|
||||
return buildmeta.ArchX86
|
||||
}
|
||||
|
||||
return ""
|
||||
}
|
||||
|
||||
// matchLibc identifies the C library from tokens.
|
||||
func matchLibc(tokens []string) buildmeta.Libc {
|
||||
has := func(s string) bool {
|
||||
for _, t := range tokens {
|
||||
if strings.Contains(t, s) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
if has("musl") {
|
||||
return buildmeta.LibcMusl
|
||||
}
|
||||
// Don't match "microsoft" — it appears in WSL kernel version strings
|
||||
// (e.g. "5.15.146.1-microsoft-standard-WSL2") and doesn't indicate MSVC.
|
||||
if has("msvc") || has("windows") {
|
||||
return buildmeta.LibcMSVC
|
||||
}
|
||||
if has("gnu") || has("glibc") || has("linux") {
|
||||
return buildmeta.LibcGNU
|
||||
}
|
||||
|
||||
return buildmeta.LibcNone
|
||||
}
|
||||
190
internal/uadetect/uadetect_test.go
Normal file
190
internal/uadetect/uadetect_test.go
Normal file
@@ -0,0 +1,190 @@
|
||||
package uadetect_test
|
||||
|
||||
import (
|
||||
"net/http"
|
||||
"testing"
|
||||
|
||||
"github.com/webinstall/webi-installers/internal/buildmeta"
|
||||
"github.com/webinstall/webi-installers/internal/uadetect"
|
||||
)
|
||||
|
||||
func TestOS(t *testing.T) {
|
||||
tests := []struct {
|
||||
ua string
|
||||
want buildmeta.OS
|
||||
}{
|
||||
// uname -srm style
|
||||
{"Darwin 23.1.0 arm64", buildmeta.OSDarwin},
|
||||
{"Darwin 20.2.0 x86_64", buildmeta.OSDarwin},
|
||||
{"Linux 6.1.0-18-amd64 x86_64", buildmeta.OSLinux},
|
||||
{"Linux 5.15.0 aarch64", buildmeta.OSLinux},
|
||||
|
||||
// WSL: Linux, not Windows (contains "microsoft" in kernel release)
|
||||
{"Linux 5.15.146.1-microsoft-standard-WSL2 x86_64", buildmeta.OSLinux},
|
||||
|
||||
// Windows
|
||||
{"MS AMD64", buildmeta.OSWindows},
|
||||
{"PowerShell/7.3.0", buildmeta.OSWindows},
|
||||
{"Microsoft Windows 10.0.19045", buildmeta.OSWindows},
|
||||
|
||||
// Msys/MINGW/Cygwin → Windows
|
||||
{"webi/curl x86_64/unknown Msys/MINGW64_NT-10.0-19045/3.5.7-463ebcdc.x86_64 libc", buildmeta.OSWindows},
|
||||
{"webi/curl+wget x86_64/unknown Msys/MSYS_NT-10.0-26200/3.6.6-1cdd4371.x86_64 libc", buildmeta.OSWindows},
|
||||
{"webi/curl x86_64/unknown Cygwin/CYGWIN_NT-10.0/2.10.0(0.325/5/3) libc", buildmeta.OSWindows},
|
||||
|
||||
// FreeBSD
|
||||
{"webi/curl amd64/unknown FreeBSD/14.3-RELEASE-p8 libc", buildmeta.OSFreeBSD},
|
||||
|
||||
// Android before Linux
|
||||
{"Android 13 aarch64", buildmeta.OSAndroid},
|
||||
{"webi/curl aarch64/unknown Android/Linux/6.6.77-android15-8 libc", buildmeta.OSAndroid},
|
||||
|
||||
// WSL: Linux, not Windows (kernel contains "microsoft")
|
||||
{"webi/curl+wget x86_64/unknown GNU/Linux/5.15.146.1-microsoft-standard-WSL2 libc", buildmeta.OSLinux},
|
||||
|
||||
// Browser-style
|
||||
{"Macintosh; Intel Mac OS X 10_15_7", buildmeta.OSDarwin},
|
||||
|
||||
// Minimal agents → assume Linux
|
||||
{"curl/8.1.2", buildmeta.OSLinux},
|
||||
{"wget/1.21", buildmeta.OSLinux},
|
||||
|
||||
// Explicit unknown
|
||||
{"-", ""},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.ua, func(t *testing.T) {
|
||||
got := uadetect.Parse(tt.ua).OS
|
||||
if got != tt.want {
|
||||
t.Errorf("Parse(%q).OS = %q, want %q", tt.ua, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestArch(t *testing.T) {
|
||||
tests := []struct {
|
||||
ua string
|
||||
want buildmeta.Arch
|
||||
}{
|
||||
{"Darwin 23.1.0 arm64", buildmeta.ArchARM64},
|
||||
{"Linux 6.1.0 aarch64", buildmeta.ArchARM64},
|
||||
{"Linux 5.4.0 x86_64", buildmeta.ArchAMD64},
|
||||
{"MS AMD64", buildmeta.ArchAMD64},
|
||||
{"Linux 5.10.0 armv7l", buildmeta.ArchARMv7},
|
||||
{"Linux 5.10.0 armv6l", buildmeta.ArchARMv6},
|
||||
{"Linux 5.4.0 ppc64le", buildmeta.ArchPPC64LE},
|
||||
{"webi/curl+wget s390x/unknown GNU/Linux/6.4.0-150700.53.6-default libc", buildmeta.ArchS390X},
|
||||
|
||||
// FreeBSD uses "amd64" not "x86_64"
|
||||
{"webi/curl amd64/unknown FreeBSD/14.3-RELEASE-p8 libc", buildmeta.ArchAMD64},
|
||||
|
||||
// Rosetta: xnu kernel info says ARM64 but actual arch is x86_64
|
||||
{"Darwin 20.2.0 Darwin Kernel Version 20.2.0; root:xnu-7195.60.75~1/RELEASE_ARM64_T8101 x86_64", buildmeta.ArchAMD64},
|
||||
|
||||
{"-", ""},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.ua, func(t *testing.T) {
|
||||
got := uadetect.Parse(tt.ua).Arch
|
||||
if got != tt.want {
|
||||
t.Errorf("Parse(%q).Arch = %q, want %q", tt.ua, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestLibc(t *testing.T) {
|
||||
tests := []struct {
|
||||
ua string
|
||||
want buildmeta.Libc
|
||||
}{
|
||||
{"Linux 6.1.0 x86_64 musl", buildmeta.LibcMusl},
|
||||
{"Linux 6.1.0 x86_64 gnu", buildmeta.LibcGNU},
|
||||
{"Linux 6.1.0 x86_64 linux", buildmeta.LibcGNU},
|
||||
{"MS AMD64 msvc", buildmeta.LibcMSVC},
|
||||
{"Microsoft Windows", buildmeta.LibcMSVC},
|
||||
{"Darwin 23.1.0 arm64", buildmeta.LibcNone},
|
||||
|
||||
// WSL: kernel version contains "microsoft" but libc is gnu, not msvc
|
||||
{"webi/curl+wget x86_64/unknown GNU/Linux/5.15.146.1-microsoft-standard-WSL2 libc", buildmeta.LibcGNU},
|
||||
|
||||
{"-", ""},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.ua, func(t *testing.T) {
|
||||
got := uadetect.Parse(tt.ua).Libc
|
||||
if got != tt.want {
|
||||
t.Errorf("Parse(%q).Libc = %q, want %q", tt.ua, got, tt.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFromRequest(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
ua string // User-Agent header
|
||||
query string // raw query string
|
||||
wantOS buildmeta.OS
|
||||
wantAr buildmeta.Arch
|
||||
}{
|
||||
{
|
||||
name: "UA header only",
|
||||
ua: "Darwin 23.1.0 arm64",
|
||||
wantOS: buildmeta.OSDarwin,
|
||||
wantAr: buildmeta.ArchARM64,
|
||||
},
|
||||
{
|
||||
name: "query params override UA",
|
||||
ua: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)",
|
||||
query: "os=linux&arch=aarch64",
|
||||
wantOS: buildmeta.OSLinux,
|
||||
wantAr: buildmeta.ArchARM64,
|
||||
},
|
||||
{
|
||||
name: "os param only",
|
||||
ua: "curl/8.1.2",
|
||||
query: "os=windows",
|
||||
wantOS: buildmeta.OSWindows,
|
||||
},
|
||||
{
|
||||
name: "arch param only",
|
||||
ua: "curl/8.1.2",
|
||||
query: "arch=arm64",
|
||||
wantAr: buildmeta.ArchARM64,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
req, _ := http.NewRequest("GET", "http://example.com/api?"+tt.query, nil)
|
||||
if tt.ua != "" {
|
||||
req.Header.Set("User-Agent", tt.ua)
|
||||
}
|
||||
got := uadetect.FromRequest(req)
|
||||
if tt.wantOS != "" && got.OS != tt.wantOS {
|
||||
t.Errorf("OS = %q, want %q", got.OS, tt.wantOS)
|
||||
}
|
||||
if tt.wantAr != "" && got.Arch != tt.wantAr {
|
||||
t.Errorf("Arch = %q, want %q", got.Arch, tt.wantAr)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestFullParse(t *testing.T) {
|
||||
r := uadetect.Parse("Darwin 23.1.0 arm64")
|
||||
if r.OS != buildmeta.OSDarwin {
|
||||
t.Errorf("OS = %q, want %q", r.OS, buildmeta.OSDarwin)
|
||||
}
|
||||
if r.Arch != buildmeta.ArchARM64 {
|
||||
t.Errorf("Arch = %q, want %q", r.Arch, buildmeta.ArchARM64)
|
||||
}
|
||||
if r.Libc != buildmeta.LibcNone {
|
||||
t.Errorf("Libc = %q, want %q", r.Libc, buildmeta.LibcNone)
|
||||
}
|
||||
}
|
||||
63
scripts/deploy-webicached.sh
Executable file
63
scripts/deploy-webicached.sh
Executable file
@@ -0,0 +1,63 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
set -u
|
||||
|
||||
# Build and deploy webicached to a target host
|
||||
|
||||
g_host="${1:-beta.webi.sh}"
|
||||
g_bin="webicached"
|
||||
g_out="agents/tmp/${g_bin}"
|
||||
g_remote_bin="~/bin/${g_bin}"
|
||||
|
||||
case "${g_host}" in
|
||||
beta.webi.sh) g_remote_conf="~/srv/beta.webinstall.dev/installers/" ;;
|
||||
next.webi.sh) g_remote_conf="~/srv/next.webinstall.dev/installers/" ;;
|
||||
*) g_remote_conf="~/srv/webid/installers/" ;;
|
||||
esac
|
||||
|
||||
fn_build() {
|
||||
b_version="$(git describe --tags --always 2> /dev/null || echo '0.0.0-dev')"
|
||||
b_commit="$(git rev-parse --short HEAD)"
|
||||
b_date="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
b_ldflags="-X main.version=${b_version} -X main.commit=${b_commit} -X main.date=${b_date}"
|
||||
|
||||
printf 'Building %s %s %s (%s)...\n' "${g_bin}" "${b_version}" "${b_commit}" "${b_date}"
|
||||
GOOS=linux GOARCH=amd64 GOAMD64=v2 go build -ldflags "${b_ldflags}" -o "${g_out}" ./cmd/webicached
|
||||
printf 'Built: %s\n' "${g_out}"
|
||||
}
|
||||
|
||||
fn_deploy() {
|
||||
printf 'Stopping %s on %s...\n' "${g_bin}" "${g_host}"
|
||||
ssh "${g_host}" "~/.local/bin/serviceman stop ${g_bin}" 2> /dev/null || true
|
||||
|
||||
printf 'Uploading binary...\n'
|
||||
scp "${g_out}" "${g_host}:${g_remote_bin}"
|
||||
|
||||
printf 'Syncing releases.conf files...\n'
|
||||
rsync -av \
|
||||
--exclude='_cache' --exclude='.git' --exclude='agents' \
|
||||
--exclude='bin' --exclude='cmd' --exclude='internal' \
|
||||
--exclude='docs' --exclude='scripts' --exclude='node_modules' \
|
||||
--include='*/' --include='releases.conf' --exclude='*' \
|
||||
./ "${g_host}:${g_remote_conf}"
|
||||
|
||||
printf 'Starting %s...\n' "${g_bin}"
|
||||
ssh "${g_host}" "~/.local/bin/serviceman start ${g_bin}"
|
||||
}
|
||||
|
||||
fn_verify() {
|
||||
printf 'Waiting 5s for startup...\n'
|
||||
sleep 5
|
||||
|
||||
printf 'Checking version...\n'
|
||||
ssh "${g_host}" "${g_remote_bin} -V"
|
||||
|
||||
printf 'Checking logs...\n'
|
||||
ssh "${g_host}" "sudo journalctl -u ${g_bin} --no-pager -n 5"
|
||||
}
|
||||
|
||||
fn_build
|
||||
fn_deploy
|
||||
fn_verify
|
||||
|
||||
printf '\nDone. %s deployed to %s.\n' "${g_bin}" "${g_host}"
|
||||
111
skills/deploy-webicached/SKILL.md
Normal file
111
skills/deploy-webicached/SKILL.md
Normal file
@@ -0,0 +1,111 @@
|
||||
---
|
||||
name: deploy-webicached
|
||||
description: Deploy webicached binary to beta.webi.sh. Use when building, uploading, or restarting the cache daemon. Covers cross-compile, conf sync, service management.
|
||||
---
|
||||
|
||||
## One-step deploy
|
||||
|
||||
```sh
|
||||
./scripts/deploy-webicached.sh beta.webi.sh
|
||||
```
|
||||
|
||||
Builds with version ldflags, stops service, uploads, syncs conf, starts, verifies.
|
||||
|
||||
## Manual steps (if needed)
|
||||
|
||||
### Build
|
||||
|
||||
```sh
|
||||
VERSION="$(git describe --tags --always)"
|
||||
COMMIT="$(git rev-parse --short HEAD)"
|
||||
DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
GOOS=linux GOARCH=amd64 GOAMD64=v2 go build \
|
||||
-ldflags "-X main.version=${VERSION} -X main.commit=${COMMIT} -X main.date=${DATE}" \
|
||||
-o agents/tmp/webicached ./cmd/webicached
|
||||
```
|
||||
|
||||
MUST: Build from the `ref-webi-go` worktree (or branch containing `cmd/webicached`).
|
||||
|
||||
### Deploy
|
||||
|
||||
```sh
|
||||
ssh beta.webi.sh "serviceman stop webicached"
|
||||
scp agents/tmp/webicached beta.webi.sh:~/bin/webicached
|
||||
```
|
||||
|
||||
MUST: Stop service before scp — Linux refuses to overwrite a running binary.
|
||||
|
||||
### Sync releases.conf
|
||||
|
||||
```sh
|
||||
rsync -av --include='*/' --include='releases.conf' --exclude='*' \
|
||||
./ beta.webi.sh:~/srv/beta.webinstall.dev/installers/
|
||||
```
|
||||
|
||||
MUST: Run from the worktree root. The server has no checkout of this branch — conf files must be synced explicitly.
|
||||
|
||||
### Start
|
||||
|
||||
```sh
|
||||
ssh beta.webi.sh "serviceman start webicached"
|
||||
```
|
||||
|
||||
### Verify
|
||||
|
||||
```sh
|
||||
ssh beta.webi.sh "sleep 5 && serviceman logs webicached"
|
||||
```
|
||||
|
||||
Expected: "batch: N stale, refreshing 20" or "all packages fresh, sleeping 9s"
|
||||
|
||||
## Smoke test
|
||||
|
||||
```sh
|
||||
ssh beta.webi.sh "curl -sSf http://localhost:3080/api/releases/bat.json | head -c 100"
|
||||
ssh beta.webi.sh "curl -sSf -A 'curl/7.81.0 Linux x86_64' http://localhost:3080/api/installers/bat.sh | head -3"
|
||||
```
|
||||
|
||||
Expected: JSON array with release objects; shell script with `PKG_NAME='bat'`.
|
||||
|
||||
## Service management
|
||||
|
||||
```sh
|
||||
serviceman status webicached
|
||||
serviceman restart webicached
|
||||
serviceman logs webicached
|
||||
```
|
||||
|
||||
## Server layout
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `~/bin/webicached` | Binary |
|
||||
| `~/srv/beta.webinstall.dev/installers/` | Conf dir (releases.conf files) |
|
||||
| `~/.cache/webi/legacy/` | Cache output (fsstore, legacy JSON format) |
|
||||
| `~/.cache/webi/raw/` | Raw upstream API responses |
|
||||
| `~/srv/beta.webinstall.dev/.env.secret` | GITHUB_TOKEN |
|
||||
| `/etc/systemd/system/webicached.service` | Service unit (created by serviceman) |
|
||||
|
||||
## Flags reference
|
||||
|
||||
| Flag | Default | Purpose |
|
||||
|------|---------|---------|
|
||||
| `--conf` | `.` | Dir with `{pkg}/releases.conf` files |
|
||||
| `--legacy` | `~/.cache/webi/legacy` | Legacy cache output directory |
|
||||
| `--raw` | `~/.cache/webi/raw` | Raw upstream response cache |
|
||||
| `--token` | `$GITHUB_TOKEN` | GitHub API token |
|
||||
| `--interval` | `9s` | Delay between package fetches in a batch |
|
||||
| `--once` | false | Run once then exit |
|
||||
| `--eager` | false | Fetch all on startup (not staleness-based) |
|
||||
| `--shallow` | false | Only first page of releases |
|
||||
| `--no-fetch` | false | Classify from rawcache only |
|
||||
| `--page-delay` | `2s` | Delay between paginated API pages |
|
||||
|
||||
## One-shot refresh (specific packages)
|
||||
|
||||
```sh
|
||||
ssh beta.webi.sh ". ~/srv/beta.webinstall.dev/.env.secret && ~/bin/webicached \
|
||||
--conf ~/srv/beta.webinstall.dev/installers/ \
|
||||
--raw ~/.cache/webi/raw \
|
||||
--once bat goreleaser"
|
||||
```
|
||||
Reference in New Issue
Block a user