mirror of
https://github.com/webinstall/webi-installers.git
synced 2026-05-31 04:52:48 +00:00
Compare commits
8 Commits
ref-cache-
...
feat-nativ
| Author | SHA1 | Date | |
|---|---|---|---|
|
fdbcf36859 | ||
|
|
c6e096e00d | ||
|
|
08f04711c8 | ||
|
|
a9298f6262 | ||
|
|
38de94bdcb | ||
|
|
fe50089c1a | ||
|
|
835e0c2582 | ||
|
|
73b0b27c13 |
3
.github/ISSUE_TEMPLATE/new_installer.md
vendored
3
.github/ISSUE_TEMPLATE/new_installer.md
vendored
@@ -35,18 +35,15 @@ info, and doing a find and replace on a few file system path names.
|
||||
```
|
||||
2. Copy the example template and update with info from Official Releases:
|
||||
<https://github.com/___CHANGE/ME___/releases>
|
||||
|
||||
```bash
|
||||
rsync -av _example/ CHANGE-ME/
|
||||
```
|
||||
|
||||
- [ ] update `CHANGE-ME/release.js` to use the official repo
|
||||
- [ ] Learn how `CHANGE-ME` unpacks (i.e. as a single file? as a .tar.gz? as
|
||||
a .tar.gz with a folder named CHANGE-ME?)
|
||||
- [ ] find and replace to change the name
|
||||
- [ ] update `CHANGE-ME/install.sh` (see `bat` and `jq` as examples)
|
||||
- [ ] update `CHANGE-ME/install.ps1` (see `bat` and `jq` as examples)
|
||||
|
||||
3. Needs an updated tagline and cheat sheet
|
||||
- [ ] update `CHANGE-ME/README.md`
|
||||
- [ ] official URL
|
||||
|
||||
7
.github/workflows/node.js.yml
vendored
7
.github/workflows/node.js.yml
vendored
@@ -23,15 +23,10 @@ jobs:
|
||||
sh ./_scripts/install-ci-deps
|
||||
echo "${HOME}/.local/bin" >> $GITHUB_PATH
|
||||
echo "${HOME}/.local/opt/pwsh" >> $GITHUB_PATH
|
||||
- run: |
|
||||
git submodule init
|
||||
git submodule update
|
||||
- run: shfmt --version
|
||||
- run: shellcheck -V
|
||||
- run: node --version
|
||||
- run: npm run fmt
|
||||
- run: npm ci
|
||||
- run: npm run lint
|
||||
- env:
|
||||
GITHUB_TOKEN: ${{ github.token }}
|
||||
run: npm run test
|
||||
- run: npm run test
|
||||
|
||||
43
.gitignore
vendored
43
.gitignore
vendored
@@ -1,46 +1,7 @@
|
||||
# generated artifacts
|
||||
.env
|
||||
node_modules
|
||||
install-*.sh
|
||||
install-*.bat
|
||||
install-*.ps1
|
||||
|
||||
# Go build outputs (from go run/build in repo root)
|
||||
/classify
|
||||
/e2etest
|
||||
/fetchraw
|
||||
/inspect
|
||||
/uaparse
|
||||
/webicached
|
||||
/zigtest
|
||||
/distributables.csv
|
||||
|
||||
# local config
|
||||
.env.*
|
||||
*.env
|
||||
.env
|
||||
!example.env
|
||||
|
||||
# caches
|
||||
_cache/
|
||||
node_modules/
|
||||
|
||||
# local test fixtures (regenerated by _webi/test-live-*.js)
|
||||
testdata/
|
||||
LIVE_cache/
|
||||
distributables.csv
|
||||
|
||||
# temporary & backup files
|
||||
.*.sw*
|
||||
*.bak
|
||||
*.bak.*
|
||||
|
||||
# agent session files
|
||||
agents/
|
||||
LOCAL.md
|
||||
|
||||
# other
|
||||
.DS_Store
|
||||
desktop.ini
|
||||
.directory
|
||||
LIVE_cache
|
||||
/webid
|
||||
bin/
|
||||
|
||||
3
.gitmodules
vendored
3
.gitmodules
vendored
@@ -1,3 +0,0 @@
|
||||
[submodule "_webi/build-classifier"]
|
||||
path = _webi/build-classifier
|
||||
url = https://github.com/webinstall/webi-build-classifier.git
|
||||
@@ -1,2 +1 @@
|
||||
node_modules/**/*
|
||||
_webi/test-*.js
|
||||
DELETEMEnode_modules/**/*
|
||||
|
||||
@@ -1,7 +1,4 @@
|
||||
{
|
||||
"globals": {
|
||||
"AbortController": false
|
||||
},
|
||||
"browser": true,
|
||||
"node": true,
|
||||
"esversion": 11,
|
||||
|
||||
483
AGENTS.md
483
AGENTS.md
@@ -1,483 +0,0 @@
|
||||
# Webi Installers — Agent Guide
|
||||
|
||||
Webi installs dev tools to `~/.local/` without sudo. Each installer is a small
|
||||
package of 3-4 files. This guide tells you how to create and modify them.
|
||||
|
||||
## Domains
|
||||
|
||||
| Environment | Domains |
|
||||
| ----------- | ------------------------------------------------ |
|
||||
| Production | webi.sh, webi.ms, webinstall.dev |
|
||||
| Beta | beta.webi.sh, beta.webi.ms, beta.webinstall.dev |
|
||||
| Next | next.webi.sh, next.webi.ms, next.webinstall.dev |
|
||||
|
||||
- **webi.sh** — POSIX shell installer: `curl https://webi.sh/node | sh`
|
||||
- **webi.ms** — PowerShell installer: `curl.exe https://webi.ms/node | powershell`
|
||||
- **webinstall.dev** — canonical domain, serves both shell scripts and the
|
||||
browser-facing cheat sheet pages
|
||||
|
||||
The domain controls the default script format. You can force the same behavior
|
||||
on any domain via User-Agent:
|
||||
- `curl -A "MS" ...` — triggers PowerShell output (same as webi.ms)
|
||||
- `curl ...` (default UA) — triggers POSIX shell output (same as webi.sh)
|
||||
- `curl -A "$(uname -srm)" ...` — once the API is activated, the full
|
||||
`uname -srm` string (e.g. `Linux 6.1.0 x86_64`) guides OS/arch selection
|
||||
|
||||
## Why Webi Exists
|
||||
|
||||
Webi makes tool installation trivially repeatable for people who aren't
|
||||
sysadmins — freelance clients, junior devs, anyone who shouldn't have to care
|
||||
about PATH, permissions, or platform differences. Three things matter:
|
||||
|
||||
1. **Install without friction.** No sudo, no manual PATH edits, no "necessary
|
||||
but unimportant" steps leaking into the experience.
|
||||
2. **Know where things are.** The Files section tells you exactly what got
|
||||
created or modified. Nothing should be mysterious.
|
||||
3. **Copy-paste recipes.** The cheat sheet is what you'd send someone less
|
||||
experienced than yourself instead of a project's full README — scannable,
|
||||
concrete, easy to reference by name.
|
||||
|
||||
## Quick Start: Adding a New Installer
|
||||
|
||||
1. Identify the **package type** (see [Categories](#categories) below)
|
||||
2. Find an existing installer of the same type to use as a template
|
||||
3. Create `<name>/releases.js`, `install.sh`, `install.ps1`, `README.md`
|
||||
4. Test with the command in [Testing releases.js](#testing-releasesjs)
|
||||
5. Run formatters before committing (see [Code Style](#code-style))
|
||||
|
||||
## Directory Layout
|
||||
|
||||
```
|
||||
<package-name>/
|
||||
README.md # YAML frontmatter + docs
|
||||
releases.js # Fetches release metadata (Node.js)
|
||||
install.sh # POSIX shell installer (macOS/Linux)
|
||||
install.ps1 # PowerShell installer (Windows) — optional
|
||||
```
|
||||
|
||||
Key infrastructure directories (do not modify without good reason):
|
||||
|
||||
- `_webi/` — bootstrap templates, `transform-releases.js` (API endpoint),
|
||||
`builds-cacher.js` (reads cache JSON), `serve-installer.js` (installer
|
||||
scripts)
|
||||
- `_common/` — shared JS fetcher libraries (being phased out — Go daemon now
|
||||
fetches upstream)
|
||||
- `_example/` — canonical template for new packages
|
||||
- `_examples/` — specialized templates (goreleaser, xz-compressed)
|
||||
|
||||
## Categories
|
||||
|
||||
Ref: <https://github.com/webinstall/webi-installers/issues/412>
|
||||
|
||||
| Type | Description | Template to copy |
|
||||
| ----- | -------------------------------------- | ---------------- |
|
||||
| `bin` | Single binary in tar/zip | `koji`, `delta` |
|
||||
| `bin` | Single bare binary (no archive) | `arc`, `shfmt` |
|
||||
| `bin` | Goreleaser-style archives | `keypairs` |
|
||||
| 📦 | Self-contained package (bin/man/share) | `node`, `go` |
|
||||
| 📂 | Multiple binaries/scripts | `pg` |
|
||||
| 🔗 | Alias/redirect to another package | `ripgrep` → `rg` |
|
||||
| 📝 | Bespoke / custom install | `rustlang` |
|
||||
|
||||
## Data Architecture
|
||||
|
||||
There are two data paths. Both read from pre-generated cache — the Node.js
|
||||
server does NOT fetch upstream APIs.
|
||||
|
||||
```
|
||||
API path (JSON/TAB output):
|
||||
Request → transform-releases.js → ~/.cache/webi/legacy/{pkg}.json → filter + sort
|
||||
Vocabulary: macos, amd64, arm64, armv7l (API vocabulary)
|
||||
normalize.js: REMOVED — cache provides all fields directly
|
||||
|
||||
Installer path (bash/ps1 script output):
|
||||
Request → serve-installer.js → builds.js → builds-cacher.js
|
||||
Vocabulary: darwin, x86_64, aarch64 (build-classifier vocabulary)
|
||||
```
|
||||
|
||||
Cache is generated by the Go daemon (`webicached`) and stored flat in
|
||||
`~/.cache/webi/legacy/{pkg}.json` (resolved via `Os.homedir()` in the Node
|
||||
readers — no `_cache` symlink, no month subdirectory). Each file contains:
|
||||
- Top-level summary arrays: `oses`, `arches`, `libcs`, `formats`
|
||||
- `releases` array with pre-classified fields: `os`, `arch`, `libc`, `ext`,
|
||||
`version`, `channel`, `download`
|
||||
- `download` template string
|
||||
|
||||
### Canonical vocabulary (cache and API)
|
||||
|
||||
**OS**: `macos` (not darwin), `linux`, `windows`, `freebsd`, etc.
|
||||
**Arch**: `amd64` (not x86_64), `arm64` (not aarch64), `armv7l` (not armv7)
|
||||
**Libc**: `none` (never empty), `gnu`, `musl`, `msvc`
|
||||
**Ext**: `tar.gz`, `zip`, `exe` (no leading dot; `exe` for bare binaries)
|
||||
|
||||
## releases.js (legacy — being phased out)
|
||||
|
||||
The `{pkg}/releases.js` files previously fetched upstream release metadata.
|
||||
These are being replaced by the Go cache daemon. Existing files are kept as
|
||||
documentation of release sources but are no longer called by the server.
|
||||
|
||||
### Testing the API (current)
|
||||
|
||||
```sh
|
||||
curl -sS 'https://beta.webi.sh/api/releases/<name>.json?os=macos&arch=arm64&limit=5' | jq .
|
||||
```
|
||||
|
||||
Verify: versions present, correct OS/arch vocabulary, download URLs resolve.
|
||||
|
||||
## install.sh
|
||||
|
||||
POSIX shell (`sh`, not bash). Always wrapped in a function:
|
||||
|
||||
```sh
|
||||
#!/bin/sh
|
||||
# shellcheck disable=SC2034
|
||||
|
||||
set -e
|
||||
set -u
|
||||
|
||||
__init_pkgname() {
|
||||
# These 6 variables are required
|
||||
pkg_cmd_name="cmd"
|
||||
|
||||
pkg_dst_cmd="$HOME/.local/bin/cmd"
|
||||
pkg_dst="$pkg_dst_cmd"
|
||||
|
||||
pkg_src_cmd="$HOME/.local/opt/cmd-v$WEBI_VERSION/bin/cmd"
|
||||
pkg_src_dir="$HOME/.local/opt/cmd-v$WEBI_VERSION"
|
||||
pkg_src="$pkg_src_cmd"
|
||||
|
||||
pkg_install() {
|
||||
mkdir -p "$(dirname "$pkg_src_cmd")"
|
||||
mv ./cmd "$pkg_src_cmd"
|
||||
}
|
||||
|
||||
pkg_get_current_version() {
|
||||
cmd --version 2> /dev/null | head -n 1 | cut -d' ' -f2
|
||||
}
|
||||
}
|
||||
|
||||
__init_pkgname
|
||||
```
|
||||
|
||||
### Framework variables available in install.sh
|
||||
|
||||
Set by the webi bootstrap (`_webi/package-install.tpl.sh`):
|
||||
|
||||
| Variable | Example | Description |
|
||||
| --------------- | ------------------- | --------------------- |
|
||||
| `WEBI_VERSION` | `1.2.3` | Selected version |
|
||||
| `WEBI_PKG_URL` | `https://...` | Download URL |
|
||||
| `WEBI_PKG_FILE` | `foo-v1.2.3.tar.gz` | Download filename |
|
||||
| `WEBI_OS` | `linux` | Detected OS |
|
||||
| `WEBI_ARCH` | `amd64` | Detected architecture |
|
||||
| `WEBI_EXT` | `tar.gz` | Archive extension |
|
||||
| `WEBI_CHANNEL` | `stable` | Release channel |
|
||||
| `PKG_NAME` | `foo` | Package name |
|
||||
|
||||
### Override functions
|
||||
|
||||
| Function | Purpose |
|
||||
| --------------------------- | --------------------------------------------- |
|
||||
| `pkg_install()` | **Required.** Move files to `$pkg_src` |
|
||||
| `pkg_get_current_version()` | Parse installed version from command output |
|
||||
| `pkg_post_install()` | Post-install setup (git config, shell config) |
|
||||
| `pkg_done_message()` | Custom completion message |
|
||||
| `pkg_link()` | Override default symlink behavior |
|
||||
| `pkg_pre_install()` | Custom pre-install logic |
|
||||
|
||||
### Framework helper functions
|
||||
|
||||
| Function | Purpose |
|
||||
| ------------------------ | ---------------------------------- |
|
||||
| `webi_download()` | Download package if not cached |
|
||||
| `webi_extract()` | Extract archive by extension |
|
||||
| `webi_path_add <dir>` | Add to PATH via envman |
|
||||
| `webi_link()` | Create versioned symlinks |
|
||||
| `webi_check_installed()` | Check if version already installed |
|
||||
|
||||
### pkg_install patterns
|
||||
|
||||
**Bare binary in archive root:**
|
||||
|
||||
```sh
|
||||
mv ./cmd "$pkg_src_cmd"
|
||||
```
|
||||
|
||||
**Binary in a subdirectory (goreleaser-style `cmd-OS-arch/cmd`):**
|
||||
|
||||
```sh
|
||||
mv ./cmd-*/cmd "$pkg_src_cmd"
|
||||
```
|
||||
|
||||
**Flexible detection (handles multiple archive layouts):**
|
||||
|
||||
```sh
|
||||
if test -f ./cmd; then
|
||||
mv ./cmd "$pkg_src_cmd"
|
||||
elif test -e ./cmd-*/cmd; then
|
||||
mv ./cmd-*/cmd "$pkg_src_cmd"
|
||||
elif test -e ./cmd-*; then
|
||||
mv ./cmd-* "$pkg_src_cmd"
|
||||
fi
|
||||
```
|
||||
|
||||
## install.ps1
|
||||
|
||||
PowerShell for Windows. Uses `$Env:` variables. See `_example/install.ps1` for
|
||||
the full template. Key differences from install.sh:
|
||||
|
||||
- Paths use backslashes, commands end in `.exe`
|
||||
- `$Env:USERPROFILE` instead of `$HOME`
|
||||
- `Test-Path`, `Move-Item`, `Copy-Item` instead of shell equivalents
|
||||
- Downloads go to `$Env:USERPROFILE\Downloads\webi\`
|
||||
- Temp work in `.local\tmp`, use `Push-Location`/`Pop-Location`
|
||||
- Symlinks done via `Copy-Item` (not actual symlinks)
|
||||
|
||||
## README.md
|
||||
|
||||
````markdown
|
||||
---
|
||||
title: toolname
|
||||
homepage: https://github.com/owner/repo
|
||||
tagline: |
|
||||
toolname: A short one-line description.
|
||||
---
|
||||
|
||||
To update or switch versions, run `webi toolname@stable` (or `@v2`, `@beta`,
|
||||
etc).
|
||||
|
||||
### Files
|
||||
|
||||
These are the files that are created and/or modified with this installer:
|
||||
|
||||
```text
|
||||
~/.config/envman/PATH.env
|
||||
~/.local/bin/toolname
|
||||
~/.local/opt/toolname-VERSION/bin/toolname
|
||||
```
|
||||
|
||||
## Cheat Sheet
|
||||
|
||||
> `toolname` does X. Brief description.
|
||||
|
||||
### How to use toolname
|
||||
|
||||
```sh
|
||||
toolname --example
|
||||
```
|
||||
````
|
||||
|
||||
Note: **Files goes above Cheat Sheet**, not inside it.
|
||||
|
||||
### Cheat Sheet tone and style
|
||||
|
||||
Webi cheat sheets are **opinionated quick-reference guides**, not comprehensive
|
||||
documentation. Think "colleague's sticky note" — not the project's official
|
||||
README.
|
||||
|
||||
The tool is the topic, but **the problem is the reason**. Cheat sheets are
|
||||
organized around tasks the reader already wants to do — the tool is how they get
|
||||
there. Headings reference the tool (the reader came to this page on purpose),
|
||||
but the content solves the underlying problem completely:
|
||||
|
||||
- "How to reverse proxy to Node" (caddy knowledge, not just node)
|
||||
- "How to run a Node app as a System Service" (serviceman knowledge)
|
||||
- "How to Enable Secure Remote Postgres Access" (openssl, pg_hba.conf, systemd)
|
||||
- "How to manually configure git to use delta" (gitconfig, not delta flags)
|
||||
- "How to make fish the default shell in iTerm2" (iTerm2 knowledge, not fish)
|
||||
|
||||
The reader's question is "how do I do X?" and the cheat sheet answers it
|
||||
completely — including configs, adjacent tools, and platform-specific
|
||||
variations. A goreleaser cheat sheet teaches you goreleaser YAML. A postgres
|
||||
cheat sheet teaches you pg_hba.conf, openssl certs, and systemd units.
|
||||
|
||||
Cheat sheets cross tool boundaries freely. Node's references caddy, serviceman,
|
||||
setcap-netbind, GitHub Actions. Postgres references serviceman, openrc, launchd.
|
||||
They link to each other's webi pages. The scope is "everything you need to
|
||||
accomplish this task," not "everything this one binary does."
|
||||
|
||||
They show the actual files and configs that matter — not documentation _about_
|
||||
configs, but the configs themselves, copy-pasteable, with inline comments
|
||||
explaining the non-obvious parts.
|
||||
|
||||
**Guidelines:**
|
||||
|
||||
- **Show the 3-5 things someone will actually do**, with copy-pasteable
|
||||
commands. Skip exhaustive flag lists and API docs.
|
||||
- **Lead with practical integration.** Show the exact `git config` lines, the
|
||||
exact hook script, the exact shell alias — don't just explain the feature and
|
||||
leave wiring up to the reader.
|
||||
- **Skip what they already know.** No need to re-explain what the tool is at
|
||||
length — the tagline and one-liner blockquote handle that. Get to the
|
||||
commands.
|
||||
- **Prefer concrete over abstract.** Instead of "you can configure X via a
|
||||
config file", show the config file contents.
|
||||
|
||||
## Shell Naming Conventions
|
||||
|
||||
**Variables:**
|
||||
|
||||
- `ALL_CAPS` — environment variables only (`PATH`, `HOME`, `WEBI_VERSION`)
|
||||
- `b_varname` — block-scoped (inside a function, loop, or conditional)
|
||||
- `g_varname` — global to the script (and sourced scripts)
|
||||
- `a_varname` — function arguments
|
||||
|
||||
**Functions and commands:**
|
||||
|
||||
- `fn_name` — helper functions (anything other than the script's main/entry
|
||||
function)
|
||||
- `cmd_name` — command aliases, e.g. `cmd_curl='curl --fail-with-body -sSL'`
|
||||
|
||||
## Code Style
|
||||
|
||||
Requires `node`, `shfmt`, `pwsh`, and `pwsh-essentials` (install all via webi).
|
||||
Run before committing:
|
||||
|
||||
```sh
|
||||
npm run fmt # prettier (JS/MD) + shfmt (sh) + pwsh-fmt (ps1)
|
||||
npm run lint # jshint + shellcheck + pwsh-lint
|
||||
```
|
||||
|
||||
Commit messages: `feat(<pkg>): add installer`, `fix(<pkg>): update install.sh`,
|
||||
`docs(<pkg>): add cheat sheet`.
|
||||
|
||||
## Naming Conventions
|
||||
|
||||
- The canonical package name is the **command name** you type: `go`, `node`,
|
||||
`rg`
|
||||
- The alternate/alias name is the project name: `golang`, `nodejs`, `ripgrep`
|
||||
- Package directories are lowercase with hyphens
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **Monorepo releases**: The GitHub API returns ALL releases for the repo. You
|
||||
must filter in `releases.js` and strip the tag prefix from the version.
|
||||
- **No `--version` flag**: Some tools lack version introspection. Comment out
|
||||
`pkg_get_current_version` — webi still works, it just can't skip reinstalls.
|
||||
- **Cache directory**: `builds-cacher.js` and `transform-releases.js` read
|
||||
exclusively from `~/.cache/webi/legacy/{pkg}.json` (resolved via
|
||||
`Os.homedir()`). No date bucketing, no `_cache` symlink, no month rollover
|
||||
hazard. If a package returns `0.0.0`, check that `~/.cache/webi/legacy/{pkg}.json`
|
||||
exists and that `webicached` is running.
|
||||
- **Goreleaser archives**: Typically contain a bare binary at the archive root
|
||||
(not nested in a directory). Use `mv ./cmd "$pkg_src_cmd"`.
|
||||
|
||||
---
|
||||
|
||||
## Go Cache Daemon
|
||||
|
||||
The Go pipeline (`cmd/webicached`) replaces the Node.js release-fetching code.
|
||||
It reads `releases.conf` files, fetches upstream release metadata, classifies
|
||||
build assets, and writes to `~/.cache/webi/legacy/` in the format the Node.js server expects.
|
||||
|
||||
### Canonical Vocabulary
|
||||
|
||||
The classifier MUST use exactly these strings. They match the production API.
|
||||
|
||||
**OS**: `macos` (NOT `darwin`), `linux`, `windows`, `freebsd`, `openbsd`,
|
||||
`netbsd`, `dragonfly`, `aix`, `illumos`, `plan9`, `solaris`, `posix_2017`
|
||||
|
||||
**Arch** — exact equivalences:
|
||||
- `amd64` (NOT `x86_64`), `x86` (NOT `i386`/`i686`/`386`)
|
||||
- `arm64` (NOT `aarch64`)
|
||||
- `armv7l` (NOT `armv7`), `armv6l` (NOT `armv6`)
|
||||
- `mipsle` (NOT `mipsel`), `mips64le` (NOT `mips64el`)
|
||||
|
||||
**Arch** — compatibility downcasts:
|
||||
- `armhf` → `armv7l`, `armv7a` → `armv7l`, `armel` → `arm`
|
||||
|
||||
**Arch** — other: `arm`, `ppc64le`, `ppc64`, `loong64`, `riscv64`, `s390x`,
|
||||
`mips`, `mips64`
|
||||
|
||||
**Libc**: `none` (never empty), `gnu`, `musl`, `msvc`
|
||||
|
||||
**Ext**: `tar.gz`, `tar.xz`, `zip`, `exe`, `7z`, `pkg`, `msi`
|
||||
(no leading dot; `exe` for bare binaries)
|
||||
|
||||
### releases.conf
|
||||
|
||||
Each package directory contains a `releases.conf` that tells the daemon where
|
||||
to fetch releases. Format is `key = value`, one per line. `#` comments and
|
||||
blank lines are ignored.
|
||||
|
||||
#### Source types (mutually exclusive — pick one)
|
||||
|
||||
```ini
|
||||
# GitHub binary releases (most common)
|
||||
github_releases = sharkdp/bat
|
||||
|
||||
# GitHub source tarballs (with optional git fallback)
|
||||
github_sources = bnnanet/serviceman
|
||||
git_url = https://github.com/bnnanet/serviceman.git
|
||||
|
||||
# Git tag enumeration (vim plugins, shell scripts — git_url alone)
|
||||
git_url = https://github.com/tpope/vim-commentary.git
|
||||
|
||||
# Gitea (full URL required, or short form + base_url)
|
||||
gitea_releases = https://git.rootprojects.org/root/pathman
|
||||
|
||||
# GitLab (defaults to gitlab.com)
|
||||
gitlab_releases = owner/repo
|
||||
|
||||
# HashiCorp releases API
|
||||
hashicorp_product = terraform
|
||||
|
||||
# Custom source (servicemandist, nodedist, zigdist, etc.)
|
||||
source = nodedist
|
||||
url = https://nodejs.org/download/release
|
||||
```
|
||||
|
||||
#### Filtering, versioning, and platform
|
||||
|
||||
```ini
|
||||
tag_prefix = bun- # monorepo: strip prefix from version
|
||||
version_prefixes = jq- # strip from version string (space-separated)
|
||||
asset_filter = MinGit # filename must contain this substring
|
||||
exclude = busybox -src- -docs- # skip assets containing these (space-separated)
|
||||
os = posix_2017 # restrict ALL versions to this OS (blanket)
|
||||
alias_of = rg # mirrors another package's releases
|
||||
```
|
||||
|
||||
#### Design rules
|
||||
|
||||
- `os` is a blanket tag on ALL versions. Only use for packages that are always
|
||||
POSIX-only. For version-dependent OS tagging, use a custom `TagVariants` in
|
||||
`internal/releases/{pkg}/variants.go`.
|
||||
- `git_url` can be primary (gittag source when it's the only key) or secondary
|
||||
fallback alongside `github_sources`/`gitea_sources`.
|
||||
- Full URL forms accepted for github/gitea/gitlab (e.g.
|
||||
`github_releases = https://github.com/sharkdp/bat`).
|
||||
|
||||
### Testing
|
||||
|
||||
Test tools: `cmd/e2etest` (pipeline comparison), `cmd/comparecache` (cache diff),
|
||||
`cmd/inspect` (single-package debug). Run each with `--help` for usage.
|
||||
|
||||
### Classifier vs Per-Package Tagger
|
||||
|
||||
The general classifier (`internal/classify/`) handles patterns common across
|
||||
many projects. It MUST NOT contain one-off logic for a single package.
|
||||
|
||||
Per-package taggers (`internal/releases/{pkg}/variants.go`) handle
|
||||
project-specific knowledge. Read the existing taggers for conventions.
|
||||
|
||||
MUST: Derive arch/OS from concrete evidence — not blanket defaults.
|
||||
MUST: New general classifier patterns must apply to 2-3+ packages.
|
||||
|
||||
### Deploying
|
||||
|
||||
```sh
|
||||
./scripts/deploy-webicached.sh beta.webi.sh
|
||||
./scripts/deploy-webicached.sh next.webi.sh
|
||||
```
|
||||
|
||||
First-time setup on a new host uses `serviceman`:
|
||||
|
||||
```sh
|
||||
serviceman add --name webicached \
|
||||
--workdir ~/srv/webid/installers/ -- \
|
||||
~/bin/webicached \
|
||||
--envfile ~/srv/webid/.env.secret \
|
||||
--conf ~/srv/webid/installers/ \
|
||||
--raw ~/.cache/webi/raw
|
||||
```
|
||||
@@ -10,6 +10,7 @@
|
||||
|
||||
- You'll be asked to make changes if you don't run the code formatters and
|
||||
linters:
|
||||
|
||||
- Node / JavaScript:
|
||||
- [prettier](https://webinstall.dev/prettier)
|
||||
```sh
|
||||
@@ -20,6 +21,7 @@
|
||||
npm run lint
|
||||
```
|
||||
- Bash
|
||||
|
||||
- [shfmt](https://webinstall.dev/shfmt)
|
||||
```sh
|
||||
npm run shfmt
|
||||
|
||||
11
README.md
11
README.md
@@ -98,10 +98,9 @@ You just fill in the blanks.
|
||||
Just create an empty directory and run the tests until you get a good result.
|
||||
|
||||
```sh
|
||||
git clone git@github.com:webinstall/webi-installers.git
|
||||
pushd ./webi-installers/
|
||||
git submodule update --init
|
||||
npm clean-install
|
||||
git clone git@github.com:webinstall/packages.git
|
||||
pushd packages
|
||||
npm install
|
||||
```
|
||||
|
||||
```sh
|
||||
@@ -145,8 +144,8 @@ It looks like this:
|
||||
`releases.js`:
|
||||
|
||||
```js
|
||||
module.exports = function () {
|
||||
return github(null, owner, repo).then(function (all) {
|
||||
module.exports = function (request) {
|
||||
return github(request, owner, repo).then(function (all) {
|
||||
// if you need to do something special, you can do it here
|
||||
// ...
|
||||
return all;
|
||||
|
||||
64
_common/brew.js
Normal file
64
_common/brew.js
Normal file
@@ -0,0 +1,64 @@
|
||||
'use strict';
|
||||
|
||||
/**
|
||||
* Gets a releases from 'brew'.
|
||||
*
|
||||
* @param request
|
||||
* @param {string} formula
|
||||
* @returns {PromiseLike<any> | Promise<any>}
|
||||
*/
|
||||
function getAllReleases(request, formula) {
|
||||
if (!formula) {
|
||||
return Promise.reject('missing formula for brew');
|
||||
}
|
||||
return request({
|
||||
url: 'https://formulae.brew.sh/api/formula/' + formula + '.json',
|
||||
fail: true, // https://git.coolaj86.com/coolaj86/request.js/issues/2
|
||||
json: true,
|
||||
})
|
||||
.then(failOnBadStatus)
|
||||
.then(function (resp) {
|
||||
var ver = resp.body.versions.stable;
|
||||
var dl = (
|
||||
resp.body.bottle.stable.files.high_sierra ||
|
||||
resp.body.bottle.stable.files.catalina
|
||||
).url.replace(new RegExp(ver.replace(/\./g, '\\.'), 'g'), '{{ v }}');
|
||||
return [
|
||||
{
|
||||
version: ver,
|
||||
download: dl.replace(/{{ v }}/g, ver),
|
||||
},
|
||||
].concat(
|
||||
resp.body.versioned_formulae.map(function (f) {
|
||||
var ver = f.replace(/.*@/, '');
|
||||
return {
|
||||
version: ver,
|
||||
download: dl,
|
||||
};
|
||||
}),
|
||||
);
|
||||
})
|
||||
.catch(function (err) {
|
||||
console.error('Error fetching MariaDB versions (brew)');
|
||||
console.error(err);
|
||||
return [];
|
||||
});
|
||||
}
|
||||
|
||||
function failOnBadStatus(resp) {
|
||||
if (resp.statusCode >= 400) {
|
||||
var err = new Error('Non-successful status code: ' + resp.statusCode);
|
||||
err.code = 'ESTATUS';
|
||||
err.response = resp;
|
||||
throw err;
|
||||
}
|
||||
return resp;
|
||||
}
|
||||
|
||||
module.exports = getAllReleases;
|
||||
|
||||
if (module === require.main) {
|
||||
getAllReleases(require('@root/request'), 'mariadb').then(function (all) {
|
||||
console.info(JSON.stringify(all, null, 2));
|
||||
});
|
||||
}
|
||||
218
_common/git-tag.js
Normal file
218
_common/git-tag.js
Normal file
@@ -0,0 +1,218 @@
|
||||
'use strict';
|
||||
|
||||
require('dotenv').config({ path: '.env' });
|
||||
|
||||
var Crypto = require('crypto');
|
||||
var util = require('util');
|
||||
var exec = util.promisify(require('child_process').exec);
|
||||
var Fs = require('node:fs/promises');
|
||||
var FsSync = require('node:fs');
|
||||
var Path = require('node:path');
|
||||
|
||||
var repoBaseDir = process.env.REPO_BASE_DIR || '';
|
||||
if (!repoBaseDir) {
|
||||
repoBaseDir = Path.resolve('./_repos');
|
||||
// for stderr
|
||||
console.error(`[Warn] REPO_BASE_DIR= not set, ${repoBaseDir}`);
|
||||
}
|
||||
|
||||
var Repos = {};
|
||||
|
||||
Repos.clone = async function (repoPath, gitUrl) {
|
||||
let uuid = Crypto.randomUUID();
|
||||
let tmpPath = `${repoPath}.${uuid}.tmp`;
|
||||
let bakPath = `${repoPath}.${uuid}.dup`;
|
||||
await exec(`git clone --bare --filter=tree:0 ${gitUrl} ${tmpPath}`);
|
||||
|
||||
try {
|
||||
FsSync.accessSync(repoPath);
|
||||
return;
|
||||
} catch (e) {
|
||||
if (e.code !== 'ENOENT') {
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
// sync to avoid race conditions
|
||||
try {
|
||||
FsSync.renameSync(repoPath, bakPath);
|
||||
} catch (e) {
|
||||
if (e.code !== 'ENOENT') {
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
FsSync.renameSync(tmpPath, repoPath);
|
||||
|
||||
await Fs.rm(bakPath, { force: true, recursive: true });
|
||||
};
|
||||
|
||||
Repos.checkExists = async function (repoPath) {
|
||||
let err = await Fs.access(repoPath).catch(Object);
|
||||
if (!err) {
|
||||
return true;
|
||||
}
|
||||
|
||||
if (err.code !== 'ENOENT') {
|
||||
throw err;
|
||||
}
|
||||
return false;
|
||||
};
|
||||
|
||||
Repos.fetch = async function (repoPath) {
|
||||
await exec(`git --git-dir=${repoPath} fetch`);
|
||||
};
|
||||
|
||||
Repos.getTags = async function (repoPath) {
|
||||
var { stdout } = await exec(`git --git-dir=${repoPath} tag`);
|
||||
var rawTags = stdout.trim().split('\n');
|
||||
|
||||
let tags = [];
|
||||
for (let tag of rawTags) {
|
||||
// ex: v1, v2, v1.1, 1.1.0-rc
|
||||
let maybeVersionRe = /^(v\d+|v?\d+\.\d+)/;
|
||||
let maybeVersion = maybeVersionRe.test(tag);
|
||||
if (maybeVersion) {
|
||||
tags.push(tag);
|
||||
}
|
||||
}
|
||||
|
||||
tags = tags.reverse();
|
||||
return tags;
|
||||
};
|
||||
|
||||
Repos.getTipInfo = async function (repoPath) {
|
||||
var { stdout } = await exec(
|
||||
`git --git-dir=${repoPath} rev-parse --abbrev-ref HEAD`,
|
||||
);
|
||||
var branch = stdout.trim();
|
||||
|
||||
var info = await Repos.getCommitInfo(repoPath, 'HEAD');
|
||||
info.commitish = branch;
|
||||
|
||||
return info;
|
||||
};
|
||||
|
||||
Repos.getCommitInfo = async function (repoPath, commitish) {
|
||||
var { stdout } = await exec(
|
||||
`git --git-dir=${repoPath} log -1 --format="%h %H %ad %cd" --date=iso-strict ${commitish}`,
|
||||
);
|
||||
stdout = stdout.trim();
|
||||
var commitParts = stdout.split(/\s+/g);
|
||||
return {
|
||||
commitish: commitish,
|
||||
commit_id: `${commitParts[0]}`,
|
||||
commit: `${commitParts[1]}`,
|
||||
date: commitParts[2],
|
||||
date_authored: commitParts[3],
|
||||
};
|
||||
};
|
||||
|
||||
/**
|
||||
* Lists GitHub Releases (w/ uploaded assets)
|
||||
*
|
||||
* @param request
|
||||
* @param {string} owner
|
||||
* @param {string} gitUrl
|
||||
* @returns {PromiseLike<any> | Promise<any>}
|
||||
*/
|
||||
async function getAllReleases(gitUrl) {
|
||||
let all = {
|
||||
releases: [],
|
||||
download: '',
|
||||
};
|
||||
|
||||
let repoName = gitUrl.split('/').pop();
|
||||
repoName = repoName.replace(/\.git$/, '');
|
||||
|
||||
let repoPath = `${repoBaseDir}/${repoName}.git`;
|
||||
|
||||
let isCloned = await Repos.checkExists(repoPath);
|
||||
if (!isCloned) {
|
||||
await Repos.clone(repoPath, gitUrl);
|
||||
} else {
|
||||
await Repos.fetch(repoPath);
|
||||
}
|
||||
|
||||
let commitInfos = [];
|
||||
let tags = await Repos.getTags(repoPath);
|
||||
for (let tag of tags) {
|
||||
let commitInfo = await Repos.getCommitInfo(repoPath, tag);
|
||||
Object.assign(commitInfo, { version: tag, channel: '' });
|
||||
commitInfos.push(commitInfo);
|
||||
}
|
||||
|
||||
{
|
||||
let tipInfo = await Repos.getTipInfo(repoPath);
|
||||
// "2024-01-01T00:00:00-05:00" => "2024-01-01T05:00:00"
|
||||
let date = new Date(tipInfo.date);
|
||||
// "2024-01-01T05:00:00" => "v2024.01.01-05.00.00"
|
||||
let version = date.toISOString();
|
||||
// strip '.000Z'
|
||||
version = version.replace(/\.\d+Z/, '');
|
||||
version = version.replace(/[:\-]/g, '.');
|
||||
version = version.replace(/T/, '-');
|
||||
Object.assign(tipInfo, { version: `v${version}`, channel: '' });
|
||||
if (commitInfos.length > 1) {
|
||||
tipInfo.channel = 'beta';
|
||||
}
|
||||
commitInfos.push(tipInfo);
|
||||
}
|
||||
|
||||
let releases = [];
|
||||
for (let commitInfo of commitInfos) {
|
||||
let version = commitInfo.version.replace(/^v/, '');
|
||||
|
||||
let date = new Date(commitInfo.date);
|
||||
let isoDate = date.toISOString();
|
||||
isoDate = isoDate.replace(/\.\d+Z/, '');
|
||||
|
||||
// tags and HEAD qualify for '--branch <branchish>'
|
||||
let branch = commitInfo.commitish;
|
||||
|
||||
let rel = {
|
||||
name: `${repoName}-v${version}`,
|
||||
version: version,
|
||||
git_tag: branch,
|
||||
git_commit_hash: commitInfo.commit_id,
|
||||
lts: false,
|
||||
channel: commitInfo.channel,
|
||||
date: isoDate,
|
||||
os: '*',
|
||||
arch: '*',
|
||||
ext: 'git',
|
||||
download: gitUrl,
|
||||
};
|
||||
|
||||
releases.push(rel);
|
||||
}
|
||||
all.releases = releases;
|
||||
|
||||
return all;
|
||||
}
|
||||
|
||||
module.exports = getAllReleases;
|
||||
|
||||
if (module === require.main) {
|
||||
(async function main() {
|
||||
let testRepos = [
|
||||
// just a few tags, and a different HEAD
|
||||
'https://github.com/tpope/vim-commentary.git',
|
||||
// no tags, just HEAD
|
||||
'https://github.com/ziglang/zig.vim.git',
|
||||
// many, many tags
|
||||
//'https://github.com/dense-analysis/ale.git',
|
||||
];
|
||||
for (let url of testRepos) {
|
||||
let all = await getAllReleases(url);
|
||||
|
||||
all = require('../_webi/normalize.js')(all);
|
||||
console.info(JSON.stringify(all, null, 2));
|
||||
}
|
||||
})()
|
||||
.then(function () {
|
||||
process.exit(0);
|
||||
})
|
||||
.catch(function (err) {
|
||||
console.error(err);
|
||||
});
|
||||
}
|
||||
39
_common/gitea.js
Normal file
39
_common/gitea.js
Normal file
@@ -0,0 +1,39 @@
|
||||
'use strict';
|
||||
|
||||
var ghRelease = require('./github.js');
|
||||
|
||||
/**
|
||||
* Gets the releases for 'ripgrep'. This function could be trimmed down and made
|
||||
* for use with any github release.
|
||||
*
|
||||
* @param request
|
||||
* @param {string} owner
|
||||
* @param {string} repo
|
||||
* @returns {PromiseLike<any> | Promise<any>}
|
||||
*/
|
||||
function getAllReleases(request, owner, repo, baseurl) {
|
||||
if (!baseurl) {
|
||||
return Promise.reject('missing baseurl');
|
||||
}
|
||||
return ghRelease(request, owner, repo, baseurl + '/api/v1').then(
|
||||
function (all) {
|
||||
return all;
|
||||
},
|
||||
);
|
||||
}
|
||||
|
||||
module.exports = getAllReleases;
|
||||
|
||||
if (module === require.main) {
|
||||
getAllReleases(
|
||||
require('@root/request'),
|
||||
'coolaj86',
|
||||
'go-pathman',
|
||||
'https://git.coolaj86.com',
|
||||
).then(
|
||||
//getAllReleases(require('@root/request'), 'root', 'serviceman', 'https://git.rootprojects.org').then(
|
||||
function (all) {
|
||||
console.info(JSON.stringify(all, null, 2));
|
||||
},
|
||||
);
|
||||
}
|
||||
133
_common/github-source.js
Normal file
133
_common/github-source.js
Normal file
@@ -0,0 +1,133 @@
|
||||
'use strict';
|
||||
|
||||
require('dotenv').config();
|
||||
|
||||
/**
|
||||
* Gets the releases for 'ripgrep'. This function could be trimmed down and made
|
||||
* for use with any github release.
|
||||
*
|
||||
* @param request
|
||||
* @param {string} owner
|
||||
* @param {string} repo
|
||||
* @returns {PromiseLike<any> | Promise<any>}
|
||||
*/
|
||||
async function getAllReleases(
|
||||
request,
|
||||
owner,
|
||||
repo,
|
||||
oses,
|
||||
arches,
|
||||
baseurl = 'https://api.github.com',
|
||||
) {
|
||||
if (!owner) {
|
||||
return Promise.reject('missing owner for repo');
|
||||
}
|
||||
if (!repo) {
|
||||
return Promise.reject('missing repo name');
|
||||
}
|
||||
|
||||
let req = {
|
||||
url: `${baseurl}/repos/${owner}/${repo}/releases`,
|
||||
json: true,
|
||||
};
|
||||
|
||||
// TODO I really don't like global config, find a way to do better
|
||||
if (process.env.GITHUB_USERNAME) {
|
||||
req.auth = {
|
||||
user: process.env.GITHUB_USERNAME,
|
||||
pass: process.env.GITHUB_TOKEN,
|
||||
};
|
||||
}
|
||||
|
||||
let resp = await request(req);
|
||||
let gHubResp = resp.body;
|
||||
let all = {
|
||||
releases: [],
|
||||
// TODO make this ':baseurl' + ':releasename'
|
||||
download: '',
|
||||
};
|
||||
|
||||
for (let release of gHubResp) {
|
||||
// TODO tags aren't always semver / sensical
|
||||
let tag = release['tag_name'];
|
||||
let lts = /(\b|_)(lts)(\b|_)/.test(release['tag_name']);
|
||||
let channel = 'stable';
|
||||
if (release['prerelease']) {
|
||||
channel = 'beta';
|
||||
}
|
||||
let date = release['published_at'] || '';
|
||||
date = date.replace(/T.*/, '');
|
||||
|
||||
let urls = [release.tarball_url, release.zipball_url];
|
||||
for (let url of urls) {
|
||||
let resp = await request({
|
||||
method: 'HEAD',
|
||||
followRedirect: true,
|
||||
followAllRedirects: true,
|
||||
followOriginalHttpMethod: true,
|
||||
url: url,
|
||||
stream: true,
|
||||
});
|
||||
// Workaround for bug where method changes to GET
|
||||
resp.destroy();
|
||||
|
||||
// content-disposition: attachment; filename=BeyondCodeBootcamp-DuckDNS.sh-v1.0.1-0-ga2f4bde.zip
|
||||
let name = resp.headers['content-disposition'].replace(
|
||||
/.*filename=([^;]+)(;|$)/,
|
||||
'$1',
|
||||
);
|
||||
all.releases.push({
|
||||
name: name,
|
||||
version: tag,
|
||||
lts: lts,
|
||||
channel: channel,
|
||||
date: date,
|
||||
os: '', // will be guessed by download filename
|
||||
arch: '', // will be guessed by download filename
|
||||
ext: '', // will be normalized
|
||||
download: resp.request.uri.href,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (oses) {
|
||||
return combinate(all, oses, arches);
|
||||
}
|
||||
|
||||
return all;
|
||||
}
|
||||
|
||||
function combinate(all, oses, arches) {
|
||||
let releases = all.releases;
|
||||
// ex: arches = ['amd64', 'arm64', 'armv7l', 'armv6l', 'x86'];
|
||||
// ex: oses = ['macos', 'linux', 'bsd', 'posix'];
|
||||
|
||||
let combos = [];
|
||||
for (let release of releases) {
|
||||
for (let arch of arches) {
|
||||
for (let os of oses) {
|
||||
let combo = {
|
||||
arch: arch,
|
||||
os: os,
|
||||
};
|
||||
let rel = Object.assign({}, release, combo);
|
||||
combos.push(rel);
|
||||
}
|
||||
}
|
||||
}
|
||||
all.releases = combos;
|
||||
|
||||
return all;
|
||||
}
|
||||
|
||||
module.exports = getAllReleases;
|
||||
|
||||
if (module === require.main) {
|
||||
getAllReleases(
|
||||
require('@root/request'),
|
||||
'BeyondCodeBootcamp',
|
||||
'DuckDNS.sh',
|
||||
).then(function (all) {
|
||||
console.info(JSON.stringify(all, null, 2));
|
||||
});
|
||||
}
|
||||
102
_common/github.js
Normal file
102
_common/github.js
Normal file
@@ -0,0 +1,102 @@
|
||||
'use strict';
|
||||
|
||||
require('dotenv').config();
|
||||
|
||||
/**
|
||||
* Lists GitHub Releases (w/ uploaded assets)
|
||||
*
|
||||
* @param request
|
||||
* @param {string} owner
|
||||
* @param {string} repo
|
||||
* @returns {PromiseLike<any> | Promise<any>}
|
||||
*/
|
||||
async function getAllReleases(
|
||||
request,
|
||||
owner,
|
||||
repo,
|
||||
baseurl = 'https://api.github.com',
|
||||
) {
|
||||
if (!owner) {
|
||||
throw new Error('missing owner for repo');
|
||||
}
|
||||
if (!repo) {
|
||||
throw new Error('missing repo name');
|
||||
}
|
||||
|
||||
var req = {
|
||||
url: `${baseurl}/repos/${owner}/${repo}/releases`,
|
||||
json: true,
|
||||
};
|
||||
|
||||
// TODO I really don't like global config, find a way to do better
|
||||
if (process.env.GITHUB_USERNAME) {
|
||||
req.auth = {
|
||||
user: process.env.GITHUB_USERNAME,
|
||||
pass: process.env.GITHUB_TOKEN,
|
||||
};
|
||||
}
|
||||
|
||||
let resp = await request(req);
|
||||
if (!resp.ok) {
|
||||
console.error('Bad Resp Headers:', resp.headers);
|
||||
console.error('Bad Resp Body:', resp.body);
|
||||
throw new Error('the elusive releases BOOGEYMAN strikes again');
|
||||
}
|
||||
|
||||
let gHubResp = resp.body;
|
||||
let all = {
|
||||
releases: [],
|
||||
// todo make this ':baseurl' + ':releasename'
|
||||
download: '',
|
||||
};
|
||||
|
||||
try {
|
||||
gHubResp.forEach(transformReleases);
|
||||
} catch (e) {
|
||||
console.error(e.message);
|
||||
console.error('Error Headers:', resp.headers);
|
||||
console.error('Error Body:', resp.body);
|
||||
throw e;
|
||||
}
|
||||
|
||||
function transformReleases(release) {
|
||||
for (let asset of release['assets']) {
|
||||
let name = asset['name'];
|
||||
let date = release['published_at']?.replace(/T.*/, '');
|
||||
let download = asset['browser_download_url'];
|
||||
|
||||
// TODO tags aren't always semver / sensical
|
||||
let version = release['tag_name'];
|
||||
let channel;
|
||||
if (release['prerelease']) {
|
||||
// -rcX, -preview, -beta, etc will be checked in _webi/normalize.js
|
||||
channel = 'beta';
|
||||
}
|
||||
let lts = /(\b|_)(lts)(\b|_)/.test(release['tag_name']);
|
||||
|
||||
all.releases.push({
|
||||
name: name,
|
||||
version: version,
|
||||
lts: lts,
|
||||
channel: channel,
|
||||
date: date,
|
||||
os: '', // will be guessed by download filename
|
||||
arch: '', // will be guessed by download filename
|
||||
ext: '', // will be normalized
|
||||
download: download,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return all;
|
||||
}
|
||||
|
||||
module.exports = getAllReleases;
|
||||
|
||||
if (module === require.main) {
|
||||
getAllReleases(require('@root/request'), 'BurntSushi', 'ripgrep').then(
|
||||
function (all) {
|
||||
console.info(JSON.stringify(all, null, 2));
|
||||
},
|
||||
);
|
||||
}
|
||||
@@ -23,7 +23,7 @@ install:
|
||||
```text
|
||||
~/.config/envman/PATH.env
|
||||
~/.local/bin/foo
|
||||
~/.local/opt/foo/
|
||||
~/.local/opt/foo
|
||||
```
|
||||
|
||||
## Cheat Sheet
|
||||
|
||||
@@ -19,13 +19,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
|
||||
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
|
||||
|
||||
# Fetch archive
|
||||
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
Write-Output "Downloading foobar from $Env:WEBI_PKG_URL to $pkg_download"
|
||||
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
|
||||
& Move-Item "$pkg_download.part" "$pkg_download"
|
||||
}
|
||||
|
||||
if (!(Test-Path -Path "$pkg_src_cmd")) {
|
||||
IF (!(Test-Path -Path "$pkg_src_cmd")) {
|
||||
Write-Output "Installing foobar"
|
||||
|
||||
# TODO: create package-specific temp directory
|
||||
|
||||
@@ -1,3 +0,0 @@
|
||||
# Example releases.conf — uses ripgrep as a sample project.
|
||||
# Copy this file into your package directory and adjust.
|
||||
github_releases = BurntSushi/ripgrep
|
||||
28
_example/releases.js
Normal file
28
_example/releases.js
Normal file
@@ -0,0 +1,28 @@
|
||||
'use strict';
|
||||
|
||||
var github = require('../_common/github.js');
|
||||
var owner = 'BurntSushi';
|
||||
var repo = 'ripgrep';
|
||||
|
||||
/******************************************************************************/
|
||||
/** Note: Delete this Comment! **/
|
||||
/** **/
|
||||
/** Need a an example that filters out miscellaneous release files? **/
|
||||
/** See `deno`, `gitea`, or `caddy` **/
|
||||
/** **/
|
||||
/******************************************************************************/
|
||||
|
||||
module.exports = function (request) {
|
||||
return github(request, owner, repo).then(function (all) {
|
||||
return all;
|
||||
});
|
||||
};
|
||||
|
||||
if (module === require.main) {
|
||||
module.exports(require('@root/request')).then(function (all) {
|
||||
all = require('../_webi/normalize.js')(all);
|
||||
// just select the first 5 for demonstration
|
||||
all.releases = all.releases.slice(0, 5);
|
||||
console.info(JSON.stringify(all, null, 2));
|
||||
});
|
||||
}
|
||||
@@ -4,6 +4,7 @@
|
||||
|
||||
//var pkg = require('../package.json');
|
||||
var os = require('os');
|
||||
//var request = require('@root/request');
|
||||
//var promisify = require('util').promisify;
|
||||
//var exec = promisify(require('child_process').exec);
|
||||
var exec = require('child_process').exec;
|
||||
|
||||
@@ -1,100 +0,0 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
set -u
|
||||
|
||||
# Deploy the Node.js webinstall (installer server) to a target host.
|
||||
# Usage: _scripts/deploy-webinstall <host>
|
||||
# e.g. _scripts/deploy-webinstall beta.webi.sh
|
||||
# _scripts/deploy-webinstall next.webi.sh
|
||||
|
||||
fn_main() {
|
||||
g_host="${1:-}"
|
||||
if test -z "${g_host}"; then
|
||||
printf 'Usage: %s <host>\n' "$0" >&2
|
||||
printf ' e.g. %s beta.webi.sh\n' "$0" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# beta.webi.sh → ~/srv/beta.webinstall.dev/installers/
|
||||
# next.webi.sh → ~/srv/next.webinstall.dev/installers/
|
||||
b_subdomain="${g_host%%.*}"
|
||||
g_dest="~/srv/${b_subdomain}.webinstall.dev/installers/"
|
||||
|
||||
# Verify the build-classifier submodule is populated. rsync will
|
||||
# happily push an empty submodule directory, after which Node
|
||||
# crashes at startup with `Cannot find module './build-classifier/
|
||||
# host-targets.js'`. New worktrees from `git worktree add` don't
|
||||
# init submodules by default.
|
||||
if ! test -f ./_webi/build-classifier/host-targets.js; then
|
||||
printf 'fatal: _webi/build-classifier submodule not initialized\n' >&2
|
||||
printf ' run: git submodule update --init _webi/build-classifier\n' >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
printf '%s\n' "Deploying to ${g_host}:${g_dest} ..."
|
||||
|
||||
rsync -avz --delete \
|
||||
--exclude='.git' \
|
||||
--exclude='node_modules' \
|
||||
--exclude='_cache' \
|
||||
--exclude='LIVE_cache' \
|
||||
--exclude='webicached' \
|
||||
--exclude='classify' \
|
||||
--exclude='fetchraw' \
|
||||
--exclude='inspect' \
|
||||
--exclude='e2etest' \
|
||||
--exclude='zigtest' \
|
||||
--exclude='distributables.csv' \
|
||||
--exclude='agents/' \
|
||||
./ "${g_host}:${g_dest}"
|
||||
|
||||
printf '%s\n' "Restarting webinstall ..."
|
||||
|
||||
# shellcheck disable=SC2029
|
||||
ssh "${g_host}" "
|
||||
. ~/.config/envman/PATH.env 2>/dev/null
|
||||
rm -rf ${g_dest}_cache 2>/dev/null
|
||||
serviceman restart webinstall
|
||||
"
|
||||
|
||||
printf '%s\n' "Waiting for restart ..."
|
||||
sleep 3
|
||||
|
||||
fn_smoke_test
|
||||
}
|
||||
|
||||
fn_smoke_test() {
|
||||
printf '%s\n' "Smoke testing https://${g_host}/ ..."
|
||||
|
||||
b_headers="$(curl -sI --max-time 5 "https://${g_host}/api/releases/go.json?limit=1")"
|
||||
|
||||
if printf '%s' "${b_headers}" | grep -qi 'X-Robots-Tag: noindex'; then
|
||||
printf '%s\n' " [ok] X-Robots-Tag: noindex"
|
||||
else
|
||||
printf '%s\n' " [FAIL] Missing X-Robots-Tag header"
|
||||
fi
|
||||
|
||||
if printf '%s' "${b_headers}" | grep -qi 'rel="canonical"'; then
|
||||
printf '%s\n' " [ok] Link: canonical"
|
||||
else
|
||||
printf '%s\n' " [FAIL] Missing canonical Link header"
|
||||
fi
|
||||
|
||||
b_version="$(curl -sS --max-time 5 "https://${g_host}/api/releases/go.json?limit=1" | jq -r '.[0].version')"
|
||||
if test "${b_version}" != "0.0.0" && test -n "${b_version}"; then
|
||||
printf '%s\n' " [ok] API returns go ${b_version}"
|
||||
else
|
||||
printf '%s\n' " [FAIL] API returned error or empty for go"
|
||||
fi
|
||||
|
||||
b_shebang="$(curl -sS --max-time 5 "https://${g_host}/node" | head -1)"
|
||||
if test "${b_shebang}" = "#!/bin/sh"; then
|
||||
printf '%s\n' " [ok] Installer path serves shell script"
|
||||
else
|
||||
printf '%s\n' " [FAIL] Installer path unexpected: ${b_shebang}"
|
||||
fi
|
||||
|
||||
printf '%s\n' "Done."
|
||||
}
|
||||
|
||||
fn_main "$@"
|
||||
@@ -15,8 +15,8 @@ foreach ($my_dir in $my_dirs) {
|
||||
$my_ps1 = [System.IO.Path]::GetRelativePath($my_cwd, $my_file.FullName)
|
||||
$my_dir = [System.IO.Path]::GetDirectoryName($my_file.FullName)
|
||||
|
||||
if (-not (Test-Path -PathType Leaf -Path $my_ps1) -or
|
||||
-not (Test-Path -PathType Container -Path $my_dir)) {
|
||||
if (-Not (Test-Path -PathType Leaf -Path $my_ps1) -or
|
||||
-Not (Test-Path -PathType Container -Path $my_dir)) {
|
||||
Write-Host (" SKIP {0} (non-regular file or parent directory)" -f $my_ps1)
|
||||
continue
|
||||
}
|
||||
@@ -31,7 +31,7 @@ foreach ($my_dir in $my_dirs) {
|
||||
$my_new_file | Set-Content -Path $my_ps1
|
||||
|
||||
$my_new_file = $my_new_file + "`n"
|
||||
if ($text -ne $my_new_file) {
|
||||
IF ($text -ne $my_new_file) {
|
||||
$my_status = 1
|
||||
}
|
||||
}
|
||||
|
||||
@@ -15,8 +15,8 @@ foreach ($my_dir in $my_dirs) {
|
||||
$my_ps1 = [System.IO.Path]::GetRelativePath($my_cwd, $my_file.FullName)
|
||||
$my_dir = [System.IO.Path]::GetDirectoryName($my_file.FullName)
|
||||
|
||||
if (-not (Test-Path -PathType Leaf -Path $my_ps1) -or
|
||||
-not (Test-Path -PathType Container -Path $my_dir)) {
|
||||
if (-Not (Test-Path -PathType Leaf -Path $my_ps1) -or
|
||||
-Not (Test-Path -PathType Container -Path $my_dir)) {
|
||||
Write-Host (" SKIP {0} (non-regular file or parent directory)" -f $my_ps1)
|
||||
continue
|
||||
}
|
||||
@@ -39,7 +39,7 @@ foreach ($my_dir in $my_dirs) {
|
||||
$my_new_file | Set-Content -Path $my_ps1
|
||||
|
||||
$my_new_file = $my_new_file + "`n"
|
||||
if ($my_old_file -ne $my_new_file) {
|
||||
IF ($my_old_file -ne $my_new_file) {
|
||||
$my_status = 1
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env pwsh
|
||||
|
||||
if (!(Test-Path -Path "$Env:USERPROFILE\.vim\pack\plugins\start")) {
|
||||
IF (!(Test-Path -Path "$Env:USERPROFILE\.vim\pack\plugins\start")) {
|
||||
New-Item -Path "$Env:USERPROFILE\.vim\pack\plugins\start" -ItemType Directory -Force | Out-Null
|
||||
}
|
||||
Remove-Item -Path "$Env:USERPROFILE\.vim\pack\plugins\start\example" -Recurse -ErrorAction Ignore
|
||||
|
||||
Submodule _webi/build-classifier deleted from f9cc9f3e19
@@ -1,72 +0,0 @@
|
||||
'use strict';
|
||||
|
||||
let Path = require('node:path');
|
||||
|
||||
let BuildsCacher = require('./builds-cacher.js');
|
||||
// let Parallel = require('./parallel.js');
|
||||
|
||||
var INSTALLERS_DIR = Path.join(__dirname, '..');
|
||||
var CACHE_DIR = Path.join(__dirname, '../_cache');
|
||||
|
||||
async function main() {
|
||||
let bc = BuildsCacher.create({
|
||||
caches: CACHE_DIR,
|
||||
installers: INSTALLERS_DIR,
|
||||
});
|
||||
|
||||
// let dirs = await bc.getProjectsByType();
|
||||
// let projNames = Object.keys(dirs.valid);
|
||||
|
||||
let lastUpdate;
|
||||
|
||||
let projName = 'k9s';
|
||||
{
|
||||
let packages = await bc.getPackages({
|
||||
//Releases: Releases,
|
||||
name: projName,
|
||||
date: new Date(),
|
||||
});
|
||||
lastUpdate = packages.updated;
|
||||
console.info(
|
||||
`Last update for '${projName}': ${packages.updated} (${packages.releases.length} assets)`,
|
||||
);
|
||||
}
|
||||
|
||||
console.info('Waiting 5s');
|
||||
{
|
||||
setTimeout(async function () {
|
||||
let packages = await bc.getPackages({
|
||||
//Releases: Releases,
|
||||
name: projName,
|
||||
date: new Date(),
|
||||
});
|
||||
console.info(
|
||||
`Last update for '${projName}': ${packages.updated} (${packages.releases.length} assets)`,
|
||||
);
|
||||
if (lastUpdate < packages.updated) {
|
||||
console.info(`PASS`);
|
||||
} else {
|
||||
console.info(`MAYBE fail`);
|
||||
}
|
||||
}, 5 * 1000);
|
||||
}
|
||||
|
||||
//let parallel = 25;
|
||||
//await Parallel.run(parallel, projNames, getAll);
|
||||
//async function getAll(name) {
|
||||
// void (await bc.getPackages({
|
||||
// //Releases: Releases,
|
||||
// name: name,
|
||||
// date: new Date(),
|
||||
// }));
|
||||
//}
|
||||
}
|
||||
|
||||
main()
|
||||
.then(function () {
|
||||
console.log('Done');
|
||||
})
|
||||
.catch(function (e) {
|
||||
console.error(e.stack || e);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -1,907 +0,0 @@
|
||||
'use strict';
|
||||
|
||||
var BuildsCacher = module.exports;
|
||||
|
||||
let Fs = require('node:fs/promises');
|
||||
let Os = require('node:os');
|
||||
let Path = require('node:path');
|
||||
|
||||
let LEGACY_CACHE_DIR = Path.join(Os.homedir(), '.cache/webi/legacy');
|
||||
|
||||
let HostTargets = require('./build-classifier/host-targets.js');
|
||||
let Lexver = require('./build-classifier/lexver.js');
|
||||
let Triplet = require('./build-classifier/triplet.js');
|
||||
|
||||
var ALIAS_RE = /^alias: ([\w.-]+)$/m;
|
||||
|
||||
var LEGACY_ARCH_MAP = {
|
||||
'*': 'ANYARCH',
|
||||
arm64: 'aarch64',
|
||||
armv6l: 'armv6',
|
||||
armv7l: 'armv7',
|
||||
amd64: 'x86_64',
|
||||
mipsle: 'mipsel',
|
||||
mips64le: 'mips64el',
|
||||
mipsr6le: 'mipsr6el',
|
||||
mips64r6le: 'mips64r6el',
|
||||
// yes... el for arm and mips, but le for ppc
|
||||
// (perhaps the joke got old?)
|
||||
ppc64el: 'ppc64le',
|
||||
386: 'x86',
|
||||
};
|
||||
var LEGACY_OS_MAP = {
|
||||
'*': 'ANYOS',
|
||||
macos: 'darwin',
|
||||
posix: 'posix_2017',
|
||||
};
|
||||
|
||||
var TERMS_META = [
|
||||
// pattern
|
||||
'{ARCH}',
|
||||
'{EXT}',
|
||||
'{LIBC}',
|
||||
'{NAME}',
|
||||
'{OS}',
|
||||
'{VENDOR}',
|
||||
// // os-/arch-indepedent
|
||||
// 'ANYARCH',
|
||||
// 'ANYOS',
|
||||
// // libc
|
||||
// 'none',
|
||||
// channel
|
||||
'beta',
|
||||
'dev',
|
||||
'preview',
|
||||
'stable',
|
||||
];
|
||||
|
||||
/** @typedef {String} TripletString - {arch}-{vendor}-{os}-{libc} */
|
||||
/** @typedef {String} VersionString */
|
||||
/** @typedef {Object.<VersionString, Array<BuildAsset>>} PackagesByRelease */
|
||||
|
||||
/**
|
||||
* @typedef ProjectInfo
|
||||
* @prop {Array<BuildAsset>} releases
|
||||
* @prop {Array<BuildAsset>} packages
|
||||
* @prop {Object.<TripletString, PackagesByRelease>} releasesByTriplet
|
||||
* @prop {Array<import('./build-classifier/types.js').ArchString>} arches
|
||||
* @prop {Array<import('./build-classifier/types.js').OsString>} oses
|
||||
* @prop {Array<import('./build-classifier/types.js').LibcString>} libcs
|
||||
* @prop {Array<String>} channels
|
||||
* @prop {Array<String>} formats
|
||||
* @prop {Array<String>} triplets
|
||||
* @prop {Array<String>} versions
|
||||
* @prop {Array<String>} lexvers
|
||||
* @prop {Object.<String, String>} lexversMap
|
||||
*/
|
||||
|
||||
/**
|
||||
* @typedef BuildAsset
|
||||
* @prop {String} name
|
||||
* @prop {String} version
|
||||
* @prop {Boolean} lts
|
||||
* @prop {String} date
|
||||
* @prop {String} arch
|
||||
* @prop {String} os
|
||||
* @prop {String} libc
|
||||
* @prop {String} ext
|
||||
* @prop {String} download
|
||||
*/
|
||||
|
||||
/**
|
||||
* @typedef VersionTarget
|
||||
* @prop {String} version
|
||||
* @prop {Boolean} lts
|
||||
* @prop {String} channel
|
||||
*/
|
||||
|
||||
/** @typedef {TargetTriplet & HostTargetPartial} HostTarget */
|
||||
/** @typedef {import('./build-classifier/types.js').TargetTriplet} TargetTriplet */
|
||||
/**
|
||||
* @typedef HostTargetPartial
|
||||
* @prop {String} target.triplet - os-vendor-arch-libc
|
||||
* @prop {Error} target.error
|
||||
*/
|
||||
|
||||
async function getPartialHeader(path) {
|
||||
let readme = `${path}/README.md`;
|
||||
let head = await readFirstBytes(readme).catch(function (err) {
|
||||
if (err.code !== 'ENOENT') {
|
||||
console.warn(`warn: ${path}: ${err.message}`);
|
||||
}
|
||||
return null;
|
||||
});
|
||||
|
||||
return head;
|
||||
}
|
||||
|
||||
// let fsOpen = util.promisify(Fs.open);
|
||||
// let fsRead = util.promisify(Fs.read);
|
||||
async function readFirstBytes(path) {
|
||||
let start = 0;
|
||||
let n = 1024;
|
||||
let fh = await Fs.open(path, 'r');
|
||||
let buf = Buffer.alloc(n);
|
||||
let result = await fh.read(buf, start, n);
|
||||
let str = result.buffer.toString('utf8');
|
||||
await fh.close();
|
||||
|
||||
return str;
|
||||
}
|
||||
|
||||
BuildsCacher.create = function ({ ALL_TERMS, installers }) {
|
||||
let installersDir = installers;
|
||||
|
||||
if (!ALL_TERMS) {
|
||||
ALL_TERMS = Triplet.TERMS_PRIMARY_MAP;
|
||||
}
|
||||
|
||||
let bc = {};
|
||||
bc.ALL_TERMS = ALL_TERMS;
|
||||
bc.orphanTerms = Object.assign({}, bc.ALL_TERMS);
|
||||
bc.unknownTerms = {};
|
||||
bc.usedTerms = {};
|
||||
bc.formats = [];
|
||||
bc._triplets = {};
|
||||
bc._targetsByBuildIdCache = {};
|
||||
bc._caches = {};
|
||||
bc._allFormats = {};
|
||||
bc._allTriplets = {};
|
||||
// Per-name lock: serializes cold-cache getPackages so concurrent
|
||||
// callers can't corrupt bc._caches[name] via a transformAndUpdate race.
|
||||
bc._inflight = {};
|
||||
|
||||
for (let term of TERMS_META) {
|
||||
delete bc.orphanTerms[term];
|
||||
}
|
||||
|
||||
bc.getProjectsByType = async function () {
|
||||
let dirs = {
|
||||
hidden: {},
|
||||
errors: {},
|
||||
alias: {},
|
||||
invalid: {},
|
||||
selfhosted: {},
|
||||
valid: {},
|
||||
};
|
||||
|
||||
let entries = await Fs.readdir(installersDir, { withFileTypes: true });
|
||||
for (let entry of entries) {
|
||||
let meta = await bc.getProjectTypeByEntry(entry);
|
||||
dirs[meta.type][entry.name] = meta.detail;
|
||||
}
|
||||
|
||||
return dirs;
|
||||
};
|
||||
|
||||
/**
|
||||
* Get project type and detail - alias, selfhosted, valid (and the invalids)
|
||||
* @param {String} name - filename
|
||||
*/
|
||||
bc.getProjectType = async function (name) {
|
||||
let filepath = Path.join(installersDir, name);
|
||||
let entry;
|
||||
try {
|
||||
entry = await Fs.lstat(filepath);
|
||||
Object.assign(entry, { name: name });
|
||||
} catch (e) {
|
||||
return { type: 'errors', detail: 'not found' };
|
||||
}
|
||||
let info = await bc.getProjectTypeByEntry(entry);
|
||||
|
||||
return info;
|
||||
};
|
||||
|
||||
/**
|
||||
* Get project type and detail - alias, selfhosted, valid (and the invalids)
|
||||
* @param {fs.Stats|fs.Dirent} entry
|
||||
*/
|
||||
bc.getProjectTypeByEntry = async function (entry) {
|
||||
let path = Path.join(installersDir, entry.name);
|
||||
|
||||
// skip non-installer dirs
|
||||
if (entry.isSymbolicLink()) {
|
||||
let link = await Fs.readlink(path);
|
||||
return { type: 'alias', detail: link };
|
||||
}
|
||||
|
||||
if (!entry.isDirectory()) {
|
||||
return { type: 'hidden', detail: '!directory' };
|
||||
}
|
||||
if (entry.name === 'node_modules') {
|
||||
return { type: 'hidden', detail: 'node_modules' };
|
||||
}
|
||||
if (entry.name.startsWith('_')) {
|
||||
return { type: 'hidden', detail: '_*' };
|
||||
}
|
||||
if (entry.name.startsWith('.')) {
|
||||
return { type: 'hidden', detail: '.*' };
|
||||
}
|
||||
if (entry.name.startsWith('~')) {
|
||||
return { type: 'hidden', detail: '~*' };
|
||||
}
|
||||
if (entry.name.endsWith('~')) {
|
||||
return { type: 'hidden', detail: '*~' };
|
||||
}
|
||||
|
||||
// skip invalid installers
|
||||
let head = await getPartialHeader(path);
|
||||
if (!head) {
|
||||
return { type: 'invalid', detail: '!README.md' };
|
||||
}
|
||||
|
||||
let alias = head.match(ALIAS_RE);
|
||||
if (alias) {
|
||||
let link = alias[1];
|
||||
return { type: 'alias', detail: link };
|
||||
}
|
||||
|
||||
let cacheFile = `${LEGACY_CACHE_DIR}/${entry.name}.json`;
|
||||
let hasCacheFile = await Fs.access(cacheFile)
|
||||
.then(function () {
|
||||
return true;
|
||||
})
|
||||
.catch(function () {
|
||||
return false;
|
||||
});
|
||||
if (!hasCacheFile) {
|
||||
return { type: 'selfhosted', detail: true };
|
||||
}
|
||||
|
||||
return { type: 'valid', detail: true };
|
||||
};
|
||||
|
||||
// Typically a package is organized by release (ex: go has 1.20, 1.21, etc),
|
||||
// but we will organize by the build (ex: go1.20-darwin-arm64.tar.gz, etc).
|
||||
bc.getPackages = async function (args) {
|
||||
let name = args.name;
|
||||
let warm = bc._caches[name];
|
||||
if (warm) {
|
||||
return _doGetPackages(args);
|
||||
}
|
||||
let inflight = bc._inflight[name];
|
||||
if (inflight) {
|
||||
return inflight;
|
||||
}
|
||||
let p = _doGetPackages(args).finally(function () {
|
||||
delete bc._inflight[name];
|
||||
});
|
||||
bc._inflight[name] = p;
|
||||
return p;
|
||||
};
|
||||
|
||||
async function _doGetPackages({ name }) {
|
||||
let dataFile = `${LEGACY_CACHE_DIR}/${name}.json`;
|
||||
let tsFile = `${LEGACY_CACHE_DIR}/${name}.updated.txt`;
|
||||
|
||||
let tsDate;
|
||||
{
|
||||
let secondsStr = await Fs.readFile(tsFile, 'ascii').catch(function (err) {
|
||||
if (err.code !== 'ENOENT') {
|
||||
throw err;
|
||||
}
|
||||
return '0';
|
||||
});
|
||||
secondsStr = secondsStr.trim();
|
||||
let seconds = parseFloat(secondsStr) || 0;
|
||||
let ms = seconds * 1000;
|
||||
tsDate = new Date(ms);
|
||||
}
|
||||
|
||||
let projInfo = bc._caches[name];
|
||||
|
||||
let meta = {
|
||||
// version info
|
||||
versions: projInfo?.versions || [],
|
||||
lexvers: projInfo?.lexvers || [],
|
||||
lexversMap: projInfo?.lexversMap || {},
|
||||
// culled release assets
|
||||
packages: projInfo?.packages || [],
|
||||
releasesByTriplet: projInfo?.releasesByTriplet || {},
|
||||
// target info
|
||||
triplets: projInfo?.triplets || [],
|
||||
oses: projInfo?.oses || [],
|
||||
arches: projInfo?.arches || [],
|
||||
libcs: projInfo?.libcs || [],
|
||||
formats: projInfo?.formats || [],
|
||||
// TODO channels: projInfo?.channels || [],
|
||||
};
|
||||
|
||||
if (!projInfo) {
|
||||
let json = await Fs.readFile(dataFile, 'ascii').catch(
|
||||
async function (err) {
|
||||
if (err.code !== 'ENOENT') {
|
||||
throw err;
|
||||
}
|
||||
|
||||
return null;
|
||||
},
|
||||
);
|
||||
|
||||
try {
|
||||
projInfo = JSON.parse(json);
|
||||
} catch (e) {
|
||||
console.error(`error: ${dataFile}:\n\t${e.message}`);
|
||||
projInfo = null;
|
||||
}
|
||||
}
|
||||
if (!projInfo) {
|
||||
return meta;
|
||||
}
|
||||
let latestProjInfo = await BuildsCacher.transformAndUpdate(
|
||||
name,
|
||||
projInfo,
|
||||
meta,
|
||||
tsDate,
|
||||
bc,
|
||||
);
|
||||
bc._caches[name] = latestProjInfo;
|
||||
|
||||
return latestProjInfo;
|
||||
}
|
||||
|
||||
/**
|
||||
* Given a list of acceptable formats, get the sorted list of of formats.
|
||||
* Actually used (as per node _webi/lint-builds.js):
|
||||
* .7z
|
||||
* .app.zip
|
||||
* .dmg
|
||||
* .exe
|
||||
* .exe.xz
|
||||
* .git
|
||||
* .gz
|
||||
* .msi
|
||||
* .pkg
|
||||
* .pkg.tar.zst
|
||||
* .sh
|
||||
* .tar.gz
|
||||
* .tar.xz
|
||||
* .xz
|
||||
* .zip
|
||||
*/
|
||||
bc.getSortedFormats = function (formats) {
|
||||
/* jshint maxcomplexity: 25 */
|
||||
formats.sort();
|
||||
let id = formats.join(',');
|
||||
if (bc._allFormats[id]) {
|
||||
return bc._allFormats[id];
|
||||
}
|
||||
|
||||
// we don't know how to handle any of these yet
|
||||
// let exclude = [];
|
||||
// let isAndroid = false;
|
||||
// if (!isAndroid) {
|
||||
// exclude.push('.apk');
|
||||
// }
|
||||
// let isDebian = false;
|
||||
// if (!isDebian) {
|
||||
// exclude.push('.deb');
|
||||
// }
|
||||
// let isEnterpriseLinux = false;
|
||||
// if (!isEnterpriseLinux) {
|
||||
// exclude.push('.rpm');
|
||||
// }
|
||||
// let isArch = false;
|
||||
// if (!isArch) {
|
||||
// exclude.push('.pkg.tar.zst');
|
||||
// }
|
||||
|
||||
let hasExe = formats.includes('exe') || formats.includes('.exe');
|
||||
|
||||
/** @type {Array<String>} */
|
||||
let exts = [];
|
||||
|
||||
let hasXz = formats.includes('xz') || formats.includes('.xz');
|
||||
if (hasXz) {
|
||||
exts.push('.tar.xz');
|
||||
if (hasExe) {
|
||||
exts.push('.exe.xz');
|
||||
}
|
||||
exts.push('.xz');
|
||||
}
|
||||
let hasZst = formats.includes('zst') || formats.includes('.zst');
|
||||
if (hasZst) {
|
||||
exts.push('.tar.zst');
|
||||
exts.push('.zst');
|
||||
}
|
||||
let hasZip = formats.includes('zip') || formats.includes('.zip');
|
||||
if (hasZip) {
|
||||
exts.push('.zip');
|
||||
}
|
||||
let has7z = false;
|
||||
if (has7z) {
|
||||
exts.push('.7z');
|
||||
}
|
||||
// let hasBz2 = formats.includes('bz2') || formats.includes('.bz2');
|
||||
// if (hasBz2) {
|
||||
// exts.push('.bz2');
|
||||
// }
|
||||
if (hasExe) {
|
||||
if (!hasZip) {
|
||||
exts.push('.zip');
|
||||
}
|
||||
exts.push('.tar.gz');
|
||||
exts.push('.gz');
|
||||
exts.push('.exe');
|
||||
exts.push('.msi');
|
||||
//exts.push('.msixbundle');
|
||||
} else {
|
||||
exts.push('.tar.gz');
|
||||
exts.push('.gz');
|
||||
exts.push('.sh');
|
||||
}
|
||||
let hasGit = formats.includes('git') || formats.includes('.git');
|
||||
if (hasGit) {
|
||||
exts.push('.git');
|
||||
}
|
||||
|
||||
// Fallbacks
|
||||
// (we include everything to bubble an extract error over not found)
|
||||
exts.push('.app.zip');
|
||||
exts.push('.dmg');
|
||||
exts.push('.pkg');
|
||||
if (!hasXz) {
|
||||
exts.push('.tar.xz');
|
||||
if (hasExe) {
|
||||
exts.push('.exe.xz');
|
||||
}
|
||||
exts.push('.xz');
|
||||
}
|
||||
if (!hasZip) {
|
||||
if (!hasExe) {
|
||||
exts.push('.zip');
|
||||
}
|
||||
}
|
||||
if (!hasZst) {
|
||||
exts.push('.tar.zst');
|
||||
exts.push('.zst');
|
||||
}
|
||||
if (!has7z) {
|
||||
exts.push('.7z');
|
||||
}
|
||||
// if (!hasRar) {
|
||||
// exts.push('.rar');
|
||||
// }
|
||||
// exts.push('.tar.bz2');
|
||||
// exts.push('.bz2');
|
||||
|
||||
bc._allFormats[id] = exts;
|
||||
return exts;
|
||||
};
|
||||
|
||||
bc.selectPackage = function (packages, formats) {
|
||||
if (packages.length === 1) {
|
||||
return packages[0];
|
||||
}
|
||||
|
||||
let exts = bc.getSortedFormats(formats);
|
||||
for (let ext of exts) {
|
||||
for (let build of packages) {
|
||||
if (build.ext === ext) {
|
||||
return build;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return packages[0];
|
||||
};
|
||||
|
||||
/**
|
||||
* @param {ProjectInfo} projInfo
|
||||
*/
|
||||
bc.enumerateLatestVersions = function (projInfo) {
|
||||
let lexPrefix = '';
|
||||
let matchInfo = Lexver.matchSorted(projInfo.lexvers, lexPrefix);
|
||||
let verInfo = {
|
||||
default: projInfo.lexversMap[matchInfo.default],
|
||||
previous: projInfo.lexversMap[matchInfo.previous],
|
||||
stable: projInfo.lexversMap[matchInfo.stable],
|
||||
latest: projInfo.lexversMap[matchInfo.latest],
|
||||
};
|
||||
|
||||
return verInfo;
|
||||
};
|
||||
|
||||
/**
|
||||
* @param {ProjectInfo} projInfo
|
||||
* @param {HostTarget} hostTarget
|
||||
* @param {VersionTarget} verTarget
|
||||
*/
|
||||
bc.findMatchingPackages = function (projInfo, hostTarget, verTarget) {
|
||||
let matchInfo = bc._enumerateVersions(projInfo, verTarget.version);
|
||||
let triplets = bc._enumerateTriplets(hostTarget);
|
||||
//console.log('dbg: matchInfo', matchInfo);
|
||||
|
||||
if (matchInfo) {
|
||||
for (let _triplet of triplets) {
|
||||
let targetReleases = projInfo.releasesByTriplet[_triplet];
|
||||
if (!targetReleases) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Make sure that these releases are the expected version
|
||||
// (ex: jq1.7 => darwin-arm64-libc, jq1.6 => darwin-x86_64-libc)
|
||||
for (let matchver of matchInfo.matches) {
|
||||
let ver = projInfo.lexversMap[matchver] || matchver;
|
||||
let packages = targetReleases[ver];
|
||||
if (!packages) {
|
||||
continue;
|
||||
}
|
||||
|
||||
let match = {
|
||||
triplet: _triplet,
|
||||
packages: packages,
|
||||
latest: projInfo.versions[0],
|
||||
version: ver,
|
||||
versions: matchInfo,
|
||||
};
|
||||
return match;
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
// Version-first iteration, not triplet-first: take the newest
|
||||
// version even when its only build lives in a fallback triplet
|
||||
// (e.g. serviceman v1.0.1 only exists at posix_2017-ANYARCH-none).
|
||||
for (let lexver of projInfo.lexvers) {
|
||||
let ver = projInfo.lexversMap[lexver] || lexver;
|
||||
|
||||
for (let _triplet of triplets) {
|
||||
let targetReleases = projInfo.releasesByTriplet[_triplet];
|
||||
if (!targetReleases) {
|
||||
continue;
|
||||
}
|
||||
|
||||
let packages = targetReleases[ver];
|
||||
if (!packages) {
|
||||
continue;
|
||||
}
|
||||
|
||||
let pkg = packages[0];
|
||||
if (verTarget.lts) {
|
||||
if (!pkg.lts) {
|
||||
continue;
|
||||
}
|
||||
|
||||
let match = {
|
||||
triplet: _triplet,
|
||||
packages: packages,
|
||||
latest: projInfo.versions[0],
|
||||
version: ver,
|
||||
versions: matchInfo,
|
||||
};
|
||||
return match;
|
||||
}
|
||||
|
||||
let wantChannel = verTarget.channel || 'stable';
|
||||
let isChannel = pkg.channel || 'stable';
|
||||
if (wantChannel === 'stable') {
|
||||
if (isChannel !== 'stable') {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
// latest, beta, alpha, rc, preview
|
||||
|
||||
let match = {
|
||||
triplet: _triplet,
|
||||
packages: packages,
|
||||
latest: projInfo.versions[0],
|
||||
version: ver,
|
||||
versions: matchInfo,
|
||||
};
|
||||
return match;
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
};
|
||||
|
||||
bc._enumerateTriplets = function (hostTarget) {
|
||||
let id = [hostTarget.os, hostTarget.arch, hostTarget.libc].join(',');
|
||||
let triplets = bc._allTriplets[id] || [];
|
||||
if (triplets.length > 0) {
|
||||
return triplets;
|
||||
}
|
||||
|
||||
// Prefer platform-specific matches over ANYOS/ANYARCH fallbacks.
|
||||
// This ensures e.g. darwin-aarch64-none matches before
|
||||
// ANYOS-ANYARCH-none (.git source URLs from old releases).
|
||||
let oses = [];
|
||||
if (hostTarget.os === 'windows') {
|
||||
oses = ['windows', 'ANYOS'];
|
||||
} else if (hostTarget.os === 'android') {
|
||||
oses = ['android', 'linux', 'posix_2017', 'posix_2024', 'ANYOS'];
|
||||
} else {
|
||||
oses = [hostTarget.os, 'posix_2017', 'posix_2024', 'ANYOS'];
|
||||
}
|
||||
|
||||
let waterfall = HostTargets.WATERFALL[hostTarget.os] || {};
|
||||
let arches = waterfall[hostTarget.arch] ||
|
||||
HostTargets.WATERFALL.ANYOS[hostTarget.arch] || [hostTarget.arch];
|
||||
arches = arches.concat(['ANYARCH']);
|
||||
// termsToTarget omits libc for plain UAs; 'libc' → waterfall ['none','libc',...]
|
||||
let libc = hostTarget.libc || 'libc';
|
||||
let libcs = waterfall[libc] ||
|
||||
HostTargets.WATERFALL.ANYOS[libc] || [libc];
|
||||
|
||||
// Extend the glibc-host waterfall: the table only lists [none, libc]
|
||||
// but Rust projects (bat, rg) and node ship libc='gnu' builds, and
|
||||
// static musl builds also run on glibc hosts.
|
||||
if (libc === 'libc' && !libcs.includes('gnu')) {
|
||||
libcs = ['none', 'gnu', 'musl', 'libc'];
|
||||
}
|
||||
|
||||
for (let os of oses) {
|
||||
for (let arch of arches) {
|
||||
for (let libc of libcs) {
|
||||
let triplet = `${os}-${arch}-${libc}`;
|
||||
triplets.push(triplet);
|
||||
}
|
||||
}
|
||||
}
|
||||
bc._allTriplets[id] = triplets;
|
||||
|
||||
return triplets;
|
||||
};
|
||||
|
||||
bc._enumerateVersions = function (projInfo, ver) {
|
||||
if (!ver) {
|
||||
return null;
|
||||
}
|
||||
let lexPrefix = Lexver.parsePrefix(ver);
|
||||
let matchInfo = Lexver.matchSorted(projInfo.lexvers, lexPrefix);
|
||||
|
||||
return matchInfo;
|
||||
};
|
||||
|
||||
return bc;
|
||||
};
|
||||
|
||||
BuildsCacher._classify = function (bc, projInfo, build) {
|
||||
/* jshint maxcomplexity: 30 */
|
||||
// Cache entries arrive pre-classified (os/arch/libc/ext set). Skip
|
||||
// maybeInstallable for those — it false-rejects names ending in a
|
||||
// version tag (`serviceman-v1.0.1`, `v1.0.1.zip`).
|
||||
let cacheClassified =
|
||||
build.os && build.arch && build.libc && build.ext;
|
||||
if (!cacheClassified) {
|
||||
let maybeInstallable = Triplet.maybeInstallable(projInfo, build);
|
||||
if (!maybeInstallable) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
if (LEGACY_OS_MAP[build.os]) {
|
||||
build.os = LEGACY_OS_MAP[build.os];
|
||||
}
|
||||
if (LEGACY_ARCH_MAP[build.arch]) {
|
||||
build.arch = LEGACY_ARCH_MAP[build.arch];
|
||||
}
|
||||
|
||||
// because some packages are shimmed to match a single download against
|
||||
let preTarget = Object.assign({ os: '', arch: '', libc: '' }, build);
|
||||
|
||||
let targetId = `${preTarget.os}:${preTarget.arch}:${preTarget.libc}`;
|
||||
let buildId = `${projInfo.name}:${targetId}@${build.download}`;
|
||||
//console.log(`dbg: buildId`, buildId);
|
||||
let target = bc._targetsByBuildIdCache[buildId];
|
||||
if (target) {
|
||||
Object.assign(build, { target: target, triplet: target.triplet });
|
||||
return target;
|
||||
}
|
||||
|
||||
let pattern = Triplet.toPattern(projInfo, build);
|
||||
//console.log(`dbg: pattern`, pattern);
|
||||
if (!pattern) {
|
||||
let err = new Error(`no pattern generated for ${projInfo.name}`);
|
||||
err.code = 'E_BUILD_NO_PATTERN';
|
||||
target = { error: err };
|
||||
bc._targetsByBuildIdCache[buildId] = target;
|
||||
return target;
|
||||
}
|
||||
|
||||
let rawTerms = pattern.split(/[_\{\}\/\.\-]+/g);
|
||||
//console.log(`dbg: rawTerms`, rawTerms);
|
||||
for (let term of rawTerms) {
|
||||
delete bc.orphanTerms[term];
|
||||
bc.usedTerms[term] = true;
|
||||
}
|
||||
|
||||
// {NAME}/{NAME}-{VER}-Windows-x86_64_v2-musl.exe =>
|
||||
// {NAME}.windows.x86_64v2.musl.exe
|
||||
let terms = Triplet.patternToTerms(pattern);
|
||||
//console.log(`dbg: terms`, terms);
|
||||
if (!terms.length) {
|
||||
let err = new Error(`'${terms}' was trimmed to ''`);
|
||||
target = { error: err };
|
||||
bc._targetsByBuildIdCache[buildId] = target;
|
||||
return target;
|
||||
}
|
||||
|
||||
for (let term of terms) {
|
||||
if (!term) {
|
||||
continue;
|
||||
}
|
||||
|
||||
if (bc.ALL_TERMS[term]) {
|
||||
delete bc.orphanTerms[term];
|
||||
bc.usedTerms[term] = true;
|
||||
continue;
|
||||
}
|
||||
|
||||
bc.unknownTerms[term] = true;
|
||||
}
|
||||
|
||||
// Skip termsToTarget for cache-classified entries: it false-flags
|
||||
// e.g. .git URLs as os=ANYOS while the cache says os=posix_2017,
|
||||
// and the mismatch check throws.
|
||||
target = { triplet: '' };
|
||||
if (cacheClassified) {
|
||||
target.os = build.os;
|
||||
target.arch = build.arch;
|
||||
target.libc = build.libc;
|
||||
target.vendor = build.vendor || 'unknown';
|
||||
target.android = false;
|
||||
target.unknownTerms = [];
|
||||
} else {
|
||||
try {
|
||||
void Triplet.termsToTarget(target, projInfo, build, terms);
|
||||
} catch (e) {
|
||||
console.error(`PACKAGE FORMAT CHANGE for '${projInfo.name}':`);
|
||||
console.error(e.message);
|
||||
console.error(build);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
target.triplet = `${target.arch}-${target.vendor}-${target.os}-${target.libc}`;
|
||||
|
||||
{
|
||||
// TODO I don't love this hidden behavior
|
||||
// perhaps classify should just happen when the package is loaded
|
||||
// (and the sanity error should be removed, or thrown after the loop is complete)
|
||||
let hasTriplet = projInfo.triplets.includes(target.triplet);
|
||||
if (!hasTriplet) {
|
||||
projInfo.triplets.push(target.triplet);
|
||||
}
|
||||
let hasOs = projInfo.oses.includes(target.os);
|
||||
if (!hasOs) {
|
||||
projInfo.oses.push(target.os);
|
||||
}
|
||||
let hasArch = projInfo.arches.includes(target.arch);
|
||||
if (!hasArch) {
|
||||
projInfo.arches.push(target.arch);
|
||||
}
|
||||
let hasLibc = projInfo.libcs.includes(target.libc);
|
||||
if (!hasLibc) {
|
||||
projInfo.libcs.push(target.libc);
|
||||
}
|
||||
|
||||
if (!build.ext) {
|
||||
build.ext = Triplet.buildToPackageType(build);
|
||||
}
|
||||
if (build.ext) {
|
||||
if (!build.ext.startsWith('.')) {
|
||||
build.ext = `.${build.ext}`;
|
||||
}
|
||||
}
|
||||
let hasExt = projInfo.formats.includes(build.ext);
|
||||
if (!hasExt) {
|
||||
projInfo.formats.push(build.ext);
|
||||
}
|
||||
let hasGlobalExt = bc.formats.includes(build.ext);
|
||||
if (!hasGlobalExt) {
|
||||
bc.formats.push(build.ext);
|
||||
}
|
||||
}
|
||||
|
||||
bc._triplets[target.triplet] = true;
|
||||
bc._targetsByBuildIdCache[buildId] = target;
|
||||
|
||||
let triple = [target.arch, target.vendor, target.os, target.libc];
|
||||
for (let term of triple) {
|
||||
if (!bc.ALL_TERMS[term]) {
|
||||
throw new Error(
|
||||
`[SANITY FAIL] '${projInfo.name}' '${target.triplet}' generated unknown term '${term}'`,
|
||||
);
|
||||
}
|
||||
|
||||
delete bc.orphanTerms[term];
|
||||
bc.usedTerms[term] = true;
|
||||
}
|
||||
|
||||
return target;
|
||||
};
|
||||
|
||||
BuildsCacher.transformAndUpdate = function (name, projInfo, meta, date, bc) {
|
||||
meta.packages = [];
|
||||
|
||||
let updated = date.valueOf();
|
||||
|
||||
Object.assign(projInfo, { name, updated }, meta);
|
||||
for (let build of projInfo.releases) {
|
||||
let buildTarget = BuildsCacher._classify(bc, projInfo, build);
|
||||
if (!buildTarget) {
|
||||
// ignore known, non-package extensions
|
||||
continue;
|
||||
}
|
||||
|
||||
if (buildTarget.error) {
|
||||
let err = buildTarget.error;
|
||||
let code = err.code || '';
|
||||
console.error(`[ERROR]: ${code} ${projInfo.name}: ${build.name}`);
|
||||
console.error(`>>> ${err.message} <<<`);
|
||||
console.error(projInfo);
|
||||
console.error(build);
|
||||
console.error(`^^^ ${err.message} ^^^`);
|
||||
console.error(err.stack);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!build.name) {
|
||||
build.name = build.download.replace(/.*\//, '');
|
||||
}
|
||||
|
||||
build.target = buildTarget;
|
||||
meta.packages.push(build);
|
||||
}
|
||||
|
||||
BuildsCacher.updateReleasesByTriplet(meta);
|
||||
BuildsCacher.updateAndSortVersions(projInfo, meta);
|
||||
|
||||
Object.assign(projInfo, { name, updated }, meta);
|
||||
return projInfo;
|
||||
};
|
||||
|
||||
// TODO
|
||||
// - tag channels
|
||||
BuildsCacher.updateAndSortVersions = function (projInfo, meta) {
|
||||
for (let build of projInfo.packages) {
|
||||
let hasVersion = meta.versions.includes(build.version);
|
||||
if (!hasVersion) {
|
||||
build.lexver = Lexver.parseVersion(build.version);
|
||||
meta.lexversMap[build.lexver] = build.version;
|
||||
}
|
||||
}
|
||||
|
||||
meta.lexvers = Object.keys(meta.lexversMap);
|
||||
meta.lexvers.sort();
|
||||
meta.lexvers.reverse();
|
||||
|
||||
meta.versions = [];
|
||||
for (let lexver of meta.lexvers) {
|
||||
let version = meta.lexversMap[lexver];
|
||||
meta.versions.push(version);
|
||||
}
|
||||
|
||||
projInfo.packages.sort(function (a, b) {
|
||||
if (a.lexver > b.lexver) {
|
||||
return -1;
|
||||
}
|
||||
if (a.lexver < b.lexver) {
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
});
|
||||
};
|
||||
|
||||
BuildsCacher.updateReleasesByTriplet = function (meta) {
|
||||
for (let build of meta.packages) {
|
||||
let target = build.target;
|
||||
|
||||
let triplet = `${target.os}-${target.arch}-${target.libc}`;
|
||||
if (!meta.releasesByTriplet[triplet]) {
|
||||
meta.releasesByTriplet[triplet] = {};
|
||||
}
|
||||
|
||||
let buildsByRelease = meta.releasesByTriplet[triplet];
|
||||
if (!buildsByRelease[build.version]) {
|
||||
buildsByRelease[build.version] = [];
|
||||
}
|
||||
|
||||
let packages = buildsByRelease[build.version];
|
||||
packages.push(build);
|
||||
}
|
||||
};
|
||||
@@ -1,37 +0,0 @@
|
||||
'use strict';
|
||||
|
||||
let Builds = module.exports;
|
||||
|
||||
let Path = require('node:path');
|
||||
|
||||
let BuildsCacher = require('./builds-cacher.js');
|
||||
// let HostTargets = require('./build-classifier/host-targets.js');
|
||||
let Parallel = require('./parallel.js');
|
||||
|
||||
var INSTALLERS_DIR = Path.join(__dirname, '..');
|
||||
var CACHE_DIR = Path.join(__dirname, '../_cache');
|
||||
|
||||
let bc = BuildsCacher.create({
|
||||
caches: CACHE_DIR,
|
||||
installers: INSTALLERS_DIR,
|
||||
});
|
||||
|
||||
Builds.init = async function () {
|
||||
let dirs = await bc.getProjectsByType();
|
||||
let projNames = Object.keys(dirs.valid);
|
||||
|
||||
let parallel = 25;
|
||||
await Parallel.run(parallel, projNames, getAll);
|
||||
async function getAll(name) {
|
||||
void (await bc.getPackages({
|
||||
name: name,
|
||||
date: new Date(),
|
||||
}));
|
||||
}
|
||||
};
|
||||
|
||||
Builds.enumerateLatestVersions = bc.enumerateLatestVersions;
|
||||
Builds.findMatchingPackages = bc.findMatchingPackages;
|
||||
Builds.getPackage = bc.getPackages;
|
||||
Builds.getProjectType = bc.getProjectType;
|
||||
Builds.selectPackage = bc.selectPackage;
|
||||
@@ -1,90 +0,0 @@
|
||||
'use strict';
|
||||
|
||||
let Fs = require('node:fs/promises');
|
||||
let Os = require('node:os');
|
||||
let Path = require('node:path');
|
||||
|
||||
let BuildsCacher = require('./builds-cacher.js');
|
||||
let Triplet = require('./build-classifier/triplet.js');
|
||||
|
||||
let LEGACY_CACHE_DIR = Path.join(Os.homedir(), '.cache/webi/legacy');
|
||||
|
||||
async function main() {
|
||||
let projName = process.argv[2];
|
||||
if (!projName) {
|
||||
console.error(``);
|
||||
console.error(`USAGE`);
|
||||
console.error(``);
|
||||
console.error(` classify-one <project-name>`);
|
||||
console.error(``);
|
||||
console.error(`EXAMPLE`);
|
||||
console.error(``);
|
||||
console.error(` classify-one caddy`);
|
||||
console.error(``);
|
||||
return;
|
||||
}
|
||||
|
||||
let tsDate = new Date(0);
|
||||
let meta = {
|
||||
// version info
|
||||
versions: [],
|
||||
lexvers: [],
|
||||
lexversMap: {},
|
||||
// culled release assets
|
||||
packages: [],
|
||||
releasesByTriplet: {},
|
||||
// target info
|
||||
triplets: [],
|
||||
oses: [],
|
||||
arches: [],
|
||||
libcs: [],
|
||||
formats: [],
|
||||
};
|
||||
|
||||
let dataFile = Path.join(LEGACY_CACHE_DIR, `${projName}.json`);
|
||||
let json = await Fs.readFile(dataFile, 'utf8');
|
||||
let projInfo = JSON.parse(json);
|
||||
|
||||
// let packages = await Builds.getPackage({ name: projName });
|
||||
// console.log(packages);
|
||||
|
||||
let bc = {};
|
||||
bc.ALL_TERMS = Triplet.TERMS_PRIMARY_MAP;
|
||||
bc.orphanTerms = Object.assign({}, bc.ALL_TERMS);
|
||||
bc.unknownTerms = {};
|
||||
bc.usedTerms = {};
|
||||
bc.formats = [];
|
||||
bc._targetsByBuildIdCache = {};
|
||||
bc._triplets = {};
|
||||
|
||||
let transformed = BuildsCacher.transformAndUpdate(
|
||||
projName,
|
||||
projInfo,
|
||||
meta,
|
||||
tsDate,
|
||||
bc,
|
||||
);
|
||||
|
||||
console.log(`[DEBUG] transformed`);
|
||||
let sample = transformed.packages.slice(0, 20);
|
||||
console.log('packages:', sample, ':packages');
|
||||
let firstTriplet = Object.keys(transformed.releasesByTriplet)[0];
|
||||
let firstVersion = transformed.versions[0];
|
||||
console.log(
|
||||
`releasesByTriplet[${firstTriplet}][${firstVersion}]:`,
|
||||
transformed.releasesByTriplet[firstTriplet]?.[firstVersion],
|
||||
':releasesByTriplet',
|
||||
);
|
||||
console.log('versions:', transformed.versions, ':versions');
|
||||
console.log('triplets:', transformed.triplets, ':triplets');
|
||||
console.log('oses:', transformed.oses, ':oses');
|
||||
console.log('arches:', transformed.arches, ':arches');
|
||||
console.log('libcs:', transformed.libcs, ':libcs');
|
||||
console.log('formats:', transformed.formats, ':formats');
|
||||
console.log(Object.keys(transformed));
|
||||
}
|
||||
|
||||
main().catch(function (err) {
|
||||
console.error('Error:');
|
||||
console.error(err);
|
||||
});
|
||||
@@ -6,7 +6,7 @@
|
||||
############################################################
|
||||
New-Item -Path "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
|
||||
New-Item -Path "$Env:USERPROFILE\.local\bin" -ItemType Directory -Force | Out-Null
|
||||
if ($null -eq $Env:WEBI_HOST -or $Env:WEBI_HOST -eq "") { $Env:WEBI_HOST = "https://webinstall.dev" }
|
||||
IF ($null -eq $Env:WEBI_HOST -or $Env:WEBI_HOST -eq "") { $Env:WEBI_HOST = "https://webinstall.dev" }
|
||||
curl.exe -s -A "windows" "$Env:WEBI_HOST/packages/webi/webi-pwsh.ps1" -o "$Env:USERPROFILE\.local\bin\webi-pwsh.ps1"
|
||||
Set-ExecutionPolicy -Scope Process Bypass
|
||||
& "$Env:USERPROFILE\.local\bin\webi-pwsh.ps1" "{{ exename }}"
|
||||
|
||||
@@ -267,23 +267,12 @@ webi_bootstrap() { (
|
||||
fn_checksum() {
|
||||
a_filepath="${1}"
|
||||
|
||||
if command -v sha1sum > /dev/null; then
|
||||
sha1sum "${a_filepath}" | cut -d' ' -f1 | cut -c 1-8
|
||||
return 0
|
||||
fi
|
||||
|
||||
cmd_shasum='sha1sum'
|
||||
if command -v shasum > /dev/null; then
|
||||
shasum "${a_filepath}" | cut -d' ' -f1 | cut -c 1-8
|
||||
return 0
|
||||
cmd_shasum='shasum'
|
||||
fi
|
||||
|
||||
if command -v sha1 > /dev/null; then
|
||||
sha1 "${a_filepath}" | cut -d'=' -f2 | cut -c 2-9
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo >&2 " warn: no sha1 sum program"
|
||||
date '+%F %H:%M'
|
||||
$cmd_shasum "${a_filepath}" | cut -d' ' -f1 | cut -c 1-8
|
||||
}
|
||||
|
||||
##############################################
|
||||
|
||||
@@ -93,18 +93,16 @@ Installers.renderBash = async function (
|
||||
['WEBI_GIT_TAG', rel.git_tag], // TODO replace with branch
|
||||
['WEBI_LTS', rel.lts],
|
||||
['WEBI_CHANNEL', rel.channel],
|
||||
['WEBI_EXT', rel.ext],
|
||||
['WEBI_EXT', rel.ext.replace(/tar.*/, 'tar')],
|
||||
['WEBI_FORMATS', formats.join(',')],
|
||||
['WEBI_PKG_URL', rel.download],
|
||||
['WEBI_PKG_PATHNAME', pkgFile],
|
||||
['WEBI_PKG_FILE', pkgFile], // TODO replace with pathname
|
||||
['PKG_NAME', pkg],
|
||||
['PKG_STABLE', rel.stable],
|
||||
['PKG_LATEST', rel.latest],
|
||||
['PKG_OSES', (rel.oses || []).join(' ')],
|
||||
['PKG_ARCHES', (rel.arches || []).join(' ')],
|
||||
['PKG_LIBCS', (rel.libcs || []).join(' ')],
|
||||
['PKG_FORMATS', (rel.formats || []).join(' ')],
|
||||
['PKG_OSES', rel.oses],
|
||||
['PKG_ARCHES', rel.arches],
|
||||
['PKG_LIBCS', rel.libcs],
|
||||
['PKG_FORMATS', (rel.formats || []).join(',')],
|
||||
];
|
||||
|
||||
for (let env of envReplacements) {
|
||||
|
||||
@@ -1,336 +0,0 @@
|
||||
#!/usr/bin/env node
|
||||
'use strict';
|
||||
|
||||
let Fs = require('node:fs/promises');
|
||||
let Path = require('node:path');
|
||||
|
||||
let BuildsCacher = require('./builds-cacher.js');
|
||||
let HostTargets = require('./build-classifier/host-targets.js');
|
||||
let Parallel = require('./parallel.js');
|
||||
|
||||
var INSTALLERS_DIR = Path.join(__dirname, '..');
|
||||
var CACHE_DIR = Path.join(__dirname, '../_cache');
|
||||
|
||||
let UserAgentsMap = require('./build-classifier/uas.json');
|
||||
let uas = Object.keys(UserAgentsMap);
|
||||
let uaTargetsMap = {};
|
||||
for (let ua of uas) {
|
||||
let terms = ua.split(/[\s\/]+/g);
|
||||
let target = {};
|
||||
void HostTargets.termsToTarget(target, terms);
|
||||
if (!target) {
|
||||
continue;
|
||||
}
|
||||
if (target.errors.length) {
|
||||
throw target.errors[0];
|
||||
}
|
||||
if (!target.os) {
|
||||
// TODO make target null, or create error for this
|
||||
console.warn(`no os for terms: ${terms}`);
|
||||
//throw new Error(`terms: ${terms}`);
|
||||
continue;
|
||||
}
|
||||
if (!target.arch) {
|
||||
// TODO make target null, or create error for this
|
||||
console.warn(`no arch for terms: ${terms}`);
|
||||
//throw new Error(`terms: ${terms}`);
|
||||
continue;
|
||||
}
|
||||
if (!target.libc) {
|
||||
// TODO make target null, or create error for this
|
||||
console.warn(`no libc for terms: ${terms}`);
|
||||
//throw new Error(`terms: ${terms}`);
|
||||
continue;
|
||||
}
|
||||
let triplet = `${target.os}-${target.arch}-${target.libc}`;
|
||||
uaTargetsMap[triplet] = target;
|
||||
}
|
||||
let uaTargets = [];
|
||||
let triplets = Object.keys(uaTargetsMap);
|
||||
for (let triplet of triplets) {
|
||||
let target = uaTargetsMap[triplet];
|
||||
uaTargets.push(target);
|
||||
}
|
||||
|
||||
function showDirs(dirs) {
|
||||
{
|
||||
let errors = Object.keys(dirs.errors);
|
||||
console.error('');
|
||||
console.error(`Errors: ${errors.length}`);
|
||||
for (let name of errors) {
|
||||
let err = dirs.errors[name];
|
||||
console.error(`${name}/: ${err.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
let hidden = Object.keys(dirs.hidden);
|
||||
console.debug('');
|
||||
console.debug(`Hidden: ${hidden.length}`);
|
||||
for (let name of hidden) {
|
||||
let kind = dirs.hidden[name];
|
||||
if (kind === '!directory') {
|
||||
console.debug(` ${name}`);
|
||||
} else {
|
||||
console.debug(` ${name}/`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
let alias = Object.keys(dirs.alias);
|
||||
console.debug('');
|
||||
console.debug(`Alias: ${alias.length}`);
|
||||
for (let name of alias) {
|
||||
let kind = dirs.alias[name];
|
||||
if (kind === 'symlink') {
|
||||
console.debug(` ${name} => ...`);
|
||||
} else {
|
||||
console.debug(` ${name}/`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
let invalids = Object.keys(dirs.invalid);
|
||||
console.warn('');
|
||||
console.warn(`Invalid: ${invalids.length}`);
|
||||
for (let name of invalids) {
|
||||
console.warn(` ${name}/`);
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
let selfhosted = Object.keys(dirs.selfhosted);
|
||||
console.info('');
|
||||
console.info(`Self-Hosted: ${selfhosted.length}`);
|
||||
for (let name of selfhosted) {
|
||||
console.info(` ${name}/`);
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
let valids = Object.keys(dirs.valid);
|
||||
console.info('');
|
||||
console.info(`Found: ${valids.length}`);
|
||||
for (let name of valids) {
|
||||
console.info(` ${name}/`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let bc = BuildsCacher.create({
|
||||
caches: CACHE_DIR,
|
||||
installers: INSTALLERS_DIR,
|
||||
});
|
||||
|
||||
async function main() {
|
||||
/* jshint maxcomplexity: 25 */
|
||||
// TODO
|
||||
// node ./_webi/lint-builds.js caddy@beta 'x86_64/unknown Darwin libc'
|
||||
//
|
||||
//let [projName, userAgent] = process.argv.slice(2);
|
||||
let projName = process.argv[2];
|
||||
// create test case for zoxide, goreleaser, go, yq, caddy, rg
|
||||
|
||||
let dirs = await bc.getProjectsByType();
|
||||
if (!projName) {
|
||||
showDirs(dirs);
|
||||
console.info('');
|
||||
}
|
||||
|
||||
let rows = [];
|
||||
let triples = [];
|
||||
let valids = Object.keys(dirs.valid);
|
||||
|
||||
if (projName) {
|
||||
if (!valids.includes(projName)) {
|
||||
throw new Error(`'${projName}' is not a valid installable project`);
|
||||
}
|
||||
valids = [projName];
|
||||
}
|
||||
//valids = ['atomicparsley', 'caddy', 'macos'];
|
||||
//valids = ['atomicparsley'];
|
||||
|
||||
console.info('');
|
||||
console.info(`Fetching project release assets`);
|
||||
let parallel = 25;
|
||||
let projects = [];
|
||||
await Parallel.run(parallel, valids, getAll);
|
||||
async function getAll(name, i) {
|
||||
console.info(` ${name}`);
|
||||
let projInfo = await bc.getPackages({
|
||||
//Releases: Releases,
|
||||
name: name,
|
||||
date: new Date(),
|
||||
});
|
||||
projects[i] = projInfo;
|
||||
}
|
||||
|
||||
console.info(`Classifying build assets for...`);
|
||||
for (let projInfo of projects) {
|
||||
console.info(` ${projInfo.name}`);
|
||||
|
||||
let nStr = projInfo.releases.length.toString();
|
||||
let n = nStr.padStart(5, ' ');
|
||||
let row = `##### ${n}\t${projInfo.name}\tv`;
|
||||
rows.push(row);
|
||||
|
||||
// ignore known, non-package extensions
|
||||
for (let build of projInfo.releases) {
|
||||
let target = bc.classify(projInfo, build);
|
||||
if (!target) {
|
||||
// non-build file
|
||||
continue;
|
||||
}
|
||||
|
||||
if (target.error) {
|
||||
let e = target.error;
|
||||
if (e.code === 'E_BUILD_NO_PATTERN') {
|
||||
console.warn(`>>> ${e.message} <<<`);
|
||||
console.warn(projInfo);
|
||||
console.warn(build);
|
||||
console.warn(`^^^ ${e.message} ^^^`);
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
|
||||
if (target.unknownTerms?.length) {
|
||||
let msg = `${projInfo.name}: unrecognized term(s) '${target.unknownTerms}' in '${build.download}'`;
|
||||
let err = new Error(msg);
|
||||
throw err;
|
||||
}
|
||||
|
||||
triples.push(target.triplet);
|
||||
// if (!build.version) {
|
||||
// throw new Error(`no version for ${pkg.name} ${build.name}`);
|
||||
// }
|
||||
// // For debug printing versions
|
||||
// console.error(build.version);
|
||||
rows.push(`${target.triplet}\t${projInfo.name}\t${build.version}`);
|
||||
}
|
||||
}
|
||||
|
||||
console.info(`Fetching builds for`);
|
||||
for (let projInfo of projects) {
|
||||
console.info('');
|
||||
console.info('');
|
||||
console.info(` ${projInfo.name}`);
|
||||
|
||||
for (let target of uaTargets) {
|
||||
let libc = target.libc || 'libc';
|
||||
let hostTriplet = `${target.os}-${target.arch}-${libc}`;
|
||||
console.info('');
|
||||
console.info(` target: ${hostTriplet}`);
|
||||
let match = bc.findMatchingPackages(projInfo, target, {
|
||||
ver: '',
|
||||
});
|
||||
if (!match) {
|
||||
console.info(
|
||||
` project: ${projInfo.name}: missing build for os '${target.os}'`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!match.releases) {
|
||||
console.info(
|
||||
` project: ${projInfo.name}: missing build for os '${target.os}-${target.arch}-${libc}'`,
|
||||
);
|
||||
} else if (match.triplet === hostTriplet) {
|
||||
let releaseNames = Object.keys(match.releases);
|
||||
console.info(` selected ${releaseNames.length}`);
|
||||
} else {
|
||||
let releaseNames = Object.keys(match.releases);
|
||||
console.info(
|
||||
` selected ${releaseNames.length} (${match.triplet} fallback)`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let tsv = rows.join('\n');
|
||||
console.info('');
|
||||
console.info('#rows', rows.length);
|
||||
await Fs.writeFile('builds.tsv', tsv, 'utf8');
|
||||
|
||||
console.info('');
|
||||
console.info('Triplets Detected:');
|
||||
let triplets = Object.keys(bc._triplets);
|
||||
if (triplets.length) {
|
||||
triplets.sort();
|
||||
console.info(' ', triplets.join('\n '));
|
||||
} else {
|
||||
console.info(' (none)');
|
||||
}
|
||||
|
||||
console.info('');
|
||||
console.info('New / Unknown Terms:');
|
||||
let unknowns = Object.keys(bc.unknownTerms);
|
||||
if (unknowns.length) {
|
||||
unknowns.sort();
|
||||
console.warn(' ', unknowns.join('\n '));
|
||||
} else {
|
||||
console.info(' (none)');
|
||||
}
|
||||
|
||||
console.info('');
|
||||
console.info('Unused Terms:');
|
||||
let unuseds = Object.keys(bc.orphanTerms);
|
||||
if (unuseds.length) {
|
||||
unuseds.sort();
|
||||
console.warn(' ', unuseds.join('\n '));
|
||||
} else {
|
||||
console.info(' (none)');
|
||||
}
|
||||
|
||||
console.info('');
|
||||
console.info('Formats:');
|
||||
if (bc.formats.length) {
|
||||
let formats = bc.formats.slice();
|
||||
formats.sort();
|
||||
if (!formats[0]) {
|
||||
formats[0] = '(bin)';
|
||||
}
|
||||
console.warn(' ', formats.join('\n '));
|
||||
} else {
|
||||
console.info(' (none)');
|
||||
}
|
||||
|
||||
// sort -u -k1 builds.tsv | rg -v '^#|^https?:' | rg -i arm
|
||||
// cut -f1 builds.tsv | sort -u -k1 | rg -v '^#|^https?:' | rg -i arm
|
||||
}
|
||||
|
||||
if (module === require.main) {
|
||||
let times = [];
|
||||
let now = Date.now();
|
||||
main()
|
||||
.then(async function () {
|
||||
let then = Date.now();
|
||||
let delta = then - now;
|
||||
times.push(delta);
|
||||
now = then;
|
||||
await main();
|
||||
then = Date.now();
|
||||
delta = then - now;
|
||||
times.push(delta);
|
||||
})
|
||||
.then(function () {
|
||||
console.info('');
|
||||
console.info('Run times');
|
||||
for (let delta of times) {
|
||||
let s = delta / 1000;
|
||||
console.info(` ${s}`);
|
||||
}
|
||||
|
||||
function forceExit() {
|
||||
console.warn(`warn: dangling event loop reference`);
|
||||
process.exit(0);
|
||||
}
|
||||
let exitTimeout = setTimeout(forceExit, 250);
|
||||
exitTimeout.unref();
|
||||
})
|
||||
.catch(function (err) {
|
||||
console.error(err.stack || err);
|
||||
process.exit(1);
|
||||
});
|
||||
}
|
||||
258
_webi/normalize.js
Normal file
258
_webi/normalize.js
Normal file
@@ -0,0 +1,258 @@
|
||||
'use strict';
|
||||
|
||||
// this may need customizations between packages
|
||||
var osMap = {
|
||||
macos: /(\b|_)(apple|os(\s_-)?x\b|mac|darwin|iPhone|iOS|iPad)/i,
|
||||
linux: /(\b|_)(linux)/i,
|
||||
freebsd: /(\b|_)(freebsd)/i,
|
||||
windows: /(\b|_)(win|microsoft|msft)/i,
|
||||
sunos: /(\b|_)(sun)/i,
|
||||
aix: /(\b|_)(aix)/i,
|
||||
};
|
||||
|
||||
var maps = {
|
||||
oses: {},
|
||||
arches: {},
|
||||
libcs: {},
|
||||
formats: {},
|
||||
};
|
||||
|
||||
Object.keys(osMap).forEach(function (name) {
|
||||
maps.oses[name] = true;
|
||||
});
|
||||
|
||||
var formats = ['zip', 'xz', 'tar', 'pkg', 'msi', 'git', 'exe', 'dmg', 'git'];
|
||||
formats.forEach(function (name) {
|
||||
maps.formats[name] = true;
|
||||
});
|
||||
|
||||
// evaluation order matters
|
||||
// (i.e. otherwise x86 and x64 can cross match)
|
||||
var arches = [
|
||||
// arm64/aarch64 has very high specificity, so it comes first
|
||||
'arm64',
|
||||
// arm 7 is also generic aarch/arm/arm32
|
||||
'armv7l',
|
||||
// arm6 can run on armv7
|
||||
'armv6l',
|
||||
// amd64 is more likely and less often specified than arm64
|
||||
'amd64',
|
||||
'x86',
|
||||
'ppc64le',
|
||||
'ppc64',
|
||||
's390x',
|
||||
];
|
||||
// Used for detecting system arch from package download url, for example:
|
||||
//
|
||||
// https://git.com/org/foo/releases/v0.7.9/foo-aarch64-linux-musl.tar.gz
|
||||
// https://git.com/org/foo/releases/v0.7.9/foo-arm-linux-musleabihf.tar.gz
|
||||
// https://git.com/org/foo/releases/v0.7.9/foo-armv7-linux-musleabihf.tar.gz
|
||||
// https://git.com/org/foo/releases/v0.7.9/foo-x86_64-linux-musl.tar.gz
|
||||
//
|
||||
var archMap = {
|
||||
arm64: /(\b|_)(aarch64|arm64)/i,
|
||||
armv7l: /(\b|_)(arm32|arm[_\-]?v?7l?)/i,
|
||||
armv6l: /(\b|_)(arm|aarch32|arm[_\-]?v?6l?)(\b|_)/i,
|
||||
//amd64: /(amd.?64|x64|[_\-]64)/i,
|
||||
amd64:
|
||||
/(\b|_|amd|(dar)?win(dows)?|mac(os)?|linux|osx|x)64([_\-]?bit)?(\b|_)/i,
|
||||
//x86: /(86)(\b|_)/i,
|
||||
x86: /(\b|_|amd|(dar)?win(dows)?|mac(os)?|linux|osx|x)(86|32)([_\-]?bit)(\b|_)/i,
|
||||
ppc64le: /(\b|_)(ppc64le)/i,
|
||||
ppc64: /(\b|_)(ppc64)(\b|_)/i,
|
||||
s390x: /(\b|_)(s390x)/i,
|
||||
};
|
||||
arches.forEach(function (name) {
|
||||
maps.arches[name] = true;
|
||||
});
|
||||
|
||||
var libcs = ['none', 'musl', 'gnu', 'msvc', 'libc'];
|
||||
libcs.forEach(function (name) {
|
||||
maps.libcs[name] = true;
|
||||
});
|
||||
|
||||
function normalize(all) {
|
||||
/* jshint maxcomplexity:50 */
|
||||
/* jshint maxdepth:10 */
|
||||
var supported = {
|
||||
oses: {},
|
||||
arches: {},
|
||||
libcs: {},
|
||||
formats: {},
|
||||
};
|
||||
|
||||
for (let rel of all.releases) {
|
||||
rel.version = rel.version.replace(/^v/i, '');
|
||||
|
||||
if (!rel.name) {
|
||||
rel.name = rel.download.replace(/.*\//, '');
|
||||
}
|
||||
|
||||
if (!rel.os) {
|
||||
rel.os = 'unknown';
|
||||
|
||||
let osNames = Object.keys(osMap);
|
||||
for (let osName of osNames) {
|
||||
let relName = rel.name || rel.download;
|
||||
let osRegExp = osMap[osName];
|
||||
let matches = osRegExp.test(relName);
|
||||
if (matches) {
|
||||
rel.os = osName;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
supported.oses[rel.os] = true;
|
||||
|
||||
if (!rel.arch) {
|
||||
for (let arch of arches) {
|
||||
let name = rel.name || rel.download;
|
||||
let isArch = name.match(archMap[arch]);
|
||||
if (isArch) {
|
||||
rel.arch = arch;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (!rel.arch) {
|
||||
if ('macos' === rel.os) {
|
||||
rel.arch = 'amd64';
|
||||
}
|
||||
}
|
||||
supported.arches[rel.arch] = true;
|
||||
|
||||
// note: depends on rel.os
|
||||
if (!rel.libc) {
|
||||
let isMusl;
|
||||
let isMsvc;
|
||||
let isStatic;
|
||||
let isGnu;
|
||||
|
||||
// extra blocks to prevent copy pasta errors
|
||||
|
||||
{
|
||||
let muslRe = /(\b|_)(musl)(\b|_)/i;
|
||||
isMusl = muslRe.test(rel.download) || muslRe.test(rel.name);
|
||||
}
|
||||
|
||||
{
|
||||
let msvcRe = /(\b|_)(msvc)(\b|_)/i;
|
||||
isMsvc = msvcRe.test(rel.download) || msvcRe.test(rel.name);
|
||||
}
|
||||
|
||||
{
|
||||
let staticRe = /(\b|_)(static)(\b|_)/i;
|
||||
isStatic = staticRe.test(rel.download) || staticRe.test(rel.name);
|
||||
}
|
||||
|
||||
{
|
||||
let gnuRe = /(\b|_)(gnu|glibc|libc)(\b|_)/i;
|
||||
isGnu = gnuRe.test(rel.download) || gnuRe.test(rel.name);
|
||||
}
|
||||
|
||||
if (isMusl) {
|
||||
// we specifically tag things that need musl++ in their own releases
|
||||
rel.libc = 'none';
|
||||
} else if (isStatic) {
|
||||
rel.libc = 'none';
|
||||
} else if (isGnu) {
|
||||
rel.libc = 'gnu';
|
||||
if (rel.os === 'windows') {
|
||||
// windows gnu is static
|
||||
rel.libc = 'none';
|
||||
} else if (rel.os === 'darwin') {
|
||||
// if glibc is required on macos, it'll be static
|
||||
rel.libc = 'none';
|
||||
}
|
||||
} else if (isMsvc) {
|
||||
rel.libc = 'msvc';
|
||||
} else {
|
||||
// The default is no requirement for any particular libc
|
||||
// (Go, Zig, POSIX Shell, JS, etc)
|
||||
// and hopefully we never have to worry about mingw and friends
|
||||
rel.libc = 'none';
|
||||
}
|
||||
}
|
||||
supported.libcs[rel.libc] = true;
|
||||
|
||||
var tarExt;
|
||||
if (!rel.ext) {
|
||||
// pkg-v1.0.tar.gz => ['gz', 'tar', '0', 'pkg-v1']
|
||||
// pkg-v1.0.tar => ['tar', '0' ,'pkg-v1']
|
||||
// pkg-v1.0.zip => ['zip', '0', 'pkg-v1']
|
||||
var exts = (rel.name || rel.download).split('.');
|
||||
if (1 === exts.length) {
|
||||
// for bare releases in the format of foo-linux-amd64
|
||||
rel.ext = 'exe';
|
||||
}
|
||||
exts = exts.reverse().slice(0, 2);
|
||||
if ('tar' === exts[1]) {
|
||||
rel.ext = exts.reverse().join('.');
|
||||
tarExt = 'tar';
|
||||
} else if ('tgz' === exts[0]) {
|
||||
rel.ext = 'tar.gz';
|
||||
tarExt = 'tar';
|
||||
} else {
|
||||
rel.ext = exts[0];
|
||||
}
|
||||
if (/\-|linux|mac|os[_\-]?x|arm|amd|86|64|mip/i.test(rel.ext)) {
|
||||
// for bare releases in the format of foo.linux-amd64
|
||||
rel.ext = 'exe';
|
||||
}
|
||||
}
|
||||
supported.formats[tarExt || rel.ext] = true;
|
||||
|
||||
if (!rel.channel) {
|
||||
// basically like this: (+.-_)(beta|rc)(0-9)(+.-_)
|
||||
// matches:
|
||||
// - v1.0-beta
|
||||
// - v1.0-beta1.1
|
||||
// - v1.0-beta-11
|
||||
// won't match:
|
||||
// - v1.0beta
|
||||
// - v1.0-beta1b
|
||||
let isBetaRe = /(\b|_)(preview|rc|beta|alpha)(\d+)(\b|_)/;
|
||||
let isBeta = isBetaRe.test(rel.name);
|
||||
if (isBeta) {
|
||||
rel.channel = 'beta';
|
||||
} else {
|
||||
rel.channel = 'stable';
|
||||
}
|
||||
}
|
||||
|
||||
if (all.download) {
|
||||
rel.download = all.download.replace(/{{ download }}/, rel.download);
|
||||
}
|
||||
}
|
||||
|
||||
all.oses = Object.keys(supported.oses).filter(function (name) {
|
||||
return maps.oses[name];
|
||||
});
|
||||
all.arches = Object.keys(supported.arches).filter(function (name) {
|
||||
return maps.arches[name];
|
||||
});
|
||||
all.libcs = Object.keys(supported.libcs).filter(function (name) {
|
||||
return maps.libcs[name];
|
||||
});
|
||||
all.formats = Object.keys(supported.formats).filter(function (name) {
|
||||
return maps.formats[name];
|
||||
});
|
||||
|
||||
return all;
|
||||
}
|
||||
|
||||
module.exports = normalize;
|
||||
module.exports._debug = function (all) {
|
||||
all = normalize(all);
|
||||
all.releases = all.releases
|
||||
.filter(function (r) {
|
||||
return ['windows', 'macos', 'linux'].includes(r.os) && 'amd64' === r.arch;
|
||||
})
|
||||
.slice(0, 10);
|
||||
return all;
|
||||
};
|
||||
// NOT in order of priority (which would be tar, xz, zip, ...)
|
||||
module.exports.formats = formats;
|
||||
module.exports.arches = arches;
|
||||
module.exports.libcs = libcs;
|
||||
module.exports.formatsMap = maps.formats;
|
||||
@@ -45,15 +45,15 @@ $TDim = "${Esc}[2m"
|
||||
$TReset = "${Esc}[0m"
|
||||
|
||||
function Invoke-DownloadUrl {
|
||||
param (
|
||||
Param (
|
||||
[string]$URL,
|
||||
[string]$Params,
|
||||
[string]$Path,
|
||||
[switch]$Force
|
||||
)
|
||||
|
||||
if (Test-Path -Path "$Path") {
|
||||
if (-not $Force.IsPresent) {
|
||||
IF (Test-Path -Path "$Path") {
|
||||
IF (-Not $Force.IsPresent) {
|
||||
Write-Host " ${TDim}Found${TReset} $Path"
|
||||
return
|
||||
}
|
||||
@@ -65,7 +65,7 @@ function Invoke-DownloadUrl {
|
||||
|
||||
Write-Host " Downloading ${TDim}from${TReset}"
|
||||
Write-Host " ${TDim}${URL}${TReset}"
|
||||
if ($Params.Length -ne 0) {
|
||||
IF ($Params.Length -ne 0) {
|
||||
Write-Host " ?$Params"
|
||||
$URL = "${URL}?${Params}"
|
||||
}
|
||||
@@ -80,18 +80,18 @@ function Get-UserAgent {
|
||||
# This is the canonical CPU arch when the process is emulated
|
||||
$my_arch = "$Env:PROCESSOR_ARCHITEW6432"
|
||||
|
||||
if ($my_arch -eq $null -or $my_arch -eq "") {
|
||||
IF ($my_arch -eq $null -or $my_arch -eq "") {
|
||||
# This is the canonical CPU arch when the process is native
|
||||
$my_arch = "$Env:PROCESSOR_ARCHITECTURE"
|
||||
}
|
||||
|
||||
if ($my_arch -eq "AMD64") {
|
||||
IF ($my_arch -eq "AMD64") {
|
||||
# Because PowerShell is sometimes AMD64 on Windows 10 ARM
|
||||
# See https://oofhours.com/2020/02/04/powershell-on-windows-10-arm64/
|
||||
$my_os_arch = (Get-CimInstance -ClassName Win32_OperatingSystem).OSArchitecture
|
||||
$my_os_arch = wmic os get osarchitecture
|
||||
|
||||
# Using -clike because of the trailing newline
|
||||
if ($my_os_arch -clike "ARM 64*") {
|
||||
IF ($my_os_arch -clike "ARM 64*") {
|
||||
$my_arch = "ARM64"
|
||||
}
|
||||
}
|
||||
@@ -124,7 +124,7 @@ function webi_path_add($pathname) {
|
||||
$exists_in_path = $true
|
||||
}
|
||||
}
|
||||
if (-not $exists_in_path) {
|
||||
if (-Not $exists_in_path) {
|
||||
$all_user_paths = "${pathname};${all_user_paths}".Trim(';')
|
||||
[Environment]::SetEnvironmentVariable("Path", $all_user_paths, "User")
|
||||
$null = Sync-EnvPath
|
||||
|
||||
@@ -27,7 +27,6 @@ __bootstrap_webi() {
|
||||
#PKG_LIBCS=
|
||||
#PKG_FORMATS=
|
||||
#PKG_LATEST=
|
||||
#PKG_STABLE=
|
||||
WEBI_PKG_DOWNLOAD=""
|
||||
WEBI_DOWNLOAD_DIR="${HOME}/Downloads"
|
||||
if command -v xdg-user-dir > /dev/null; then
|
||||
@@ -37,7 +36,7 @@ __bootstrap_webi() {
|
||||
fi
|
||||
fi
|
||||
|
||||
WEBI_PKG_PATH="${WEBI_DOWNLOAD_DIR}/webi/${PKG_NAME:-error}/${WEBI_VERSION:-stable}"
|
||||
WEBI_PKG_PATH="${WEBI_DOWNLOAD_DIR}/webi/${PKG_NAME:-error}/${WEBI_VERSION:-latest}"
|
||||
|
||||
# get the special formatted version
|
||||
# (i.e. "go is go1.14" while node is "node v12.10.8")
|
||||
@@ -128,10 +127,6 @@ __bootstrap_webi() {
|
||||
echo ""
|
||||
echo " $(t_err "Error: no '${PKG_NAME:-"Unknown Package"}@${WEBI_TAG:-"Unknown Tag"}' release for '${WEBI_OS:-"Unknown OS"}' (${WEBI_LIBC:-"Unknown Libc"}) on '${WEBI_ARCH:-"Unknown CPU"}' as one of '${WEBI_FORMATS:-"Unknown File Type"}'")"
|
||||
echo ""
|
||||
echo " Latest Stable: ${PKG_STABLE}"
|
||||
if test "${PKG_LATEST}" != "${PKG_STABLE}"; then
|
||||
echo " Next Version: ${PKG_LATEST}"
|
||||
fi
|
||||
echo " CPUs: $PKG_ARCHES"
|
||||
echo " OSes: $PKG_OSES"
|
||||
echo " libcs: $PKG_LIBCS"
|
||||
@@ -201,32 +196,20 @@ __bootstrap_webi() {
|
||||
my_dl_rel="$(
|
||||
fn_sub_home "${WEBI_PKG_PATH}/${WEBI_PKG_FILE}"
|
||||
)"
|
||||
if test "$WEBI_EXT" = "tar.zst"; then
|
||||
echo " Extracting $(t_path "${my_dl_rel}")"
|
||||
unzstd -c --keep "${WEBI_PKG_PATH}/$WEBI_PKG_FILE" | tar xf -
|
||||
elif test "$WEBI_EXT" = "tar.xz"; then
|
||||
echo " Extracting $(t_path "${my_dl_rel}")"
|
||||
unxz -c -k "${WEBI_PKG_PATH}/$WEBI_PKG_FILE" | tar xf -
|
||||
elif test "$WEBI_EXT" = "tar.gz"; then
|
||||
echo " Extracting $(t_path "${my_dl_rel}")"
|
||||
tar xzf "${WEBI_PKG_PATH}/$WEBI_PKG_FILE"
|
||||
elif test "$WEBI_EXT" = "tar.bz2"; then
|
||||
echo " Extracting $(t_path "${my_dl_rel}")"
|
||||
tar xjf "${WEBI_PKG_PATH}/$WEBI_PKG_FILE"
|
||||
elif test "$WEBI_EXT" = "tar"; then
|
||||
if [ "tar" = "$WEBI_EXT" ]; then
|
||||
echo " Extracting $(t_path "${my_dl_rel}")"
|
||||
tar xf "${WEBI_PKG_PATH}/$WEBI_PKG_FILE"
|
||||
elif test "$WEBI_EXT" = "zip" || test "$WEBI_EXT" = "app.zip"; then
|
||||
elif [ "zip" = "$WEBI_EXT" ]; then
|
||||
echo " Extracting $(t_path "${my_dl_rel}")"
|
||||
unzip "${WEBI_PKG_PATH}/$WEBI_PKG_FILE" > __unzip__.log
|
||||
elif test "$WEBI_EXT" = "exe"; then
|
||||
elif [ "exe" = "$WEBI_EXT" ]; then
|
||||
echo " Moving $(t_path "${my_dl_rel}")"
|
||||
echo " to $(t_path "$(fn_sub_home "$(pwd)")")"
|
||||
mv "${WEBI_PKG_PATH}/$WEBI_PKG_FILE" .
|
||||
elif test "$WEBI_EXT" = "git"; then
|
||||
elif [ "git" = "$WEBI_EXT" ]; then
|
||||
echo " Moving $(t_path "${my_dl_rel}")"
|
||||
mv "${WEBI_PKG_PATH}/$WEBI_PKG_FILE" .
|
||||
elif test "$WEBI_EXT" = "xz"; then
|
||||
elif [ "xz" = "$WEBI_EXT" ]; then
|
||||
echo " Inflating $(t_path "${my_dl_rel}")"
|
||||
unxz -c "${WEBI_PKG_PATH}/$WEBI_PKG_FILE" > "$(basename "$WEBI_PKG_FILE")"
|
||||
else
|
||||
@@ -239,6 +222,8 @@ __bootstrap_webi() {
|
||||
webi_path_add() {
|
||||
my_path="${1}"
|
||||
|
||||
fn_envman_init
|
||||
|
||||
# \v was chosen as it is extremely unlikely for a filename
|
||||
# \1 could be an even better choice, but needs more testing.
|
||||
# (currently tested working on: linux & mac)
|
||||
@@ -401,6 +386,7 @@ __bootstrap_webi() {
|
||||
export _webi_tmp="${_webi_tmp:-"$HOME/.local/opt/webi-tmp.d"}"
|
||||
|
||||
mkdir -p "${WEBI_PKG_PATH}"
|
||||
mkdir -p "$HOME/.local/bin"
|
||||
mkdir -p "$HOME/.local/opt"
|
||||
|
||||
if test -e ~/.local/bin; then
|
||||
@@ -409,7 +395,6 @@ __bootstrap_webi() {
|
||||
echo " Creating$(t_path ' ~/.local/bin')"
|
||||
mkdir -p "$HOME/.local/bin"
|
||||
fi
|
||||
fn_envman_init
|
||||
|
||||
##
|
||||
##
|
||||
@@ -790,23 +775,12 @@ webi_upgrade() { (
|
||||
fn_checksum() {
|
||||
a_filepath="${1}"
|
||||
|
||||
if command -v sha1sum > /dev/null; then
|
||||
sha1sum "${a_filepath}" | cut -d' ' -f1 | cut -c 1-8
|
||||
return 0
|
||||
fi
|
||||
|
||||
cmd_shasum='sha1sum'
|
||||
if command -v shasum > /dev/null; then
|
||||
shasum "${a_filepath}" | cut -d' ' -f1 | cut -c 1-8
|
||||
return 0
|
||||
cmd_shasum='shasum'
|
||||
fi
|
||||
|
||||
if command -v sha1 > /dev/null; then
|
||||
sha1 "${a_filepath}" | cut -d'=' -f2 | cut -c 2-9
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo >&2 " warn: no sha1 sum program"
|
||||
date '+%F %H:%M'
|
||||
$cmd_shasum "${a_filepath}" | cut -d' ' -f1 | cut -c 1-8
|
||||
}
|
||||
|
||||
##############################################
|
||||
|
||||
@@ -1,44 +0,0 @@
|
||||
'use strict';
|
||||
|
||||
var Parallel = module.exports;
|
||||
Parallel.run = async function (limit, arr, fn) {
|
||||
let index = 0;
|
||||
let actives = [];
|
||||
let results = [];
|
||||
limit = Math.min(limit, arr.length);
|
||||
|
||||
function launch() {
|
||||
let _index = index;
|
||||
let p = fn(arr[_index], _index, arr);
|
||||
|
||||
// some tasks may be synchronous
|
||||
// so we must push before removing
|
||||
actives.push(p);
|
||||
|
||||
p.then(function _resolve(result) {
|
||||
let i = actives.indexOf(p);
|
||||
actives.splice(i, 1);
|
||||
results[_index] = result;
|
||||
});
|
||||
|
||||
index += 1;
|
||||
}
|
||||
|
||||
// start tasks in parallel, up to limit
|
||||
for (; actives.length < limit; ) {
|
||||
launch();
|
||||
}
|
||||
|
||||
// keep the task queue full
|
||||
for (; index < arr.length; ) {
|
||||
// wait for one task to complete
|
||||
await Promise.race(actives);
|
||||
// add one task again
|
||||
launch();
|
||||
}
|
||||
|
||||
// wait for all remaining tasks
|
||||
await Promise.all(actives);
|
||||
|
||||
return results;
|
||||
};
|
||||
@@ -2,14 +2,17 @@
|
||||
|
||||
var InstallerServer = module.exports;
|
||||
|
||||
let Fs = require('fs/promises');
|
||||
let Path = require('path');
|
||||
var Fs = require('fs/promises');
|
||||
var path = require('path');
|
||||
|
||||
let HostTargets = require('./build-classifier/host-targets.js');
|
||||
let Builds = require('./builds.js');
|
||||
let Installers = require('./installers.js');
|
||||
var uaDetect = require('./ua-detect.js');
|
||||
var Projects = require('./projects.js');
|
||||
var Installers = require('./installers.js');
|
||||
|
||||
InstallerServer.INSTALLERS_DIR = Path.join(__dirname, '..');
|
||||
// handlers caching and transformation, probably should be broken down
|
||||
var Releases = require('./transform-releases.js');
|
||||
|
||||
InstallerServer.INSTALLERS_DIR = path.join(__dirname, '..');
|
||||
InstallerServer.serveInstaller = async function (
|
||||
baseurl,
|
||||
ua,
|
||||
@@ -19,210 +22,113 @@ InstallerServer.serveInstaller = async function (
|
||||
formats,
|
||||
libc,
|
||||
) {
|
||||
let unameAgent = ua;
|
||||
let projectName = pkg;
|
||||
let [rel, tmplParams] = await InstallerServer.helper({
|
||||
unameAgent,
|
||||
projectName,
|
||||
let [rel, opts] = await InstallerServer.helper({
|
||||
ua,
|
||||
pkg,
|
||||
tag,
|
||||
formats,
|
||||
libc,
|
||||
});
|
||||
Object.assign(tmplParams, {
|
||||
Object.assign(opts, {
|
||||
baseurl,
|
||||
});
|
||||
|
||||
var pkgdir = Path.join(InstallerServer.INSTALLERS_DIR, projectName);
|
||||
var pkgdir = path.join(InstallerServer.INSTALLERS_DIR, pkg);
|
||||
if ('ps1' === ext) {
|
||||
return Installers.renderPowerShell(pkgdir, rel, tmplParams);
|
||||
return Installers.renderPowerShell(pkgdir, rel, opts);
|
||||
}
|
||||
return Installers.renderBash(pkgdir, rel, tmplParams);
|
||||
return Installers.renderBash(pkgdir, rel, opts);
|
||||
};
|
||||
InstallerServer.helper = async function ({ ua, pkg, tag, formats, libc }) {
|
||||
// TODO put some of this in a middleware? or common function?
|
||||
|
||||
// TODO put some of this in a middleware? or common function?
|
||||
// TODO maybe move package/version/lts/channel detection into getReleases
|
||||
InstallerServer.helper = async function ({
|
||||
unameAgent,
|
||||
projectName,
|
||||
tag,
|
||||
formats,
|
||||
libc,
|
||||
}) {
|
||||
console.log(`dbg: Installer User-Agent: ${unameAgent}`);
|
||||
// TODO maybe move package/version/lts/channel detection into getReleases
|
||||
var ver = tag.replace(/^v/, '');
|
||||
var lts;
|
||||
var channel;
|
||||
|
||||
let releaseTarget = toReleaseTarget(tag);
|
||||
let hostFormats = formats;
|
||||
let terms = unameAgent.split(/[\s\/]+/g);
|
||||
let hostTarget = {};
|
||||
try {
|
||||
void HostTargets.termsToTarget(hostTarget, terms);
|
||||
} catch (e) {
|
||||
// if we can't guarantee the results...
|
||||
// "in the face of ambiguity, refuse the temptation to guess"
|
||||
throw e;
|
||||
}
|
||||
console.log(`dbg: Installer Host Target:`);
|
||||
console.log(hostTarget);
|
||||
|
||||
if (!hostTarget.os) {
|
||||
throw new Error(`OS could not be identified by User-Agent '${unameAgent}'`);
|
||||
switch (ver) {
|
||||
case 'latest':
|
||||
ver = '';
|
||||
channel = 'stable';
|
||||
break;
|
||||
case 'lts':
|
||||
lts = true;
|
||||
channel = 'stable';
|
||||
ver = '';
|
||||
break;
|
||||
case 'stable':
|
||||
channel = 'stable';
|
||||
ver = '';
|
||||
break;
|
||||
case 'beta':
|
||||
channel = 'beta';
|
||||
ver = '';
|
||||
break;
|
||||
case 'dev':
|
||||
channel = 'dev';
|
||||
ver = '';
|
||||
break;
|
||||
}
|
||||
|
||||
console.log(`dbg: Get Project Installer Type for '${projectName}':`);
|
||||
let proj = await Builds.getProjectType(projectName);
|
||||
if (proj.type === 'alias') {
|
||||
console.log(`dbg: alias`, proj);
|
||||
projectName = proj.detail;
|
||||
proj = await Builds.getProjectType(projectName); // an alias should never resolve to an alias
|
||||
var myOs = uaDetect.os(ua);
|
||||
var myArch = uaDetect.arch(ua);
|
||||
var myLibc;
|
||||
if (libc) {
|
||||
myLibc = uaDetect.libc(libc);
|
||||
}
|
||||
console.log(`dbg: proj`, proj);
|
||||
|
||||
let validTypes = ['selfhosted', 'valid'];
|
||||
if (!validTypes.includes(proj.type)) {
|
||||
let msg = `'${projectName}' doesn't have an installer: '${proj.type}': '${proj.detail}'`;
|
||||
let err = new Error(msg);
|
||||
err.code = 'ENOENT';
|
||||
throw err;
|
||||
if (!myLibc) {
|
||||
myLibc = uaDetect.libc(ua);
|
||||
}
|
||||
if (!myLibc) {
|
||||
myLibc = 'libc';
|
||||
}
|
||||
|
||||
let tmplParams = {
|
||||
pkg: projectName,
|
||||
tag: tag,
|
||||
os: hostTarget.os,
|
||||
arch: hostTarget.arch,
|
||||
libc: hostTarget.libc,
|
||||
formats: hostFormats,
|
||||
let cfg = await Projects.get(pkg);
|
||||
let releaseQuery = {
|
||||
pkg: cfg.alias || pkg,
|
||||
ver,
|
||||
os: myOs,
|
||||
arch: myArch,
|
||||
libc: myLibc,
|
||||
lts,
|
||||
channel,
|
||||
// TODO use formats for sorting, not exclusion
|
||||
// (it's better to install xz or report an error to install zip)
|
||||
formats,
|
||||
limit: 1,
|
||||
};
|
||||
Object.assign(tmplParams, releaseTarget);
|
||||
console.log('tmplParams', tmplParams);
|
||||
|
||||
let errPackage = {
|
||||
name: 'doesntexist.ext',
|
||||
version: '0.0.0',
|
||||
lts: '-',
|
||||
channel: 'error',
|
||||
date: '1970-01-01',
|
||||
os: hostTarget.os || '-',
|
||||
arch: hostTarget.arch || '-',
|
||||
libc: hostTarget.libc || '-',
|
||||
ext: 'err',
|
||||
download: 'https://example.com/doesntexist.ext',
|
||||
comment:
|
||||
'No matches found. Could be bad or missing version info' +
|
||||
',' +
|
||||
"Check query parameters. Should be something like '/api/releases/{package}@{version}.tab?os={macos|linux|windows|-}&arch={amd64|x86|aarch64|arm64|armv7l|-}&libc={musl|gnu|msvc|libc|static}&limit=10'",
|
||||
let rels = await Releases.getReleases(releaseQuery);
|
||||
|
||||
var rel = rels.releases[0];
|
||||
var opts = {
|
||||
pkg: cfg.alias || pkg,
|
||||
ver,
|
||||
tag,
|
||||
os: myOs,
|
||||
arch: myArch,
|
||||
libc: myLibc,
|
||||
lts,
|
||||
channel,
|
||||
formats,
|
||||
limit: 1,
|
||||
};
|
||||
|
||||
if (proj.type === 'selfhosted') {
|
||||
return [errPackage, tmplParams];
|
||||
}
|
||||
|
||||
let projInfo = await Builds.getPackage({
|
||||
name: projectName,
|
||||
date: new Date(),
|
||||
});
|
||||
let latestVersions = Builds.enumerateLatestVersions(projInfo);
|
||||
//console.log('projInfo', projInfo);
|
||||
|
||||
let buildTargetInfo = {
|
||||
triplets: projInfo.triplets,
|
||||
oses: projInfo.oses,
|
||||
arches: projInfo.arches,
|
||||
libcs: projInfo.libcs,
|
||||
formats: projInfo.formats,
|
||||
latest: latestVersions.latest,
|
||||
stable: latestVersions.stable,
|
||||
};
|
||||
|
||||
// TODO .findMatchingPackages() should probably account for this
|
||||
let hasOs = projInfo.oses.includes(hostTarget.os);
|
||||
let maybePosix = !hasOs && hostTarget.os !== 'windows';
|
||||
if (maybePosix) {
|
||||
let posixes = ['posix_2017', 'posix_2024'];
|
||||
for (let posixYear of posixes) {
|
||||
let hasPosix = projInfo.oses.includes(posixYear);
|
||||
if (hasPosix) {
|
||||
hasOs = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (!hasOs) {
|
||||
hasOs = projInfo.oses.includes('ANYOS');
|
||||
}
|
||||
|
||||
if (!hasOs) {
|
||||
let pkg1 = Object.assign(buildTargetInfo, errPackage);
|
||||
return [pkg1, tmplParams];
|
||||
}
|
||||
|
||||
let targetRelease = Builds.findMatchingPackages(
|
||||
projInfo,
|
||||
hostTarget,
|
||||
releaseTarget,
|
||||
rel = Object.assign(
|
||||
{
|
||||
oses: rels.oses,
|
||||
arches: rels.arches,
|
||||
libcs: rels.libcs,
|
||||
formats: rels.formats,
|
||||
},
|
||||
rel,
|
||||
);
|
||||
// { triplet: `${os}-${arch}-${libc}`, packages: targetPackages
|
||||
// , latest: projInfo.versions[0], versions: matchInfo
|
||||
// }
|
||||
|
||||
if (!targetRelease?.packages) {
|
||||
let pkg1 = Object.assign(buildTargetInfo, errPackage);
|
||||
return [pkg1, tmplParams];
|
||||
}
|
||||
|
||||
let buildPkg = Builds.selectPackage(targetRelease.packages, hostFormats);
|
||||
let ext = buildPkg.ext || '.exe';
|
||||
if (ext.startsWith('.')) {
|
||||
ext = ext.slice(1);
|
||||
}
|
||||
|
||||
let version = targetRelease.version;
|
||||
if (version.startsWith('v')) {
|
||||
version = version.slice(1);
|
||||
}
|
||||
|
||||
buildPkg = Object.assign(buildTargetInfo, buildPkg, { ext, version });
|
||||
console.log('dbg: buildPkg', buildPkg);
|
||||
console.log('dbg: tmplParams', tmplParams);
|
||||
return [buildPkg, tmplParams];
|
||||
return [rel, opts];
|
||||
};
|
||||
|
||||
let channelNames = [
|
||||
'stable',
|
||||
// 'hotfix',
|
||||
'latest',
|
||||
'rc',
|
||||
'preview',
|
||||
'pre',
|
||||
'dev',
|
||||
'beta',
|
||||
'alpha',
|
||||
];
|
||||
|
||||
function toReleaseTarget(tag) {
|
||||
tag = tag.replace(/^v/, '');
|
||||
|
||||
let releaseTarget = {
|
||||
channel: '',
|
||||
lts: false,
|
||||
version: '',
|
||||
};
|
||||
|
||||
if (tag === 'lts') {
|
||||
releaseTarget.lts = true;
|
||||
releaseTarget.channel = 'stable';
|
||||
} else if (channelNames.includes(tag)) {
|
||||
releaseTarget.channel = tag;
|
||||
} else {
|
||||
releaseTarget.version = tag;
|
||||
}
|
||||
|
||||
return releaseTarget;
|
||||
}
|
||||
|
||||
var CURL_PIPE_PS1_BOOT = Path.join(__dirname, 'curl-pipe-bootstrap.tpl.ps1');
|
||||
var CURL_PIPE_SH_BOOT = Path.join(__dirname, 'curl-pipe-bootstrap.tpl.sh');
|
||||
var CURL_PIPE_PS1_BOOT = path.join(__dirname, 'curl-pipe-bootstrap.tpl.ps1');
|
||||
var CURL_PIPE_SH_BOOT = path.join(__dirname, 'curl-pipe-bootstrap.tpl.sh');
|
||||
var BAD_SH_RE = /[<>'"`$\\]/;
|
||||
|
||||
InstallerServer.getPosixCurlPipeBootstrap = async function ({
|
||||
@@ -244,6 +150,7 @@ InstallerServer.getPosixCurlPipeBootstrap = async function ({
|
||||
let name = env[0];
|
||||
let value = env[1];
|
||||
|
||||
// TODO create REs once, in higher scope
|
||||
let envRe = new RegExp(
|
||||
`^[ \\t]*#?[ \\t]*(export[ \\t])?[ \\t]*(${name})=.*`,
|
||||
'm',
|
||||
@@ -291,6 +198,9 @@ InstallerServer.getPwshCurlPipeBootstrap = async function ({
|
||||
let tplRe = new RegExp(`{{ (${name}) }}`, 'g');
|
||||
bootTxt = bootTxt.replace(tplRe, `${value}`);
|
||||
|
||||
// let envRe = new RegExp(`^[ \\t]*#?[ \\t]*($$${name})[ \\t]*=.*`, 'im');
|
||||
// bootTxt = bootTxt.replace(envRe, `$$${name} = '${value}'`);
|
||||
|
||||
let setRe = new RegExp(
|
||||
`(#[ \\t]*)?(\\$${name})[ \\t]*=[ \\t]['"].*['"][ \\t]`,
|
||||
'im',
|
||||
|
||||
@@ -1,208 +0,0 @@
|
||||
'use strict';
|
||||
|
||||
let Fs = require('node:fs/promises');
|
||||
let Path = require('node:path');
|
||||
|
||||
let Releases = require('./transform-releases.js');
|
||||
|
||||
let TESTDATA_DIR = Path.join(__dirname, 'testdata');
|
||||
|
||||
// These mirror what the live API returns for /api/releases/{pkg}@stable.json?...
|
||||
let FILTERED_CASES = [
|
||||
{ pkg: 'bat', os: 'macos', arch: 'amd64' },
|
||||
{ pkg: 'bat', os: 'macos', arch: 'arm64' },
|
||||
{ pkg: 'bat', os: 'linux', arch: 'amd64' },
|
||||
{ pkg: 'bat', os: 'windows', arch: 'amd64' },
|
||||
{ pkg: 'go', os: 'macos', arch: 'amd64' },
|
||||
{ pkg: 'go', os: 'macos', arch: 'arm64' },
|
||||
{ pkg: 'go', os: 'linux', arch: 'amd64' },
|
||||
{ pkg: 'go', os: 'windows', arch: 'amd64' },
|
||||
{ pkg: 'rg', os: 'macos', arch: 'amd64' },
|
||||
{ pkg: 'rg', os: 'macos', arch: 'arm64' },
|
||||
{ pkg: 'rg', os: 'linux', arch: 'amd64' },
|
||||
{ pkg: 'rg', os: 'windows', arch: 'amd64' },
|
||||
{ pkg: 'caddy', os: 'macos', arch: 'amd64' },
|
||||
{ pkg: 'caddy', os: 'macos', arch: 'arm64' },
|
||||
{ pkg: 'caddy', os: 'linux', arch: 'amd64' },
|
||||
{ pkg: 'caddy', os: 'windows', arch: 'amd64' },
|
||||
];
|
||||
|
||||
// Fields to compare between live and local
|
||||
let COMPARE_FIELDS = ['version', 'os', 'arch', 'ext', 'libc', 'channel', 'download'];
|
||||
|
||||
async function main() {
|
||||
let failures = 0;
|
||||
let passes = 0;
|
||||
let skips = 0;
|
||||
|
||||
// Test 1: Unfiltered release list — compare structure and field values
|
||||
console.log('=== Test 1: Unfiltered /api/releases/{pkg}.json ===');
|
||||
console.log('');
|
||||
for (let pkg of ['bat', 'go', 'node', 'rg', 'jq', 'caddy']) {
|
||||
let liveFile = `${TESTDATA_DIR}/live_${pkg}.json`;
|
||||
let liveExists = await Fs.access(liveFile).then(
|
||||
function () { return true; },
|
||||
function () { return false; },
|
||||
);
|
||||
if (!liveExists) {
|
||||
console.log(` SKIP ${pkg}: no golden data`);
|
||||
skips++;
|
||||
continue;
|
||||
}
|
||||
|
||||
let liveJson = await Fs.readFile(liveFile, 'utf8');
|
||||
let liveReleases = JSON.parse(liveJson);
|
||||
|
||||
let localResult = await Releases.getReleases({
|
||||
pkg: pkg,
|
||||
ver: '',
|
||||
os: '',
|
||||
arch: '',
|
||||
libc: '',
|
||||
lts: false,
|
||||
channel: '',
|
||||
formats: [],
|
||||
limit: 100,
|
||||
});
|
||||
|
||||
let localReleases = localResult.releases;
|
||||
|
||||
// Compare OS vocabulary
|
||||
let liveOses = [...new Set(liveReleases.map(function (r) { return r.os; }))].sort();
|
||||
let localOses = [...new Set(localReleases.map(function (r) { return r.os; }))].sort();
|
||||
let osMatch = JSON.stringify(liveOses) === JSON.stringify(localOses);
|
||||
if (!osMatch) {
|
||||
console.log(` FAIL ${pkg} OS values: live=${JSON.stringify(liveOses)} local=${JSON.stringify(localOses)}`);
|
||||
failures++;
|
||||
} else {
|
||||
console.log(` PASS ${pkg} OS values: ${JSON.stringify(liveOses)}`);
|
||||
passes++;
|
||||
}
|
||||
|
||||
// Compare arch vocabulary
|
||||
let liveArches = [...new Set(liveReleases.map(function (r) { return r.arch; }))].sort();
|
||||
let localArches = [...new Set(localReleases.map(function (r) { return r.arch; }))].sort();
|
||||
let archMatch = JSON.stringify(liveArches) === JSON.stringify(localArches);
|
||||
if (!archMatch) {
|
||||
console.log(` FAIL ${pkg} arch values: live=${JSON.stringify(liveArches)} local=${JSON.stringify(localArches)}`);
|
||||
failures++;
|
||||
} else {
|
||||
console.log(` PASS ${pkg} arch values: ${JSON.stringify(liveArches)}`);
|
||||
passes++;
|
||||
}
|
||||
|
||||
// Compare latest version
|
||||
let liveLatest = liveReleases[0]?.version;
|
||||
let localLatest = localReleases[0]?.version;
|
||||
if (liveLatest !== localLatest) {
|
||||
// Version differences may be expected if cache is newer/older
|
||||
console.log(` WARN ${pkg} latest version: live=${liveLatest} local=${localLatest}`);
|
||||
} else {
|
||||
console.log(` PASS ${pkg} latest version: ${liveLatest}`);
|
||||
passes++;
|
||||
}
|
||||
|
||||
// Compare ext vocabulary
|
||||
let liveExts = [...new Set(liveReleases.map(function (r) { return r.ext; }))].sort();
|
||||
let localExts = [...new Set(localReleases.map(function (r) { return r.ext; }))].sort();
|
||||
let extMatch = JSON.stringify(liveExts) === JSON.stringify(localExts);
|
||||
if (!extMatch) {
|
||||
console.log(` FAIL ${pkg} ext values: live=${JSON.stringify(liveExts)} local=${JSON.stringify(localExts)}`);
|
||||
failures++;
|
||||
} else {
|
||||
console.log(` PASS ${pkg} ext values: ${JSON.stringify(liveExts)}`);
|
||||
passes++;
|
||||
}
|
||||
|
||||
// Compare that version strings don't have 'v' prefix
|
||||
let localHasVPrefix = localReleases.some(function (r) {
|
||||
return r.version.startsWith('v');
|
||||
});
|
||||
if (localHasVPrefix) {
|
||||
console.log(` FAIL ${pkg} versions have 'v' prefix (should be stripped)`);
|
||||
failures++;
|
||||
} else {
|
||||
console.log(` PASS ${pkg} no 'v' prefix on versions`);
|
||||
passes++;
|
||||
}
|
||||
}
|
||||
|
||||
// Test 2: Filtered queries — compare selected package for specific OS/arch
|
||||
console.log('');
|
||||
console.log('=== Test 2: Filtered /api/releases/{pkg}@stable.json?os=...&arch=... ===');
|
||||
console.log('');
|
||||
for (let tc of FILTERED_CASES) {
|
||||
let fname = `live_${tc.pkg}_os_${tc.os}_arch_${tc.arch}.json`;
|
||||
let liveFile = `${TESTDATA_DIR}/${fname}`;
|
||||
let liveExists = await Fs.access(liveFile).then(
|
||||
function () { return true; },
|
||||
function () { return false; },
|
||||
);
|
||||
if (!liveExists) {
|
||||
skips++;
|
||||
continue;
|
||||
}
|
||||
|
||||
let liveJson = await Fs.readFile(liveFile, 'utf8');
|
||||
let liveReleases = JSON.parse(liveJson);
|
||||
let liveFirst = liveReleases[0];
|
||||
if (!liveFirst || liveFirst.channel === 'error') {
|
||||
console.log(` SKIP ${tc.pkg} ${tc.os}/${tc.arch}: live returned error/empty`);
|
||||
skips++;
|
||||
continue;
|
||||
}
|
||||
|
||||
let localResult = await Releases.getReleases({
|
||||
pkg: tc.pkg,
|
||||
ver: '',
|
||||
os: tc.os,
|
||||
arch: tc.arch,
|
||||
libc: '',
|
||||
lts: false,
|
||||
channel: 'stable',
|
||||
formats: ['tar', 'zip', 'exe', 'xz'],
|
||||
limit: 1,
|
||||
});
|
||||
let localFirst = localResult.releases[0];
|
||||
|
||||
if (!localFirst || localFirst.channel === 'error') {
|
||||
console.log(` FAIL ${tc.pkg} ${tc.os}/${tc.arch}: local returned error/empty, live had ${liveFirst.version}`);
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
|
||||
let diffs = [];
|
||||
for (let field of COMPARE_FIELDS) {
|
||||
let liveVal = String(liveFirst[field] || '');
|
||||
let localVal = String(localFirst[field] || '');
|
||||
if (liveVal !== localVal) {
|
||||
// Version differences are OK if cache age differs
|
||||
if (field === 'version' || field === 'download' || field === 'date') {
|
||||
continue;
|
||||
}
|
||||
diffs.push(`${field}: live=${liveVal} local=${localVal}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (diffs.length > 0) {
|
||||
console.log(` FAIL ${tc.pkg} ${tc.os}/${tc.arch}: ${diffs.join(', ')}`);
|
||||
failures++;
|
||||
} else {
|
||||
let ver = localFirst.version;
|
||||
let ext = localFirst.ext;
|
||||
console.log(` PASS ${tc.pkg} ${tc.os}/${tc.arch}: v${ver} .${ext}`);
|
||||
passes++;
|
||||
}
|
||||
}
|
||||
|
||||
console.log('');
|
||||
console.log(`=== Results: ${passes} passed, ${failures} failed, ${skips} skipped ===`);
|
||||
if (failures > 0) {
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch(function (err) {
|
||||
console.error(err.stack);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -1,77 +0,0 @@
|
||||
'use strict';
|
||||
|
||||
// Broad sweep: test that all cached packages resolve on macOS arm64
|
||||
// and Linux amd64. Catches any package that completely fails to resolve.
|
||||
//
|
||||
// Usage: node _webi/test-broad-resolve.js
|
||||
|
||||
var Path = require('node:path');
|
||||
var InstallerServer = require('./serve-installer.js');
|
||||
var Builds = require('./builds.js');
|
||||
var BuildsCacher = require('./builds-cacher.js');
|
||||
|
||||
var UA_CASES = [
|
||||
{ label: 'macOS arm64', ua: 'aarch64/unknown Darwin/24.2.0 libc' },
|
||||
{ label: 'Linux amd64', ua: 'x86_64/unknown Linux/5.15.0 libc' },
|
||||
];
|
||||
|
||||
async function main() {
|
||||
console.log('Initializing build cache...');
|
||||
await Builds.init();
|
||||
console.log('');
|
||||
|
||||
var bc = BuildsCacher.create({
|
||||
caches: Path.join(__dirname, '../_cache'),
|
||||
installers: Path.join(__dirname, '..'),
|
||||
});
|
||||
var dirs = await bc.getProjectsByType();
|
||||
var pkgs = Object.keys(dirs.valid).sort();
|
||||
console.log('Testing ' + pkgs.length + ' packages...');
|
||||
console.log('');
|
||||
|
||||
var pass = 0;
|
||||
var fail = 0;
|
||||
var failures = [];
|
||||
|
||||
for (var i = 0; i < pkgs.length; i++) {
|
||||
var pkg = pkgs[i];
|
||||
for (var j = 0; j < UA_CASES.length; j++) {
|
||||
var tc = UA_CASES[j];
|
||||
try {
|
||||
var r = await InstallerServer.helper({
|
||||
unameAgent: tc.ua,
|
||||
projectName: pkg,
|
||||
tag: 'stable',
|
||||
formats: ['tar', 'exe', 'zip', 'xz', 'dmg'],
|
||||
libc: '',
|
||||
});
|
||||
var p = r[0];
|
||||
if (p.channel === 'error' || p.ext === 'err') {
|
||||
failures.push(pkg + ' ' + tc.label + ': error (v' + p.version + ')');
|
||||
fail++;
|
||||
} else {
|
||||
pass++;
|
||||
}
|
||||
} catch (e) {
|
||||
failures.push(pkg + ' ' + tc.label + ': ' + e.message.substring(0, 60));
|
||||
fail++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (failures.length > 0) {
|
||||
console.log('Failures:');
|
||||
for (var k = 0; k < failures.length; k++) {
|
||||
console.log(' FAIL ' + failures[k]);
|
||||
}
|
||||
console.log('');
|
||||
}
|
||||
|
||||
var total = pkgs.length * UA_CASES.length;
|
||||
console.log('=== ' + pass + '/' + total + ' passed (' + fail + ' failed) ===');
|
||||
}
|
||||
|
||||
main().catch(function (err) {
|
||||
console.error(err.stack);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -1,277 +0,0 @@
|
||||
'use strict';
|
||||
|
||||
// Tests that _cache JSON files are ready for direct use by the API endpoint
|
||||
// (transform-releases.js) WITHOUT needing normalize.js to fix up fields.
|
||||
//
|
||||
// These tests drive GOER to add missing features to ExportLegacy so that
|
||||
// normalize.js can be eliminated from the API path entirely.
|
||||
//
|
||||
// Usage: node _webi/test-cache-api-ready.js
|
||||
|
||||
var Fs = require('node:fs');
|
||||
var Os = require('node:os');
|
||||
var Path = require('node:path');
|
||||
|
||||
var CACHE_DIR = Path.join(Os.homedir(), '.cache/webi/legacy');
|
||||
|
||||
// Packages to spot-check (mix of github_releases, custom releases.js, gittag)
|
||||
var CHECK_PKGS = [
|
||||
'bat',
|
||||
'caddy',
|
||||
'go',
|
||||
'hugo',
|
||||
'jq',
|
||||
'node',
|
||||
'rg',
|
||||
'terraform',
|
||||
'zig',
|
||||
];
|
||||
|
||||
function loadReleases(dir, pkg) {
|
||||
var file = Path.join(dir, pkg + '.json');
|
||||
if (!Fs.existsSync(file)) {
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
return JSON.parse(Fs.readFileSync(file, 'utf8'));
|
||||
} catch (e) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async function main() {
|
||||
var passes = 0;
|
||||
var failures = 0;
|
||||
|
||||
if (!Fs.existsSync(CACHE_DIR)) {
|
||||
console.error('No cache directory at ' + CACHE_DIR);
|
||||
process.exit(1);
|
||||
}
|
||||
console.log('Testing ' + CACHE_DIR);
|
||||
console.log('');
|
||||
|
||||
// ================================================================
|
||||
// Test 1: Version has no 'v' prefix
|
||||
// ================================================================
|
||||
console.log('=== Test 1: Version v-prefix stripped ===');
|
||||
console.log('');
|
||||
|
||||
for (var vi = 0; vi < CHECK_PKGS.length; vi++) {
|
||||
var vpkg = CHECK_PKGS[vi];
|
||||
var vdata = loadReleases(CACHE_DIR, vpkg);
|
||||
if (!vdata) {
|
||||
console.log(' SKIP ' + vpkg + ': no data');
|
||||
continue;
|
||||
}
|
||||
|
||||
var vPrefixed = 0;
|
||||
for (var vri = 0; vri < vdata.releases.length; vri++) {
|
||||
var ver = vdata.releases[vri].version || '';
|
||||
if (ver.startsWith('v')) {
|
||||
vPrefixed++;
|
||||
}
|
||||
}
|
||||
|
||||
if (vPrefixed === 0) {
|
||||
console.log(' PASS ' + vpkg + ': no v-prefixed versions');
|
||||
passes++;
|
||||
} else {
|
||||
console.log(' FAIL ' + vpkg + ': ' + vPrefixed + '/' + vdata.releases.length + ' versions have v prefix');
|
||||
failures++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 2: libc is never empty string (should be "none", "gnu", "musl", "msvc")
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 2: libc never empty ===');
|
||||
console.log('');
|
||||
|
||||
for (var li = 0; li < CHECK_PKGS.length; li++) {
|
||||
var lpkg = CHECK_PKGS[li];
|
||||
var ldata = loadReleases(CACHE_DIR, lpkg);
|
||||
if (!ldata) {
|
||||
console.log(' SKIP ' + lpkg + ': no data');
|
||||
continue;
|
||||
}
|
||||
|
||||
var emptyLibc = 0;
|
||||
for (var lri = 0; lri < ldata.releases.length; lri++) {
|
||||
if (ldata.releases[lri].libc === '') {
|
||||
emptyLibc++;
|
||||
}
|
||||
}
|
||||
|
||||
if (emptyLibc === 0) {
|
||||
console.log(' PASS ' + lpkg + ': all entries have libc set');
|
||||
passes++;
|
||||
} else {
|
||||
console.log(' FAIL ' + lpkg + ': ' + emptyLibc + '/' + ldata.releases.length + ' entries have empty libc (should be "none")');
|
||||
failures++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 3: ext has no leading dot
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 3: ext has no leading dot ===');
|
||||
console.log('');
|
||||
|
||||
for (var ei = 0; ei < CHECK_PKGS.length; ei++) {
|
||||
var epkg = CHECK_PKGS[ei];
|
||||
var edata = loadReleases(CACHE_DIR, epkg);
|
||||
if (!edata) {
|
||||
console.log(' SKIP ' + epkg + ': no data');
|
||||
continue;
|
||||
}
|
||||
|
||||
var dotExt = 0;
|
||||
for (var eri = 0; eri < edata.releases.length; eri++) {
|
||||
var ext = edata.releases[eri].ext || '';
|
||||
if (ext.startsWith('.')) {
|
||||
dotExt++;
|
||||
}
|
||||
}
|
||||
|
||||
if (dotExt === 0) {
|
||||
console.log(' PASS ' + epkg + ': no leading dots on ext');
|
||||
passes++;
|
||||
} else {
|
||||
console.log(' FAIL ' + epkg + ': ' + dotExt + '/' + edata.releases.length + ' entries have leading dot on ext');
|
||||
failures++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 4: bare binaries have ext "exe" (not empty)
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 4: bare binaries have ext "exe" ===');
|
||||
console.log('');
|
||||
|
||||
var barePkgs = ['jq', 'bat', 'rg', 'caddy'];
|
||||
for (var bi = 0; bi < barePkgs.length; bi++) {
|
||||
var bpkg = barePkgs[bi];
|
||||
var bdata = loadReleases(CACHE_DIR, bpkg);
|
||||
if (!bdata) {
|
||||
console.log(' SKIP ' + bpkg + ': no data');
|
||||
continue;
|
||||
}
|
||||
|
||||
var emptyExt = 0;
|
||||
for (var bri = 0; bri < bdata.releases.length; bri++) {
|
||||
var brel = bdata.releases[bri];
|
||||
if (brel.ext === '' && brel.name && !brel.name.includes('.')) {
|
||||
emptyExt++;
|
||||
}
|
||||
}
|
||||
|
||||
if (emptyExt === 0) {
|
||||
console.log(' PASS ' + bpkg + ': no bare binaries with empty ext');
|
||||
passes++;
|
||||
} else {
|
||||
console.log(' FAIL ' + bpkg + ': ' + emptyExt + ' bare binaries have empty ext (should be "exe")');
|
||||
failures++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 5: Top-level summary arrays present
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 5: Summary arrays (oses, arches, libcs, formats) ===');
|
||||
console.log('');
|
||||
|
||||
for (var si = 0; si < CHECK_PKGS.length; si++) {
|
||||
var spkg = CHECK_PKGS[si];
|
||||
var sdata = loadReleases(CACHE_DIR, spkg);
|
||||
if (!sdata) {
|
||||
console.log(' SKIP ' + spkg + ': no data');
|
||||
continue;
|
||||
}
|
||||
|
||||
var missing = [];
|
||||
if (!Array.isArray(sdata.oses)) { missing.push('oses'); }
|
||||
if (!Array.isArray(sdata.arches)) { missing.push('arches'); }
|
||||
if (!Array.isArray(sdata.libcs)) { missing.push('libcs'); }
|
||||
if (!Array.isArray(sdata.formats)) { missing.push('formats'); }
|
||||
|
||||
if (missing.length === 0) {
|
||||
// Verify they contain the right values
|
||||
var hasMacos = sdata.oses.includes('macos') || sdata.oses.includes('darwin');
|
||||
var hasLinux = sdata.oses.includes('linux');
|
||||
var ok = true;
|
||||
if (sdata.releases.some(function (r) { return r.os === 'macos' || r.os === 'darwin'; }) && !hasMacos) {
|
||||
console.log(' FAIL ' + spkg + ': oses array missing macos/darwin');
|
||||
failures++;
|
||||
ok = false;
|
||||
}
|
||||
if (sdata.releases.some(function (r) { return r.os === 'linux'; }) && !hasLinux) {
|
||||
console.log(' FAIL ' + spkg + ': oses array missing linux');
|
||||
failures++;
|
||||
ok = false;
|
||||
}
|
||||
if (ok) {
|
||||
console.log(' PASS ' + spkg + ': has oses, arches, libcs, formats');
|
||||
passes++;
|
||||
}
|
||||
} else {
|
||||
console.log(' FAIL ' + spkg + ': missing top-level arrays: ' + missing.join(', '));
|
||||
failures++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 6: Version sort order — stable before beta, newest first
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 6: Version sort order ===');
|
||||
console.log('');
|
||||
|
||||
var sortPkgs = ['go', 'node', 'terraform'];
|
||||
for (var oi = 0; oi < sortPkgs.length; oi++) {
|
||||
var opkg = sortPkgs[oi];
|
||||
var odata = loadReleases(CACHE_DIR, opkg);
|
||||
if (!odata) {
|
||||
console.log(' SKIP ' + opkg + ': no data');
|
||||
continue;
|
||||
}
|
||||
|
||||
// Find first stable release
|
||||
var firstStable = odata.releases.find(function (r) { return r.channel === 'stable'; });
|
||||
// Find first beta release
|
||||
var firstBeta = odata.releases.find(function (r) { return r.channel === 'beta'; });
|
||||
|
||||
if (!firstStable) {
|
||||
console.log(' SKIP ' + opkg + ': no stable release');
|
||||
continue;
|
||||
}
|
||||
|
||||
// The first entry overall should be a stable release (newest stable > any beta)
|
||||
// unless the beta is a newer version number
|
||||
var firstEntry = odata.releases[0];
|
||||
if (firstEntry.channel === 'stable') {
|
||||
console.log(' PASS ' + opkg + ': first entry is stable (' + firstEntry.version + ')');
|
||||
passes++;
|
||||
} else {
|
||||
console.log(' FAIL ' + opkg + ': first entry is ' + firstEntry.channel + ' (' + firstEntry.version + '), expected stable (' + firstStable.version + ')');
|
||||
failures++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Summary
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Results: ' + passes + ' passed, ' + failures + ' failed ===');
|
||||
if (failures > 0) {
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch(function (err) {
|
||||
console.error(err.stack);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -1,687 +0,0 @@
|
||||
'use strict';
|
||||
|
||||
// Cache compatibility tests: verify that Go-generated cache files work
|
||||
// correctly with the Node build-classifier and installer resolution pipeline.
|
||||
//
|
||||
// Tests cover:
|
||||
// 1. Cache completeness — all packages have releases, required fields present
|
||||
// 2. Format selection — correct archive format per platform
|
||||
// 3. Edge-case platforms — FreeBSD, ARM variants, musl/Alpine
|
||||
// 4. Script generation — installer scripts render without error
|
||||
// 5. API compat — transform-releases output matches expected vocabulary
|
||||
// 6. Version format — no 'v' prefix, valid semver-ish
|
||||
// 7. Channel detection — stable vs beta correctly identified
|
||||
//
|
||||
// Usage: node _webi/test-cache-compat.js
|
||||
|
||||
var Fs = require('node:fs');
|
||||
var Path = require('node:path');
|
||||
|
||||
var InstallerServer = require('./serve-installer.js');
|
||||
var Builds = require('./builds.js');
|
||||
var Releases = require('./transform-releases.js');
|
||||
|
||||
var CACHE_DIR = Path.join(require('node:os').homedir(), '.cache/webi/legacy');
|
||||
|
||||
// ====================================================================
|
||||
// Test 1: Cache completeness — every package has releases with valid fields
|
||||
// ====================================================================
|
||||
|
||||
// Packages that are expected to have binary releases (not gittag-only)
|
||||
var BINARY_PKGS = [
|
||||
'bat',
|
||||
'caddy',
|
||||
'cmake',
|
||||
'delta',
|
||||
'deno',
|
||||
'fd',
|
||||
'fzf',
|
||||
'gh',
|
||||
'go',
|
||||
'goreleaser',
|
||||
'hugo',
|
||||
'jq',
|
||||
'k9s',
|
||||
'node',
|
||||
'ollama',
|
||||
'rg',
|
||||
'shellcheck',
|
||||
'shfmt',
|
||||
'syncthing',
|
||||
'terraform',
|
||||
'xz',
|
||||
'yq',
|
||||
'zig',
|
||||
'zoxide',
|
||||
];
|
||||
|
||||
// Packages that are gittag/source-only — must have releases but os/arch may be special
|
||||
var GITTAG_PKGS = [
|
||||
'aliasman',
|
||||
'serviceman',
|
||||
'vim-airline',
|
||||
'vim-go',
|
||||
'vim-sensible',
|
||||
];
|
||||
|
||||
// ====================================================================
|
||||
// Test 2: Format selection — correct ext per platform
|
||||
// ====================================================================
|
||||
|
||||
// For these packages, verify the resolved format is correct for each platform.
|
||||
// Linux/macOS should get .tar.gz or .tar.xz (not .zip except for specific packages).
|
||||
// Windows should get .zip or .exe (not .tar.gz).
|
||||
var FORMAT_CASES = [
|
||||
// Go projects — should be .tar.gz on Linux/macOS, .zip on Windows
|
||||
{
|
||||
label: 'go Linux amd64 format',
|
||||
pkg: 'go',
|
||||
ua: 'x86_64/unknown Linux/5.15.0 libc',
|
||||
expectExt: 'tar.gz',
|
||||
},
|
||||
{
|
||||
label: 'go Windows amd64 format',
|
||||
pkg: 'go',
|
||||
ua: 'x86_64/unknown Windows/10.0.19041 msvc',
|
||||
expectExt: 'zip',
|
||||
},
|
||||
{
|
||||
label: 'go macOS arm64 format',
|
||||
pkg: 'go',
|
||||
ua: 'aarch64/unknown Darwin/24.2.0 libc',
|
||||
expectExt: 'tar.gz',
|
||||
},
|
||||
// Rust projects — should be .tar.gz on Linux/macOS, .zip on Windows
|
||||
{
|
||||
label: 'bat Linux amd64 format',
|
||||
pkg: 'bat',
|
||||
ua: 'x86_64/unknown Linux/5.15.0 libc',
|
||||
expectExt: 'tar.gz',
|
||||
},
|
||||
{
|
||||
label: 'bat Windows amd64 format',
|
||||
pkg: 'bat',
|
||||
ua: 'x86_64/unknown Windows/10.0.19041 msvc',
|
||||
expectExt: 'zip',
|
||||
},
|
||||
{
|
||||
label: 'rg Linux amd64 format',
|
||||
pkg: 'rg',
|
||||
ua: 'x86_64/unknown Linux/5.15.0 libc',
|
||||
expectExt: 'tar.gz',
|
||||
},
|
||||
{
|
||||
label: 'rg Windows amd64 format',
|
||||
pkg: 'rg',
|
||||
ua: 'x86_64/unknown Windows/10.0.19041 msvc',
|
||||
expectExt: 'zip',
|
||||
},
|
||||
// Node — uses .tar.xz on Linux/macOS
|
||||
{
|
||||
label: 'node Linux amd64 format',
|
||||
pkg: 'node',
|
||||
ua: 'x86_64/unknown Linux/5.15.0 libc',
|
||||
expectExt: 'tar.xz',
|
||||
},
|
||||
{
|
||||
label: 'node macOS arm64 format',
|
||||
pkg: 'node',
|
||||
ua: 'aarch64/unknown Darwin/24.2.0 libc',
|
||||
expectExt: 'tar.xz',
|
||||
},
|
||||
// delta — Rust, .tar.gz on Linux/macOS
|
||||
{
|
||||
label: 'delta Linux amd64 format',
|
||||
pkg: 'delta',
|
||||
ua: 'x86_64/unknown Linux/5.15.0 libc',
|
||||
expectExt: 'tar.gz',
|
||||
},
|
||||
// shfmt — Go, bare exe on Linux, .exe on Windows
|
||||
{
|
||||
label: 'shfmt Linux amd64 format',
|
||||
pkg: 'shfmt',
|
||||
ua: 'x86_64/unknown Linux/5.15.0 libc',
|
||||
expectExt: 'exe',
|
||||
},
|
||||
];
|
||||
|
||||
// ====================================================================
|
||||
// Test 3: Edge-case platforms
|
||||
// ====================================================================
|
||||
|
||||
var EDGE_CASES = [
|
||||
// Linux ARM variants
|
||||
{
|
||||
label: 'go Linux aarch64',
|
||||
pkg: 'go',
|
||||
ua: 'aarch64/unknown Linux/5.15.0 libc',
|
||||
expectOs: 'linux',
|
||||
expectNotError: true,
|
||||
},
|
||||
{
|
||||
label: 'node Linux aarch64',
|
||||
pkg: 'node',
|
||||
ua: 'aarch64/unknown Linux/5.15.0 libc',
|
||||
expectOs: 'linux',
|
||||
expectNotError: true,
|
||||
},
|
||||
// Alpine/musl — bat should resolve to musl build
|
||||
{
|
||||
label: 'bat Linux musl amd64',
|
||||
pkg: 'bat',
|
||||
ua: 'x86_64/unknown Linux/5.15.0 musl',
|
||||
expectOs: 'linux',
|
||||
expectNotError: true,
|
||||
},
|
||||
{
|
||||
label: 'rg Linux musl amd64',
|
||||
pkg: 'rg',
|
||||
ua: 'x86_64/unknown Linux/5.15.0 musl',
|
||||
expectOs: 'linux',
|
||||
expectNotError: true,
|
||||
},
|
||||
// node musl — separate musl build
|
||||
{
|
||||
label: 'node Linux musl amd64',
|
||||
pkg: 'node',
|
||||
ua: 'x86_64/unknown Linux/5.15.0 musl',
|
||||
expectOs: 'linux',
|
||||
expectNotError: true,
|
||||
},
|
||||
// FreeBSD — packages that have freebsd builds
|
||||
{
|
||||
label: 'syncthing FreeBSD amd64',
|
||||
pkg: 'syncthing',
|
||||
ua: 'x86_64/unknown FreeBSD/14.0 libc',
|
||||
expectOs: 'freebsd',
|
||||
expectNotError: true,
|
||||
},
|
||||
{
|
||||
label: 'caddy FreeBSD amd64',
|
||||
pkg: 'caddy',
|
||||
ua: 'x86_64/unknown FreeBSD/14.0 libc',
|
||||
expectOs: 'freebsd',
|
||||
expectNotError: true,
|
||||
},
|
||||
// Windows aarch64 — should fallback to amd64 for most packages
|
||||
{
|
||||
label: 'go Windows aarch64',
|
||||
pkg: 'go',
|
||||
ua: 'aarch64/unknown Windows/10.0.22000 msvc',
|
||||
expectOs: 'windows',
|
||||
expectNotError: true,
|
||||
},
|
||||
];
|
||||
|
||||
// ====================================================================
|
||||
// Test 4: Script generation smoke tests
|
||||
// ====================================================================
|
||||
|
||||
var SCRIPT_CASES = [
|
||||
{ pkg: 'bat', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'bat Linux' },
|
||||
{ pkg: 'go', ua: 'aarch64/unknown Darwin/24.2.0 libc', label: 'go macOS' },
|
||||
{ pkg: 'node', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'node Linux' },
|
||||
{ pkg: 'rg', ua: 'x86_64/unknown Windows/10.0.19041 msvc', label: 'rg Windows' },
|
||||
{ pkg: 'jq', ua: 'aarch64/unknown Darwin/24.2.0 libc', label: 'jq macOS' },
|
||||
{ pkg: 'caddy', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'caddy Linux' },
|
||||
{ pkg: 'shellcheck', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'shellcheck Linux' },
|
||||
{ pkg: 'shfmt', ua: 'aarch64/unknown Darwin/24.2.0 libc', label: 'shfmt macOS' },
|
||||
{ pkg: 'hugo', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'hugo Linux' },
|
||||
{ pkg: 'delta', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'delta Linux' },
|
||||
{ pkg: 'fd', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'fd Linux' },
|
||||
{ pkg: 'fzf', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'fzf Linux' },
|
||||
{ pkg: 'zoxide', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'zoxide Linux' },
|
||||
{ pkg: 'k9s', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'k9s Linux' },
|
||||
{ pkg: 'yq', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'yq Linux' },
|
||||
];
|
||||
|
||||
// ====================================================================
|
||||
// Test 5: API compat — transform-releases vocabulary
|
||||
// ====================================================================
|
||||
|
||||
// The /api/releases/ endpoint uses normalize.js which has its own OS/arch names.
|
||||
// Verify that the Go cache produces correct values after normalization.
|
||||
var API_VOCAB_CASES = [
|
||||
{
|
||||
label: 'bat has macos+linux+windows',
|
||||
pkg: 'bat',
|
||||
expectOses: ['linux', 'macos', 'windows'],
|
||||
},
|
||||
{
|
||||
label: 'go has macos+linux+windows',
|
||||
pkg: 'go',
|
||||
expectOses: ['linux', 'macos', 'windows'],
|
||||
},
|
||||
{
|
||||
label: 'node has linux+macos+windows',
|
||||
pkg: 'node',
|
||||
expectOses: ['linux', 'macos', 'windows'],
|
||||
},
|
||||
{
|
||||
label: 'syncthing has freebsd+linux+macos+windows',
|
||||
pkg: 'syncthing',
|
||||
expectOses: ['freebsd', 'linux', 'macos', 'windows'],
|
||||
},
|
||||
{
|
||||
label: 'bat has amd64+arm64 arches',
|
||||
pkg: 'bat',
|
||||
expectArches: ['amd64', 'arm64'],
|
||||
},
|
||||
{
|
||||
label: 'go has amd64+arm64 arches',
|
||||
pkg: 'go',
|
||||
expectArches: ['amd64', 'arm64'],
|
||||
},
|
||||
];
|
||||
|
||||
// ====================================================================
|
||||
// Test 6: Version format validation
|
||||
// ====================================================================
|
||||
|
||||
var VERSION_CHECKS = [
|
||||
'bat',
|
||||
'go',
|
||||
'node',
|
||||
'rg',
|
||||
'caddy',
|
||||
'hugo',
|
||||
'terraform',
|
||||
'jq',
|
||||
'cmake',
|
||||
'zig',
|
||||
];
|
||||
|
||||
// ====================================================================
|
||||
// Test 7: Channel detection
|
||||
// ====================================================================
|
||||
|
||||
// Packages that should have both stable and beta releases
|
||||
var CHANNEL_CASES = [
|
||||
{
|
||||
label: 'go has stable releases',
|
||||
pkg: 'go',
|
||||
channel: 'stable',
|
||||
expectMinCount: 10,
|
||||
},
|
||||
{
|
||||
label: 'node has stable releases',
|
||||
pkg: 'node',
|
||||
channel: 'stable',
|
||||
expectMinCount: 10,
|
||||
},
|
||||
{
|
||||
label: 'cmake has stable releases',
|
||||
pkg: 'cmake',
|
||||
channel: 'stable',
|
||||
expectMinCount: 5,
|
||||
},
|
||||
];
|
||||
|
||||
// ====================================================================
|
||||
// Runner
|
||||
// ====================================================================
|
||||
|
||||
async function main() {
|
||||
var passes = 0;
|
||||
var failures = 0;
|
||||
var knowns = 0;
|
||||
var skips = 0;
|
||||
|
||||
console.log('Initializing build cache...');
|
||||
await Builds.init();
|
||||
|
||||
if (!Fs.existsSync(CACHE_DIR)) {
|
||||
console.error('No cache directory at ' + CACHE_DIR);
|
||||
process.exit(1);
|
||||
}
|
||||
var cachePath = CACHE_DIR;
|
||||
console.log('Using cache: ' + cachePath);
|
||||
console.log('');
|
||||
|
||||
// ================================================================
|
||||
// Test 1: Cache completeness
|
||||
// ================================================================
|
||||
console.log('=== Test 1: Cache Completeness ===');
|
||||
console.log('');
|
||||
|
||||
for (var bi = 0; bi < BINARY_PKGS.length; bi++) {
|
||||
var bpkg = BINARY_PKGS[bi];
|
||||
var bfile = Path.join(cachePath, bpkg + '.json');
|
||||
if (!Fs.existsSync(bfile)) {
|
||||
console.log(' FAIL ' + bpkg + ': cache file missing');
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
var bdata = JSON.parse(Fs.readFileSync(bfile, 'utf8'));
|
||||
if (!bdata.releases || bdata.releases.length === 0) {
|
||||
console.log(' FAIL ' + bpkg + ': 0 releases');
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
// Check that at least one release has a download URL
|
||||
var hasDownload = bdata.releases.some(function (r) {
|
||||
return r.download && r.download.startsWith('http');
|
||||
});
|
||||
if (!hasDownload) {
|
||||
console.log(' FAIL ' + bpkg + ': no releases with download URLs');
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
console.log(' PASS ' + bpkg + ': ' + bdata.releases.length + ' releases');
|
||||
passes++;
|
||||
}
|
||||
|
||||
for (var gi = 0; gi < GITTAG_PKGS.length; gi++) {
|
||||
var gpkg = GITTAG_PKGS[gi];
|
||||
var gfile = Path.join(cachePath, gpkg + '.json');
|
||||
if (!Fs.existsSync(gfile)) {
|
||||
console.log(' FAIL ' + gpkg + ' (gittag): cache file missing');
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
var gdata = JSON.parse(Fs.readFileSync(gfile, 'utf8'));
|
||||
if (!gdata.releases || gdata.releases.length === 0) {
|
||||
console.log(' FAIL ' + gpkg + ' (gittag): 0 releases');
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
console.log(' PASS ' + gpkg + ' (gittag): ' + gdata.releases.length + ' releases');
|
||||
passes++;
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 2: Format selection
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 2: Format Selection ===');
|
||||
console.log('');
|
||||
|
||||
for (var fi = 0; fi < FORMAT_CASES.length; fi++) {
|
||||
var ftc = FORMAT_CASES[fi];
|
||||
try {
|
||||
var fr = await InstallerServer.helper({
|
||||
unameAgent: ftc.ua,
|
||||
projectName: ftc.pkg,
|
||||
tag: 'stable',
|
||||
formats: ['tar', 'exe', 'zip', 'xz', 'dmg'],
|
||||
libc: '',
|
||||
});
|
||||
var fpkg = fr[0];
|
||||
if (fpkg.channel === 'error') {
|
||||
console.log(' FAIL ' + ftc.label + ': resolved to error');
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
if (fpkg.ext !== ftc.expectExt) {
|
||||
console.log(' FAIL ' + ftc.label + ': got .' + fpkg.ext + ' want .' + ftc.expectExt);
|
||||
failures++;
|
||||
} else {
|
||||
console.log(' PASS ' + ftc.label + ': .' + fpkg.ext);
|
||||
passes++;
|
||||
}
|
||||
} catch (e) {
|
||||
console.log(' ERROR ' + ftc.label + ': ' + e.message);
|
||||
failures++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 3: Edge-case platforms
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 3: Edge-Case Platforms ===');
|
||||
console.log('');
|
||||
|
||||
for (var ei = 0; ei < EDGE_CASES.length; ei++) {
|
||||
var etc = EDGE_CASES[ei];
|
||||
try {
|
||||
var er = await InstallerServer.helper({
|
||||
unameAgent: etc.ua,
|
||||
projectName: etc.pkg,
|
||||
tag: 'stable',
|
||||
formats: ['tar', 'exe', 'zip', 'xz', 'dmg'],
|
||||
libc: '',
|
||||
});
|
||||
var epkg = er[0];
|
||||
if (etc.expectNotError && epkg.channel === 'error') {
|
||||
console.log(' FAIL ' + etc.label + ': resolved to error (v' + epkg.version + ')');
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
if (etc.expectOs && epkg.os !== etc.expectOs) {
|
||||
console.log(' FAIL ' + etc.label + ': os=' + epkg.os + ' want=' + etc.expectOs);
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
console.log(' PASS ' + etc.label + ': v' + epkg.version + ' .' + epkg.ext);
|
||||
passes++;
|
||||
} catch (e) {
|
||||
if (etc.known) {
|
||||
console.log(' KNOWN ' + etc.label + ': ' + e.message);
|
||||
knowns++;
|
||||
} else {
|
||||
console.log(' FAIL ' + etc.label + ': ' + e.message);
|
||||
failures++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 4: Script generation smoke tests
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 4: Script Generation ===');
|
||||
console.log('');
|
||||
|
||||
for (var si = 0; si < SCRIPT_CASES.length; si++) {
|
||||
var stc = SCRIPT_CASES[si];
|
||||
try {
|
||||
var script = await InstallerServer.serveInstaller(
|
||||
'https://webi.sh',
|
||||
stc.ua,
|
||||
stc.pkg,
|
||||
'stable',
|
||||
'sh',
|
||||
['tar', 'exe', 'zip', 'xz', 'dmg'],
|
||||
'',
|
||||
);
|
||||
|
||||
// Script must contain WEBI_PKG_URL and WEBI_VERSION
|
||||
var hasUrl = /WEBI_PKG_URL='[^']+'/m.test(script);
|
||||
var hasVersion = /WEBI_VERSION='[^']+'/m.test(script);
|
||||
var hasExt = /WEBI_EXT='[^']+'/m.test(script);
|
||||
|
||||
if (!hasUrl) {
|
||||
console.log(' FAIL ' + stc.label + ': missing WEBI_PKG_URL');
|
||||
failures++;
|
||||
} else if (!hasVersion) {
|
||||
console.log(' FAIL ' + stc.label + ': missing WEBI_VERSION');
|
||||
failures++;
|
||||
} else if (!hasExt) {
|
||||
console.log(' FAIL ' + stc.label + ': missing WEBI_EXT');
|
||||
failures++;
|
||||
} else {
|
||||
var vMatch = script.match(/WEBI_VERSION='([^']+)'/);
|
||||
var extMatch = script.match(/WEBI_EXT='([^']+)'/);
|
||||
console.log(' PASS ' + stc.label + ': v' + vMatch[1] + ' .' + extMatch[1]);
|
||||
passes++;
|
||||
}
|
||||
} catch (e) {
|
||||
console.log(' FAIL ' + stc.label + ': ' + e.message.substring(0, 80));
|
||||
failures++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 5: API compat — transform-releases vocabulary
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 5: API Vocabulary (transform-releases) ===');
|
||||
console.log('');
|
||||
|
||||
for (var ai = 0; ai < API_VOCAB_CASES.length; ai++) {
|
||||
var atc = API_VOCAB_CASES[ai];
|
||||
try {
|
||||
var ares = await Releases.getReleases({
|
||||
pkg: atc.pkg,
|
||||
ver: '',
|
||||
os: '',
|
||||
arch: '',
|
||||
libc: '',
|
||||
lts: false,
|
||||
channel: '',
|
||||
formats: [],
|
||||
limit: 100,
|
||||
});
|
||||
var arels = ares.releases;
|
||||
|
||||
if (atc.expectOses) {
|
||||
var actualOses = [];
|
||||
for (var oi = 0; oi < arels.length; oi++) {
|
||||
if (actualOses.indexOf(arels[oi].os) === -1) {
|
||||
actualOses.push(arels[oi].os);
|
||||
}
|
||||
}
|
||||
actualOses.sort();
|
||||
var missingOses = [];
|
||||
for (var mi = 0; mi < atc.expectOses.length; mi++) {
|
||||
if (actualOses.indexOf(atc.expectOses[mi]) === -1) {
|
||||
missingOses.push(atc.expectOses[mi]);
|
||||
}
|
||||
}
|
||||
if (missingOses.length > 0) {
|
||||
console.log(' FAIL ' + atc.label + ': missing ' + JSON.stringify(missingOses) + ' (has ' + JSON.stringify(actualOses) + ')');
|
||||
failures++;
|
||||
} else {
|
||||
console.log(' PASS ' + atc.label + ': ' + JSON.stringify(atc.expectOses));
|
||||
passes++;
|
||||
}
|
||||
}
|
||||
|
||||
if (atc.expectArches) {
|
||||
var actualArches = [];
|
||||
for (var ari = 0; ari < arels.length; ari++) {
|
||||
if (arels[ari].arch && actualArches.indexOf(arels[ari].arch) === -1) {
|
||||
actualArches.push(arels[ari].arch);
|
||||
}
|
||||
}
|
||||
actualArches.sort();
|
||||
var missingArches = [];
|
||||
for (var mai = 0; mai < atc.expectArches.length; mai++) {
|
||||
if (actualArches.indexOf(atc.expectArches[mai]) === -1) {
|
||||
missingArches.push(atc.expectArches[mai]);
|
||||
}
|
||||
}
|
||||
if (missingArches.length > 0) {
|
||||
console.log(' FAIL ' + atc.label + ': missing ' + JSON.stringify(missingArches) + ' (has ' + JSON.stringify(actualArches) + ')');
|
||||
failures++;
|
||||
} else {
|
||||
console.log(' PASS ' + atc.label);
|
||||
passes++;
|
||||
}
|
||||
}
|
||||
} catch (e) {
|
||||
console.log(' FAIL ' + atc.label + ': ' + e.message);
|
||||
failures++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 6: Version format
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 6: Version Format ===');
|
||||
console.log('');
|
||||
|
||||
// Version format check: the Go cache may have 'v' prefixes (that's OK —
|
||||
// normalize.js and serve-installer.js strip them). What matters is that
|
||||
// after normalization via transform-releases, versions have no 'v' prefix.
|
||||
for (var vi = 0; vi < VERSION_CHECKS.length; vi++) {
|
||||
var vpkg = VERSION_CHECKS[vi];
|
||||
try {
|
||||
var vres = await Releases.getReleases({
|
||||
pkg: vpkg,
|
||||
ver: '',
|
||||
os: '',
|
||||
arch: '',
|
||||
libc: '',
|
||||
lts: false,
|
||||
channel: '',
|
||||
formats: [],
|
||||
limit: 10,
|
||||
});
|
||||
var vrels = vres.releases;
|
||||
var badVersions = [];
|
||||
for (var vri = 0; vri < vrels.length; vri++) {
|
||||
var ver = vrels[vri].version;
|
||||
if (!ver) {
|
||||
badVersions.push('(empty)');
|
||||
break;
|
||||
}
|
||||
if (ver.startsWith('v')) {
|
||||
badVersions.push(ver);
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (badVersions.length > 0) {
|
||||
console.log(' FAIL ' + vpkg + ': v-prefix not stripped: ' + badVersions[0]);
|
||||
failures++;
|
||||
} else {
|
||||
console.log(' PASS ' + vpkg + ': latest=' + (vrels[0] ? vrels[0].version : '?'));
|
||||
passes++;
|
||||
}
|
||||
} catch (e) {
|
||||
console.log(' FAIL ' + vpkg + ': ' + e.message);
|
||||
failures++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 7: Channel detection
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 7: Channel Detection ===');
|
||||
console.log('');
|
||||
|
||||
for (var ci = 0; ci < CHANNEL_CASES.length; ci++) {
|
||||
var ctc = CHANNEL_CASES[ci];
|
||||
try {
|
||||
var cres = await Releases.getReleases({
|
||||
pkg: ctc.pkg,
|
||||
ver: '',
|
||||
os: '',
|
||||
arch: '',
|
||||
libc: '',
|
||||
lts: false,
|
||||
channel: ctc.channel,
|
||||
formats: [],
|
||||
limit: 100,
|
||||
});
|
||||
var count = cres.releases.length;
|
||||
if (count < ctc.expectMinCount) {
|
||||
console.log(' FAIL ' + ctc.label + ': only ' + count + ' (want >=' + ctc.expectMinCount + ')');
|
||||
failures++;
|
||||
} else {
|
||||
console.log(' PASS ' + ctc.label + ': ' + count + ' releases');
|
||||
passes++;
|
||||
}
|
||||
} catch (e) {
|
||||
console.log(' FAIL ' + ctc.label + ': ' + e.message);
|
||||
failures++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Summary
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Results: ' + passes + ' passed, ' + failures + ' failed, ' + knowns + ' known, ' + skips + ' skipped ===');
|
||||
if (failures > 0) {
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch(function (err) {
|
||||
console.error(err.stack);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -1,339 +0,0 @@
|
||||
'use strict';
|
||||
|
||||
// Fleet-wide diff: compare a candidate host (e.g. beta.webi.sh) against
|
||||
// production for every cached package, across multiple OS/arch combos.
|
||||
// Outputs TSV for grep/sort.
|
||||
//
|
||||
// Usage:
|
||||
// node _webi/test-fleet-diff.js
|
||||
// --cand-url=https://beta.webi.sh
|
||||
// --prod-url=https://webinstall.dev # default
|
||||
// --kind=api # default; or "installer"
|
||||
// --pkgs=bat,go,... # default: all cached packages
|
||||
// --concurrency=8 # default
|
||||
// --out=fleet-api.tsv # default: stdout
|
||||
|
||||
let Fs = require('node:fs/promises');
|
||||
let Os = require('node:os');
|
||||
let Path = require('node:path');
|
||||
let Https = require('node:https');
|
||||
|
||||
let CACHE_DIR = Path.join(Os.homedir(), '.cache/webi/legacy');
|
||||
|
||||
function arg(name, dflt) {
|
||||
for (let a of process.argv) {
|
||||
if (a.startsWith(`--${name}=`)) {
|
||||
return a.slice(name.length + 3);
|
||||
}
|
||||
}
|
||||
return dflt;
|
||||
}
|
||||
|
||||
let CAND_URL = arg('cand-url', 'https://beta.webi.sh').replace(/\/+$/, '');
|
||||
let PROD_URL = arg('prod-url', 'https://webinstall.dev').replace(/\/+$/, '');
|
||||
let KIND = arg('kind', 'api');
|
||||
let PKGS_ARG = arg('pkgs', '');
|
||||
let CONCURRENCY = parseInt(arg('concurrency', '8'), 10);
|
||||
let OUT = arg('out', '');
|
||||
|
||||
// OS/arch matrix for API mode
|
||||
let API_MATRIX = [
|
||||
{ os: 'macos', arch: 'amd64' },
|
||||
{ os: 'macos', arch: 'arm64' },
|
||||
{ os: 'linux', arch: 'amd64' },
|
||||
{ os: 'linux', arch: 'arm64' },
|
||||
{ os: 'linux', arch: 'armv7l' },
|
||||
{ os: 'windows', arch: 'amd64' },
|
||||
{ os: 'freebsd', arch: 'amd64' },
|
||||
];
|
||||
|
||||
// UA strings for installer mode
|
||||
let INSTALLER_MATRIX = [
|
||||
{ label: 'macos_arm64', ua: 'aarch64/unknown Darwin/24.2.0 libc' },
|
||||
{ label: 'macos_amd64', ua: 'x86_64/unknown Darwin/23.0.0 libc' },
|
||||
{ label: 'linux_amd64', ua: 'x86_64/unknown Linux/5.15.0 libc' },
|
||||
{ label: 'linux_arm64', ua: 'aarch64/unknown Linux/5.15.0 libc' },
|
||||
{ label: 'linux_musl', ua: 'x86_64/unknown Linux/5.15.0 musl' },
|
||||
{ label: 'windows_amd64', ua: 'x86_64/unknown Windows/10.0.19041 msvc' },
|
||||
];
|
||||
|
||||
function httpsGet(url, headers) {
|
||||
return new Promise(function (resolve, reject) {
|
||||
let opts = { headers: headers || {}, timeout: 15000 };
|
||||
let req = Https.get(url, opts, function (res) {
|
||||
// Follow one redirect
|
||||
if (res.statusCode >= 300 && res.statusCode < 400 && res.headers.location) {
|
||||
let redir = res.headers.location;
|
||||
if (redir.startsWith('/')) {
|
||||
let m = url.match(/^(https?:\/\/[^/]+)/);
|
||||
redir = (m ? m[1] : '') + redir;
|
||||
}
|
||||
Https.get(redir, opts, function (res2) {
|
||||
let data = '';
|
||||
res2.on('data', function (c) { data += c; });
|
||||
res2.on('end', function () { resolve({ status: res2.statusCode, body: data }); });
|
||||
}).on('error', reject);
|
||||
return;
|
||||
}
|
||||
let data = '';
|
||||
res.on('data', function (c) { data += c; });
|
||||
res.on('end', function () { resolve({ status: res.statusCode, body: data }); });
|
||||
});
|
||||
req.on('error', reject);
|
||||
req.on('timeout', function () { req.destroy(new Error('timeout')); });
|
||||
});
|
||||
}
|
||||
|
||||
async function listCachedPkgs() {
|
||||
let entries = await Fs.readdir(CACHE_DIR);
|
||||
return entries
|
||||
.filter(function (n) { return n.endsWith('.json') && !n.endsWith('.updated.txt'); })
|
||||
.map(function (n) { return n.slice(0, -5); })
|
||||
.sort();
|
||||
}
|
||||
|
||||
function safeFirst(json) {
|
||||
try {
|
||||
let arr = JSON.parse(json);
|
||||
if (!Array.isArray(arr) || arr.length === 0) {
|
||||
return null;
|
||||
}
|
||||
return arr[0];
|
||||
} catch (e) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function parseInstallerVars(script) {
|
||||
let vars = {};
|
||||
let re = /^(?:export\s+)?(WEBI_\w+|PKG_NAME)='([^']*)'/gm;
|
||||
let m;
|
||||
while ((m = re.exec(script)) !== null) {
|
||||
vars[m[1]] = m[2];
|
||||
}
|
||||
return vars;
|
||||
}
|
||||
|
||||
async function diffApi(pkg, os, arch) {
|
||||
let qs = `?os=${os}&arch=${arch}&limit=1`;
|
||||
let candUrl = `${CAND_URL}/api/releases/${pkg}@stable.json${qs}`;
|
||||
let prodUrl = `${PROD_URL}/api/releases/${pkg}@stable.json${qs}`;
|
||||
|
||||
let cand, prod;
|
||||
try {
|
||||
[cand, prod] = await Promise.all([httpsGet(candUrl), httpsGet(prodUrl)]);
|
||||
} catch (e) {
|
||||
return { pkg, os, arch, status: 'fetch_error', detail: e.message };
|
||||
}
|
||||
|
||||
if (cand.status !== 200 || prod.status !== 200) {
|
||||
return {
|
||||
pkg, os, arch, status: 'http_error',
|
||||
detail: `cand=${cand.status} prod=${prod.status}`,
|
||||
};
|
||||
}
|
||||
|
||||
let candFirst = safeFirst(cand.body);
|
||||
let prodFirst = safeFirst(prod.body);
|
||||
|
||||
let candErr = !candFirst || candFirst.channel === 'error';
|
||||
let prodErr = !prodFirst || prodFirst.channel === 'error';
|
||||
|
||||
if (candErr && prodErr) {
|
||||
return { pkg, os, arch, status: 'both_error', detail: '' };
|
||||
}
|
||||
if (candErr && !prodErr) {
|
||||
return {
|
||||
pkg, os, arch, status: 'cand_only_error',
|
||||
detail: `prod=${prodFirst.version}/${prodFirst.ext}`,
|
||||
};
|
||||
}
|
||||
if (!candErr && prodErr) {
|
||||
return {
|
||||
pkg, os, arch, status: 'prod_only_error',
|
||||
detail: `cand=${candFirst.version}/${candFirst.ext}`,
|
||||
};
|
||||
}
|
||||
|
||||
// Both succeeded — diff the key fields
|
||||
let diffs = [];
|
||||
for (let f of ['os', 'arch', 'libc', 'ext']) {
|
||||
if (candFirst[f] !== prodFirst[f]) {
|
||||
diffs.push(`${f}:cand=${candFirst[f]}|prod=${prodFirst[f]}`);
|
||||
}
|
||||
}
|
||||
|
||||
let ver = candFirst.version === prodFirst.version
|
||||
? candFirst.version
|
||||
: `cand=${candFirst.version}|prod=${prodFirst.version}`;
|
||||
|
||||
return {
|
||||
pkg, os, arch,
|
||||
status: diffs.length === 0 ? 'match' : 'diff',
|
||||
detail: diffs.length === 0 ? `v${ver} ${candFirst.ext}` : `v${ver} ${diffs.join(',')}`,
|
||||
};
|
||||
}
|
||||
|
||||
async function diffInstaller(pkg, label, ua) {
|
||||
let candUrl = `${CAND_URL}/api/installers/${pkg}@stable.sh`;
|
||||
let prodUrl = `${PROD_URL}/api/installers/${pkg}@stable.sh`;
|
||||
let headers = { 'User-Agent': ua };
|
||||
|
||||
let cand, prod;
|
||||
try {
|
||||
[cand, prod] = await Promise.all([
|
||||
httpsGet(candUrl, headers),
|
||||
httpsGet(prodUrl, headers),
|
||||
]);
|
||||
} catch (e) {
|
||||
return { pkg, label, status: 'fetch_error', detail: e.message };
|
||||
}
|
||||
|
||||
if (cand.status !== 200 || prod.status !== 200) {
|
||||
return {
|
||||
pkg, label, status: 'http_error',
|
||||
detail: `cand=${cand.status} prod=${prod.status}`,
|
||||
};
|
||||
}
|
||||
|
||||
let candVars = parseInstallerVars(cand.body);
|
||||
let prodVars = parseInstallerVars(prod.body);
|
||||
|
||||
let candHas = candVars.WEBI_PKG_URL && candVars.WEBI_EXT && candVars.WEBI_EXT !== 'err';
|
||||
let prodHas = prodVars.WEBI_PKG_URL && prodVars.WEBI_EXT && prodVars.WEBI_EXT !== 'err';
|
||||
|
||||
if (!candHas && !prodHas) {
|
||||
return { pkg, label, status: 'both_error', detail: '' };
|
||||
}
|
||||
if (!candHas && prodHas) {
|
||||
return {
|
||||
pkg, label, status: 'cand_only_error',
|
||||
detail: `prod=${prodVars.WEBI_VERSION}/${prodVars.WEBI_EXT}`,
|
||||
};
|
||||
}
|
||||
if (candHas && !prodHas) {
|
||||
return {
|
||||
pkg, label, status: 'prod_only_error',
|
||||
detail: `cand=${candVars.WEBI_VERSION}/${candVars.WEBI_EXT}`,
|
||||
};
|
||||
}
|
||||
|
||||
// Diff WEBI_OS, WEBI_ARCH, WEBI_EXT (PKG_NAME may differ for aliases)
|
||||
let diffs = [];
|
||||
for (let v of ['WEBI_OS', 'WEBI_ARCH', 'WEBI_EXT']) {
|
||||
if (candVars[v] !== prodVars[v]) {
|
||||
diffs.push(`${v}:cand=${candVars[v]}|prod=${prodVars[v]}`);
|
||||
}
|
||||
}
|
||||
|
||||
let ver = candVars.WEBI_VERSION === prodVars.WEBI_VERSION
|
||||
? candVars.WEBI_VERSION
|
||||
: `cand=${candVars.WEBI_VERSION}|prod=${prodVars.WEBI_VERSION}`;
|
||||
|
||||
return {
|
||||
pkg, label,
|
||||
status: diffs.length === 0 ? 'match' : 'diff',
|
||||
detail: diffs.length === 0 ? `v${ver} ${candVars.WEBI_EXT}` : `v${ver} ${diffs.join(',')}`,
|
||||
};
|
||||
}
|
||||
|
||||
async function pool(items, fn, concurrency) {
|
||||
let results = new Array(items.length);
|
||||
let i = 0;
|
||||
async function worker() {
|
||||
while (true) {
|
||||
let idx = i++;
|
||||
if (idx >= items.length) {
|
||||
return;
|
||||
}
|
||||
try {
|
||||
results[idx] = await fn(items[idx], idx);
|
||||
} catch (e) {
|
||||
results[idx] = { status: 'exception', detail: e.message, _item: items[idx] };
|
||||
}
|
||||
}
|
||||
}
|
||||
let workers = [];
|
||||
for (let k = 0; k < concurrency; k++) {
|
||||
workers.push(worker());
|
||||
}
|
||||
await Promise.all(workers);
|
||||
return results;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
let pkgs;
|
||||
if (PKGS_ARG) {
|
||||
pkgs = PKGS_ARG.split(',').filter(Boolean);
|
||||
} else {
|
||||
pkgs = await listCachedPkgs();
|
||||
}
|
||||
|
||||
console.error(`Comparing ${pkgs.length} packages: ${CAND_URL} (cand) vs ${PROD_URL} (prod)`);
|
||||
console.error(`Mode: ${KIND}, concurrency: ${CONCURRENCY}`);
|
||||
|
||||
let jobs = [];
|
||||
if (KIND === 'api') {
|
||||
for (let pkg of pkgs) {
|
||||
for (let combo of API_MATRIX) {
|
||||
jobs.push({ pkg, os: combo.os, arch: combo.arch });
|
||||
}
|
||||
}
|
||||
} else if (KIND === 'installer') {
|
||||
for (let pkg of pkgs) {
|
||||
for (let combo of INSTALLER_MATRIX) {
|
||||
jobs.push({ pkg, label: combo.label, ua: combo.ua });
|
||||
}
|
||||
}
|
||||
} else {
|
||||
console.error(`Unknown kind: ${KIND}`);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
let started = Date.now();
|
||||
let results = await pool(jobs, async function (job) {
|
||||
if (KIND === 'api') {
|
||||
return diffApi(job.pkg, job.os, job.arch);
|
||||
}
|
||||
return diffInstaller(job.pkg, job.label, job.ua);
|
||||
}, CONCURRENCY);
|
||||
let elapsed = ((Date.now() - started) / 1000).toFixed(1);
|
||||
|
||||
// TSV output
|
||||
let lines = [];
|
||||
if (KIND === 'api') {
|
||||
lines.push(['pkg', 'os', 'arch', 'status', 'detail'].join('\t'));
|
||||
for (let r of results) {
|
||||
lines.push([r.pkg, r.os, r.arch, r.status, r.detail || ''].join('\t'));
|
||||
}
|
||||
} else {
|
||||
lines.push(['pkg', 'target', 'status', 'detail'].join('\t'));
|
||||
for (let r of results) {
|
||||
lines.push([r.pkg, r.label, r.status, r.detail || ''].join('\t'));
|
||||
}
|
||||
}
|
||||
let body = lines.join('\n') + '\n';
|
||||
|
||||
if (OUT) {
|
||||
await Fs.writeFile(OUT, body, 'utf8');
|
||||
console.error(`Wrote ${OUT}`);
|
||||
} else {
|
||||
process.stdout.write(body);
|
||||
}
|
||||
|
||||
// Summary to stderr
|
||||
let counts = {};
|
||||
for (let r of results) {
|
||||
counts[r.status] = (counts[r.status] || 0) + 1;
|
||||
}
|
||||
console.error('');
|
||||
console.error(`=== Summary (${elapsed}s, ${results.length} jobs) ===`);
|
||||
for (let s of Object.keys(counts).sort()) {
|
||||
console.error(` ${s}: ${counts[s]}`);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch(function (err) {
|
||||
console.error(err.stack);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -1,444 +0,0 @@
|
||||
'use strict';
|
||||
|
||||
let InstallerServer = require('./serve-installer.js');
|
||||
let Builds = require('./builds.js');
|
||||
|
||||
// Real User-Agent strings sent by webi bootstrap scripts.
|
||||
//
|
||||
// Libc taxonomy:
|
||||
// none = static build, no runtime libc dep (often built with musl, but self-contained)
|
||||
// musl = requires musl C/C++ runtime at runtime (e.g. node-musl)
|
||||
// gnu = requires glibc at runtime (crashes on musl-only/Alpine)
|
||||
// libc = host UA value meaning "I have glibc" (not used in release metadata)
|
||||
//
|
||||
// Known issues:
|
||||
//
|
||||
// 1. WATERFALL libc vs gnu: The WATERFALL maps `libc` => ['none', 'libc']
|
||||
// but never tries 'gnu'. Packages with glibc-linked builds (libc='gnu' in
|
||||
// Go cache) won't match for hosts reporting 'libc'. Fix: update WATERFALL
|
||||
// to `libc: ['none', 'gnu', 'libc']` in build-classifier submodule.
|
||||
//
|
||||
// 2. Go cache .git regression: The Go cache includes .git source repo URLs
|
||||
// as releases, creating ANYOS/ANYARCH triplets. These match before
|
||||
// platform-specific binaries. Fix: exclude .git from Go cache output.
|
||||
|
||||
let UA_CASES = [
|
||||
// === macOS (no libc issue — darwin uses libc='none') ===
|
||||
{
|
||||
label: 'bat macOS arm64',
|
||||
pkg: 'bat',
|
||||
ua: 'aarch64/unknown Darwin/24.2.0 libc',
|
||||
expectOs: 'darwin',
|
||||
expectArch: 'aarch64',
|
||||
expectExt: 'tar.gz',
|
||||
},
|
||||
{
|
||||
label: 'bat macOS amd64',
|
||||
pkg: 'bat',
|
||||
ua: 'x86_64/unknown Darwin/23.0.0 libc',
|
||||
expectOs: 'darwin',
|
||||
expectArch: 'x86_64',
|
||||
expectExt: 'tar.gz',
|
||||
},
|
||||
{
|
||||
label: 'go macOS arm64',
|
||||
pkg: 'go',
|
||||
ua: 'aarch64/unknown Darwin/24.2.0 libc',
|
||||
expectOs: 'darwin',
|
||||
expectArch: 'aarch64',
|
||||
expectExt: 'tar.gz',
|
||||
},
|
||||
{
|
||||
label: 'node macOS arm64',
|
||||
pkg: 'node',
|
||||
ua: 'aarch64/unknown Darwin/24.2.0 libc',
|
||||
expectOs: 'darwin',
|
||||
expectArch: 'aarch64',
|
||||
expectExt: 'tar.xz',
|
||||
},
|
||||
{
|
||||
label: 'rg macOS arm64',
|
||||
pkg: 'rg',
|
||||
ua: 'aarch64/unknown Darwin/24.2.0 libc',
|
||||
expectOs: 'darwin',
|
||||
expectArch: 'aarch64',
|
||||
expectExt: 'tar.gz',
|
||||
},
|
||||
|
||||
// === macOS universal2 — packages where recent darwin builds are universal-only ===
|
||||
// These currently resolve to ancient versions because universal2 entries are
|
||||
// dropped by the classifier. The GOER's legacy export needs to emit these
|
||||
// with arch: "x86_64" so the classifier accepts them. The darwin WATERFALL
|
||||
// (aarch64 falls back to x86_64) handles aarch64 users.
|
||||
{
|
||||
label: 'cmake macOS arm64 (universal2)',
|
||||
pkg: 'cmake',
|
||||
ua: 'aarch64/unknown Darwin/24.2.0 libc',
|
||||
expectOs: 'darwin',
|
||||
expectExt: 'tar.gz',
|
||||
expectMinVersion: '4.0.0',
|
||||
known: true,
|
||||
},
|
||||
{
|
||||
label: 'cmake macOS amd64 (universal2)',
|
||||
pkg: 'cmake',
|
||||
ua: 'x86_64/unknown Darwin/23.0.0 libc',
|
||||
expectOs: 'darwin',
|
||||
expectExt: 'tar.gz',
|
||||
expectMinVersion: '4.0.0',
|
||||
known: true,
|
||||
},
|
||||
{
|
||||
label: 'hugo macOS arm64 (universal2)',
|
||||
pkg: 'hugo',
|
||||
ua: 'aarch64/unknown Darwin/24.2.0 libc',
|
||||
expectOs: 'darwin',
|
||||
expectExt: 'tar.gz',
|
||||
expectMinVersion: '0.140.0',
|
||||
known: true,
|
||||
},
|
||||
{
|
||||
label: 'hugo macOS amd64 (universal2)',
|
||||
pkg: 'hugo',
|
||||
ua: 'x86_64/unknown Darwin/23.0.0 libc',
|
||||
expectOs: 'darwin',
|
||||
expectExt: 'tar.gz',
|
||||
expectMinVersion: '0.140.0',
|
||||
known: true,
|
||||
},
|
||||
|
||||
// === Windows ===
|
||||
{
|
||||
label: 'bat Windows amd64',
|
||||
pkg: 'bat',
|
||||
ua: 'x86_64/unknown Windows/10.0.19041 msvc',
|
||||
expectOs: 'windows',
|
||||
expectArch: 'x86_64',
|
||||
expectExt: 'zip',
|
||||
},
|
||||
{
|
||||
label: 'go Windows amd64',
|
||||
pkg: 'go',
|
||||
ua: 'x86_64/unknown Windows/10.0.19041 msvc',
|
||||
expectOs: 'windows',
|
||||
expectArch: 'x86_64',
|
||||
expectExt: 'zip',
|
||||
},
|
||||
|
||||
// === Linux musl (Alpine/Docker) ===
|
||||
{
|
||||
label: 'bat Linux musl',
|
||||
pkg: 'bat',
|
||||
ua: 'x86_64/unknown Linux/5.15.0 musl',
|
||||
expectOs: 'linux',
|
||||
expectArch: 'x86_64',
|
||||
expectExt: 'tar.gz',
|
||||
},
|
||||
|
||||
// === Linux glibc — packages with libc='none' in cache ===
|
||||
{
|
||||
label: 'go Linux amd64',
|
||||
pkg: 'go',
|
||||
ua: 'x86_64/unknown Linux/5.15.0 libc',
|
||||
expectOs: 'linux',
|
||||
expectArch: 'x86_64',
|
||||
expectExt: 'tar.gz',
|
||||
},
|
||||
// === Linux glibc — packages with libc='gnu' in cache ===
|
||||
// These previously failed (WATERFALL libc→gnu gap). Fixed by adding
|
||||
// 'gnu' to the libc candidates for glibc hosts in _enumerateTriplets.
|
||||
{
|
||||
label: 'bat Linux amd64',
|
||||
pkg: 'bat',
|
||||
ua: 'x86_64/unknown Linux/5.15.0 libc',
|
||||
expectOs: 'linux',
|
||||
expectArch: 'x86_64',
|
||||
expectExt: 'tar.gz',
|
||||
},
|
||||
{
|
||||
label: 'rg Linux amd64',
|
||||
pkg: 'rg',
|
||||
ua: 'x86_64/unknown Linux/5.15.0 libc',
|
||||
expectOs: 'linux',
|
||||
expectArch: 'x86_64',
|
||||
expectExt: 'tar.gz',
|
||||
},
|
||||
{
|
||||
label: 'node Linux amd64',
|
||||
pkg: 'node',
|
||||
ua: 'x86_64/unknown Linux/5.15.0 libc',
|
||||
expectOs: 'linux',
|
||||
expectArch: 'x86_64',
|
||||
expectExt: 'tar.xz',
|
||||
},
|
||||
|
||||
// === Packages with .git source URLs in old releases ===
|
||||
// These previously failed (ANYOS .git matched before platform binary).
|
||||
// Fixed by putting specific OS before ANYOS in triplet enumeration.
|
||||
{
|
||||
label: 'jq macOS arm64',
|
||||
pkg: 'jq',
|
||||
ua: 'aarch64/unknown Darwin/24.2.0 libc',
|
||||
expectOs: 'darwin',
|
||||
expectArch: 'aarch64',
|
||||
expectExt: 'exe',
|
||||
},
|
||||
{
|
||||
label: 'caddy macOS arm64',
|
||||
pkg: 'caddy',
|
||||
ua: 'aarch64/unknown Darwin/24.2.0 libc',
|
||||
expectOs: 'darwin',
|
||||
expectArch: 'aarch64',
|
||||
expectExt: 'tar.gz',
|
||||
},
|
||||
{
|
||||
label: 'caddy Linux amd64',
|
||||
pkg: 'caddy',
|
||||
ua: 'x86_64/unknown Linux/5.15.0 libc',
|
||||
expectOs: 'linux',
|
||||
expectArch: 'x86_64',
|
||||
expectExt: 'tar.gz',
|
||||
},
|
||||
];
|
||||
|
||||
async function main() {
|
||||
let failures = 0;
|
||||
let passes = 0;
|
||||
let knowns = 0;
|
||||
let errors = 0;
|
||||
|
||||
console.log('Initializing build cache...');
|
||||
await Builds.init();
|
||||
console.log('');
|
||||
|
||||
console.log('=== Installer Resolution Tests ===');
|
||||
console.log('');
|
||||
|
||||
for (let tc of UA_CASES) {
|
||||
try {
|
||||
let [pkg, params] = await InstallerServer.helper({
|
||||
unameAgent: tc.ua,
|
||||
projectName: tc.pkg,
|
||||
tag: 'stable',
|
||||
formats: ['tar', 'exe', 'zip', 'xz', 'dmg'],
|
||||
libc: '',
|
||||
});
|
||||
|
||||
// Known issue — just verify it fails as expected
|
||||
if (tc.known) {
|
||||
let isError = pkg.channel === 'error' || !pkg.download || pkg.download.includes('doesntexist') || pkg.ext === 'git';
|
||||
let isStale = false;
|
||||
if (tc.expectMinVersion && pkg.version) {
|
||||
let got = pkg.version.replace(/^v/, '').split('.').map(Number);
|
||||
let want = tc.expectMinVersion.split('.').map(Number);
|
||||
for (let i = 0; i < want.length; i++) {
|
||||
if ((got[i] || 0) < want[i]) { isStale = true; break; }
|
||||
if ((got[i] || 0) > want[i]) { break; }
|
||||
}
|
||||
}
|
||||
if (isError || isStale) {
|
||||
let detail = isStale ? `stale v${pkg.version} < v${tc.expectMinVersion}` : '';
|
||||
console.log(` KNOWN ${tc.label}${detail ? ': ' + detail : ''}`);
|
||||
knowns++;
|
||||
} else {
|
||||
console.log(` PASS ${tc.label} (known issue resolved!): v${pkg.version} .${pkg.ext} ${(pkg.download || '').split('/').pop()}`);
|
||||
passes++;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (pkg.channel === 'error') {
|
||||
console.log(` FAIL ${tc.label}: resolved to error package`);
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
|
||||
let diffs = [];
|
||||
|
||||
if (tc.expectOs && pkg.os !== tc.expectOs) {
|
||||
diffs.push(`os: got=${pkg.os} want=${tc.expectOs}`);
|
||||
}
|
||||
if (tc.expectArch && pkg.arch !== tc.expectArch) {
|
||||
diffs.push(`arch: got=${pkg.arch} want=${tc.expectArch}`);
|
||||
}
|
||||
if (tc.expectExt && pkg.ext !== tc.expectExt) {
|
||||
diffs.push(`ext: got=${pkg.ext} want=${tc.expectExt}`);
|
||||
}
|
||||
|
||||
if (!pkg.version || pkg.version === '0.0.0') {
|
||||
diffs.push('version: missing or zero');
|
||||
}
|
||||
|
||||
if (!pkg.download || pkg.download.includes('doesntexist')) {
|
||||
diffs.push('download: missing or error');
|
||||
}
|
||||
|
||||
if (diffs.length > 0) {
|
||||
console.log(` FAIL ${tc.label}: ${diffs.join(', ')}`);
|
||||
failures++;
|
||||
} else {
|
||||
console.log(` PASS ${tc.label}: v${pkg.version} .${pkg.ext} ${pkg.download.split('/').pop()}`);
|
||||
passes++;
|
||||
}
|
||||
} catch (err) {
|
||||
if (tc.known) {
|
||||
console.log(` KNOWN ${tc.label} (error: ${err.message})`);
|
||||
knowns++;
|
||||
continue;
|
||||
}
|
||||
console.log(` ERROR ${tc.label}: ${err.message}`);
|
||||
errors++;
|
||||
}
|
||||
}
|
||||
|
||||
console.log('');
|
||||
console.log(`=== Results: ${passes} passed, ${failures} failed, ${knowns} known, ${errors} errors ===`);
|
||||
if (failures > 0 || errors > 0) {
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// Cache value validation: the classifier re-parses filenames and rejects
|
||||
// entries where the cache os/arch doesn't match. These checks prevent
|
||||
// regressions where someone "normalizes" cache values in a way that
|
||||
// breaks the classifier.
|
||||
console.log('');
|
||||
console.log('=== Cache Value Validation ===');
|
||||
console.log('');
|
||||
let cacheFailures = await validateCacheValues();
|
||||
if (cacheFailures > 0) {
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
// Verify that cache os/arch values match what the Node classifier expects
|
||||
// to extract from the download filename. The classifier is a submodule and
|
||||
// is NOT being modified — the cache must emit values it already recognizes.
|
||||
//
|
||||
// Known bug (LIVE_cache): the Go legacy export previously translated
|
||||
// solaris/illumos → sunos in the cache, but the filenames still say
|
||||
// solaris/illumos. The classifier detects the filename value and rejects
|
||||
// the entry when it doesn't match. Same issue with universal2 arch.
|
||||
//
|
||||
// Rule: cache os/arch must match the filename, not some "canonical" form.
|
||||
|
||||
// Cache os/arch values must match what the Node classifier extracts from the
|
||||
// download filename. The classifier already recognizes solaris, illumos, sunos,
|
||||
// armhf, armel, etc. — these are not new values. The only value the classifier
|
||||
// does NOT recognize is "universal2" — use "x86_64" instead.
|
||||
//
|
||||
// matchField: which field to check in the release entry ('name' or 'download')
|
||||
let CACHE_CHECKS = [
|
||||
// The classifier knows "solaris" as an OS. Filenames/URLs say "solaris".
|
||||
// Do NOT translate to "sunos" — that creates a mismatch and drops the entry.
|
||||
{
|
||||
label: 'terraform solaris entries have os=solaris (not sunos)',
|
||||
pkg: 'terraform',
|
||||
matchField: 'download',
|
||||
filenameMatch: /solaris/,
|
||||
field: 'os',
|
||||
expect: 'solaris',
|
||||
},
|
||||
{
|
||||
label: 'syncthing solaris entries have os=solaris (not sunos)',
|
||||
pkg: 'syncthing',
|
||||
matchField: 'download',
|
||||
filenameMatch: /solaris/,
|
||||
field: 'os',
|
||||
expect: 'solaris',
|
||||
},
|
||||
// The classifier knows "illumos" as an OS. Don't translate to sunos.
|
||||
{
|
||||
label: 'syncthing illumos entries have os=illumos (not sunos)',
|
||||
pkg: 'syncthing',
|
||||
matchField: 'download',
|
||||
filenameMatch: /illumos/,
|
||||
field: 'os',
|
||||
expect: 'illumos',
|
||||
},
|
||||
// node.js uses "sunos" in filenames — cache must say "sunos" (already correct)
|
||||
{
|
||||
label: 'node sunos entries have os=sunos',
|
||||
pkg: 'node',
|
||||
matchField: 'name',
|
||||
filenameMatch: /sunos/,
|
||||
field: 'os',
|
||||
expect: 'sunos',
|
||||
},
|
||||
// The classifier maps "universal" in filenames → x86_64. The classifier does
|
||||
// NOT recognize "universal2". Cache must say arch="x86_64" for these entries.
|
||||
// aarch64 users get them via the darwin WATERFALL (aarch64 → x86_64 fallback).
|
||||
{
|
||||
label: 'cmake universal entries have arch=x86_64 (not universal2)',
|
||||
pkg: 'cmake',
|
||||
matchField: 'download',
|
||||
filenameMatch: /universal/,
|
||||
field: 'arch',
|
||||
expect: 'x86_64',
|
||||
},
|
||||
{
|
||||
label: 'hugo universal entries have arch=x86_64 (not universal2)',
|
||||
pkg: 'hugo',
|
||||
matchField: 'download',
|
||||
filenameMatch: /universal/,
|
||||
field: 'arch',
|
||||
expect: 'x86_64',
|
||||
},
|
||||
];
|
||||
|
||||
async function validateCacheValues() {
|
||||
let Os = require('node:os');
|
||||
let Path = require('path');
|
||||
let Fs = require('fs');
|
||||
|
||||
let cachePath = Path.join(Os.homedir(), '.cache/webi/legacy');
|
||||
if (!Fs.existsSync(cachePath)) {
|
||||
console.log(' SKIP: no cache directory at ' + cachePath);
|
||||
return 0;
|
||||
}
|
||||
|
||||
let failures = 0;
|
||||
|
||||
for (let check of CACHE_CHECKS) {
|
||||
let filePath = Path.join(cachePath, `${check.pkg}.json`);
|
||||
if (!Fs.existsSync(filePath)) {
|
||||
console.log(` SKIP ${check.label}: no cache file`);
|
||||
continue;
|
||||
}
|
||||
|
||||
let data = JSON.parse(Fs.readFileSync(filePath, 'utf8'));
|
||||
let matchField = check.matchField || 'name';
|
||||
let matched = data.releases.filter(function (r) {
|
||||
return check.filenameMatch.test(r[matchField]);
|
||||
});
|
||||
|
||||
if (matched.length === 0) {
|
||||
console.log(` SKIP ${check.label}: no matching filenames`);
|
||||
continue;
|
||||
}
|
||||
|
||||
let wrong = matched.filter(function (r) {
|
||||
return r[check.field] !== check.expect;
|
||||
});
|
||||
|
||||
if (wrong.length > 0) {
|
||||
let sample = wrong[0];
|
||||
console.log(
|
||||
` FAIL ${check.label}: ${wrong.length}/${matched.length} entries have` +
|
||||
` ${check.field}="${sample[check.field]}" (want "${check.expect}")` +
|
||||
` e.g. ${sample.name}`,
|
||||
);
|
||||
failures++;
|
||||
} else {
|
||||
console.log(` PASS ${check.label}: ${matched.length} entries OK`);
|
||||
}
|
||||
}
|
||||
|
||||
console.log('');
|
||||
console.log(`=== Cache Validation: ${CACHE_CHECKS.length - failures} passed, ${failures} failed ===`);
|
||||
return failures;
|
||||
}
|
||||
|
||||
main().catch(function (err) {
|
||||
console.error(err.stack);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -1,450 +0,0 @@
|
||||
'use strict';
|
||||
|
||||
// Compare _cache vs LIVE_cache for correctness and compatibility.
|
||||
//
|
||||
// Rules (from NODER_PURPOSE.md):
|
||||
// - _cache should be more complete and more correct than LIVE_cache
|
||||
// - Must NOT introduce new tags (OS, arch, libc) that don't exist in LIVE_cache
|
||||
// - Must NOT break compatibility with existing data
|
||||
//
|
||||
// Usage: node _webi/test-live-cache-diff.js
|
||||
|
||||
var Fs = require('node:fs');
|
||||
var Os = require('node:os');
|
||||
var Path = require('node:path');
|
||||
|
||||
// CACHE_DIR is the live cache produced by webicached (flat layout).
|
||||
// LIVE_DIR is the historical snapshot taken pre-cutover (month-bucketed).
|
||||
var CACHE_DIR = Path.join(Os.homedir(), '.cache/webi/legacy');
|
||||
var LIVE_DIR = Path.join(__dirname, '..', 'LIVE_cache');
|
||||
|
||||
// resolveLayout: figures out whether dir uses the flat layout
|
||||
// (~/.cache/webi/legacy/<pkg>.json) or the legacy month-bucketed layout
|
||||
// (LIVE_cache/<YYYY-MM>/<pkg>.json) and returns the directory to read from.
|
||||
function resolveLayout(dir) {
|
||||
if (!Fs.existsSync(dir)) {
|
||||
return null;
|
||||
}
|
||||
var entries = Fs.readdirSync(dir);
|
||||
var months = entries
|
||||
.filter(function (d) { return /^\d{4}-\d{2}$/.test(d); })
|
||||
.sort()
|
||||
.reverse();
|
||||
if (months[0]) {
|
||||
return Path.join(dir, months[0]);
|
||||
}
|
||||
// Flat layout (cache files directly under dir).
|
||||
return dir;
|
||||
}
|
||||
|
||||
function loadReleases(layoutPath, pkg) {
|
||||
var file = Path.join(layoutPath, pkg + '.json');
|
||||
if (!Fs.existsSync(file)) {
|
||||
return null;
|
||||
}
|
||||
try {
|
||||
return JSON.parse(Fs.readFileSync(file, 'utf8'));
|
||||
} catch (e) {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function uniqueValues(releases, field) {
|
||||
var seen = {};
|
||||
for (var i = 0; i < releases.length; i++) {
|
||||
var val = releases[i][field];
|
||||
if (val !== null && val !== undefined && val !== '') {
|
||||
seen[val] = true;
|
||||
}
|
||||
}
|
||||
return Object.keys(seen).sort();
|
||||
}
|
||||
|
||||
async function main() {
|
||||
var passes = 0;
|
||||
var failures = 0;
|
||||
var warns = 0;
|
||||
|
||||
var cachePath = resolveLayout(CACHE_DIR);
|
||||
var livePath = resolveLayout(LIVE_DIR);
|
||||
|
||||
if (!cachePath) {
|
||||
console.error('No _cache directory found');
|
||||
process.exit(1);
|
||||
}
|
||||
if (!livePath) {
|
||||
console.error('No LIVE_cache directory found');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log('Using cache: ' + cachePath + ' vs ' + livePath);
|
||||
console.log('');
|
||||
|
||||
// Get all packages that exist in both caches
|
||||
var cacheFiles = Fs.readdirSync(cachePath)
|
||||
.filter(function (f) { return f.endsWith('.json'); })
|
||||
.map(function (f) { return f.replace('.json', ''); });
|
||||
var liveFiles = Fs.readdirSync(livePath)
|
||||
.filter(function (f) { return f.endsWith('.json'); })
|
||||
.map(function (f) { return f.replace('.json', ''); });
|
||||
|
||||
var cacheSet = {};
|
||||
var liveSet = {};
|
||||
cacheFiles.forEach(function (f) { cacheSet[f] = true; });
|
||||
liveFiles.forEach(function (f) { liveSet[f] = true; });
|
||||
|
||||
var common = cacheFiles.filter(function (f) { return liveSet[f]; });
|
||||
|
||||
// ================================================================
|
||||
// Test 1: Global vocabulary — no new OS/arch/libc tags
|
||||
// ================================================================
|
||||
console.log('=== Test 1: No New Tags (OS/Arch/Libc) ===');
|
||||
console.log('');
|
||||
|
||||
var allLiveOs = {};
|
||||
var allLiveArch = {};
|
||||
var allLiveLibc = {};
|
||||
var allCacheOs = {};
|
||||
var allCacheArch = {};
|
||||
var allCacheLibc = {};
|
||||
|
||||
for (var ci = 0; ci < common.length; ci++) {
|
||||
var pkg = common[ci];
|
||||
var liveData = loadReleases(livePath, pkg);
|
||||
var cacheData = loadReleases(cachePath, pkg);
|
||||
if (!liveData || !cacheData) { continue; }
|
||||
|
||||
uniqueValues(liveData.releases, 'os').forEach(function (v) { allLiveOs[v] = true; });
|
||||
uniqueValues(liveData.releases, 'arch').forEach(function (v) { allLiveArch[v] = true; });
|
||||
uniqueValues(liveData.releases, 'libc').forEach(function (v) { allLiveLibc[v] = true; });
|
||||
uniqueValues(cacheData.releases, 'os').forEach(function (v) { allCacheOs[v] = true; });
|
||||
uniqueValues(cacheData.releases, 'arch').forEach(function (v) { allCacheArch[v] = true; });
|
||||
uniqueValues(cacheData.releases, 'libc').forEach(function (v) { allCacheLibc[v] = true; });
|
||||
}
|
||||
|
||||
var newOs = Object.keys(allCacheOs).filter(function (v) { return !allLiveOs[v]; }).sort();
|
||||
var newArch = Object.keys(allCacheArch).filter(function (v) { return !allLiveArch[v]; }).sort();
|
||||
var newLibc = Object.keys(allCacheLibc).filter(function (v) { return !allLiveLibc[v]; }).sort();
|
||||
|
||||
if (newOs.length > 0) {
|
||||
console.log(' FAIL new OS values in _cache not in LIVE_cache: ' + JSON.stringify(newOs));
|
||||
failures++;
|
||||
} else {
|
||||
console.log(' PASS no new OS values');
|
||||
passes++;
|
||||
}
|
||||
|
||||
if (newArch.length > 0) {
|
||||
console.log(' FAIL new arch values in _cache not in LIVE_cache: ' + JSON.stringify(newArch));
|
||||
failures++;
|
||||
} else {
|
||||
console.log(' PASS no new arch values');
|
||||
passes++;
|
||||
}
|
||||
|
||||
if (newLibc.length > 0) {
|
||||
console.log(' FAIL new libc values in _cache not in LIVE_cache: ' + JSON.stringify(newLibc));
|
||||
failures++;
|
||||
} else {
|
||||
console.log(' PASS no new libc values');
|
||||
passes++;
|
||||
}
|
||||
|
||||
// Show what LIVE has that _cache doesn't (informational)
|
||||
var missingOs = Object.keys(allLiveOs).filter(function (v) { return !allCacheOs[v]; }).sort();
|
||||
var missingArch = Object.keys(allLiveArch).filter(function (v) { return !allCacheArch[v]; }).sort();
|
||||
if (missingOs.length > 0) {
|
||||
console.log(' INFO OS in LIVE but not _cache: ' + JSON.stringify(missingOs));
|
||||
}
|
||||
if (missingArch.length > 0) {
|
||||
console.log(' INFO arch in LIVE but not _cache: ' + JSON.stringify(missingArch));
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 2: Per-package release count — _cache should have >= LIVE
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 2: Release Count (per package) ===');
|
||||
console.log('');
|
||||
|
||||
// LIVE_cache includes junk entries (.pem, .sig, .sha256, .deb, .rpm, etc.)
|
||||
// that Go correctly filters out. Only count installable entries.
|
||||
var junkExts = /\.(pem|sig|asc|sha256|sha512|sha256sum|sha512sum|deb|rpm|apk|sbom|json|txt|sum|md5|cosign-bundle|intoto-jsonl)$/i;
|
||||
|
||||
function countInstallable(releases) {
|
||||
var n = 0;
|
||||
for (var i = 0; i < releases.length; i++) {
|
||||
if (!junkExts.test(releases[i].name || '')) {
|
||||
n++;
|
||||
}
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
var countIssues = [];
|
||||
for (var pi = 0; pi < common.length; pi++) {
|
||||
var ppkg = common[pi];
|
||||
var pLive = loadReleases(livePath, ppkg);
|
||||
var pCache = loadReleases(cachePath, ppkg);
|
||||
if (!pLive || !pCache) { continue; }
|
||||
|
||||
var liveCount = countInstallable(pLive.releases);
|
||||
var cacheCount = pCache.releases.length;
|
||||
|
||||
// _cache should have at least as many installable releases as LIVE
|
||||
var ratio = liveCount > 0 ? cacheCount / liveCount : 1;
|
||||
if (ratio < 0.5 && liveCount > 10) {
|
||||
countIssues.push({ pkg: ppkg, live: liveCount, cache: cacheCount, ratio: ratio });
|
||||
}
|
||||
}
|
||||
|
||||
if (countIssues.length === 0) {
|
||||
console.log(' PASS all packages have adequate release counts');
|
||||
passes++;
|
||||
} else {
|
||||
for (var ii = 0; ii < countIssues.length; ii++) {
|
||||
var issue = countIssues[ii];
|
||||
console.log(' FAIL ' + issue.pkg + ': _cache=' + issue.cache + ' LIVE=' + issue.live + ' (ratio=' + issue.ratio.toFixed(2) + ')');
|
||||
failures++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 3: Per-package OS coverage — _cache should have same core OSes
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 3: OS Coverage (per package) ===');
|
||||
console.log('');
|
||||
|
||||
var coreOses = ['darwin', 'linux', 'windows'];
|
||||
var osIssues = [];
|
||||
for (var oi = 0; oi < common.length; oi++) {
|
||||
var opkg = common[oi];
|
||||
var oLive = loadReleases(livePath, opkg);
|
||||
var oCache = loadReleases(cachePath, opkg);
|
||||
if (!oLive || !oCache) { continue; }
|
||||
|
||||
var liveOses = uniqueValues(oLive.releases, 'os');
|
||||
var cacheOses = uniqueValues(oCache.releases, 'os');
|
||||
|
||||
// For each core OS in LIVE, it should also be in _cache
|
||||
for (var coi = 0; coi < coreOses.length; coi++) {
|
||||
var os = coreOses[coi];
|
||||
if (liveOses.indexOf(os) >= 0 && cacheOses.indexOf(os) < 0) {
|
||||
osIssues.push({ pkg: opkg, os: os });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (osIssues.length === 0) {
|
||||
console.log(' PASS all packages have matching core OS coverage');
|
||||
passes++;
|
||||
} else {
|
||||
for (var oii = 0; oii < osIssues.length; oii++) {
|
||||
var oisue = osIssues[oii];
|
||||
console.log(' FAIL ' + oisue.pkg + ': missing os=' + oisue.os + ' (present in LIVE)');
|
||||
failures++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 4: Per-package new tags — flag packages introducing new values
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 4: Per-Package New Tags ===');
|
||||
console.log('');
|
||||
|
||||
var tagIssues = [];
|
||||
var tagSkipped = 0;
|
||||
for (var ti = 0; ti < common.length; ti++) {
|
||||
var tpkg = common[ti];
|
||||
var tLive = loadReleases(livePath, tpkg);
|
||||
var tCache = loadReleases(cachePath, tpkg);
|
||||
if (!tLive || !tCache) { continue; }
|
||||
|
||||
var tLiveOs = {};
|
||||
var tLiveArch = {};
|
||||
uniqueValues(tLive.releases, 'os').forEach(function (v) { tLiveOs[v] = true; });
|
||||
uniqueValues(tLive.releases, 'arch').forEach(function (v) { tLiveArch[v] = true; });
|
||||
|
||||
// Skip packages where LIVE has no classified entries (all empty os/arch).
|
||||
// These are github_releases packages where classification happens at query
|
||||
// time. Our _cache filling in values is an improvement, not a regression.
|
||||
var liveOsKeys = Object.keys(tLiveOs);
|
||||
var liveArchKeys = Object.keys(tLiveArch);
|
||||
if (liveOsKeys.length === 0 && liveArchKeys.length === 0) {
|
||||
tagSkipped++;
|
||||
continue;
|
||||
}
|
||||
|
||||
var tCacheOs = uniqueValues(tCache.releases, 'os');
|
||||
var tCacheArch = uniqueValues(tCache.releases, 'arch');
|
||||
|
||||
var tNewOs = tCacheOs.filter(function (v) { return !tLiveOs[v]; });
|
||||
var tNewArch = tCacheArch.filter(function (v) { return !tLiveArch[v]; });
|
||||
|
||||
if (tNewOs.length > 0 || tNewArch.length > 0) {
|
||||
tagIssues.push({
|
||||
pkg: tpkg,
|
||||
newOs: tNewOs,
|
||||
newArch: tNewArch,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (tagSkipped > 0) {
|
||||
console.log(' INFO skipped ' + tagSkipped + ' packages with no LIVE classification (unclassified github_releases)');
|
||||
}
|
||||
if (tagIssues.length === 0) {
|
||||
console.log(' PASS no pre-classified packages introduce new tags');
|
||||
passes++;
|
||||
} else {
|
||||
for (var tii = 0; tii < tagIssues.length; tii++) {
|
||||
var tissue = tagIssues[tii];
|
||||
var parts = [];
|
||||
if (tissue.newOs.length > 0) {
|
||||
parts.push('os: ' + JSON.stringify(tissue.newOs));
|
||||
}
|
||||
if (tissue.newArch.length > 0) {
|
||||
parts.push('arch: ' + JSON.stringify(tissue.newArch));
|
||||
}
|
||||
console.log(' WARN ' + tissue.pkg + ': new tags: ' + parts.join(', '));
|
||||
warns++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 5: Latest stable version — _cache should be >= LIVE
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 5: Latest Stable Version ===');
|
||||
console.log('');
|
||||
|
||||
var stableCheckPkgs = ['bat', 'go', 'node', 'rg', 'caddy', 'jq', 'hugo', 'terraform'];
|
||||
for (var si = 0; si < stableCheckPkgs.length; si++) {
|
||||
var spkg = stableCheckPkgs[si];
|
||||
var sLive = loadReleases(livePath, spkg);
|
||||
var sCache = loadReleases(cachePath, spkg);
|
||||
if (!sLive || !sCache) {
|
||||
console.log(' SKIP ' + spkg + ': missing data');
|
||||
continue;
|
||||
}
|
||||
|
||||
// Find first stable release in each
|
||||
var liveStable = sLive.releases.find(function (r) { return r.channel === 'stable'; });
|
||||
var cacheStable = sCache.releases.find(function (r) { return r.channel === 'stable'; });
|
||||
|
||||
if (!liveStable || !cacheStable) {
|
||||
console.log(' SKIP ' + spkg + ': no stable release found');
|
||||
continue;
|
||||
}
|
||||
|
||||
var lv = (liveStable.version || '').replace(/^v/, '');
|
||||
var cv = (cacheStable.version || '').replace(/^v/, '');
|
||||
|
||||
if (lv === cv) {
|
||||
console.log(' PASS ' + spkg + ': ' + cv);
|
||||
passes++;
|
||||
} else {
|
||||
// Just warn — versions may differ due to cache age
|
||||
console.log(' WARN ' + spkg + ': LIVE=' + lv + ' _cache=' + cv);
|
||||
warns++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 6: Download URLs — all entries should have valid URLs
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 6: Download URL Validity ===');
|
||||
console.log('');
|
||||
|
||||
var urlIssues = [];
|
||||
for (var ui = 0; ui < common.length; ui++) {
|
||||
var upkg = common[ui];
|
||||
var uCache = loadReleases(cachePath, upkg);
|
||||
if (!uCache) { continue; }
|
||||
|
||||
var emptyUrls = 0;
|
||||
var badUrls = 0;
|
||||
for (var uri = 0; uri < uCache.releases.length; uri++) {
|
||||
var rel = uCache.releases[uri];
|
||||
var url = rel.download || '';
|
||||
if (url === '') {
|
||||
emptyUrls++;
|
||||
} else if (!/^https?:\/\//.test(url)) {
|
||||
badUrls++;
|
||||
}
|
||||
}
|
||||
if (emptyUrls > 0 || badUrls > 0) {
|
||||
urlIssues.push({ pkg: upkg, empty: emptyUrls, bad: badUrls });
|
||||
}
|
||||
}
|
||||
|
||||
if (urlIssues.length === 0) {
|
||||
console.log(' PASS all packages have valid download URLs');
|
||||
passes++;
|
||||
} else {
|
||||
for (var uii = 0; uii < urlIssues.length; uii++) {
|
||||
var uissue = urlIssues[uii];
|
||||
var uparts = [];
|
||||
if (uissue.empty > 0) { uparts.push(uissue.empty + ' empty'); }
|
||||
if (uissue.bad > 0) { uparts.push(uissue.bad + ' malformed'); }
|
||||
console.log(' FAIL ' + uissue.pkg + ': ' + uparts.join(', '));
|
||||
failures++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 7: Required fields — all entries should have version + name
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 7: Required Fields ===');
|
||||
console.log('');
|
||||
|
||||
var fieldIssues = [];
|
||||
for (var fi = 0; fi < common.length; fi++) {
|
||||
var fpkg = common[fi];
|
||||
var fCache = loadReleases(cachePath, fpkg);
|
||||
if (!fCache) { continue; }
|
||||
|
||||
var noVersion = 0;
|
||||
var noName = 0;
|
||||
for (var fri = 0; fri < fCache.releases.length; fri++) {
|
||||
var frel = fCache.releases[fri];
|
||||
if (!frel.version) { noVersion++; }
|
||||
if (!frel.name && !frel.download) { noName++; }
|
||||
}
|
||||
if (noVersion > 0 || noName > 0) {
|
||||
fieldIssues.push({ pkg: fpkg, noVersion: noVersion, noName: noName });
|
||||
}
|
||||
}
|
||||
|
||||
if (fieldIssues.length === 0) {
|
||||
console.log(' PASS all packages have version and name/download');
|
||||
passes++;
|
||||
} else {
|
||||
for (var fii = 0; fii < fieldIssues.length; fii++) {
|
||||
var fissue = fieldIssues[fii];
|
||||
var fparts = [];
|
||||
if (fissue.noVersion > 0) { fparts.push(fissue.noVersion + ' missing version'); }
|
||||
if (fissue.noName > 0) { fparts.push(fissue.noName + ' missing name+download'); }
|
||||
console.log(' FAIL ' + fissue.pkg + ': ' + fparts.join(', '));
|
||||
failures++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Summary
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Results: ' + passes + ' passed, ' + failures + ' failed, ' + warns + ' warnings ===');
|
||||
if (failures > 0) {
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch(function (err) {
|
||||
console.error(err.stack);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -1,577 +0,0 @@
|
||||
'use strict';
|
||||
|
||||
// Comprehensive live-vs-local comparison test.
|
||||
// Fetches from a remote API (default: webinstall.dev) and compares against
|
||||
// local cache-only output to catch regressions.
|
||||
//
|
||||
// Usage:
|
||||
// node _webi/test-live-compare.js # compare against prod
|
||||
// node _webi/test-live-compare.js --refresh # refresh golden data
|
||||
// node _webi/test-live-compare.js --base-url=https://beta.webi.sh
|
||||
// node _webi/test-live-compare.js --all # all cached pkgs
|
||||
// node _webi/test-live-compare.js --all --tsv # TSV output
|
||||
// node _webi/test-live-compare.js --base-url=https://webi.sh \
|
||||
// --cand-url=https://beta.webi.sh # remote-vs-remote
|
||||
// # (Test 4 only;
|
||||
// # no local cache
|
||||
// # needed)
|
||||
|
||||
let Fs = require('node:fs/promises');
|
||||
let Os = require('node:os');
|
||||
let Path = require('node:path');
|
||||
let Https = require('node:https');
|
||||
|
||||
let Releases = require('./transform-releases.js');
|
||||
let InstallerServer = require('./serve-installer.js');
|
||||
let Builds = require('./builds.js');
|
||||
|
||||
let TESTDATA_DIR = Path.join(__dirname, 'testdata');
|
||||
let CACHE_DIR = Path.join(Os.homedir(), '.cache/webi/legacy');
|
||||
|
||||
let REFRESH = process.argv.includes('--refresh');
|
||||
let ALL_PKGS = process.argv.includes('--all');
|
||||
let TSV = process.argv.includes('--tsv');
|
||||
|
||||
let BASE_URL = 'https://webinstall.dev';
|
||||
let CAND_URL = '';
|
||||
for (let arg of process.argv) {
|
||||
if (arg.startsWith('--base-url=')) {
|
||||
BASE_URL = arg.slice('--base-url='.length).replace(/\/+$/, '');
|
||||
} else if (arg.startsWith('--cand-url=')) {
|
||||
CAND_URL = arg.slice('--cand-url='.length).replace(/\/+$/, '');
|
||||
}
|
||||
}
|
||||
|
||||
// Packages to test — mix of Go-built, Rust-built, and C/C++ projects
|
||||
let TEST_PKGS = ['bat', 'go', 'node', 'rg', 'jq', 'caddy'];
|
||||
|
||||
async function listCachedPkgs() {
|
||||
let entries;
|
||||
try {
|
||||
entries = await Fs.readdir(CACHE_DIR);
|
||||
} catch (e) {
|
||||
console.error(`No cache directory: ${CACHE_DIR}`);
|
||||
return [];
|
||||
}
|
||||
let pkgs = entries
|
||||
.filter(function (n) { return n.endsWith('.json'); })
|
||||
.map(function (n) { return n.slice(0, -5); })
|
||||
.sort();
|
||||
return pkgs;
|
||||
}
|
||||
|
||||
// OS/arch combos for filtered release API tests
|
||||
let RELEASE_API_CASES = [
|
||||
{ os: 'macos', arch: 'amd64' },
|
||||
{ os: 'macos', arch: 'arm64' },
|
||||
{ os: 'linux', arch: 'amd64' },
|
||||
{ os: 'windows', arch: 'amd64' },
|
||||
];
|
||||
|
||||
// Older / channel-specific version specs that map to webi <pkg>@<spec>
|
||||
// invocations. Each case exercises a code path that the unfiltered
|
||||
// "@stable" sweep above would never hit:
|
||||
// - LTS filter (lts=true)
|
||||
// - Channel filter (channel=beta)
|
||||
// - Major-series prefix (ver=20 → /^20\b/)
|
||||
// - Older minor (ver=0.18 → /^0.18\b/)
|
||||
// - Older major (ver=1.21 → /^1.21\b/) for projects with deep history
|
||||
let VERSION_SPEC_CASES = [
|
||||
{ pkg: 'node', spec: 'lts', ver: '', channel: '', lts: true, os: 'linux', arch: 'amd64' },
|
||||
{ pkg: 'node', spec: '20', ver: '20', channel: '', lts: false, os: 'linux', arch: 'amd64' },
|
||||
{ pkg: 'node', spec: 'beta', ver: '', channel: 'beta', lts: false, os: 'linux', arch: 'amd64' },
|
||||
{ pkg: 'go', spec: '1.22', ver: '1.22', channel: '', lts: false, os: 'linux', arch: 'amd64' },
|
||||
{ pkg: 'go', spec: '1.21', ver: '1.21', channel: '', lts: false, os: 'macos', arch: 'arm64' },
|
||||
{ pkg: 'bat', spec: '0.20', ver: '0.20', channel: '', lts: false, os: 'linux', arch: 'amd64' },
|
||||
{ pkg: 'bat', spec: '0.18', ver: '0.18', channel: '', lts: false, os: 'linux', arch: 'amd64' },
|
||||
{ pkg: 'rg', spec: '13', ver: '13', channel: '', lts: false, os: 'linux', arch: 'amd64' },
|
||||
{ pkg: 'caddy', spec: '2.7', ver: '2.7', channel: '', lts: false, os: 'linux', arch: 'amd64' },
|
||||
];
|
||||
|
||||
// UA strings for installer resolution tests
|
||||
let INSTALLER_CASES = [
|
||||
{ label: 'macOS arm64', ua: 'aarch64/unknown Darwin/24.2.0 libc' },
|
||||
{ label: 'macOS amd64', ua: 'x86_64/unknown Darwin/23.0.0 libc' },
|
||||
{ label: 'Linux musl', ua: 'x86_64/unknown Linux/5.15.0 musl' },
|
||||
{ label: 'Windows amd64', ua: 'x86_64/unknown Windows/10.0.19041 msvc' },
|
||||
];
|
||||
|
||||
// Known differences between Go cache and production (not regressions)
|
||||
// Extensions that the Go cache correctly excludes (non-installable formats)
|
||||
// OR that the Go cache includes but shouldn't (man pages, etc.)
|
||||
let KNOWN_EXT_DIFFS = new Set([
|
||||
'deb', 'rpm', 'sha256', 'sig', 'pem', 'sbom', 'txt',
|
||||
'1', '2', '3', '4', '5', '6', '7', '8', // man page extensions
|
||||
]);
|
||||
|
||||
function httpsGet(url) {
|
||||
return new Promise(function (resolve, reject) {
|
||||
Https.get(url, function (res) {
|
||||
let data = '';
|
||||
res.on('data', function (chunk) {
|
||||
data += chunk;
|
||||
});
|
||||
res.on('end', function () {
|
||||
if (res.statusCode !== 200) {
|
||||
reject(new Error(`HTTP ${res.statusCode}: ${url}`));
|
||||
return;
|
||||
}
|
||||
resolve(data);
|
||||
});
|
||||
}).on('error', reject);
|
||||
});
|
||||
}
|
||||
|
||||
async function fetchLiveReleases(pkg, os, arch, limit) {
|
||||
let url = `${BASE_URL}/api/releases/${pkg}@stable.json?limit=${limit || 100}`;
|
||||
if (os) {
|
||||
url += `&os=${os}`;
|
||||
}
|
||||
if (arch) {
|
||||
url += `&arch=${arch}`;
|
||||
}
|
||||
let json = await httpsGet(url);
|
||||
return JSON.parse(json);
|
||||
}
|
||||
|
||||
async function fetchAtSpec(baseUrl, pkg, spec, os, arch) {
|
||||
let url = `${baseUrl}/api/releases/${pkg}@${spec}.json?limit=1`;
|
||||
if (os) {
|
||||
url += `&os=${os}`;
|
||||
}
|
||||
if (arch) {
|
||||
url += `&arch=${arch}`;
|
||||
}
|
||||
let json = await httpsGet(url);
|
||||
return JSON.parse(json);
|
||||
}
|
||||
|
||||
async function fetchLiveInstaller(pkg, ua) {
|
||||
return new Promise(function (resolve, reject) {
|
||||
let url = `${BASE_URL}/${pkg}@stable.sh`;
|
||||
let opts = {
|
||||
headers: { 'User-Agent': ua },
|
||||
};
|
||||
Https.get(url, opts, function (res) {
|
||||
// Follow redirects
|
||||
if (res.statusCode >= 300 && res.statusCode < 400 && res.headers.location) {
|
||||
let redir = res.headers.location;
|
||||
if (redir.startsWith('/')) {
|
||||
redir = BASE_URL + redir;
|
||||
}
|
||||
Https.get(redir, opts, function (res2) {
|
||||
let data = '';
|
||||
res2.on('data', function (chunk) {
|
||||
data += chunk;
|
||||
});
|
||||
res2.on('end', function () {
|
||||
resolve(data);
|
||||
});
|
||||
}).on('error', reject);
|
||||
return;
|
||||
}
|
||||
let data = '';
|
||||
res.on('data', function (chunk) {
|
||||
data += chunk;
|
||||
});
|
||||
res.on('end', function () {
|
||||
resolve(data);
|
||||
});
|
||||
}).on('error', reject);
|
||||
});
|
||||
}
|
||||
|
||||
function parseInstallerVars(scriptText) {
|
||||
let vars = {};
|
||||
// Match both `export WEBI_FOO='...'` and `WEBI_FOO='...'`
|
||||
let names = ['WEBI_PKG_URL', 'WEBI_VERSION', 'WEBI_EXT', 'WEBI_OS', 'WEBI_ARCH', 'PKG_NAME'];
|
||||
for (let name of names) {
|
||||
let re = new RegExp("^(?:export\\s+)?" + name + "='([^']*)'", 'm');
|
||||
let m = scriptText.match(re);
|
||||
if (m) {
|
||||
vars[name] = m[1];
|
||||
}
|
||||
}
|
||||
return vars;
|
||||
}
|
||||
|
||||
async function saveGolden(name, data) {
|
||||
await Fs.mkdir(TESTDATA_DIR, { recursive: true });
|
||||
let file = Path.join(TESTDATA_DIR, name);
|
||||
await Fs.writeFile(file, JSON.stringify(data), 'utf8');
|
||||
}
|
||||
|
||||
async function loadGolden(name) {
|
||||
let file = Path.join(TESTDATA_DIR, name);
|
||||
try {
|
||||
let json = await Fs.readFile(file, 'utf8');
|
||||
return JSON.parse(json);
|
||||
} catch (e) {
|
||||
if (e.code === 'ENOENT') {
|
||||
return null;
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
|
||||
async function main() {
|
||||
let passes = 0;
|
||||
let failures = 0;
|
||||
let skips = 0;
|
||||
let knowns = 0;
|
||||
|
||||
console.log('Initializing build cache...');
|
||||
await Builds.init();
|
||||
console.log('');
|
||||
|
||||
// ================================================================
|
||||
// Test 1: Release API — unfiltered
|
||||
// ================================================================
|
||||
console.log('=== Test 1: Unfiltered /api/releases/{pkg}.json ===');
|
||||
console.log('');
|
||||
|
||||
for (let pkg of TEST_PKGS) {
|
||||
let goldenName = `live_${pkg}.json`;
|
||||
let liveReleases;
|
||||
|
||||
if (REFRESH) {
|
||||
try {
|
||||
liveReleases = await fetchLiveReleases(pkg);
|
||||
await saveGolden(goldenName, liveReleases);
|
||||
console.log(` refreshed ${goldenName}`);
|
||||
} catch (e) {
|
||||
console.log(` SKIP ${pkg}: fetch error: ${e.message}`);
|
||||
skips++;
|
||||
continue;
|
||||
}
|
||||
} else {
|
||||
liveReleases = await loadGolden(goldenName);
|
||||
if (!liveReleases) {
|
||||
console.log(` SKIP ${pkg}: no golden data (run with --refresh)`);
|
||||
skips++;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
let localResult = await Releases.getReleases({
|
||||
pkg: pkg,
|
||||
ver: '',
|
||||
os: '',
|
||||
arch: '',
|
||||
libc: '',
|
||||
lts: false,
|
||||
channel: '',
|
||||
formats: [],
|
||||
limit: 100,
|
||||
});
|
||||
let localReleases = localResult.releases;
|
||||
|
||||
// Compare OS vocabulary — check that local has the core OSes that live has
|
||||
let liveOses = [...new Set(liveReleases.map(function (r) { return r.os; }))].sort();
|
||||
let localOses = [...new Set(localReleases.map(function (r) { return r.os; }))].sort();
|
||||
let coreOses = ['linux', 'macos', 'windows'];
|
||||
let liveCore = liveOses.filter(function (o) { return coreOses.includes(o); }).sort();
|
||||
let localCore = localOses.filter(function (o) { return coreOses.includes(o); }).sort();
|
||||
// Local should have at least the core OSes that live has
|
||||
let missingCore = liveCore.filter(function (o) { return !localCore.includes(o); });
|
||||
if (missingCore.length > 0) {
|
||||
console.log(` FAIL ${pkg} OS: missing core OSes: ${JSON.stringify(missingCore)}`);
|
||||
failures++;
|
||||
} else {
|
||||
console.log(` PASS ${pkg} OS: ${JSON.stringify(localCore)}`);
|
||||
passes++;
|
||||
}
|
||||
|
||||
// Compare ext vocabulary (excluding known non-installable formats)
|
||||
let liveExts = [...new Set(liveReleases.map(function (r) { return r.ext; }))].sort();
|
||||
let localExts = [...new Set(localReleases.map(function (r) { return r.ext; }))].sort();
|
||||
let liveExtsFiltered = liveExts.filter(function (e) { return !KNOWN_EXT_DIFFS.has(e); });
|
||||
let localExtsFiltered = localExts.filter(function (e) { return !KNOWN_EXT_DIFFS.has(e); });
|
||||
let extMatch = true;
|
||||
for (let ext of localExtsFiltered) {
|
||||
if (!liveExtsFiltered.includes(ext)) {
|
||||
// Local has a real ext that live doesn't — may be due to limit or sampling
|
||||
// Only fail if it's clearly wrong (not a standard installable format)
|
||||
let installable = ['tar.gz', 'tar.xz', 'tar.bz2', 'zip', 'exe', 'dmg', 'pkg', 'msi', '7z', 'xz'];
|
||||
if (!installable.includes(ext)) {
|
||||
console.log(` FAIL ${pkg} ext: local has unexpected '${ext}'`);
|
||||
failures++;
|
||||
extMatch = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (extMatch) {
|
||||
console.log(` PASS ${pkg} ext: ${JSON.stringify(localExtsFiltered)}`);
|
||||
passes++;
|
||||
}
|
||||
|
||||
// Version format — no 'v' prefix
|
||||
let hasVPrefix = localReleases.some(function (r) {
|
||||
return r.version && r.version.startsWith('v');
|
||||
});
|
||||
if (hasVPrefix) {
|
||||
console.log(` FAIL ${pkg}: versions have 'v' prefix`);
|
||||
failures++;
|
||||
} else {
|
||||
console.log(` PASS ${pkg}: no 'v' prefix`);
|
||||
passes++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 2: Release API — filtered by OS/arch
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 2: Filtered /api/releases/{pkg}@stable.json?os=...&arch=... ===');
|
||||
console.log('');
|
||||
|
||||
for (let pkg of TEST_PKGS) {
|
||||
for (let tc of RELEASE_API_CASES) {
|
||||
let goldenName = `live_${pkg}_os_${tc.os}_arch_${tc.arch}.json`;
|
||||
let liveReleases;
|
||||
|
||||
if (REFRESH) {
|
||||
try {
|
||||
liveReleases = await fetchLiveReleases(pkg, tc.os, tc.arch, 1);
|
||||
await saveGolden(goldenName, liveReleases);
|
||||
} catch (e) {
|
||||
skips++;
|
||||
continue;
|
||||
}
|
||||
} else {
|
||||
liveReleases = await loadGolden(goldenName);
|
||||
if (!liveReleases) {
|
||||
skips++;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
let liveFirst = liveReleases[0];
|
||||
if (!liveFirst || liveFirst.channel === 'error') {
|
||||
skips++;
|
||||
continue;
|
||||
}
|
||||
|
||||
let localResult = await Releases.getReleases({
|
||||
pkg: pkg,
|
||||
ver: '',
|
||||
os: tc.os,
|
||||
arch: tc.arch,
|
||||
libc: '',
|
||||
lts: false,
|
||||
channel: 'stable',
|
||||
formats: [],
|
||||
limit: 1,
|
||||
});
|
||||
let localFirst = localResult.releases[0];
|
||||
|
||||
if (!localFirst || localFirst.channel === 'error') {
|
||||
console.log(` FAIL ${pkg} ${tc.os}/${tc.arch}: local returned error/empty`);
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
|
||||
let diffs = [];
|
||||
// Compare os, arch, ext (skip version/download since cache age may differ)
|
||||
if (liveFirst.os !== localFirst.os) {
|
||||
diffs.push(`os: live=${liveFirst.os} local=${localFirst.os}`);
|
||||
}
|
||||
if (liveFirst.arch !== localFirst.arch) {
|
||||
diffs.push(`arch: live=${liveFirst.arch} local=${localFirst.arch}`);
|
||||
}
|
||||
if (liveFirst.ext !== localFirst.ext) {
|
||||
if (KNOWN_EXT_DIFFS.has(liveFirst.ext)) {
|
||||
// Live returns a non-installable format (deb, pem, etc.) — known
|
||||
console.log(` KNOWN ${pkg} ${tc.os}/${tc.arch}: live ext '${liveFirst.ext}' excluded by Go cache, local='${localFirst.ext}'`);
|
||||
knowns++;
|
||||
continue;
|
||||
}
|
||||
diffs.push(`ext: live=${liveFirst.ext} local=${localFirst.ext}`);
|
||||
}
|
||||
|
||||
if (diffs.length > 0) {
|
||||
console.log(` FAIL ${pkg} ${tc.os}/${tc.arch}: ${diffs.join(', ')}`);
|
||||
failures++;
|
||||
} else {
|
||||
console.log(` PASS ${pkg} ${tc.os}/${tc.arch}: v${localFirst.version} .${localFirst.ext}`);
|
||||
passes++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 3: Installer resolution — compare rendered script vars
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log('=== Test 3: Installer script variables (local serveInstaller vs live) ===');
|
||||
console.log('');
|
||||
|
||||
for (let pkg of ['bat', 'go', 'rg']) {
|
||||
for (let tc of INSTALLER_CASES) {
|
||||
// Get local result
|
||||
let localVars;
|
||||
try {
|
||||
let script = await InstallerServer.serveInstaller(
|
||||
'https://webi.sh',
|
||||
tc.ua,
|
||||
pkg,
|
||||
'stable',
|
||||
'sh',
|
||||
['tar', 'exe', 'zip', 'xz', 'dmg'],
|
||||
'',
|
||||
);
|
||||
localVars = parseInstallerVars(script);
|
||||
} catch (e) {
|
||||
console.log(` ERROR ${pkg} ${tc.label}: ${e.message}`);
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!localVars.WEBI_PKG_URL || localVars.WEBI_PKG_URL === '') {
|
||||
// Check if this is a known issue
|
||||
if (localVars.WEBI_EXT === 'err') {
|
||||
console.log(` KNOWN ${pkg} ${tc.label}: no match (WATERFALL gap)`);
|
||||
knowns++;
|
||||
continue;
|
||||
}
|
||||
console.log(` FAIL ${pkg} ${tc.label}: empty WEBI_PKG_URL`);
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Verify the URL looks like a real download
|
||||
let url = localVars.WEBI_PKG_URL;
|
||||
let hasRealDomain = url.includes('github.com') ||
|
||||
url.includes('dl.google.com') ||
|
||||
url.includes('nodejs.org') ||
|
||||
url.includes('jqlang');
|
||||
if (!hasRealDomain) {
|
||||
console.log(` FAIL ${pkg} ${tc.label}: suspicious URL: ${url}`);
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Verify version is set and has no 'v' prefix
|
||||
if (!localVars.WEBI_VERSION || localVars.WEBI_VERSION === '0.0.0') {
|
||||
console.log(` FAIL ${pkg} ${tc.label}: bad version: ${localVars.WEBI_VERSION}`);
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Verify ext is a real installable format
|
||||
let ext = localVars.WEBI_EXT;
|
||||
let goodExts = ['tar.gz', 'tar.xz', 'zip', 'exe', 'dmg', 'pkg', 'msi', '7z'];
|
||||
if (!goodExts.includes(ext)) {
|
||||
console.log(` FAIL ${pkg} ${tc.label}: bad ext: ${ext}`);
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
|
||||
console.log(` PASS ${pkg} ${tc.label}: v${localVars.WEBI_VERSION} .${ext} ${url.split('/').pop()}`);
|
||||
passes++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Test 4: @version path-form parity (webi <pkg>@<spec>)
|
||||
// Exercises lts/channel/version filters that the unfiltered sweep
|
||||
// doesn't touch. Two modes:
|
||||
// - --cand-url=<url> set: remote (BASE_URL) vs remote (CAND_URL).
|
||||
// No local cache needed.
|
||||
// - --cand-url unset: remote (BASE_URL) vs in-process Releases.
|
||||
// ================================================================
|
||||
console.log('');
|
||||
if (CAND_URL) {
|
||||
console.log(`=== Test 4: @version filter parity (${BASE_URL} vs ${CAND_URL}) ===`);
|
||||
} else {
|
||||
console.log('=== Test 4: @version filter parity (remote vs local resolver) ===');
|
||||
}
|
||||
console.log('');
|
||||
|
||||
for (let tc of VERSION_SPEC_CASES) {
|
||||
let label = `${tc.pkg}@${tc.spec} ${tc.os}/${tc.arch}`;
|
||||
|
||||
let baseFirst;
|
||||
try {
|
||||
let baseRel = await fetchAtSpec(BASE_URL, tc.pkg, tc.spec, tc.os, tc.arch);
|
||||
baseFirst = baseRel[0];
|
||||
} catch (e) {
|
||||
console.log(` SKIP ${label}: base fetch error: ${e.message}`);
|
||||
skips++;
|
||||
continue;
|
||||
}
|
||||
if (!baseFirst || baseFirst.channel === 'error') {
|
||||
console.log(` SKIP ${label}: base returned error/empty`);
|
||||
skips++;
|
||||
continue;
|
||||
}
|
||||
|
||||
let candFirst;
|
||||
if (CAND_URL) {
|
||||
try {
|
||||
let candRel = await fetchAtSpec(CAND_URL, tc.pkg, tc.spec, tc.os, tc.arch);
|
||||
candFirst = candRel[0];
|
||||
} catch (e) {
|
||||
console.log(` FAIL ${label}: cand fetch error: ${e.message}`);
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
} else {
|
||||
let localResult = await Releases.getReleases({
|
||||
pkg: tc.pkg,
|
||||
ver: tc.ver,
|
||||
os: tc.os,
|
||||
arch: tc.arch,
|
||||
libc: '',
|
||||
lts: tc.lts,
|
||||
channel: tc.channel,
|
||||
formats: [],
|
||||
limit: 1,
|
||||
});
|
||||
candFirst = localResult.releases && localResult.releases[0];
|
||||
}
|
||||
if (!candFirst || candFirst.channel === 'error') {
|
||||
console.log(` FAIL ${label}: cand returned error/empty (base=${baseFirst.version})`);
|
||||
failures++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Both should satisfy the requested spec. A version diff is only a
|
||||
// real failure if cand picked something the spec excludes (e.g.
|
||||
// requested '20' but got '26'). Same prefix on both sides is just
|
||||
// cache-age skew (e.g. 1.21.13 vs 1.21.14).
|
||||
if (baseFirst.version !== candFirst.version) {
|
||||
let prefix = tc.ver;
|
||||
let candMatches = !prefix || new RegExp('^' + prefix + '\\b').test(candFirst.version);
|
||||
let baseMatches = !prefix || new RegExp('^' + prefix + '\\b').test(baseFirst.version);
|
||||
if (!candMatches && baseMatches) {
|
||||
console.log(` FAIL ${label}: cand=${candFirst.version} doesn't match spec; base=${baseFirst.version}`);
|
||||
failures++;
|
||||
} else if (candMatches && !baseMatches) {
|
||||
console.log(` PASS ${label}: cand=${candFirst.version} (base=${baseFirst.version} is wrong)`);
|
||||
passes++;
|
||||
} else {
|
||||
console.log(` PASS ${label}: cand=${candFirst.version} base=${baseFirst.version} (both match '${prefix}'; cache-age skew)`);
|
||||
passes++;
|
||||
}
|
||||
} else {
|
||||
console.log(` PASS ${label}: v${candFirst.version}`);
|
||||
passes++;
|
||||
}
|
||||
}
|
||||
|
||||
// ================================================================
|
||||
// Summary
|
||||
// ================================================================
|
||||
console.log('');
|
||||
console.log(`=== Results: ${passes} passed, ${failures} failed, ${knowns} known, ${skips} skipped ===`);
|
||||
if (failures > 0) {
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch(function (err) {
|
||||
console.error(err.stack);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -1,202 +0,0 @@
|
||||
'use strict';
|
||||
|
||||
// Direct side-by-side comparison: fetch the live installer script from
|
||||
// a remote host with the same UA, and compare WEBI_* variables against
|
||||
// what our local serveInstaller() produces.
|
||||
//
|
||||
// This is the most direct behavioral equivalence test — if the same UA
|
||||
// produces the same WEBI_PKG_URL, WEBI_VERSION, WEBI_EXT, and WEBI_OS,
|
||||
// then the user gets the same binary.
|
||||
//
|
||||
// Usage:
|
||||
// node _webi/test-live-installer-diff.js
|
||||
// node _webi/test-live-installer-diff.js --base-url=https://beta.webi.sh
|
||||
|
||||
let Https = require('node:https');
|
||||
let InstallerServer = require('./serve-installer.js');
|
||||
let Builds = require('./builds.js');
|
||||
|
||||
let BASE_URL = 'https://webinstall.dev';
|
||||
for (let arg of process.argv) {
|
||||
if (arg.startsWith('--base-url=')) {
|
||||
BASE_URL = arg.slice('--base-url='.length).replace(/\/+$/, '');
|
||||
}
|
||||
}
|
||||
|
||||
let CASES = [
|
||||
// bat — Rust project, gnu-linked Linux builds
|
||||
{ pkg: 'bat', ua: 'aarch64/unknown Darwin/24.2.0 libc', label: 'bat macOS arm64' },
|
||||
{ pkg: 'bat', ua: 'x86_64/unknown Darwin/23.0.0 libc', label: 'bat macOS amd64' },
|
||||
{ pkg: 'bat', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'bat Linux amd64' },
|
||||
{ pkg: 'bat', ua: 'x86_64/unknown Linux/5.15.0 musl', label: 'bat Linux musl' },
|
||||
{ pkg: 'bat', ua: 'x86_64/unknown Windows/10.0.19041 msvc', label: 'bat Windows amd64' },
|
||||
// go — Go project, static builds (libc='none')
|
||||
{ pkg: 'go', ua: 'aarch64/unknown Darwin/24.2.0 libc', label: 'go macOS arm64' },
|
||||
{ pkg: 'go', ua: 'x86_64/unknown Darwin/23.0.0 libc', label: 'go macOS amd64' },
|
||||
{ pkg: 'go', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'go Linux amd64' },
|
||||
{ pkg: 'go', ua: 'x86_64/unknown Windows/10.0.19041 msvc', label: 'go Windows amd64' },
|
||||
// node — C++ project, gnu-linked Linux builds, separate musl build
|
||||
{ pkg: 'node', ua: 'aarch64/unknown Darwin/24.2.0 libc', label: 'node macOS arm64' },
|
||||
{ pkg: 'node', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'node Linux amd64',
|
||||
known: 'live fails (WATERFALL gap), local correctly resolves gnu build' },
|
||||
{ pkg: 'node', ua: 'x86_64/unknown Linux/5.15.0 musl', label: 'node Linux musl' },
|
||||
// rg — Rust project, gnu-linked Linux builds
|
||||
{ pkg: 'rg', ua: 'aarch64/unknown Darwin/24.2.0 libc', label: 'rg macOS arm64' },
|
||||
{ pkg: 'rg', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'rg Linux amd64' },
|
||||
{ pkg: 'rg', ua: 'x86_64/unknown Linux/5.15.0 musl', label: 'rg Linux musl' },
|
||||
{ pkg: 'rg', ua: 'x86_64/unknown Windows/10.0.19041 msvc', label: 'rg Windows amd64' },
|
||||
// jq — C project, had .git source URLs in old releases
|
||||
{ pkg: 'jq', ua: 'aarch64/unknown Darwin/24.2.0 libc', label: 'jq macOS arm64' },
|
||||
{ pkg: 'jq', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'jq Linux amd64' },
|
||||
{ pkg: 'jq', ua: 'x86_64/unknown Windows/10.0.19041 msvc', label: 'jq Windows amd64' },
|
||||
// caddy — Go project, had .git source URLs in old releases
|
||||
{ pkg: 'caddy', ua: 'aarch64/unknown Darwin/24.2.0 libc', label: 'caddy macOS arm64' },
|
||||
{ pkg: 'caddy', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'caddy Linux amd64' },
|
||||
{ pkg: 'caddy', ua: 'x86_64/unknown Windows/10.0.19041 msvc', label: 'caddy Windows amd64' },
|
||||
// Additional packages for broader coverage
|
||||
{ pkg: 'shellcheck', ua: 'aarch64/unknown Darwin/24.2.0 libc', label: 'shellcheck macOS arm64' },
|
||||
{ pkg: 'shellcheck', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'shellcheck Linux amd64' },
|
||||
{ pkg: 'shfmt', ua: 'aarch64/unknown Darwin/24.2.0 libc', label: 'shfmt macOS arm64' },
|
||||
{ pkg: 'shfmt', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'shfmt Linux amd64' },
|
||||
{ pkg: 'fd', ua: 'aarch64/unknown Darwin/24.2.0 libc', label: 'fd macOS arm64' },
|
||||
{ pkg: 'fd', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'fd Linux amd64' },
|
||||
{ pkg: 'hugo', ua: 'aarch64/unknown Darwin/24.2.0 libc', label: 'hugo macOS arm64',
|
||||
known: 'classifier rejects darwin-universal as x86_64!=universal2' },
|
||||
{ pkg: 'hugo', ua: 'x86_64/unknown Linux/5.15.0 libc', label: 'hugo Linux amd64' },
|
||||
// Alias tests — these should resolve to the real package
|
||||
{ pkg: 'golang', ua: 'aarch64/unknown Darwin/24.2.0 libc', label: 'golang alias macOS arm64' },
|
||||
{ pkg: 'ripgrep', ua: 'aarch64/unknown Darwin/24.2.0 libc', label: 'ripgrep alias macOS arm64' },
|
||||
];
|
||||
|
||||
// Variables that must match between live and local for the install to work
|
||||
let CRITICAL_VARS = ['PKG_NAME', 'WEBI_OS', 'WEBI_ARCH', 'WEBI_EXT'];
|
||||
// Variables where version differences are OK (cache age)
|
||||
let VERSION_VARS = ['WEBI_VERSION', 'WEBI_PKG_URL', 'WEBI_PKG_FILE'];
|
||||
|
||||
function fetchLiveInstaller(pkg, ua) {
|
||||
return new Promise(function (resolve, reject) {
|
||||
let url = `${BASE_URL}/api/installers/${pkg}@stable.sh`;
|
||||
let opts = { headers: { 'User-Agent': ua } };
|
||||
Https.get(url, opts, function (res) {
|
||||
let data = '';
|
||||
res.on('data', function (chunk) { data += chunk; });
|
||||
res.on('end', function () { resolve(data); });
|
||||
}).on('error', reject);
|
||||
});
|
||||
}
|
||||
|
||||
function parseVars(script) {
|
||||
let vars = {};
|
||||
// Match: PKG_NAME='bat' or WEBI_VERSION='1.2.3' (with or without export)
|
||||
let re = /^(?:export\s+)?(WEBI_\w+|PKG_NAME)='([^']*)'/gm;
|
||||
let m;
|
||||
while ((m = re.exec(script)) !== null) {
|
||||
vars[m[1]] = m[2];
|
||||
}
|
||||
return vars;
|
||||
}
|
||||
|
||||
async function main() {
|
||||
let passes = 0;
|
||||
let failures = 0;
|
||||
let knowns = 0;
|
||||
let errors = 0;
|
||||
|
||||
console.log('Initializing build cache...');
|
||||
await Builds.init();
|
||||
console.log('');
|
||||
|
||||
console.log('=== Live vs Local Installer Diff ===');
|
||||
console.log('');
|
||||
|
||||
for (let tc of CASES) {
|
||||
// Fetch live
|
||||
let liveScript;
|
||||
try {
|
||||
liveScript = await fetchLiveInstaller(tc.pkg, tc.ua);
|
||||
} catch (e) {
|
||||
console.log(` SKIP ${tc.label}: fetch error: ${e.message}`);
|
||||
continue;
|
||||
}
|
||||
let liveVars = parseVars(liveScript);
|
||||
|
||||
if (!liveVars.WEBI_PKG_URL) {
|
||||
console.log(` SKIP ${tc.label}: live returned no WEBI_PKG_URL`);
|
||||
continue;
|
||||
}
|
||||
|
||||
// Render local
|
||||
let localScript;
|
||||
try {
|
||||
localScript = await InstallerServer.serveInstaller(
|
||||
BASE_URL,
|
||||
tc.ua,
|
||||
tc.pkg,
|
||||
'stable',
|
||||
'sh',
|
||||
['tar', 'exe', 'zip', 'xz', 'dmg'],
|
||||
'',
|
||||
);
|
||||
} catch (e) {
|
||||
console.log(` ERROR ${tc.label}: local error: ${e.message}`);
|
||||
errors++;
|
||||
continue;
|
||||
}
|
||||
let localVars = parseVars(localScript);
|
||||
|
||||
if (tc.known) {
|
||||
let localExt = localVars.WEBI_EXT || 'err';
|
||||
let liveExt = liveVars.WEBI_EXT || '?';
|
||||
if (localExt !== liveExt) {
|
||||
console.log(` KNOWN ${tc.label}: ${tc.known} (live=${liveExt} local=${localExt})`);
|
||||
knowns++;
|
||||
} else {
|
||||
console.log(` PASS ${tc.label}: known issue resolved! v${localVars.WEBI_VERSION} .${localExt}`);
|
||||
passes++;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!localVars.WEBI_PKG_URL || localVars.WEBI_EXT === 'err') {
|
||||
console.log(` KNOWN ${tc.label}: local failed to resolve (live=${liveVars.WEBI_EXT})`);
|
||||
knowns++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Compare critical vars
|
||||
let diffs = [];
|
||||
for (let v of CRITICAL_VARS) {
|
||||
let liveVal = liveVars[v] || '';
|
||||
let localVal = localVars[v] || '';
|
||||
if (liveVal !== localVal) {
|
||||
diffs.push(`${v}: live='${liveVal}' local='${localVal}'`);
|
||||
}
|
||||
}
|
||||
|
||||
// Log version info (informational, not failure)
|
||||
let versionNote = '';
|
||||
if (liveVars.WEBI_VERSION !== localVars.WEBI_VERSION) {
|
||||
versionNote = ` (version: live=${liveVars.WEBI_VERSION} local=${localVars.WEBI_VERSION})`;
|
||||
}
|
||||
|
||||
if (diffs.length > 0) {
|
||||
console.log(` FAIL ${tc.label}: ${diffs.join(', ')}${versionNote}`);
|
||||
failures++;
|
||||
} else {
|
||||
let file = (localVars.WEBI_PKG_URL || '').split('/').pop();
|
||||
console.log(` PASS ${tc.label}: v${localVars.WEBI_VERSION} .${localVars.WEBI_EXT} ${file}${versionNote}`);
|
||||
passes++;
|
||||
}
|
||||
}
|
||||
|
||||
console.log('');
|
||||
console.log(`=== Results: ${passes} passed, ${failures} failed, ${knowns} known, ${errors} errors ===`);
|
||||
if (failures > 0 || errors > 0) {
|
||||
process.exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
main().catch(function (err) {
|
||||
console.error(err.stack);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -32,7 +32,7 @@ if (/\b-?-h(elp)?\b/.test(process.argv.join(' '))) {
|
||||
var os = require('os');
|
||||
var fs = require('fs');
|
||||
var path = require('path');
|
||||
var Builds = require('./builds.js');
|
||||
var Releases = require('./transform-releases.js');
|
||||
var Installers = require('./installers.js');
|
||||
var ServeInstaller = require('./serve-installer.js');
|
||||
|
||||
@@ -49,7 +49,7 @@ var baseurl = 'https://webinstall.dev';
|
||||
var maxLen = 0;
|
||||
console.info('');
|
||||
console.info('Has the necessary files?');
|
||||
['README.md', 'install.sh', 'install.ps1']
|
||||
['README.md', 'releases.js', 'install.sh', 'install.ps1']
|
||||
.map(function (node) {
|
||||
maxLen = Math.max(maxLen, node.length);
|
||||
return node;
|
||||
@@ -65,8 +65,7 @@ console.info('Has the necessary files?');
|
||||
});
|
||||
|
||||
console.info('');
|
||||
let projName = pkgdir.split('/').filter(Boolean).pop();
|
||||
Builds.getPackage({ name: projName }).then(async function (/*projInfo*/) {
|
||||
Releases.get(path.join(process.cwd(), pkgdir)).then(async function (all) {
|
||||
var pkgname = path.basename(pkgdir.replace(/\/$/, ''));
|
||||
var nodeOs = os.platform();
|
||||
var nodeOsRelease = os.release();
|
||||
@@ -83,8 +82,8 @@ Builds.getPackage({ name: projName }).then(async function (/*projInfo*/) {
|
||||
var formats = ['exe', 'xz', 'tar', 'zip', 'git'];
|
||||
|
||||
let [rel, opts] = await ServeInstaller.helper({
|
||||
unameAgent: `${nodeOs}/${nodeOsRelease} ${nodeArch}/unknown ${nodeLibc}`,
|
||||
projectName: pkgname,
|
||||
ua: `${nodeOs}/${nodeOsRelease} ${nodeArch}/unknown ${nodeLibc}`,
|
||||
pkg: pkgname,
|
||||
tag: pkgtag || '',
|
||||
formats: formats,
|
||||
libc: nodeLibc,
|
||||
|
||||
@@ -2,30 +2,72 @@
|
||||
|
||||
var Releases = module.exports;
|
||||
|
||||
var Fs = require('node:fs/promises');
|
||||
var Os = require('node:os');
|
||||
var path = require('path');
|
||||
var request = require('@root/request');
|
||||
var _normalize = require('./normalize.js');
|
||||
|
||||
var cache = {};
|
||||
//var staleAge = 5 * 1000;
|
||||
//var expiredAge = 15 * 1000;
|
||||
var staleAge = 5 * 60 * 1000;
|
||||
var expiredAge = 15 * 60 * 1000;
|
||||
|
||||
var LEGACY_CACHE_DIR = path.join(Os.homedir(), '.cache/webi/legacy');
|
||||
let installerDir = path.join(__dirname, '..');
|
||||
|
||||
// Sort releases by ext preference and libc within the same version.
|
||||
// The cache is already sorted by version (stable before beta, newest first),
|
||||
// so we only re-order within the same version string.
|
||||
Releases.get = async function (pkgdir) {
|
||||
let get;
|
||||
try {
|
||||
get = require(path.join(pkgdir, 'releases.js'));
|
||||
} catch (e) {
|
||||
let err = new Error('no releases.js for', pkgdir.split(/[\/\\]+/).pop());
|
||||
err.code = 'E_NO_RELEASE';
|
||||
throw err;
|
||||
}
|
||||
|
||||
let all = await get(request);
|
||||
|
||||
return _normalize(all);
|
||||
};
|
||||
|
||||
// TODO needs a proper test, and more accurate (though perhaps far less simple) code
|
||||
function createFormatsSorter(formats) {
|
||||
return function sortByExtLibc(a, b) {
|
||||
if (a.version !== b.version) {
|
||||
// Array.sort is stable (V8, ES2019), so returning 0 across
|
||||
// versions preserves the cache's pre-sorted version-desc order.
|
||||
return 0;
|
||||
return function sortByVerExt(a, b) {
|
||||
function lexver(semver) {
|
||||
// v1.20.156 => 00001.00020.00156.zzzzz
|
||||
// TODO BUG: v1.20.156-rc2 => 00001.00020.00156.rc2zz
|
||||
var parts = semver.split(/[+\.\-]/g);
|
||||
while (parts.length < 4) {
|
||||
parts.push('');
|
||||
}
|
||||
return parts
|
||||
.map(function (num, i) {
|
||||
if (3 === i) {
|
||||
return num.toString().padEnd(10, 'z');
|
||||
}
|
||||
return num.toString().padStart(10, '0');
|
||||
})
|
||||
.join('.');
|
||||
}
|
||||
|
||||
var aver = lexver(a.version);
|
||||
var bver = lexver(b.version);
|
||||
if (aver > bver) {
|
||||
//console.log(aver, '>', bver);
|
||||
return -1;
|
||||
}
|
||||
if (aver < bver) {
|
||||
//console.log(aver, '<', bver);
|
||||
return 1;
|
||||
}
|
||||
|
||||
var aExtPri = formats.indexOf(a.ext.replace(/tar\..*/, 'tar'));
|
||||
var bExtPri = formats.indexOf(b.ext.replace(/tar\..*/, 'tar'));
|
||||
if (aExtPri > bExtPri) {
|
||||
//console.log(a.ext, aExtPri, '>', b.ext, bExtPri);
|
||||
return -1;
|
||||
}
|
||||
if (aExtPri < bExtPri) {
|
||||
//console.log(a.ext, aExtPri, '<', b.ext, bExtPri);
|
||||
return 1;
|
||||
}
|
||||
|
||||
@@ -42,39 +84,99 @@ function createFormatsSorter(formats) {
|
||||
}
|
||||
|
||||
async function getCachedReleases(pkg) {
|
||||
// returns { download: '', releases: [{ version, date, os, arch, lts, channel, download}] }
|
||||
// returns { download: '<template string>', releases: [{ version, date, os, arch, lts, channel, download}] }
|
||||
|
||||
if (cache[pkg]) {
|
||||
return cache[pkg];
|
||||
async function chainCachePromise(fn) {
|
||||
cache[pkg].promise = cache[pkg].promise.then(fn);
|
||||
return cache[pkg].promise;
|
||||
}
|
||||
|
||||
let dataFile = `${LEGACY_CACHE_DIR}/${pkg}.json`;
|
||||
async function sleep(ms) {
|
||||
return await new Promise(function (resolve, reject) {
|
||||
setTimeout(resolve, ms);
|
||||
});
|
||||
}
|
||||
|
||||
let json = await Fs.readFile(dataFile, 'utf8').catch(function (err) {
|
||||
if (err.code === 'ENOENT') {
|
||||
return null;
|
||||
async function putCache() {
|
||||
var age = Date.now() - cache[pkg].updatedAt;
|
||||
if (age < staleAge) {
|
||||
//console.debug('NOT STALE ANYMORE - updated in previous promise');
|
||||
return cache[pkg].all;
|
||||
}
|
||||
throw err;
|
||||
});
|
||||
|
||||
if (!json) {
|
||||
let empty = { download: '', releases: [] };
|
||||
cache[pkg] = empty;
|
||||
return empty;
|
||||
//console.debug('DOWNLOADING NEW "%s" releases', pkg);
|
||||
var pkgdir = path.join(installerDir, pkg);
|
||||
|
||||
// workaround for request timeout seeming to not work
|
||||
let complete = false;
|
||||
await Promise.race([
|
||||
Releases.get(pkgdir)
|
||||
.catch(function (err) {
|
||||
if ('E_NO_RELEASE' === err.code) {
|
||||
let all = { _error: 'E_NO_RELEASE', download: '', releases: [] };
|
||||
return all;
|
||||
}
|
||||
|
||||
throw err;
|
||||
})
|
||||
.catch(function (err) {
|
||||
let hasReleases = cache[pkg].all?.releases?.length > 1;
|
||||
if (!hasReleases) {
|
||||
throw err;
|
||||
}
|
||||
|
||||
console.error(`Error: the BOOGEYMAN got us!`);
|
||||
console.error(err.stack);
|
||||
|
||||
return cache[pkg].all;
|
||||
})
|
||||
.then(function (all) {
|
||||
// Note: it is possible for slightly older data
|
||||
// to replace slightly newer data, but this is better
|
||||
// than being in a cycle where release updates _always_
|
||||
// take longer than expected.
|
||||
//console.debug('DOWNLOADED NEW "%s" releases', pkg);
|
||||
cache[pkg].updatedAt = Date.now();
|
||||
cache[pkg].all = all;
|
||||
complete = true;
|
||||
}),
|
||||
sleep(15000).then(function () {
|
||||
if (complete) {
|
||||
return;
|
||||
}
|
||||
console.error(`request timeout waiting for '${pkg}' release info`);
|
||||
}),
|
||||
]);
|
||||
|
||||
return cache[pkg].all;
|
||||
}
|
||||
|
||||
let all;
|
||||
try {
|
||||
all = JSON.parse(json);
|
||||
} catch (e) {
|
||||
console.error(`error: ${dataFile}:\n\t${e.message}`);
|
||||
let empty = { download: '', releases: [] };
|
||||
cache[pkg] = empty;
|
||||
return empty;
|
||||
if (!cache[pkg]) {
|
||||
cache[pkg] = {
|
||||
updatedAt: 0,
|
||||
all: { download: '', releases: [] },
|
||||
promise: Promise.resolve(),
|
||||
};
|
||||
}
|
||||
|
||||
cache[pkg] = all;
|
||||
return all;
|
||||
var bgRenewal;
|
||||
var age = Date.now() - cache[pkg].updatedAt;
|
||||
var fresh = age < staleAge;
|
||||
if (!fresh) {
|
||||
bgRenewal = chainCachePromise(putCache);
|
||||
}
|
||||
|
||||
var tooStale = age > expiredAge;
|
||||
if (!tooStale) {
|
||||
return await cache[pkg].all;
|
||||
}
|
||||
|
||||
return await Promise.race([
|
||||
bgRenewal,
|
||||
sleep(5000).then(function () {
|
||||
return cache[pkg].all;
|
||||
}),
|
||||
]);
|
||||
}
|
||||
|
||||
async function filterReleases(
|
||||
@@ -85,22 +187,15 @@ async function filterReleases(
|
||||
// sort the most compatible format first
|
||||
// (i.e. so that we don't do .pkg on linux except on purpose)
|
||||
var rformats = formats.slice(0).reverse();
|
||||
var sortByExtLibc = createFormatsSorter(rformats);
|
||||
var sortByVerExt = createFormatsSorter(rformats);
|
||||
var reVer = new RegExp('^' + ver + '\\b');
|
||||
|
||||
function selectMatches(rel) {
|
||||
/* jshint maxcomplexity: 25 */
|
||||
if (os) {
|
||||
// '*' = any OS (matches anything, including windows).
|
||||
// 'posix' / 'posix_20xx' = any POSIX OS (matches linux, macos,
|
||||
// freebsd, etc., but NOT windows).
|
||||
let isPosix = rel.os === 'posix' || rel.os.startsWith('posix_20');
|
||||
let osMatches =
|
||||
rel.os === '*' ||
|
||||
rel.os === os ||
|
||||
(isPosix && os !== 'windows');
|
||||
if (!osMatches) {
|
||||
return false;
|
||||
if (rel.os !== '*') {
|
||||
if (rel.os !== os) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -112,10 +207,7 @@ async function filterReleases(
|
||||
}
|
||||
}
|
||||
|
||||
// libc='libc' is serve-releases.js's default when the caller
|
||||
// didn't pin one — treat it as 'no preference', not a filter.
|
||||
let isMeaningfulLibc = libc && libc !== 'libc' && rel.libc !== 'none';
|
||||
if (isMeaningfulLibc) {
|
||||
if (rel.libc !== 'none') {
|
||||
let releaseRequiresMusl = rel.libc === 'musl';
|
||||
// goal: handle non-glibc (Alpine / Docker / musl)
|
||||
let osHasMusl = libc === 'musl';
|
||||
@@ -162,7 +254,7 @@ async function filterReleases(
|
||||
return true;
|
||||
}
|
||||
|
||||
var sortedRels = all.releases.filter(selectMatches).sort(sortByExtLibc);
|
||||
var sortedRels = all.releases.filter(selectMatches).sort(sortByVerExt);
|
||||
//console.log(sortedRels.slice(0, 4));
|
||||
|
||||
return sortedRels.slice(0, limit || 1000);
|
||||
@@ -321,8 +413,8 @@ Releases.getReleases = function ({
|
||||
};
|
||||
|
||||
if (require.main === module) {
|
||||
return Releases
|
||||
.getReleases({
|
||||
return module
|
||||
.exports({
|
||||
pkg: 'node',
|
||||
ver: '',
|
||||
os: 'macos',
|
||||
|
||||
@@ -38,8 +38,9 @@ function getOs(ua) {
|
||||
// It's the year of the Linux Desktop!
|
||||
// See also http://www.mslinux.org/
|
||||
// 'linux' must be tested before 'Microsoft' because WSL
|
||||
// (TODO: does this affect cygwin / msysgit?)
|
||||
return 'linux';
|
||||
} else if (/^ms$|Microsoft|Windows|win32|win|PowerShell|CYGWIN|MINGW/i.test(ua)) {
|
||||
} else if (/^ms$|Microsoft|Windows|win32|win|PowerShell/i.test(ua)) {
|
||||
// 'win' must be tested after 'darwin'
|
||||
return 'windows';
|
||||
} else if (/Linux|curl|wget/i.test(ua)) {
|
||||
|
||||
@@ -72,8 +72,8 @@ gc "feat: new feature"
|
||||
|
||||
### Common aliases
|
||||
|
||||
Use *alias*es to make other tools you find around webi even _more_ convenient ⚡️
|
||||
(and powerful 💪).
|
||||
Use *alias*es to make other tools you find around webi even _more_ convenient
|
||||
⚡️ (and powerful 💪).
|
||||
|
||||
```sh
|
||||
aliasman curl 'curlie'
|
||||
|
||||
@@ -1,2 +0,0 @@
|
||||
github_sources = BeyondCodeBootcamp/aliasman
|
||||
git_url = https://github.com/BeyondCodeBootcamp/aliasman.git
|
||||
30
aliasman/releases.js
Normal file
30
aliasman/releases.js
Normal file
@@ -0,0 +1,30 @@
|
||||
'use strict';
|
||||
|
||||
var githubSource = require('../_common/github-source.js');
|
||||
var owner = 'BeyondCodeBootcamp';
|
||||
var repo = 'aliasman';
|
||||
|
||||
module.exports = function (request) {
|
||||
let arches = [
|
||||
'amd64',
|
||||
'arm64',
|
||||
'armv6l',
|
||||
'armv7l',
|
||||
'ppc64le',
|
||||
'ppc64',
|
||||
's390x',
|
||||
'x86',
|
||||
];
|
||||
let oses = ['freebsd', 'linux', 'macos', 'posix'];
|
||||
return githubSource(request, owner, repo, oses, arches).then(function (all) {
|
||||
all._names = ['aliasman', 'legacy'];
|
||||
return all;
|
||||
});
|
||||
};
|
||||
|
||||
if (module === require.main) {
|
||||
module.exports(require('@root/request')).then(function (all) {
|
||||
all = require('../_webi/normalize.js')(all);
|
||||
console.info(JSON.stringify(all, null, 2));
|
||||
});
|
||||
}
|
||||
@@ -19,13 +19,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
|
||||
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
|
||||
|
||||
# Fetch archive
|
||||
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
Write-Output "Downloading archiver from $Env:WEBI_PKG_URL to $pkg_download"
|
||||
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
|
||||
& Move-Item "$pkg_download.part" "$pkg_download"
|
||||
}
|
||||
|
||||
if (!(Test-Path -Path "$pkg_src_cmd")) {
|
||||
IF (!(Test-Path -Path "$pkg_src_cmd")) {
|
||||
Write-Output "Installing archiver"
|
||||
|
||||
# TODO: create package-specific temp directory
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
github_releases = mholt/archiver
|
||||
21
arc/releases.js
Normal file
21
arc/releases.js
Normal file
@@ -0,0 +1,21 @@
|
||||
'use strict';
|
||||
|
||||
var github = require('../_common/github.js');
|
||||
var owner = 'mholt';
|
||||
var repo = 'archiver';
|
||||
|
||||
module.exports = function (request) {
|
||||
return github(request, owner, repo).then(function (all) {
|
||||
all._names = ['archiver', 'arc'];
|
||||
return all;
|
||||
});
|
||||
};
|
||||
|
||||
if (module === require.main) {
|
||||
module.exports(require('@root/request')).then(function (all) {
|
||||
all = require('../_webi/normalize.js')(all);
|
||||
// just select the first 5 for demonstration
|
||||
all.releases = all.releases.slice(0, 5);
|
||||
console.info(JSON.stringify(all, null, 2));
|
||||
});
|
||||
}
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/bin/pwsh
|
||||
|
||||
Write-Output "'archiver@$Env:WEBI_TAG' is an alias for 'arc@$Env:WEBI_VERSION'"
|
||||
if ($null -eq $Env:WEBI_HOST -or $Env:WEBI_HOST -eq "") { $Env:WEBI_HOST = "https://webinstall.dev" }
|
||||
IF ($null -eq $Env:WEBI_HOST -or $Env:WEBI_HOST -eq "") { $Env:WEBI_HOST = "https://webinstall.dev" }
|
||||
curl.exe -A MS -fsSL "$Env:WEBI_HOST/arc@$Env:WEBI_VERSION" | powershell
|
||||
|
||||
@@ -20,7 +20,7 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
|
||||
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
|
||||
|
||||
# Fetch archive
|
||||
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
Write-Output "Checking for (or Installing) MSVC Runtime..."
|
||||
& "$Env:USERPROFILE\.local\bin\webi-pwsh.ps1" vcruntime
|
||||
|
||||
@@ -29,7 +29,7 @@ if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
& Move-Item "$pkg_download.part" "$pkg_download"
|
||||
}
|
||||
|
||||
if (!(Test-Path -Path "$pkg_src_cmd")) {
|
||||
IF (!(Test-Path -Path "$pkg_src_cmd")) {
|
||||
Write-Output "Installing AtomicParsley"
|
||||
|
||||
# TODO: create package-specific temp directory
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
github_releases = wez/atomicparsley
|
||||
79
atomicparsley/releases.js
Normal file
79
atomicparsley/releases.js
Normal file
@@ -0,0 +1,79 @@
|
||||
'use strict';
|
||||
|
||||
var github = require('../_common/github.js');
|
||||
var owner = 'wez';
|
||||
var repo = 'atomicparsley';
|
||||
|
||||
let targets = {
|
||||
x86win: {
|
||||
os: 'windows',
|
||||
arch: 'x86',
|
||||
libc: 'msvc',
|
||||
},
|
||||
x64win: {
|
||||
os: 'windows',
|
||||
arch: 'amd64',
|
||||
// https://github.com/wez/atomicparsley/issues/6#issuecomment-1364523028
|
||||
libc: 'msvc',
|
||||
},
|
||||
x64mac: {
|
||||
os: 'macos',
|
||||
arch: 'amd64',
|
||||
},
|
||||
x64lin: {
|
||||
os: 'linux',
|
||||
arch: 'amd64',
|
||||
libc: 'gnu',
|
||||
},
|
||||
x64musl: {
|
||||
os: 'linux',
|
||||
arch: 'amd64',
|
||||
libc: 'musl',
|
||||
},
|
||||
};
|
||||
|
||||
module.exports = function (request) {
|
||||
return github(request, owner, repo).then(function (all) {
|
||||
for (let rel of all.releases) {
|
||||
let windows32 = rel.name.includes('WindowsX86.');
|
||||
if (windows32) {
|
||||
Object.assign(rel, targets.x86win);
|
||||
continue;
|
||||
}
|
||||
|
||||
let windows64 = rel.name.includes('Windows.');
|
||||
if (windows64) {
|
||||
Object.assign(rel, targets.x64win);
|
||||
continue;
|
||||
}
|
||||
|
||||
let macos64 = rel.name.includes('MacOS');
|
||||
if (macos64) {
|
||||
Object.assign(rel, targets.x64mac);
|
||||
continue;
|
||||
}
|
||||
|
||||
let musl64 = rel.name.includes('Alpine');
|
||||
if (musl64) {
|
||||
Object.assign(rel, targets.x64musl);
|
||||
continue;
|
||||
}
|
||||
|
||||
let lin64 = rel.name.includes('Linux.');
|
||||
if (lin64) {
|
||||
Object.assign(rel, targets.x64lin);
|
||||
continue;
|
||||
}
|
||||
}
|
||||
all._names = ['AtomicParsley', 'atomicparsley'];
|
||||
return all;
|
||||
});
|
||||
};
|
||||
|
||||
if (module === require.main) {
|
||||
module.exports(require('@root/request')).then(function (all) {
|
||||
all = require('../_webi/normalize.js')(all);
|
||||
console.info(JSON.stringify(all));
|
||||
//console.info(JSON.stringify(all, null, 2));
|
||||
});
|
||||
}
|
||||
@@ -20,13 +20,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
|
||||
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
|
||||
|
||||
# Fetch archive
|
||||
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
Write-Output "Downloading awless from $Env:WEBI_PKG_URL to $pkg_download"
|
||||
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
|
||||
& Move-Item "$pkg_download.part" "$pkg_download"
|
||||
}
|
||||
|
||||
if (!(Test-Path -Path "$pkg_src_cmd")) {
|
||||
IF (!(Test-Path -Path "$pkg_src_cmd")) {
|
||||
Write-Output "Installing awless"
|
||||
|
||||
# TODO: create package-specific temp directory
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
github_releases = wallix/awless
|
||||
22
awless/releases.js
Normal file
22
awless/releases.js
Normal file
@@ -0,0 +1,22 @@
|
||||
'use strict';
|
||||
|
||||
var github = require('../_common/github.js');
|
||||
var owner = 'wallix';
|
||||
var repo = 'awless';
|
||||
|
||||
module.exports = function (request) {
|
||||
return github(request, owner, repo).then(function (all) {
|
||||
// remove checksums and .deb
|
||||
all.releases = all.releases.filter(function (rel) {
|
||||
return !/(\.txt)|(\.deb)$/i.test(rel.name);
|
||||
});
|
||||
return all;
|
||||
});
|
||||
};
|
||||
|
||||
if (module === require.main) {
|
||||
module.exports(require('@root/request')).then(function (all) {
|
||||
all = require('../_webi/normalize.js')(all);
|
||||
console.info(JSON.stringify(all));
|
||||
});
|
||||
}
|
||||
@@ -3,7 +3,7 @@
|
||||
$VERNAME = "$Env:PKG_NAME-v$Env:WEBI_VERSION.exe"
|
||||
$EXENAME = "$Env:PKG_NAME.exe"
|
||||
# Fetch archive
|
||||
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
Write-Output "Downloading $Env:PKG_NAME from $Env:WEBI_PKG_URL to $Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
|
||||
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part"
|
||||
& Move-Item "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part" "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
|
||||
@@ -11,11 +11,11 @@ if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
|
||||
# Fetch MSVC Runtime
|
||||
Write-Output "Checking for MSVC Runtime..."
|
||||
if (-not (Test-Path "\Windows\System32\vcruntime140.dll")) {
|
||||
IF (-not (Test-Path "\Windows\System32\vcruntime140.dll")) {
|
||||
& "$Env:USERPROFILE\.local\bin\webi-pwsh.ps1" vcruntime
|
||||
}
|
||||
|
||||
if (!(Test-Path -Path "$Env:USERPROFILE\.local\bin\$VERNAME")) {
|
||||
IF (!(Test-Path -Path "$Env:USERPROFILE\.local\bin\$VERNAME")) {
|
||||
Write-Output "Installing $Env:PKG_NAME"
|
||||
# TODO: temp directory
|
||||
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
github_releases = sharkdp/bat
|
||||
20
bat/releases.js
Normal file
20
bat/releases.js
Normal file
@@ -0,0 +1,20 @@
|
||||
'use strict';
|
||||
|
||||
var github = require('../_common/github.js');
|
||||
var owner = 'sharkdp';
|
||||
var repo = 'bat';
|
||||
|
||||
module.exports = function (request) {
|
||||
return github(request, owner, repo).then(function (all) {
|
||||
return all;
|
||||
});
|
||||
};
|
||||
|
||||
if (module === require.main) {
|
||||
module.exports(require('@root/request')).then(function (all) {
|
||||
all = require('../_webi/normalize.js')(all);
|
||||
all.releases = all.releases.slice(0, 10);
|
||||
//console.info(JSON.stringify(all));
|
||||
console.info(JSON.stringify(all, null, 2));
|
||||
});
|
||||
}
|
||||
@@ -6,23 +6,15 @@ main() { (
|
||||
sed '1,/^#~\/.local\/bin\/brew-updater/d' "${0}" > ~/.local/bin/brew-update-hourly
|
||||
chmod a+x ~/.local/bin/brew-update-hourly
|
||||
|
||||
echo "Checking for serviceman..."
|
||||
~/.local/bin/webi serviceman
|
||||
if ! command -v serviceman > /dev/null; then
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
fi
|
||||
serviceman --version
|
||||
|
||||
serviceman add --agent \
|
||||
env PATH="$PATH" serviceman add --user \
|
||||
--workdir ~/.local/opt/brew/ \
|
||||
--name sh.brew.updater -- \
|
||||
~/.local/bin/brew-update-hourly
|
||||
); }
|
||||
|
||||
if ! main; then
|
||||
exit 1
|
||||
if main; then
|
||||
exit 0
|
||||
fi
|
||||
exit 0
|
||||
|
||||
#~/.local/bin/brew-updater
|
||||
#!/bin/sh
|
||||
|
||||
@@ -132,7 +132,9 @@ file)
|
||||
```
|
||||
3. Add your project to the system launcher, running as the current user
|
||||
```sh
|
||||
serviceman add --name 'my-project' --daemon -- \
|
||||
sudo env PATH="$PATH" \
|
||||
serviceman add --path="$PATH" --system \
|
||||
--username "$(whoami)" --name my-project -- \
|
||||
bun run ./my-project.js
|
||||
```
|
||||
4. Restart the logging service
|
||||
@@ -153,6 +155,6 @@ For **macOS**:
|
||||
```
|
||||
3. Add your project to the system launcher, running as the current user
|
||||
```sh
|
||||
serviceman add --agent --name 'my-project' -- \
|
||||
serviceman add --path="$PATH" --user --name my-project -- \
|
||||
bun run ./my-project.js
|
||||
```
|
||||
|
||||
@@ -1,5 +0,0 @@
|
||||
github_releases = oven-sh/bun
|
||||
tag_prefix = bun-
|
||||
default_x86_64 = x86_64_v3
|
||||
x86_64_v2 = baseline
|
||||
variants = profile
|
||||
39
bun/releases.js
Normal file
39
bun/releases.js
Normal file
@@ -0,0 +1,39 @@
|
||||
'use strict';
|
||||
|
||||
var github = require('../_common/github.js');
|
||||
var owner = 'oven-sh';
|
||||
var repo = 'bun';
|
||||
|
||||
module.exports = function (request) {
|
||||
return github(request, owner, repo).then(function (all) {
|
||||
all.releases = all.releases
|
||||
.filter(function (r) {
|
||||
let isDebug = r.name.includes('-profile');
|
||||
if (isDebug) {
|
||||
return false;
|
||||
}
|
||||
|
||||
let isAncient = r.name.includes('-baseline');
|
||||
if (isAncient) {
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
})
|
||||
.map(function (r) {
|
||||
// bun-v0.5.1 => v0.5.1
|
||||
r.version = r.version.replace(/bun-/g, '');
|
||||
return r;
|
||||
});
|
||||
return all;
|
||||
});
|
||||
};
|
||||
|
||||
if (module === require.main) {
|
||||
module.exports(require('@root/request')).then(function (all) {
|
||||
all = require('../_webi/normalize.js')(all);
|
||||
// just select the first 5 for demonstration
|
||||
all.releases = all.releases.slice(0, 5);
|
||||
console.info(JSON.stringify(all, null, 2));
|
||||
});
|
||||
}
|
||||
@@ -819,10 +819,10 @@ To avoid the nitty-gritty details of `launchd` plist files, you can use
|
||||
2. Use Serviceman to create a _launchd_ plist file
|
||||
|
||||
```sh
|
||||
my_username="$(id -u -n)"
|
||||
my_username="$( id -u -n )"
|
||||
|
||||
serviceman add --agent --name 'caddy' --workdir ./ -- \
|
||||
caddy run --envfile ~/.config/caddy/env --config ./Caddyfile --adapter caddyfile
|
||||
serviceman add --user --name caddy -- \
|
||||
caddy run --config ./Caddyfile --envfile ~/.config/caddy/env
|
||||
```
|
||||
|
||||
(this will create `~/Library/LaunchAgents/caddy.plist`)
|
||||
@@ -837,8 +837,8 @@ This process creates a _User-Level_ service in `~/Library/LaunchAgents`. To
|
||||
create a _System-Level_ service in `/Library/LaunchDaemons/` instead:
|
||||
|
||||
```sh
|
||||
serviceman add --name 'caddy' --workdir ./ --daemon -- \
|
||||
caddy run --envfile ~/.config/caddy/env --config ./Caddyfile --adapter caddyfile
|
||||
sudo serviceman add --system --name caddy -- \
|
||||
caddy run --config ./Caddyfile --envfile ~/.config/caddy/env
|
||||
```
|
||||
|
||||
### How to run Caddy as a Windows Service
|
||||
@@ -856,7 +856,7 @@ serviceman add --name 'caddy' --workdir ./ --daemon -- \
|
||||
3. Create a **Startup Registry Entry** with Serviceman.
|
||||
```sh
|
||||
serviceman.exe add --name caddy -- \
|
||||
caddy run --envfile ~/.config/caddy/env --config ./Caddyfile --adapter caddyfile
|
||||
caddy run --config ./Caddyfile --envfile ~/.config/caddy/env
|
||||
```
|
||||
4. You can manage the service directly with Serviceman. For example:
|
||||
```sh
|
||||
@@ -901,8 +901,10 @@ See the notes below to run as a **User Service** or use the JSON Config.
|
||||
```
|
||||
4. Use Serviceman to create a _systemd_ config file.
|
||||
```sh
|
||||
serviceman add --name 'caddy' --daemon -- \
|
||||
caddy run --envfile ~/.config/caddy/env --config ./Caddyfile --adapter caddyfile
|
||||
my_username="$( id -u -n )"
|
||||
sudo env PATH="$PATH" \
|
||||
serviceman add --system --username "${my_username}" --name caddy -- \
|
||||
caddy run --config ./Caddyfile --envfile ~/.config/caddy/env
|
||||
```
|
||||
(this will create `/etc/systemd/system/caddy.service`)
|
||||
5. Manage the service with `systemctl` and `journalctl`:
|
||||
@@ -913,10 +915,10 @@ See the notes below to run as a **User Service** or use the JSON Config.
|
||||
|
||||
To create a **User Service** instead:
|
||||
|
||||
- use `--agent` when running `serviceman`:
|
||||
- don't use `sudo`, but do use `--user` when running `serviceman`:
|
||||
```sh
|
||||
serviceman add --agent --name caddy -- \
|
||||
caddy run --envfile ~/.config/caddy/env --config ./Caddyfile --adapter caddyfile
|
||||
serviceman add --user --name caddy -- \
|
||||
caddy run --config ./Caddyfile --envfile ~/.config/caddy/env
|
||||
```
|
||||
(this will create `~/.config/systemd/user/`)
|
||||
- user the `--user` flag to manage services and logs:
|
||||
@@ -1181,8 +1183,7 @@ To prevent search engine and browser confusion
|
||||
- _DO NOT_ prevent crawling via `robots.txt` \
|
||||
(counter-intuitive, but pages _must_ be crawled for links to _NOT_ be indexed)
|
||||
- _all_ domains using public TLS certs _will_ be indexed by default \
|
||||
(they are all linked to and crawled from various Certificate Transparency
|
||||
reports)
|
||||
(they are all linked to and crawled from various Certificate Transparency reports)
|
||||
- follow these guidelines even if the dev sites use HTTP Basic Auth
|
||||
|
||||
```Caddyfile
|
||||
@@ -1362,13 +1363,19 @@ See also: <https://caddyserver.com/docs/running>
|
||||
2. Generate the `service` file: \
|
||||
- JSON Config
|
||||
```sh
|
||||
serviceman add --name 'caddy' --daemon -- \
|
||||
caddy run --resume --envfile ./caddy.env
|
||||
my_app_user="$( id -u -n )"
|
||||
sudo env PATH="${PATH}" \
|
||||
serviceman add --system --cap-net-bind \
|
||||
--username "${my_app_user}" --name caddy -- \
|
||||
caddy run --resume --envfile ./caddy.env
|
||||
```
|
||||
- Caddyfile
|
||||
```sh
|
||||
serviceman add --name 'caddy' --daemon -- \
|
||||
caddy run --config ./Caddyfile --envfile ./caddy.env
|
||||
my_app_user="$( id -u -n )"
|
||||
sudo env PATH="${PATH}" \
|
||||
serviceman add --system --cap-net-bind \
|
||||
--username "${my_app_user}" --name caddy -- \
|
||||
caddy run --config ./Caddyfile --envfile ./caddy.env
|
||||
```
|
||||
3. Reload `systemd` config files, the logging service (it may not be started on
|
||||
a new VPS), and caddy
|
||||
|
||||
@@ -20,13 +20,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
|
||||
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
|
||||
|
||||
# Fetch archive
|
||||
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
Write-Output "Downloading caddy from $Env:WEBI_PKG_URL to $pkg_download"
|
||||
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
|
||||
& Move-Item "$pkg_download.part" "$pkg_download"
|
||||
}
|
||||
|
||||
if (!(Test-Path -Path "$pkg_src_cmd")) {
|
||||
IF (!(Test-Path -Path "$pkg_src_cmd")) {
|
||||
Write-Output "Installing caddy"
|
||||
|
||||
# TODO: create package-specific temp directory
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
github_releases = caddyserver/caddy
|
||||
27
caddy/releases.js
Normal file
27
caddy/releases.js
Normal file
@@ -0,0 +1,27 @@
|
||||
'use strict';
|
||||
|
||||
var github = require('../_common/github.js');
|
||||
var owner = 'caddyserver';
|
||||
var repo = 'caddy';
|
||||
|
||||
module.exports = function (request) {
|
||||
return github(request, owner, repo).then(function (all) {
|
||||
// remove checksums and .deb
|
||||
all.releases = all.releases.filter(function (rel) {
|
||||
let isOneOffAsset = rel.download.includes('buildable-artifact');
|
||||
if (isOneOffAsset) {
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
});
|
||||
return all;
|
||||
});
|
||||
};
|
||||
|
||||
if (module === require.main) {
|
||||
module.exports(require('@root/request')).then(function (all) {
|
||||
all = require('../_webi/normalize.js')(all);
|
||||
console.info(JSON.stringify(all));
|
||||
});
|
||||
}
|
||||
@@ -19,13 +19,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
|
||||
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
|
||||
|
||||
# Fetch archive
|
||||
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
Write-Output "Downloading chromedriver from $Env:WEBI_PKG_URL to $pkg_download"
|
||||
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
|
||||
& Move-Item "$pkg_download.part" "$pkg_download"
|
||||
}
|
||||
|
||||
if (!(Test-Path -Path "$pkg_src_cmd")) {
|
||||
IF (!(Test-Path -Path "$pkg_src_cmd")) {
|
||||
Write-Output "Installing chromedriver"
|
||||
|
||||
# TODO: create package-specific temp directory
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
source = chromedist
|
||||
84
chromedriver/releases.js
Normal file
84
chromedriver/releases.js
Normal file
@@ -0,0 +1,84 @@
|
||||
'use strict';
|
||||
|
||||
// See <https://googlechromelabs.github.io/chrome-for-testing/>
|
||||
var releaseApiUrl =
|
||||
'https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json';
|
||||
|
||||
// {
|
||||
// "timestamp": "2023-11-15T21:08:56.730Z",
|
||||
// "versions": [
|
||||
// {
|
||||
// "version": "121.0.6120.0",
|
||||
// "revision": "1222902",
|
||||
// "downloads": {
|
||||
// "chrome": [],
|
||||
// "chromedriver": [
|
||||
// {
|
||||
// "platform": "linux64",
|
||||
// "url": "https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/121.0.6120.0/linux64/chromedriver-linux64.zip"
|
||||
// },
|
||||
// {
|
||||
// "platform": "mac-arm64",
|
||||
// "url": "https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/121.0.6120.0/mac-arm64/chromedriver-mac-arm64.zip"
|
||||
// },
|
||||
// {
|
||||
// "platform": "mac-x64",
|
||||
// "url": "https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/121.0.6120.0/mac-x64/chromedriver-mac-x64.zip"
|
||||
// },
|
||||
// {
|
||||
// "platform": "win32",
|
||||
// "url": "https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/121.0.6120.0/win32/chromedriver-win32.zip"
|
||||
// },
|
||||
// {
|
||||
// "platform": "win64",
|
||||
// "url": "https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/121.0.6120.0/win64/chromedriver-win64.zip"
|
||||
// }
|
||||
// ],
|
||||
// "chrome-headless-shell": []
|
||||
// }
|
||||
// }
|
||||
// ]
|
||||
// }
|
||||
|
||||
module.exports = async function (request) {
|
||||
let resp = await request({
|
||||
url: releaseApiUrl,
|
||||
json: true,
|
||||
});
|
||||
|
||||
let builds = [];
|
||||
for (let release of resp.body.versions) {
|
||||
if (!release.downloads.chromedriver) {
|
||||
continue;
|
||||
}
|
||||
|
||||
let version = release.version;
|
||||
for (let asset of release.downloads.chromedriver) {
|
||||
let build = {
|
||||
version: version,
|
||||
download: asset.url,
|
||||
// I' not sure that this is actually statically built but it
|
||||
// seems to be and at worst we'll just get bug reports for Apline
|
||||
libc: 'none',
|
||||
};
|
||||
|
||||
builds.push(build);
|
||||
}
|
||||
}
|
||||
|
||||
let all = {
|
||||
download: '',
|
||||
releases: builds,
|
||||
};
|
||||
|
||||
return all;
|
||||
};
|
||||
|
||||
if (module === require.main) {
|
||||
module.exports(require('@root/request')).then(function (all) {
|
||||
all = require('../_webi/normalize.js')(all);
|
||||
// just select the latest 5 for demonstration
|
||||
all.releases = all.releases.slice(-20);
|
||||
console.info(JSON.stringify(all, null, 2));
|
||||
});
|
||||
}
|
||||
@@ -1,47 +0,0 @@
|
||||
---
|
||||
title: cilium
|
||||
homepage: https://github.com/cilium/cilium-cli
|
||||
tagline: |
|
||||
cilium: manage & troubleshoot Kubernetes clusters running Cilium
|
||||
---
|
||||
|
||||
To update or switch versions, run `webi cilium@stable` (or `@v2`, `@beta`,etc).
|
||||
|
||||
### Files
|
||||
|
||||
These are the files / directories that are created and/or modified with this
|
||||
install:
|
||||
|
||||
```text
|
||||
~/.config/envman/PATH.env
|
||||
~/.local/bin/cilium
|
||||
~/.local/opt/cilium/
|
||||
```
|
||||
|
||||
## Cheat Sheet
|
||||
|
||||
> Cilium is an open source, cloud native solution for providing, securing, and
|
||||
> observing network connectivity between workloads, fueled by the revolutionary
|
||||
> Kernel technology eBPF.
|
||||
|
||||
Quick Start User Guide:
|
||||
|
||||
<https://docs.cilium.io/en/stable/gettingstarted/k8s-install-default/#k8s-install-quick>
|
||||
|
||||
To install the default version of the Cilium image:
|
||||
|
||||
```sh
|
||||
cilium install
|
||||
```
|
||||
|
||||
To upgrade to a specific version of the Cilium image:
|
||||
|
||||
```sh
|
||||
cilium upgrade --version v1.15.3
|
||||
```
|
||||
|
||||
To check the status of the current Cilium deployment:
|
||||
|
||||
```sh
|
||||
cilium status
|
||||
```
|
||||
@@ -1,45 +0,0 @@
|
||||
#!/usr/bin/env pwsh
|
||||
|
||||
##################
|
||||
# Install cilium #
|
||||
##################
|
||||
|
||||
$pkg_cmd_name = "cilium"
|
||||
|
||||
$pkg_dst_cmd = "$Env:USERPROFILE\.local\bin\cilium.exe"
|
||||
$pkg_dst = "$pkg_dst_cmd"
|
||||
|
||||
$pkg_src_cmd = "$Env:USERPROFILE\.local\opt\cilium-v$Env:WEBI_VERSION\bin\cilium.exe"
|
||||
$pkg_src_bin = "$Env:USERPROFILE\.local\opt\cilium-v$Env:WEBI_VERSION\bin"
|
||||
$pkg_src_dir = "$Env:USERPROFILE\.local\opt\cilium-v$Env:WEBI_VERSION"
|
||||
$pkg_src = "$pkg_src_cmd"
|
||||
|
||||
New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
|
||||
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
|
||||
|
||||
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
Write-Output "Downloading cilium from $Env:WEBI_PKG_URL to $pkg_download"
|
||||
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
|
||||
& Move-Item "$pkg_download.part" "$pkg_download"
|
||||
}
|
||||
|
||||
if (!(Test-Path -Path "$pkg_src_cmd")) {
|
||||
Write-Output "Installing cilium"
|
||||
Push-Location .local\tmp
|
||||
|
||||
Remove-Item -Path ".\cilium-v*" -Recurse -ErrorAction Ignore
|
||||
Remove-Item -Path ".\cilium.exe" -Recurse -ErrorAction Ignore
|
||||
|
||||
Write-Output "Unpacking $pkg_download"
|
||||
& tar xf "$pkg_download"
|
||||
|
||||
Write-Output "Install Location: $pkg_src_cmd"
|
||||
New-Item "$pkg_src_bin" -ItemType Directory -Force
|
||||
Move-Item -Path ".\cilium-*\cilium.exe" -Destination "$pkg_src_bin"
|
||||
|
||||
Pop-Location
|
||||
}
|
||||
|
||||
Write-Output "Copying into '$pkg_dst_cmd' from '$pkg_src_cmd'"
|
||||
Remove-Item -Path "$pkg_dst_cmd" -Recurse -ErrorAction Ignore
|
||||
Copy-Item -Path "$pkg_src" -Destination "$pkg_dst" -Recurse
|
||||
@@ -1,39 +0,0 @@
|
||||
#!/bin/sh
|
||||
|
||||
__init_cilium() {
|
||||
set -e
|
||||
set -u
|
||||
|
||||
##################
|
||||
# Install cilium #
|
||||
##################
|
||||
|
||||
pkg_cmd_name="cilium"
|
||||
|
||||
pkg_dst_cmd="$HOME/.local/bin/cilium"
|
||||
pkg_dst="$pkg_dst_cmd"
|
||||
|
||||
pkg_src_cmd="$HOME/.local/opt/cilium-v$WEBI_VERSION/bin/cilium"
|
||||
pkg_src_dir="$HOME/.local/opt/cilium-v$WEBI_VERSION"
|
||||
pkg_src="$pkg_src_cmd"
|
||||
|
||||
WEBI_SINGLE=true
|
||||
|
||||
# pkg_install must be defined by every package
|
||||
pkg_install() {
|
||||
# ~/.local/opt/cilium-v0.16.16/bin
|
||||
mkdir -p "$(dirname "${pkg_src_cmd}")"
|
||||
|
||||
# mv ./hugo ~/.local/opt/cilium-v0.16.16/bin/
|
||||
mv ./cilium "${pkg_src_cmd}"
|
||||
}
|
||||
|
||||
pkg_get_current_version() {
|
||||
cilium version 2> /dev/null |
|
||||
head -n 1 |
|
||||
cut -d ' ' -f 2
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
__init_cilium
|
||||
@@ -1 +0,0 @@
|
||||
github_releases = cilium/cilium-cli
|
||||
@@ -20,13 +20,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
|
||||
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
|
||||
|
||||
# Fetch archive
|
||||
if (!(Test-Path -Path "$pkg_download")) {
|
||||
IF (!(Test-Path -Path "$pkg_download")) {
|
||||
Write-Output "Downloading cmake from $Env:WEBI_PKG_URL to $pkg_download"
|
||||
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
|
||||
& Move-Item "$pkg_download.part" "$pkg_download"
|
||||
}
|
||||
|
||||
if (!(Test-Path -Path "$pkg_src_dir")) {
|
||||
IF (!(Test-Path -Path "$pkg_src_dir")) {
|
||||
Write-Output "Installing cmake"
|
||||
|
||||
# TODO: create package-specific temp directory
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
github_releases = Kitware/CMake
|
||||
53
cmake/releases.js
Normal file
53
cmake/releases.js
Normal file
@@ -0,0 +1,53 @@
|
||||
'use strict';
|
||||
|
||||
var github = require('../_common/github.js');
|
||||
var owner = 'Kitware';
|
||||
var repo = 'CMake';
|
||||
|
||||
module.exports = function (request) {
|
||||
return github(request, owner, repo).then(function (all) {
|
||||
for (let rel of all.releases) {
|
||||
if (rel.version.startsWith('v')) {
|
||||
rel._version = rel.version.slice(1);
|
||||
}
|
||||
|
||||
{
|
||||
let linuxRe = /(\b|_)(linux|gnu)(\b|_)/i;
|
||||
let isLinux = linuxRe.test(rel.download) || linuxRe.test(rel.name);
|
||||
|
||||
if (isLinux) {
|
||||
let muslRe = /(\b|_)(musl|alpine)(\b|_)/i;
|
||||
let isMusl = muslRe.test(rel.download) || muslRe.test(rel.name);
|
||||
if (isMusl) {
|
||||
rel.libc = 'musl';
|
||||
} else {
|
||||
rel.libc = 'gnu';
|
||||
}
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
||||
{
|
||||
let windowsRe = /(\b|_)(win\d*|windows\d*)(\b|_)/i;
|
||||
let isWindows =
|
||||
windowsRe.test(rel.download) || windowsRe.test(rel.name);
|
||||
|
||||
if (isWindows) {
|
||||
rel.libc = 'msvc';
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return all;
|
||||
});
|
||||
};
|
||||
|
||||
if (module === require.main) {
|
||||
module.exports(require('@root/request')).then(function (all) {
|
||||
all = require('../_webi/normalize.js')(all);
|
||||
// just select the first 5 for demonstration
|
||||
all.releases = all.releases.slice(0, 5);
|
||||
console.info(JSON.stringify(all, null, 2));
|
||||
});
|
||||
}
|
||||
@@ -1,925 +0,0 @@
|
||||
// Command webicached is the release cache daemon. It fetches releases
|
||||
// from upstream sources, classifies build assets, and writes them to
|
||||
// the _cache/ directory in the format the Node.js server expects.
|
||||
//
|
||||
// This is the Go replacement for the Node.js release-fetching pipeline.
|
||||
// It reads releases.conf files to discover packages, fetches from the
|
||||
// configured source, classifies assets, and writes to fsstore.
|
||||
//
|
||||
// Default mode: classify all from existing rawcache on startup, then
|
||||
// fetch+refresh one package per tick (round-robin, 15m default).
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// go run ./cmd/webicached # default: round-robin, one per tick
|
||||
// go run ./cmd/webicached -eager # fetch all packages on startup
|
||||
// go run ./cmd/webicached -once -no-fetch # classify from rawcache and exit
|
||||
// go run ./cmd/webicached bat goreleaser # only these packages
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"math/rand/v2"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/joho/godotenv"
|
||||
"github.com/webinstall/webi-installers/internal/classifypkg"
|
||||
"github.com/webinstall/webi-installers/internal/httpclient"
|
||||
"github.com/webinstall/webi-installers/internal/installerconf"
|
||||
"github.com/webinstall/webi-installers/internal/rawcache"
|
||||
"github.com/webinstall/webi-installers/internal/releases/chromedist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/flutterdist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/gitea"
|
||||
"github.com/webinstall/webi-installers/internal/releases/github"
|
||||
"github.com/webinstall/webi-installers/internal/releases/githubish"
|
||||
"github.com/webinstall/webi-installers/internal/releases/gittag"
|
||||
"github.com/webinstall/webi-installers/internal/releases/golang"
|
||||
"github.com/webinstall/webi-installers/internal/releases/gpgdist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/hashicorp"
|
||||
"github.com/webinstall/webi-installers/internal/releases/iterm2dist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/juliadist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/mariadbdist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/nodedist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/servicemandist"
|
||||
"github.com/webinstall/webi-installers/internal/releases/zigdist"
|
||||
"github.com/webinstall/webi-installers/internal/storage"
|
||||
"github.com/webinstall/webi-installers/internal/storage/fsstore"
|
||||
)
|
||||
|
||||
var (
|
||||
name = "webicached"
|
||||
version = "0.0.0-dev"
|
||||
commit = "0000000"
|
||||
date = "0001-01-01"
|
||||
licenseYear = "2024"
|
||||
licenseOwner = "AJ ONeal"
|
||||
licenseType = "MPL-2.0"
|
||||
)
|
||||
|
||||
func printVersion(w io.Writer) {
|
||||
b_ver := strings.TrimPrefix(version, "v")
|
||||
_, _ = fmt.Fprintf(w, "%s v%s %s (%s)\n", name, b_ver, commit[:7], date)
|
||||
_, _ = fmt.Fprintf(w, "Copyright (C) %s %s\n", licenseYear, licenseOwner)
|
||||
_, _ = fmt.Fprintf(w, "Licensed under %s\n", licenseType)
|
||||
}
|
||||
|
||||
type MainConfig struct {
|
||||
envFile string
|
||||
confDir string
|
||||
cacheDir string
|
||||
rawDir string
|
||||
token string
|
||||
once bool
|
||||
noFetch bool
|
||||
shallow bool
|
||||
eager bool
|
||||
interval time.Duration
|
||||
pageDelay time.Duration
|
||||
}
|
||||
|
||||
// WebiCache holds the configuration for the cache daemon.
|
||||
type WebiCache struct {
|
||||
ConfDir string // root directory with {pkg}/releases.conf files
|
||||
Store storage.Store // classified asset storage (fsstore)
|
||||
RawDir string // raw upstream response cache
|
||||
Client *http.Client // HTTP client for upstream calls
|
||||
Auth *githubish.Auth // GitHub API auth (optional)
|
||||
Shallow bool // fetch only the first page of releases
|
||||
NoFetch bool // skip fetching, classify from existing raw data only
|
||||
PageDelay time.Duration // delay between paginated API requests
|
||||
}
|
||||
|
||||
// delayTransport wraps an http.RoundTripper to add a delay between requests.
|
||||
type delayTransport struct {
|
||||
base http.RoundTripper
|
||||
delay time.Duration
|
||||
last time.Time
|
||||
}
|
||||
|
||||
func (t *delayTransport) RoundTrip(req *http.Request) (*http.Response, error) {
|
||||
if !t.last.IsZero() && t.delay > 0 {
|
||||
if wait := t.delay - time.Since(t.last); wait > 0 {
|
||||
time.Sleep(wait)
|
||||
}
|
||||
}
|
||||
t.last = time.Now()
|
||||
return t.base.RoundTrip(req)
|
||||
}
|
||||
|
||||
func main() {
|
||||
if len(os.Args) > 1 {
|
||||
switch os.Args[1] {
|
||||
case "-V", "-version", "--version", "version":
|
||||
printVersion(os.Stdout)
|
||||
os.Exit(0)
|
||||
case "help", "-help", "--help":
|
||||
printVersion(os.Stdout)
|
||||
fmt.Fprintln(os.Stdout, "")
|
||||
fs := flag.NewFlagSet(os.Args[0], flag.ContinueOnError)
|
||||
fs.SetOutput(os.Stdout)
|
||||
registerFlags(fs, &MainConfig{})
|
||||
fs.Usage()
|
||||
os.Exit(0)
|
||||
}
|
||||
}
|
||||
|
||||
cfg := MainConfig{}
|
||||
fs := flag.NewFlagSet(os.Args[0], flag.ContinueOnError)
|
||||
registerFlags(fs, &cfg)
|
||||
if err := fs.Parse(os.Args[1:]); err != nil {
|
||||
if errors.Is(err, flag.ErrHelp) {
|
||||
os.Exit(0)
|
||||
}
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
cfg.cacheDir = expandHome(cfg.cacheDir)
|
||||
cfg.rawDir = expandHome(cfg.rawDir)
|
||||
|
||||
if cfg.envFile != "" {
|
||||
if err := godotenv.Load(cfg.envFile); err != nil {
|
||||
log.Fatalf("envfile: %v", err)
|
||||
}
|
||||
}
|
||||
if cfg.token == "" {
|
||||
cfg.token = os.Getenv("GITHUB_TOKEN")
|
||||
}
|
||||
|
||||
fss, err := fsstore.New(cfg.cacheDir)
|
||||
if err != nil {
|
||||
log.Fatalf("fsstore: %v", err)
|
||||
}
|
||||
var store storage.Store = fss
|
||||
|
||||
var auth *githubish.Auth
|
||||
if cfg.token != "" {
|
||||
auth = &githubish.Auth{Token: cfg.token}
|
||||
}
|
||||
|
||||
client := httpclient.New()
|
||||
if cfg.pageDelay > 0 {
|
||||
client.Transport = &delayTransport{
|
||||
base: client.Transport,
|
||||
delay: cfg.pageDelay,
|
||||
}
|
||||
}
|
||||
|
||||
wc := &WebiCache{
|
||||
ConfDir: cfg.confDir,
|
||||
Store: store,
|
||||
RawDir: cfg.rawDir,
|
||||
Client: client,
|
||||
Auth: auth,
|
||||
Shallow: cfg.shallow,
|
||||
NoFetch: cfg.noFetch,
|
||||
PageDelay: cfg.pageDelay,
|
||||
}
|
||||
|
||||
filterPkgs := fs.Args()
|
||||
|
||||
if cfg.eager {
|
||||
wc.Run(filterPkgs)
|
||||
if cfg.once {
|
||||
return
|
||||
}
|
||||
} else if cfg.once {
|
||||
wc.Run(filterPkgs)
|
||||
return
|
||||
} else {
|
||||
saved := wc.NoFetch
|
||||
wc.NoFetch = true
|
||||
wc.Run(filterPkgs)
|
||||
wc.NoFetch = saved
|
||||
}
|
||||
|
||||
packages, err := discover(wc.ConfDir)
|
||||
if err != nil {
|
||||
log.Fatalf("discover: %v", err)
|
||||
}
|
||||
nameSet := make(map[string]bool, len(filterPkgs))
|
||||
for _, a := range filterPkgs {
|
||||
nameSet[a] = true
|
||||
}
|
||||
if len(filterPkgs) > 0 {
|
||||
var filtered []pkgConf
|
||||
for _, p := range packages {
|
||||
if nameSet[p.name] {
|
||||
filtered = append(filtered, p)
|
||||
}
|
||||
}
|
||||
packages = filtered
|
||||
}
|
||||
|
||||
var real []pkgConf
|
||||
for _, pkg := range packages {
|
||||
if pkg.conf.AliasOf == "" {
|
||||
real = append(real, pkg)
|
||||
}
|
||||
}
|
||||
|
||||
// rescanNew appends any conf files added since the last scan.
|
||||
// Returns true when at least one new package was added so the caller
|
||||
// can restart the batch loop and process new packages immediately.
|
||||
rescanNew := func() bool {
|
||||
discovered, err := discover(wc.ConfDir)
|
||||
if err != nil {
|
||||
log.Printf("rescan: %v", err)
|
||||
return false
|
||||
}
|
||||
known := make(map[string]bool, len(real))
|
||||
for _, p := range real {
|
||||
known[p.name] = true
|
||||
}
|
||||
added := false
|
||||
for _, p := range discovered {
|
||||
if p.conf.AliasOf != "" || known[p.name] {
|
||||
continue
|
||||
}
|
||||
if len(filterPkgs) > 0 && !nameSet[p.name] {
|
||||
continue
|
||||
}
|
||||
log.Printf("discovered new package: %s (source=%s)", p.name, p.conf.Source)
|
||||
real = append(real, p)
|
||||
added = true
|
||||
}
|
||||
return added
|
||||
}
|
||||
|
||||
log.Printf("refreshing %d packages, interval %s, batch size 20 (ctrl-c to stop)", len(real), cfg.interval)
|
||||
for {
|
||||
// Rescan before computing staleness so newly added conf files are
|
||||
// included immediately. New packages have a zero timestamp and sort
|
||||
// to the front of the stale list, so they are processed next.
|
||||
rescanNew()
|
||||
|
||||
stale := wc.stalest(real)
|
||||
if len(stale) == 0 {
|
||||
log.Printf("all packages fresh, sleeping %s", cfg.interval)
|
||||
time.Sleep(cfg.interval)
|
||||
continue
|
||||
}
|
||||
|
||||
batch := stale
|
||||
if len(batch) > 20 {
|
||||
batch = batch[:20]
|
||||
}
|
||||
rand.Shuffle(len(batch), func(i, j int) {
|
||||
batch[i], batch[j] = batch[j], batch[i]
|
||||
})
|
||||
|
||||
log.Printf("batch: %d stale, refreshing %d (most stale first)", len(stale), len(batch))
|
||||
for _, pkg := range batch {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute)
|
||||
if err := wc.refreshPackage(ctx, pkg); err != nil {
|
||||
log.Printf(" ERROR %s: %v", pkg.name, err)
|
||||
}
|
||||
cancel()
|
||||
time.Sleep(cfg.interval)
|
||||
// Rescan mid-batch so new packages preempt remaining batch items.
|
||||
if rescanNew() {
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func registerFlags(fs *flag.FlagSet, cfg *MainConfig) {
|
||||
fs.StringVar(&cfg.envFile, "envfile", "", "path to .env file to load before running")
|
||||
fs.StringVar(&cfg.confDir, "conf", ".", "root directory containing {pkg}/releases.conf files")
|
||||
fs.StringVar(&cfg.cacheDir, "legacy", "~/.cache/webi/legacy", "legacy cache directory (fsstore root)")
|
||||
fs.StringVar(&cfg.rawDir, "raw", "~/.cache/webi/raw", "raw cache directory for upstream responses")
|
||||
fs.StringVar(&cfg.token, "token", "", "GitHub API token (or set $GITHUB_TOKEN)")
|
||||
fs.BoolVar(&cfg.once, "once", false, "run once then exit (no periodic refresh)")
|
||||
fs.BoolVar(&cfg.noFetch, "no-fetch", false, "skip fetching, classify from existing raw data only")
|
||||
fs.BoolVar(&cfg.shallow, "shallow", false, "fetch only the first page of releases (latest)")
|
||||
fs.BoolVar(&cfg.eager, "eager", false, "fetch all packages on startup (default: one per tick)")
|
||||
fs.DurationVar(&cfg.interval, "interval", 9*time.Second, "delay between individual package fetches")
|
||||
fs.DurationVar(&cfg.pageDelay, "page-delay", 2*time.Second, "delay between paginated API requests")
|
||||
}
|
||||
|
||||
func expandHome(path string) string {
|
||||
if !strings.HasPrefix(path, "~/") {
|
||||
return path
|
||||
}
|
||||
home, err := os.UserHomeDir()
|
||||
if err != nil {
|
||||
return path
|
||||
}
|
||||
return filepath.Join(home, path[2:])
|
||||
}
|
||||
|
||||
// stalest returns packages sorted by most stale first (oldest UpdatedAt).
|
||||
// Packages with no cache entry or empty assets are considered most stale.
|
||||
func (wc *WebiCache) stalest(packages []pkgConf) []pkgConf {
|
||||
type stamped struct {
|
||||
pkg pkgConf
|
||||
updatedAt time.Time
|
||||
}
|
||||
|
||||
var stale []stamped
|
||||
ctx := context.Background()
|
||||
for _, pkg := range packages {
|
||||
data, err := wc.Store.Load(ctx, pkg.name)
|
||||
var t time.Time
|
||||
hasAssets := false
|
||||
if err == nil && data != nil {
|
||||
t = data.UpdatedAt
|
||||
hasAssets = len(data.Assets) > 0
|
||||
}
|
||||
// Never fetched, or has no assets despite having a timestamp
|
||||
// (e.g. classified from empty rawcache), or older than 10 minutes.
|
||||
if t.IsZero() || !hasAssets || time.Since(t) > 10*time.Minute {
|
||||
stale = append(stale, stamped{pkg: pkg, updatedAt: t})
|
||||
}
|
||||
}
|
||||
|
||||
sort.SliceStable(stale, func(i, j int) bool {
|
||||
ti, tj := stale[i].updatedAt, stale[j].updatedAt
|
||||
if ti.Equal(tj) {
|
||||
return stale[i].pkg.name < stale[j].pkg.name
|
||||
}
|
||||
return ti.Before(tj)
|
||||
})
|
||||
|
||||
result := make([]pkgConf, len(stale))
|
||||
for i, s := range stale {
|
||||
result[i] = s.pkg
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// Run discovers packages and refreshes each one.
|
||||
func (wc *WebiCache) Run(filterPkgs []string) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
|
||||
defer cancel()
|
||||
|
||||
packages, err := discover(wc.ConfDir)
|
||||
if err != nil {
|
||||
log.Printf("discover: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
if len(filterPkgs) > 0 {
|
||||
nameSet := make(map[string]bool, len(filterPkgs))
|
||||
for _, a := range filterPkgs {
|
||||
nameSet[a] = true
|
||||
}
|
||||
var filtered []pkgConf
|
||||
for _, p := range packages {
|
||||
if nameSet[p.name] {
|
||||
filtered = append(filtered, p)
|
||||
}
|
||||
}
|
||||
packages = filtered
|
||||
}
|
||||
|
||||
var real []pkgConf
|
||||
for _, pkg := range packages {
|
||||
if pkg.conf.AliasOf != "" {
|
||||
continue
|
||||
}
|
||||
real = append(real, pkg)
|
||||
}
|
||||
|
||||
log.Printf("refreshing %d packages", len(real))
|
||||
runStart := time.Now()
|
||||
|
||||
for _, pkg := range real {
|
||||
if err := wc.refreshPackage(ctx, pkg); err != nil {
|
||||
log.Printf(" ERROR %s: %v", pkg.name, err)
|
||||
}
|
||||
}
|
||||
|
||||
log.Printf("refreshed %d packages in %s", len(real), time.Since(runStart))
|
||||
}
|
||||
|
||||
type pkgConf struct {
|
||||
name string
|
||||
conf *installerconf.Conf
|
||||
}
|
||||
|
||||
func discover(dir string) ([]pkgConf, error) {
|
||||
pattern := filepath.Join(dir, "*", "releases.conf")
|
||||
matches, err := filepath.Glob(pattern)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
var packages []pkgConf
|
||||
for _, path := range matches {
|
||||
pkgDir := filepath.Dir(path)
|
||||
name := filepath.Base(pkgDir)
|
||||
if strings.HasPrefix(name, "_") {
|
||||
continue
|
||||
}
|
||||
|
||||
// If the package directory is a symlink, treat it as an alias
|
||||
// of the symlink target (e.g. rust.vim → vim-rust).
|
||||
fi, err := os.Lstat(filepath.Join(dir, name))
|
||||
if err != nil {
|
||||
log.Printf("warning: %s: %v", name, err)
|
||||
continue
|
||||
}
|
||||
if fi.Mode()&os.ModeSymlink != 0 {
|
||||
target, err := os.Readlink(filepath.Join(dir, name))
|
||||
if err != nil {
|
||||
log.Printf("warning: readlink %s: %v", name, err)
|
||||
continue
|
||||
}
|
||||
packages = append(packages, pkgConf{
|
||||
name: name,
|
||||
conf: &installerconf.Conf{AliasOf: target},
|
||||
})
|
||||
continue
|
||||
}
|
||||
|
||||
conf, err := installerconf.Read(path)
|
||||
if err != nil {
|
||||
log.Printf("warning: %s: %v", path, err)
|
||||
continue
|
||||
}
|
||||
packages = append(packages, pkgConf{name: name, conf: conf})
|
||||
}
|
||||
|
||||
sort.Slice(packages, func(i, j int) bool {
|
||||
return packages[i].name < packages[j].name
|
||||
})
|
||||
return packages, nil
|
||||
}
|
||||
|
||||
// refreshPackage does the full pipeline for one package:
|
||||
// fetch raw → classify → write to fsstore.
|
||||
func (wc *WebiCache) refreshPackage(ctx context.Context, pkg pkgConf) error {
|
||||
pkgStart := time.Now()
|
||||
name := pkg.name
|
||||
conf := pkg.conf
|
||||
|
||||
// Step 1: Fetch raw upstream data to rawcache (unless -no-fetch).
|
||||
if !wc.NoFetch {
|
||||
shallow := wc.Shallow
|
||||
if !shallow {
|
||||
d, err := rawcache.Open(filepath.Join(wc.RawDir, name))
|
||||
if err == nil && d.Populated() {
|
||||
shallow = true
|
||||
}
|
||||
}
|
||||
fetchStart := time.Now()
|
||||
if err := wc.fetchRaw(ctx, pkg, shallow); err != nil {
|
||||
return fmt.Errorf("fetch: %w", err)
|
||||
}
|
||||
log.Printf(" %s: fetch %s", name, time.Since(fetchStart))
|
||||
}
|
||||
|
||||
// Step 2: Classify raw data into assets, tag variants, apply config.
|
||||
classifyStart := time.Now()
|
||||
d, err := rawcache.Open(filepath.Join(wc.RawDir, name))
|
||||
if err != nil {
|
||||
return fmt.Errorf("rawcache open: %w", err)
|
||||
}
|
||||
|
||||
// Open supplementary gittag raw cache if available (for packages with
|
||||
// git_url that use a non-gittag source type like servicemandist).
|
||||
var gitTagDir *rawcache.Dir
|
||||
if conf.GitURL != "" && conf.Source != "gittag" {
|
||||
gd, gdErr := rawcache.Open(filepath.Join(wc.RawDir, "_gittag", name))
|
||||
if gdErr == nil && gd.Populated() {
|
||||
gitTagDir = gd
|
||||
}
|
||||
}
|
||||
|
||||
assets, err := classifypkg.Package(name, conf, d, gitTagDir)
|
||||
if err != nil {
|
||||
return fmt.Errorf("classify: %w", err)
|
||||
}
|
||||
classifyDur := time.Since(classifyStart)
|
||||
|
||||
// Step 3: Write to fsstore.
|
||||
writeStart := time.Now()
|
||||
tx, err := wc.Store.BeginRefresh(ctx, name)
|
||||
if err != nil {
|
||||
return fmt.Errorf("begin refresh: %w", err)
|
||||
}
|
||||
if err := tx.Put(assets); err != nil {
|
||||
tx.Rollback()
|
||||
return fmt.Errorf("put: %w", err)
|
||||
}
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return fmt.Errorf("commit: %w", err)
|
||||
}
|
||||
writeDur := time.Since(writeStart)
|
||||
|
||||
log.Printf(" %s: %d assets (classify %s, write %s, total %s)",
|
||||
name, len(assets), classifyDur, writeDur, time.Since(pkgStart))
|
||||
return nil
|
||||
}
|
||||
|
||||
// --- Fetch raw ---
|
||||
|
||||
func (wc *WebiCache) fetchRaw(ctx context.Context, pkg pkgConf, shallow bool) error {
|
||||
switch pkg.conf.Source {
|
||||
case "github", "githubsource":
|
||||
if err := wc.fetchGitHub(ctx, pkg.name, pkg.conf, shallow); err != nil {
|
||||
return err
|
||||
}
|
||||
case "nodedist":
|
||||
return wc.fetchNodeDist(ctx, pkg.name, pkg.conf)
|
||||
case "gittag":
|
||||
return wc.fetchGitTag(ctx, pkg.name, pkg.conf, shallow)
|
||||
case "gitea":
|
||||
return wc.fetchGitea(ctx, pkg.name, pkg.conf, shallow)
|
||||
case "chromedist":
|
||||
return fetchChromeDist(ctx, wc.Client, wc.RawDir, pkg.name)
|
||||
case "flutterdist":
|
||||
return fetchFlutterDist(ctx, wc.Client, wc.RawDir, pkg.name)
|
||||
case "golang":
|
||||
return fetchGolang(ctx, wc.Client, wc.RawDir, pkg.name)
|
||||
case "gpgdist":
|
||||
return fetchGPGDist(ctx, wc.Client, wc.RawDir, pkg.name)
|
||||
case "hashicorp":
|
||||
return fetchHashiCorp(ctx, wc.Client, wc.RawDir, pkg.name, pkg.conf)
|
||||
case "iterm2dist":
|
||||
return fetchITerm2Dist(ctx, wc.Client, wc.RawDir, pkg.name)
|
||||
case "juliadist":
|
||||
return fetchJuliaDist(ctx, wc.Client, wc.RawDir, pkg.name)
|
||||
case "mariadbdist":
|
||||
return fetchMariaDBDist(ctx, wc.Client, wc.RawDir, pkg.name)
|
||||
case "servicemandist":
|
||||
if err := servicemandist.Fetch(ctx, wc.Client, wc.RawDir, pkg.name, wc.Auth, shallow); err != nil {
|
||||
return err
|
||||
}
|
||||
case "zigdist":
|
||||
return fetchZigDist(ctx, wc.Client, wc.RawDir, pkg.name)
|
||||
default:
|
||||
log.Printf(" %s: source %q not yet supported, skipping", pkg.name, pkg.conf.Source)
|
||||
return nil
|
||||
}
|
||||
|
||||
// For non-gittag sources with a git_url, also clone the repo to get
|
||||
// commit hashes. Git entries are classified from this data in
|
||||
// refreshPackage, not from the main raw cache.
|
||||
if pkg.conf.GitURL != "" && pkg.conf.Source != "gittag" {
|
||||
gitShallow := shallow
|
||||
if !wc.Shallow {
|
||||
gd, gdErr := rawcache.Open(filepath.Join(wc.RawDir, "_gittag", pkg.name))
|
||||
if gdErr == nil && !gd.Populated() {
|
||||
gitShallow = false
|
||||
}
|
||||
}
|
||||
if err := wc.fetchGitTagSupplementary(ctx, pkg.name, pkg.conf.GitURL, gitShallow); err != nil {
|
||||
log.Printf(" %s: supplementary gittag fetch: %v", pkg.name, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// fetchGitTagSupplementary clones a git repo to get commit hashes for
|
||||
// packages that use a non-gittag source type (servicemandist, githubsource)
|
||||
// but also have a git_url for source installs.
|
||||
func (wc *WebiCache) fetchGitTagSupplementary(ctx context.Context, pkgName, gitURL string, shallow bool) error {
|
||||
d, err := rawcache.Open(filepath.Join(wc.RawDir, "_gittag", pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
repoDir := filepath.Join(wc.RawDir, "_repos")
|
||||
os.MkdirAll(repoDir, 0o755)
|
||||
|
||||
for batch, err := range gittag.Fetch(ctx, gitURL, repoDir) {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, entry := range batch {
|
||||
tag := entry.Version
|
||||
if tag == "" {
|
||||
tag = "HEAD-" + entry.CommitHash
|
||||
}
|
||||
data, _ := json.Marshal(entry)
|
||||
d.Merge(tag, data)
|
||||
}
|
||||
if shallow {
|
||||
break
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (wc *WebiCache) fetchGitHub(ctx context.Context, pkgName string, conf *installerconf.Conf, shallow bool) error {
|
||||
owner, repo := conf.Owner, conf.Repo
|
||||
if owner == "" || repo == "" {
|
||||
return fmt.Errorf("missing owner or repo")
|
||||
}
|
||||
|
||||
d, err := rawcache.Open(filepath.Join(wc.RawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
tagPrefix := conf.TagPrefix
|
||||
for batch, err := range github.Fetch(ctx, wc.Client, owner, repo, wc.Auth) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("github %s/%s: %w", owner, repo, err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
if rel.Draft {
|
||||
continue
|
||||
}
|
||||
tag := rel.TagName
|
||||
if tagPrefix != "" && !strings.HasPrefix(tag, tagPrefix) {
|
||||
continue
|
||||
}
|
||||
data, _ := json.Marshal(rel)
|
||||
d.Merge(tag, data)
|
||||
}
|
||||
if shallow {
|
||||
break
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (wc *WebiCache) fetchNodeDist(ctx context.Context, pkgName string, conf *installerconf.Conf) error {
|
||||
baseURL := conf.BaseURL
|
||||
if baseURL == "" {
|
||||
return fmt.Errorf("missing url")
|
||||
}
|
||||
|
||||
d, err := rawcache.Open(filepath.Join(wc.RawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Fetch from primary URL. Tag with "official/" prefix so unofficial
|
||||
// entries for the same version don't overwrite.
|
||||
for batch, err := range nodedist.Fetch(ctx, wc.Client, baseURL) {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, entry := range batch {
|
||||
data, _ := json.Marshal(entry)
|
||||
d.Merge("official/"+entry.Version, data)
|
||||
}
|
||||
}
|
||||
|
||||
// Fetch from unofficial URL if configured (e.g. Node.js unofficial builds
|
||||
// which add musl, riscv64, loong64 targets).
|
||||
if unofficialURL := conf.Extra["unofficial_url"]; unofficialURL != "" {
|
||||
for batch, err := range nodedist.Fetch(ctx, wc.Client, unofficialURL) {
|
||||
if err != nil {
|
||||
log.Printf("warning: %s unofficial fetch: %v", pkgName, err)
|
||||
break
|
||||
}
|
||||
for _, entry := range batch {
|
||||
data, _ := json.Marshal(entry)
|
||||
d.Merge("unofficial/"+entry.Version, data)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func (wc *WebiCache) fetchGitTag(ctx context.Context, pkgName string, conf *installerconf.Conf, shallow bool) error {
|
||||
gitURL := conf.BaseURL
|
||||
if gitURL == "" {
|
||||
return fmt.Errorf("missing url")
|
||||
}
|
||||
|
||||
d, err := rawcache.Open(filepath.Join(wc.RawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
repoDir := filepath.Join(wc.RawDir, "_repos")
|
||||
os.MkdirAll(repoDir, 0o755)
|
||||
|
||||
for batch, err := range gittag.Fetch(ctx, gitURL, repoDir) {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, entry := range batch {
|
||||
tag := entry.Version
|
||||
if tag == "" {
|
||||
tag = "HEAD-" + entry.CommitHash
|
||||
}
|
||||
data, _ := json.Marshal(entry)
|
||||
d.Merge(tag, data)
|
||||
}
|
||||
if shallow {
|
||||
break
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (wc *WebiCache) fetchGitea(ctx context.Context, pkgName string, conf *installerconf.Conf, shallow bool) error {
|
||||
baseURL, owner, repo := conf.BaseURL, conf.Owner, conf.Repo
|
||||
if baseURL == "" || owner == "" || repo == "" {
|
||||
return fmt.Errorf("missing base_url, owner, or repo")
|
||||
}
|
||||
|
||||
d, err := rawcache.Open(filepath.Join(wc.RawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range gitea.Fetch(ctx, wc.Client, baseURL, owner, repo, nil) {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, rel := range batch {
|
||||
if rel.Draft {
|
||||
continue
|
||||
}
|
||||
data, _ := json.Marshal(rel)
|
||||
d.Merge(rel.TagName, data)
|
||||
}
|
||||
if shallow {
|
||||
break
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchChromeDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range chromedist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("chromedist: %w", err)
|
||||
}
|
||||
for _, ver := range batch {
|
||||
data, _ := json.Marshal(ver)
|
||||
d.Merge(ver.Version, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchFlutterDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range flutterdist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("flutterdist: %w", err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
// Key by version+channel+os for uniqueness.
|
||||
key := rel.Version + "-" + rel.Channel + "-" + rel.OS
|
||||
data, _ := json.Marshal(rel)
|
||||
d.Merge(key, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchGolang(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range golang.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("golang: %w", err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
data, _ := json.Marshal(rel)
|
||||
d.Merge(rel.Version, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchGPGDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range gpgdist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("gpgdist: %w", err)
|
||||
}
|
||||
for _, entry := range batch {
|
||||
data, _ := json.Marshal(entry)
|
||||
d.Merge(entry.Version, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchHashiCorp(ctx context.Context, client *http.Client, rawDir, pkgName string, conf *installerconf.Conf) error {
|
||||
product := conf.Repo
|
||||
if product == "" {
|
||||
product = pkgName
|
||||
}
|
||||
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for idx, err := range hashicorp.Fetch(ctx, client, product) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("hashicorp %s: %w", product, err)
|
||||
}
|
||||
for ver, vdata := range idx.Versions {
|
||||
data, _ := json.Marshal(vdata)
|
||||
d.Merge(ver, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchITerm2Dist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range iterm2dist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("iterm2dist: %w", err)
|
||||
}
|
||||
for _, entry := range batch {
|
||||
key := entry.Version
|
||||
if entry.Channel == "beta" {
|
||||
key += "-beta"
|
||||
}
|
||||
data, _ := json.Marshal(entry)
|
||||
d.Merge(key, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchJuliaDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range juliadist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("juliadist: %w", err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
data, _ := json.Marshal(rel)
|
||||
d.Merge(rel.Version, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchMariaDBDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range mariadbdist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("mariadbdist: %w", err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
data, _ := json.Marshal(rel)
|
||||
d.Merge(rel.ReleaseID, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fetchZigDist(ctx context.Context, client *http.Client, rawDir, pkgName string) error {
|
||||
d, err := rawcache.Open(filepath.Join(rawDir, pkgName))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
for batch, err := range zigdist.Fetch(ctx, client) {
|
||||
if err != nil {
|
||||
return fmt.Errorf("zigdist: %w", err)
|
||||
}
|
||||
for _, rel := range batch {
|
||||
data, _ := json.Marshal(rel)
|
||||
d.Merge(rel.Version, data)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -1,55 +0,0 @@
|
||||
---
|
||||
title: XCode Command Line Tools
|
||||
homepage: https://webinstall.dev/commandlinetools
|
||||
tagline: |
|
||||
The XCode Command Line Tools include git, swift, make, clang, and other developer tools
|
||||
---
|
||||
|
||||
## Cheat Sheet
|
||||
|
||||
> The developer tools provided by Apple for macOS.
|
||||
|
||||
- git
|
||||
- swift
|
||||
- make
|
||||
- clang
|
||||
- etc
|
||||
|
||||
This is also part of [webi-essentials](../webi-essentials/).
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- Files
|
||||
- Manual Install
|
||||
- macOS
|
||||
- Linux
|
||||
- Alpine
|
||||
- Windows
|
||||
|
||||
### Files
|
||||
|
||||
These are the files / directories that are created and/or modified with this
|
||||
install:
|
||||
|
||||
```sh
|
||||
/Library/Developer/CommandLineTools/
|
||||
```
|
||||
|
||||
### How to Install Manually
|
||||
|
||||
It's very easy to start the installer:
|
||||
|
||||
```sh
|
||||
xcode-select --install
|
||||
```
|
||||
|
||||
The trick is to also have a mechanism to know when it has finished:
|
||||
|
||||
```sh
|
||||
while ! test -x /Library/Developer/CommandLineTools/usr/bin/git ||
|
||||
! test -x /Library/Developer/CommandLineTools/usr/bin/make; do
|
||||
sleep 0.25
|
||||
done
|
||||
|
||||
echo "Command Line Tools Installed"
|
||||
```
|
||||
@@ -1,62 +0,0 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
set -u
|
||||
|
||||
fn_install_xcode_commandlinetools() { (
|
||||
b_os="$(uname -s)"
|
||||
if test "${b_os}" != 'Darwin'; then
|
||||
echo >&2 'XCode Command Line Tools are for macOS only'
|
||||
return 1
|
||||
fi
|
||||
|
||||
# streamline the output to be pretty
|
||||
fn_check_pkg '/Library/Developer/CommandLineTools/usr/bin/clang' 'clang'
|
||||
fn_check_pkg '/Library/Developer/CommandLineTools/usr/bin/git' 'git'
|
||||
fn_check_pkg '/Library/Developer/CommandLineTools/usr/bin/make' 'make'
|
||||
echo >&2 ""
|
||||
|
||||
# git
|
||||
if xcode-select -p > /dev/null 2> /dev/null; then
|
||||
echo ""
|
||||
return 0
|
||||
fi
|
||||
|
||||
cmd_xcode_cli_tools_install="xcode-select --install"
|
||||
echo " Running $(t_cmd "${cmd_xcode_cli_tools_install}")"
|
||||
$cmd_xcode_cli_tools_install 2> /dev/null
|
||||
echo ""
|
||||
echo ">>> $(t_attn 'ACTION REQUIRED') <<<"
|
||||
echo ""
|
||||
echo " $(t_attn "Click") '$(t_bold 'Install')' $(t_attn "in the pop-up")"
|
||||
echo " (it may appear $(t_em 'under') this window)"
|
||||
echo ""
|
||||
echo "^^^ $(t_attn 'ACTION REQUIRED') ^^^"
|
||||
echo ""
|
||||
printf " waiting %s to finish installing Command Line Developer Tools ..." "$(t_em 'for you')"
|
||||
while ! test -x /Library/Developer/CommandLineTools/usr/bin/git ||
|
||||
! test -x /Library/Developer/CommandLineTools/usr/bin/make; do
|
||||
sleep 0.25
|
||||
done
|
||||
echo " $(t_info 'OK')"
|
||||
echo " Installed to $(t_path '/Library/Developer/CommandLineTools/')"
|
||||
sleep 1
|
||||
); }
|
||||
|
||||
fn_check_pkg() { (
|
||||
a_pkg="${1}"
|
||||
a_pkgname="${2:-$a_pkg}"
|
||||
|
||||
printf >&2 ' %s %s %s' \
|
||||
"$(t_dim "Checking for")" \
|
||||
"$(t_pkg "${a_pkgname}")" \
|
||||
"$(t_dim "...")"
|
||||
|
||||
if command -v "${a_pkg}" > /dev/null; then
|
||||
echo >&2 " $(t_dim 'OK')"
|
||||
return 0
|
||||
fi
|
||||
|
||||
echo >&2 ' missing'
|
||||
); }
|
||||
|
||||
fn_install_xcode_commandlinetools
|
||||
@@ -3,13 +3,13 @@
|
||||
$VERNAME = "$Env:PKG_NAME-v$Env:WEBI_VERSION.exe"
|
||||
$EXENAME = "$Env:PKG_NAME.exe"
|
||||
# Fetch archive
|
||||
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
Write-Output "Downloading $Env:PKG_NAME from $Env:WEBI_PKG_URL to $Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
|
||||
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part"
|
||||
& Move-Item "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part" "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
|
||||
}
|
||||
|
||||
if (!(Test-Path -Path "$Env:USERPROFILE\.local\opt\$Env:PKG_NAME-v$Env:WEBI_VERSION\bin\$VERNAME")) {
|
||||
IF (!(Test-Path -Path "$Env:USERPROFILE\.local\opt\$Env:PKG_NAME-v$Env:WEBI_VERSION\bin\$VERNAME")) {
|
||||
Write-Output "Installing $Env:PKG_NAME"
|
||||
# TODO: temp directory
|
||||
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
github_releases = kivikakk/comrak
|
||||
40
comrak/releases.js
Normal file
40
comrak/releases.js
Normal file
@@ -0,0 +1,40 @@
|
||||
'use strict';
|
||||
|
||||
var github = require('../_common/github.js');
|
||||
var owner = 'kivikakk';
|
||||
var repo = 'comrak';
|
||||
|
||||
var ODDITIES = ['-musleabihf.1-'];
|
||||
|
||||
module.exports = function (request) {
|
||||
return github(request, owner, repo).then(function (all) {
|
||||
let builds = [];
|
||||
|
||||
loopBuilds: for (let build of all.releases) {
|
||||
let isOddity;
|
||||
for (let oddity of ODDITIES) {
|
||||
isOddity = build.name.includes(oddity);
|
||||
if (isOddity) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (isOddity) {
|
||||
continue;
|
||||
}
|
||||
|
||||
builds.push(build);
|
||||
}
|
||||
|
||||
all.releases = builds;
|
||||
return all;
|
||||
});
|
||||
};
|
||||
|
||||
if (module === require.main) {
|
||||
module.exports(require('@root/request')).then(function (all) {
|
||||
all = require('../_webi/normalize.js')(all);
|
||||
all.releases = all.releases.slice(0, 10);
|
||||
//console.info(JSON.stringify(all));
|
||||
console.info(JSON.stringify(all, null, 2));
|
||||
});
|
||||
}
|
||||
@@ -20,13 +20,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
|
||||
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
|
||||
|
||||
# Fetch archive
|
||||
if (!(Test-Path -Path "$pkg_download")) {
|
||||
IF (!(Test-Path -Path "$pkg_download")) {
|
||||
Write-Output "Downloading crabz from $Env:WEBI_PKG_URL to $pkg_download"
|
||||
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
|
||||
& Move-Item "$pkg_download.part" "$pkg_download"
|
||||
}
|
||||
|
||||
if (!(Test-Path -Path "$pkg_src_cmd")) {
|
||||
IF (!(Test-Path -Path "$pkg_src_cmd")) {
|
||||
Write-Output "Installing crabz"
|
||||
|
||||
# TODO: create package-specific temp directory
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
github_releases = sstadick/crabz
|
||||
31
crabz/releases.js
Normal file
31
crabz/releases.js
Normal file
@@ -0,0 +1,31 @@
|
||||
'use strict';
|
||||
|
||||
var github = require('../_common/github.js');
|
||||
var owner = 'sstadick';
|
||||
var repo = 'crabz';
|
||||
|
||||
module.exports = async function (request) {
|
||||
let all = await github(request, owner, repo);
|
||||
|
||||
let releases = [];
|
||||
for (let rel of all.releases) {
|
||||
let isSrc = rel.download.includes('-src.');
|
||||
if (isSrc) {
|
||||
continue;
|
||||
}
|
||||
|
||||
releases.push(rel);
|
||||
}
|
||||
all.releases = releases;
|
||||
|
||||
return all;
|
||||
};
|
||||
|
||||
if (module === require.main) {
|
||||
module.exports(require('@root/request')).then(function (all) {
|
||||
all = require('../_webi/normalize.js')(all);
|
||||
// just select the first 5 for demonstration
|
||||
all.releases = all.releases.slice(0, 5);
|
||||
console.info(JSON.stringify(all, null, 2));
|
||||
});
|
||||
}
|
||||
@@ -3,13 +3,13 @@
|
||||
$VERNAME = "$Env:PKG_NAME-v$Env:WEBI_VERSION.exe"
|
||||
$EXENAME = "$Env:PKG_NAME.exe"
|
||||
# Fetch archive
|
||||
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
|
||||
Write-Output "Downloading $Env:PKG_NAME from $Env:WEBI_PKG_URL to $Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
|
||||
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part"
|
||||
& Move-Item "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part" "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
|
||||
}
|
||||
|
||||
if (!(Test-Path -Path "$Env:USERPROFILE\.local\bin\$VERNAME")) {
|
||||
IF (!(Test-Path -Path "$Env:USERPROFILE\.local\bin\$VERNAME")) {
|
||||
Write-Output "Installing $Env:PKG_NAME"
|
||||
# TODO: temp directory
|
||||
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
github_releases = rs/curlie
|
||||
21
curlie/releases.js
Normal file
21
curlie/releases.js
Normal file
@@ -0,0 +1,21 @@
|
||||
'use strict';
|
||||
|
||||
var github = require('../_common/github.js');
|
||||
var owner = 'rs';
|
||||
var repo = 'curlie';
|
||||
|
||||
module.exports = function (request) {
|
||||
return github(request, owner, repo).then(function (all) {
|
||||
all._names = ['curlie', 'curl-httpie'];
|
||||
return all;
|
||||
});
|
||||
};
|
||||
|
||||
if (module === require.main) {
|
||||
module.exports(require('@root/request')).then(function (all) {
|
||||
all = require('../_webi/normalize.js')(all);
|
||||
all.releases = all.releases.slice(0, 10);
|
||||
//console.info(JSON.stringify(all));
|
||||
console.info(JSON.stringify(all, null, 2));
|
||||
});
|
||||
}
|
||||
@@ -100,7 +100,8 @@ mkdir -p ~/.dashcore/wallets/
|
||||
mkdir -p /mnt/slc1_vol_100g/dashcore/_data
|
||||
mkdir -p /mnt/slc1_vol_100g/dashcore/_caches
|
||||
|
||||
serviceman add --name 'dashd' --daemon -- \
|
||||
sudo env PATH="$PATH" serviceman add \
|
||||
--system --user "$my_user" --path "$PATH" --name dashd --force -- \
|
||||
dashd \
|
||||
-usehd \
|
||||
-conf="$HOME/.dashcore/dash.conf" \
|
||||
|
||||
@@ -1,14 +1,7 @@
|
||||
# for exposing RPCs, building APIs
|
||||
txindex=1
|
||||
addressindex=1
|
||||
timestampindex=1
|
||||
spentindex=1
|
||||
# listen as server (explicit default)
|
||||
listen=1
|
||||
# because its already run as a service (systemd, openrc)
|
||||
daemon=0
|
||||
# for evonodes
|
||||
#server=1
|
||||
|
||||
[main]
|
||||
rpcuser=RPCUSER_MAIN
|
||||
|
||||
@@ -17,7 +17,7 @@ fn_usage() { (
|
||||
); }
|
||||
|
||||
fn_datadir_help() { (
|
||||
my_vol="${1}"
|
||||
my_vol="${1:-}"
|
||||
my_user="$(
|
||||
id -n -u
|
||||
)"
|
||||
@@ -25,7 +25,6 @@ fn_datadir_help() { (
|
||||
id -n -g
|
||||
)"
|
||||
|
||||
my_mount="$(dirname "${my_vol}")"
|
||||
echo >&2 ""
|
||||
echo >&2 "ERROR"
|
||||
echo >&2 " '${my_vol}' is not writable"
|
||||
@@ -33,8 +32,8 @@ fn_datadir_help() { (
|
||||
echo >&2 "SOLUTION"
|
||||
echo >&2 " 1. Mount a large (50gb+) volume"
|
||||
echo >&2 ""
|
||||
echo >&2 " sudo mkdir -p ${my_mount}"
|
||||
echo >&2 " sudo mount /dev/sdx1 ${my_mount}"
|
||||
echo >&2 " sudo mkdir -p /mnt/EXAMPLE"
|
||||
echo >&2 " sudo mount /dev/sdx1 /mnt/EXAMPLE"
|
||||
echo >&2 ""
|
||||
echo >&2 " 2. Create a 'dashcore' inside of it"
|
||||
echo >&2 ""
|
||||
@@ -84,8 +83,20 @@ fn_srv_install() { (
|
||||
my_name="dashd-${my_netname}"
|
||||
fi
|
||||
|
||||
my_system_args=""
|
||||
my_kernel="$(
|
||||
uname -s
|
||||
)"
|
||||
if test "Darwin" != "${my_kernel}"; then
|
||||
my_user="$(
|
||||
id -u -n
|
||||
)"
|
||||
my_system_args="--system --username ${my_user}"
|
||||
fi
|
||||
|
||||
# shellcheck disable=SC2016,SC1090
|
||||
echo "serviceman add --name \"${my_name}\" --" \
|
||||
echo 'sudo env PATH="$PATH"' \
|
||||
"serviceman add ${my_system_args} --path \"\$PATH\" --name \"${my_name}\" --force --" \
|
||||
"dashd " \
|
||||
"${my_net_flag}" \
|
||||
-usehd \
|
||||
@@ -95,16 +106,16 @@ fn_srv_install() { (
|
||||
"-datadir=\"${my_datadir}\"" \
|
||||
"-blocksdir=\"${my_blocksdir}\""
|
||||
|
||||
echo ""
|
||||
echo "Installing latest 'serviceman'..."
|
||||
echo ""
|
||||
"$HOME/.local/bin/webi" serviceman > /dev/null
|
||||
if ! command -v serviceman > /dev/null; then
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
fi
|
||||
serviceman --version
|
||||
if ! command -v dashd > /dev/null; then
|
||||
export PATH="$HOME/.local/opt/dashcore/bin:$PATH"
|
||||
echo ""
|
||||
echo "Installing 'serviceman'..."
|
||||
echo ""
|
||||
{
|
||||
curl -fsSL "${WEBI_HOST}/serviceman" | sh
|
||||
} > /dev/null
|
||||
|
||||
# shellcheck disable=SC1090
|
||||
. ~/.config/envman/PATH.env || true
|
||||
fi
|
||||
|
||||
mkdir -p "$HOME/.dashcore/wallets/"
|
||||
@@ -119,7 +130,8 @@ fn_srv_install() { (
|
||||
cd "${my_vol}" || return 1
|
||||
# leave options unquoted so they're interpreted separately
|
||||
# shellcheck disable=SC2086
|
||||
serviceman add --name "${my_name}" -- \
|
||||
sudo env PATH="${PATH}" \
|
||||
serviceman add ${my_system_args} --path "${PATH}" --name "${my_name}" --force -- \
|
||||
dashd \
|
||||
${my_net_flag} \
|
||||
-usehd \
|
||||
|
||||
@@ -5,38 +5,32 @@ set -u
|
||||
__install_dashcore_utils() {
|
||||
webi_download \
|
||||
"$WEBI_HOST/packages/dashcore-utils/dash-qt-hd" \
|
||||
"$HOME/.local/bin/dash-qt-hd" \
|
||||
"dash-qt-hd"
|
||||
"$HOME/.local/bin/dash-qt-hd"
|
||||
chmod a+x "$HOME/.local/bin/dash-qt-hd"
|
||||
|
||||
webi_download \
|
||||
"$WEBI_HOST/packages/dashcore-utils/dash-qt-testnet" \
|
||||
"$HOME/.local/bin/dash-qt-testnet" \
|
||||
"dash-qt-testnet"
|
||||
"$HOME/.local/bin/dash-qt-testnet"
|
||||
chmod a+x "$HOME/.local/bin/dash-qt-testnet"
|
||||
|
||||
webi_download \
|
||||
"$WEBI_HOST/packages/dashcore-utils/dashd-hd" \
|
||||
"$HOME/.local/bin/dashd-hd" \
|
||||
"dashd-hd"
|
||||
"$HOME/.local/bin/dashd-hd"
|
||||
chmod a+x "$HOME/.local/bin/dashd-hd"
|
||||
|
||||
webi_download \
|
||||
"$WEBI_HOST/packages/dashcore-utils/dashd-testnet" \
|
||||
"$HOME/.local/bin/dashd-testnet" \
|
||||
"dashd-testnet"
|
||||
"$HOME/.local/bin/dashd-testnet"
|
||||
chmod a+x "$HOME/.local/bin/dashd-testnet"
|
||||
|
||||
webi_download \
|
||||
"$WEBI_HOST/packages/dashcore-utils/dashd-hd-service-install" \
|
||||
"$HOME/.local/bin/dashd-hd-service-install" \
|
||||
"dashd-hd-service-install"
|
||||
"$HOME/.local/bin/dashd-hd-service-install"
|
||||
chmod a+x "$HOME/.local/bin/dashd-hd-service-install"
|
||||
|
||||
webi_download \
|
||||
"$WEBI_HOST/packages/dashcore-utils/dashd-testnet-service-install" \
|
||||
"$HOME/.local/bin/dashd-testnet-service-install" \
|
||||
"dashd-testnet-service-install"
|
||||
"$HOME/.local/bin/dashd-testnet-service-install"
|
||||
chmod a+x "$HOME/.local/bin/dashd-testnet-service-install"
|
||||
|
||||
if ! test -e "${HOME}/.dashcore"; then
|
||||
@@ -50,8 +44,7 @@ __install_dashcore_utils() {
|
||||
|
||||
webi_download \
|
||||
"$WEBI_HOST/packages/dashcore-utils/dash.example.conf" \
|
||||
"$HOME/.dashcore/dash.example.conf" \
|
||||
"dash.example.conf"
|
||||
"$HOME/.dashcore/dash.example.conf"
|
||||
|
||||
if ! grep -q rpcuser ~/.dashcore/dash.conf; then
|
||||
cat ~/.dashcore/dash.example.conf >> ~/.dashcore/dash.conf
|
||||
|
||||
@@ -147,8 +147,8 @@ dash-qt \
|
||||
CoinJoin aids in preventing some bad actors and malicious observers being able
|
||||
to easily reconstruct details about your transactions from the publicly
|
||||
available data by creating many excess transactions. \
|
||||
(be aware, however, that dedicated bad actors can use sophisticated software
|
||||
that will reveal much of the same information over time)
|
||||
(be aware, however, that dedicated bad actors can use sophisticated software that
|
||||
will reveal much of the same information over time)
|
||||
|
||||
`dash-qt` does not enable CoinJoin mixing by default.
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user