Compare commits

..

1 Commits

Author SHA1 Message Date
AJ ONeal
b135d5c48f fix: catch and log floating update promise 2025-02-07 11:33:22 +00:00
260 changed files with 5634 additions and 1610 deletions

View File

@@ -35,18 +35,15 @@ info, and doing a find and replace on a few file system path names.
```
2. Copy the example template and update with info from Official Releases:
<https://github.com/___CHANGE/ME___/releases>
```bash
rsync -av _example/ CHANGE-ME/
```
- [ ] update `CHANGE-ME/release.js` to use the official repo
- [ ] Learn how `CHANGE-ME` unpacks (i.e. as a single file? as a .tar.gz? as
a .tar.gz with a folder named CHANGE-ME?)
- [ ] find and replace to change the name
- [ ] update `CHANGE-ME/install.sh` (see `bat` and `jq` as examples)
- [ ] update `CHANGE-ME/install.ps1` (see `bat` and `jq` as examples)
3. Needs an updated tagline and cheat sheet
- [ ] update `CHANGE-ME/README.md`
- [ ] official URL

372
AGENTS.md
View File

@@ -1,372 +0,0 @@
# Webi Installers — Agent Guide
Webi installs dev tools to `~/.local/` without sudo. Each installer is a small
package of 3-4 files. This guide tells you how to create and modify them.
## Why Webi Exists
Webi makes tool installation trivially repeatable for people who aren't
sysadmins — freelance clients, junior devs, anyone who shouldn't have to care
about PATH, permissions, or platform differences. Three things matter:
1. **Install without friction.** No sudo, no manual PATH edits, no "necessary
but unimportant" steps leaking into the experience.
2. **Know where things are.** The Files section tells you exactly what got
created or modified. Nothing should be mysterious.
3. **Copy-paste recipes.** The cheat sheet is what you'd send someone less
experienced than yourself instead of a project's full README — scannable,
concrete, easy to reference by name.
## Quick Start: Adding a New Installer
1. Identify the **package type** (see [Categories](#categories) below)
2. Find an existing installer of the same type to use as a template
3. Create `<name>/releases.js`, `install.sh`, `install.ps1`, `README.md`
4. Test with the command in [Testing releases.js](#testing-releasesjs)
5. Run formatters before committing (see [Code Style](#code-style))
## Directory Layout
```
<package-name>/
README.md # YAML frontmatter + docs
releases.js # Fetches release metadata (Node.js)
install.sh # POSIX shell installer (macOS/Linux)
install.ps1 # PowerShell installer (Windows) — optional
```
Key infrastructure directories (do not modify without good reason):
- `_webi/` — bootstrap templates, `normalize.js` (auto-detects OS/arch/ext from
filenames)
- `_common/` — shared JS: `github.js`, `githubish.js`, `gitea.js`, `fetcher.js`
- `_example/` — canonical template for new packages
- `_examples/` — specialized templates (goreleaser, xz-compressed)
## Categories
Ref: <https://github.com/webinstall/webi-installers/issues/412>
| Type | Description | Template to copy |
| ----- | -------------------------------------- | ---------------- |
| `bin` | Single binary in tar/zip | `koji`, `delta` |
| `bin` | Single bare binary (no archive) | `arc`, `shfmt` |
| `bin` | Goreleaser-style archives | `keypairs` |
| 📦 | Self-contained package (bin/man/share) | `node`, `go` |
| 📂 | Multiple binaries/scripts | `pg` |
| 🔗 | Alias/redirect to another package | `ripgrep``rg` |
| 📝 | Bespoke / custom install | `rustlang` |
## releases.js
Fetches release metadata and returns a normalized object. Most packages use
GitHub releases:
```js
'use strict';
var github = require('../_common/github.js');
var owner = 'OWNER';
var repo = 'REPO';
let Releases = module.exports;
Releases.latest = async function () {
let all = await github(null, owner, repo);
return all;
};
Releases.sample = async function () {
let normalize = require('../_webi/normalize.js');
let all = await Releases.latest();
all = normalize(all);
all.releases = all.releases.slice(0, 5);
return all;
};
if (module === require.main) {
(async function () {
let samples = await Releases.sample();
console.info(JSON.stringify(samples, null, 2));
})();
}
```
### Common release transformations
**Strip version prefix** (monorepo or tool-prefixed tags):
```js
// e.g. "tools/monorel/v0.6.5" → "v0.6.5"
rel.version = rel.version.replace(/^tools\/monorel\//, '');
// e.g. "cli-v1.2.3" → "v1.2.3"
rel.version = rel.version.replace(/^cli-/, '');
```
**Filter releases** (monorepo with multiple tools, or unwanted assets):
```js
all.releases = all.releases.filter(function (rel) {
// Keep only releases for this tool
return rel.version.startsWith('tools/monorel/');
});
```
Apply transformations inside `Releases.latest`, before returning `all`.
**Available sources** beyond `github.js`:
- `_common/gitea.js` — Gitea servers
- `_common/git-tag.js` — Git tag listing
- Custom fetch from any JSON API (see `go/releases.js`, `terraform/releases.js`)
### Testing releases.js
```sh
node -e "
let Releases = require('./<name>/releases.js');
Releases.sample().then(function (all) {
console.log(JSON.stringify(all, null, 2));
});
"
```
Verify: versions are clean semver (`0.6.5` not `tools/monorel/v0.6.5`), OS/arch
detected correctly, download URLs resolve.
## install.sh
POSIX shell (`sh`, not bash). Always wrapped in a function:
```sh
#!/bin/sh
# shellcheck disable=SC2034
set -e
set -u
__init_pkgname() {
# These 6 variables are required
pkg_cmd_name="cmd"
pkg_dst_cmd="$HOME/.local/bin/cmd"
pkg_dst="$pkg_dst_cmd"
pkg_src_cmd="$HOME/.local/opt/cmd-v$WEBI_VERSION/bin/cmd"
pkg_src_dir="$HOME/.local/opt/cmd-v$WEBI_VERSION"
pkg_src="$pkg_src_cmd"
pkg_install() {
mkdir -p "$(dirname "$pkg_src_cmd")"
mv ./cmd "$pkg_src_cmd"
}
pkg_get_current_version() {
cmd --version 2> /dev/null | head -n 1 | cut -d' ' -f2
}
}
__init_pkgname
```
### Framework variables available in install.sh
Set by the webi bootstrap (`_webi/package-install.tpl.sh`):
| Variable | Example | Description |
| --------------- | ------------------- | --------------------- |
| `WEBI_VERSION` | `1.2.3` | Selected version |
| `WEBI_PKG_URL` | `https://...` | Download URL |
| `WEBI_PKG_FILE` | `foo-v1.2.3.tar.gz` | Download filename |
| `WEBI_OS` | `linux` | Detected OS |
| `WEBI_ARCH` | `amd64` | Detected architecture |
| `WEBI_EXT` | `tar.gz` | Archive extension |
| `WEBI_CHANNEL` | `stable` | Release channel |
| `PKG_NAME` | `foo` | Package name |
### Override functions
| Function | Purpose |
| --------------------------- | --------------------------------------------- |
| `pkg_install()` | **Required.** Move files to `$pkg_src` |
| `pkg_get_current_version()` | Parse installed version from command output |
| `pkg_post_install()` | Post-install setup (git config, shell config) |
| `pkg_done_message()` | Custom completion message |
| `pkg_link()` | Override default symlink behavior |
| `pkg_pre_install()` | Custom pre-install logic |
### Framework helper functions
| Function | Purpose |
| ------------------------ | ---------------------------------- |
| `webi_download()` | Download package if not cached |
| `webi_extract()` | Extract archive by extension |
| `webi_path_add <dir>` | Add to PATH via envman |
| `webi_link()` | Create versioned symlinks |
| `webi_check_installed()` | Check if version already installed |
### pkg_install patterns
**Bare binary in archive root:**
```sh
mv ./cmd "$pkg_src_cmd"
```
**Binary in a subdirectory (goreleaser-style `cmd-OS-arch/cmd`):**
```sh
mv ./cmd-*/cmd "$pkg_src_cmd"
```
**Flexible detection (handles multiple archive layouts):**
```sh
if test -f ./cmd; then
mv ./cmd "$pkg_src_cmd"
elif test -e ./cmd-*/cmd; then
mv ./cmd-*/cmd "$pkg_src_cmd"
elif test -e ./cmd-*; then
mv ./cmd-* "$pkg_src_cmd"
fi
```
## install.ps1
PowerShell for Windows. Uses `$Env:` variables. See `_example/install.ps1` for
the full template. Key differences from install.sh:
- Paths use backslashes, commands end in `.exe`
- `$Env:USERPROFILE` instead of `$HOME`
- `Test-Path`, `Move-Item`, `Copy-Item` instead of shell equivalents
- Downloads go to `$Env:USERPROFILE\Downloads\webi\`
- Temp work in `.local\tmp`, use `Push-Location`/`Pop-Location`
- Symlinks done via `Copy-Item` (not actual symlinks)
## README.md
````markdown
---
title: toolname
homepage: https://github.com/owner/repo
tagline: |
toolname: A short one-line description.
---
To update or switch versions, run `webi toolname@stable` (or `@v2`, `@beta`,
etc).
### Files
These are the files that are created and/or modified with this installer:
```text
~/.config/envman/PATH.env
~/.local/bin/toolname
~/.local/opt/toolname-VERSION/bin/toolname
```
## Cheat Sheet
> `toolname` does X. Brief description.
### How to use toolname
```sh
toolname --example
```
````
Note: **Files goes above Cheat Sheet**, not inside it.
### Cheat Sheet tone and style
Webi cheat sheets are **opinionated quick-reference guides**, not comprehensive
documentation. Think "colleague's sticky note" — not the project's official
README.
The tool is the topic, but **the problem is the reason**. Cheat sheets are
organized around tasks the reader already wants to do — the tool is how they get
there. Headings reference the tool (the reader came to this page on purpose),
but the content solves the underlying problem completely:
- "How to reverse proxy to Node" (caddy knowledge, not just node)
- "How to run a Node app as a System Service" (serviceman knowledge)
- "How to Enable Secure Remote Postgres Access" (openssl, pg_hba.conf, systemd)
- "How to manually configure git to use delta" (gitconfig, not delta flags)
- "How to make fish the default shell in iTerm2" (iTerm2 knowledge, not fish)
The reader's question is "how do I do X?" and the cheat sheet answers it
completely — including configs, adjacent tools, and platform-specific
variations. A goreleaser cheat sheet teaches you goreleaser YAML. A postgres
cheat sheet teaches you pg_hba.conf, openssl certs, and systemd units.
Cheat sheets cross tool boundaries freely. Node's references caddy, serviceman,
setcap-netbind, GitHub Actions. Postgres references serviceman, openrc, launchd.
They link to each other's webi pages. The scope is "everything you need to
accomplish this task," not "everything this one binary does."
They show the actual files and configs that matter — not documentation _about_
configs, but the configs themselves, copy-pasteable, with inline comments
explaining the non-obvious parts.
**Guidelines:**
- **Show the 3-5 things someone will actually do**, with copy-pasteable
commands. Skip exhaustive flag lists and API docs.
- **Lead with practical integration.** Show the exact `git config` lines, the
exact hook script, the exact shell alias — don't just explain the feature and
leave wiring up to the reader.
- **Skip what they already know.** No need to re-explain what the tool is at
length — the tagline and one-liner blockquote handle that. Get to the
commands.
- **Prefer concrete over abstract.** Instead of "you can configure X via a
config file", show the config file contents.
## Shell Naming Conventions
**Variables:**
- `ALL_CAPS` — environment variables only (`PATH`, `HOME`, `WEBI_VERSION`)
- `b_varname` — block-scoped (inside a function, loop, or conditional)
- `g_varname` — global to the script (and sourced scripts)
- `a_varname` — function arguments
**Functions and commands:**
- `fn_name` — helper functions (anything other than the script's main/entry
function)
- `cmd_name` — command aliases, e.g. `cmd_curl='curl --fail-with-body -sSL'`
## Code Style
Requires `node`, `shfmt`, `pwsh`, and `pwsh-essentials` (install all via webi).
Run before committing:
```sh
npm run fmt # prettier (JS/MD) + shfmt (sh) + pwsh-fmt (ps1)
npm run lint # jshint + shellcheck + pwsh-lint
```
Commit messages: `feat(<pkg>): add installer`, `fix(<pkg>): update install.sh`,
`docs(<pkg>): add cheat sheet`.
## Naming Conventions
- The canonical package name is the **command name** you type: `go`, `node`,
`rg`
- The alternate/alias name is the project name: `golang`, `nodejs`, `ripgrep`
- Package directories are lowercase with hyphens
## Common Pitfalls
- **Monorepo releases**: The GitHub API returns ALL releases for the repo. You
must filter in `releases.js` and strip the tag prefix from the version.
- **No `--version` flag**: Some tools lack version introspection. Comment out
`pkg_get_current_version` — webi still works, it just can't skip reinstalls.
- **normalize.js auto-detection**: OS/arch/ext are guessed from download
filenames. If the tool uses non-standard naming, you may need to set `os`,
`arch`, or `ext` explicitly in `releases.js`.
- **Goreleaser archives**: Typically contain a bare binary at the archive root
(not nested in a directory). Use `mv ./cmd "$pkg_src_cmd"`.

View File

@@ -10,6 +10,7 @@
- You'll be asked to make changes if you don't run the code formatters and
linters:
- Node / JavaScript:
- [prettier](https://webinstall.dev/prettier)
```sh
@@ -20,6 +21,7 @@
npm run lint
```
- Bash
- [shfmt](https://webinstall.dev/shfmt)
```sh
npm run shfmt

62
_common/brew.js Normal file
View File

@@ -0,0 +1,62 @@
'use strict';
let Fetcher = require('../_common/fetcher.js');
/**
* Gets releases from 'brew'.
*
* @param {null} _
* @param {string} formula
* @returns {Promise<any>}
*/
async function getDistributables(_, formula) {
if (!formula) {
return Promise.reject('missing formula for brew');
}
let resp;
try {
let url = `https://formulae.brew.sh/api/formula/${formula}.json`;
resp = await Fetcher.fetch(url, {
headers: { Accept: 'application/json' },
});
} catch (e) {
/** @type {Error & { code: string, response: { status: number, body: string } }} */ //@ts-expect-error
let err = e;
if (err.code === 'E_FETCH_RELEASES') {
err.message = `failed to fetch '${formula}' release data from 'brew': ${err.response.status} ${err.response.body}`;
}
throw e;
}
let body = JSON.parse(resp.body);
var ver = body.versions.stable;
var dl = (
body.bottle.stable.files.high_sierra || body.bottle.stable.files.catalina
).url.replace(new RegExp(ver.replace(/\./g, '\\.'), 'g'), '{{ v }}');
return [
{
version: ver,
download: dl.replace(/{{ v }}/g, ver),
},
].concat(
body.versioned_formulae.map(
/** @param {String} f */
function (f) {
var ver = f.replace(/.*@/, '');
return {
version: ver,
download: dl,
};
},
),
);
}
module.exports = getDistributables;
if (module === require.main) {
getDistributables(null, 'mariadb').then(function (all) {
console.info(JSON.stringify(all, null, 2));
});
}

56
_common/fetcher.js Normal file
View File

@@ -0,0 +1,56 @@
'use strict';
let Fetcher = module.exports;
/**
* @typedef ResponseSummary
* @prop {Boolean} ok
* @prop {Headers} headers
* @prop {Number} status
* @prop {String} body
*/
/**
* @param {String} url
* @param {RequestInit} opts
* @returns {Promise<ResponseSummary>}
*/
Fetcher.fetch = async function (url, opts) {
let resp = await fetch(url, opts);
let summary = Fetcher.throwIfNotOk(resp);
return summary;
};
/**
* @param {Response} resp
* @returns {Promise<ResponseSummary>}
*/
Fetcher.throwIfNotOk = async function (resp) {
let text = await resp.text();
if (!resp.ok) {
let headers = Array.from(resp.headers);
console.error('[Fetcher] error: Response Headers:', headers);
console.error('[Fetcher] error: Response Text:', text);
let err = new Error(`fetch was not ok`);
Object.assign({
status: 503,
code: 'E_FETCH_RELEASES',
response: {
status: resp.status,
headers: headers,
body: text,
},
});
throw err;
}
let summary = {
ok: resp.ok,
headers: resp.headers,
status: resp.status,
body: text,
};
return summary;
};

218
_common/git-tag.js Normal file
View File

@@ -0,0 +1,218 @@
'use strict';
require('dotenv').config({ path: '.env' });
var Crypto = require('crypto');
var util = require('util');
var exec = util.promisify(require('child_process').exec);
var Fs = require('node:fs/promises');
var FsSync = require('node:fs');
var Path = require('node:path');
var repoBaseDir = process.env.REPO_BASE_DIR || '';
if (!repoBaseDir) {
repoBaseDir = Path.resolve('./_repos');
// for stderr
console.error(`[Warn] REPO_BASE_DIR= not set, ${repoBaseDir}`);
}
var Repos = {};
Repos.clone = async function (repoPath, gitUrl) {
let uuid = Crypto.randomUUID();
let tmpPath = `${repoPath}.${uuid}.tmp`;
let bakPath = `${repoPath}.${uuid}.dup`;
await exec(`git clone --bare --filter=tree:0 ${gitUrl} ${tmpPath}`);
try {
FsSync.accessSync(repoPath);
return;
} catch (e) {
if (e.code !== 'ENOENT') {
throw e;
}
}
// sync to avoid race conditions
try {
FsSync.renameSync(repoPath, bakPath);
} catch (e) {
if (e.code !== 'ENOENT') {
throw e;
}
}
FsSync.renameSync(tmpPath, repoPath);
await Fs.rm(bakPath, { force: true, recursive: true });
};
Repos.checkExists = async function (repoPath) {
let err = await Fs.access(repoPath).catch(Object);
if (!err) {
return true;
}
if (err.code !== 'ENOENT') {
throw err;
}
return false;
};
Repos.fetch = async function (repoPath) {
await exec(`git --git-dir=${repoPath} fetch`);
};
Repos.getTags = async function (repoPath) {
var { stdout } = await exec(`git --git-dir=${repoPath} tag`);
var rawTags = stdout.trim().split('\n');
let tags = [];
for (let tag of rawTags) {
// ex: v1, v2, v1.1, 1.1.0-rc
let maybeVersionRe = /^(v\d+|v?\d+\.\d+)/;
let maybeVersion = maybeVersionRe.test(tag);
if (maybeVersion) {
tags.push(tag);
}
}
tags = tags.reverse();
return tags;
};
Repos.getTipInfo = async function (repoPath) {
var { stdout } = await exec(
`git --git-dir=${repoPath} rev-parse --abbrev-ref HEAD`,
);
var branch = stdout.trim();
var info = await Repos.getCommitInfo(repoPath, 'HEAD');
info.commitish = branch;
return info;
};
Repos.getCommitInfo = async function (repoPath, commitish) {
var { stdout } = await exec(
`git --git-dir=${repoPath} log -1 --format="%h %H %ad %cd" --date=iso-strict ${commitish}`,
);
stdout = stdout.trim();
var commitParts = stdout.split(/\s+/g);
return {
commitish: commitish,
commit_id: `${commitParts[0]}`,
commit: `${commitParts[1]}`,
date: commitParts[2],
date_authored: commitParts[3],
};
};
/**
* Lists GitHub Releases (w/ uploaded assets)
*
* @param request
* @param {string} owner
* @param {string} gitUrl
* @returns {PromiseLike<any> | Promise<any>}
*/
async function getDistributables(gitUrl) {
let all = {
releases: [],
download: '',
};
let repoName = gitUrl.split('/').pop();
repoName = repoName.replace(/\.git$/, '');
let repoPath = `${repoBaseDir}/${repoName}.git`;
let isCloned = await Repos.checkExists(repoPath);
if (!isCloned) {
await Repos.clone(repoPath, gitUrl);
} else {
await Repos.fetch(repoPath);
}
let commitInfos = [];
let tags = await Repos.getTags(repoPath);
for (let tag of tags) {
let commitInfo = await Repos.getCommitInfo(repoPath, tag);
Object.assign(commitInfo, { version: tag, channel: '' });
commitInfos.push(commitInfo);
}
{
let tipInfo = await Repos.getTipInfo(repoPath);
// "2024-01-01T00:00:00-05:00" => "2024-01-01T05:00:00"
let date = new Date(tipInfo.date);
// "2024-01-01T05:00:00" => "v2024.01.01-05.00.00"
let version = date.toISOString();
// strip '.000Z'
version = version.replace(/\.\d+Z/, '');
version = version.replace(/[:\-]/g, '.');
version = version.replace(/T/, '-');
Object.assign(tipInfo, { version: `v${version}`, channel: '' });
if (commitInfos.length > 1) {
tipInfo.channel = 'beta';
}
commitInfos.push(tipInfo);
}
let releases = [];
for (let commitInfo of commitInfos) {
let version = commitInfo.version.replace(/^v/, '');
let date = new Date(commitInfo.date);
let isoDate = date.toISOString();
isoDate = isoDate.replace(/\.\d+Z/, '');
// tags and HEAD qualify for '--branch <branchish>'
let branch = commitInfo.commitish;
let rel = {
name: `${repoName}-v${version}`,
version: version,
git_tag: branch,
git_commit_hash: commitInfo.commit_id,
lts: false,
channel: commitInfo.channel,
date: isoDate,
os: '*',
arch: '*',
ext: 'git',
download: gitUrl,
};
releases.push(rel);
}
all.releases = releases;
return all;
}
module.exports = getDistributables;
if (module === require.main) {
(async function main() {
let testRepos = [
// just a few tags, and a different HEAD
'https://github.com/tpope/vim-commentary.git',
// no tags, just HEAD
'https://github.com/ziglang/zig.vim.git',
// many, many tags
//'https://github.com/dense-analysis/ale.git',
];
for (let url of testRepos) {
let all = await getDistributables(url);
all = require('../_webi/normalize.js')(all);
console.info(JSON.stringify(all, null, 2));
}
})()
.then(function () {
process.exit(0);
})
.catch(function (err) {
console.error(err);
});
}

47
_common/gitea.js Normal file
View File

@@ -0,0 +1,47 @@
'use strict';
var GitHubish = require('./githubish.js');
/**
* Lists Gitea Releases (w/ uploaded assets)
*
* @param {null} _ - deprecated
* @param {String} owner
* @param {String} repo
* @param {String} baseurl
* @param {String} [username]
* @param {String} [token]
*/
async function getDistributables(
_,
owner,
repo,
baseurl,
username = '',
token = '',
) {
baseurl = `${baseurl}/api/v1`;
let all = await GitHubish.getDistributables({
owner,
repo,
baseurl,
username,
token,
});
return all;
}
module.exports = getDistributables;
if (module === require.main) {
getDistributables(
null,
'root',
'pathman',
'https://git.rootprojects.org',
'',
'',
).then(function (all) {
console.info(JSON.stringify(all, null, 2));
});
}

40
_common/github-source.js Normal file
View File

@@ -0,0 +1,40 @@
'use strict';
require('dotenv').config({ path: '.env' });
let GitHubSource = module.exports;
let GitHubishSource = require('./githubish-source.js');
/**
* @param {Object} opts
* @param {String} opts.owner
* @param {String} opts.repo
* @param {String} [opts.baseurl]
* @param {String} [opts.username]
* @param {String} [opts.token]
*/
GitHubSource.getDistributables = async function ({
owner,
repo,
baseurl = 'https://api.github.com',
username = process.env.GITHUB_USERNAME || '',
token = process.env.GITHUB_TOKEN || '',
}) {
let all = await GitHubishSource.getDistributables({
owner,
repo,
baseurl,
username,
token,
});
return all;
};
if (module === require.main) {
GitHubSource.getDistributables(null, 'BeyondCodeBootcamp', 'DuckDNS.sh').then(
function (all) {
console.info(JSON.stringify(all, null, 2));
},
);
}

42
_common/github.js Normal file
View File

@@ -0,0 +1,42 @@
'use strict';
require('dotenv').config({ path: '.env' });
let GitHubish = require('./githubish.js');
/**
* Lists GitHub Releases (w/ uploaded assets)
*
* @param {null} _ - deprecated
* @param {String} owner
* @param {String} repo
* @param {String} [baseurl]
* @param {String} [username]
* @param {String} [token]
*/
module.exports = async function (
_,
owner,
repo,
baseurl = 'https://api.github.com',
username = process.env.GITHUB_USERNAME || '',
token = process.env.GITHUB_TOKEN || '',
) {
let all = await GitHubish.getDistributables({
owner,
repo,
baseurl,
username,
token,
});
return all;
};
let GitHub = module.exports;
GitHub.getDistributables = module.exports;
if (module === require.main) {
GitHub.getDistributables(null, 'BurntSushi', 'ripgrep').then(function (all) {
console.info(JSON.stringify(all, null, 2));
});
}

174
_common/githubish-source.js Normal file
View File

@@ -0,0 +1,174 @@
'use strict';
let Fetcher = require('../_common/fetcher.js');
let GitHubishSource = module.exports;
/**
* Lists GitHub-Like Releases (source tarball & zip)
*
* @param {Object} opts
* @param {String} opts.owner
* @param {String} opts.repo
* @param {String} opts.baseurl
* @param {String} [opts.username]
* @param {String} [opts.token]
*/
GitHubishSource.getDistributables = async function ({
owner,
repo,
baseurl,
username = '',
token = '',
}) {
if (!owner) {
throw new Error('missing owner for repo');
}
if (!repo) {
throw new Error('missing repo name');
}
if (!baseurl) {
throw new Error('missing baseurl');
}
let url = `${baseurl}/repos/${owner}/${repo}/releases`;
let opts = {
headers: {
'Content-Type': 'appplication/json',
},
};
if (token) {
let userpass = `${username}:${token}`;
let basicAuth = btoa(userpass);
Object.assign(opts.headers, {
Authorization: `Basic ${basicAuth}`,
});
}
let resp;
try {
resp = await Fetcher.fetch(url, opts);
} catch (e) {
/** @type {Error & { code: string, response: { status: number, body: string } }} */ //@ts-expect-error
let err = e;
if (err.code === 'E_FETCH_RELEASES') {
err.message = `failed to fetch '${baseurl}' (githubish-source, user '${username}) release data: ${err.response.status} ${err.response.body}`;
}
throw e;
}
let gHubResp = JSON.parse(resp.body);
let all = {
/** @type {Array<BuildInfo>} */
releases: [],
download: '',
};
for (let release of gHubResp) {
let dists = GitHubishSource.releaseToDistributables(release);
for (let dist of dists) {
let updates =
await GitHubishSource.followDistributableDownloadAttachment(dist);
Object.assign(dist, updates);
all.releases.push(dist);
}
}
return all;
};
/**
* @typedef BuildInfo
* @prop {String} [name] - name to use instead of filename for hash urls
* @prop {String} version
* @prop {String} [_version]
* @prop {String} [arch]
* @prop {String} channel
* @prop {String} date
* @prop {String} download
* @prop {String} [ext]
* @prop {String} [_filename]
* @prop {String} [hash]
* @prop {String} [libc]
* @prop {Boolean} [_musl]
* @prop {Boolean} [lts]
* @prop {String} [size]
* @prop {String} os
*/
/**
* @param {any} ghRelease - TODO
* @returns {Array<BuildInfo>}
*/
GitHubishSource.releaseToDistributables = function (ghRelease) {
let ghTag = ghRelease['tag_name']; // TODO tags aren't always semver / sensical
let lts = /(\b|_)(lts)(\b|_)/.test(ghRelease['tag_name']);
let channel = 'stable';
if (ghRelease['prerelease']) {
channel = 'beta';
}
let date = ghRelease['published_at'] || '';
date = date.replace(/T.*/, '');
let urls = [ghRelease.tarball_url, ghRelease.zipball_url];
/** @type {Array<BuildInfo>} */
let dists = [];
for (let url of urls) {
dists.push({
name: '',
version: ghTag,
lts: lts,
channel: channel,
date: date,
os: '*',
arch: '*',
libc: '',
ext: '',
download: url,
});
}
return dists;
};
/**
* @param {BuildInfo} dist
*/
GitHubishSource.followDistributableDownloadAttachment = async function (dist) {
let abortCtrl = new AbortController();
let resp = await fetch(dist.download, {
method: 'HEAD',
redirect: 'follow',
signal: abortCtrl.signal,
});
let headers = Object.fromEntries(resp.headers);
// Workaround for bug where METHOD changes to GET
abortCtrl.abort();
await resp.text().catch(function (err) {
if (err.name !== 'AbortError') {
throw err;
}
});
// ex: content-disposition: attachment; filename=BeyondCodeBootcamp-DuckDNS.sh-v1.0.1-0-ga2f4bde.zip
// => BeyondCodeBootcamp-DuckDNS.sh-v1.0.1-0-ga2f4bde.zip
let name = headers['content-disposition'].replace(
/.*filename=([^;]+)(;|$)/,
'$1',
);
let download = resp.url;
return { name, download };
};
if (module === require.main) {
GitHubishSource.getDistributables({
owner: 'BeyondCodeBootcamp',
repo: 'DuckDNS.sh',
baseurl: 'https://api.github.com',
}).then(function (all) {
console.info(JSON.stringify(all, null, 2));
});
}

137
_common/githubish.js Normal file
View File

@@ -0,0 +1,137 @@
'use strict';
let Fetcher = require('../_common/fetcher.js');
/**
* @typedef DistributableRaw
* @prop {String} name
* @prop {String} version
* @prop {Boolean} lts
* @prop {String} [channel]
* @prop {String} date
* @prop {String} os
* @prop {String} arch
* @prop {String} ext
* @prop {String} download
*/
let GitHubish = module.exports;
/**
* Lists GitHub-Like Releases (w/ uploaded assets)
*
* @param {Object} opts
* @param {String} opts.owner
* @param {String} opts.repo
* @param {String} opts.baseurl
* @param {String} [opts.username]
* @param {String} [opts.token]
*/
GitHubish.getDistributables = async function ({
owner,
repo,
baseurl,
username = '',
token = '',
}) {
if (!owner) {
throw new Error('missing owner for repo');
}
if (!repo) {
throw new Error('missing repo name');
}
if (!baseurl) {
throw new Error('missing baseurl');
}
let url = `${baseurl}/repos/${owner}/${repo}/releases`;
let opts = {
headers: {
'Content-Type': 'appplication/json',
},
};
if (token) {
let userpass = `${username}:${token}`;
let basicAuth = btoa(userpass);
Object.assign(opts.headers, {
Authorization: `Basic ${basicAuth}`,
});
}
let resp;
try {
resp = await Fetcher.fetch(url, opts);
} catch (e) {
/** @type {Error & { code: string, response: { status: number, body: string } }} */ //@ts-expect-error
let err = e;
if (err.code === 'E_FETCH_RELEASES') {
err.message = `failed to fetch '${baseurl}' (githubish, user '${username}) release data: ${err.response.status} ${err.response.body}`;
}
throw e;
}
let gHubResp = JSON.parse(resp.body);
let all = {
/** @type {Array<DistributableRaw>} */
releases: [],
// todo make this ':baseurl' + ':releasename'
download: '',
};
try {
gHubResp.forEach(transformReleases);
} catch (e) {
/** @type {Error & { code: string, response: { status: number, body: string } }} */ //@ts-expect-error
let err = e;
console.error(err.message);
console.error('Error Headers:', resp.headers);
console.error('Error Body:', resp.body);
let msg = `failed to transform releases from '${baseurl}' with user '${username}'`;
throw new Error(msg);
}
/**
* @param {any} release - TODO
*/
function transformReleases(release) {
for (let asset of release['assets']) {
let name = asset['name'];
let date = release['published_at']?.replace(/T.*/, '');
let download = asset['browser_download_url'];
// TODO tags aren't always semver / sensical
let version = release['tag_name'];
let channel;
if (release['prerelease']) {
// -rcX, -preview, -beta, etc will be checked in _webi/normalize.js
channel = 'beta';
}
let lts = /(\b|_)(lts)(\b|_)/.test(release['tag_name']);
all.releases.push({
name: name,
version: version,
lts: lts,
channel: channel,
date: date,
os: '', // will be guessed by download filename
arch: '', // will be guessed by download filename
ext: '', // will be normalized
download: download,
});
}
}
return all;
};
if (module === require.main) {
GitHubish.getDistributables({
owner: 'BurntSushi',
repo: 'ripgrep',
baseurl: 'https://api.github.com',
}).then(function (all) {
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -19,13 +19,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading foobar from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing foobar"
# TODO: create package-specific temp directory

37
_example/releases.js Normal file
View File

@@ -0,0 +1,37 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'BurntSushi';
var repo = 'ripgrep';
/******************************************************************************/
/** Note: Delete this Comment! **/
/** **/
/** Need a an example that filters out miscellaneous release files? **/
/** See `deno`, `gitea`, or `caddy` **/
/** **/
/******************************************************************************/
let Releases = module.exports;
Releases.latest = async function () {
let all = await github(null, owner, repo);
return all;
};
Releases.sample = async function () {
let normalize = require('../_webi/normalize.js');
let all = await Releases.latest();
all = normalize(all);
// just select the first 5 for demonstration
all.releases = all.releases.slice(0, 5);
return all;
};
if (module === require.main) {
(async function () {
let samples = await Releases.sample();
console.info(JSON.stringify(samples, null, 2));
})();
}

View File

@@ -15,8 +15,8 @@ foreach ($my_dir in $my_dirs) {
$my_ps1 = [System.IO.Path]::GetRelativePath($my_cwd, $my_file.FullName)
$my_dir = [System.IO.Path]::GetDirectoryName($my_file.FullName)
if (-not (Test-Path -PathType Leaf -Path $my_ps1) -or
-not (Test-Path -PathType Container -Path $my_dir)) {
if (-Not (Test-Path -PathType Leaf -Path $my_ps1) -or
-Not (Test-Path -PathType Container -Path $my_dir)) {
Write-Host (" SKIP {0} (non-regular file or parent directory)" -f $my_ps1)
continue
}
@@ -31,7 +31,7 @@ foreach ($my_dir in $my_dirs) {
$my_new_file | Set-Content -Path $my_ps1
$my_new_file = $my_new_file + "`n"
if ($text -ne $my_new_file) {
IF ($text -ne $my_new_file) {
$my_status = 1
}
}

View File

@@ -15,8 +15,8 @@ foreach ($my_dir in $my_dirs) {
$my_ps1 = [System.IO.Path]::GetRelativePath($my_cwd, $my_file.FullName)
$my_dir = [System.IO.Path]::GetDirectoryName($my_file.FullName)
if (-not (Test-Path -PathType Leaf -Path $my_ps1) -or
-not (Test-Path -PathType Container -Path $my_dir)) {
if (-Not (Test-Path -PathType Leaf -Path $my_ps1) -or
-Not (Test-Path -PathType Container -Path $my_dir)) {
Write-Host (" SKIP {0} (non-regular file or parent directory)" -f $my_ps1)
continue
}
@@ -39,7 +39,7 @@ foreach ($my_dir in $my_dirs) {
$my_new_file | Set-Content -Path $my_ps1
$my_new_file = $my_new_file + "`n"
if ($my_old_file -ne $my_new_file) {
IF ($my_old_file -ne $my_new_file) {
$my_status = 1
}
}

View File

@@ -1,6 +1,6 @@
#!/usr/bin/env pwsh
if (!(Test-Path -Path "$Env:USERPROFILE\.vim\pack\plugins\start")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\.vim\pack\plugins\start")) {
New-Item -Path "$Env:USERPROFILE\.vim\pack\plugins\start" -ItemType Directory -Force | Out-Null
}
Remove-Item -Path "$Env:USERPROFILE\.vim\pack\plugins\start\example" -Recurse -ErrorAction Ignore

View File

@@ -13,6 +13,7 @@ async function main() {
caches: CACHE_DIR,
installers: INSTALLERS_DIR,
});
bc.freshenRandomPackage(600 * 1000);
// let dirs = await bc.getProjectsByType();
// let projNames = Object.keys(dirs.valid);

View File

@@ -3,16 +3,13 @@
var BuildsCacher = module.exports;
let Fs = require('node:fs/promises');
let Os = require('node:os');
let Path = require('node:path');
let LEGACY_CACHE_DIR = Path.join(Os.homedir(), '.cache/webi/legacy');
let HostTargets = require('./build-classifier/host-targets.js');
let Lexver = require('./build-classifier/lexver.js');
let Triplet = require('./build-classifier/triplet.js');
var ALIAS_RE = /^alias: ([\w.-]+)$/m;
var ALIAS_RE = /^alias: (\w+)$/m;
var LEGACY_ARCH_MAP = {
'*': 'ANYARCH',
@@ -129,8 +126,61 @@ async function readFirstBytes(path) {
return str;
}
BuildsCacher.create = function ({ ALL_TERMS, installers }) {
let promises = {};
async function getLatestBuilds(Releases, installersDir, cacheDir, name, date) {
console.info(`[INFO] getLatestBuilds: ${name}`);
if (!Releases) {
Releases = require(`${installersDir}/${name}/releases.js`);
}
// TODO update all releases files with module.exports.xxxx = 'foo';
if (!Releases.latest) {
Releases.latest = Releases;
}
let id = `${cacheDir}/${name}`;
if (!promises[id]) {
promises[id] = Promise.resolve();
}
promises[id] = promises[id].then(async function () {
return await getLatestBuildsInner(Releases, cacheDir, name, date);
});
return await promises[id];
}
async function getLatestBuildsInner(Releases, cacheDir, name, date) {
let data = await Releases.latest();
if (!date) {
date = new Date();
}
let isoDate = date.toISOString();
let yearMonth = isoDate.slice(0, 7);
// TODO hash file
let dataFile = `${cacheDir}/${yearMonth}/${name}.json`;
// TODO fsstat releases.js vs require-ing time as well
let tsFile = `${cacheDir}/${yearMonth}/${name}.updated.txt`;
let dirPath = Path.dirname(dataFile);
await Fs.mkdir(dirPath, { recursive: true });
let json = JSON.stringify(data, null, 2);
await Fs.writeFile(dataFile, json, 'utf8');
let seconds = date.valueOf();
let ms = seconds / 1000;
let msStr = ms.toFixed(3);
await Fs.writeFile(tsFile, msStr, 'utf8');
return data;
}
BuildsCacher.create = function ({ ALL_TERMS, installers, caches }) {
let installersDir = installers;
let cacheDir = caches;
if (!ALL_TERMS) {
ALL_TERMS = Triplet.TERMS_PRIMARY_MAP;
@@ -145,11 +195,9 @@ BuildsCacher.create = function ({ ALL_TERMS, installers }) {
bc._triplets = {};
bc._targetsByBuildIdCache = {};
bc._caches = {};
bc._staleAge = 15 * 60 * 1000;
bc._allFormats = {};
bc._allTriplets = {};
// Per-name lock: serializes cold-cache getPackages so concurrent
// callers can't corrupt bc._caches[name] via a transformAndUpdate race.
bc._inflight = {};
for (let term of TERMS_META) {
delete bc.orphanTerms[term];
@@ -168,6 +216,19 @@ BuildsCacher.create = function ({ ALL_TERMS, installers }) {
let entries = await Fs.readdir(installersDir, { withFileTypes: true });
for (let entry of entries) {
let meta = await bc.getProjectTypeByEntry(entry);
if (meta.type === 'not_found') {
let err = meta.detail;
console.error('');
console.error('PROBLEM');
console.error(` ${err.message}`);
console.error('');
console.error('SOLUTION');
console.error(' npm clean-install');
console.error('');
throw new Error(
'[SANITY FAIL] should never have missing modules in prod',
);
}
dirs[meta.type][entry.name] = meta.detail;
}
@@ -236,16 +297,19 @@ BuildsCacher.create = function ({ ALL_TERMS, installers }) {
return { type: 'alias', detail: link };
}
let cacheFile = `${LEGACY_CACHE_DIR}/${entry.name}.json`;
let hasCacheFile = await Fs.access(cacheFile)
.then(function () {
return true;
})
.catch(function () {
return false;
});
if (!hasCacheFile) {
return { type: 'selfhosted', detail: true };
let releasesPath = Path.join(path, 'releases.js');
try {
void require(releasesPath);
} catch (err) {
if (err.code !== 'MODULE_NOT_FOUND') {
return { type: 'errors', detail: err };
}
if (err.message.includes(`Cannot find module '${releasesPath}'`)) {
return { type: 'selfhosted', detail: true };
}
return { type: 'not_found', detail: err };
}
return { type: 'valid', detail: true };
@@ -253,26 +317,14 @@ BuildsCacher.create = function ({ ALL_TERMS, installers }) {
// Typically a package is organized by release (ex: go has 1.20, 1.21, etc),
// but we will organize by the build (ex: go1.20-darwin-arm64.tar.gz, etc).
bc.getPackages = async function (args) {
let name = args.name;
let warm = bc._caches[name];
if (warm) {
return _doGetPackages(args);
bc.getPackages = async function ({ Releases, name, date }) {
if (!date) {
date = new Date();
}
let inflight = bc._inflight[name];
if (inflight) {
return inflight;
}
let p = _doGetPackages(args).finally(function () {
delete bc._inflight[name];
});
bc._inflight[name] = p;
return p;
};
async function _doGetPackages({ name }) {
let dataFile = `${LEGACY_CACHE_DIR}/${name}.json`;
let tsFile = `${LEGACY_CACHE_DIR}/${name}.updated.txt`;
let isoDate = date.toISOString();
let yearMonth = isoDate.slice(0, 7);
let dataFile = `${cacheDir}/${yearMonth}/${name}.json`;
let tsFile = `${cacheDir}/${yearMonth}/${name}.updated.txt`;
let tsDate;
{
@@ -326,7 +378,7 @@ BuildsCacher.create = function ({ ALL_TERMS, installers }) {
}
}
if (!projInfo) {
return meta;
projInfo = await getLatestBuilds(Releases, installersDir, cacheDir, name);
}
let latestProjInfo = await BuildsCacher.transformAndUpdate(
name,
@@ -337,8 +389,69 @@ BuildsCacher.create = function ({ ALL_TERMS, installers }) {
);
bc._caches[name] = latestProjInfo;
return latestProjInfo;
}
process.nextTick(async function () {
let now = date.valueOf();
let age = now - projInfo.updated;
let fresh = age < bc._staleAge;
if (fresh) {
return;
}
projInfo = await getLatestBuilds(Releases, installersDir, cacheDir, name)
.then(function () {
let latestProjInfo = BuildsCacher.transformAndUpdate(
name,
projInfo,
meta,
date,
bc,
);
bc._caches[name] = latestProjInfo;
})
.catch(function (err) {
console.error(`Fail when fetching latest builds for '${name}'`);
console.error(err);
});
});
return projInfo;
};
// Makes sure that packages are updated once an hour, on average
bc._staleNames = [];
bc._freshenTimeout = null;
bc.freshenRandomPackage = async function (minDelay) {
if (!minDelay) {
minDelay = 15 * 1000;
}
if (bc._staleNames.length === 0) {
let dirs = await bc.getProjectsByType();
bc._staleNames = Object.keys(dirs.valid);
bc._staleNames.sort(function () {
return 0.5 - Math.random();
});
}
let name = bc._staleNames.pop();
void (await bc.getPackages({
//Releases: Releases,
name: name,
date: new Date(),
}));
console.info(`[INFO] freshenRandomPackage: ${name}`);
let hour = 60 * 60 * 1000;
let delay = minDelay;
let spread = hour / bc._staleNames.length;
let seed = Math.random();
delay += seed * spread;
clearTimeout(bc._freshenTimeout);
bc._freshenTimeout = setTimeout(bc.freshenRandomPackage, delay);
bc._freshenTimeout.unref();
};
/**
* Given a list of acceptable formats, get the sorted list of of formats.
@@ -430,10 +543,7 @@ BuildsCacher.create = function ({ ALL_TERMS, installers }) {
exts.push('.gz');
exts.push('.sh');
}
let hasGit = formats.includes('git') || formats.includes('.git');
if (hasGit) {
exts.push('.git');
}
exts.push('.git');
// Fallbacks
// (we include everything to bubble an extract error over not found)
@@ -542,19 +652,29 @@ BuildsCacher.create = function ({ ALL_TERMS, installers }) {
return null;
}
// Version-first iteration, not triplet-first: take the newest
// version even when its only build lives in a fallback triplet
// (e.g. serviceman v1.0.1 only exists at posix_2017-ANYARCH-none).
for (let lexver of projInfo.lexvers) {
let ver = projInfo.lexversMap[lexver] || lexver;
for (let _triplet of triplets) {
let targetReleases = projInfo.releasesByTriplet[_triplet];
if (!targetReleases) {
continue;
}
for (let _triplet of triplets) {
let targetReleases = projInfo.releasesByTriplet[_triplet];
if (!targetReleases) {
continue;
}
let versions = Object.keys(targetReleases);
//console.log('dbg: targetRelease versions', versions);
let lexvers = [];
for (let version of versions) {
let lexPrefix = Lexver.parseVersion(version);
lexvers.push(lexPrefix);
}
lexvers.sort();
lexvers.reverse();
// TODO get the other matchInfo props
// Make sure that these releases are the expected version
// (ex: jq1.7 => darwin-arm64-libc, jq1.6 => darwin-x86_64-libc)
for (let matchver of lexvers) {
let ver = projInfo.lexversMap[matchver] || matchver;
let packages = targetReleases[ver];
//console.log('dbg: packages', packages);
if (!packages) {
continue;
}
@@ -605,32 +725,22 @@ BuildsCacher.create = function ({ ALL_TERMS, installers }) {
return triplets;
}
// Prefer platform-specific matches over ANYOS/ANYARCH fallbacks.
// This ensures e.g. darwin-aarch64-none matches before
// ANYOS-ANYARCH-none (.git source URLs from old releases).
let oses = [];
if (hostTarget.os === 'windows') {
oses = ['windows', 'ANYOS'];
oses = ['ANYOS', 'windows'];
} else if (hostTarget.os === 'android') {
oses = ['android', 'linux', 'posix_2017', 'posix_2024', 'ANYOS'];
oses = ['ANYOS', 'posix_2017', 'posix_2024', 'android', 'linux'];
} else {
oses = [hostTarget.os, 'posix_2017', 'posix_2024', 'ANYOS'];
oses = ['ANYOS', 'posix_2017', 'posix_2024', hostTarget.os];
}
let waterfall = HostTargets.WATERFALL[hostTarget.os] || {};
let arches = waterfall[hostTarget.arch] ||
HostTargets.WATERFALL.ANYOS[hostTarget.arch] || [hostTarget.arch];
arches = arches.concat(['ANYARCH']);
arches = ['ANYARCH'].concat(arches);
let libcs = waterfall[hostTarget.libc] ||
HostTargets.WATERFALL.ANYOS[hostTarget.libc] || [hostTarget.libc];
// Extend the glibc-host waterfall: the table only lists [none, libc]
// but Rust projects (bat, rg) and node ship libc='gnu' builds, and
// static musl builds also run on glibc hosts.
if (hostTarget.libc === 'libc' && !libcs.includes('gnu')) {
libcs = ['none', 'gnu', 'musl', 'libc'];
}
for (let os of oses) {
for (let arch of arches) {
for (let libc of libcs) {
@@ -658,17 +768,10 @@ BuildsCacher.create = function ({ ALL_TERMS, installers }) {
};
BuildsCacher._classify = function (bc, projInfo, build) {
/* jshint maxcomplexity: 30 */
// Cache entries arrive pre-classified (os/arch/libc/ext set). Skip
// maybeInstallable for those — it false-rejects names ending in a
// version tag (`serviceman-v1.0.1`, `v1.0.1.zip`).
let cacheClassified =
build.os && build.arch && build.libc && build.ext;
if (!cacheClassified) {
let maybeInstallable = Triplet.maybeInstallable(projInfo, build);
if (!maybeInstallable) {
return null;
}
/* jshint maxcomplexity: 25 */
let maybeInstallable = Triplet.maybeInstallable(projInfo, build);
if (!maybeInstallable) {
return null;
}
if (LEGACY_OS_MAP[build.os]) {
@@ -732,26 +835,16 @@ BuildsCacher._classify = function (bc, projInfo, build) {
bc.unknownTerms[term] = true;
}
// Skip termsToTarget for cache-classified entries: it false-flags
// e.g. .git URLs as os=ANYOS while the cache says os=posix_2017,
// and the mismatch check throws.
// {NAME}.windows.x86_64v2.musl.exe
// windows-x86_64_v2-musl
target = { triplet: '' };
if (cacheClassified) {
target.os = build.os;
target.arch = build.arch;
target.libc = build.libc;
target.vendor = build.vendor || 'unknown';
target.android = false;
target.unknownTerms = [];
} else {
try {
void Triplet.termsToTarget(target, projInfo, build, terms);
} catch (e) {
console.error(`PACKAGE FORMAT CHANGE for '${projInfo.name}':`);
console.error(e.message);
console.error(build);
return null;
}
try {
void Triplet.termsToTarget(target, projInfo, build, terms);
} catch (e) {
console.error(`PACKAGE FORMAT CHANGE for '${projInfo.name}':`);
console.error(e.message);
console.error(build);
return null;
}
target.triplet = `${target.arch}-${target.vendor}-${target.os}-${target.libc}`;

View File

@@ -15,8 +15,11 @@ let bc = BuildsCacher.create({
caches: CACHE_DIR,
installers: INSTALLERS_DIR,
});
bc.freshenRandomPackage(600 * 1000);
Builds.init = async function () {
bc.freshenRandomPackage(600 * 1000);
let dirs = await bc.getProjectsByType();
let projNames = Object.keys(dirs.valid);
@@ -24,6 +27,7 @@ Builds.init = async function () {
await Parallel.run(parallel, projNames, getAll);
async function getAll(name) {
void (await bc.getPackages({
//Releases: Releases,
name: name,
date: new Date(),
}));

View File

@@ -1,14 +1,11 @@
'use strict';
let Fs = require('node:fs/promises');
let Os = require('node:os');
let Path = require('node:path');
// let Builds = require('./builds.js');
let BuildsCacher = require('./builds-cacher.js');
let Triplet = require('./build-classifier/triplet.js');
let LEGACY_CACHE_DIR = Path.join(Os.homedir(), '.cache/webi/legacy');
async function main() {
let projName = process.argv[2];
if (!projName) {
@@ -39,11 +36,16 @@ async function main() {
arches: [],
libcs: [],
formats: [],
// TODO channels: [],
};
let dataFile = Path.join(LEGACY_CACHE_DIR, `${projName}.json`);
let json = await Fs.readFile(dataFile, 'utf8');
let projInfo = JSON.parse(json);
let installersDir = Path.join(__dirname, '..');
let Releases = require(`${installersDir}/${projName}/releases.js`);
if (!Releases.latest) {
Releases.latest = Releases;
}
let projInfo = await Releases.latest();
// let packages = await Builds.getPackage({ name: projName });
// console.log(packages);
@@ -68,11 +70,9 @@ async function main() {
console.log(`[DEBUG] transformed`);
let sample = transformed.packages.slice(0, 20);
console.log('packages:', sample, ':packages');
let firstTriplet = Object.keys(transformed.releasesByTriplet)[0];
let firstVersion = transformed.versions[0];
console.log(
`releasesByTriplet[${firstTriplet}][${firstVersion}]:`,
transformed.releasesByTriplet[firstTriplet]?.[firstVersion],
'releasesByTriplet:',
transformed.releasesByTriplet['linux-x86_64-none'][transformed.versions[0]],
':releasesByTriplet',
);
console.log('versions:', transformed.versions, ':versions');

View File

@@ -6,7 +6,7 @@
############################################################
New-Item -Path "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
New-Item -Path "$Env:USERPROFILE\.local\bin" -ItemType Directory -Force | Out-Null
if ($null -eq $Env:WEBI_HOST -or $Env:WEBI_HOST -eq "") { $Env:WEBI_HOST = "https://webinstall.dev" }
IF ($null -eq $Env:WEBI_HOST -or $Env:WEBI_HOST -eq "") { $Env:WEBI_HOST = "https://webinstall.dev" }
curl.exe -s -A "windows" "$Env:WEBI_HOST/packages/webi/webi-pwsh.ps1" -o "$Env:USERPROFILE\.local\bin\webi-pwsh.ps1"
Set-ExecutionPolicy -Scope Process Bypass
& "$Env:USERPROFILE\.local\bin\webi-pwsh.ps1" "{{ exename }}"

View File

@@ -139,6 +139,8 @@ async function main() {
console.info('');
}
bc.freshenRandomPackage(600 * 1000);
let rows = [];
let triples = [];
let valids = Object.keys(dirs.valid);

259
_webi/normalize.js Normal file
View File

@@ -0,0 +1,259 @@
'use strict';
// this may need customizations between packages
var osMap = {
macos: /(\b|_)(apple|os(\s_-)?x\b|mac|darwin|iPhone|iOS|iPad)/i,
linux: /(\b|_)(linux)/i,
freebsd: /(\b|_)(freebsd)/i,
windows: /(\b|_)(win|microsoft|msft)/i,
sunos: /(\b|_)(sun)/i,
aix: /(\b|_)(aix)/i,
};
var maps = {
oses: {},
arches: {},
libcs: {},
formats: {},
};
Object.keys(osMap).forEach(function (name) {
maps.oses[name] = true;
});
var formats = ['zip', 'xz', 'tar', 'pkg', 'msi', 'git', 'exe', 'dmg', 'git'];
formats.forEach(function (name) {
maps.formats[name] = true;
});
// evaluation order matters
// (i.e. otherwise x86 and x64 can cross match)
var arches = [
// arm64/aarch64 has very high specificity, so it comes first
'arm64',
// arm 7 is also generic aarch/arm/arm32
'armv7l',
// arm6 can run on armv7
'armv6l',
// amd64 is more likely and less often specified than arm64
'amd64',
'x86',
'ppc64le',
'ppc64',
's390x',
];
// Used for detecting system arch from package download url, for example:
//
// https://git.com/org/foo/releases/v0.7.9/foo-aarch64-linux-musl.tar.gz
// https://git.com/org/foo/releases/v0.7.9/foo-arm-linux-musleabihf.tar.gz
// https://git.com/org/foo/releases/v0.7.9/foo-armv7-linux-musleabihf.tar.gz
// https://git.com/org/foo/releases/v0.7.9/foo-x86_64-linux-musl.tar.gz
//
var archMap = {
arm64: /(\b|_)(aarch64|arm64)/i,
armv7l: /(\b|_)(arm32|arm[_\-]?v?7l?)/i,
armv6l: /(\b|_)(arm|aarch32|arm[_\-]?v?6l?)(\b|_)/i,
//amd64: /(amd.?64|x64|[_\-]64)/i,
amd64:
/(\b|_|amd|(dar)?win(dows)?|mac(os)?|linux|osx|x)64([_\-]?bit)?(\b|_)/i,
//x86: /(86)(\b|_)/i,
x86: /(\b|_|amd|(dar)?win(dows)?|mac(os)?|linux|osx|x)(86|32|i?386)([_\-]?bit)?(\b|_)/i,
ppc64le: /(\b|_)(ppc64le)/i,
ppc64: /(\b|_)(ppc64)(\b|_)/i,
s390x: /(\b|_)(s390x)/i,
};
arches.forEach(function (name) {
maps.arches[name] = true;
});
var libcs = ['none', 'musl', 'gnu', 'msvc', 'libc'];
libcs.forEach(function (name) {
maps.libcs[name] = true;
});
function normalize(all) {
/* jshint maxcomplexity:50 */
/* jshint maxdepth:10 */
var supported = {
oses: {},
arches: {},
libcs: {},
formats: {},
};
for (let rel of all.releases) {
rel.version = rel.version.replace(/^v/i, '');
if (!rel.name) {
rel.name = rel.download.replace(/.*\//, '');
}
if (!rel.os) {
rel.os = 'unknown';
let osNames = Object.keys(osMap);
for (let osName of osNames) {
let relName = rel.name || rel.download;
let osRegExp = osMap[osName];
let matches = osRegExp.test(relName);
if (matches) {
rel.os = osName;
break;
}
}
}
supported.oses[rel.os] = true;
if (!rel.arch) {
for (let arch of arches) {
let name = rel.name || rel.download;
let isArch = name.match(archMap[arch]);
if (isArch) {
rel.arch = arch;
break;
}
}
}
if (!rel.arch) {
if ('macos' === rel.os) {
rel.arch = 'amd64';
}
}
supported.arches[rel.arch] = true;
// note: depends on rel.os
if (!rel.libc) {
let isMusl;
let isMsvc;
let isStatic;
let isGnu;
// extra blocks to prevent copy pasta errors
{
let muslRe = /(\b|_)(musl)(\b|_)/i;
isMusl = muslRe.test(rel.download) || muslRe.test(rel.name);
}
{
let msvcRe = /(\b|_)(msvc)(\b|_)/i;
isMsvc = msvcRe.test(rel.download) || msvcRe.test(rel.name);
}
{
let staticRe = /(\b|_)(static)(\b|_)/i;
isStatic = staticRe.test(rel.download) || staticRe.test(rel.name);
}
{
let gnuRe = /(\b|_)(gnu|glibc|libc)(\b|_)/i;
isGnu = gnuRe.test(rel.download) || gnuRe.test(rel.name);
}
if (isMusl) {
// we specifically tag things that need musl++ in their own releases
rel.libc = 'none';
} else if (isStatic) {
rel.libc = 'none';
} else if (isGnu) {
rel.libc = 'gnu';
if (rel.os === 'windows') {
// windows gnu is static
rel.libc = 'none';
} else if (rel.os === 'darwin') {
// if glibc is required on macos, it'll be static
rel.libc = 'none';
}
} else if (isMsvc) {
rel.libc = 'msvc';
} else {
// The default is no requirement for any particular libc
// (Go, Zig, POSIX Shell, JS, etc)
// and hopefully we never have to worry about mingw and friends
rel.libc = 'none';
}
}
supported.libcs[rel.libc] = true;
var tarExt;
if (!rel.ext) {
// pkg-v1.0.tar.gz => ['gz', 'tar', '0', 'pkg-v1']
// pkg-v1.0.tar => ['tar', '0' ,'pkg-v1']
// pkg-v1.0.zip => ['zip', '0', 'pkg-v1']
var exts = (rel.name || rel.download).split('.');
if (1 === exts.length) {
// for bare releases in the format of foo-linux-amd64
rel.ext = 'exe';
}
exts = exts.reverse().slice(0, 2);
if ('tar' === exts[1]) {
rel.ext = exts.reverse().join('.');
tarExt = 'tar';
} else if ('tgz' === exts[0]) {
rel.ext = 'tar.gz';
tarExt = 'tar';
} else {
rel.ext = exts[0];
}
if (/\-|linux|mac|os[_\-]?x|arm|amd|86|64|mip/i.test(rel.ext)) {
// for bare releases in the format of foo.linux-amd64
rel.ext = 'exe';
}
}
supported.formats[tarExt || rel.ext] = true;
if (!rel.channel) {
// basically like this: (+.-_)(beta|rc)(0-9)(+.-_)
// matches:
// - v1.0-beta
// - v1.0-beta1.1
// - v1.0-beta-11
// won't match:
// - v1.0beta
// - v1.0-beta1b
let isBetaRe =
/(\b|_)(alpha|beta|dev|developer|prev|preview|rc)(\d+)(\b|_)/;
let isBeta = isBetaRe.test(rel.name);
if (isBeta) {
rel.channel = 'beta';
} else {
rel.channel = 'stable';
}
}
if (all.download) {
rel.download = all.download.replace(/{{ download }}/, rel.download);
}
}
all.oses = Object.keys(supported.oses).filter(function (name) {
return maps.oses[name];
});
all.arches = Object.keys(supported.arches).filter(function (name) {
return maps.arches[name];
});
all.libcs = Object.keys(supported.libcs).filter(function (name) {
return maps.libcs[name];
});
all.formats = Object.keys(supported.formats).filter(function (name) {
return maps.formats[name];
});
return all;
}
module.exports = normalize;
module.exports._debug = function (all) {
all = normalize(all);
all.releases = all.releases
.filter(function (r) {
return ['windows', 'macos', 'linux'].includes(r.os) && 'amd64' === r.arch;
})
.slice(0, 10);
return all;
};
// NOT in order of priority (which would be tar, xz, zip, ...)
module.exports.formats = formats;
module.exports.arches = arches;
module.exports.libcs = libcs;
module.exports.formatsMap = maps.formats;

View File

@@ -45,15 +45,15 @@ $TDim = "${Esc}[2m"
$TReset = "${Esc}[0m"
function Invoke-DownloadUrl {
param (
Param (
[string]$URL,
[string]$Params,
[string]$Path,
[switch]$Force
)
if (Test-Path -Path "$Path") {
if (-not $Force.IsPresent) {
IF (Test-Path -Path "$Path") {
IF (-Not $Force.IsPresent) {
Write-Host " ${TDim}Found${TReset} $Path"
return
}
@@ -65,7 +65,7 @@ function Invoke-DownloadUrl {
Write-Host " Downloading ${TDim}from${TReset}"
Write-Host " ${TDim}${URL}${TReset}"
if ($Params.Length -ne 0) {
IF ($Params.Length -ne 0) {
Write-Host " ?$Params"
$URL = "${URL}?${Params}"
}
@@ -80,18 +80,18 @@ function Get-UserAgent {
# This is the canonical CPU arch when the process is emulated
$my_arch = "$Env:PROCESSOR_ARCHITEW6432"
if ($my_arch -eq $null -or $my_arch -eq "") {
IF ($my_arch -eq $null -or $my_arch -eq "") {
# This is the canonical CPU arch when the process is native
$my_arch = "$Env:PROCESSOR_ARCHITECTURE"
}
if ($my_arch -eq "AMD64") {
IF ($my_arch -eq "AMD64") {
# Because PowerShell is sometimes AMD64 on Windows 10 ARM
# See https://oofhours.com/2020/02/04/powershell-on-windows-10-arm64/
$my_os_arch = (Get-CimInstance -ClassName Win32_OperatingSystem).OSArchitecture
# Using -clike because of the trailing newline
if ($my_os_arch -clike "ARM 64*") {
IF ($my_os_arch -clike "ARM 64*") {
$my_arch = "ARM64"
}
}
@@ -124,7 +124,7 @@ function webi_path_add($pathname) {
$exists_in_path = $true
}
}
if (-not $exists_in_path) {
if (-Not $exists_in_path) {
$all_user_paths = "${pathname};${all_user_paths}".Trim(';')
[Environment]::SetEnvironmentVariable("Path", $all_user_paths, "User")
$null = Sync-EnvPath

View File

@@ -49,7 +49,7 @@ var baseurl = 'https://webinstall.dev';
var maxLen = 0;
console.info('');
console.info('Has the necessary files?');
['README.md', 'install.sh', 'install.ps1']
['README.md', 'releases.js', 'install.sh', 'install.ps1']
.map(function (node) {
maxLen = Math.max(maxLen, node.length);
return node;

View File

@@ -2,30 +2,75 @@
var Releases = module.exports;
var Fs = require('node:fs/promises');
var Os = require('node:os');
var path = require('path');
var _normalize = require('./normalize.js');
var cache = {};
//var staleAge = 5 * 1000;
//var expiredAge = 15 * 1000;
var staleAge = 5 * 60 * 1000;
var expiredAge = 15 * 60 * 1000;
var LEGACY_CACHE_DIR = path.join(Os.homedir(), '.cache/webi/legacy');
let installerDir = path.join(__dirname, '..');
// Sort releases by ext preference and libc within the same version.
// The cache is already sorted by version (stable before beta, newest first),
// so we only re-order within the same version string.
Releases.get = async function (pkgdir) {
let get;
try {
get = require(`${pkgdir}/releases.js`);
// TODO update all releases files with module.exports.xxxx = 'foo';
if (!get.latest) {
get.latest = get;
}
} catch (e) {
let err = new Error('no releases.js for', pkgdir.split(/[\/\\]+/).pop());
err.code = 'E_NO_RELEASE';
throw err;
}
let all = await get.latest();
return _normalize(all);
};
// TODO needs a proper test, and more accurate (though perhaps far less simple) code
function createFormatsSorter(formats) {
return function sortByExtLibc(a, b) {
if (a.version !== b.version) {
// Array.sort is stable (V8, ES2019), so returning 0 across
// versions preserves the cache's pre-sorted version-desc order.
return 0;
return function sortByVerExt(a, b) {
function lexver(semver) {
// v1.20.156 => 00001.00020.00156.zzzzz
// TODO BUG: v1.20.156-rc2 => 00001.00020.00156.rc2zz
var parts = semver.split(/[+\.\-]/g);
while (parts.length < 4) {
parts.push('');
}
return parts
.map(function (num, i) {
if (3 === i) {
return num.toString().padEnd(10, 'z');
}
return num.toString().padStart(10, '0');
})
.join('.');
}
var aver = lexver(a.version);
var bver = lexver(b.version);
if (aver > bver) {
//console.log(aver, '>', bver);
return -1;
}
if (aver < bver) {
//console.log(aver, '<', bver);
return 1;
}
var aExtPri = formats.indexOf(a.ext.replace(/tar\..*/, 'tar'));
var bExtPri = formats.indexOf(b.ext.replace(/tar\..*/, 'tar'));
if (aExtPri > bExtPri) {
//console.log(a.ext, aExtPri, '>', b.ext, bExtPri);
return -1;
}
if (aExtPri < bExtPri) {
//console.log(a.ext, aExtPri, '<', b.ext, bExtPri);
return 1;
}
@@ -42,39 +87,99 @@ function createFormatsSorter(formats) {
}
async function getCachedReleases(pkg) {
// returns { download: '', releases: [{ version, date, os, arch, lts, channel, download}] }
// returns { download: '<template string>', releases: [{ version, date, os, arch, lts, channel, download}] }
if (cache[pkg]) {
return cache[pkg];
async function chainCachePromise(fn) {
cache[pkg].promise = cache[pkg].promise.then(fn);
return cache[pkg].promise;
}
let dataFile = `${LEGACY_CACHE_DIR}/${pkg}.json`;
async function sleep(ms) {
return await new Promise(function (resolve, reject) {
setTimeout(resolve, ms);
});
}
let json = await Fs.readFile(dataFile, 'utf8').catch(function (err) {
if (err.code === 'ENOENT') {
return null;
async function putCache() {
var age = Date.now() - cache[pkg].updatedAt;
if (age < staleAge) {
//console.debug('NOT STALE ANYMORE - updated in previous promise');
return cache[pkg].all;
}
throw err;
});
if (!json) {
let empty = { download: '', releases: [] };
cache[pkg] = empty;
return empty;
//console.debug('DOWNLOADING NEW "%s" releases', pkg);
var pkgdir = path.join(installerDir, pkg);
// workaround for request timeout seeming to not work
let complete = false;
await Promise.race([
Releases.get(pkgdir)
.catch(function (err) {
if ('E_NO_RELEASE' === err.code) {
let all = { _error: 'E_NO_RELEASE', download: '', releases: [] };
return all;
}
throw err;
})
.catch(function (err) {
let hasReleases = cache[pkg].all?.releases?.length > 1;
if (!hasReleases) {
throw err;
}
console.error(`Error: the BOOGEYMAN got us!`);
console.error(err.stack);
return cache[pkg].all;
})
.then(function (all) {
// Note: it is possible for slightly older data
// to replace slightly newer data, but this is better
// than being in a cycle where release updates _always_
// take longer than expected.
//console.debug('DOWNLOADED NEW "%s" releases', pkg);
cache[pkg].updatedAt = Date.now();
cache[pkg].all = all;
complete = true;
}),
sleep(15000).then(function () {
if (complete) {
return;
}
console.error(`request timeout waiting for '${pkg}' release info`);
}),
]);
return cache[pkg].all;
}
let all;
try {
all = JSON.parse(json);
} catch (e) {
console.error(`error: ${dataFile}:\n\t${e.message}`);
let empty = { download: '', releases: [] };
cache[pkg] = empty;
return empty;
if (!cache[pkg]) {
cache[pkg] = {
updatedAt: 0,
all: { download: '', releases: [] },
promise: Promise.resolve(),
};
}
cache[pkg] = all;
return all;
var bgRenewal;
var age = Date.now() - cache[pkg].updatedAt;
var fresh = age < staleAge;
if (!fresh) {
bgRenewal = chainCachePromise(putCache);
}
var tooStale = age > expiredAge;
if (!tooStale) {
return await cache[pkg].all;
}
return await Promise.race([
bgRenewal,
sleep(5000).then(function () {
return cache[pkg].all;
}),
]);
}
async function filterReleases(
@@ -85,22 +190,15 @@ async function filterReleases(
// sort the most compatible format first
// (i.e. so that we don't do .pkg on linux except on purpose)
var rformats = formats.slice(0).reverse();
var sortByExtLibc = createFormatsSorter(rformats);
var sortByVerExt = createFormatsSorter(rformats);
var reVer = new RegExp('^' + ver + '\\b');
function selectMatches(rel) {
/* jshint maxcomplexity: 25 */
if (os) {
// '*' = any OS (matches anything, including windows).
// 'posix' / 'posix_20xx' = any POSIX OS (matches linux, macos,
// freebsd, etc., but NOT windows).
let isPosix = rel.os === 'posix' || rel.os.startsWith('posix_20');
let osMatches =
rel.os === '*' ||
rel.os === os ||
(isPosix && os !== 'windows');
if (!osMatches) {
return false;
if (rel.os !== '*') {
if (rel.os !== os) {
return false;
}
}
}
@@ -112,10 +210,7 @@ async function filterReleases(
}
}
// libc='libc' is serve-releases.js's default when the caller
// didn't pin one — treat it as 'no preference', not a filter.
let isMeaningfulLibc = libc && libc !== 'libc' && rel.libc !== 'none';
if (isMeaningfulLibc) {
if (rel.libc !== 'none') {
let releaseRequiresMusl = rel.libc === 'musl';
// goal: handle non-glibc (Alpine / Docker / musl)
let osHasMusl = libc === 'musl';
@@ -162,7 +257,7 @@ async function filterReleases(
return true;
}
var sortedRels = all.releases.filter(selectMatches).sort(sortByExtLibc);
var sortedRels = all.releases.filter(selectMatches).sort(sortByVerExt);
//console.log(sortedRels.slice(0, 4));
return sortedRels.slice(0, limit || 1000);
@@ -321,8 +416,8 @@ Releases.getReleases = function ({
};
if (require.main === module) {
return Releases
.getReleases({
return module
.exports({
pkg: 'node',
ver: '',
os: 'macos',

View File

@@ -72,8 +72,8 @@ gc "feat: new feature"
### Common aliases
Use *alias*es to make other tools you find around webi even _more_ convenient ⚡️
(and powerful 💪).
Use *alias*es to make other tools you find around webi even _more_ convenient
⚡️ (and powerful 💪).
```sh
aliasman curl 'curlie'

22
aliasman/releases.js Normal file
View File

@@ -0,0 +1,22 @@
'use strict';
let Releases = module.exports;
let GitHubSource = require('../_common/github-source.js');
let owner = 'BeyondCodeBootcamp';
let repo = 'aliasman';
Releases.latest = async function () {
let all = await GitHubSource.getDistributables({ owner, repo });
for (let pkg of all.releases) {
pkg.os = 'posix_2017';
}
return all;
};
if (module === require.main) {
Releases.latest().then(function (all) {
all = require('../_webi/normalize.js')(all);
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -19,13 +19,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading archiver from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing archiver"
# TODO: create package-specific temp directory

21
arc/releases.js Normal file
View File

@@ -0,0 +1,21 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'mholt';
var repo = 'archiver';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
all._names = ['archiver', 'arc'];
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
// just select the first 5 for demonstration
all.releases = all.releases.slice(0, 5);
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -1,5 +1,5 @@
#!/bin/pwsh
Write-Output "'archiver@$Env:WEBI_TAG' is an alias for 'arc@$Env:WEBI_VERSION'"
if ($null -eq $Env:WEBI_HOST -or $Env:WEBI_HOST -eq "") { $Env:WEBI_HOST = "https://webinstall.dev" }
IF ($null -eq $Env:WEBI_HOST -or $Env:WEBI_HOST -eq "") { $Env:WEBI_HOST = "https://webinstall.dev" }
curl.exe -A MS -fsSL "$Env:WEBI_HOST/arc@$Env:WEBI_VERSION" | powershell

View File

@@ -20,7 +20,7 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Checking for (or Installing) MSVC Runtime..."
& "$Env:USERPROFILE\.local\bin\webi-pwsh.ps1" vcruntime
@@ -29,7 +29,7 @@ if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing AtomicParsley"
# TODO: create package-specific temp directory

79
atomicparsley/releases.js Normal file
View File

@@ -0,0 +1,79 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'wez';
var repo = 'atomicparsley';
let targets = {
x86win: {
os: 'windows',
arch: 'x86',
libc: 'msvc',
},
x64win: {
os: 'windows',
arch: 'amd64',
// https://github.com/wez/atomicparsley/issues/6#issuecomment-1364523028
libc: 'msvc',
},
x64mac: {
os: 'macos',
arch: 'amd64',
},
x64lin: {
os: 'linux',
arch: 'amd64',
libc: 'gnu',
},
x64musl: {
os: 'linux',
arch: 'amd64',
libc: 'musl',
},
};
module.exports = function () {
return github(null, owner, repo).then(function (all) {
for (let rel of all.releases) {
let windows32 = rel.name.includes('WindowsX86.');
if (windows32) {
Object.assign(rel, targets.x86win);
continue;
}
let windows64 = rel.name.includes('Windows.');
if (windows64) {
Object.assign(rel, targets.x64win);
continue;
}
let macos64 = rel.name.includes('MacOS');
if (macos64) {
Object.assign(rel, targets.x64mac);
continue;
}
let musl64 = rel.name.includes('Alpine');
if (musl64) {
Object.assign(rel, targets.x64musl);
continue;
}
let lin64 = rel.name.includes('Linux.');
if (lin64) {
Object.assign(rel, targets.x64lin);
continue;
}
}
all._names = ['AtomicParsley', 'atomicparsley'];
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
console.info(JSON.stringify(all));
//console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -20,13 +20,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading awless from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing awless"
# TODO: create package-specific temp directory

22
awless/releases.js Normal file
View File

@@ -0,0 +1,22 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'wallix';
var repo = 'awless';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
// remove checksums and .deb
all.releases = all.releases.filter(function (rel) {
return !/(\.txt)|(\.deb)$/i.test(rel.name);
});
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
console.info(JSON.stringify(all));
});
}

View File

@@ -3,7 +3,7 @@
$VERNAME = "$Env:PKG_NAME-v$Env:WEBI_VERSION.exe"
$EXENAME = "$Env:PKG_NAME.exe"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading $Env:PKG_NAME from $Env:WEBI_PKG_URL to $Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part"
& Move-Item "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part" "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
@@ -11,11 +11,11 @@ if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
# Fetch MSVC Runtime
Write-Output "Checking for MSVC Runtime..."
if (-not (Test-Path "\Windows\System32\vcruntime140.dll")) {
IF (-not (Test-Path "\Windows\System32\vcruntime140.dll")) {
& "$Env:USERPROFILE\.local\bin\webi-pwsh.ps1" vcruntime
}
if (!(Test-Path -Path "$Env:USERPROFILE\.local\bin\$VERNAME")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\.local\bin\$VERNAME")) {
Write-Output "Installing $Env:PKG_NAME"
# TODO: temp directory

View File

@@ -28,22 +28,6 @@ __init_bat() {
# chmod a+x ~/.local/opt/bat-v0.15.4/bin/bat
chmod a+x "$pkg_src_cmd"
# install completions if present (autocomplete/)
if test -d ./bat-*/autocomplete; then
mkdir -p "$pkg_src_dir/share/bash-completion/completions"
mkdir -p "$pkg_src_dir/share/fish/vendor_completions.d"
mkdir -p "$pkg_src_dir/share/zsh/site-functions"
mv ./bat-*/autocomplete/bat.bash "$pkg_src_dir/share/bash-completion/completions/bat" 2>/dev/null || true
mv ./bat-*/autocomplete/bat.fish "$pkg_src_dir/share/fish/vendor_completions.d/bat.fish" 2>/dev/null || true
mv ./bat-*/autocomplete/bat.zsh "$pkg_src_dir/share/zsh/site-functions/_bat" 2>/dev/null || true
fi
# install man page if present
if test -f ./bat-*/bat.1; then
mkdir -p "$pkg_src_dir/share/man/man1"
mv ./bat-*/bat.1 "$pkg_src_dir/share/man/man1/bat.1"
fi
if ! [ -e ~/.config/bat/config ]; then
mkdir -p ~/.config/bat/
touch ~/.config/bat/config

20
bat/releases.js Normal file
View File

@@ -0,0 +1,20 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'sharkdp';
var repo = 'bat';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
all.releases = all.releases.slice(0, 10);
//console.info(JSON.stringify(all));
console.info(JSON.stringify(all, null, 2));
});
}

47
bun/releases.js Normal file
View File

@@ -0,0 +1,47 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'oven-sh';
var repo = 'bun';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
all.releases = all.releases
.filter(function (r) {
let isDebug = r.name.includes('-profile');
if (isDebug) {
return false;
}
let isAncient = r.name.includes('-baseline');
if (isAncient) {
return false;
}
let isMusl = r.name.includes('-musl');
if (isMusl) {
r._musl = true;
r.libc = 'musl';
} else if (r.os === 'linux') {
r.libc = 'gnu';
}
return true;
})
.map(function (r) {
// bun-v0.5.1 => v0.5.1
r.version = r.version.replace(/bun-/g, '');
return r;
});
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
// just select the first 5 for demonstration
all.releases = all.releases.slice(0, 5);
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -1181,8 +1181,7 @@ To prevent search engine and browser confusion
- _DO NOT_ prevent crawling via `robots.txt` \
(counter-intuitive, but pages _must_ be crawled for links to _NOT_ be indexed)
- _all_ domains using public TLS certs _will_ be indexed by default \
(they are all linked to and crawled from various Certificate Transparency
reports)
(they are all linked to and crawled from various Certificate Transparency reports)
- follow these guidelines even if the dev sites use HTTP Basic Auth
```Caddyfile

View File

@@ -20,13 +20,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading caddy from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing caddy"
# TODO: create package-specific temp directory

27
caddy/releases.js Normal file
View File

@@ -0,0 +1,27 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'caddyserver';
var repo = 'caddy';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
// remove checksums and .deb
all.releases = all.releases.filter(function (rel) {
let isOneOffAsset = rel.download.includes('buildable-artifact');
if (isOneOffAsset) {
return false;
}
return true;
});
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
console.info(JSON.stringify(all));
});
}

View File

@@ -19,13 +19,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading chromedriver from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing chromedriver"
# TODO: create package-specific temp directory

101
chromedriver/releases.js Normal file
View File

@@ -0,0 +1,101 @@
'use strict';
let Fetcher = require('../_common/fetcher.js');
// See <https://googlechromelabs.github.io/chrome-for-testing/>
const releaseApiUrl =
'https://googlechromelabs.github.io/chrome-for-testing/known-good-versions-with-downloads.json';
// {
// "timestamp": "2023-11-15T21:08:56.730Z",
// "versions": [
// {
// "version": "121.0.6120.0",
// "revision": "1222902",
// "downloads": {
// "chrome": [],
// "chromedriver": [
// {
// "platform": "linux64",
// "url": "https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/121.0.6120.0/linux64/chromedriver-linux64.zip"
// },
// {
// "platform": "mac-arm64",
// "url": "https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/121.0.6120.0/mac-arm64/chromedriver-mac-arm64.zip"
// },
// {
// "platform": "mac-x64",
// "url": "https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/121.0.6120.0/mac-x64/chromedriver-mac-x64.zip"
// },
// {
// "platform": "win32",
// "url": "https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/121.0.6120.0/win32/chromedriver-win32.zip"
// },
// {
// "platform": "win64",
// "url": "https://edgedl.me.gvt1.com/edgedl/chrome/chrome-for-testing/121.0.6120.0/win64/chromedriver-win64.zip"
// }
// ],
// "chrome-headless-shell": []
// }
// }
// ]
// }
module.exports = async function () {
let resp;
try {
resp = await Fetcher.fetch(releaseApiUrl, {
headers: { Accept: 'application/json' },
});
} catch (e) {
/** @type {Error & { code: string, response: { status: number, body: string } }} */ //@ts-expect-error
let err = e;
if (err.code === 'E_FETCH_RELEASES') {
err.message = `failed to fetch 'chromedriver' release data: ${err.response.status} ${err.response.body}`;
}
throw e;
}
let data = JSON.parse(resp.body);
let builds = [];
for (let release of data.versions) {
if (!release.downloads.chromedriver) {
continue;
}
let version = release.version;
for (let asset of release.downloads.chromedriver) {
let build = {
version: version,
download: asset.url,
// I' not sure that this is actually statically built but it
// seems to be and at worst we'll just get bug reports for Alpine
libc: 'none',
};
builds.push(build);
}
}
let all = {
download: '',
releases: builds,
};
return all;
};
if (module === require.main) {
module
.exports()
.then(function (all) {
all = require('../_webi/normalize.js')(all);
// just select the latest 20 for demonstration
all.releases = all.releases.slice(-20);
console.info(JSON.stringify(all, null, 2));
})
.catch(function (err) {
console.error('Error:', err);
});
}

View File

@@ -17,13 +17,13 @@ $pkg_src = "$pkg_src_cmd"
New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading cilium from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing cilium"
Push-Location .local\tmp

21
cilium/releases.js Normal file
View File

@@ -0,0 +1,21 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'cilium';
var repo = 'cilium-cli';
module.exports = async function () {
let all = await github(null, owner, repo);
return all;
};
if (module === require.main) {
(async function () {
let normalize = require('../_webi/normalize.js');
let all = await module.exports();
all = normalize(all);
// just select the first 5 for demonstration
all.releases = all.releases.slice(0, 5);
console.info(JSON.stringify(all, null, 2));
})();
}

View File

@@ -20,13 +20,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$pkg_download")) {
IF (!(Test-Path -Path "$pkg_download")) {
Write-Output "Downloading cmake from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_dir")) {
IF (!(Test-Path -Path "$pkg_src_dir")) {
Write-Output "Installing cmake"
# TODO: create package-specific temp directory

53
cmake/releases.js Normal file
View File

@@ -0,0 +1,53 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'Kitware';
var repo = 'CMake';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
for (let rel of all.releases) {
if (rel.version.startsWith('v')) {
rel._version = rel.version.slice(1);
}
{
let linuxRe = /(\b|_)(linux|gnu)(\b|_)/i;
let isLinux = linuxRe.test(rel.download) || linuxRe.test(rel.name);
if (isLinux) {
let muslRe = /(\b|_)(musl|alpine)(\b|_)/i;
let isMusl = muslRe.test(rel.download) || muslRe.test(rel.name);
if (isMusl) {
rel.libc = 'musl';
} else {
rel.libc = 'gnu';
}
continue;
}
}
{
let windowsRe = /(\b|_)(win\d*|windows\d*)(\b|_)/i;
let isWindows =
windowsRe.test(rel.download) || windowsRe.test(rel.name);
if (isWindows) {
rel.libc = 'msvc';
continue;
}
}
}
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
// just select the first 5 for demonstration
all.releases = all.releases.slice(0, 5);
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -3,13 +3,13 @@
$VERNAME = "$Env:PKG_NAME-v$Env:WEBI_VERSION.exe"
$EXENAME = "$Env:PKG_NAME.exe"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading $Env:PKG_NAME from $Env:WEBI_PKG_URL to $Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part"
& Move-Item "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part" "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
}
if (!(Test-Path -Path "$Env:USERPROFILE\.local\opt\$Env:PKG_NAME-v$Env:WEBI_VERSION\bin\$VERNAME")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\.local\opt\$Env:PKG_NAME-v$Env:WEBI_VERSION\bin\$VERNAME")) {
Write-Output "Installing $Env:PKG_NAME"
# TODO: temp directory

40
comrak/releases.js Normal file
View File

@@ -0,0 +1,40 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'kivikakk';
var repo = 'comrak';
var ODDITIES = ['-musleabihf.1-'];
module.exports = function () {
return github(null, owner, repo).then(function (all) {
let builds = [];
loopBuilds: for (let build of all.releases) {
let isOddity;
for (let oddity of ODDITIES) {
isOddity = build.name.includes(oddity);
if (isOddity) {
break;
}
}
if (isOddity) {
continue;
}
builds.push(build);
}
all.releases = builds;
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
all.releases = all.releases.slice(0, 10);
//console.info(JSON.stringify(all));
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -20,13 +20,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$pkg_download")) {
IF (!(Test-Path -Path "$pkg_download")) {
Write-Output "Downloading crabz from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing crabz"
# TODO: create package-specific temp directory

31
crabz/releases.js Normal file
View File

@@ -0,0 +1,31 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'sstadick';
var repo = 'crabz';
module.exports = async function () {
let all = await github(null, owner, repo);
let releases = [];
for (let rel of all.releases) {
let isSrc = rel.download.includes('-src.');
if (isSrc) {
continue;
}
releases.push(rel);
}
all.releases = releases;
return all;
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
// just select the first 5 for demonstration
all.releases = all.releases.slice(0, 5);
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -3,13 +3,13 @@
$VERNAME = "$Env:PKG_NAME-v$Env:WEBI_VERSION.exe"
$EXENAME = "$Env:PKG_NAME.exe"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading $Env:PKG_NAME from $Env:WEBI_PKG_URL to $Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part"
& Move-Item "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part" "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
}
if (!(Test-Path -Path "$Env:USERPROFILE\.local\bin\$VERNAME")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\.local\bin\$VERNAME")) {
Write-Output "Installing $Env:PKG_NAME"
# TODO: temp directory

21
curlie/releases.js Normal file
View File

@@ -0,0 +1,21 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'rs';
var repo = 'curlie';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
all._names = ['curlie', 'curl-httpie'];
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
all.releases = all.releases.slice(0, 10);
//console.info(JSON.stringify(all));
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -147,8 +147,8 @@ dash-qt \
CoinJoin aids in preventing some bad actors and malicious observers being able
to easily reconstruct details about your transactions from the publicly
available data by creating many excess transactions. \
(be aware, however, that dedicated bad actors can use sophisticated software
that will reveal much of the same information over time)
(be aware, however, that dedicated bad actors can use sophisticated software that
will reveal much of the same information over time)
`dash-qt` does not enable CoinJoin mixing by default.

View File

@@ -20,13 +20,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading dashcore from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_dir")) {
IF (!(Test-Path -Path "$pkg_src_dir")) {
Write-Output "Installing dashcore"
# TODO: create package-specific temp directory

31
dashcore/releases.js Normal file
View File

@@ -0,0 +1,31 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'dashpay';
var repo = 'dash';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
all.releases.forEach(function (rel) {
if (rel.name.includes('osx64')) {
rel.os = 'macos';
}
if (rel.version.startsWith('v')) {
rel._version = rel.version.slice(1);
}
});
all._names = ['dashd', 'dashcore'];
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
// just select the first 5 for demonstration
all.releases = all.releases.slice(0, 5);
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -20,13 +20,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading dashd from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_dir")) {
IF (!(Test-Path -Path "$pkg_src_dir")) {
Write-Output "Installing dashd"
# TODO: create package-specific temp directory

3
dashd/releases.js Normal file
View File

@@ -0,0 +1,3 @@
'use strict';
module.exports = require('../dashcore/releases.js');

View File

@@ -19,13 +19,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading dashmsg from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing dashmsg"
# TODO: create package-specific temp directory

18
dashmsg/releases.js Normal file
View File

@@ -0,0 +1,18 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'dashhive';
var repo = 'dashmsg';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -19,13 +19,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading delta from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing delta"
# TODO: create package-specific temp directory

20
delta/releases.js Normal file
View File

@@ -0,0 +1,20 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'dandavison';
var repo = 'delta';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
// just select the first 15 for demonstration
all.releases = all.releases.slice(0, 15);
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -18,7 +18,7 @@ etc).
The obligatory Hello World
```sh
deno run https://docs.deno.com/examples/scripts/hello_world.ts
deno run https://deno.land/std/examples/welcome.ts
```
Run a local file

View File

@@ -1,14 +1,14 @@
#!/usr/bin/env pwsh
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading $Env:PKG_NAME from $Env:WEBI_PKG_URL to $Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
#Invoke-WebRequest https://nodejs.org/dist/v12.16.2/node-v12.16.2-win-x64.zip -OutFile node-v12.16.2-win-x64.zip
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part"
& Move-Item "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part" "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
}
if (!(Test-Path -Path "$Env:USERPROFILE\.local\opt\$Env:PKG_NAME-v$Env:WEBI_VERSION")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\.local\opt\$Env:PKG_NAME-v$Env:WEBI_VERSION")) {
Write-Output "Installing $Env:PKG_NAME"
# TODO: temp directory

42
deno/releases.js Normal file
View File

@@ -0,0 +1,42 @@
'use strict';
var path = require('path');
var github = require('../_common/github.js');
var owner = 'denoland';
var repo = 'deno';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
// remove checksums and .deb
all.releases = all.releases
.filter(function (rel) {
let isMeta = rel.name.endsWith('.d.ts');
if (isMeta) {
return false;
}
return true;
})
.map(function (rel) {
var ext;
if (!rel.name.match(rel.version)) {
ext = path.extname(rel.name);
rel.filename =
rel.name.slice(0, rel.name.length - ext.length) +
'-' +
rel.version +
ext;
}
return rel;
});
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -20,18 +20,18 @@ $pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch MSVC Runtime
Write-Output "Checking for MSVC Runtime..."
if (-not (Test-Path "\Windows\System32\vcruntime140.dll")) {
IF (-not (Test-Path "\Windows\System32\vcruntime140.dll")) {
& "$Env:USERPROFILE\.local\bin\webi-pwsh.ps1" vcruntime
}
# Fetch archive
if (!(Test-Path -Path "$pkg_download")) {
IF (!(Test-Path -Path "$pkg_download")) {
Write-Output "Downloading dotenv-linter from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing dotenv-linter"
# TODO: create package-specific temp directory

20
dotenv-linter/releases.js Normal file
View File

@@ -0,0 +1,20 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'dotenv-linter';
var repo = 'dotenv-linter';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
// just select the first 5 for demonstration
all.releases = all.releases.slice(0, 5);
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -19,13 +19,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading dotenv from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing dotenv"
# TODO: create package-specific temp directory

18
dotenv/releases.js Normal file
View File

@@ -0,0 +1,18 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'therootcompany';
var repo = 'dotenv';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
console.info(JSON.stringify(all));
});
}

22
duckdns.sh/releases.js Normal file
View File

@@ -0,0 +1,22 @@
'use strict';
let Releases = module.exports;
let GitHubSource = require('../_common/github-source.js');
let owner = 'BeyondCodeBootcamp';
let repo = 'DuckDNS.sh';
Releases.latest = async function () {
let all = await GitHubSource.getDistributables({ owner, repo });
for (let pkg of all.releases) {
pkg.os = 'posix_2017';
}
return all;
};
if (module === require.main) {
Releases.latest().then(function (all) {
all = require('../_webi/normalize.js')(all);
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -3,13 +3,13 @@
$VERNAME = "$Env:PKG_NAME-v$Env:WEBI_VERSION.exe"
$EXENAME = "$Env:PKG_NAME.exe"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading $Env:PKG_NAME from $Env:WEBI_PKG_URL to $Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part"
& Move-Item "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part" "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
}
if (!(Test-Path -Path "$Env:USERPROFILE\.local\bin\$VERNAME")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\.local\bin\$VERNAME")) {
Write-Output "Installing $Env:PKG_NAME"
# TODO: temp directory

View File

@@ -27,22 +27,6 @@ __init_fd() {
# chmod a+x "$HOME/.local/opt/fd-v8.1.1/bin/fd"
chmod a+x "$pkg_src_cmd"
# install completions if present (autocomplete/{fd.bash,fd.fish,_fd})
if test -d ./fd-*/autocomplete; then
mkdir -p "$pkg_src_dir/share/bash-completion/completions"
mkdir -p "$pkg_src_dir/share/fish/vendor_completions.d"
mkdir -p "$pkg_src_dir/share/zsh/site-functions"
mv ./fd-*/autocomplete/fd.bash "$pkg_src_dir/share/bash-completion/completions/fd" 2>/dev/null || true
mv ./fd-*/autocomplete/fd.fish "$pkg_src_dir/share/fish/vendor_completions.d/fd.fish" 2>/dev/null || true
mv ./fd-*/autocomplete/_fd "$pkg_src_dir/share/zsh/site-functions/_fd" 2>/dev/null || true
fi
# install man page if present
if test -f ./fd-*/fd.1; then
mkdir -p "$pkg_src_dir/share/man/man1"
mv ./fd-*/fd.1 "$pkg_src_dir/share/man/man1/fd.1"
fi
}
}

31
fd/releases.js Normal file
View File

@@ -0,0 +1,31 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'sharkdp';
var repo = 'fd';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
let builds = [];
for (let build of all.releases) {
if (build.name === 'fd') {
continue;
}
builds.push(build);
}
all.releases = builds;
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
all.releases = all.releases.slice(0, 10);
//console.info(JSON.stringify(all));
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -19,13 +19,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading ffmpeg from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
Move-Item -Path "$pkg_download.part" -Destination "$pkg_download" -Force
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing ffmpeg"
# TODO: create package-specific temp directory

45
ffmpeg/releases.js Normal file
View File

@@ -0,0 +1,45 @@
'use strict';
var path = require('path');
var github = require('../_common/github.js');
var owner = 'eugeneware';
var repo = 'ffmpeg-static';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
all.releases = all.releases
.filter(function (rel) {
let isFfmpeg = rel.name.includes('ffmpeg');
if (!isFfmpeg) {
return;
}
// remove README and LICENSE
return !['.README', '.LICENSE'].includes(path.extname(rel.name));
})
.map(function (rel) {
rel.version = rel.version.replace(/^b/, '');
if (/win32/.test(rel.name)) {
rel.os = 'windows';
rel.ext = 'exe';
}
if (/ia32/.test(rel.name)) {
rel.arch = '386';
} else if (/x64/.test(rel.name)) {
rel.arch = 'amd64';
}
return rel;
});
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
console.info(JSON.stringify(all));
});
}

View File

@@ -20,13 +20,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (-not (Test-Path -Path "$pkg_download")) {
IF (-Not (Test-Path -Path "$pkg_download")) {
Write-Output "Downloading ffuf from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (-not (Test-Path -Path "$pkg_src_cmd")) {
IF (-Not (Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing ffuf"
# TODO: create package-specific temp directory

20
ffuf/releases.js Normal file
View File

@@ -0,0 +1,20 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'ffuf';
var repo = 'ffuf';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
// just select the first 5 for demonstration
all.releases = all.releases.slice(0, 5);
console.info(JSON.stringify(all, null, 2));
});
}

39
fish/releases.js Normal file
View File

@@ -0,0 +1,39 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'fish-shell';
var repo = 'fish-shell';
var ODDITIES = ['bundledpcre'];
module.exports = function () {
return github(null, owner, repo).then(function (all) {
all.releases = all.releases
.map(function (rel) {
for (let oddity of ODDITIES) {
let isOddity = rel.name.includes(oddity);
if (isOddity) {
return;
}
}
// We can extract the macos bins from the .app
if (/\.app\.zip$/.test(rel.name)) {
rel.os = 'macos';
rel.arch = 'amd64';
return rel;
}
})
.filter(Boolean);
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
// just select the first 5 for demonstration
all.releases = all.releases.slice(0, 5);
console.info(JSON.stringify(all, null, 2));
});
}

132
flutter/releases.js Normal file
View File

@@ -0,0 +1,132 @@
'use strict';
let Fetcher = require('../_common/fetcher.js');
let FLUTTER_OSES = ['macos', 'linux', 'windows'];
/**
* stable, beta, dev
* @type {Object.<String, Boolean>}
*/
let channelMap = {};
// This can be spot-checked against
// https://docs.flutter.dev/release/archive?tab=windows
// The release URLs are
// - https://storage.googleapis.com/flutter_infra_release/releases/releases_macos.json
// - https://storage.googleapis.com/flutter_infra_release/releases/releases_linux.json
// - https://storage.googleapis.com/flutter_infra_release/releases/releases_windows.json
// The old release URLs are
// - https://storage.googleapis.com/flutter_infra/releases/releases_macos.json
// - https://storage.googleapis.com/flutter_infra/releases/releases_linux.json
// - https://storage.googleapis.com/flutter_infra/releases/releases_windows.json
// The data looks like
// {
// "base_url": "https://storage.googleapis.com/flutter_infra/releases",
// "current_release": {
// "beta": "b22742018b3edf16c6cadd7b76d9db5e7f9064b5",
// "dev": "fa5883b78e566877613ad1ccb48dd92075cb5c23",
// "stable": "02c026b03cd31dd3f867e5faeb7e104cce174c5f"
// },
// "releases": [
// {
// "hash": "fa5883b78e566877613ad1ccb48dd92075cb5c23",
// "channel": "dev",
// "version": "2.3.0-16.0.pre",
// "release_date": "2021-05-27T23:58:47.683121Z",
// "archive": "dev/macos/flutter_macos_2.3.0-16.0.pre-dev.zip",
// "sha256": "f572b42d36714e6c58a3ed170b93bb414e2ced3ca4bde5094fbe18061cbcba6c"
// },
// {
// "hash": "02c026b03cd31dd3f867e5faeb7e104cce174c5f",
// "channel": "stable",
// "version": "2.2.1",
// "release_date": "2021-05-27T23:06:07.243882Z",
// "archive": "stable/macos/flutter_macos_2.2.1-stable.zip",
// "sha256": "6373d39ec563c337600baf42a42b258420208e4523d85479373e113d61d748df"
// },
// {
// "hash": "b22742018b3edf16c6cadd7b76d9db5e7f9064b5",
// "channel": "beta",
// "version": "2.2.0",
// "release_date": "2021-05-19T21:14:59.281482Z",
// "archive": "beta/macos/flutter_macos_2.2.0-beta.zip",
// "sha256": "31ab530e708f8d1274712211253a27a4ce7d676f139d30f2ec021df22382f052"
// }
// ]
// }
/**
* @typedef BuildInfo
* @prop {String} version
* @prop {String} [_version]
* @prop {Boolean} lts
* @prop {String} channel
* @prop {String} date
* @prop {String} download
* @prop {String} [_filename]
*/
module.exports = async function () {
let all = {
download: '',
/** @type {Array<BuildInfo>} */
releases: [],
/** @type {Array<String>} */
channels: [],
};
for (let osname of FLUTTER_OSES) {
let resp;
try {
let url = `https://storage.googleapis.com/flutter_infra_release/releases/releases_${osname}.json`;
resp = await Fetcher.fetch(url, {
headers: { Accept: 'application/json' },
});
} catch (e) {
/** @type {Error & { code: string, response: { status: number, body: string } }} */ //@ts-expect-error
let err = e;
if (err.code === 'E_FETCH_RELEASES') {
err.message = `failed to fetch 'flutter' release data for ${osname}: ${err.response.status} ${err.response.body}`;
}
throw e;
}
let data = JSON.parse(resp.body);
let osBaseUrl = data.base_url;
let osReleases = data.releases;
for (let asset of osReleases) {
if (!channelMap[asset.channel]) {
channelMap[asset.channel] = true;
}
all.releases.push({
version: asset.version,
_version: `${asset.version}-${asset.channel}`,
lts: false,
channel: asset.channel,
date: asset.release_date.replace(/T.*/, ''),
download: `${osBaseUrl}/${asset.archive}`,
_filename: asset.archive,
});
}
}
all.channels = Object.keys(channelMap);
// note: versions have a waterfall relationship with channels:
// - a release that is in beta today may become stable tomorrow
// - semver prereleases are either beta or dev
return all;
};
if (module === require.main) {
module.exports().then(function (all) {
all.releases = all.releases.slice(25);
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -3,13 +3,13 @@
$VERNAME = "$Env:PKG_NAME-v$Env:WEBI_VERSION.exe"
$EXENAME = "$Env:PKG_NAME.exe"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading $Env:PKG_NAME from $Env:WEBI_PKG_URL to $Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part"
& Move-Item "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE.part" "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
}
if (!(Test-Path -Path "$Env:USERPROFILE\.local\bin\$VERNAME")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\.local\bin\$VERNAME")) {
Write-Output "Installing $Env:PKG_NAME"
# TODO: temp directory

20
fzf/releases.js Normal file
View File

@@ -0,0 +1,20 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'junegunn';
var repo = 'fzf';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
all.releases = all.releases.slice(0, 10);
//console.info(JSON.stringify(all));
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -19,13 +19,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading gh from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing gh"
# TODO: create package-specific temp directory

View File

@@ -25,12 +25,6 @@ __init_gh() {
# mv ./gh-*/gh ~/.local/opt/gh-v0.99.9/bin/gh
mv ./"$pkg_cmd_name"*/bin/gh "$pkg_src_cmd"
# install man pages if present
if test -d ./"$pkg_cmd_name"*/share/man; then
mkdir -p "$pkg_src_dir/share"
mv ./"$pkg_cmd_name"*/share/man "$pkg_src_dir/share/man"
fi
}
# pkg_get_current_version is recommended, but (soon) not required

20
gh/releases.js Normal file
View File

@@ -0,0 +1,20 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'cli';
var repo = 'cli';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
// just select the first 5 for demonstration
all.releases = all.releases.slice(0, 5);
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -1,5 +1,5 @@
#!/bin/pwsh
Write-Output "'git-gpg-init@$Env:WEBI_TAG' is an alias for 'git-config-gpg@$Env:WEBI_VERSION'"
if ($null -eq $Env:WEBI_HOST -or $Env:WEBI_HOST -eq "") { $Env:WEBI_HOST = "https://webinstall.dev" }
IF ($null -eq $Env:WEBI_HOST -or $Env:WEBI_HOST -eq "") { $Env:WEBI_HOST = "https://webinstall.dev" }
curl.exe -A MS -fsSL "$Env:WEBI_HOST/git-config-gpg@$Env:WEBI_VERSION" | powershell

View File

@@ -11,13 +11,13 @@ $pkg_dst_cmd = "$pkg_dst\cmd\$pkg_cmd_name"
$pkg_dst_bin = "$pkg_dst\cmd"
# Fetch archive
if (!(Test-Path -Path "$pkg_download")) {
IF (!(Test-Path -Path "$pkg_download")) {
Write-Output "Downloading $Env:PKG_NAME from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src")) {
IF (!(Test-Path -Path "$pkg_src")) {
Write-Output "Installing $pkg_cmd_name"
# TODO: temp directory
@@ -30,12 +30,12 @@ if (!(Test-Path -Path "$pkg_src")) {
# Unpack archive
# Windows BSD-tar handles zip. Imagine that.
Write-Output "Unpacking $pkg_download"
if (!(Test-Path -Path "$pkg_cmd_name-v$Env:WEBI_VERSION")) {
IF (!(Test-Path -Path "$pkg_cmd_name-v$Env:WEBI_VERSION")) {
New-Item -Path "$pkg_cmd_name-v$Env:WEBI_VERSION" -ItemType Directory -Force | Out-Null
}
($none = Push-Location "$pkg_cmd_name-v$Env:WEBI_VERSION") | Out-Null
($none = Push-Location "$pkg_cmd_name-v$Env:WEBI_VERSION") | Out-Null
& tar xf "$pkg_download"
($none = Pop-Location) | Out-Null
($none = Pop-Location) | Out-Null
# Settle unpacked archive into place
Write-Output "New Name: $pkg_cmd_name-v$Env:WEBI_VERSION"

33
git/releases.js Normal file
View File

@@ -0,0 +1,33 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'git-for-windows';
var repo = 'git';
module.exports = function () {
// TODO support mac and linux tarballs
return github(null, owner, repo).then(function (all) {
// See https://github.com/git-for-windows/git/wiki/MinGit
// also consider https://github.com/git-for-windows/git/wiki/Silent-or-Unattended-Installation
all.releases = all.releases
.filter(function (rel) {
rel.os = 'windows';
rel._version = rel.version.replace(/\.windows.1.*/, '');
rel._version = rel._version.replace(/\.windows(\.\d)/, '$1');
return (
/MinGit/i.test(rel.name || rel.download) &&
!/busybox/i.test(rel.name || rel.download)
);
})
.slice(0, 20);
all._names = ['MinGit', 'git'];
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -19,13 +19,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading gitdeploy from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing gitdeploy"
# TODO: create package-specific temp directory

18
gitdeploy/releases.js Normal file
View File

@@ -0,0 +1,18 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'therootcompany';
var repo = 'gitdeploy';
module.exports = function () {
return github(null, owner, repo).then(function (all) {
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
console.info(JSON.stringify(all));
});
}

View File

@@ -19,19 +19,19 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
Write-Output "Checking for Git..."
if (-not (Get-Command -Name "git" -ErrorAction Silent)) {
IF (-Not (Get-Command -Name "git" -ErrorAction Silent)) {
& "$HOME\.local\bin\webi-pwsh.ps1" git
$null = Sync-EnvPath
}
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading gitea from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing gitea"
# TODO: create package-specific temp directory

34
gitea/releases.js Normal file
View File

@@ -0,0 +1,34 @@
'use strict';
var github = require('../_common/github.js');
var owner = 'go-gitea';
var repo = 'gitea';
var ODDITIES = ['-gogit-', '-docs-'];
module.exports = function () {
return github(null, owner, repo).then(function (all) {
// remove checksums and .deb
all.releases = all.releases.filter(function (rel) {
for (let oddity of ODDITIES) {
let isOddity = rel.name.includes(oddity);
if (isOddity) {
return false;
}
}
return true;
});
// "windows-4.0" as a nod to Windows NT ¯\_(ツ)_/¯
all._names = ['gitea', '-4.0-'];
return all;
});
};
if (module === require.main) {
module.exports().then(function (all) {
all = require('../_webi/normalize.js')(all);
console.info(JSON.stringify(all));
});
}

View File

@@ -16,13 +16,13 @@ if (!(Get-Command "git.exe" -ErrorAction SilentlyContinue)) {
}
# Fetch archive
if (!(Test-Path -Path "$pkg_download")) {
IF (!(Test-Path -Path "$pkg_download")) {
Write-Output "Downloading $Env:PKG_NAME from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src")) {
IF (!(Test-Path -Path "$pkg_src")) {
Write-Output "Installing $pkg_cmd_name"
# TODO: temp directory
@@ -50,7 +50,7 @@ if (!(Test-Path -Path "$pkg_src")) {
Write-Output "Copying into '$pkg_dst' from '$pkg_src'"
Remove-Item -Path "$pkg_dst" -Recurse -ErrorAction Ignore
Copy-Item -Path "$pkg_src" -Destination "$pkg_dst" -Recurse
if (!(Test-Path -Path go\bin)) { New-Item -Path go\bin -ItemType Directory -Force | Out-Null }
IF (!(Test-Path -Path go\bin)) { New-Item -Path go\bin -ItemType Directory -Force | Out-Null }
# Add to path
webi_path_add ~/.local/opt/go/bin

130
go/releases.js Normal file
View File

@@ -0,0 +1,130 @@
'use strict';
let Fetcher = require('../_common/fetcher.js');
/** @type {Object.<String, String>} */
let osMap = {
darwin: 'macos',
};
/** @type {Object.<String, String>} */
let archMap = {
386: 'x86',
};
let ODDITIES = ['bootstrap', '-arm6.'];
/**
* @param {String} filename
*/
function isOdd(filename) {
for (let oddity of ODDITIES) {
let isOddity = filename.includes(oddity);
if (isOddity) {
return true;
}
}
}
/**
* @typedef BuildInfo
* @prop {String} version
* @prop {String} [_version]
* @prop {String} arch
* @prop {String} channel
* @prop {String} date
* @prop {String} download
* @prop {String} ext
* @prop {String} [_filename]
* @prop {String} hash
* @prop {Boolean} lts
* @prop {String} os
*/
async function getDistributables() {
/*
{
version: 'go1.13.8',
stable: true,
files: [
{
filename: 'go1.13.8.src.tar.gz',
os: '',
arch: '',
version: 'go1.13.8',
sha256:
'b13bf04633d4d8cf53226ebeaace8d4d2fd07ae6fa676d0844a688339debec34',
size: 21631178,
kind: 'source'
}
]
};
*/
let resp;
try {
let url = 'https://golang.org/dl/?mode=json&include=all';
resp = await Fetcher.fetch(url, {
headers: { Accept: 'application/json' },
});
} catch (e) {
/** @type {Error & { code: string, response: { status: number, body: string } }} */ //@ts-expect-error
let err = e;
if (err.code === 'E_FETCH_RELEASES') {
err.message = `failed to fetch 'Go' release data: ${err.response.status} ${err.response.body}`;
}
throw e;
}
let goReleases = JSON.parse(resp.body);
let all = {
/** @type {Array<BuildInfo>} */
releases: [],
download: '',
};
for (let release of goReleases) {
// Strip 'go' prefix, standardize version
let parts = release.version.slice(2).split('.');
while (parts.length < 3) {
parts.push('0');
}
let version = parts.join('.');
let fileversion = release.version.slice(2);
for (let asset of release.files) {
if (isOdd(asset.filename)) {
continue;
}
let filename = asset.filename;
let os = osMap[asset.os] || asset.os || '-';
let arch = archMap[asset.arch] || asset.arch || '-';
let build = {
version: version,
_version: fileversion,
lts: (parts[0] > 0 && release.stable) || false,
channel: (release.stable && 'stable') || 'beta',
date: '1970-01-01', // the world may never know
os: os,
arch: arch,
ext: '', // let normalize run the split/test/join
hash: '-', // not ready to standardize this yet
download: `https://dl.google.com/go/${filename}`,
};
all.releases.push(build);
}
}
return all;
}
module.exports = getDistributables;
if (module === require.main) {
getDistributables().then(function (all) {
all = require('../_webi/normalize.js')(all);
//@ts-expect-error
all.releases = all.releases.slice(0, 10);
console.info(JSON.stringify(all, null, 2));
});
}

View File

@@ -16,13 +16,13 @@ if (!(Get-Command "git.exe" -ErrorAction SilentlyContinue)) {
}
# Fetch archive
if (!(Test-Path -Path "$pkg_download")) {
IF (!(Test-Path -Path "$pkg_download")) {
Write-Output "Downloading $Env:PKG_NAME from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src")) {
IF (!(Test-Path -Path "$pkg_src")) {
Write-Output "Installing $pkg_cmd_name"
# TODO: temp directory
@@ -50,7 +50,7 @@ if (!(Test-Path -Path "$pkg_src")) {
Write-Output "Copying into '$pkg_dst' from '$pkg_src'"
Remove-Item -Path "$pkg_dst" -Recurse -ErrorAction Ignore
Copy-Item -Path "$pkg_src" -Destination "$pkg_dst" -Recurse
if (!(Test-Path -Path go\bin)) { New-Item -Path go\bin -ItemType Directory -Force | Out-Null }
IF (!(Test-Path -Path go\bin)) { New-Item -Path go\bin -ItemType Directory -Force | Out-Null }
# Add to path
webi_path_add ~/.local/opt/go/bin

3
golang/releases.js Normal file
View File

@@ -0,0 +1,3 @@
'use strict';
module.exports = require('../go/releases.js');

View File

@@ -122,8 +122,8 @@ goreleaser --clean
Check the console output to make sure that there are no messages about a failed
publish. \
If all is well you should see the git tag on the releases page updated with a
ChangeLog and the published binaries.
If all is well you should the git tag on the releases page updated with a ChangeLog
and the published binaries.
### How to Publish to Gitea and others

View File

@@ -19,13 +19,13 @@ New-Item "$Env:USERPROFILE\Downloads\webi" -ItemType Directory -Force | Out-Null
$pkg_download = "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE"
# Fetch archive
if (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
IF (!(Test-Path -Path "$Env:USERPROFILE\Downloads\webi\$Env:WEBI_PKG_FILE")) {
Write-Output "Downloading goreleaser from $Env:WEBI_PKG_URL to $pkg_download"
& curl.exe -A "$Env:WEBI_UA" -fsSL "$Env:WEBI_PKG_URL" -o "$pkg_download.part"
& Move-Item "$pkg_download.part" "$pkg_download"
}
if (!(Test-Path -Path "$pkg_src_cmd")) {
IF (!(Test-Path -Path "$pkg_src_cmd")) {
Write-Output "Installing goreleaser"
# TODO: create package-specific temp directory

View File

@@ -25,23 +25,6 @@ __init_goreleaser() {
# mv ./goreleaser-*/goreleaser ~/.local/opt/goreleaser-v1.21.2/bin/goreleaser
mv ./goreleaser "$pkg_src_cmd"
# install completions if present (completions/{goreleaser.bash,.fish,.zsh})
if test -d ./completions; then
mkdir -p "$pkg_src_dir/share/bash-completion/completions"
mkdir -p "$pkg_src_dir/share/fish/vendor_completions.d"
mkdir -p "$pkg_src_dir/share/zsh/site-functions"
mv ./completions/goreleaser.bash "$pkg_src_dir/share/bash-completion/completions/goreleaser" 2>/dev/null || true
mv ./completions/goreleaser.fish "$pkg_src_dir/share/fish/vendor_completions.d/goreleaser.fish" 2>/dev/null || true
mv ./completions/goreleaser.zsh "$pkg_src_dir/share/zsh/site-functions/_goreleaser" 2>/dev/null || true
fi
# install man page if present (manpages/goreleaser.1.gz)
if test -d ./manpages; then
mkdir -p "$pkg_src_dir/share/man/man1"
mv ./manpages/*.1.gz "$pkg_src_dir/share/man/man1/" 2>/dev/null || true
mv ./manpages/*.1 "$pkg_src_dir/share/man/man1/" 2>/dev/null || true
fi
}
# pkg_get_current_version is recommended, but (soon) not required

Some files were not shown because too many files have changed in this diff Show More