# Production Notes: Node.js Cache-Only Migration

## Current State

The Node.js server no longer fetches from upstream APIs. It reads only from
`_cache/YYYY-MM/{pkg}.json` files generated by the Go `webicached` daemon.
All per-package `releases.js` files and `_common/` fetcher libraries have been
deleted — they are no longer needed.

### Changes Made

- **builds.js**: Removed `freshenRandomPackage()` calls and background refresh
- **builds-cacher.js**: Removed `getLatestBuilds()`, stale re-fetch, and
  `freshenRandomPackage()`. Missing cache files return empty metadata.
- **builds-cacher.js `getProjectTypeByEntry()`**: Replaced `require(releases.js)`
  with cache file existence check. Packages are `valid` if they have a cache
  file, `selfhosted` if they don't.
- **transform-releases.js**: Legacy release API now reads from cache files
  instead of fetching upstream. Pre-classified fields are cleared before
  `normalize()` so the output matches the legacy format (e.g. `darwin` → `macos`,
  versions without `v` prefix).
- **classify-one.js**: Updated to read from cache instead of `releases.js`.
  Fixed hardcoded triplet key.
- **Deleted 94 `{pkg}/releases.js` files** and **8 `_common/*.js` files**

### Data Format Note

The Go cache uses different OS/arch names than the legacy normalize.js:
- Go: `darwin` → normalize.js: `macos`
- Go: `amd64` → normalize.js: `amd64` (same)
- Go: `aarch64` → normalize.js: `arm64`

For the legacy `/api/releases/` endpoint (`transform-releases.js`), we clear
pre-classified fields and re-normalize from filenames to match production output.

For the installer endpoint (`builds-cacher.js` + `serve-installer.js`), the Go
naming is used directly — the build-classifier handles its own mappings.

### Known Intentional Differences from Production

The Go cache filters releases more aggressively than the old Node normalize.js:
- **Excluded**: `.deb`, `.rpm`, `.sha256`, `.sig`, `.pem`, `.sbom`, `.txt` files
  (non-installable metadata/package-manager assets)
- **OS `unknown`**: Some assets that normalize.js couldn't classify get `os: "unknown"`.
  The Go cache either classifies them correctly or excludes them.
- These are improvements — the filtered results better reflect what webi can
  actually install.

### Libc Taxonomy

| Value  | Meaning |
|--------|---------|
| `none` | Static build — no runtime libc dependency. Often built with musl but fully self-contained. Works everywhere. |
| `musl` | Requires musl C/C++ runtime at runtime (e.g. `node-musl`). NOT the same as a static musl build. |
| `gnu`  | Requires glibc at runtime. Crashes on musl-only systems (Alpine). |
| `libc` | Host-reported UA value meaning "I have standard C library" (typically glibc). Not used in release metadata. |

### Known Issues Needing Resolution

- **WATERFALL libc vs gnu**: The Go cache correctly uses `libc='gnu'` for
  glibc-linked Linux binaries (Rust projects like bat, rg; also node). The
  build-classifier WATERFALL maps `libc` => `['none', 'libc']` but never tries
  `gnu`, so Linux glibc hosts can't match these packages. Fix needed in
  `host-targets.js`: `libc: ['none', 'gnu', 'libc']` (`none` first = prefer
  static, `gnu` second = glibc host can run gnu-linked, `libc` last = fallback).
  See QUESTIONS.md in the Go agent worktree.
- **Go cache `.git` source URLs (regression)**: The Go cache includes `.git`
  source repo URLs as release assets. These get classified as
  `ANYOS`/`ANYARCH` and match before platform-specific binaries because
  the triplet enumeration tries ANYOS first. Production doesn't have this
  issue — the old `releases.js` fetchers never included `.git` URLs. Fix:
  exclude `.git` URLs from Go cache output.

### Known Pre-existing Issues

- **`transform-releases.js` self-test**: The `if (require.main === module)`
  block calls `module.exports({...})` but the module exports an object.
- **armhf classification warnings**: `arm-unknown-linux-gnueabihf` triggers
  "wrong arch" warnings in the build-classifier (armhf vs armv6).
- **Go illumos/solaris warnings**: Go's illumos and solaris builds trigger
  "wrong os" warnings (expected `sunos`).
- **webinstall.dev `serve-releases.js` libc bug**: Line 37 calls
  `UaDetect.arch(req.query.libc)` instead of `UaDetect.libc(...)`. This
  means the libc query parameter always evaluates to `'error'`, effectively
  disabling libc filtering in the release API. (Bug is in the webinstall.dev
  repo, not webi-installers.)

## Public API Endpoints

The HTTP routing is NOT in this repo — an external server calls into the Node
modules. Here's the complete endpoint catalog:

### 1. Bootstrap / Installer Scripts

```
GET /{package}@{tag}
GET /{package}@{tag}.sh
GET /{package}@{tag}.ps1
```

**Handler**: `serve-installer.js:serveInstaller(baseurl, ua, pkg, tag, ext, formats, libc)`

**Flow**:
1. Parse User-Agent → `build-classifier/host-targets.js:termsToTarget()` → `{os, arch, libc}`
2. Resolve alias → `builds-cacher.js:getProjectType()` (symlink or README alias)
3. Load cache → `builds-cacher.js:getPackages()` → reads `_cache/YYYY-MM/{pkg}.json`
4. Classify → `builds-cacher.js:transformAndUpdate()` → triplets, versions, formats
5. Match → `builds-cacher.js:findMatchingPackages()` → filter by OS/arch/libc/version
6. Select → `builds-cacher.js:selectPackage()` → pick preferred format
7. Render → `installers.js:renderBash()` or `renderPowerShell()` with template vars

**UA format** (sent by webi bootstrap): `{arch}/unknown {OS}/{version} {libc}`
- e.g. `aarch64/unknown Darwin/24.2.0 libc`
- e.g. `x86_64/unknown Linux/5.15.0 musl`

**Template variables injected**: `WEBI_VERSION`, `WEBI_PKG_URL`, `WEBI_PKG_FILE`,
`WEBI_OS`, `WEBI_ARCH`, `WEBI_EXT`, `WEBI_CHANNEL`, `PKG_NAME`

### 2. Release Metadata API (Legacy)

```
GET /api/releases/{package}.json
GET /api/releases/{package}@{version}.json
GET /api/releases/{package}.tab
GET /api/releases/{package}@{version}.tab
```

**Handler**: `transform-releases.js:getReleases({pkg, ver, os, arch, libc, lts, channel, formats, limit})`

**Flow**:
1. Read cache → `_cache/YYYY-MM/{pkg}.json`
2. Re-normalize → `normalize.js` (clears pre-classified fields, re-detects from filenames)
3. Filter → `filterReleases()` by query params
4. Sort → by version (descending), then format preference

**Query params**: `os`, `arch`, `libc`, `lts`, `channel`, `formats`, `limit`

**Response (JSON)**: `{ oses, arches, libcs, formats, releases: [{name, version, lts, channel, date, os, arch, ext, download, libc}] }`

**Response (TSV)**: `version \t lts \t channel \t date \t os \t arch \t ext \t - \t download`

**Key format details**:
- OS: `macos` (not `darwin`), `linux`, `windows`
- Arch: `arm64` (not `aarch64`), `amd64`, `armv6l`, `armv7l`, `x86`
- Versions: no `v` prefix (`0.26.1` not `v0.26.1`)

### 3. Curl-pipe Bootstrap

```
GET /{package}@{tag} (with curl/wget User-Agent)
```

**Handler**: `serve-installer.js:getPosixCurlPipeBootstrap({baseurl, pkg, ver})`
or `getPwshCurlPipeBootstrap({baseurl, pkg, ver, exename})`

Sets env vars `WEBI_PKG`, `WEBI_HOST`, `WEBI_CHECKSUM` in the bootstrap template.

### 4. Package Metadata

```
GET /packages/{package}/README.md (or other assets)
```

**Handler**: `packages.js:get(name)` → reads README.md frontmatter via `frontmarker.js`

**Response**: `{ title, tagline, description, bash, windows }`

### 5. Debug

```
GET /api/debug
```

**Handler**: `ua-detect.js:request(req)` — returns detected OS/arch/libc from UA.

## Testing

### Automated compatibility test

```sh
# Refresh golden data from live site
node _webi/test-api-compat.js --refresh   # (not yet implemented)

# Run comparison
node _webi/test-api-compat.js
```

### Manual smoke tests

```sh
# builds-cacher: load packages from cache
node -e "let B = require('./_webi/builds.js'); B.init().then(...);"

# transform-releases: legacy API from cache
node -e "let R = require('./_webi/transform-releases.js'); R.getReleases({pkg:'bat', ...});"

# serve-installer: full end-to-end resolution
node -e "let I = require('./_webi/serve-installer.js'); I.helper({unameAgent:'aarch64/unknown Darwin/24.2.0 libc', ...});"

# classify-one: dev tool reads cache
node _webi/classify-one.js bat
```

## Project Type Detection

`builds-cacher.js:getProjectTypeByEntry()` classifies packages:

| Type | Meaning | Check |
|------|---------|-------|
| `alias` | Symlink or README has `alias: x` | symlink or README frontmatter |
| `valid` | Has cache file | `_cache/YYYY-MM/{name}.json` exists |
| `selfhosted` | No cache file | cache file missing |
| `hidden` | System dirs, `_*`, `.*` etc | naming convention |
| `invalid` | No README.md | file check |

## Package Counts

- **99 packages** detected as `valid` (have cache files)
- **58 packages** detected as `selfhosted` (no cache)
- **33 packages** detected as `alias`