ref(builds): replace releases.js type check with cache file check

Project type detection no longer require()s {pkg}/releases.js to
determine if a package is valid. Instead it checks for a cache file
at _cache/YYYY-MM/{pkg}.json. This means:

- Packages with Go-generated cache but no releases.js (vim plugins,
  pg-essentials, etc.) are now correctly detected as 'valid'
- The 'not_found' type (broken npm deps) is removed — no longer relevant
- releases.js files are no longer loaded at runtime for type detection
This commit is contained in:
AJ ONeal
2026-03-11 01:52:49 -06:00
parent 81c3d67a4f
commit 0f78339725
2 changed files with 79 additions and 26 deletions

66
PROD_NOTES.md Normal file
View File

@@ -0,0 +1,66 @@
# Production Notes: Node.js Cache-Only Migration
## Current State
The Node.js server no longer fetches from upstream APIs. It reads only from
`_cache/YYYY-MM/{pkg}.json` files generated by the Go `webicached` daemon.
### Completed
- **builds.js**: Removed `freshenRandomPackage()` calls and background refresh
- **builds-cacher.js**: Removed `getLatestBuilds()`, stale re-fetch, and
`freshenRandomPackage()`. Missing cache files return empty metadata.
### In Progress
- **Remove `releases.js` runtime dependency**: The `getProjectTypeByEntry()`
function still `require()`s each package's `releases.js` to determine if a
package is `valid` (has releases.js) vs `selfhosted` (no releases.js). This
distinction controlled whether Node fetched upstream. Now that all data comes
from cache, the check should use cache file existence instead.
### Pending
- **`transform-releases.js`**: Legacy release API. Not `require()`d anywhere in
this repo — called by an external HTTP server. Still fetches upstream via
`Releases.get()``require({pkg}/releases.js)``.latest()`. Needs to be
made cache-only if this API endpoint is still live.
## Public API Endpoints (live at webinstall.dev)
Tested against production 2026-03-11:
- `GET /api/releases/{pkg}.json` — Returns raw JSON array of release objects
(via `transform-releases.js` + `normalize.js`). Each object has: name,
version, lts, channel, date, os, arch, ext, download, libc.
- `GET /api/releases/{pkg}.tab` — Tab-separated release data
- `GET /{pkg}@{tag}` — Returns installer script (bash or ps1 based on UA)
## Project Type Detection
`builds-cacher.js:getProjectTypeByEntry()` classifies packages:
| Type | Meaning | Current check |
|------|---------|---------------|
| `alias` | Symlink or README has `alias: x` | symlink or README frontmatter |
| `valid` | Has releases.js | `require(releases.js)` succeeds |
| `selfhosted` | No releases.js | `require(releases.js)` throws MODULE_NOT_FOUND |
| `hidden` | System dirs, `_*`, `.*` etc | naming convention |
| `invalid` | No README.md | file check |
**`serve-installer.js` behavior by type:**
- `valid` → loads cache, resolves version/triplet, renders installer
- `selfhosted` → returns error package immediately (no cache lookup)
- Others → throws ENOENT
**Problem**: With cache-only mode, some `selfhosted` packages now have Go-generated
cache files (vim plugins, pg-essentials, etc.) but are never served because the
type check short-circuits them to error.
## Package Counts
- **101 packages** have Go-generated cache files
- **~97 packages** have `releases.js` files
- **~90 packages** are `selfhosted` (README but no releases.js)
- **13 packages** have cache but no releases.js (vim plugins, pg-essentials, etc.)
- **5 releases.js** files have no cache (aliases: golang, ripgrep; plus _example, macos, zig.vim)

View File

@@ -163,19 +163,6 @@ BuildsCacher.create = function ({ ALL_TERMS, installers, caches }) {
let entries = await Fs.readdir(installersDir, { withFileTypes: true });
for (let entry of entries) {
let meta = await bc.getProjectTypeByEntry(entry);
if (meta.type === 'not_found') {
let err = meta.detail;
console.error('');
console.error('PROBLEM');
console.error(` ${err.message}`);
console.error('');
console.error('SOLUTION');
console.error(' npm clean-install');
console.error('');
throw new Error(
'[SANITY FAIL] should never have missing modules in prod',
);
}
dirs[meta.type][entry.name] = meta.detail;
}
@@ -244,19 +231,19 @@ BuildsCacher.create = function ({ ALL_TERMS, installers, caches }) {
return { type: 'alias', detail: link };
}
let releasesPath = Path.join(path, 'releases.js');
try {
void require(releasesPath);
} catch (err) {
if (err.code !== 'MODULE_NOT_FOUND') {
return { type: 'errors', detail: err };
}
if (err.message.includes(`Cannot find module '${releasesPath}'`)) {
return { type: 'selfhosted', detail: true };
}
return { type: 'not_found', detail: err };
let date = new Date();
let yearMonth = date.toISOString().slice(0, 7);
let cacheFile = `${cacheDir}/${yearMonth}/${entry.name}.json`;
let hasCacheFile = await Fs.access(cacheFile).then(
function () {
return true;
},
function () {
return false;
},
);
if (!hasCacheFile) {
return { type: 'selfhosted', detail: true };
}
return { type: 'valid', detail: true };