Commit Graph

3 Commits

Author SHA1 Message Date
AJ ONeal
2e052fa553 wire 9 custom source fetchers into webicached
Add fetch + classify functions for all custom source types:
- chromedist (chromedriver): Chrome for Testing JSON index
- flutterdist (flutter): Google Storage per-OS release indexes
- golang (go): golang.org/dl JSON API
- gpgdist (gpg): SourceForge RSS scraping
- hashicorp (terraform): releases.hashicorp.com product index
- iterm2dist (iterm2): HTML scraping of downloads page
- juliadist (julia): S3 versions.json with platform files
- mariadbdist (mariadb): two-step REST API (majors → releases)
- zigdist (zig): mixed-schema JSON with platform keys

All 9 fetcher packages already existed in internal/releases/ but
were not wired into webicached's fetchRaw/classifyPackage switches.
Now all 103 packages produce classified cache output.
2026-03-10 11:23:41 -06:00
AJ ONeal
b51e9e2998 add comparecache tool and LIVE_cache comparison checklist
- cmd/comparecache: compares Go cache vs Node.js LIVE_cache at filename
  level, categorizes differences (meta-filtering, version depth, source
  tarballs, unsupported sources, real asset differences)
- COMPARISON.md: per-package checklist with 91 live packages categorized
- webicached: add -no-fetch flag to classify from existing raw data only
- GO_WEBI.md: update Phase 1 checkboxes for completed items
2026-03-10 11:17:37 -06:00
AJ ONeal
0e6d90e011 feat: add webicached — release cache daemon
Combines fetch + classify + write into one pipeline:
1. Reads releases.conf to discover packages
2. Fetches raw upstream data to rawcache
3. Classifies assets (OS, arch, libc, format)
4. Applies config transforms (exclude, version prefix strip)
5. Writes to fsstore in Node.js-compatible _cache/ format

Supports github, nodedist, gittag, and gitea sources. Other sources
(golang, zigdist, flutter, etc.) are skipped with a log message —
they'll be added as needed.

Can run as a one-shot (-once) or periodic daemon (-interval 15m).
2026-03-10 10:58:17 -06:00