feat: incremental digest engine — reuse Gemini results when inventory unchanged

Each (category × period) cell in news_digest.json now carries a hash of its
entry pool (sorted URLs). Before calling Gemini, the engine checks:
  1. Hash matches stored hash (same inventory entries)
  2. Last analyzed within MAX_STALENESS_DAYS=30

Both conditions true → reuse existing ranked list, 0 Gemini API calls.
Only cells where the inventory actually changed (new/removed entries) trigger
a fresh Gemini ranking.

Expected savings: re-runs with code-only changes → 0 API calls (down from 66).
New ingestion batch → only affected categories call Gemini (5-10 of 26 typical).
Stale refresh (>30d) → full refresh at most once per month.

Also:
- Remove day-level GitHub Actions cache for digest (redundant: news_digest.json
  is committed to the repo and is the persistent store for _meta)
- Always run news_digest.py in both 04.1 and 09 workflows (it's cheap when 0
  Gemini calls are needed)
- Trending badge now reads _meta.last_updated (actual analysis date) instead of
  file mtime (which changes on every render/commit)
- Remove invalid extra_head key from v2-mkdocs.yml (left over from rollback)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

.github/workflows/04.1.agentic_v2_publish.yml

@@ -61,19 +61,6 @@ jobs:
         run: |
           pip install --no-cache-dir pydantic PyGithub httpx fake-useragent pytz python-dotenv pyyaml tenacity
 
-      - name: Get current date for digest cache key
-        id: digest-date
-        run: echo "date=$(date -u +%Y-%m-%d)" >> $GITHUB_OUTPUT
-
-      - name: Restore News Digest Cache
-        id: cache-digest
-        uses: actions/cache/restore@v5
-        with:
-          path: data/news_digest.json
-          key: news-digest-${{ steps.digest-date.outputs.date }}
-          restore-keys: |
-            news-digest-
-
       - name: Execute Video Portal Generator
         env:
           PYTHONPATH: ${{ github.workspace }}
@@ -99,13 +86,16 @@ jobs:
         run: |
           python -u -m src.enrichment || echo "Enrichment pipeline skipped (no token or error)"
 
-      - name: Generate News Digest
-        if: steps.cache-digest.outputs.cache-hit != 'true'
+      - name: Generate News Digest (Incremental)
         env:
           PYTHONPATH: ${{ github.workspace }}
           GEMINI_API_KEY_1: ${{ secrets.GEMINI_API_KEY_1 }}
           GEMINI_API_KEY_2: ${{ secrets.GEMINI_API_KEY_2 }}
         run: |
+          # The engine loads news_digest.json from the repo, checks per-cell
+          # hashes, and only calls Gemini for categories with new entries or
+          # results older than MAX_STALENESS_DAYS (30).  Typical re-runs with
+          # no inventory changes cost 0 Gemini API calls.
           python -u -m src.news_digest || echo "News digest generation skipped (no API key or error)"
 
       - name: Generate RSS Feed
@@ -114,13 +104,6 @@ jobs:
         run: |
           python -u -m src.rss_generator || echo "RSS generation skipped"
 
-      - name: Save News Digest Cache
-        if: steps.cache-digest.outputs.cache-hit != 'true' && hashFiles('data/news_digest.json') != ''
-        uses: actions/cache/save@v5
-        with:
-          path: data/news_digest.json
-          key: news-digest-${{ steps.digest-date.outputs.date }}
-
       - name: Run V2 Publisher (Render-Only)
         env:
           PYTHONPATH: ${{ github.workspace }}

.github/workflows/09.weekly_digest.yml

@@ -33,35 +33,17 @@ jobs:
       - name: Install Dependencies
         run: pip install -r requirements.txt
 
-      - name: Get current date for digest cache key
-        id: digest-date
-        run: echo "date=$(date -u +%Y-%m-%d)" >> $GITHUB_OUTPUT
-
-      - name: Restore News Digest Cache
-        id: cache-digest
-        uses: actions/cache/restore@v5
-        with:
-          path: data/news_digest.json
-          key: news-digest-${{ steps.digest-date.outputs.date }}
-          restore-keys: |
-            news-digest-
-
-      - name: Generate News Digest (Gemini)
-        if: steps.cache-digest.outputs.cache-hit != 'true'
+      - name: Generate News Digest (Incremental)
         env:
           PYTHONPATH: ${{ github.workspace }}
           GEMINI_API_KEY_1: ${{ secrets.GEMINI_API_KEY_1 }}
           GEMINI_API_KEY_2: ${{ secrets.GEMINI_API_KEY_2 }}
         run: |
+          # Incremental engine: reuses stored results for unchanged categories.
+          # On Monday, only categories with new entries since the last run
+          # (or older than MAX_STALENESS_DAYS=30) trigger Gemini API calls.
           python -u -m src.news_digest
 
-      - name: Save News Digest Cache
-        if: steps.cache-digest.outputs.cache-hit != 'true' && hashFiles('data/news_digest.json') != ''
-        uses: actions/cache/save@v5
-        with:
-          path: data/news_digest.json
-          key: news-digest-${{ steps.digest-date.outputs.date }}
-
       - name: Generate RSS Feed
         env:
           PYTHONPATH: ${{ github.workspace }}