mirror of
https://github.com/nubenetes/awesome-kubernetes.git
synced 2026-08-18 03:46:27 +00:00
feat: standardize YouTube enrichment across all pipelines and modularize Video Hub workflow
This commit is contained in:
Nubenetes Bot
+19
-118
@@ -5,17 +5,12 @@ All notable changes to this project will be documented in this file.
|
||||
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
||||
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [2.0.0-rc.12] - 2026-05-22
|
||||
## [2.0.0-rc.17] - 2026-05-22
|
||||
|
||||
### Fixed
|
||||
- **YouTube Mosaic Exemption**: Implemented high-precision detection for the YouTube mosaic in `docs/index.md`, ensuring that only the primary high-density block is exempt from health checks, while other YouTube links (before the mosaic and in collapsible notes) are properly checked.
|
||||
- **README Metrics Restoration**: Fixed the git checkout depth (`fetch-depth: 0`) in the V2 publisher workflow to restore accurate historical metric generation for the README.
|
||||
- **Iframe Health Checks**: Added support for detecting and checking YouTube links within `<iframe>` tags in the index file.
|
||||
|
||||
## [2.0.0-rc.11] - 2026-05-22
|
||||
|
||||
### Added
|
||||
- **YouTube Mosaic Update**: Added YouTuber @mouredev (Brais Moure) to the YouTube mosaic in both V1 and V2 index portals.
|
||||
### Changed
|
||||
- **Centralized YouTube Enrichment**: Standardized YouTube metadata extraction across all curation pipelines (V1 and V2). All workflows now fetch real titles and descriptions from YouTube to ensure high-fidelity AI summaries.
|
||||
- **Workflow Modularization**: Reverted the consolidation of the Video Hub into the Publisher pipeline, maintaining it as a standalone workflow (`agentic_v2_videos.yml`) for better operational flexibility.
|
||||
- **Improved Video Hub Design**: Redesigned the V2 Elite Video Hub with high-fidelity categorization, technology tags, and individual collapsible blocks for better user experience and flow.
|
||||
|
||||
## [2.0.0-rc.14] - 2026-05-22
|
||||
|
||||
@@ -28,120 +23,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
### Added
|
||||
- **GEMINI Mandate Codification**: Officially codified architectural and UI standards in `GEMINI.md`, including the V1 vs V2 metrics protocol and branding protection rules.
|
||||
|
||||
## [2.0.0-rc.12] - 2026-05-22
|
||||
|
||||
### Changed
|
||||
### Fixed
|
||||
- **YouTube Mosaic Exemption**: Implemented high-precision detection for the YouTube mosaic in `docs/index.md`, ensuring that only the primary high-density block is exempt from health checks.
|
||||
- **README Metrics Restoration**: Fixed the git checkout depth (`fetch-depth: 0`) in the V2 publisher workflow to restore accurate historical metric generation.
|
||||
- **Iframe Health Checks**: Added support for detecting and checking YouTube links within `<iframe>` tags in the index file.
|
||||
|
||||
## [2.0.0-rc.11] - 2026-05-22
|
||||
|
||||
### Added
|
||||
- **YouTube Mosaic Update**: Added YouTuber @mouredev (Brais Moure) to the YouTube mosaic in both V1 and V2 index portals.
|
||||
|
||||
## [2.0.0-rc.10] - 2026-05-22
|
||||
|
||||
### Added
|
||||
- **V2 Elite Synchronization**: Successfully synchronized the curated elite edition and updated README metrics via the automated publisher pipeline.
|
||||
|
||||
## [2.0.0-rc.9] - 2026-05-22
|
||||
|
||||
## [2.0.0-rc.9] - 2026-05-22
|
||||
|
||||
### Changed
|
||||
- **UI Branding Optimization**: Removed redundant brand mentions in the V2 Portal header for a cleaner and more professional appearance.
|
||||
|
||||
## [2.0.0-rc.8] - 2026-05-22
|
||||
|
||||
## [2.0.0-rc.8] - 2026-05-22
|
||||
|
||||
### Added
|
||||
- **Comparative Knowledge Metrics**: Enhanced the V2 Portal index with V1 vs V2 inventory comparison, showing the total base resources analyzed and the elite selection density ratio.
|
||||
- **Agentic Efficiency Tracking**: Integrated real-time calculation of the curation delta between the master archive and the high-density portal.
|
||||
|
||||
|
||||
## [2.0.0-rc.7] - 2026-05-22
|
||||
|
||||
### Fixed
|
||||
- **GitHub Metadata Metrics**: Corrected the coverage calculation logic in `v2_optimizer.py` to properly identify GitHub repositories by ensuring URLs are correctly injected into the resource objects during the audit phase.
|
||||
- **Knowledge Graph Integrity**: Resolved the 0% reporting bug in the V2 Portal index, ensuring accurate tracking of resource maturity.
|
||||
|
||||
### Changed
|
||||
- **Pipeline Orchestration**: Successfully validated the new decoupled workflow system by performing a manual execution of the Metadata and Publisher engines.
|
||||
|
||||
|
||||
## [2.0.0-rc.6] - 2026-05-22
|
||||
|
||||
### Added
|
||||
- **High-Density Inline Summaries**: Replaced collapsible blocks with always-visible summaries appended directly to the resource line using an em-dash (—) separator for immediate readability.
|
||||
|
||||
### Fixed
|
||||
- **V2 Rendering Engine**: Resolved a critical `TypeError` caused by incorrect indentation in the link rendering logic.
|
||||
- **Workflow Reliability**: Standardized all GitHub Actions to use Python module execution mode (`python -m src.module`), eliminating `ModuleNotFoundError` issues.
|
||||
|
||||
## [2.0.0-rc.5] - 2026-05-22
|
||||
|
||||
### Added
|
||||
- **Decoupled V2 Architecture**: Refactored the monolithic V2 builder into four specialized micro-workflows (**Health Monitor**, **Metadata Engine**, **AI Curator**, and **Publisher**) to optimize compute quotas and Gemini token consumption.
|
||||
- **Minimalist Inline UI**: Implemented native HTML5 `<details>` summaries with `inline-block` behavior in the V2 Portal, significantly increasing vertical information density.
|
||||
- **Knowledge Coverage Metrics**: Integrated real-time AI Enrichment and GitHub Metadata coverage tracking into PR reports and the V2 index.
|
||||
- **Automated Scheduling**: Configured monthly automated runs for Health (1st) and Metadata (15th) agents.
|
||||
|
||||
### Fixed
|
||||
- **V2 Rendering Logic**: Fixed a bug where links without a defined hierarchy or at top-level categories were omitted; introduced the **Standard Reference** fallback section.
|
||||
- **Markdown Linting**: Resolved TOC anchor broken links (MD051) in `README.md`.
|
||||
|
||||
## [2.0.0-rc.4] - 2026-05-22
|
||||
|
||||
### Added
|
||||
- **Mosaic Expansion**: Added Olena Kutsenko's YouTube channel to the index mosaic on both V1 and V2, including custom profile picture extraction and asset synchronization.
|
||||
|
||||
## [2.0.0-rc.3] - 2026-05-21
|
||||
|
||||
### Added
|
||||
- **Flash-First Architecture**: Strategic transition to Gemini Flash/Lite models for high-density analysis, achieving >90% cost reduction and radical throughput gains.
|
||||
- **Multi-Phase Incremental Persistence**: Dual-layer auto-save mechanism (Metadata & AI phases) ensuring zero data loss during long-running CI/CD workflows.
|
||||
- **Industrial Telemetry Layer**: Deep AI observability with token counting, payload estimation, and transparent error diagnostics.
|
||||
- **GitHub Actions Cache Integration**: High-resilience persistence layer to reuse enriched metadata across interrupted or queued runs.
|
||||
- **Emergency PR Workflow**: Lightweight "Safety Off-ramp" to publish current cached state without AI overhead.
|
||||
|
||||
### Changed
|
||||
- **Optimized Batching Strategy**: Balanced batch sizes (50 items) and safety delays (2s) to maximize API quota utilization (TPM/RPM).
|
||||
- **Hardened Error Handling**: Transitioned from silent failure loops to strict Circuit Breaker (Exit Code 42) logic.
|
||||
- **Unified Global Logging**: Enforced `PYTHONUNBUFFERED` logging across the entire workflow ecosystem for real-time transparency.
|
||||
- **V2 UI Refinement**: Optimized header contrast for Light/Dark modes and fixed semantic cross-link duplication.
|
||||
|
||||
### Fixed
|
||||
- **API Timeout Bottleneck**: Increased HTTP request timeouts to 180s to accommodate high-density token generation.
|
||||
- **Markdown Linter Compliance**: Resolved MD024, MD025, MD031, and MD039 violations through "Nuclear" whitespace sanitization and header depth management.
|
||||
- **Workflow Permissions**: Optimized security model to allow automated PRs without requiring conflictive workflow-write permissions.
|
||||
|
||||
## [2.0.0-rc.2] - 2026-05-20
|
||||
|
||||
### Added
|
||||
- **Announce Banner**: Added a global announcement banner to V1 directing users to the V2 Elite Portal.
|
||||
- **Dependency Caching**: Implemented `cache: pip` via `requirements.txt` to significantly speed up build times.
|
||||
- **MkDocs Features**: Activated native Privacy Plugin, Pruned Navigation, Social Cards in V1, Code Copy, Tab Sync, and Tooltips.
|
||||
|
||||
### Changed
|
||||
- **CI/CD Deployment**: Migrated deployment to use native GitHub Pages artifacts (`upload-pages-artifact`, `deploy-pages`) instead of the `gh-pages` branch.
|
||||
- **V2 Elite Aesthetic**: Upgraded UI to "Cyber Cloud" style featuring high-contrast pure black backgrounds, neon cyan accents, and advanced glassmorphism.
|
||||
|
||||
## [2.0.0-rc.1] - 2026-05-19
|
||||
|
||||
### Added
|
||||
- **Nubenetes Elite Portal (V2)**: A high-density, agentic-curated edition of the archive.
|
||||
- **Automotive Container Metaphor**: New modernized visual identity with a custom PNG asset.
|
||||
- **Agentic AI Orchestration**: Multi-tier model coordination (Gemini Pro/Flash) with adaptive rate limiting.
|
||||
- **Maturity Taxonomy**: Professional 5-tier classification system for all Elite resources.
|
||||
- **Semantic Cross-Linking**: Autonomous identification of related architectural patterns.
|
||||
- **Platinum Maintenance Suite**: Automated triage, social preview cards, and dependency guarding.
|
||||
- **Real-time Web Grounding**: Live data verification using MCP-style search for high-precision curation.
|
||||
- **Industrial Learning Flow**: O'Reilly-style technical progression in V2 documentation.
|
||||
|
||||
### Changed
|
||||
- **Unified Metadata Database**: Centralized all link lifecycle data in `data/inventory.yaml`.
|
||||
- **Global Currency Standard**: Adopted Euro (€) as the primary currency for all cost and economic analysis.
|
||||
- **Navigation Overhaul**: Flat, high-density navigation structure for the Elite portal.
|
||||
- **README Metrics Engine**: Transitioned to a fully database-driven metric extraction system.
|
||||
|
||||
### Fixed
|
||||
- **Image Path Resolution**: Implemented flat asset routing to prevent broken relative paths.
|
||||
- **Markdown Rendering**: Fixed HTML/Markdown interop using `markdown="1"` attributes.
|
||||
- **Trigger Loop Prevention**: Hardened CI/CD workflows against infinite automation loops.
|
||||
- **URL Normalization**: Standardized canonical formats to prevent semantic drift.
|
||||
|
||||
### Security
|
||||
- **Credential Protection**: Hardened environment variable management across all agentic scripts.
|
||||
- **License Guard**: Automated monitoring of permissive-to-restrictive license transitions.
|
||||
|
||||
---
|
||||
*Generated by Nubenetes Agentic Intelligence*
|
||||
|
||||
+10
-1
@@ -9,7 +9,7 @@ from datetime import datetime
|
||||
from typing import List, Dict, Optional, Tuple
|
||||
from src.config import GH_TOKEN, TARGET_REPO, GEMINI_API_KEY, NUBENETES_CATEGORIES, MADRID_TZ, INVENTORY_DIR
|
||||
from src.gitops_manager import RepositoryController
|
||||
from src.gemini_utils import call_gemini_with_retry, normalize_url, clean_toc_text
|
||||
from src.gemini_utils import call_gemini_with_retry, normalize_url, clean_toc_text, fetch_youtube_metadata
|
||||
from src.logger import log_event
|
||||
|
||||
# Configuration
|
||||
@@ -42,6 +42,15 @@ async def _get_github_activity(url: str) -> Dict:
|
||||
return {}
|
||||
|
||||
async def _deep_fetch_content(url: str) -> Tuple[str, Dict]:
|
||||
# MANDATE 25: Special handling for YouTube
|
||||
if "youtube.com" in url or "youtu.be" in url:
|
||||
log_event(f" [YT] Detected YouTube link: {url}. Fetching native metadata...")
|
||||
meta = await fetch_youtube_metadata(url)
|
||||
if meta:
|
||||
# Combine title and description to feed the AI
|
||||
content = f"TITLE: {meta['raw_title']}\nDESCRIPTION: {meta['raw_description']}"
|
||||
return content, {"og_image": f"https://img.youtube.com/vi/{url.split('v=')[-1].split('&')[0]}/maxresdefault.jpg" if "v=" in url else ""}
|
||||
|
||||
headers = {"User-Agent": "Mozilla/5.0"}
|
||||
try:
|
||||
async with httpx.AsyncClient(headers=headers, follow_redirects=True, timeout=15.0) as client:
|
||||
|
||||
+1
-23
@@ -4,32 +4,10 @@ import re
|
||||
import asyncio
|
||||
import httpx
|
||||
from src.logger import log_event
|
||||
from src.gemini_utils import call_gemini_with_retry
|
||||
from src.gemini_utils import call_gemini_with_retry, fetch_youtube_metadata
|
||||
|
||||
INVENTORY_PATH = "data/inventory.yaml"
|
||||
|
||||
async def fetch_youtube_metadata(url: str):
|
||||
"""Fetches basic metadata from YouTube page without API key."""
|
||||
try:
|
||||
async with httpx.AsyncClient(follow_redirects=True, timeout=10.0) as client:
|
||||
# Convert embed to watch URL if needed for better meta tags
|
||||
watch_url = url.replace("/embed/", "/watch?v=").split("?")[0]
|
||||
resp = await client.get(watch_url, headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"})
|
||||
if resp.status_code != 200:
|
||||
return None
|
||||
|
||||
html = resp.text
|
||||
title_match = re.search(r'<title>(.*?)</title>', html)
|
||||
desc_match = re.search(r'name="description" content="(.*?)"', html)
|
||||
|
||||
title = title_match.group(1).replace(" - YouTube", "") if title_match else "YouTube Video"
|
||||
description = desc_match.group(1) if desc_match else ""
|
||||
|
||||
return {"raw_title": title, "raw_description": description}
|
||||
except Exception as e:
|
||||
log_event(f"Error fetching YT metadata for {url}: {e}")
|
||||
return None
|
||||
|
||||
async def enrich_video_entry(url: str, entry: dict):
|
||||
log_event(f"[*] Enriching video: {url}")
|
||||
meta = await fetch_youtube_metadata(url)
|
||||
|
||||
@@ -417,3 +417,49 @@ async def call_gemini_with_retry(prompt: str, response_format: str = "json", max
|
||||
|
||||
log_event(" [!] QUOTA EXHAUSTED: All keys and models rate-limited. Triggering Tenacity backoff...")
|
||||
raise GeminiQuotaExhausted(f"Critical Gemini failure after adaptive tiering.\n{diagnostics.get_report()}")
|
||||
|
||||
async def fetch_youtube_metadata(url: str) -> Optional[Dict]:
|
||||
"""
|
||||
Fetches basic metadata (title, description) from a YouTube page.
|
||||
Used for pre-enriching AI prompts with real content data.
|
||||
"""
|
||||
try:
|
||||
# Convert embed/short URLs to standard watch URLs for better meta tags
|
||||
clean_url = url.split("?")[0].split("&")[0]
|
||||
if "/embed/" in clean_url:
|
||||
vid = clean_url.split("/embed/")[-1]
|
||||
watch_url = f"https://www.youtube.com/watch?v={vid}"
|
||||
elif "youtu.be/" in clean_url:
|
||||
vid = clean_url.split("youtu.be/")[-1]
|
||||
watch_url = f"https://www.youtube.com/watch?v={vid}"
|
||||
else:
|
||||
watch_url = clean_url
|
||||
|
||||
headers = {
|
||||
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
|
||||
"Accept-Language": "en-US,en;q=0.9"
|
||||
}
|
||||
|
||||
async with httpx.AsyncClient(follow_redirects=True, timeout=10.0) as client:
|
||||
resp = await client.get(watch_url, headers=headers)
|
||||
if resp.status_code != 200:
|
||||
return None
|
||||
|
||||
html = resp.text
|
||||
# Use regex to find title and description in meta tags
|
||||
title_match = re.search(r'<title>(.*?)</title>', html)
|
||||
desc_match = re.search(r'name="description" content="(.*?)"', html)
|
||||
|
||||
title = title_match.group(1).replace(" - YouTube", "") if title_match else "YouTube Video"
|
||||
description = desc_match.group(1) if desc_match else ""
|
||||
|
||||
# Clean description from encoded characters
|
||||
description = re.sub(r'\\u[0-9a-fA-F]{4}', '', description)
|
||||
|
||||
return {
|
||||
"raw_title": title.strip(),
|
||||
"raw_description": description.strip()[:2000] # Limit size
|
||||
}
|
||||
except Exception as e:
|
||||
log_event(f" [!] YouTube metadata fetch failed for {url}: {e}")
|
||||
return None
|
||||
|
||||
+13
-2
@@ -7,7 +7,7 @@ import httpx
|
||||
from datetime import datetime
|
||||
from typing import List, Dict, Set, Any, Tuple
|
||||
from src.config import GEMINI_API_KEYS, GH_TOKEN, TARGET_REPO, MADRID_TZ, INVENTORY_PATH
|
||||
from src.gemini_utils import call_gemini_with_retry, normalize_url, clean_toc_text, get_github_activity
|
||||
from src.gemini_utils import call_gemini_with_retry, normalize_url, clean_toc_text, get_github_activity, fetch_youtube_metadata
|
||||
from src.logger import log_event
|
||||
|
||||
def nuclear_strip(text: str) -> str:
|
||||
@@ -405,6 +405,17 @@ class V2VisionEngine:
|
||||
total_batches = (total_grounded + BATCH_SIZE_GROUNDED - 1) // BATCH_SIZE_GROUNDED
|
||||
log_event(f" [🌟] Grounded-Track: Processing Batch {batch_num}/{total_batches} (Grounding active)...")
|
||||
|
||||
# MANDATE 25: Pre-enrich YouTube links with real metadata
|
||||
enriched_batch = []
|
||||
for item in batch:
|
||||
url = item["url"]
|
||||
if "youtube.com" in url or "youtu.be" in url:
|
||||
log_event(f" [YT] Pre-fetching metadata for: {url}")
|
||||
meta = await fetch_youtube_metadata(url)
|
||||
if meta:
|
||||
item["description"] = f"TITLE: {meta['raw_title']}\nDESCRIPTION: {meta['raw_description']}"
|
||||
enriched_batch.append(item)
|
||||
|
||||
prompt = (
|
||||
f"You are the Nubenetes Technical Analyst (2026).\n"
|
||||
f"{dynamic_mandates}\n"
|
||||
@@ -412,7 +423,7 @@ class V2VisionEngine:
|
||||
"PHASE 5: DOUBLE-EVIDENCE SYNTHESIS & RICH SUMMARY (GROUNDED)\n"
|
||||
"- Cross-reference provided title/desc with search grounding.\n"
|
||||
"Respond ONLY JSON: {{\"results\": [{{ \"idx\": int, \"year\": \"YYYY\", \"stars\": 0-5, \"hierarchy\": [\"Area\", \"Topic\", ...], \"tags\": [\"...\"], \"summary\": \"Synthesis...\", \"language\": \"...\", \"type\": \"...\", \"complexity\": \"...\", \"is_microservice\": bool }}, ...]}}\n\n"
|
||||
"LINKS:\n" + "\n".join([f"{idx}. {l['title']} ({l['url']})" for idx, l in enumerate(batch)])
|
||||
"LINKS:\n" + "\n".join([f"{idx}. {l['title']} ({l['url']}) | Input Context: {l.get('description', 'N/A')}" for idx, l in enumerate(enriched_batch)])
|
||||
)
|
||||
try:
|
||||
data = await call_gemini_with_retry(prompt, prefer_flash=True, use_grounding=True, role="Analyst-Grounded")
|
||||
|
||||
Reference in New Issue
Block a user