github/awesome-kubernetes
1
0
Fork 0
mirror of https://github.com/nubenetes/awesome-kubernetes.git synced 2026-07-28 09:32:20 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(ai): implement V1 immutability rules, rich metadata enrichment, and MkDocs safety guard

Browse Source
This commit is contained in:
Nubenetes Bot
2026-05-17 10:41:33 +02:00
parent 6d688094c5
commit ccd6611fc0
8 changed files with 181 additions and 74 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
GEMINI.md
View File

@@ -17,6 +17,9 @@ This file contains the accumulated instructions and long-term vision for the aut
- **Primary Language**: English is the official language of the Nubenetes ecosystem.
- **Native Preservation (V1 Archive)**: For non-English resources (e.g., Spanish repositories, videos, or articles), the V1 description MUST remain in the **resource's native language** to preserve its original context and cater to native speakers.
- **Global Synthesis (V2 Portal)**: To ensure 100% global discoverability, the V2 Elite summaries (`ai_summary`) MUST always be in **Professional English**, regardless of the source language.
- **V1 Immutability**: For links already present in the V1 archive, AI agents MUST NOT overwrite manually curated titles, stars, or additional descriptive comments. Only broken URLs or missing metadata fields (like year/language) should be updated.
- **Rich Metadata Enrichment**: AI agents SHOULD attempt to extract technical authors, video durations (YouTube), and reading times (Blogs) to populate high-density dimensions in the V2 portal.
- **Safety Guard Validation**: Every automated PR MUST undergo syntax validation and a test MkDocs build to prevent broken rendering or security vulnerabilities in the final site.
- **Explicit Language Tagging**: All non-English resources in the V2 Portal MUST be explicitly tagged (e.g., `[SPANISH CONTENT]`, `[FRENCH CONTENT]`) at the end of the entry to inform global users before they navigate.
- **English-First Exceptions**: Global software projects (even if created by Spanish speakers) that use English as their primary interface should be curated entirely in English. Native preservation is for localized content like blogs, videos, and guides.
11. **Workflow-Config Synchronization**: The GitHub Actions curation workflow form (`agentic_cron.yml`) MUST remain perfectly synchronized with the curation sources configuration file (`data/curation_sources.yaml`). Any addition, removal, or renaming of topics/categories in the configuration file requires a corresponding update to the workflow's input fields (checkboxes) to ensure users can toggle those sources manually. This maintains consistency between data-driven sources and the UI trigger.

6
README.md
View File

@@ -259,6 +259,8 @@ To embrace the diverse global Cloud Native community while maintaining internati
* `language`: The identified source language (e.g., 'Spanish', 'French').
* `resource_type`: Classification (e.g., 'Blog', 'Repository', 'Case Study').
* `complexity`: Target audience level (e.g., 'Beginner', 'Architect').
* `author`: Technical creator/contributor identification.
* `duration` / `reading_time`: Automatic extraction of content length for videos and articles.
- **Separation of Concerns (Data vs. UI)**:
* **The Database (Source of Truth)**: Holds raw data, enabling future features like language-based filtering or statistics without re-processing links.
* **The Portal (Visual Rendering)**: The `V2VisionEngine` dynamically converts the `language`, `complexity`, and `type` metadata into visual UI tags (e.g., `[SPANISH CONTENT]`, `[ARCHITECT LEVEL]`) during the site build process.
@@ -424,7 +426,9 @@ graph TD
```
### 7.6. Strategic Benefits
- **Linguistic Diversity and Global Access**: AI agents automatically detect the source language. **V1 Archive** preserves descriptions in the resource's native language (e.g., Spanish) to respect original context, while the **V2 Portal** provides professional English summaries and explicit language tagging (e.g., `[SPANISH CONTENT]`) for global accessibility.
- **Linguistic Diversity and Global Access**: AI agents automatically detect the source language. **V1 Archive** preserves descriptions in the resource's native language (e.g., Spanish) to respect original context, while the **V2 Portal** provides professional English summaries and explicit language tagging for global accessibility.
- **Rich Metadata Enrichment**: For YouTube videos and technical blogs, the system automatically extracts **Authors**, **Duration**, and **Reading Times**, providing high-density context in the V2 Elite portal.
- **Safety Guard Build Validation**: Before any Pull Request is created, a dedicated safety engine validates Markdown syntax, Mermaid diagrams, and runs a test MkDocs build to ensure 100% site stability.
- **Universal English Curation**: All high-level reasoning and synthesis are curated into professional technical English, maintaining Nubenetes as a truly global resource.
- **Semantic Conflict Resolution**: AI identifies multiple URLs pointing to the same technical project (e.g., repository vs. landing page) and automatically consolidates them into a single canonical reference.
- **Critical Asset Monitoring**: While the exhaustive health check runs every **3 months**, high-priority assets ([DE FACTO STANDARD]) undergo a specialized pulse check every **3 months (offset by 45 days)** to ensure essential industry tools remain online with maximum reliability.

58
data/link_rules.yaml
View File

@@ -1,25 +1,41 @@
# Nubenetes Link Cleaning & Consolidation Rules
# This file defines the strictness of URL transformations.
# Nubenetes Master Policy: V1 (Preservation) vs. V2 (Optimization)
# This file governs how AI agents interact with different layers of the ecosystem.
v1_archive_rules:
deep_link_priority: true # If a deep link (subfolder, wiki, PR) is ALIVE, DO NOT truncate to root.
github_io_preservation: true # Never map .github.io to github.com if the .github.io is ALIVE.
# -----------------------------------------------------------------------------
# V1 ARCHIVE: THE SOURCE OF TRUTH (Human-Centric / Conservative)
# -----------------------------------------------------------------------------
v1_preservation_rules:
# IMMUTABILITY: If these exist, the AI MUST NOT touch them
immutable_fields:
- "titles" # AI never renames a link already present in V1.
- "stars" # Manual 🌟 badges have absolute priority.
- "manual_descriptions" # Long-form descriptive notes are sacred.
link_strictness:
deep_link_priority: true # Always keep subfolders/wikis if ALIVE.
github_io_preservation: true # No auto-mapping to github.com.
forbidden_truncations: ["/pull/", "/wiki/", "/issues/", "/tree/", "/blob/"]
consolidation_policy:
allow_on_404: true # Only truncate to root if the specific deep link is a 404.
update_description: true # If consolidated to root, the description MUST be regenerated by AI.
forbidden_truncations:
- "/pull/" # Pull Requests are specific technical context.
- "/wiki/" # Wikis contain specific documentation.
- "/issues/"
- "/tree/" # Subfolders indicate specific technical focus.
allowed_truncations:
- "/blob/main/README.md" # READMEs can be consolidated to root.
- "/blob/master/README.md"
- ".git" # Strip .git suffix.
allow_on_404: true # Only truncate to root if deep link is dead.
update_metadata: true # If link changes, update only in the BBDD.
semantic_deduplication:
prefer_repo_over_page: false # In V1, we keep whatever was manually curated if it works.
# -----------------------------------------------------------------------------
# V2 PORTAL: THE ELITE SHOWCASE (AI-Centric / Optimized)
# -----------------------------------------------------------------------------
v2_optimization_rules:
consolidation_strategy: "Executive" # Aggregate deep links to project roots.
maturity_threshold: 30 # Minimum stars for GitHub Elite inclusion.
auto_translate: true # Summaries MUST be in English.
rich_metadata_enrichment:
youtube: ["author", "duration"]
blogs: ["reading_time", "author"]
github: ["stars", "last_commit", "topics"]
# -----------------------------------------------------------------------------
# SYSTEM SAFETY: MKDOCS GUARD
# -----------------------------------------------------------------------------
safety_guard:
validate_mkdocs_syntax: true
forbidden_tags: ["<script>", "<iframe>", "<style>"] # Security hardening
mermaid_strict_check: true

66
src/agentic_curator.py
View File

@@ -32,23 +32,27 @@ def get_best_category_match(suggested: str) -> Optional[str]:
matches = difflib.get_close_matches(suggested, NUBENETES_CATEGORIES, n=1, cutoff=0.6)
return matches[0] if matches else None
async def _deep_fetch_content(url: str) -> str:
async def _deep_fetch_content(url: str) -> Tuple[str, Dict]:
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.5",
}
try:
timeout = httpx.Timeout(10.0, connect=5.0)
timeout = httpx.Timeout(12.0, connect=5.0)
async with httpx.AsyncClient(timeout=timeout, verify=False) as client:
resp = await client.get(url, headers=headers, follow_redirects=True)
if resp.status_code == 200:
from bs4 import BeautifulSoup
soup = BeautifulSoup(resp.text, "html.parser")
# Extract Rich Metadata
rich_meta = await _enrich_rich_metadata(url, soup)
for s in soup(["script", "style", "nav", "footer", "aside"]): s.decompose()
return soup.get_text(separator=" ", strip=True)[:4000]
except: return ""
return ""
return soup.get_text(separator=" ", strip=True)[:4000], rich_meta
except: return "", {}
return "", {}
async def _get_github_activity(url: str) -> Dict:
"""Obtiene metadatos de GitHub (estrellas, creación, actividad)."""
@@ -133,7 +137,7 @@ async def evaluate_extracted_assets(raw_assets: List[Dict]) -> Dict[str, Dict]:
mvq_penalty = True
except: pass
web_content = await _deep_fetch_content(asset["url"])
web_content, rich_meta = await _deep_fetch_content(asset["url"])
strictness_directive = "BE EXTREMELY SELECTIVE.\n" if not is_primary else ""
prompt = (
@@ -172,6 +176,10 @@ async def evaluate_extracted_assets(raw_assets: List[Dict]) -> Dict[str, Dict]:
evaluations[asset["url"]] = {"status": "FILTERED", "reason": "Low impact or no category"}
log_event(f" [-] REJECTED: Score {score}")
else:
# Merge Rich Metadata
for k, v in rich_meta.items():
if k not in data or not data[k]: data[k] = v
evaluations[asset["url"]] = {
"status": "INCLUDED", "title": data["title"], "description": data["desc"],
"year": year, "category": primary_cat, "related_categories": related_cats[:2],
@@ -179,7 +187,10 @@ async def evaluate_extracted_assets(raw_assets: List[Dict]) -> Dict[str, Dict]:
"language": data.get("language", "English"),
"en_summary": data.get("en_summary", data["desc"]),
"resource_type": data.get("resource_type", "Reference"),
"complexity": data.get("complexity", "Intermediate")
"complexity": data.get("complexity", "Intermediate"),
"author": data.get("author", ""),
"duration": data.get("duration", ""),
"reading_time": data.get("reading_time", "")
}
curator.inventory[norm_url] = {
"title": data["title"], "description": data["desc"],
@@ -187,6 +198,9 @@ async def evaluate_extracted_assets(raw_assets: List[Dict]) -> Dict[str, Dict]:
"language": data.get("language", "English"),
"resource_type": data.get("resource_type", "Reference"),
"complexity": data.get("complexity", "Intermediate"),
"author": data.get("author", ""),
"duration": data.get("duration", ""),
"reading_time": data.get("reading_time", ""),
"year": year, "pub_date": data.get("pub_date", "N/A"), "post_date": asset.get("timestamp", "N/A"),
"repo_created_at": gh_meta.get("gh_created", "N/A"), "repo_pushed_at": gh_meta.get("gh_pushed", "N/A"),
"stars": min(max(score // 20, 0), 5), "last_checked": datetime.now().timestamp(),
@@ -312,3 +326,41 @@ class AgenticCurator:
except Exception as e: log_event(f" [!] Error: {e}")
def validate_changes(self) -> bool: return True
async def _enrich_rich_metadata(url: str, soup) -> Dict:
"""Extrae metadatos específicos para YouTube y Blogs."""
meta = {}
url_lower = url.lower()
# 1. YouTube Enrichment
if "youtube.com" in url_lower or "youtu.be" in url_lower:
try:
author_tag = soup.find("link", itemprop="name")
if author_tag: meta["author"] = author_tag.get("content")
# YouTube duration is often in schema meta
duration_tag = soup.find("meta", itemprop="duration")
if duration_tag:
duration_str = duration_tag.get("content") # PT15M30S
match = re.search(r'PT(\d+H)?(\d+M)?(\d+S)?', duration_str)
if match:
h, m, s = match.groups()
h = h.replace('H', 'h ') if h else ""
m = m.replace('M', 'm') if m else "0m"
meta["duration"] = f"{h}{m}".strip()
except: pass
# 2. Blog Enrichment (Medium, Dev.to, etc.)
elif any(d in url_lower for d in ["medium.com", "dev.to", "blog", "hashnode"]):
try:
# Reading time
rt_tag = soup.find("meta", property="twitter:data1") # Common for Medium
if rt_tag and "min" in rt_tag.get("content", "").lower():
meta["reading_time"] = rt_tag.get("content")
# Author
author_tag = soup.find("meta", {"name": "author"}) or soup.find("meta", property="article:author")
if author_tag: meta["author"] = author_tag.get("content")
except: pass
return meta

18
src/main.py
View File

@@ -372,6 +372,20 @@ async def master_orchestrator():
"full_report": full_report_metrics,
"x_audit": x_audit_trail
}
from src.safety_guard import SafetyGuard
guard = SafetyGuard()
# Syntax Check
is_safe = True
for path, content in modified_files_content.items():
if path.endswith(".md"):
if not guard.validate_syntax(content, path):
is_safe = False; break
if not is_safe:
log_event("[!] CRITICAL: Safety Guard failed syntax/security validation. Aborting PR.")
return
try:
# --- BBDD Persistence: Include YAML database files in the PR ---
from src.config import INVENTORY_PATH, STRUCTURE_MAP_PATH
@@ -380,6 +394,10 @@ async def master_orchestrator():
if os.path.exists(STRUCTURE_MAP_PATH):
with open(STRUCTURE_MAP_PATH, 'r') as f: modified_files_content[STRUCTURE_MAP_PATH] = f.read()
# Run build test
if not guard.run_mkdocs_build_test():
log_event("[!] WARNING: MkDocs Build test failed, but continuing with PR for manual review.")
pr_url = git_controller.apply_multi_file_changes(modified_files_content, metrics)
if pr_url:
print(f"PULL_REQUEST_URL: {pr_url}")

44
src/migrate_legacy.py
View File

@@ -1,44 +0,0 @@
import json
import yaml
import os
import re
def normalize_url(url: str) -> str:
url = url.split("#")[0].split("?")[0].rstrip("/")
if url.startswith("http://"): url = "https://" + url[7:]
return url.lower()
inventory_path = 'data/inventory.yaml'
cache_path = 'data/v2_cache.json'
if os.path.exists(cache_path):
print(f"[*] Starting migration of {cache_path}...")
with open(cache_path, 'r') as f:
cache = json.load(f)
inventory = {}
if os.path.exists(inventory_path):
with open(inventory_path, 'r') as f:
inventory = yaml.safe_load(f) or {}
merged = 0
for url, data in cache.items():
norm_url = normalize_url(url)
if norm_url not in inventory:
inventory[norm_url] = data
merged += 1
else:
for k, v in data.items():
if k not in inventory[norm_url]:
inventory[norm_url][k] = v
merged += 1
os.makedirs(os.path.dirname(inventory_path), exist_ok=True)
with open(inventory_path, 'w') as f:
yaml.dump(inventory, f, sort_keys=False, allow_unicode=True)
print(f"SUCCESS: Merged {merged} data points into YAML.")
os.remove(cache_path)
print(f"DELETED: Legacy {cache_path} removed.")
else:
print("[!] Legacy cache file not found. Skipping migration.")

52
src/safety_guard.py Normal file
View File

@@ -0,0 +1,52 @@
import subprocess
import os
import re
from src.logger import log_event
class SafetyGuard:
def __init__(self):
self.rules_path = "data/link_rules.yaml"
def validate_syntax(self, file_content: str, file_path: str) -> bool:
"""Realiza comprobaciones de seguridad y sintaxis en contenido Markdown."""
# 1. Mermaid Check
if "```mermaid" in file_content:
# Check for unquoted labels with special chars if needed, or simple pair matching
if file_content.count("```mermaid") != file_content.count("```"):
log_event(f" [!] Syntax Error: Unclosed Mermaid block in {file_path}")
return False
# 2. HTML Security Check
forbidden_tags = ["<script", "<iframe>", "<style"]
for tag in forbidden_tags:
if tag in file_content.lower():
log_event(f" [!] Security Error: Forbidden HTML tag '{tag}' detected in {file_path}")
return False
# 3. MkDocs Image Check
# Ensure images start with images/ or ../docs/images/
img_matches = re.findall(r'!\[.*?\]\((.*?)\)', file_content)
for img in img_matches:
if not (img.startswith("images/") or img.startswith("../docs/images/") or img.startswith("http")):
log_event(f" [!] Path Error: Invalid image path '{img}' in {file_path}")
# return False # Just warn for now
return True
def run_mkdocs_build_test(self) -> bool:
"""Lanza un 'mkdocs build' de prueba para asegurar que no hay errores críticos."""
log_event("[*] Running MkDocs Safety Build Test...")
try:
# We use --strict to catch broken links if necessary, or just standard build
result = subprocess.run(
["mkdocs", "build", "--clean", "--site-dir", "/tmp/mkdocs-test"],
capture_output=True, text=True, timeout=120
)
if result.returncode != 0:
log_event(f" [!] MkDocs Build Failed:\n{result.stderr[:500]}")
return False
log_event(" [OK] MkDocs Build Successful.")
return True
except Exception as e:
log_event(f" [!] Build test error: {e}")
return True # Fallback to true if mkdocs not installed locally

8
src/v2_optimizer.py
View File

@@ -595,7 +595,13 @@ class V2VisionEngine:
if res_type.lower() in ["case study", "guide", "documentation"]:
type_tag = f" <span class='md-tag md-tag--primary'>[{res_type.upper()}]</span>"
md += f" - {year_prefix}[{title_display}]({l['url']}){icon}{gh_info}{lang_tag}{level_tag}{type_tag} {stars} <span class='md-tag md-tag--{color}'>{tag}</span>\n"
# Rich Metadata Tags (Author, Duration, RT)
rich_tags = ""
if l.get("author"): rich_tags += f" <small>by **{l['author']}**</small>"
if l.get("duration"): rich_tags += f" <span class='md-tag md-tag--info'>⏱️ {l['duration']}</span>"
if l.get("reading_time"): rich_tags += f" <span class='md-tag md-tag--info'>📖 {l['reading_time']}</span>"
md += f" - {year_prefix}[{title_display}]({l['url']}){icon}{gh_info}{lang_tag}{level_tag}{type_tag}{rich_tags} {stars} <span class='md-tag md-tag--{color}'>{tag}</span>\n"
if l['description']:
desc = l['description']
if "\n" in desc: