github/awesome-kubernetes
1
0
Fork 0
mirror of https://github.com/nubenetes/awesome-kubernetes.git synced 2026-07-15 19:32:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #294 from nubenetes/develop

Browse Source 
fix(rss): bypass bot protection with httpx and fake-useragent
This commit is contained in:
Inaki
2026-06-01 14:29:50 +02:00
committed by GitHub
parent 94b41030ab d317ff1734
commit cfb60c8d47
2 changed files with 17 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
README.md
View File

@@ -140,7 +140,7 @@ Additionally, as of May 2026, Nubenetes has reached the **Platinum Operational T
| :--- | :--- |
| **Total Technical Resources (Links)** | **18006+** |
| **Specialized MD Pages** | **161** |
| **Total Commits** | **5689+** |
| **Total Commits** | **5691+** |
| **Primary AI Engine** | **Google Gemini (Agentic)** |
<!-- HEART_STATS_END -->
@@ -178,7 +178,7 @@ The growth of Nubenetes reflects the acceleration of the Cloud Native ecosystem.
| 6 | 2023 | 30 | 123 | Maintenance & Refinement |
| 7 | 2024 | 53 | 218 | Curation Strategy Pivot |
| 8 | 2025 | 5 | 20 | Stability & Research Phase |
| 9 | 2026 | 2130 | 8,796 | **Agentic AI Surge** (May 2026 Inception) |
| 9 | 2026 | 2132 | 8,805 | **Agentic AI Surge** (May 2026 Inception) |
<!-- ANNUAL_GROWTH_END -->
<!-- ANNUAL_CHART_START -->
@@ -194,8 +194,8 @@ xychart-beta
title "Nubenetes Annual Growth Metrics (2018–2026)"
x-axis ["2018", "2019", "2020", "2021", "2022", "2023", "2024", "2025", "2026"]
y-axis "Volume (Commits / Estimated New Refs)" 0 --> 9000
bar [1445, 586, 8449, 2193, 1660, 123, 218, 20, 8796]
bar [350, 142, 2046, 531, 402, 30, 53, 5, 2130]
bar [1445, 586, 8449, 2193, 1660, 123, 218, 20, 8805]
bar [350, 142, 2046, 531, 402, 30, 53, 5, 2132]
```
<!-- ANNUAL_CHART_END -->
@@ -205,7 +205,7 @@ xychart-beta
| :--- | :---: | :---: | :--- |
| 2026-04 | 25 | 103 | Active Curation |
| 2026-05 | 2101 | 8,677 | **Agentic Inception (Gemini Era)** |
| 2026-06 | 4 | 16 | Active Curation |
| 2026-06 | 6 | 24 | Active Curation |
<!-- MONTHLY_SURGE_END -->
### 2.4. Content Distribution and Semantic Clustering

15
src/ingestion_rss.py
View File

@@ -1,6 +1,8 @@
import feedparser
import asyncio
import re
import httpx
from fake_useragent import UserAgent
from datetime import datetime
from typing import List, Dict, Optional
from src.logger import log_event
@@ -18,13 +20,20 @@ class RSSDataExtractor:
async def fetch_links_since(self, since_date: datetime, feeds: List[str]) -> List[Dict]:
all_articles = []
ua = UserAgent()
for url in feeds:
self.log_audit("Discovery", None, f"Parsing feed: {url}")
try:
# Use a thread for feedparser as it's blocking
feed = await asyncio.to_thread(feedparser.parse, url)
async with httpx.AsyncClient(follow_redirects=True, timeout=30.0) as client:
headers = {'User-Agent': ua.random}
response = await client.get(url, headers=headers)
response.raise_for_status()
content = response.content
if feed.bozo:
# Use a thread for feedparser as it's blocking
feed = await asyncio.to_thread(feedparser.parse, content)
if feed.bozo and len(feed.entries) == 0:
self.log_audit("Parsing", False, f"Malformed feed: {url}")
continue