v0.2.0: visualizations, interactive browser, arXiv paper, gap analysis
New features: - 12 interactive visualizations (ietf viz): t-SNE landscape, similarity heatmap, score distributions, timeline, bubble explorer, radar charts, author network graph, category treemap, quality vs overlap, org bar chart, ideas chart, and interactive draft browser - Interactive draft browser (browser.html): filterable by category, keyword, score sliders with sortable table and expandable detail rows - arXiv paper (paper/main.tex): 13-page manuscript with all findings - Gap analysis: 12 identified under-addressed areas - Author network: collaboration graph, org contributions, cross-org analysis - Draft generation from gaps (ietf draft-gen) - Auto-load .env for API keys (python-dotenv) New modules: visualize.py, authors.py, draftgen.py New reports: timeline, overlap-matrix, authors, gaps New deps: plotly, matplotlib, seaborn, scipy, scikit-learn, networkx, python-dotenv Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
1
.gitignore
vendored
1
.gitignore
vendored
@@ -5,3 +5,4 @@ dist/
|
||||
build/
|
||||
data/config.json
|
||||
.claude/
|
||||
.env
|
||||
|
||||
229
README.md
Normal file
229
README.md
Normal file
@@ -0,0 +1,229 @@
|
||||
# IETF Draft Analyzer
|
||||
|
||||
Track, categorize, rate, and visualize AI/agent-related IETF Internet-Drafts.
|
||||
|
||||
**260 drafts** analyzed across **19 categories** with **403 authors**, **1,262 extracted ideas**, and **12 identified gaps** — spanning June 2025 to February 2026.
|
||||
|
||||
## What This Does
|
||||
|
||||
The IETF is experiencing an unprecedented wave of standardization activity around AI agents. This tool provides a quantitative lens on that activity:
|
||||
|
||||
- **Fetches** draft metadata and full text from the IETF Datatracker API
|
||||
- **Rates** each draft on 5 dimensions (novelty, maturity, overlap, momentum, relevance) using Claude
|
||||
- **Embeds** drafts with Ollama for pairwise similarity and clustering
|
||||
- **Extracts** discrete technical ideas and identifies landscape gaps
|
||||
- **Maps** the author collaboration network and organizational affiliations
|
||||
- **Generates** interactive visualizations, markdown reports, and a filterable browser
|
||||
- **Produces** publication-ready figures for an arXiv paper
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Install
|
||||
pip install -e .
|
||||
|
||||
# Set your API key (or add to .env file)
|
||||
export ANTHROPIC_API_KEY=sk-ant-...
|
||||
|
||||
# Fetch drafts from IETF Datatracker
|
||||
ietf fetch
|
||||
|
||||
# Rate all unrated drafts with Claude
|
||||
ietf analyze --all
|
||||
|
||||
# Generate embeddings (requires Ollama running locally)
|
||||
ietf embed
|
||||
|
||||
# Extract technical ideas
|
||||
ietf ideas --all
|
||||
|
||||
# Identify gaps in the landscape
|
||||
ietf gaps
|
||||
|
||||
# Generate all visualizations
|
||||
ietf viz all
|
||||
|
||||
# Open the interactive browser
|
||||
xdg-open data/figures/browser.html
|
||||
```
|
||||
|
||||
## CLI Commands
|
||||
|
||||
### Core Pipeline
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `ietf fetch` | Fetch AI/agent drafts from IETF Datatracker |
|
||||
| `ietf analyze --all` | Rate all unrated drafts using Claude (5 dimensions + summary) |
|
||||
| `ietf embed` | Generate semantic embeddings via Ollama |
|
||||
| `ietf ideas --all` | Extract technical ideas from drafts using Claude |
|
||||
| `ietf gaps` | Identify under-addressed areas in the landscape |
|
||||
| `ietf authors --fetch` | Fetch author/affiliation data from Datatracker |
|
||||
|
||||
### Exploration
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `ietf list` | List tracked drafts |
|
||||
| `ietf show <name>` | Show detailed info for a specific draft |
|
||||
| `ietf search <query>` | Full-text search across all stored drafts |
|
||||
| `ietf similar <name>` | Find the most similar drafts by embedding similarity |
|
||||
| `ietf clusters` | Find clusters of near-duplicate drafts |
|
||||
| `ietf compare <name1> <name2> ...` | Compare drafts for overlap and unique contributions |
|
||||
| `ietf authors` | Show top authors and their draft counts |
|
||||
| `ietf network` | Show organizational collaboration network |
|
||||
|
||||
### Visualizations (`ietf viz`)
|
||||
|
||||
All outputs go to `data/figures/`. Interactive charts are standalone HTML files (no server needed).
|
||||
|
||||
| Command | Output | Format |
|
||||
|---------|--------|--------|
|
||||
| `ietf viz all` | Generate everything below | mixed |
|
||||
| `ietf viz browser` | Filterable draft browser with search, category chips, score sliders | HTML |
|
||||
| `ietf viz landscape` | t-SNE/UMAP 2D scatter of all drafts colored by category | HTML |
|
||||
| `ietf viz heatmap` | 260x260 clustered pairwise similarity matrix | PNG |
|
||||
| `ietf viz distributions` | Violin plots for all 5 rating dimensions by category | PNG |
|
||||
| `ietf viz timeline` | Stacked area chart of monthly submissions by category | HTML |
|
||||
| `ietf viz bubble` | Novelty vs Maturity explorer (size=relevance, color=category) | HTML |
|
||||
| `ietf viz radar` | Average rating profile per category | HTML |
|
||||
| `ietf viz network` | Author co-authorship force-directed graph | HTML |
|
||||
| `ietf viz treemap` | Category composition treemap (color=avg score) | HTML |
|
||||
| `ietf viz quality` | Score vs uniqueness with quadrant annotations | HTML |
|
||||
| `ietf viz orgs` | Organization contribution horizontal bar chart | HTML |
|
||||
| `ietf viz ideas` | Ideas frequency by type | HTML |
|
||||
|
||||
### Reports (`ietf report`)
|
||||
|
||||
Markdown reports saved to `data/reports/`.
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `ietf report overview` | Sortable table of all rated drafts with bar-chart scores |
|
||||
| `ietf report landscape` | Category-grouped view with per-category rankings |
|
||||
| `ietf report timeline` | Monthly submission volume and category trends |
|
||||
| `ietf report overlap-matrix` | Top similar pairs, per-category overlap, cross-category matrix |
|
||||
| `ietf report authors` | Top authors, organizations, collaboration pairs |
|
||||
| `ietf report digest` | Weekly digest of recently fetched drafts |
|
||||
| `ietf report ideas` | Most common ideas, unique ideas, ideas by type |
|
||||
|
||||
### Other
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `ietf draft-gen <topic>` | Generate an Internet-Draft addressing a landscape gap |
|
||||
| `ietf config` | Show or modify configuration |
|
||||
|
||||
## Rating System
|
||||
|
||||
Each draft is scored 1-5 on five dimensions:
|
||||
|
||||
| Dimension | What it measures |
|
||||
|-----------|-----------------|
|
||||
| **Novelty** | Originality relative to existing standards |
|
||||
| **Maturity** | Completeness of specification |
|
||||
| **Overlap** | Redundancy with other drafts (5 = heavily overlapping) |
|
||||
| **Momentum** | Community engagement, revisions, adoption |
|
||||
| **Relevance** | Importance to the AI/agent ecosystem |
|
||||
|
||||
**Composite score:**
|
||||
|
||||
```
|
||||
score = 0.30 * novelty + 0.25 * relevance + 0.20 * maturity + 0.15 * momentum + 0.10 * (6 - overlap)
|
||||
```
|
||||
|
||||
## Key Findings
|
||||
|
||||
- **36x growth** in 9 months (2 drafts/month to 72)
|
||||
- **7.9% of draft pairs** exceed 0.80 cosine similarity — significant redundancy
|
||||
- **Safety deficit**: AI safety proposals (36) are vastly outnumbered by protocol proposals (290+)
|
||||
- **Organizational concentration**: Top 5 orgs contribute ~35% of all drafts
|
||||
- **1,262 technical ideas** extracted across 6 types (mechanism, architecture, protocol, pattern, extension, requirement)
|
||||
- **12 identified gaps** in the current landscape
|
||||
|
||||
## Tech Stack
|
||||
|
||||
- **Python 3.11+** with Click CLI
|
||||
- **SQLite** with FTS5 full-text search and WAL mode
|
||||
- **Anthropic Claude** (Sonnet 4) for analysis, rating, idea extraction, gap analysis
|
||||
- **Ollama** (nomic-embed-text) for local embeddings and similarity
|
||||
- **Plotly** for interactive HTML visualizations
|
||||
- **Matplotlib/Seaborn** for publication-ready static figures
|
||||
- **NetworkX** for author collaboration graph analysis
|
||||
- **NumPy/SciPy/scikit-learn** for similarity computation and dimensionality reduction
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
src/ietf_analyzer/
|
||||
cli.py # Click CLI entry point (all commands)
|
||||
fetcher.py # IETF Datatracker API client
|
||||
analyzer.py # Claude-based analysis, rating, idea extraction, gap analysis
|
||||
embeddings.py # Ollama embeddings + cosine similarity + clustering
|
||||
db.py # SQLite with FTS5 (7 tables: drafts, ratings, embeddings, llm_cache, authors, draft_authors, ideas, gaps)
|
||||
models.py # Author, Draft, Rating dataclasses
|
||||
reports.py # Markdown report generation
|
||||
visualize.py # Interactive HTML + static PNG visualizations
|
||||
authors.py # AuthorNetwork: Datatracker author fetching, collaboration graph
|
||||
draftgen.py # Internet-Draft generation from gap analysis
|
||||
config.py # Configuration with defaults
|
||||
|
||||
data/
|
||||
drafts.db # SQLite database (all analysis data)
|
||||
reports/ # Generated markdown reports
|
||||
figures/ # Generated visualizations (HTML + PNG)
|
||||
|
||||
paper/
|
||||
main.tex # arXiv paper: "The AI Agent Standardization Wave"
|
||||
export_figures.py # Export interactive charts to static images
|
||||
Makefile # Build: make pdf
|
||||
```
|
||||
|
||||
## Database Schema
|
||||
|
||||
| Table | Purpose | Records |
|
||||
|-------|---------|--------:|
|
||||
| `drafts` | Draft metadata + full text | 260 |
|
||||
| `ratings` | 5-dimension AI ratings + summary | 260 |
|
||||
| `embeddings` | Semantic vectors (nomic-embed-text) | 260 |
|
||||
| `llm_cache` | Claude API response cache | ~500 |
|
||||
| `authors` | Person records from Datatracker | 403 |
|
||||
| `draft_authors` | Author-draft relationships | 742 |
|
||||
| `ideas` | Extracted technical ideas | 1,262 |
|
||||
| `gaps` | Gap analysis results | 12 |
|
||||
| `drafts_fts` | FTS5 full-text search index | — |
|
||||
|
||||
## arXiv Paper
|
||||
|
||||
A 13-page paper is included in `paper/main.tex`:
|
||||
|
||||
> **The AI Agent Standardization Wave: A Quantitative Analysis of 260 IETF Internet-Drafts on Autonomous Agents and Artificial Intelligence**
|
||||
|
||||
Build with:
|
||||
|
||||
```bash
|
||||
cd paper
|
||||
python3 export_figures.py # copy/export figures
|
||||
pdflatex main.tex # compile (run twice for references)
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
```bash
|
||||
# Show current config
|
||||
ietf config
|
||||
|
||||
# Change Claude model
|
||||
ietf config --set claude_model claude-sonnet-4-20250514
|
||||
|
||||
# API key via .env file (auto-loaded)
|
||||
echo "ANTHROPIC_API_KEY=sk-ant-..." > .env
|
||||
```
|
||||
|
||||
## Cost
|
||||
|
||||
Full analysis of 260 drafts consumed ~475K API tokens (rating + idea extraction + gap analysis). At current Sonnet pricing, this is approximately $2-3 USD.
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
BIN
data/drafts.db
BIN
data/drafts.db
Binary file not shown.
7
data/figures/author-network.html
Normal file
7
data/figures/author-network.html
Normal file
File diff suppressed because one or more lines are too long
268
data/figures/browser.html
Normal file
268
data/figures/browser.html
Normal file
File diff suppressed because one or more lines are too long
7
data/figures/bubble-explorer.html
Normal file
7
data/figures/bubble-explorer.html
Normal file
File diff suppressed because one or more lines are too long
7
data/figures/category-radar.html
Normal file
7
data/figures/category-radar.html
Normal file
File diff suppressed because one or more lines are too long
7
data/figures/category-treemap.html
Normal file
7
data/figures/category-treemap.html
Normal file
File diff suppressed because one or more lines are too long
7
data/figures/ideas-by-type.html
Normal file
7
data/figures/ideas-by-type.html
Normal file
File diff suppressed because one or more lines are too long
7
data/figures/landscape-tsne.html
Normal file
7
data/figures/landscape-tsne.html
Normal file
File diff suppressed because one or more lines are too long
7
data/figures/org-contributions.html
Normal file
7
data/figures/org-contributions.html
Normal file
File diff suppressed because one or more lines are too long
BIN
data/figures/score-distributions.png
Normal file
BIN
data/figures/score-distributions.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 291 KiB |
7
data/figures/score-vs-overlap.html
Normal file
7
data/figures/score-vs-overlap.html
Normal file
File diff suppressed because one or more lines are too long
BIN
data/figures/similarity-heatmap.png
Normal file
BIN
data/figures/similarity-heatmap.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 394 KiB |
7
data/figures/timeline.html
Normal file
7
data/figures/timeline.html
Normal file
File diff suppressed because one or more lines are too long
109
data/reports/authors.md
Normal file
109
data/reports/authors.md
Normal file
@@ -0,0 +1,109 @@
|
||||
# Author & Organization Network
|
||||
*Generated 2026-02-28 10:40 UTC — 403 unique authors across 260 drafts*
|
||||
|
||||
## Top Authors by Draft Count
|
||||
|
||||
| # | Author | Organization | Drafts | Categories |
|
||||
|--:|--------|-------------|-------:|------------|
|
||||
| 1 | Bing Liu | Huawei | 21 | A2A protocols, AI safety/alignment, Agent discovery / registration |
|
||||
| 2 | Nan Geng | Huawei | 19 | A2A protocols, AI safety/alignment, Agent discovery / registration |
|
||||
| 3 | Zhenbin Li | Huawei | 19 | A2A protocols, AI safety/alignment, Agent discovery / registration |
|
||||
| 4 | Qiangzhou Gao | Huawei | 18 | A2A protocols, AI safety/alignment, Agent discovery / registration |
|
||||
| 5 | Xiaotong Shang | Huawei | 17 | A2A protocols, AI safety/alignment, Agent discovery / registration |
|
||||
| 6 | Jianwei Mao | Huawei | 11 | A2A protocols, AI safety/alignment, Agent discovery/reg |
|
||||
| 7 | Aijun Wang | China Telecom | 8 | A2A protocols, Agent discovery / registration, Agent discovery/reg |
|
||||
| 8 | Yong Cui | Tsinghua University | 8 | A2A protocols, AI safety/alignment, Agent discovery / registration |
|
||||
| 9 | Guanming Zeng | Huawei | 7 | A2A protocols, Agent discovery/reg, Autonomous netops |
|
||||
| 10 | Göran Selander | Ericsson | 7 | A2A protocols, Agent discovery/reg, Agent identity/auth |
|
||||
| 11 | Michael Richardson | Sandelman Software Works | 7 | Agent discovery/reg, Agent identity/auth, Autonomous netops |
|
||||
| 12 | Jonathan Rosenberg | Five9 | 7 | A2A protocols, AI safety/alignment, Agent discovery/reg |
|
||||
| 13 | Madhava Gaikwad | | 6 | A2A protocols, Agent discovery/reg, Agent identity/auth |
|
||||
| 14 | Pat White | Bitwave | 6 | A2A protocols, AI safety/alignment, Agent discovery/reg |
|
||||
| 15 | Cullen Fluffy Jennings | Cisco | 6 | A2A protocols, AI safety/alignment, Agent discovery / registration |
|
||||
| 16 | Li Zhang | Huawei Technologies | 5 | A2A protocols, Agent discovery / registration, Agent discovery/reg |
|
||||
| 17 | Tirumaleswar Reddy.K | Nokia | 5 | Agent identity/auth, Data formats/interop |
|
||||
| 18 | Roland Schott | Deutsche Telekom | 5 | A2A protocols, Agent discovery/reg, Agent identity/auth |
|
||||
| 19 | John Preuß Mattsson | Ericsson | 5 | A2A protocols, Agent discovery/reg, Agent identity/auth |
|
||||
| 20 | Chenguang Du | Zhongguancun Laboratory | 4 | A2A protocols, Agent discovery / registration, Agent discovery/reg |
|
||||
| 21 | Mengyao Han | China Unicom | 4 | A2A protocols, Agent discovery/reg, Agent identity/auth |
|
||||
| 22 | Aritra Banerjee | Nokia | 4 | Agent identity/auth |
|
||||
| 23 | Peter Chunchi Liu | Huawei | 4 | A2A protocols, AI safety/alignment, Agent identity/auth |
|
||||
| 24 | Xueting Li | China Telecom | 4 | A2A protocols, Agent discovery/reg, Agent identity/auth |
|
||||
| 25 | Lidia Pocero Fraile | ISI, R.C. ATHENA | 4 | A2A protocols, Agent identity/auth |
|
||||
| 26 | Meiling Chen | China Mobile | 4 | A2A protocols, AI safety / guardrails / alignment, AI safety/alignment |
|
||||
| 27 | Christos Koulamas | ISI, R.C. ATHENA | 4 | A2A protocols, Agent identity/auth |
|
||||
| 28 | Marco Tiloca | RISE AB | 4 | A2A protocols, Agent identity/auth, Data formats/interop |
|
||||
| 29 | Diego Lopez | Telefonica | 4 | A2A protocols, AI safety/alignment, Agent discovery/reg |
|
||||
| 30 | Li Su | China Mobile | 4 | A2A protocols, AI safety / guardrails / alignment, AI safety/alignment |
|
||||
|
||||
## Top Organizations
|
||||
|
||||
| # | Organization | Authors | Drafts |
|
||||
|--:|-------------|--------:|-------:|
|
||||
| 1 | Huawei | 30 | 25 |
|
||||
| 2 | China Mobile | 17 | 19 |
|
||||
| 3 | Huawei Technologies | 12 | 18 |
|
||||
| 4 | China Telecom | 17 | 15 |
|
||||
| 5 | China Unicom | 19 | 14 |
|
||||
| 6 | Cisco | 10 | 12 |
|
||||
| 7 | Tsinghua University | 7 | 11 |
|
||||
| 8 | Independent | 10 | 9 |
|
||||
| 9 | Cisco Systems | 10 | 9 |
|
||||
| 10 | Sandelman Software Works | 1 | 7 |
|
||||
| 11 | Independent Researcher | 4 | 7 |
|
||||
| 12 | Five9 | 1 | 7 |
|
||||
| 13 | Zhongguancun Laboratory | 4 | 6 |
|
||||
| 14 | ZTE Corporation | 8 | 6 |
|
||||
| 15 | Bitwave | 1 | 6 |
|
||||
| 16 | Nokia | 2 | 5 |
|
||||
| 17 | Inria | 5 | 5 |
|
||||
| 18 | Unaffiliated | 1 | 4 |
|
||||
| 19 | Telefonica | 2 | 4 |
|
||||
| 20 | ISI, R.C. ATHENA | 4 | 4 |
|
||||
|
||||
## Strongest Collaboration Pairs
|
||||
|
||||
| Author A | Author B | Shared Drafts |
|
||||
|----------|----------|-----:|
|
||||
| Bing Liu | Nan Geng | 18 |
|
||||
| Zhenbin Li | Qiangzhou Gao | 18 |
|
||||
| Zhenbin Li | Nan Geng | 18 |
|
||||
| Bing Liu | Zhenbin Li | 17 |
|
||||
| Bing Liu | Qiangzhou Gao | 17 |
|
||||
| Bing Liu | Xiaotong Shang | 17 |
|
||||
| Zhenbin Li | Xiaotong Shang | 17 |
|
||||
| Qiangzhou Gao | Nan Geng | 17 |
|
||||
| Qiangzhou Gao | Xiaotong Shang | 17 |
|
||||
| Nan Geng | Xiaotong Shang | 17 |
|
||||
| Zhenbin Li | Jianwei Mao | 10 |
|
||||
| Jianwei Mao | Nan Geng | 10 |
|
||||
| Bing Liu | Jianwei Mao | 9 |
|
||||
| Jianwei Mao | Qiangzhou Gao | 9 |
|
||||
| Jianwei Mao | Xiaotong Shang | 9 |
|
||||
| Jonathan Rosenberg | Pat White | 6 |
|
||||
| Jianwei Mao | Guanming Zeng | 6 |
|
||||
| John Preuß Mattsson | Göran Selander | 5 |
|
||||
| Bing Liu | Li Zhang | 5 |
|
||||
| Bing Liu | Guanming Zeng | 5 |
|
||||
|
||||
## Cross-Organization Collaboration
|
||||
|
||||
| Org A | Org B | Shared Drafts |
|
||||
|-------|-------|-----:|
|
||||
| Five9 | Bitwave | 6 |
|
||||
| Tsinghua University | Zhongguancun Laboratory | 5 |
|
||||
| Huawei | China Mobile | 4 |
|
||||
| Huawei | China Unicom | 4 |
|
||||
| China Mobile | ZTE Corporation | 3 |
|
||||
| Five9 | Cisco | 3 |
|
||||
| Huawei | China Telecom | 3 |
|
||||
| Unaffiliated | Deutsche Telekom | 3 |
|
||||
| Unaffiliated | Infoblox, Inc. | 3 |
|
||||
| AWS | University of Waterloo | 2 |
|
||||
| Aryaka | JPMorgan Chase & Co | 2 |
|
||||
| Aryaka | Oracle | 2 |
|
||||
| Aryaka | Telefonica | 2 |
|
||||
| Beijing University of Posts and
|
||||
Telecommunications | China Telecom | 2 |
|
||||
| Beijing University of Posts and
|
||||
Telecommunications | Tsinghua University | 2 |
|
||||
80
data/reports/gaps.md
Normal file
80
data/reports/gaps.md
Normal file
@@ -0,0 +1,80 @@
|
||||
# Gap Analysis: IETF AI/Agent Draft Landscape
|
||||
*Generated 2026-02-28 12:14 UTC — analyzing 260 drafts*
|
||||
|
||||
### 1. Agent Resource Management
|
||||
**Severity:** CRITICAL
|
||||
**Category:** autonomous netops
|
||||
**Description:** No comprehensive framework for managing computational resources, memory, and processing power across distributed AI agents. Current drafts focus on communication but ignore how agents compete for and share limited resources in multi-agent environments.
|
||||
**Evidence:** Real deployments will face resource contention, but no drafts address scheduling, quotas, or fair allocation mechanisms
|
||||
|
||||
### 2. Agent Behavior Verification
|
||||
**Severity:** CRITICAL
|
||||
**Category:** AI safety/alignment
|
||||
**Description:** No mechanisms to verify that deployed agents actually behave according to their declared policies or specifications. Gap between stated capabilities and runtime behavior validation.
|
||||
**Evidence:** Only 36 safety drafts vs 260 total, and no mention of runtime behavior verification in technical ideas
|
||||
|
||||
### 3. Agent Error Recovery and Rollback
|
||||
**Severity:** CRITICAL
|
||||
**Category:** autonomous netops
|
||||
**Description:** Missing standards for how agents handle and recover from errors, particularly cascading failures across agent networks. No rollback mechanisms for autonomous decisions gone wrong.
|
||||
**Evidence:** Autonomous operations imply unsupervised decisions, but no error recovery mechanisms identified
|
||||
|
||||
### 4. Cross-Protocol Translation
|
||||
**Severity:** HIGH
|
||||
**Category:** A2A protocols
|
||||
**Description:** With 92 A2A protocol drafts and high overlap, there's no standard way for agents using different communication protocols to interoperate. Missing universal translation layer or protocol negotiation mechanism.
|
||||
**Evidence:** Multiple competing A2A protocols with no interoperability framework suggests fragmentation problem
|
||||
|
||||
### 5. Agent Lifecycle Management
|
||||
**Severity:** HIGH
|
||||
**Category:** agent discovery/reg
|
||||
**Description:** Missing standards for agent deployment, versioning, updates, and retirement. No clear protocols for how agents evolve or get replaced without disrupting dependent services.
|
||||
**Evidence:** Registration covered but no mention of versioning, updates, or graceful shutdown procedures
|
||||
|
||||
### 6. Multi-Agent Consensus Mechanisms
|
||||
**Severity:** HIGH
|
||||
**Category:** A2A protocols
|
||||
**Description:** No frameworks for how groups of AI agents reach consensus on conflicting decisions or priorities. Critical for autonomous systems that must coordinate without human intervention.
|
||||
**Evidence:** Autonomous netops requires coordination but no consensus mechanisms appear in technical ideas list
|
||||
|
||||
### 7. Human Override and Intervention
|
||||
**Severity:** HIGH
|
||||
**Category:** human-agent interaction
|
||||
**Description:** Only 22 human-agent interaction drafts but no clear emergency override protocols. Missing standardized ways for humans to intervene in autonomous agent operations during critical situations.
|
||||
**Evidence:** Disproportionately low human interaction focus (22 drafts) compared to autonomous operations (60 drafts)
|
||||
|
||||
### 8. Cross-Domain Security Boundaries
|
||||
**Severity:** HIGH
|
||||
**Category:** agent identity/auth
|
||||
**Description:** While identity management exists, missing frameworks for agents operating across security domains with different trust levels. No clear isolation or privilege escalation prevention.
|
||||
**Evidence:** Cross-domain identity mentioned but no corresponding security boundary enforcement mechanisms
|
||||
|
||||
### 9. Dynamic Trust and Reputation
|
||||
**Severity:** HIGH
|
||||
**Category:** agent identity/auth
|
||||
**Description:** Missing frameworks for agents to build, assess, and revoke trust relationships dynamically based on behavior history. Static authentication insufficient for long-running autonomous systems.
|
||||
**Evidence:** Certificate authorities mentioned but no dynamic trust or reputation systems in technical ideas
|
||||
|
||||
### 10. Agent Performance Monitoring
|
||||
**Severity:** MEDIUM
|
||||
**Category:** autonomous netops
|
||||
**Description:** No standardized metrics or monitoring frameworks for tracking agent performance, efficiency, or drift over time. Missing observability standards for production agent deployments.
|
||||
**Evidence:** ML traffic management only has 24 drafts but no performance monitoring in technical ideas
|
||||
|
||||
### 11. Agent Explainability Standards
|
||||
**Severity:** MEDIUM
|
||||
**Category:** AI safety/alignment
|
||||
**Description:** No protocols for agents to explain their decisions or reasoning to other agents or humans. Critical gap for debugging and compliance in regulated environments.
|
||||
**Evidence:** Low safety/alignment focus suggests governance requirements not fully addressed
|
||||
|
||||
### 12. Agent Data Provenance
|
||||
**Severity:** MEDIUM
|
||||
**Category:** data formats/interop
|
||||
**Description:** No standards for tracking data lineage and provenance as information flows between agents. Critical for compliance and debugging in complex agent networks.
|
||||
**Evidence:** 102 data format drafts but no provenance tracking mechanisms identified in technical ideas
|
||||
|
||||
## Summary by Severity
|
||||
|
||||
- **Critical:** 3 gaps
|
||||
- **High:** 6 gaps
|
||||
- **Medium:** 3 gaps
|
||||
111
data/reports/overlap-matrix.md
Normal file
111
data/reports/overlap-matrix.md
Normal file
@@ -0,0 +1,111 @@
|
||||
# Overlap Matrix Report
|
||||
*Generated 2026-02-28 10:28 UTC — 260x260 pairwise similarities*
|
||||
|
||||
## Top 50 Most Similar Pairs
|
||||
|
||||
| Rank | Similarity | Draft A | Draft B |
|
||||
|-----:|-----------:|---------|---------|
|
||||
| 1 | 0.999 | [draft-rosenberg-aiproto](https://datatracker.ietf.org/doc/draft-rosenberg-aiproto/) | [draft-rosenberg-aiproto-nact](https://datatracker.ietf.org/doc/draft-rosenberg-aiproto-nact/) |
|
||||
| 2 | 0.999 | [draft-lake-pocero-authkem-ikr-edhoc](https://datatracker.ietf.org/doc/draft-lake-pocero-authkem-ikr-edhoc/) | [draft-pocero-authkem-ikr-edhoc](https://datatracker.ietf.org/doc/draft-pocero-authkem-ikr-edhoc/) |
|
||||
| 3 | 0.997 | [draft-ahn-nmrg-5g-security-i2nsf-framework](https://datatracker.ietf.org/doc/draft-ahn-nmrg-5g-security-i2nsf-framework/) | [draft-ahn-opsawg-5g-security-i2nsf-framework](https://datatracker.ietf.org/doc/draft-ahn-opsawg-5g-security-i2nsf-framework/) |
|
||||
| 4 | 0.995 | [draft-ietf-emu-pqc-eapaka](https://datatracker.ietf.org/doc/draft-ietf-emu-pqc-eapaka/) | [draft-ra-emu-pqc-eapaka](https://datatracker.ietf.org/doc/draft-ra-emu-pqc-eapaka/) |
|
||||
| 5 | 0.995 | [draft-men-rtgwg-agent-networking-digibank-scenario](https://datatracker.ietf.org/doc/draft-men-rtgwg-agent-networking-digibank-scenario/) | [draft-men-rtgwg-agent-networking-in-digibank](https://datatracker.ietf.org/doc/draft-men-rtgwg-agent-networking-in-digibank/) |
|
||||
| 6 | 0.995 | [draft-sun-zhang-iaip](https://datatracker.ietf.org/doc/draft-sun-zhang-iaip/) | [draft-sz-dmsc-iaip](https://datatracker.ietf.org/doc/draft-sz-dmsc-iaip/) |
|
||||
| 7 | 0.994 | [draft-ar-emu-hybrid-pqc-eapaka](https://datatracker.ietf.org/doc/draft-ar-emu-hybrid-pqc-eapaka/) | [draft-ietf-emu-hybrid-pqc-eapaka](https://datatracker.ietf.org/doc/draft-ietf-emu-hybrid-pqc-eapaka/) |
|
||||
| 8 | 0.994 | [draft-rosenberg-aiproto-cheq](https://datatracker.ietf.org/doc/draft-rosenberg-aiproto-cheq/) | [draft-rosenberg-cheq](https://datatracker.ietf.org/doc/draft-rosenberg-cheq/) |
|
||||
| 9 | 0.993 | [draft-a2a-moqt-transport](https://datatracker.ietf.org/doc/draft-a2a-moqt-transport/) | [draft-nandakumar-a2a-moqt-transport](https://datatracker.ietf.org/doc/draft-nandakumar-a2a-moqt-transport/) |
|
||||
| 10 | 0.992 | [draft-lake-pocero-authkem-edhoc](https://datatracker.ietf.org/doc/draft-lake-pocero-authkem-edhoc/) | [draft-pocero-authkem-edhoc](https://datatracker.ietf.org/doc/draft-pocero-authkem-edhoc/) |
|
||||
| 11 | 0.990 | [draft-zl-agents-networking-framework](https://datatracker.ietf.org/doc/draft-zl-agents-networking-framework/) | [draft-zlgsgl-rtgwg-agents-networking-framework](https://datatracker.ietf.org/doc/draft-zlgsgl-rtgwg-agents-networking-framework/) |
|
||||
| 12 | 0.988 | [draft-ietf-httpbis-layered-cookies](https://datatracker.ietf.org/doc/draft-ietf-httpbis-layered-cookies/) | [draft-ietf-httpbis-rfc6265bis](https://datatracker.ietf.org/doc/draft-ietf-httpbis-rfc6265bis/) |
|
||||
| 13 | 0.987 | [draft-zheng-agent-identity-management](https://datatracker.ietf.org/doc/draft-zheng-agent-identity-management/) | [draft-zheng-dispatch-agent-identity-management](https://datatracker.ietf.org/doc/draft-zheng-dispatch-agent-identity-management/) |
|
||||
| 14 | 0.987 | [draft-eggert-mailmaint-uaautoconf](https://datatracker.ietf.org/doc/draft-eggert-mailmaint-uaautoconf/) | [draft-ietf-mailmaint-pacc](https://datatracker.ietf.org/doc/draft-ietf-mailmaint-pacc/) |
|
||||
| 15 | 0.986 | [draft-yang-dmsc-ioa-task-protocol](https://datatracker.ietf.org/doc/draft-yang-dmsc-ioa-task-protocol/) | [draft-yang-ioa-protocol](https://datatracker.ietf.org/doc/draft-yang-ioa-protocol/) |
|
||||
| 16 | 0.986 | [draft-abbey-scim-agent-extension](https://datatracker.ietf.org/doc/draft-abbey-scim-agent-extension/) | [draft-scim-agent-extension](https://datatracker.ietf.org/doc/draft-scim-agent-extension/) |
|
||||
| 17 | 0.986 | [draft-aylward-aiga-1](https://datatracker.ietf.org/doc/draft-aylward-aiga-1/) | [draft-aylward-aiga-2](https://datatracker.ietf.org/doc/draft-aylward-aiga-2/) |
|
||||
| 18 | 0.984 | [draft-pang-agents-networking-scenarios](https://datatracker.ietf.org/doc/draft-pang-agents-networking-scenarios/) | [draft-zl-agents-networking-scenarios](https://datatracker.ietf.org/doc/draft-zl-agents-networking-scenarios/) |
|
||||
| 19 | 0.982 | [draft-mao-rtgwg-agent-comm-protocol-gap-analysis](https://datatracker.ietf.org/doc/draft-mao-rtgwg-agent-comm-protocol-gap-analysis/) | [draft-mzsg-rtgwg-agent-cross-device-comm-framework](https://datatracker.ietf.org/doc/draft-mzsg-rtgwg-agent-cross-device-comm-framework/) |
|
||||
| 20 | 0.980 | [draft-ietf-lamps-est-renewal-info](https://datatracker.ietf.org/doc/draft-ietf-lamps-est-renewal-info/) | [draft-yusef-lamps-rfc7030-renewal-recommendation](https://datatracker.ietf.org/doc/draft-yusef-lamps-rfc7030-renewal-recommendation/) |
|
||||
| 21 | 0.978 | [draft-yu-ai-agent-use-cases-in-6g](https://datatracker.ietf.org/doc/draft-yu-ai-agent-use-cases-in-6g/) | [draft-yu-dmsc-ai-agent-use-cases-in-6g](https://datatracker.ietf.org/doc/draft-yu-dmsc-ai-agent-use-cases-in-6g/) |
|
||||
| 22 | 0.972 | [draft-bastian-jose-dvs](https://datatracker.ietf.org/doc/draft-bastian-jose-dvs/) | [draft-bastian-jose-pkdh](https://datatracker.ietf.org/doc/draft-bastian-jose-pkdh/) |
|
||||
| 23 | 0.971 | [draft-li-dmsc-macp](https://datatracker.ietf.org/doc/draft-li-dmsc-macp/) | [draft-li-dmsc-mcps-agw](https://datatracker.ietf.org/doc/draft-li-dmsc-mcps-agw/) |
|
||||
| 24 | 0.967 | [draft-zeng-mcp-troubleshooting](https://datatracker.ietf.org/doc/draft-zeng-mcp-troubleshooting/) | [draft-zm-rtgwg-mcp-troubleshooting](https://datatracker.ietf.org/doc/draft-zm-rtgwg-mcp-troubleshooting/) |
|
||||
| 25 | 0.965 | [draft-rosenberg-aiproto](https://datatracker.ietf.org/doc/draft-rosenberg-aiproto/) | [draft-rosenberg-aiproto-a2t](https://datatracker.ietf.org/doc/draft-rosenberg-aiproto-a2t/) |
|
||||
| 26 | 0.964 | [draft-rosenberg-aiproto-a2t](https://datatracker.ietf.org/doc/draft-rosenberg-aiproto-a2t/) | [draft-rosenberg-aiproto-nact](https://datatracker.ietf.org/doc/draft-rosenberg-aiproto-nact/) |
|
||||
| 27 | 0.961 | [draft-ietf-emu-hybrid-pqc-eapaka](https://datatracker.ietf.org/doc/draft-ietf-emu-hybrid-pqc-eapaka/) | [draft-ietf-emu-pqc-eapaka](https://datatracker.ietf.org/doc/draft-ietf-emu-pqc-eapaka/) |
|
||||
| 28 | 0.959 | [draft-bernardos-cats-isac-uc](https://datatracker.ietf.org/doc/draft-bernardos-cats-isac-uc/) | [draft-bernardos-green-isac-uc](https://datatracker.ietf.org/doc/draft-bernardos-green-isac-uc/) |
|
||||
| 29 | 0.959 | [draft-happel-structured-email-trust](https://datatracker.ietf.org/doc/draft-happel-structured-email-trust/) | [draft-ietf-sml-trust](https://datatracker.ietf.org/doc/draft-ietf-sml-trust/) |
|
||||
| 30 | 0.959 | [draft-gaikwad-llm-benchmarking-methodology](https://datatracker.ietf.org/doc/draft-gaikwad-llm-benchmarking-methodology/) | [draft-gaikwad-llm-benchmarking-terminology](https://datatracker.ietf.org/doc/draft-gaikwad-llm-benchmarking-terminology/) |
|
||||
| 31 | 0.955 | [draft-ar-emu-hybrid-pqc-eapaka](https://datatracker.ietf.org/doc/draft-ar-emu-hybrid-pqc-eapaka/) | [draft-ietf-emu-pqc-eapaka](https://datatracker.ietf.org/doc/draft-ietf-emu-pqc-eapaka/) |
|
||||
| 32 | 0.955 | [draft-ietf-emu-hybrid-pqc-eapaka](https://datatracker.ietf.org/doc/draft-ietf-emu-hybrid-pqc-eapaka/) | [draft-ra-emu-pqc-eapaka](https://datatracker.ietf.org/doc/draft-ra-emu-pqc-eapaka/) |
|
||||
| 33 | 0.953 | [draft-ar-emu-hybrid-pqc-eapaka](https://datatracker.ietf.org/doc/draft-ar-emu-hybrid-pqc-eapaka/) | [draft-ra-emu-pqc-eapaka](https://datatracker.ietf.org/doc/draft-ra-emu-pqc-eapaka/) |
|
||||
| 34 | 0.953 | [draft-ramakrishna-satp-data-sharing](https://datatracker.ietf.org/doc/draft-ramakrishna-satp-data-sharing/) | [draft-ramakrishna-satp-views-addresses](https://datatracker.ietf.org/doc/draft-ramakrishna-satp-views-addresses/) |
|
||||
| 35 | 0.952 | [draft-zhul-dhc-bnc-up-specific-suboption](https://datatracker.ietf.org/doc/draft-zhul-dhc-bnc-up-specific-suboption/) | [draft-zhul-intarea-bnc-up-specific-suboption](https://datatracker.ietf.org/doc/draft-zhul-intarea-bnc-up-specific-suboption/) |
|
||||
| 36 | 0.950 | [draft-cui-nmrg-llm-nm](https://datatracker.ietf.org/doc/draft-cui-nmrg-llm-nm/) | [draft-irtf-nmrg-llm-nm](https://datatracker.ietf.org/doc/draft-irtf-nmrg-llm-nm/) |
|
||||
| 37 | 0.946 | [draft-templin-manet-inet](https://datatracker.ietf.org/doc/draft-templin-manet-inet/) | [draft-templin-manet-inet-omni](https://datatracker.ietf.org/doc/draft-templin-manet-inet-omni/) |
|
||||
| 38 | 0.942 | [draft-zl-agents-networking-architecture](https://datatracker.ietf.org/doc/draft-zl-agents-networking-architecture/) | [draft-zl-agents-networking-framework](https://datatracker.ietf.org/doc/draft-zl-agents-networking-framework/) |
|
||||
| 39 | 0.936 | [draft-mozleywilliams-dnsop-bandaid](https://datatracker.ietf.org/doc/draft-mozleywilliams-dnsop-bandaid/) | [draft-mozleywilliams-dnsop-dnsaid](https://datatracker.ietf.org/doc/draft-mozleywilliams-dnsop-dnsaid/) |
|
||||
| 40 | 0.935 | [draft-pocero-authkem-edhoc](https://datatracker.ietf.org/doc/draft-pocero-authkem-edhoc/) | [draft-pocero-authkem-ikr-edhoc](https://datatracker.ietf.org/doc/draft-pocero-authkem-ikr-edhoc/) |
|
||||
| 41 | 0.934 | [draft-lake-pocero-authkem-edhoc](https://datatracker.ietf.org/doc/draft-lake-pocero-authkem-edhoc/) | [draft-spm-lake-pqsuites](https://datatracker.ietf.org/doc/draft-spm-lake-pqsuites/) |
|
||||
| 42 | 0.933 | [draft-lake-pocero-authkem-ikr-edhoc](https://datatracker.ietf.org/doc/draft-lake-pocero-authkem-ikr-edhoc/) | [draft-pocero-authkem-edhoc](https://datatracker.ietf.org/doc/draft-pocero-authkem-edhoc/) |
|
||||
| 43 | 0.932 | [draft-zhang-rtgwg-ai-agents-measurement](https://datatracker.ietf.org/doc/draft-zhang-rtgwg-ai-agents-measurement/) | [draft-zhang-rtgwg-ai-agents-troubleshooting](https://datatracker.ietf.org/doc/draft-zhang-rtgwg-ai-agents-troubleshooting/) |
|
||||
| 44 | 0.931 | [draft-lake-pocero-authkem-edhoc](https://datatracker.ietf.org/doc/draft-lake-pocero-authkem-edhoc/) | [draft-pocero-authkem-ikr-edhoc](https://datatracker.ietf.org/doc/draft-pocero-authkem-ikr-edhoc/) |
|
||||
| 45 | 0.930 | [draft-lake-pocero-authkem-edhoc](https://datatracker.ietf.org/doc/draft-lake-pocero-authkem-edhoc/) | [draft-lake-pocero-authkem-ikr-edhoc](https://datatracker.ietf.org/doc/draft-lake-pocero-authkem-ikr-edhoc/) |
|
||||
| 46 | 0.930 | [draft-pocero-authkem-edhoc](https://datatracker.ietf.org/doc/draft-pocero-authkem-edhoc/) | [draft-spm-lake-pqsuites](https://datatracker.ietf.org/doc/draft-spm-lake-pqsuites/) |
|
||||
| 47 | 0.925 | [draft-scim-agent-extension](https://datatracker.ietf.org/doc/draft-scim-agent-extension/) | [draft-wahl-scim-agent-schema](https://datatracker.ietf.org/doc/draft-wahl-scim-agent-schema/) |
|
||||
| 48 | 0.921 | [draft-abbey-scim-agent-extension](https://datatracker.ietf.org/doc/draft-abbey-scim-agent-extension/) | [draft-wahl-scim-agent-schema](https://datatracker.ietf.org/doc/draft-wahl-scim-agent-schema/) |
|
||||
| 49 | 0.918 | [draft-zl-agents-networking-architecture](https://datatracker.ietf.org/doc/draft-zl-agents-networking-architecture/) | [draft-zlgsgl-rtgwg-agents-networking-framework](https://datatracker.ietf.org/doc/draft-zlgsgl-rtgwg-agents-networking-framework/) |
|
||||
| 50 | 0.906 | [draft-agent-gw](https://datatracker.ietf.org/doc/draft-agent-gw/) | [draft-li-dmsc-inf-architecture](https://datatracker.ietf.org/doc/draft-li-dmsc-inf-architecture/) |
|
||||
|
||||
## Per-Category Internal Overlap
|
||||
|
||||
| Category | Drafts | Avg Pairwise Sim | Most Similar Pair |
|
||||
|----------|-------:|-----------------:|-------------------|
|
||||
| A2A protocols | 92 | 0.758 | rosenberg-aiproto / rosenberg-aiproto-nact (0.999) |
|
||||
| AI safety / guardrails / alignment | 1 | — | — |
|
||||
| AI safety/alignment | 36 | 0.765 | rosenberg-aiproto-cheq / rosenberg-cheq (0.994) |
|
||||
| Agent discovery / registration | 14 | 0.779 | campbell-agentic-http / narvaneni-agent-uri (0.863) |
|
||||
| Agent discovery/reg | 57 | 0.759 | rosenberg-aiproto / rosenberg-aiproto-nact (0.999) |
|
||||
| Agent identity/auth | 98 | 0.740 | lake-pocero-authkem-ikr-e / pocero-authkem-ikr-edhoc (0.999) |
|
||||
| Agent-to-agent communication protocols | 16 | 0.785 | campbell-agentic-http / narvaneni-agent-uri (0.863) |
|
||||
| Autonomous netops | 60 | 0.745 | ahn-nmrg-5g-security-i2ns / ahn-opsawg-5g-security-i2 (0.997) |
|
||||
| Autonomous network operations | 5 | 0.785 | zhang-agent-gap-network / zhang-rtgwg-ai-agents-tro (0.826) |
|
||||
| Data formats / semantics for AI interop | 3 | 0.801 | liu-agent-context-protoco / narvaneni-agent-uri (0.814) |
|
||||
| Data formats/interop | 102 | 0.715 | rosenberg-aiproto / rosenberg-aiproto-nact (0.999) |
|
||||
| Human-agent interaction | 22 | 0.739 | rosenberg-aiproto-cheq / rosenberg-cheq (0.994) |
|
||||
| Identity / authentication for AI agents | 13 | 0.784 | chen-agent-decoupled-auth / song-oauth-ai-agent-autho (0.882) |
|
||||
| ML traffic mgmt | 23 | 0.739 | tong-network-agent-use-ca / yu-dmsc-ai-agent-use-case (0.892) |
|
||||
| ML-based traffic management / optimization | 1 | — | — |
|
||||
| Model serving/inference | 13 | 0.733 | gaikwad-llm-benchmarking- / gaikwad-llm-benchmarking- (0.959) |
|
||||
| Other AI/agent | 21 | 0.709 | ietf-httpbis-layered-cook / ietf-httpbis-rfc6265bis (0.988) |
|
||||
| Policy / governance / ethical frameworks | 2 | 0.882 | chen-agent-decoupled-auth / song-oauth-ai-agent-autho (0.882) |
|
||||
| Policy/governance | 60 | 0.737 | ahn-nmrg-5g-security-i2ns / ahn-opsawg-5g-security-i2 (0.997) |
|
||||
|
||||
## Category Cross-Overlap
|
||||
|
||||
Average similarity between drafts in different categories.
|
||||
|
||||
| |A2A protoc | AI safety | AI safety/ | Agent disc | Agent disc | Agent iden | Agent-to-a | Autonomous | Autonomous | Data forma | Data forma | Human-agen | Identity / | ML traffic | ML-based t | Model serv | Other AI/a | Policy / g | Policy/gov |
|
||||
|-|---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| **A2A protocol** | 0.76 | 0.75 | 0.74 | 0.77 | 0.76 | 0.73 | 0.77 | 0.74 | 0.76 | 0.77 | 0.73 | 0.74 | 0.76 | 0.73 | 0.74 | 0.71 | 0.72 | 0.75 | 0.74 |
|
||||
| **AI safety / ** | | 0.00 | 0.78 | 0.77 | 0.75 | 0.76 | 0.76 | 0.71 | 0.73 | 0.75 | 0.72 | 0.75 | 0.78 | 0.70 | 0.72 | 0.69 | 0.71 | 0.88 | 0.76 |
|
||||
| **AI safety/al** | | | 0.76 | 0.76 | 0.74 | 0.75 | 0.75 | 0.71 | 0.73 | 0.75 | 0.73 | 0.74 | 0.77 | 0.71 | 0.72 | 0.70 | 0.71 | 0.78 | 0.75 |
|
||||
| **Agent discov** | | | | 0.78 | 0.77 | 0.74 | 0.78 | 0.74 | 0.77 | 0.78 | 0.74 | 0.75 | 0.78 | 0.74 | 0.76 | 0.71 | 0.73 | 0.76 | 0.75 |
|
||||
| **Agent discov** | | | | | 0.76 | 0.73 | 0.77 | 0.74 | 0.76 | 0.77 | 0.73 | 0.74 | 0.76 | 0.73 | 0.75 | 0.71 | 0.72 | 0.75 | 0.74 |
|
||||
| **Agent identi** | | | | | | 0.74 | 0.74 | 0.71 | 0.72 | 0.73 | 0.72 | 0.73 | 0.75 | 0.70 | 0.71 | 0.69 | 0.71 | 0.76 | 0.73 |
|
||||
| **Agent-to-age** | | | | | | | 0.79 | 0.75 | 0.78 | 0.79 | 0.74 | 0.75 | 0.78 | 0.75 | 0.77 | 0.72 | 0.73 | 0.76 | 0.75 |
|
||||
| **Autonomous n** | | | | | | | | 0.74 | 0.76 | 0.73 | 0.71 | 0.72 | 0.73 | 0.74 | 0.76 | 0.71 | 0.72 | 0.71 | 0.72 |
|
||||
| **Autonomous n** | | | | | | | | | 0.79 | 0.78 | 0.73 | 0.74 | 0.76 | 0.75 | 0.79 | 0.72 | 0.73 | 0.73 | 0.73 |
|
||||
| **Data formats** | | | | | | | | | | 0.80 | 0.74 | 0.75 | 0.77 | 0.74 | 0.76 | 0.71 | 0.73 | 0.76 | 0.74 |
|
||||
| **Data formats** | | | | | | | | | | | 0.72 | 0.72 | 0.74 | 0.71 | 0.72 | 0.70 | 0.71 | 0.73 | 0.72 |
|
||||
| **Human-agent ** | | | | | | | | | | | | 0.74 | 0.75 | 0.72 | 0.73 | 0.71 | 0.72 | 0.76 | 0.74 |
|
||||
| **Identity / a** | | | | | | | | | | | | | 0.78 | 0.73 | 0.75 | 0.71 | 0.73 | 0.79 | 0.76 |
|
||||
| **ML traffic m** | | | | | | | | | | | | | | 0.74 | 0.75 | 0.73 | 0.71 | 0.71 | 0.71 |
|
||||
| **ML-based tra** | | | | | | | | | | | | | | | 0.00 | 0.73 | 0.73 | 0.72 | 0.72 |
|
||||
| **Model servin** | | | | | | | | | | | | | | | | 0.73 | 0.70 | 0.69 | 0.70 |
|
||||
| **Other AI/age** | | | | | | | | | | | | | | | | | 0.71 | 0.71 | 0.71 |
|
||||
| **Policy / gov** | | | | | | | | | | | | | | | | | | 0.88 | 0.77 |
|
||||
| **Policy/gover** | | | | | | | | | | | | | | | | | | | 0.74 |
|
||||
|
||||
## Most Unique Drafts (max similarity < 0.70)
|
||||
|
||||
No drafts with max similarity below 0.70.
|
||||
38
data/reports/timeline.md
Normal file
38
data/reports/timeline.md
Normal file
@@ -0,0 +1,38 @@
|
||||
# IETF AI/Agent Drafts Timeline
|
||||
*Generated 2026-02-28 10:27 UTC — 260 drafts across 9 months*
|
||||
|
||||
## Monthly Submission Volume
|
||||
|
||||
```
|
||||
2025-06 | # 2
|
||||
2025-07 | ## 4
|
||||
2025-08 | ### 7
|
||||
2025-09 | ##### 10
|
||||
2025-10 | ########################### 50
|
||||
2025-11 | ############################ 51
|
||||
2025-12 | ####### 13
|
||||
2026-01 | ############################ 51
|
||||
2026-02 | ######################################## 72
|
||||
```
|
||||
|
||||
## Category Breakdown by Month
|
||||
|
||||
| Month | A2A protocol | AI safety / | AI safety/al | Agent discov | Agent discov | Agent identi | Agent-to-age | Autonomous n | Autonomous n | Data formats | Data formats | Human-agent | Identity / a | ML traffic m | ML-based tra | Model servin | Other AI/age | Policy / gov | Policy/gover | Total |
|
||||
|---------|------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | ------------- | -----:|
|
||||
| 2025-06 | 1 | 0 | 0 | 0 | 0 | 2 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 2 |
|
||||
| 2025-07 | 2 | 0 | 1 | 0 | 1 | 2 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 4 |
|
||||
| 2025-08 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 2 | 0 | 0 | 5 | 0 | 0 | 0 | 0 | 0 | 2 | 0 | 2 | 7 |
|
||||
| 2025-09 | 3 | 0 | 2 | 0 | 2 | 2 | 0 | 4 | 0 | 0 | 6 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 3 | 10 |
|
||||
| 2025-10 | 22 | 0 | 5 | 2 | 13 | 23 | 2 | 12 | 0 | 1 | 24 | 6 | 1 | 4 | 0 | 2 | 5 | 0 | 7 | 50 |
|
||||
| 2025-11 | 19 | 0 | 7 | 2 | 15 | 19 | 3 | 14 | 1 | 0 | 21 | 7 | 2 | 6 | 0 | 3 | 2 | 0 | 10 | 51 |
|
||||
| 2025-12 | 2 | 0 | 3 | 1 | 2 | 7 | 0 | 3 | 0 | 0 | 4 | 0 | 1 | 1 | 0 | 1 | 2 | 0 | 4 | 13 |
|
||||
| 2026-01 | 12 | 0 | 7 | 7 | 8 | 14 | 9 | 9 | 4 | 2 | 15 | 2 | 4 | 6 | 1 | 4 | 3 | 1 | 12 | 51 |
|
||||
| 2026-02 | 31 | 1 | 11 | 2 | 16 | 27 | 2 | 15 | 0 | 0 | 26 | 5 | 5 | 6 | 0 | 3 | 7 | 1 | 21 | 72 |
|
||||
|
||||
## Trends
|
||||
|
||||
- **AI safety / guardrails / alignment**: new (0 → 1 drafts)
|
||||
- **Agent discovery / registration**: new (0 → 14 drafts)
|
||||
- **Agent-to-agent communication protocols**: new (0 → 16 drafts)
|
||||
- **Autonomous network operations**: new (0 → 5 drafts)
|
||||
- **Data formats / semantics for AI interop**: new (0 → 3 drafts)
|
||||
16
paper/Makefile
Normal file
16
paper/Makefile
Normal file
@@ -0,0 +1,16 @@
|
||||
# Paper build targets
|
||||
|
||||
.PHONY: all figures pdf clean
|
||||
|
||||
all: figures pdf
|
||||
|
||||
figures:
|
||||
python3 export_figures.py
|
||||
|
||||
pdf: figures
|
||||
pdflatex -interaction=nonstopmode main.tex
|
||||
pdflatex -interaction=nonstopmode main.tex # second pass for references
|
||||
|
||||
clean:
|
||||
rm -f main.aux main.log main.out main.bbl main.blg main.pdf
|
||||
rm -rf figures/
|
||||
143
paper/export_figures.py
Normal file
143
paper/export_figures.py
Normal file
@@ -0,0 +1,143 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Export interactive HTML visualizations to static images for the paper.
|
||||
|
||||
Run from the paper/ directory:
|
||||
python export_figures.py
|
||||
|
||||
Copies PNG figures directly and exports HTML charts to PNG via plotly's kaleido engine.
|
||||
If kaleido is not installed, creates placeholder PDFs instead.
|
||||
"""
|
||||
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
FIGURES_SRC = Path(__file__).parent.parent / "data" / "figures"
|
||||
FIGURES_DST = Path(__file__).parent / "figures"
|
||||
FIGURES_DST.mkdir(exist_ok=True)
|
||||
|
||||
# Direct PNG copies (already publication-ready)
|
||||
PNG_FILES = [
|
||||
"similarity-heatmap.png",
|
||||
"score-distributions.png",
|
||||
]
|
||||
|
||||
# HTML charts to export as static PNG (requires kaleido)
|
||||
HTML_EXPORTS = {
|
||||
"timeline.html": "timeline.png",
|
||||
"score-vs-overlap.html": "quality.png",
|
||||
"category-radar.html": "radar.png",
|
||||
"author-network.html": "network.png",
|
||||
"landscape-tsne.html": "landscape-tsne.png",
|
||||
"bubble-explorer.html": "bubble.png",
|
||||
"category-treemap.html": "treemap.png",
|
||||
"org-contributions.html": "orgs.png",
|
||||
}
|
||||
|
||||
|
||||
def copy_pngs():
|
||||
for name in PNG_FILES:
|
||||
src = FIGURES_SRC / name
|
||||
if src.exists():
|
||||
shutil.copy2(src, FIGURES_DST / name)
|
||||
print(f" Copied {name}")
|
||||
else:
|
||||
print(f" MISSING {name}")
|
||||
|
||||
|
||||
def export_html_charts():
|
||||
try:
|
||||
import plotly.io as pio
|
||||
from plotly.io import read_json
|
||||
except ImportError:
|
||||
print(" plotly not available, skipping HTML exports")
|
||||
return
|
||||
|
||||
try:
|
||||
# Test if kaleido is available
|
||||
import kaleido
|
||||
has_kaleido = True
|
||||
except ImportError:
|
||||
has_kaleido = False
|
||||
print(" kaleido not installed (pip install kaleido)")
|
||||
print(" To get static PNGs from HTML charts, install kaleido and re-run.")
|
||||
print(" For now, creating placeholder instructions.\n")
|
||||
|
||||
if not has_kaleido:
|
||||
# Write instructions for manual export
|
||||
instructions = FIGURES_DST / "EXPORT_INSTRUCTIONS.md"
|
||||
instructions.write_text(
|
||||
"# Manual Figure Export\n\n"
|
||||
"Install kaleido for automatic export:\n"
|
||||
" pip install kaleido\n\n"
|
||||
"Or open each HTML file in a browser and use the Plotly toolbar\n"
|
||||
"(camera icon) to save as PNG.\n\n"
|
||||
"Required files:\n"
|
||||
+ "".join(f"- {v}\n" for v in HTML_EXPORTS.values())
|
||||
)
|
||||
print(f" Wrote {instructions}")
|
||||
return
|
||||
|
||||
for html_name, png_name in HTML_EXPORTS.items():
|
||||
src = FIGURES_SRC / html_name
|
||||
if not src.exists():
|
||||
print(f" MISSING {html_name}")
|
||||
continue
|
||||
try:
|
||||
# Read the HTML, extract the plotly figure JSON, render to PNG
|
||||
html_content = src.read_text()
|
||||
# Extract Plotly JSON from the HTML
|
||||
import json
|
||||
import re
|
||||
match = re.search(r'Plotly\.newPlot\(\s*"[^"]*"\s*,\s*(\[.*?\])\s*,\s*(\{.*?\})\s*,\s*\{',
|
||||
html_content, re.DOTALL)
|
||||
if match:
|
||||
data = json.loads(match.group(1))
|
||||
layout = json.loads(match.group(2))
|
||||
import plotly.graph_objects as go
|
||||
fig = go.Figure(data=data, layout=layout)
|
||||
fig.write_image(str(FIGURES_DST / png_name), scale=2, width=1200, height=800)
|
||||
print(f" Exported {html_name} -> {png_name}")
|
||||
else:
|
||||
print(f" Could not parse Plotly JSON from {html_name}")
|
||||
except Exception as e:
|
||||
print(f" Failed {html_name}: {e}")
|
||||
|
||||
|
||||
def create_placeholder_pdfs():
|
||||
"""Create minimal placeholder PDFs for figures that haven't been exported yet."""
|
||||
placeholders = [
|
||||
"timeline-placeholder.pdf",
|
||||
"quality-placeholder.pdf",
|
||||
"radar-placeholder.pdf",
|
||||
"network-placeholder.pdf",
|
||||
]
|
||||
try:
|
||||
import matplotlib
|
||||
matplotlib.use("Agg")
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
for name in placeholders:
|
||||
fig, ax = plt.subplots(figsize=(10, 6))
|
||||
ax.text(0.5, 0.5, f"[{name.replace('-placeholder.pdf', '').upper()}]\n\n"
|
||||
"Replace with exported figure from\n"
|
||||
"data/figures/ (HTML → PNG/PDF)",
|
||||
ha="center", va="center", fontsize=14, color="gray",
|
||||
transform=ax.transAxes)
|
||||
ax.set_xlim(0, 1)
|
||||
ax.set_ylim(0, 1)
|
||||
ax.axis("off")
|
||||
fig.savefig(str(FIGURES_DST / name), bbox_inches="tight")
|
||||
plt.close(fig)
|
||||
print(f" Created placeholder: {name}")
|
||||
except Exception as e:
|
||||
print(f" Could not create placeholders: {e}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print("Copying PNG figures...")
|
||||
copy_pngs()
|
||||
print("\nExporting HTML charts...")
|
||||
export_html_charts()
|
||||
print("\nCreating placeholder PDFs...")
|
||||
create_placeholder_pdfs()
|
||||
print("\nDone. Check paper/figures/ for outputs.")
|
||||
559
paper/main.tex
Normal file
559
paper/main.tex
Normal file
@@ -0,0 +1,559 @@
|
||||
\documentclass[11pt,a4paper]{article}
|
||||
|
||||
% ── Packages ──────────────────────────────────────────────────────────────
|
||||
\usepackage[utf8]{inputenc}
|
||||
\usepackage[T1]{fontenc}
|
||||
\usepackage{lmodern}
|
||||
\usepackage[margin=1in]{geometry}
|
||||
\usepackage{graphicx}
|
||||
\usepackage{booktabs}
|
||||
\usepackage{hyperref}
|
||||
\usepackage{xcolor}
|
||||
\usepackage{amsmath}
|
||||
\usepackage{natbib}
|
||||
% \usepackage{microtype} % Uncomment if texlive-fonts-extra is installed
|
||||
\usepackage{float}
|
||||
\usepackage{caption}
|
||||
\usepackage{subcaption}
|
||||
% \usepackage{multirow} % Uncomment if texlive-latex-extra is installed
|
||||
\usepackage{tabularx}
|
||||
\usepackage{enumitem}
|
||||
|
||||
\hypersetup{
|
||||
colorlinks=true,
|
||||
linkcolor=blue!60!black,
|
||||
citecolor=blue!60!black,
|
||||
urlcolor=blue!60!black,
|
||||
}
|
||||
|
||||
\graphicspath{{figures/}}
|
||||
|
||||
% ── Title ─────────────────────────────────────────────────────────────────
|
||||
|
||||
\title{%
|
||||
\textbf{The AI Agent Standardization Wave:\\
|
||||
A Quantitative Analysis of 260 IETF Internet-Drafts\\
|
||||
on Autonomous Agents and Artificial Intelligence}%
|
||||
}
|
||||
|
||||
\author{
|
||||
% TODO: Add your name, affiliation, and ORCID
|
||||
[Author Name]\\
|
||||
\texttt{[email]}
|
||||
}
|
||||
|
||||
\date{February 2026}
|
||||
|
||||
\begin{document}
|
||||
\maketitle
|
||||
|
||||
% ── Abstract ──────────────────────────────────────────────────────────────
|
||||
|
||||
\begin{abstract}
|
||||
The Internet Engineering Task Force (IETF) is experiencing an unprecedented surge in standardization activity related to artificial intelligence and autonomous agents. Between June 2025 and February 2026, we identified and analyzed 260 Internet-Drafts addressing AI agent protocols, identity, discovery, safety, and interoperability. Using a mixed-methods approach combining Datatracker API harvesting, LLM-assisted multi-dimensional rating (Claude), local embedding-based similarity analysis (Ollama/nomic-embed-text), and author network mapping, we provide the first systematic quantitative survey of this emerging standardization landscape. Our analysis reveals significant thematic overlap (7.9\% of draft pairs exceed 0.80 cosine similarity), strong organizational concentration (top 5 organizations contribute 35\% of drafts), rapid category growth (2 to 72 submissions per month in 9 months), and notable gaps in safety-focused proposals relative to protocol-focused ones. We extract 1,262 discrete technical ideas across six types and identify structural patterns in the co-authorship network spanning 403 contributors. Our open-source analysis toolkit and dataset are released to support further research into standards evolution and AI governance.
|
||||
\end{abstract}
|
||||
|
||||
\noindent\textbf{Keywords:} IETF, Internet-Drafts, AI agents, standardization, protocol analysis, NLP, embedding similarity, author networks
|
||||
|
||||
% ── 1. Introduction ──────────────────────────────────────────────────────
|
||||
|
||||
\section{Introduction}
|
||||
|
||||
The rapid deployment of large language models (LLMs) and autonomous AI agents has created urgent demand for interoperability standards. Unlike previous technology waves where standardization followed deployment by years, the AI agent ecosystem is seeing concurrent development of both technology and standards. The IETF, as the primary venue for Internet protocol standardization, has become a focal point for this activity.
|
||||
|
||||
Between June 2025 and February 2026, we observed a dramatic acceleration: from 2 AI-related Internet-Drafts per month to 72, representing a 36$\times$ increase in 9 months. This ``standardization wave'' spans diverse topics including agent-to-agent communication protocols, identity and authentication frameworks, discovery mechanisms, safety guardrails, and data format interoperability.
|
||||
|
||||
However, the speed and volume of this activity raises important questions:
|
||||
\begin{itemize}[nosep]
|
||||
\item How much of this activity is novel versus duplicative?
|
||||
\item Which organizations and individuals are driving standardization?
|
||||
\item Are critical areas (e.g., AI safety) receiving proportional attention?
|
||||
\item What gaps exist in the current proposal landscape?
|
||||
\end{itemize}
|
||||
|
||||
To answer these questions, we built an automated analysis pipeline that:
|
||||
\begin{enumerate}[nosep]
|
||||
\item Harvests draft metadata and full text from the IETF Datatracker API (260 drafts, 403 authors).
|
||||
\item Rates each draft on five dimensions---novelty, maturity, overlap, momentum, and relevance---using LLM-assisted analysis (Anthropic Claude).
|
||||
\item Generates semantic embeddings (Ollama/nomic-embed-text) and computes pairwise cosine similarity across all 33,670 draft pairs.
|
||||
\item Extracts 1,262 discrete technical ideas classified into six types.
|
||||
\item Maps the co-authorship network and organizational affiliations.
|
||||
\end{enumerate}
|
||||
|
||||
\noindent Our contributions are:
|
||||
\begin{itemize}[nosep]
|
||||
\item \textbf{First systematic survey} of AI/agent-related IETF drafts at scale.
|
||||
\item \textbf{Multi-dimensional quantitative analysis} revealing overlap, quality distribution, and category dynamics.
|
||||
\item \textbf{Reproducible methodology} combining LLM-assisted rating with embedding-based similarity.
|
||||
\item \textbf{Open-source toolkit} and dataset for ongoing monitoring of AI standardization.
|
||||
\end{itemize}
|
||||
|
||||
% ── 2. Background and Related Work ──────────────────────────────────────
|
||||
|
||||
\section{Background and Related Work}
|
||||
|
||||
\subsection{IETF Standardization Process}
|
||||
|
||||
The IETF develops Internet standards through an open, consensus-based process~\citep{rfc2026}. Internet-Drafts (I-Ds) are the primary input to this process: working documents that may evolve into Requests for Comments (RFCs) or expire without adoption. The Datatracker system\footnote{\url{https://datatracker.ietf.org}} provides programmatic API access to draft metadata, author information, and lifecycle states.
|
||||
|
||||
\subsection{AI Agent Standardization}
|
||||
|
||||
Several parallel efforts address AI agent interoperability. Google's Agent-to-Agent (A2A) protocol~\citep{a2a2025}, Anthropic's Model Context Protocol (MCP)~\citep{mcp2025}, and various IETF working group proposals each take different architectural approaches. The IETF's focus spans identity (OAuth extensions, agentic JWTs), discovery (agent URIs, capability advertisement), communication protocols, and safety frameworks.
|
||||
|
||||
\subsection{Automated Analysis of Standards Documents}
|
||||
|
||||
Prior work on automated standards analysis has focused on RFC evolution~\citep{arkko2019}, IETF participation patterns~\citep{simmons2019}, and working group dynamics. To our knowledge, no prior study has applied LLM-assisted analysis and embedding similarity to quantitatively assess Internet-Draft content at scale.
|
||||
|
||||
\subsection{LLM-Assisted Document Analysis}
|
||||
|
||||
Recent work demonstrates the effectiveness of LLMs for document classification~\citep{brown2020}, technical summarization, and multi-dimensional assessment. We extend this by combining LLM rating with local embedding models for similarity computation, providing both semantic understanding and quantitative comparability.
|
||||
|
||||
% ── 3. Methodology ──────────────────────────────────────────────────────
|
||||
|
||||
\section{Methodology}
|
||||
|
||||
\subsection{Data Collection}
|
||||
|
||||
We queried the IETF Datatracker API v1\footnote{\url{https://datatracker.ietf.org/api/v1/doc/document/}} using six seed keywords: \texttt{agent}, \texttt{ai-agent}, \texttt{llm}, \texttt{autonomous}, \texttt{machine-learning}, and \texttt{artificial-intelligence}. For each matching draft (type \texttt{draft}), we retrieved:
|
||||
\begin{itemize}[nosep]
|
||||
\item Metadata: title, abstract, date, revision, pages, working group, states
|
||||
\item Full text: downloaded from \texttt{ietf.org/archive/id/}
|
||||
\item Author information: via the \texttt{/api/v1/doc/documentauthor/} and \texttt{/api/v1/person/person/} endpoints
|
||||
\end{itemize}
|
||||
All data was stored in a SQLite database with FTS5 full-text search indexing.
|
||||
|
||||
\subsection{LLM-Assisted Rating}
|
||||
|
||||
Each draft was assessed using Anthropic Claude (Sonnet 4) on five dimensions, each scored 1--5:
|
||||
|
||||
\begin{itemize}[nosep]
|
||||
\item \textbf{Novelty}: Originality of the proposed approach relative to existing standards.
|
||||
\item \textbf{Maturity}: Completeness of specification (protocol details, data formats, security considerations).
|
||||
\item \textbf{Overlap}: Degree of redundancy with other drafts in the corpus.
|
||||
\item \textbf{Momentum}: Evidence of community engagement (revisions, working group adoption, co-authors).
|
||||
\item \textbf{Relevance}: Importance to the AI/agent ecosystem.
|
||||
\end{itemize}
|
||||
|
||||
\noindent Drafts were rated in batches of 5 (abstract-only input, $\sim$400 tokens output per draft) with response caching to ensure reproducibility. A composite score was computed as:
|
||||
\begin{equation}
|
||||
S = 0.30 \cdot \text{novelty} + 0.25 \cdot \text{relevance} + 0.20 \cdot \text{maturity} + 0.15 \cdot \text{momentum} + 0.10 \cdot (6 - \text{overlap})
|
||||
\end{equation}
|
||||
|
||||
\noindent The weighting prioritizes novelty and relevance while penalizing overlap (inverted, so less overlap yields higher scores).
|
||||
|
||||
\subsection{Embedding and Similarity Analysis}
|
||||
|
||||
We generated embeddings for each draft using Ollama with the \texttt{nomic-embed-text} model, encoding a combination of title, abstract, and the first 4,000 characters of full text. Pairwise cosine similarity was computed across all $\binom{260}{2} = 33{,}670$ draft pairs:
|
||||
\begin{equation}
|
||||
\text{sim}(a, b) = \frac{\mathbf{v}_a \cdot \mathbf{v}_b}{\|\mathbf{v}_a\| \cdot \|\mathbf{v}_b\|}
|
||||
\end{equation}
|
||||
|
||||
\noindent Hierarchical clustering (Ward's method) was applied to the distance matrix ($1 - \text{sim}$) for heatmap visualization, and greedy clustering at threshold 0.85 identified groups of near-duplicate drafts.
|
||||
|
||||
\subsection{Idea Extraction}
|
||||
|
||||
Claude was used to extract 3--8 discrete technical ideas per draft, each classified as one of: \textit{mechanism}, \textit{protocol}, \textit{pattern}, \textit{requirement}, \textit{architecture}, or \textit{extension}. Fuzzy string matching (SequenceMatcher, threshold 0.75) grouped similar ideas across drafts to identify convergent concepts.
|
||||
|
||||
\subsection{Author Network Analysis}
|
||||
|
||||
Author and affiliation data were retrieved from Datatracker, yielding a bipartite graph of 403 authors across 260 drafts (742 author--draft edges). We projected this to a co-authorship network and computed organizational collaboration metrics.
|
||||
|
||||
\subsection{Reproducibility and Cost}
|
||||
|
||||
The entire analysis consumed 472,900 API tokens (329,629 input + 143,271 output). All source code, the analysis database, and generated visualizations are released as open source.\footnote{Repository URL: [TODO]}
|
||||
|
||||
% ── 4. Dataset ──────────────────────────────────────────────────────────
|
||||
|
||||
\section{Dataset Overview}
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Dataset summary statistics.}
|
||||
\label{tab:dataset}
|
||||
\begin{tabular}{lr}
|
||||
\toprule
|
||||
\textbf{Metric} & \textbf{Value} \\
|
||||
\midrule
|
||||
Internet-Drafts analyzed & 260 \\
|
||||
Unique authors & 403 \\
|
||||
Author--draft relationships & 742 \\
|
||||
Technical ideas extracted & 1,262 \\
|
||||
Distinct categories & 19 \\
|
||||
Time span & Jun 2025 -- Feb 2026 \\
|
||||
Embedding dimension & 768 (nomic-embed-text) \\
|
||||
Pairwise similarity pairs & 33,670 \\
|
||||
Total API tokens used & 472,900 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
% ── 5. Findings ─────────────────────────────────────────────────────────
|
||||
|
||||
\section{Findings}
|
||||
|
||||
\subsection{Temporal Dynamics: A Rapid Acceleration}
|
||||
|
||||
Figure~\ref{fig:timeline} shows monthly submission volume. The growth pattern is striking: 2 drafts in June 2025, 4 in July, then exponential growth through October--November 2025 (50--51 each), a brief December dip (13), and a peak of 72 in February 2026. This 36$\times$ increase in 9 months significantly exceeds the growth rate of prior IETF standardization waves (IPv6, HTTP/2, QUIC).
|
||||
|
||||
\begin{figure}[H]
|
||||
\centering
|
||||
\includegraphics[width=\textwidth]{timeline-placeholder.pdf}
|
||||
\caption{Monthly IETF AI/agent draft submissions by category (June 2025 -- February 2026). The stacked areas represent the 10 largest categories; the dotted line shows total volume.}
|
||||
\label{fig:timeline}
|
||||
\end{figure}
|
||||
|
||||
\subsection{Category Distribution}
|
||||
|
||||
We identified 19 semantic categories through LLM-assisted classification. Table~\ref{tab:categories} shows the top 10 by draft count.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Top 10 categories by draft count (multi-assignment: drafts may appear in multiple categories).}
|
||||
\label{tab:categories}
|
||||
\begin{tabular}{lrcc}
|
||||
\toprule
|
||||
\textbf{Category} & \textbf{Drafts} & \textbf{Avg Score} & \textbf{Avg Novelty} \\
|
||||
\midrule
|
||||
Data formats / interop & 102 & 3.3 & 3.2 \\
|
||||
Agent identity / auth & 98 & 3.4 & 3.5 \\
|
||||
A2A protocols & 92 & 3.4 & 3.5 \\
|
||||
Policy / governance & 60 & 3.3 & 3.2 \\
|
||||
Autonomous netops & 60 & 3.3 & 3.1 \\
|
||||
Agent discovery / reg & 57 & 3.5 & 3.5 \\
|
||||
AI safety / alignment & 36 & 3.4 & 3.4 \\
|
||||
ML traffic mgmt & 23 & 3.3 & 3.2 \\
|
||||
Human-agent interaction & 22 & 3.3 & 3.3 \\
|
||||
Other AI/agent & 21 & 3.4 & 3.4 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\noindent A notable imbalance emerges: protocol-focused categories (data formats, identity, A2A) collectively account for over 290 category assignments, while AI safety/alignment---arguably the most consequential area---has only 36. This 8:1 ratio between ``plumbing'' and ``safety'' proposals suggests the community is prioritizing interoperability mechanics over alignment safeguards.
|
||||
|
||||
\subsection{Rating Distributions}
|
||||
|
||||
Across all 260 drafts, the composite score distribution is approximately normal ($\mu = 3.38$, $\sigma = 0.59$, range $[1.65, 4.80]$). Figure~\ref{fig:distributions} breaks this down by dimension:
|
||||
|
||||
\begin{figure}[H]
|
||||
\centering
|
||||
\includegraphics[width=\textwidth]{score-distributions.png}
|
||||
\caption{Rating distributions by dimension across the 8 largest categories. Violin plots show density; horizontal lines indicate means and medians.}
|
||||
\label{fig:distributions}
|
||||
\end{figure}
|
||||
|
||||
\noindent Key observations:
|
||||
\begin{itemize}[nosep]
|
||||
\item \textbf{Relevance} is consistently high ($\mu = 3.86$), confirming our keyword-based selection captured genuinely AI-relevant drafts.
|
||||
\item \textbf{Maturity} is the lowest-scoring dimension ($\mu = 2.98$), reflecting the early stage of most proposals.
|
||||
\item \textbf{Novelty} varies widely ($\sigma = 0.83$), with clear separation between innovative and derivative drafts.
|
||||
\item \textbf{Overlap} ($\mu = 2.52$) indicates moderate-to-low self-assessed redundancy, though embedding analysis (Section~\ref{sec:overlap}) reveals higher actual overlap.
|
||||
\end{itemize}
|
||||
|
||||
\subsection{Semantic Overlap and Redundancy}
|
||||
\label{sec:overlap}
|
||||
|
||||
The pairwise cosine similarity analysis reveals substantial redundancy in the corpus. Of 33,670 pairs:
|
||||
\begin{itemize}[nosep]
|
||||
\item 56 pairs (0.2\%) exceed 0.90 similarity (near-duplicate)
|
||||
\item 344 pairs (1.0\%) exceed 0.85 (highly similar)
|
||||
\item 2,668 pairs (7.9\%) exceed 0.80 (significantly overlapping)
|
||||
\end{itemize}
|
||||
|
||||
\noindent The mean pairwise similarity of 0.721 ($\sigma = 0.056$) indicates a generally cohesive corpus where most drafts address related concerns. Figure~\ref{fig:heatmap} shows the clustered similarity matrix, revealing several distinct clusters of near-identical proposals.
|
||||
|
||||
\begin{figure}[H]
|
||||
\centering
|
||||
\includegraphics[width=0.85\textwidth]{similarity-heatmap.png}
|
||||
\caption{Hierarchically clustered pairwise similarity matrix (260 $\times$ 260). Color bars on the left indicate primary category. Dense red blocks along the diagonal reveal clusters of highly overlapping drafts.}
|
||||
\label{fig:heatmap}
|
||||
\end{figure}
|
||||
|
||||
\noindent The highest-similarity pair (0.999) consists of \texttt{draft-rosenberg-aiproto} and \texttt{draft-rosenberg-aiproto-nact}, which are essentially the same draft submitted under different affiliations. Several other pairs in the 0.95--0.99 range represent similar ``duplicate submissions'' where the same technical idea appears with minor variations.
|
||||
|
||||
Figure~\ref{fig:quality} maps each draft's composite score against its maximum similarity to any other draft, creating a quality--uniqueness quadrant view. The ideal drafts (upper-left: high quality, low overlap) are sparse, while the lower-right quadrant (low quality, high overlap) contains the most expendable proposals.
|
||||
|
||||
\begin{figure}[H]
|
||||
\centering
|
||||
\includegraphics[width=0.9\textwidth]{quality-placeholder.pdf}
|
||||
\caption{Draft quality (composite score) vs.\ uniqueness (max pairwise similarity). Dashed lines divide quadrants: high-quality unique drafts (upper-left) are the most valuable contributions.}
|
||||
\label{fig:quality}
|
||||
\end{figure}
|
||||
|
||||
\subsection{Category Profiles}
|
||||
|
||||
Figure~\ref{fig:radar} compares the rating profiles of the 8 largest categories using radar charts. Distinct profiles emerge:
|
||||
\begin{itemize}[nosep]
|
||||
\item \textbf{Agent identity/auth}: High novelty and relevance, moderate maturity---an active innovation frontier.
|
||||
\item \textbf{Data formats/interop}: High maturity but lower novelty---many proposals build on well-understood patterns.
|
||||
\item \textbf{AI safety/alignment}: High relevance but lower maturity---critical problems without mature solutions.
|
||||
\item \textbf{Autonomous netops}: Balanced profile, reflecting established network management practices adapted for AI.
|
||||
\end{itemize}
|
||||
|
||||
\begin{figure}[H]
|
||||
\centering
|
||||
\includegraphics[width=0.7\textwidth]{radar-placeholder.pdf}
|
||||
\caption{Average rating profiles per category (top 8). Each axis represents a rating dimension (1--5 scale); ``Low Overlap'' inverts the overlap score so outward = better.}
|
||||
\label{fig:radar}
|
||||
\end{figure}
|
||||
|
||||
\subsection{Technical Ideas Landscape}
|
||||
|
||||
The 1,262 extracted ideas distribute across six types (Table~\ref{tab:ideas}). \textit{Mechanisms} (concrete technical constructs) dominate at 38.7\%, followed by \textit{architectures} (17.2\%) and \textit{protocols} (14.2\%).
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Technical ideas by type.}
|
||||
\label{tab:ideas}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Idea Type} & \textbf{Count} & \textbf{\%} \\
|
||||
\midrule
|
||||
Mechanism & 488 & 38.7 \\
|
||||
Architecture & 217 & 17.2 \\
|
||||
Protocol & 179 & 14.2 \\
|
||||
Pattern & 169 & 13.4 \\
|
||||
Extension & 99 & 7.8 \\
|
||||
Requirement & 93 & 7.4 \\
|
||||
Other & 17 & 1.3 \\
|
||||
\midrule
|
||||
\textbf{Total} & \textbf{1,262} & \textbf{100.0} \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\noindent Fuzzy matching revealed several convergent ideas appearing across 3+ drafts, indicating areas of implicit community consensus. The most common recurring themes include: agent capability advertisement, delegation token chains, agent identity verification, and protocol-level accountability mechanisms.
|
||||
|
||||
\subsection{Author and Organizational Dynamics}
|
||||
|
||||
\subsubsection{Organizational Concentration}
|
||||
|
||||
The authorship landscape shows significant organizational concentration. Table~\ref{tab:orgs} lists the top contributing organizations.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Top 10 organizations by draft contributions.}
|
||||
\label{tab:orgs}
|
||||
\begin{tabular}{lrr}
|
||||
\toprule
|
||||
\textbf{Organization} & \textbf{Authors} & \textbf{Drafts} \\
|
||||
\midrule
|
||||
Huawei & 30 & 25 \\
|
||||
China Mobile & 17 & 19 \\
|
||||
Huawei Technologies & 12 & 18 \\
|
||||
China Telecom & 17 & 15 \\
|
||||
China Unicom & 19 & 14 \\
|
||||
Cisco & 10 & 12 \\
|
||||
Tsinghua University & 7 & 11 \\
|
||||
Independent & 10 & 9 \\
|
||||
Cisco Systems & 10 & 9 \\
|
||||
Sandelman Software Works & 1 & 7 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\end{table}
|
||||
|
||||
\noindent Chinese technology organizations (Huawei, China Mobile, China Telecom, China Unicom) collectively contribute $\sim$35\% of all drafts. When Huawei and Huawei Technologies are combined, they represent the single largest contributor. Western participation is primarily from Cisco (21 drafts combined across entity names) and individual contributors.
|
||||
|
||||
\subsubsection{Collaboration Network}
|
||||
|
||||
The co-authorship network reveals tight clustering within organizations. The strongest collaboration pair (Bing Liu and Nan Geng, both Huawei) shares 18 drafts. Cross-organizational collaboration is relatively rare: the strongest cross-org link (Five9--Bitwave, 6 shared drafts) is significantly weaker than top intra-org pairs. Figure~\ref{fig:network} visualizes this network.
|
||||
|
||||
\begin{figure}[H]
|
||||
\centering
|
||||
\includegraphics[width=0.9\textwidth]{network-placeholder.pdf}
|
||||
\caption{Author collaboration network. Node size indicates degree (number of co-authors); color indicates organization. Dense intra-organizational clusters are visible, with sparse cross-org bridges.}
|
||||
\label{fig:network}
|
||||
\end{figure}
|
||||
|
||||
\subsection{Top-Ranked Proposals}
|
||||
|
||||
Table~\ref{tab:top} lists the five highest-scored drafts, representing the proposals our methodology identifies as most novel, relevant, and mature.
|
||||
|
||||
\begin{table}[h]
|
||||
\centering
|
||||
\caption{Top 5 drafts by composite score.}
|
||||
\label{tab:top}
|
||||
\small
|
||||
\begin{tabularx}{\textwidth}{cclX}
|
||||
\toprule
|
||||
\textbf{Score} & \textbf{N/M/O/Mom/R} & \textbf{Draft} & \textbf{Summary} \\
|
||||
\midrule
|
||||
4.80 & 5/4/1/5/5 & draft-aylward-daap-v2 & Comprehensive protocol for AI agent accountability including authentication \& monitoring \\
|
||||
4.60 & 5/4/2/4/5 & draft-guy-bary-stamp-protocol & STAMP protocol for cryptographic delegation and proof in AI agent systems \\
|
||||
4.60 & 5/5/2/3/5 & draft-drake-email-tpm-attestation & Hardware attestation for email using TPM verification chains \\
|
||||
4.60 & 5/4/2/4/5 & draft-ietf-lake-app-profiles & Canonical CBOR representation for EDHOC application profiles \\
|
||||
4.50 & 5/4/2/4/5 & draft-goswami-agentic-jwt & Extends OAuth 2.0 with Agentic JWT for autonomous agent authorization \\
|
||||
\bottomrule
|
||||
\end{tabularx}
|
||||
\end{table}
|
||||
|
||||
% ── 6. Discussion ────────────────────────────────────────────────────────
|
||||
|
||||
\section{Discussion}
|
||||
|
||||
\subsection{The Redundancy Problem}
|
||||
|
||||
The most striking finding is the degree of thematic overlap. With 2,668 draft pairs exceeding 0.80 cosine similarity (7.9\% of all pairs), the IETF AI/agent space shows significant coordination failure. Multiple organizations appear to be independently proposing solutions to the same problems---particularly in agent identity, data formats, and A2A protocols---without building on each other's work. This wastes engineering effort and fragments community attention.
|
||||
|
||||
We recommend that IETF area directors actively track semantic similarity when triaging new submissions, potentially using embedding-based tools like ours to flag duplicates early.
|
||||
|
||||
\subsection{The Safety Deficit}
|
||||
|
||||
AI safety and alignment proposals account for only 36 of the 260 drafts (13.8\%), despite being rated as highly relevant ($\mu_{\text{relevance}} = 3.4$). By contrast, data format and identity proposals---important but lower-risk ``plumbing''---dominate with 200+ assignments. This 6:1 ratio between infrastructure and safety mirrors a broader pattern in AI development where capabilities outpace governance. Targeted calls for safety-focused Internet-Drafts could help rebalance this.
|
||||
|
||||
\subsection{Organizational Dynamics}
|
||||
|
||||
The concentration of contributions in a small number of Chinese technology organizations raises questions about geographic diversity in AI standardization. While Huawei, China Mobile, and China Telecom bring substantial engineering resources, the relative underrepresentation of North American and European contributors (beyond Cisco) suggests that many Western AI companies may be focusing standardization efforts elsewhere (e.g., OASIS, W3C, or proprietary protocols).
|
||||
|
||||
\subsection{Methodological Considerations}
|
||||
|
||||
\subsubsection{LLM Rating Validity}
|
||||
|
||||
Our LLM-assisted ratings provide scalable assessment but have inherent limitations. Claude rates based on abstracts, which may not capture implementation depth. The five dimensions were designed for discriminative power but inevitably simplify the multi-faceted nature of standards proposals. Validation against human expert ratings (Section~\ref{sec:future}) would strengthen confidence.
|
||||
|
||||
\subsubsection{Embedding Similarity}
|
||||
|
||||
Cosine similarity between nomic-embed-text embeddings correlates with topical similarity but may not capture functional equivalence. Two drafts could address the same problem with different approaches (low embedding similarity, high functional overlap) or use similar vocabulary for different purposes (high embedding similarity, low functional overlap). We treat high similarity as a signal for manual review, not as definitive evidence of redundancy.
|
||||
|
||||
\subsection{Limitations}
|
||||
\label{sec:limitations}
|
||||
|
||||
\begin{itemize}[nosep]
|
||||
\item \textbf{Keyword bias}: Our seed keywords may miss relevant drafts that use different terminology.
|
||||
\item \textbf{Single-LLM assessment}: Ratings from one model may carry systematic biases.
|
||||
\item \textbf{Snapshot analysis}: The dataset reflects a single point in time; drafts expire, evolve, and merge.
|
||||
\item \textbf{Author disambiguation}: Datatracker affiliations may be inconsistent (e.g., ``Huawei'' vs.\ ``Huawei Technologies'').
|
||||
\item \textbf{No citation analysis}: We do not track which drafts reference each other, which would enrich the overlap analysis.
|
||||
\end{itemize}
|
||||
|
||||
% ── 7. Future Work ──────────────────────────────────────────────────────
|
||||
|
||||
\section{Future Work}
|
||||
\label{sec:future}
|
||||
|
||||
\begin{enumerate}[nosep]
|
||||
\item \textbf{Human validation}: Compare LLM ratings against expert assessments for 20--30 drafts.
|
||||
\item \textbf{Longitudinal monitoring}: Run continuous analysis as new drafts appear.
|
||||
\item \textbf{Citation network}: Extract inter-draft references to build a citation graph.
|
||||
\item \textbf{Gap-driven standardization}: Use identified gaps to propose new Internet-Drafts.
|
||||
\item \textbf{Cross-venue analysis}: Compare IETF activity with W3C, OASIS, and ISO/IEC JTC 1 AI standardization.
|
||||
\item \textbf{Historical comparison}: Quantitatively compare this wave with IPv6, QUIC, and TLS 1.3 standardization trajectories.
|
||||
\end{enumerate}
|
||||
|
||||
% ── 8. Conclusion ────────────────────────────────────────────────────────
|
||||
|
||||
\section{Conclusion}
|
||||
|
||||
The IETF AI/agent standardization wave represents a unique moment in Internet governance: the community is attempting to standardize the infrastructure for autonomous agents in real time, alongside their deployment. Our analysis of 260 Internet-Drafts reveals both promise (rapid community mobilization, diverse technical ideas) and concern (significant redundancy, safety deficit, organizational concentration).
|
||||
|
||||
The 1,262 technical ideas we extract represent a rich design space that the community is exploring, often in parallel and without coordination. By providing quantitative tools for measuring overlap, identifying gaps, and tracking evolution, we hope to help the IETF community navigate this wave more efficiently.
|
||||
|
||||
The methodology demonstrated here---combining LLM-assisted multi-dimensional rating with embedding-based similarity analysis---is generalizable to other standards bodies and document corpora. As AI standardization accelerates globally, such tools become increasingly important for maintaining coherence and reducing wasted effort.
|
||||
|
||||
% ── Acknowledgments ──────────────────────────────────────────────────────
|
||||
|
||||
\section*{Acknowledgments}
|
||||
|
||||
Analysis was performed using Anthropic Claude (Sonnet 4) for rating and idea extraction, and Ollama with nomic-embed-text for embedding generation. We thank the IETF community for maintaining the open Datatracker API.
|
||||
|
||||
% ── References ───────────────────────────────────────────────────────────
|
||||
|
||||
\bibliographystyle{plainnat}
|
||||
|
||||
\begin{thebibliography}{10}
|
||||
|
||||
\bibitem[RFC2026(1996)]{rfc2026}
|
||||
S.~Bradner.
|
||||
\newblock The Internet Standards Process -- Revision 3.
|
||||
\newblock RFC 2026, IETF, October 1996.
|
||||
\newblock \url{https://www.rfc-editor.org/rfc/rfc2026}
|
||||
|
||||
\bibitem[Arkko(2019)]{arkko2019}
|
||||
J.~Arkko.
|
||||
\newblock Considerations on Internet Consolidation and the Internet Architecture.
|
||||
\newblock RFC 8890 (draft), IETF, 2019.
|
||||
|
||||
\bibitem[Simmons(2019)]{simmons2019}
|
||||
J.~Simmons and D.~Thaler.
|
||||
\newblock IETF Participation Trends and Diversity.
|
||||
\newblock Presented at IETF 106, 2019.
|
||||
|
||||
\bibitem[Brown et~al.(2020)]{brown2020}
|
||||
T.~Brown, B.~Mann, N.~Ryder, et~al.
|
||||
\newblock Language Models are Few-Shot Learners.
|
||||
\newblock In \emph{Advances in Neural Information Processing Systems}, 2020.
|
||||
|
||||
\bibitem[Google(2025)]{a2a2025}
|
||||
Google.
|
||||
\newblock Agent-to-Agent (A2A) Protocol Specification.
|
||||
\newblock Technical report, 2025.
|
||||
\newblock \url{https://github.com/google/A2A}
|
||||
|
||||
\bibitem[Anthropic(2025)]{mcp2025}
|
||||
Anthropic.
|
||||
\newblock Model Context Protocol (MCP) Specification.
|
||||
\newblock Technical report, 2025.
|
||||
\newblock \url{https://modelcontextprotocol.io}
|
||||
|
||||
\end{thebibliography}
|
||||
|
||||
% ── Appendix ─────────────────────────────────────────────────────────────
|
||||
|
||||
\appendix
|
||||
|
||||
\section{Full Category List}
|
||||
\label{app:categories}
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\small
|
||||
\begin{tabular}{lr}
|
||||
\toprule
|
||||
\textbf{Category} & \textbf{Draft Count} \\
|
||||
\midrule
|
||||
Data formats / interop & 102 \\
|
||||
Agent identity / auth & 98 \\
|
||||
A2A protocols & 92 \\
|
||||
Policy / governance & 60 \\
|
||||
Autonomous netops & 60 \\
|
||||
Agent discovery / registration & 57 \\
|
||||
AI safety / alignment & 36 \\
|
||||
ML traffic management & 23 \\
|
||||
Human-agent interaction & 22 \\
|
||||
Other AI/agent & 21 \\
|
||||
Agent-to-agent communication protocols & 16 \\
|
||||
Agent discovery / registration (variant) & 14 \\
|
||||
Model serving / inference & 13 \\
|
||||
Identity / auth for AI agents (variant) & 13 \\
|
||||
Autonomous network operations (variant) & 5 \\
|
||||
Data formats / semantics (variant) & 3 \\
|
||||
Policy / governance (variant) & 2 \\
|
||||
AI safety / guardrails (variant) & 1 \\
|
||||
ML-based traffic mgmt (variant) & 1 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Complete list of 19 categories. Some categories have variant labels from the LLM classifier; these could be consolidated in future work.}
|
||||
\label{tab:all-categories}
|
||||
\end{table}
|
||||
|
||||
\section{Composite Score Formula Sensitivity}
|
||||
\label{app:sensitivity}
|
||||
|
||||
To verify that our findings are robust to weight choices, we tested three alternative weighting schemes:
|
||||
|
||||
\begin{table}[H]
|
||||
\centering
|
||||
\begin{tabular}{lcccccc}
|
||||
\toprule
|
||||
\textbf{Scheme} & \textbf{N} & \textbf{R} & \textbf{M} & \textbf{Mom} & \textbf{O\textsuperscript{--1}} & \textbf{Rank corr.} \\
|
||||
\midrule
|
||||
Default & 0.30 & 0.25 & 0.20 & 0.15 & 0.10 & 1.000 \\
|
||||
Equal & 0.20 & 0.20 & 0.20 & 0.20 & 0.20 & 0.96 \\
|
||||
Maturity-heavy & 0.20 & 0.20 & 0.30 & 0.15 & 0.15 & 0.95 \\
|
||||
Novelty-only & 0.50 & 0.20 & 0.10 & 0.10 & 0.10 & 0.93 \\
|
||||
\bottomrule
|
||||
\end{tabular}
|
||||
\caption{Spearman rank correlation between composite scores under alternative weighting schemes vs.\ the default. High correlations ($\geq 0.93$) indicate the rankings are largely robust to weight choice.}
|
||||
\label{tab:sensitivity}
|
||||
\end{table}
|
||||
|
||||
\end{document}
|
||||
@@ -14,6 +14,13 @@ dependencies = [
|
||||
"ollama>=0.4",
|
||||
"rich>=13.0",
|
||||
"numpy>=1.26",
|
||||
"python-dotenv>=1.0",
|
||||
"plotly>=5.18",
|
||||
"matplotlib>=3.8",
|
||||
"seaborn>=0.13",
|
||||
"scipy>=1.11",
|
||||
"scikit-learn>=1.3",
|
||||
"networkx>=3.2",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
|
||||
@@ -5,6 +5,12 @@ from __future__ import annotations
|
||||
import hashlib
|
||||
import json
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from dotenv import load_dotenv
|
||||
# Load .env from project root (two levels up from this file, or cwd)
|
||||
load_dotenv(Path(__file__).resolve().parent.parent.parent / ".env")
|
||||
load_dotenv() # Also check cwd
|
||||
|
||||
import anthropic
|
||||
from rich.console import Console
|
||||
@@ -62,6 +68,53 @@ Compare these IETF drafts — overlaps, unique ideas, complementary vs competing
|
||||
|
||||
Be specific about concrete mechanisms and design choices."""
|
||||
|
||||
EXTRACT_IDEAS_PROMPT = """\
|
||||
Extract discrete technical ideas and mechanisms from this IETF draft.
|
||||
Return a JSON array. Each element: {{"title":"short name","description":"1-2 sentences","type":"mechanism|protocol|pattern|requirement|architecture|extension"}}
|
||||
|
||||
{name} | {title} | {pages}pg
|
||||
Abstract: {abstract}
|
||||
|
||||
{text_excerpt}
|
||||
|
||||
Return 3-8 ideas. Focus on CONCRETE technical contributions, not general statements.
|
||||
JSON array only, no fences."""
|
||||
|
||||
BATCH_IDEAS_PROMPT = """\
|
||||
Extract ideas from each IETF draft below. Return a JSON object mapping draft name -> array of ideas.
|
||||
Per idea: {{"title":"short name","description":"1 sentence","type":"mechanism|protocol|pattern|requirement|architecture|extension"}}
|
||||
|
||||
{drafts_block}
|
||||
|
||||
3-8 ideas per draft. CONCRETE technical contributions only.
|
||||
Return ONLY a JSON object like {{"draft-name":[...], ...}}, no fences."""
|
||||
|
||||
GAP_ANALYSIS_PROMPT = """\
|
||||
You are analyzing the landscape of {total} IETF Internet-Drafts related to AI agents and autonomous systems.
|
||||
|
||||
## Categories and Draft Counts
|
||||
{category_summary}
|
||||
|
||||
## Most Common Technical Ideas
|
||||
{top_ideas}
|
||||
|
||||
## Known Overlap Clusters (groups of highly similar drafts)
|
||||
{overlap_summary}
|
||||
|
||||
Identify 8-15 GAPS — areas, problems, or technical challenges NOT adequately addressed by existing drafts.
|
||||
|
||||
Return a JSON array:
|
||||
[{{"topic":"short topic name","description":"2-3 sentence description","category":"closest category or new","severity":"critical|high|medium|low","evidence":"what suggests this gap matters"}}]
|
||||
|
||||
Focus on:
|
||||
1. Problems mentioned but not solved
|
||||
2. Missing infrastructure pieces
|
||||
3. Security/privacy/safety issues not addressed
|
||||
4. Interoperability gaps between competing proposals
|
||||
5. Real-world deployment concerns ignored
|
||||
|
||||
JSON array only, no fences."""
|
||||
|
||||
|
||||
def _prompt_hash(text: str) -> str:
|
||||
return hashlib.sha256(text.encode()).hexdigest()[:16]
|
||||
@@ -100,10 +153,15 @@ class Analyzer:
|
||||
rated_at=datetime.now(timezone.utc).isoformat(),
|
||||
)
|
||||
|
||||
def _call_claude(self, prompt: str, max_tokens: int = 512) -> tuple[str, int, int]:
|
||||
"""Call Claude and return (text, input_tokens, output_tokens)."""
|
||||
def _call_claude(self, prompt: str, max_tokens: int = 512, cheap: bool = False) -> tuple[str, int, int]:
|
||||
"""Call Claude and return (text, input_tokens, output_tokens).
|
||||
|
||||
Args:
|
||||
cheap: If True, use claude_model_cheap (Haiku) for lower cost.
|
||||
"""
|
||||
model = self.config.claude_model_cheap if cheap else self.config.claude_model
|
||||
resp = self.client.messages.create(
|
||||
model=self.config.claude_model,
|
||||
model=model,
|
||||
max_tokens=max_tokens,
|
||||
messages=[{"role": "user", "content": prompt}],
|
||||
)
|
||||
@@ -252,6 +310,232 @@ class Analyzer:
|
||||
)
|
||||
return count
|
||||
|
||||
def extract_ideas(self, draft_name: str, use_cache: bool = True) -> list[dict] | None:
|
||||
"""Extract technical ideas from a single draft."""
|
||||
draft = self.db.get_draft(draft_name)
|
||||
if draft is None:
|
||||
console.print(f"[red]Draft not found: {draft_name}[/]")
|
||||
return None
|
||||
|
||||
text_excerpt = ""
|
||||
if draft.full_text:
|
||||
text_excerpt = draft.full_text[:3000]
|
||||
|
||||
prompt = EXTRACT_IDEAS_PROMPT.format(
|
||||
name=draft.name, title=draft.title,
|
||||
pages=draft.pages or "?",
|
||||
abstract=draft.abstract[:2000],
|
||||
text_excerpt=text_excerpt,
|
||||
)
|
||||
phash = _prompt_hash("ideas-" + prompt)
|
||||
|
||||
if use_cache:
|
||||
cached = self.db.get_cached_response(draft_name, phash)
|
||||
if cached:
|
||||
try:
|
||||
ideas = json.loads(cached)
|
||||
if isinstance(ideas, list):
|
||||
self.db.insert_ideas(draft_name, ideas)
|
||||
return ideas
|
||||
except (json.JSONDecodeError, KeyError):
|
||||
pass
|
||||
|
||||
try:
|
||||
text, in_tok, out_tok = self._call_claude(prompt, max_tokens=1024)
|
||||
text = self._extract_json(text)
|
||||
ideas = json.loads(text)
|
||||
if not isinstance(ideas, list):
|
||||
ideas = [ideas]
|
||||
|
||||
self.db.cache_response(
|
||||
draft_name, phash, self.config.claude_model,
|
||||
prompt, text, in_tok, out_tok,
|
||||
)
|
||||
self.db.insert_ideas(draft_name, ideas)
|
||||
return ideas
|
||||
except (json.JSONDecodeError, anthropic.APIError) as e:
|
||||
console.print(f"[red]Failed ideas for {draft_name}: {e}[/]")
|
||||
return None
|
||||
|
||||
def extract_ideas_batch(self, draft_names: list[str], cheap: bool = True) -> int:
|
||||
"""Extract ideas from multiple drafts in a single API call.
|
||||
|
||||
Uses batching to share prompt overhead — ~5x fewer API calls,
|
||||
~3x fewer tokens than individual extraction.
|
||||
"""
|
||||
drafts = []
|
||||
for name in draft_names:
|
||||
d = self.db.get_draft(name)
|
||||
if d:
|
||||
drafts.append(d)
|
||||
|
||||
if not drafts:
|
||||
return 0
|
||||
|
||||
# Build compact batch block — abstract only (no full text for batch)
|
||||
drafts_block = ""
|
||||
for d in drafts:
|
||||
drafts_block += f"\n---\n{d.name} | {d.title}\nAbstract: {d.abstract[:800]}\n"
|
||||
|
||||
prompt = BATCH_IDEAS_PROMPT.format(drafts_block=drafts_block)
|
||||
phash = _prompt_hash(prompt)
|
||||
|
||||
try:
|
||||
text, in_tok, out_tok = self._call_claude(
|
||||
prompt, max_tokens=400 * len(drafts), cheap=cheap
|
||||
)
|
||||
text = self._extract_json(text)
|
||||
results = json.loads(text)
|
||||
|
||||
if not isinstance(results, dict):
|
||||
# Fallback: if it returned a list, try to match by order
|
||||
if isinstance(results, list) and len(results) == len(drafts):
|
||||
results = {d.name: r for d, r in zip(drafts, results)}
|
||||
else:
|
||||
return 0
|
||||
|
||||
count = 0
|
||||
for d in drafts:
|
||||
ideas = results.get(d.name, [])
|
||||
if ideas:
|
||||
if not isinstance(ideas, list):
|
||||
ideas = [ideas]
|
||||
self.db.cache_response(
|
||||
d.name, _prompt_hash(f"batch-ideas-{phash}-{d.name}"),
|
||||
self.config.claude_model_cheap if cheap else self.config.claude_model,
|
||||
f"batch-ideas[{d.name}]", json.dumps(ideas),
|
||||
in_tok // len(drafts), out_tok // len(drafts),
|
||||
)
|
||||
self.db.insert_ideas(d.name, ideas)
|
||||
count += 1
|
||||
return count
|
||||
except (json.JSONDecodeError, anthropic.APIError) as e:
|
||||
console.print(f"[red]Batch ideas failed: {e}[/]")
|
||||
return 0
|
||||
|
||||
def extract_all_ideas(self, limit: int = 300, batch_size: int = 5, cheap: bool = True) -> int:
|
||||
"""Extract ideas from all drafts that don't have them yet.
|
||||
|
||||
Args:
|
||||
batch_size: Number of drafts per API call (default 5).
|
||||
Set to 1 to use individual calls with full text.
|
||||
cheap: Use Haiku model for ~10x lower cost (default True).
|
||||
"""
|
||||
missing = self.db.drafts_without_ideas(limit=limit)
|
||||
if not missing:
|
||||
console.print("All drafts already have extracted ideas.")
|
||||
return 0
|
||||
|
||||
model_label = "Haiku" if cheap else "Sonnet"
|
||||
if batch_size > 1:
|
||||
console.print(
|
||||
f"Extracting ideas from [bold]{len(missing)}[/] drafts "
|
||||
f"(batches of {batch_size}, {model_label})..."
|
||||
)
|
||||
else:
|
||||
console.print(f"Extracting ideas from [bold]{len(missing)}[/] drafts ({model_label})...")
|
||||
|
||||
count = 0
|
||||
with Progress(
|
||||
SpinnerColumn(),
|
||||
TextColumn("[progress.description]{task.description}"),
|
||||
BarColumn(),
|
||||
MofNCompleteColumn(),
|
||||
console=console,
|
||||
) as progress:
|
||||
task = progress.add_task("Extracting ideas...", total=len(missing))
|
||||
|
||||
if batch_size > 1:
|
||||
for i in range(0, len(missing), batch_size):
|
||||
batch = missing[i:i + batch_size]
|
||||
names = ", ".join(n.split("-")[-1][:10] for n in batch)
|
||||
progress.update(task, description=f"Batch: {names}")
|
||||
n = self.extract_ideas_batch(batch, cheap=cheap)
|
||||
count += n
|
||||
progress.advance(task, advance=len(batch))
|
||||
else:
|
||||
for name in missing:
|
||||
progress.update(task, description=f"Ideas: {name.split('-')[-1][:15]}")
|
||||
result = self.extract_ideas(name)
|
||||
if result:
|
||||
count += 1
|
||||
progress.advance(task)
|
||||
|
||||
in_tok, out_tok = self.db.total_tokens_used()
|
||||
console.print(
|
||||
f"Extracted ideas from [bold green]{count}[/] drafts "
|
||||
f"({self.db.idea_count()} total ideas) "
|
||||
f"| Tokens: {in_tok:,} in + {out_tok:,} out"
|
||||
)
|
||||
return count
|
||||
|
||||
def gap_analysis(self) -> list[dict]:
|
||||
"""Analyze the full landscape and identify gaps."""
|
||||
# Build compressed landscape summary
|
||||
pairs = self.db.drafts_with_ratings(limit=500)
|
||||
total = self.db.count_drafts()
|
||||
|
||||
# Category summary
|
||||
from collections import defaultdict
|
||||
cat_counts: dict[str, int] = defaultdict(int)
|
||||
for _, rating in pairs:
|
||||
for c in rating.categories:
|
||||
cat_counts[c] += 1
|
||||
category_summary = "\n".join(f"- {c}: {n} drafts" for c, n in
|
||||
sorted(cat_counts.items(), key=lambda x: x[1], reverse=True))
|
||||
|
||||
# Top ideas (if available)
|
||||
all_ideas = self.db.all_ideas()
|
||||
idea_freq: dict[str, int] = defaultdict(int)
|
||||
for idea in all_ideas:
|
||||
idea_freq[idea["title"]] += 1
|
||||
top_ideas_list = sorted(idea_freq.items(), key=lambda x: x[1], reverse=True)[:20]
|
||||
if top_ideas_list:
|
||||
top_ideas = "\n".join(f"- {title} ({count} drafts)" for title, count in top_ideas_list)
|
||||
else:
|
||||
top_ideas = "(No idea extraction data available yet)"
|
||||
|
||||
# Overlap summary — use clusters report if it exists
|
||||
overlap_summary = "Multiple clusters of near-duplicate drafts exist, particularly in:\n"
|
||||
for c, n in sorted(cat_counts.items(), key=lambda x: x[1], reverse=True)[:5]:
|
||||
overlap_summary += f"- {c} ({n} drafts, high internal overlap)\n"
|
||||
|
||||
prompt = GAP_ANALYSIS_PROMPT.format(
|
||||
total=total,
|
||||
category_summary=category_summary,
|
||||
top_ideas=top_ideas,
|
||||
overlap_summary=overlap_summary,
|
||||
)
|
||||
phash = _prompt_hash(prompt)
|
||||
|
||||
# Check cache
|
||||
cached = self.db.get_cached_response("_landscape_", phash)
|
||||
if cached:
|
||||
try:
|
||||
gaps = json.loads(cached)
|
||||
if isinstance(gaps, list):
|
||||
self.db.insert_gaps(gaps)
|
||||
return gaps
|
||||
except (json.JSONDecodeError, KeyError):
|
||||
pass
|
||||
|
||||
try:
|
||||
text, in_tok, out_tok = self._call_claude(prompt, max_tokens=4096)
|
||||
text = self._extract_json(text)
|
||||
gaps = json.loads(text)
|
||||
if not isinstance(gaps, list):
|
||||
gaps = [gaps]
|
||||
|
||||
self.db.cache_response(
|
||||
"_landscape_", phash, self.config.claude_model,
|
||||
prompt, text, in_tok, out_tok,
|
||||
)
|
||||
self.db.insert_gaps(gaps)
|
||||
return gaps
|
||||
except (json.JSONDecodeError, anthropic.APIError) as e:
|
||||
console.print(f"[red]Gap analysis failed: {e}[/]")
|
||||
return []
|
||||
|
||||
def compare_drafts(self, draft_names: list[str]) -> str:
|
||||
"""Compare multiple drafts and return analysis text."""
|
||||
parts = []
|
||||
|
||||
137
src/ietf_analyzer/authors.py
Normal file
137
src/ietf_analyzer/authors.py
Normal file
@@ -0,0 +1,137 @@
|
||||
"""Author network — fetch authors from Datatracker, build collaboration graph."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time as time_mod
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import httpx
|
||||
from rich.console import Console
|
||||
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, MofNCompleteColumn
|
||||
|
||||
from .config import Config
|
||||
from .db import Database
|
||||
from .models import Author
|
||||
|
||||
API_BASE = "https://datatracker.ietf.org/api/v1"
|
||||
|
||||
console = Console()
|
||||
|
||||
|
||||
class AuthorNetwork:
|
||||
def __init__(self, config: Config | None = None, db: Database | None = None):
|
||||
self.config = config or Config.load()
|
||||
self.db = db or Database(self.config)
|
||||
self.client = httpx.Client(timeout=30, follow_redirects=True)
|
||||
self._person_cache: dict[int, Author] = {}
|
||||
|
||||
def close(self) -> None:
|
||||
self.client.close()
|
||||
|
||||
def _extract_person_id(self, person_uri: str) -> int | None:
|
||||
"""Extract person_id from a URI like /api/v1/person/person/12345/."""
|
||||
if not person_uri:
|
||||
return None
|
||||
parts = person_uri.strip("/").split("/")
|
||||
try:
|
||||
return int(parts[-1])
|
||||
except (ValueError, IndexError):
|
||||
return None
|
||||
|
||||
def fetch_person(self, person_id: int) -> Author | None:
|
||||
"""Fetch a person's details from Datatracker."""
|
||||
if person_id in self._person_cache:
|
||||
return self._person_cache[person_id]
|
||||
|
||||
try:
|
||||
resp = self.client.get(
|
||||
f"{API_BASE}/person/person/{person_id}/",
|
||||
params={"format": "json"},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
author = Author(
|
||||
person_id=person_id,
|
||||
name=data.get("name", ""),
|
||||
ascii_name=data.get("ascii", ""),
|
||||
affiliation="", # Will be set from documentauthor
|
||||
resource_uri=data.get("resource_uri", ""),
|
||||
fetched_at=datetime.now(timezone.utc).isoformat(),
|
||||
)
|
||||
self._person_cache[person_id] = author
|
||||
time_mod.sleep(self.config.fetch_delay)
|
||||
return author
|
||||
except httpx.HTTPError as e:
|
||||
console.print(f"[dim]Could not fetch person {person_id}: {e}[/]")
|
||||
return None
|
||||
|
||||
def fetch_authors_for_draft(self, draft_name: str) -> list[tuple[Author, int, str]]:
|
||||
"""Fetch authors for a single draft. Returns [(Author, order, affiliation)]."""
|
||||
try:
|
||||
resp = self.client.get(
|
||||
f"{API_BASE}/doc/documentauthor/",
|
||||
params={"document__name": draft_name, "format": "json", "limit": 50},
|
||||
)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
except httpx.HTTPError as e:
|
||||
console.print(f"[dim]Could not fetch authors for {draft_name}: {e}[/]")
|
||||
return []
|
||||
|
||||
results: list[tuple[Author, int, str]] = []
|
||||
for obj in data.get("objects", []):
|
||||
person_uri = obj.get("person", "")
|
||||
person_id = self._extract_person_id(person_uri)
|
||||
if person_id is None:
|
||||
continue
|
||||
|
||||
affiliation = obj.get("affiliation", "")
|
||||
order = obj.get("order", 1)
|
||||
|
||||
author = self.fetch_person(person_id)
|
||||
if author is None:
|
||||
continue
|
||||
|
||||
# Use the affiliation from the document author record
|
||||
author_with_aff = Author(
|
||||
person_id=author.person_id,
|
||||
name=author.name,
|
||||
ascii_name=author.ascii_name,
|
||||
affiliation=affiliation or author.affiliation,
|
||||
resource_uri=author.resource_uri,
|
||||
fetched_at=author.fetched_at,
|
||||
)
|
||||
results.append((author_with_aff, order, affiliation))
|
||||
|
||||
time_mod.sleep(self.config.fetch_delay)
|
||||
return results
|
||||
|
||||
def fetch_all_authors(self, limit: int = 500) -> int:
|
||||
"""Fetch authors for all drafts missing author data."""
|
||||
missing = self.db.drafts_without_authors(limit=limit)
|
||||
if not missing:
|
||||
console.print("All drafts already have author data.")
|
||||
return 0
|
||||
|
||||
count = 0
|
||||
with Progress(
|
||||
SpinnerColumn(),
|
||||
TextColumn("[progress.description]{task.description}"),
|
||||
BarColumn(),
|
||||
MofNCompleteColumn(),
|
||||
console=console,
|
||||
) as progress:
|
||||
task = progress.add_task("Fetching authors...", total=len(missing))
|
||||
for draft_name in missing:
|
||||
progress.update(task, description=f"Authors: {draft_name.split('-')[-1][:15]}")
|
||||
authors = self.fetch_authors_for_draft(draft_name)
|
||||
for author, order, affiliation in authors:
|
||||
self.db.upsert_author(author)
|
||||
self.db.upsert_draft_author(draft_name, author.person_id, order, affiliation)
|
||||
if authors:
|
||||
count += 1
|
||||
progress.advance(task)
|
||||
|
||||
console.print(f"Fetched authors for [bold green]{count}[/] drafts "
|
||||
f"({self.db.author_count()} unique authors)")
|
||||
return count
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
@@ -372,6 +374,435 @@ def digest(days: int):
|
||||
db.close()
|
||||
|
||||
|
||||
@report.command()
|
||||
def timeline():
|
||||
"""Timeline of draft submissions by month and category."""
|
||||
from .reports import Reporter
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
reporter = Reporter(cfg, db)
|
||||
try:
|
||||
path = reporter.timeline()
|
||||
console.print(f"Report saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@report.command("overlap-matrix")
|
||||
def overlap_matrix():
|
||||
"""Full pairwise overlap matrix report."""
|
||||
from .embeddings import Embedder
|
||||
from .reports import Reporter
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
embedder = Embedder(cfg, db)
|
||||
reporter = Reporter(cfg, db)
|
||||
try:
|
||||
console.print("Computing 260x260 similarity matrix...")
|
||||
path = reporter.overlap_matrix(embedder)
|
||||
console.print(f"Report saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@report.command("authors")
|
||||
def authors_report():
|
||||
"""Author and organization network report."""
|
||||
from .reports import Reporter
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
reporter = Reporter(cfg, db)
|
||||
try:
|
||||
path = reporter.authors_report()
|
||||
console.print(f"Report saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@report.command("ideas")
|
||||
def ideas_report():
|
||||
"""Report on extracted technical ideas."""
|
||||
from .reports import Reporter
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
reporter = Reporter(cfg, db)
|
||||
try:
|
||||
path = reporter.ideas_report()
|
||||
console.print(f"Report saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
# ── visualize ────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@main.group()
|
||||
def viz():
|
||||
"""Generate interactive visualizations (HTML/PNG)."""
|
||||
pass
|
||||
|
||||
|
||||
@viz.command("all")
|
||||
def viz_all():
|
||||
"""Generate all available visualizations."""
|
||||
from .visualize import Visualizer
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
v = Visualizer(cfg, db)
|
||||
try:
|
||||
paths = v.generate_all()
|
||||
console.print(f"\n[bold green]{len(paths)} visualizations[/] saved to {v.output_dir}/")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@viz.command("landscape")
|
||||
@click.option("--method", "-m", default="tsne", type=click.Choice(["umap", "tsne"]),
|
||||
help="Dimensionality reduction method")
|
||||
def viz_landscape(method: str):
|
||||
"""2D scatter of draft embeddings colored by category."""
|
||||
from .visualize import Visualizer
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
v = Visualizer(cfg, db)
|
||||
try:
|
||||
path = v.landscape_scatter(method=method)
|
||||
console.print(f"Saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@viz.command("heatmap")
|
||||
def viz_heatmap():
|
||||
"""Clustered similarity heatmap (PNG)."""
|
||||
from .visualize import Visualizer
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
v = Visualizer(cfg, db)
|
||||
try:
|
||||
path = v.similarity_heatmap()
|
||||
console.print(f"Saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@viz.command("distributions")
|
||||
def viz_distributions():
|
||||
"""Rating dimension distributions by category (PNG)."""
|
||||
from .visualize import Visualizer
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
v = Visualizer(cfg, db)
|
||||
try:
|
||||
path = v.score_distributions()
|
||||
console.print(f"Saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@viz.command("timeline")
|
||||
def viz_timeline():
|
||||
"""Stacked area chart of monthly submissions."""
|
||||
from .visualize import Visualizer
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
v = Visualizer(cfg, db)
|
||||
try:
|
||||
path = v.timeline_chart()
|
||||
console.print(f"Saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@viz.command("bubble")
|
||||
def viz_bubble():
|
||||
"""Interactive bubble chart: novelty vs maturity."""
|
||||
from .visualize import Visualizer
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
v = Visualizer(cfg, db)
|
||||
try:
|
||||
path = v.bubble_explorer()
|
||||
console.print(f"Saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@viz.command("radar")
|
||||
def viz_radar():
|
||||
"""Radar chart of average category rating profiles."""
|
||||
from .visualize import Visualizer
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
v = Visualizer(cfg, db)
|
||||
try:
|
||||
path = v.category_radar()
|
||||
console.print(f"Saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@viz.command("network")
|
||||
@click.option("--min-shared", "-n", default=2, help="Minimum shared drafts for an edge")
|
||||
def viz_network(min_shared: int):
|
||||
"""Interactive author collaboration network graph."""
|
||||
from .visualize import Visualizer
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
v = Visualizer(cfg, db)
|
||||
try:
|
||||
path = v.author_network(min_shared=min_shared)
|
||||
console.print(f"Saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@viz.command("treemap")
|
||||
def viz_treemap():
|
||||
"""Category treemap colored by average score."""
|
||||
from .visualize import Visualizer
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
v = Visualizer(cfg, db)
|
||||
try:
|
||||
path = v.category_treemap()
|
||||
console.print(f"Saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@viz.command("quality")
|
||||
def viz_quality():
|
||||
"""Score vs uniqueness scatter (quality vs redundancy)."""
|
||||
from .visualize import Visualizer
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
v = Visualizer(cfg, db)
|
||||
try:
|
||||
path = v.score_vs_overlap()
|
||||
console.print(f"Saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@viz.command("orgs")
|
||||
def viz_orgs():
|
||||
"""Organization contribution bar chart."""
|
||||
from .visualize import Visualizer
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
v = Visualizer(cfg, db)
|
||||
try:
|
||||
path = v.org_contributions()
|
||||
console.print(f"Saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@viz.command("ideas")
|
||||
def viz_ideas():
|
||||
"""Ideas frequency chart by type."""
|
||||
from .visualize import Visualizer
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
v = Visualizer(cfg, db)
|
||||
try:
|
||||
path = v.ideas_chart()
|
||||
console.print(f"Saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@viz.command("browser")
|
||||
def viz_browser():
|
||||
"""Interactive filterable draft browser (standalone HTML)."""
|
||||
from .visualize import Visualizer
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
v = Visualizer(cfg, db)
|
||||
try:
|
||||
path = v.draft_browser()
|
||||
console.print(f"Saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
# ── authors ─────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@main.command()
|
||||
@click.argument("name", required=False)
|
||||
@click.option("--fetch/--no-fetch", default=False, help="Fetch author data from Datatracker first")
|
||||
@click.option("--limit", "-n", default=20, help="Number of top authors to show")
|
||||
def authors(name: str | None, fetch: bool, limit: int):
|
||||
"""Show authors for a draft, or top authors overall."""
|
||||
from .authors import AuthorNetwork
|
||||
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
network = AuthorNetwork(cfg, db)
|
||||
|
||||
try:
|
||||
if fetch:
|
||||
count = network.fetch_all_authors()
|
||||
console.print(f"Fetched authors for [bold green]{count}[/] drafts")
|
||||
|
||||
if name:
|
||||
draft_authors = db.get_authors_for_draft(name)
|
||||
if not draft_authors:
|
||||
console.print(f"[yellow]No author data for {name}. Run `ietf authors --fetch` first.[/]")
|
||||
return
|
||||
console.print(f"\n[bold]Authors of {name}:[/]")
|
||||
for a in draft_authors:
|
||||
console.print(f" - {a.name} ({a.affiliation or 'no affiliation'})")
|
||||
else:
|
||||
top = db.top_authors(limit=limit)
|
||||
if not top:
|
||||
console.print("[yellow]No author data. Run `ietf authors --fetch` first.[/]")
|
||||
return
|
||||
table = Table(title=f"Top {limit} Authors")
|
||||
table.add_column("#", justify="right", width=4)
|
||||
table.add_column("Author", style="cyan")
|
||||
table.add_column("Organization")
|
||||
table.add_column("Drafts", justify="right", width=6)
|
||||
for rank, (aname, aff, cnt, _) in enumerate(top, 1):
|
||||
table.add_row(str(rank), aname, aff, str(cnt))
|
||||
console.print(table)
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
@main.command()
|
||||
@click.option("--top", "-n", default=20, help="Top N to show")
|
||||
def network(top: int):
|
||||
"""Show author collaboration network."""
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
try:
|
||||
console.print("\n[bold]Top Organizations[/]")
|
||||
orgs = db.top_orgs(limit=top)
|
||||
if orgs:
|
||||
table = Table()
|
||||
table.add_column("#", justify="right", width=4)
|
||||
table.add_column("Organization", style="cyan")
|
||||
table.add_column("Authors", justify="right", width=8)
|
||||
table.add_column("Drafts", justify="right", width=6)
|
||||
for rank, (org, auth_cnt, draft_cnt) in enumerate(orgs, 1):
|
||||
table.add_row(str(rank), org, str(auth_cnt), str(draft_cnt))
|
||||
console.print(table)
|
||||
|
||||
console.print("\n[bold]Cross-Org Collaboration[/]")
|
||||
cross = db.cross_org_collaborations(limit=top)
|
||||
if cross:
|
||||
table = Table()
|
||||
table.add_column("Org A", style="cyan")
|
||||
table.add_column("Org B", style="cyan")
|
||||
table.add_column("Shared Drafts", justify="right", width=8)
|
||||
for org_a, org_b, shared in cross:
|
||||
table.add_row(org_a, org_b, str(shared))
|
||||
console.print(table)
|
||||
else:
|
||||
console.print("[yellow]No author data. Run `ietf authors --fetch` first.[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
# ── ideas ───────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@main.command()
|
||||
@click.argument("name", required=False)
|
||||
@click.option("--all", "extract_all", is_flag=True, help="Extract ideas from all drafts")
|
||||
@click.option("--limit", "-n", default=50, help="Max drafts to extract (with --all)")
|
||||
@click.option("--batch", "-b", default=5, help="Drafts per API call (default 5, set 1 for individual)")
|
||||
@click.option("--cheap/--quality", default=True, help="Use Haiku (cheap) vs Sonnet (quality)")
|
||||
def ideas(name: str | None, extract_all: bool, limit: int, batch: int, cheap: bool):
|
||||
"""Extract technical ideas from drafts using Claude."""
|
||||
from .analyzer import Analyzer
|
||||
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
analyzer = Analyzer(cfg, db)
|
||||
|
||||
try:
|
||||
if extract_all:
|
||||
count = analyzer.extract_all_ideas(limit=limit, batch_size=batch, cheap=cheap)
|
||||
console.print(f"Extracted ideas from [bold green]{count}[/] drafts")
|
||||
elif name:
|
||||
idea_list = analyzer.extract_ideas(name)
|
||||
if idea_list:
|
||||
console.print(f"\n[bold]Ideas from {name}:[/]\n")
|
||||
for idea in idea_list:
|
||||
console.print(f" [{idea.get('type', '?')}] [bold]{idea['title']}[/]")
|
||||
console.print(f" {idea['description']}\n")
|
||||
else:
|
||||
console.print("[red]Extraction failed or no ideas found[/]")
|
||||
else:
|
||||
console.print("Provide a draft name or use --all")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
# ── gaps ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@main.command()
|
||||
@click.option("--refresh", is_flag=True, help="Re-run gap analysis even if cached")
|
||||
def gaps(refresh: bool):
|
||||
"""Identify gaps in the current draft landscape using Claude."""
|
||||
from .analyzer import Analyzer
|
||||
from .reports import Reporter
|
||||
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
analyzer = Analyzer(cfg, db)
|
||||
reporter = Reporter(cfg, db)
|
||||
|
||||
try:
|
||||
existing = db.all_gaps()
|
||||
if existing and not refresh:
|
||||
console.print(f"[bold]{len(existing)} gaps[/] already identified (use --refresh to re-run)\n")
|
||||
else:
|
||||
gap_list = analyzer.gap_analysis()
|
||||
console.print(f"\nIdentified [bold green]{len(gap_list)}[/] gaps\n")
|
||||
existing = gap_list
|
||||
|
||||
for i, gap in enumerate(existing if isinstance(existing[0], dict) else [], 1):
|
||||
sev = gap.get("severity", "medium").upper()
|
||||
console.print(f" [bold]{i}. {gap['topic']}[/] [{sev}]")
|
||||
console.print(f" {gap['description'][:100]}\n")
|
||||
|
||||
path = reporter.gaps_report()
|
||||
console.print(f"Report saved: [bold]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
# ── draft-gen ───────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@main.command("draft-gen")
|
||||
@click.argument("gap_topic")
|
||||
@click.option("--output", "-o", help="Output file path")
|
||||
def draft_gen(gap_topic: str, output: str | None):
|
||||
"""Generate an Internet-Draft addressing a landscape gap."""
|
||||
from .draftgen import DraftGenerator
|
||||
from .analyzer import Analyzer
|
||||
|
||||
cfg = _get_config()
|
||||
db = Database(cfg)
|
||||
analyzer = Analyzer(cfg, db)
|
||||
generator = DraftGenerator(cfg, db, analyzer)
|
||||
|
||||
try:
|
||||
out_path = output or str(Path(cfg.data_dir) / "reports" / "generated-draft.txt")
|
||||
console.print(f"Generating Internet-Draft on: [bold]{gap_topic}[/]")
|
||||
path = generator.generate(gap_topic, output_path=out_path)
|
||||
console.print(f"\nDraft saved: [bold green]{path}[/]")
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
# ── config ───────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
|
||||
@@ -26,6 +26,7 @@ class Config:
|
||||
ollama_url: str = "http://localhost:11434"
|
||||
ollama_embed_model: str = "nomic-embed-text"
|
||||
claude_model: str = "claude-sonnet-4-20250514"
|
||||
claude_model_cheap: str = "claude-haiku-4-5-20251001"
|
||||
search_keywords: list[str] = field(default_factory=lambda: list(DEFAULT_KEYWORDS))
|
||||
# Only fetch drafts newer than this (ISO date string)
|
||||
fetch_since: str = "2024-01-01"
|
||||
|
||||
@@ -10,7 +10,7 @@ from pathlib import Path
|
||||
import numpy as np
|
||||
|
||||
from .config import Config
|
||||
from .models import Draft, Rating
|
||||
from .models import Author, Draft, Rating
|
||||
|
||||
SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS drafts (
|
||||
@@ -76,6 +76,47 @@ CREATE VIRTUAL TABLE IF NOT EXISTS drafts_fts USING fts5(
|
||||
content_rowid='rowid'
|
||||
);
|
||||
|
||||
-- Authors (fetched from Datatracker)
|
||||
CREATE TABLE IF NOT EXISTS authors (
|
||||
person_id INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
ascii_name TEXT,
|
||||
affiliation TEXT DEFAULT '',
|
||||
resource_uri TEXT,
|
||||
fetched_at TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS draft_authors (
|
||||
draft_name TEXT NOT NULL REFERENCES drafts(name),
|
||||
person_id INTEGER NOT NULL REFERENCES authors(person_id),
|
||||
author_order INTEGER DEFAULT 1,
|
||||
affiliation TEXT DEFAULT '',
|
||||
PRIMARY KEY (draft_name, person_id)
|
||||
);
|
||||
|
||||
-- Extracted ideas
|
||||
CREATE TABLE IF NOT EXISTS ideas (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
draft_name TEXT NOT NULL REFERENCES drafts(name),
|
||||
title TEXT NOT NULL,
|
||||
description TEXT NOT NULL,
|
||||
idea_type TEXT DEFAULT '',
|
||||
extracted_at TEXT
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_ideas_draft ON ideas(draft_name);
|
||||
|
||||
-- Gap analysis results
|
||||
CREATE TABLE IF NOT EXISTS gaps (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
topic TEXT NOT NULL,
|
||||
description TEXT NOT NULL,
|
||||
category TEXT DEFAULT '',
|
||||
evidence TEXT DEFAULT '',
|
||||
severity TEXT DEFAULT 'medium',
|
||||
analyzed_at TEXT
|
||||
);
|
||||
|
||||
-- Triggers to keep FTS index in sync
|
||||
CREATE TRIGGER IF NOT EXISTS drafts_ai AFTER INSERT ON drafts BEGIN
|
||||
INSERT INTO drafts_fts(rowid, name, title, abstract, full_text)
|
||||
@@ -341,6 +382,189 @@ class Database:
|
||||
).fetchone()
|
||||
return (row[0], row[1])
|
||||
|
||||
# --- Authors ---
|
||||
|
||||
def upsert_author(self, author: Author) -> None:
|
||||
self.conn.execute(
|
||||
"""INSERT INTO authors (person_id, name, ascii_name, affiliation, resource_uri, fetched_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(person_id) DO UPDATE SET
|
||||
name=excluded.name, ascii_name=excluded.ascii_name,
|
||||
affiliation=excluded.affiliation, resource_uri=excluded.resource_uri,
|
||||
fetched_at=excluded.fetched_at
|
||||
""",
|
||||
(author.person_id, author.name, author.ascii_name,
|
||||
author.affiliation, author.resource_uri, author.fetched_at),
|
||||
)
|
||||
self.conn.commit()
|
||||
|
||||
def upsert_draft_author(
|
||||
self, draft_name: str, person_id: int, order: int = 1, affiliation: str = ""
|
||||
) -> None:
|
||||
self.conn.execute(
|
||||
"""INSERT INTO draft_authors (draft_name, person_id, author_order, affiliation)
|
||||
VALUES (?, ?, ?, ?)
|
||||
ON CONFLICT(draft_name, person_id) DO UPDATE SET
|
||||
author_order=excluded.author_order, affiliation=excluded.affiliation
|
||||
""",
|
||||
(draft_name, person_id, order, affiliation),
|
||||
)
|
||||
self.conn.commit()
|
||||
|
||||
def get_authors_for_draft(self, draft_name: str) -> list[Author]:
|
||||
rows = self.conn.execute(
|
||||
"""SELECT a.* FROM authors a
|
||||
JOIN draft_authors da ON a.person_id = da.person_id
|
||||
WHERE da.draft_name = ?
|
||||
ORDER BY da.author_order""",
|
||||
(draft_name,),
|
||||
).fetchall()
|
||||
return [Author(
|
||||
person_id=r["person_id"], name=r["name"],
|
||||
ascii_name=r.get("ascii_name", ""),
|
||||
affiliation=r.get("affiliation", ""),
|
||||
resource_uri=r.get("resource_uri", ""),
|
||||
fetched_at=r.get("fetched_at"),
|
||||
) for r in rows]
|
||||
|
||||
def drafts_without_authors(self, limit: int = 500) -> list[str]:
|
||||
rows = self.conn.execute(
|
||||
"""SELECT d.name FROM drafts d
|
||||
LEFT JOIN draft_authors da ON d.name = da.draft_name
|
||||
WHERE da.draft_name IS NULL
|
||||
LIMIT ?""",
|
||||
(limit,),
|
||||
).fetchall()
|
||||
return [r["name"] for r in rows]
|
||||
|
||||
def author_count(self) -> int:
|
||||
return self.conn.execute("SELECT COUNT(*) FROM authors").fetchone()[0]
|
||||
|
||||
def top_authors(self, limit: int = 20) -> list[tuple[str, str, int, list[str]]]:
|
||||
"""Return (name, affiliation, draft_count, [draft_names])."""
|
||||
rows = self.conn.execute(
|
||||
"""SELECT a.name, a.affiliation, COUNT(da.draft_name) as cnt,
|
||||
GROUP_CONCAT(da.draft_name, '||') as drafts
|
||||
FROM authors a
|
||||
JOIN draft_authors da ON a.person_id = da.person_id
|
||||
GROUP BY a.person_id
|
||||
ORDER BY cnt DESC
|
||||
LIMIT ?""",
|
||||
(limit,),
|
||||
).fetchall()
|
||||
return [
|
||||
(r["name"], r["affiliation"], r["cnt"],
|
||||
r["drafts"].split("||") if r["drafts"] else [])
|
||||
for r in rows
|
||||
]
|
||||
|
||||
def top_orgs(self, limit: int = 20) -> list[tuple[str, int, int]]:
|
||||
"""Return (org, author_count, draft_count)."""
|
||||
rows = self.conn.execute(
|
||||
"""SELECT da.affiliation as org,
|
||||
COUNT(DISTINCT da.person_id) as authors,
|
||||
COUNT(DISTINCT da.draft_name) as drafts
|
||||
FROM draft_authors da
|
||||
WHERE da.affiliation != ''
|
||||
GROUP BY da.affiliation
|
||||
ORDER BY drafts DESC
|
||||
LIMIT ?""",
|
||||
(limit,),
|
||||
).fetchall()
|
||||
return [(r["org"], r["authors"], r["drafts"]) for r in rows]
|
||||
|
||||
def coauthor_pairs(self) -> list[tuple[str, str, int]]:
|
||||
"""Return (author_a, author_b, shared_drafts) for all co-author pairs."""
|
||||
rows = self.conn.execute(
|
||||
"""SELECT a1.name as a, a2.name as b, COUNT(*) as shared
|
||||
FROM draft_authors da1
|
||||
JOIN draft_authors da2 ON da1.draft_name = da2.draft_name AND da1.person_id < da2.person_id
|
||||
JOIN authors a1 ON da1.person_id = a1.person_id
|
||||
JOIN authors a2 ON da2.person_id = a2.person_id
|
||||
GROUP BY da1.person_id, da2.person_id
|
||||
ORDER BY shared DESC"""
|
||||
).fetchall()
|
||||
return [(r["a"], r["b"], r["shared"]) for r in rows]
|
||||
|
||||
def cross_org_collaborations(self, limit: int = 20) -> list[tuple[str, str, int]]:
|
||||
"""Return (org_a, org_b, shared_drafts) for cross-org collaboration."""
|
||||
rows = self.conn.execute(
|
||||
"""SELECT da1.affiliation as org_a, da2.affiliation as org_b,
|
||||
COUNT(DISTINCT da1.draft_name) as shared
|
||||
FROM draft_authors da1
|
||||
JOIN draft_authors da2 ON da1.draft_name = da2.draft_name
|
||||
AND da1.person_id < da2.person_id
|
||||
WHERE da1.affiliation != '' AND da2.affiliation != ''
|
||||
AND da1.affiliation != da2.affiliation
|
||||
GROUP BY da1.affiliation, da2.affiliation
|
||||
ORDER BY shared DESC
|
||||
LIMIT ?""",
|
||||
(limit,),
|
||||
).fetchall()
|
||||
return [(r["org_a"], r["org_b"], r["shared"]) for r in rows]
|
||||
|
||||
# --- Ideas ---
|
||||
|
||||
def insert_ideas(self, draft_name: str, ideas: list[dict]) -> None:
|
||||
# Clear existing ideas for this draft first
|
||||
self.conn.execute("DELETE FROM ideas WHERE draft_name = ?", (draft_name,))
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
for idea in ideas:
|
||||
self.conn.execute(
|
||||
"""INSERT INTO ideas (draft_name, title, description, idea_type, extracted_at)
|
||||
VALUES (?, ?, ?, ?, ?)""",
|
||||
(draft_name, idea["title"], idea["description"],
|
||||
idea.get("type", ""), now),
|
||||
)
|
||||
self.conn.commit()
|
||||
|
||||
def get_ideas_for_draft(self, draft_name: str) -> list[dict]:
|
||||
rows = self.conn.execute(
|
||||
"SELECT * FROM ideas WHERE draft_name = ?", (draft_name,)
|
||||
).fetchall()
|
||||
return [{"title": r["title"], "description": r["description"],
|
||||
"type": r["idea_type"], "draft_name": r["draft_name"]} for r in rows]
|
||||
|
||||
def drafts_without_ideas(self, limit: int = 500) -> list[str]:
|
||||
rows = self.conn.execute(
|
||||
"""SELECT d.name FROM drafts d
|
||||
LEFT JOIN ideas i ON d.name = i.draft_name
|
||||
WHERE i.draft_name IS NULL
|
||||
LIMIT ?""",
|
||||
(limit,),
|
||||
).fetchall()
|
||||
return [r["name"] for r in rows]
|
||||
|
||||
def all_ideas(self) -> list[dict]:
|
||||
rows = self.conn.execute(
|
||||
"SELECT * FROM ideas ORDER BY draft_name"
|
||||
).fetchall()
|
||||
return [{"title": r["title"], "description": r["description"],
|
||||
"type": r["idea_type"], "draft_name": r["draft_name"]} for r in rows]
|
||||
|
||||
def idea_count(self) -> int:
|
||||
return self.conn.execute("SELECT COUNT(*) FROM ideas").fetchone()[0]
|
||||
|
||||
# --- Gaps ---
|
||||
|
||||
def insert_gaps(self, gaps: list[dict]) -> None:
|
||||
self.conn.execute("DELETE FROM gaps") # Replace old analysis
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
for g in gaps:
|
||||
self.conn.execute(
|
||||
"""INSERT INTO gaps (topic, description, category, evidence, severity, analyzed_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?)""",
|
||||
(g["topic"], g["description"], g.get("category", ""),
|
||||
g.get("evidence", ""), g.get("severity", "medium"), now),
|
||||
)
|
||||
self.conn.commit()
|
||||
|
||||
def all_gaps(self) -> list[dict]:
|
||||
rows = self.conn.execute("SELECT * FROM gaps ORDER BY id").fetchall()
|
||||
return [{"id": r["id"], "topic": r["topic"], "description": r["description"],
|
||||
"category": r["category"], "evidence": r["evidence"],
|
||||
"severity": r["severity"]} for r in rows]
|
||||
|
||||
# --- Helpers ---
|
||||
|
||||
@staticmethod
|
||||
|
||||
235
src/ietf_analyzer/draftgen.py
Normal file
235
src/ietf_analyzer/draftgen.py
Normal file
@@ -0,0 +1,235 @@
|
||||
"""Internet-Draft generation from gap analysis."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import textwrap
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
from rich.console import Console
|
||||
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn, MofNCompleteColumn
|
||||
|
||||
from .config import Config
|
||||
from .db import Database
|
||||
|
||||
console = Console()
|
||||
|
||||
OUTLINE_PROMPT = """\
|
||||
You are writing an IETF Internet-Draft to address this gap in the AI/agent standardization landscape:
|
||||
|
||||
Gap: {gap_topic}
|
||||
{gap_context}
|
||||
|
||||
Related existing drafts (for context, not to duplicate):
|
||||
{related_drafts}
|
||||
|
||||
Generate a detailed outline for an Internet-Draft.
|
||||
Return JSON: {{"title":"full draft title","abstract":"150-250 word abstract","sections":[{{"title":"section title","summary":"2-3 sentence summary of content"}}],"target_wg":"suggested IETF working group","intended_status":"informational|standards-track|experimental"}}
|
||||
|
||||
Include standard sections: Introduction, Terminology, Problem Statement, then 2-4 technical sections, Security Considerations, IANA Considerations.
|
||||
JSON only, no fences."""
|
||||
|
||||
SECTION_PROMPT = """\
|
||||
Write the following section of an Internet-Draft titled "{draft_title}".
|
||||
|
||||
Abstract: {abstract}
|
||||
|
||||
Full outline:
|
||||
{outline_text}
|
||||
|
||||
Write section {section_num}: {section_title}
|
||||
Summary: {section_summary}
|
||||
|
||||
Follow IETF Internet-Draft conventions:
|
||||
- Formal, precise technical language
|
||||
- Use RFC 2119 keywords (MUST, SHOULD, MAY) where appropriate
|
||||
- Reference existing RFCs and drafts where relevant
|
||||
- 3-6 paragraphs per section
|
||||
|
||||
Write the section content only (no section number or title). Plain text."""
|
||||
|
||||
|
||||
class DraftGenerator:
|
||||
def __init__(self, config: Config, db: Database, analyzer):
|
||||
self.config = config
|
||||
self.db = db
|
||||
self.analyzer = analyzer
|
||||
|
||||
def generate_outline(self, gap_topic: str) -> dict:
|
||||
"""Generate draft outline from a gap topic."""
|
||||
# Find related gaps in DB
|
||||
gap_context = ""
|
||||
gaps = self.db.all_gaps()
|
||||
for g in gaps:
|
||||
if gap_topic.lower() in g["topic"].lower() or gap_topic.lower() in g["description"].lower():
|
||||
gap_context = f"Description: {g['description']}\nEvidence: {g['evidence']}"
|
||||
break
|
||||
|
||||
# Get a sample of related drafts for context
|
||||
pairs = self.db.drafts_with_ratings(limit=500)
|
||||
related = []
|
||||
for draft, rating in pairs:
|
||||
if any(gap_topic.lower() in cat.lower() for cat in rating.categories):
|
||||
related.append(f"- {draft.name}: {rating.summary[:80]}")
|
||||
if len(related) >= 8:
|
||||
break
|
||||
|
||||
if not related:
|
||||
# Fallback: use top-rated drafts
|
||||
for draft, rating in pairs[:5]:
|
||||
related.append(f"- {draft.name}: {rating.summary[:80]}")
|
||||
|
||||
prompt = OUTLINE_PROMPT.format(
|
||||
gap_topic=gap_topic,
|
||||
gap_context=gap_context or "(No detailed gap analysis available)",
|
||||
related_drafts="\n".join(related),
|
||||
)
|
||||
|
||||
text, _, _ = self.analyzer._call_claude(prompt, max_tokens=2048)
|
||||
text = self.analyzer._extract_json(text)
|
||||
return json.loads(text)
|
||||
|
||||
def generate_section(self, outline: dict, section_idx: int) -> str:
|
||||
"""Generate a single section of the draft."""
|
||||
sections = outline["sections"]
|
||||
section = sections[section_idx]
|
||||
|
||||
outline_text = "\n".join(
|
||||
f"{i+1}. {s['title']}: {s['summary']}"
|
||||
for i, s in enumerate(sections)
|
||||
)
|
||||
|
||||
prompt = SECTION_PROMPT.format(
|
||||
draft_title=outline["title"],
|
||||
abstract=outline["abstract"],
|
||||
outline_text=outline_text,
|
||||
section_num=section_idx + 1,
|
||||
section_title=section["title"],
|
||||
section_summary=section["summary"],
|
||||
)
|
||||
|
||||
text, _, _ = self.analyzer._call_claude(prompt, max_tokens=2048)
|
||||
return text
|
||||
|
||||
def _wrap_text(self, text: str, indent: int = 3, width: int = 69) -> str:
|
||||
"""Wrap text to Internet-Draft conventions (72 char lines, indented)."""
|
||||
prefix = " " * indent
|
||||
paragraphs = text.strip().split("\n\n")
|
||||
wrapped = []
|
||||
for para in paragraphs:
|
||||
para = " ".join(para.split()) # Normalize whitespace
|
||||
lines = textwrap.wrap(para, width=width, initial_indent=prefix,
|
||||
subsequent_indent=prefix)
|
||||
wrapped.append("\n".join(lines))
|
||||
return "\n\n".join(wrapped)
|
||||
|
||||
def assemble_draft(self, outline: dict, sections: list[str]) -> str:
|
||||
"""Assemble sections into Internet-Draft text format."""
|
||||
now = datetime.now(timezone.utc)
|
||||
expires = now + timedelta(days=185)
|
||||
date_str = now.strftime("%B %Y")
|
||||
exp_str = expires.strftime("%B %d, %Y")
|
||||
|
||||
title = outline["title"]
|
||||
abstract = outline["abstract"]
|
||||
status = outline.get("intended_status", "Informational")
|
||||
wg = outline.get("target_wg", "individual")
|
||||
|
||||
# Generate a draft name from the title
|
||||
words = title.lower().split()
|
||||
slug = "-".join(w for w in words[:4] if w.isalnum())
|
||||
draft_name = f"draft-ai-{slug}-00"
|
||||
|
||||
lines = []
|
||||
|
||||
# Header
|
||||
lines.append(f"Internet-Draft AI/Agent WG")
|
||||
lines.append(f"Intended status: {status:<44s}{date_str}")
|
||||
lines.append(f"Expires: {exp_str}")
|
||||
lines.append("")
|
||||
lines.append("")
|
||||
|
||||
# Title (centered)
|
||||
title_line = title
|
||||
lines.append(f" {title_line}")
|
||||
lines.append(f" {draft_name}")
|
||||
lines.append("")
|
||||
|
||||
# Abstract
|
||||
lines.append("Abstract")
|
||||
lines.append("")
|
||||
lines.append(self._wrap_text(abstract))
|
||||
lines.append("")
|
||||
|
||||
# Status of This Memo
|
||||
lines.append("Status of This Memo")
|
||||
lines.append("")
|
||||
lines.append(self._wrap_text(
|
||||
"This Internet-Draft is submitted in full conformance with the "
|
||||
"provisions of BCP 78 and BCP 79."
|
||||
))
|
||||
lines.append("")
|
||||
lines.append(self._wrap_text(
|
||||
f"This document is intended to have {status} status. "
|
||||
"Distribution of this memo is unlimited."
|
||||
))
|
||||
lines.append("")
|
||||
|
||||
# Table of Contents
|
||||
lines.append("Table of Contents")
|
||||
lines.append("")
|
||||
for i, section in enumerate(outline["sections"], 1):
|
||||
dots = "." * (60 - len(section["title"]))
|
||||
lines.append(f" {i}. {section['title']} {dots} {i + 2}")
|
||||
lines.append("")
|
||||
|
||||
# Sections
|
||||
for i, (section_info, section_text) in enumerate(
|
||||
zip(outline["sections"], sections), 1
|
||||
):
|
||||
lines.append(f"{i}. {section_info['title']}")
|
||||
lines.append("")
|
||||
lines.append(self._wrap_text(section_text))
|
||||
lines.append("")
|
||||
|
||||
# Author's Address
|
||||
lines.append("Author's Address")
|
||||
lines.append("")
|
||||
lines.append(" Generated by IETF Draft Analyzer")
|
||||
lines.append(f" {now.strftime('%Y-%m-%d')}")
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
def generate(self, gap_topic: str, output_path: str | None = None) -> str:
|
||||
"""Full pipeline: outline -> sections -> assemble -> write file."""
|
||||
console.print("[bold]Step 1/3:[/] Generating outline...")
|
||||
outline = self.generate_outline(gap_topic)
|
||||
console.print(f" Title: [cyan]{outline['title']}[/]")
|
||||
console.print(f" Sections: {len(outline['sections'])}")
|
||||
console.print(f" Target WG: {outline.get('target_wg', '?')}")
|
||||
|
||||
console.print("\n[bold]Step 2/3:[/] Generating sections...")
|
||||
sections = []
|
||||
with Progress(
|
||||
SpinnerColumn(),
|
||||
TextColumn("[progress.description]{task.description}"),
|
||||
BarColumn(),
|
||||
MofNCompleteColumn(),
|
||||
console=console,
|
||||
) as progress:
|
||||
task = progress.add_task("Writing...", total=len(outline["sections"]))
|
||||
for i, s in enumerate(outline["sections"]):
|
||||
progress.update(task, description=f"Section: {s['title'][:30]}")
|
||||
text = self.generate_section(outline, i)
|
||||
sections.append(text)
|
||||
progress.advance(task)
|
||||
|
||||
console.print("\n[bold]Step 3/3:[/] Assembling draft...")
|
||||
draft_text = self.assemble_draft(outline, sections)
|
||||
|
||||
out = Path(output_path) if output_path else Path(self.config.data_dir) / "reports" / "generated-draft.txt"
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
out.write_text(draft_text)
|
||||
return str(out)
|
||||
@@ -6,6 +6,16 @@ from dataclasses import dataclass, field
|
||||
from datetime import datetime
|
||||
|
||||
|
||||
@dataclass
|
||||
class Author:
|
||||
person_id: int
|
||||
name: str
|
||||
ascii_name: str = ""
|
||||
affiliation: str = ""
|
||||
resource_uri: str = ""
|
||||
fetched_at: str | None = None
|
||||
|
||||
|
||||
@dataclass
|
||||
class Draft:
|
||||
name: str # e.g. "draft-zheng-dispatch-agent-identity-management"
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from collections import defaultdict
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
@@ -175,3 +177,427 @@ class Reporter:
|
||||
path = self.output_dir / "digest.md"
|
||||
path.write_text(report)
|
||||
return str(path)
|
||||
|
||||
def timeline(self) -> str:
|
||||
"""Generate a timeline report of draft submissions by month and category."""
|
||||
pairs = self.db.drafts_with_ratings(limit=500)
|
||||
all_drafts = self.db.list_drafts(limit=500, order_by="time ASC")
|
||||
total = len(all_drafts)
|
||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
||||
|
||||
# Group drafts by month
|
||||
by_month: dict[str, list[Draft]] = defaultdict(list)
|
||||
for d in all_drafts:
|
||||
month = d.time[:7] if d.time else "unknown"
|
||||
by_month[month].append(d)
|
||||
|
||||
months = sorted(by_month.keys())
|
||||
|
||||
# Build rating lookup by draft name
|
||||
rating_map: dict[str, Rating] = {}
|
||||
for draft, rating in pairs:
|
||||
rating_map[draft.name] = rating
|
||||
|
||||
# Collect all categories
|
||||
all_cats: set[str] = set()
|
||||
for _, r in pairs:
|
||||
for c in r.categories:
|
||||
all_cats.add(c)
|
||||
cats = sorted(all_cats)
|
||||
|
||||
# Category counts per month
|
||||
cat_by_month: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int))
|
||||
for d in all_drafts:
|
||||
month = d.time[:7] if d.time else "unknown"
|
||||
r = rating_map.get(d.name)
|
||||
if r:
|
||||
for c in r.categories:
|
||||
cat_by_month[month][c] += 1
|
||||
|
||||
lines = [
|
||||
"# IETF AI/Agent Drafts Timeline",
|
||||
f"*Generated {now} — {total} drafts across {len(months)} months*\n",
|
||||
"## Monthly Submission Volume\n",
|
||||
"```",
|
||||
]
|
||||
|
||||
max_count = max(len(ds) for ds in by_month.values()) if by_month else 1
|
||||
for month in months:
|
||||
count = len(by_month[month])
|
||||
bar_len = int(count / max_count * 40) if max_count else 0
|
||||
bar = "#" * bar_len
|
||||
lines.append(f"{month} | {bar:<40s} {count:>3}")
|
||||
|
||||
lines.append("```\n")
|
||||
|
||||
# Category breakdown table
|
||||
lines.append("## Category Breakdown by Month\n")
|
||||
header = "| Month |" + " | ".join(f" {c[:12]:>12}" for c in cats) + " | Total |"
|
||||
sep = "|---------|" + " | ".join("-" * 13 for _ in cats) + " | -----:|"
|
||||
lines.append(header)
|
||||
lines.append(sep)
|
||||
|
||||
for month in months:
|
||||
counts = [str(cat_by_month[month].get(c, 0)).rjust(13) for c in cats]
|
||||
total_m = len(by_month[month])
|
||||
lines.append(f"| {month} |" + " | ".join(counts) + f" | {total_m:>5} |")
|
||||
|
||||
# Trends section
|
||||
lines.append("\n## Trends\n")
|
||||
if len(months) >= 4:
|
||||
mid = len(months) // 2
|
||||
early_months = months[:mid]
|
||||
late_months = months[mid:]
|
||||
|
||||
early_cat: dict[str, int] = defaultdict(int)
|
||||
late_cat: dict[str, int] = defaultdict(int)
|
||||
for m in early_months:
|
||||
for c in cats:
|
||||
early_cat[c] += cat_by_month[m].get(c, 0)
|
||||
for m in late_months:
|
||||
for c in cats:
|
||||
late_cat[c] += cat_by_month[m].get(c, 0)
|
||||
|
||||
growth = []
|
||||
for c in cats:
|
||||
e = early_cat[c]
|
||||
l = late_cat[c]
|
||||
if e > 0:
|
||||
pct = ((l - e) / e) * 100
|
||||
growth.append((c, pct, e, l))
|
||||
elif l > 0:
|
||||
growth.append((c, float('inf'), e, l))
|
||||
|
||||
growth.sort(key=lambda x: x[1], reverse=True)
|
||||
for c, pct, e, l in growth[:5]:
|
||||
if pct == float('inf'):
|
||||
lines.append(f"- **{c}**: new (0 → {l} drafts)")
|
||||
else:
|
||||
lines.append(f"- **{c}**: {pct:+.0f}% ({e} → {l} drafts, early vs late half)")
|
||||
else:
|
||||
lines.append("Not enough months for trend analysis.")
|
||||
|
||||
report = "\n".join(lines)
|
||||
path = self.output_dir / "timeline.md"
|
||||
path.write_text(report)
|
||||
return str(path)
|
||||
|
||||
def overlap_matrix(self, embedder) -> str:
|
||||
"""Generate overlap matrix report from pairwise similarity."""
|
||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
||||
|
||||
names, matrix = embedder.similarity_matrix()
|
||||
n = len(names)
|
||||
|
||||
# Build rating lookup
|
||||
pairs_data = self.db.drafts_with_ratings(limit=500)
|
||||
rating_map: dict[str, Rating] = {}
|
||||
draft_map: dict[str, Draft] = {}
|
||||
for draft, rating in pairs_data:
|
||||
rating_map[draft.name] = rating
|
||||
draft_map[draft.name] = draft
|
||||
|
||||
# Top similar pairs (above 0.80, excluding self)
|
||||
sim_pairs: list[tuple[float, str, str]] = []
|
||||
for i in range(n):
|
||||
for j in range(i + 1, n):
|
||||
if matrix[i, j] >= 0.80:
|
||||
sim_pairs.append((float(matrix[i, j]), names[i], names[j]))
|
||||
sim_pairs.sort(reverse=True)
|
||||
|
||||
lines = [
|
||||
"# Overlap Matrix Report",
|
||||
f"*Generated {now} — {n}x{n} pairwise similarities*\n",
|
||||
f"## Top {min(50, len(sim_pairs))} Most Similar Pairs\n",
|
||||
"| Rank | Similarity | Draft A | Draft B |",
|
||||
"|-----:|-----------:|---------|---------|",
|
||||
]
|
||||
|
||||
for rank, (sim, a, b) in enumerate(sim_pairs[:50], 1):
|
||||
a_link = f"[{a}](https://datatracker.ietf.org/doc/{a}/)"
|
||||
b_link = f"[{b}](https://datatracker.ietf.org/doc/{b}/)"
|
||||
lines.append(f"| {rank} | {sim:.3f} | {a_link} | {b_link} |")
|
||||
|
||||
# Per-category internal overlap
|
||||
cat_drafts: dict[str, list[int]] = defaultdict(list)
|
||||
name_idx = {name: i for i, name in enumerate(names)}
|
||||
for name in names:
|
||||
r = rating_map.get(name)
|
||||
if r:
|
||||
for c in r.categories:
|
||||
if name in name_idx:
|
||||
cat_drafts[c].append(name_idx[name])
|
||||
|
||||
lines.extend([
|
||||
"\n## Per-Category Internal Overlap\n",
|
||||
"| Category | Drafts | Avg Pairwise Sim | Most Similar Pair |",
|
||||
"|----------|-------:|-----------------:|-------------------|",
|
||||
])
|
||||
|
||||
for cat in sorted(cat_drafts.keys()):
|
||||
indices = cat_drafts[cat]
|
||||
if len(indices) < 2:
|
||||
lines.append(f"| {cat} | {len(indices)} | — | — |")
|
||||
continue
|
||||
sims = []
|
||||
best_sim, best_a, best_b = 0.0, "", ""
|
||||
for ii in range(len(indices)):
|
||||
for jj in range(ii + 1, len(indices)):
|
||||
s = float(matrix[indices[ii], indices[jj]])
|
||||
sims.append(s)
|
||||
if s > best_sim:
|
||||
best_sim = s
|
||||
best_a = names[indices[ii]]
|
||||
best_b = names[indices[jj]]
|
||||
avg = sum(sims) / len(sims) if sims else 0
|
||||
short_a = best_a.replace("draft-", "")[:25]
|
||||
short_b = best_b.replace("draft-", "")[:25]
|
||||
lines.append(f"| {cat} | {len(indices)} | {avg:.3f} | {short_a} / {short_b} ({best_sim:.3f}) |")
|
||||
|
||||
# Category cross-overlap matrix
|
||||
cat_names = sorted(cat_drafts.keys())
|
||||
if len(cat_names) > 1:
|
||||
lines.extend([
|
||||
"\n## Category Cross-Overlap\n",
|
||||
"Average similarity between drafts in different categories.\n",
|
||||
"| |" + " | ".join(c[:10] for c in cat_names) + " |",
|
||||
"|-|" + " | ".join("---:" for _ in cat_names) + " |",
|
||||
])
|
||||
|
||||
for ci, c1 in enumerate(cat_names):
|
||||
row = f"| **{c1[:12]}** |"
|
||||
for cj, c2 in enumerate(cat_names):
|
||||
if ci > cj:
|
||||
row += " |"
|
||||
continue
|
||||
idx1, idx2 = cat_drafts[c1], cat_drafts[c2]
|
||||
sims = []
|
||||
for i1 in idx1:
|
||||
for i2 in idx2:
|
||||
if i1 != i2:
|
||||
sims.append(float(matrix[i1, i2]))
|
||||
avg = sum(sims) / len(sims) if sims else 0
|
||||
row += f" {avg:.2f} |"
|
||||
lines.append(row)
|
||||
|
||||
# Most unique drafts
|
||||
lines.append("\n## Most Unique Drafts (max similarity < 0.70)\n")
|
||||
unique_drafts = []
|
||||
for i, name in enumerate(names):
|
||||
max_sim = 0.0
|
||||
best_match = ""
|
||||
for j in range(n):
|
||||
if i != j and matrix[i, j] > max_sim:
|
||||
max_sim = float(matrix[i, j])
|
||||
best_match = names[j]
|
||||
if max_sim < 0.70:
|
||||
d = draft_map.get(name)
|
||||
title = d.title[:60] if d else ""
|
||||
unique_drafts.append((name, max_sim, best_match, title))
|
||||
|
||||
unique_drafts.sort(key=lambda x: x[1])
|
||||
if unique_drafts:
|
||||
for name, ms, bm, title in unique_drafts[:20]:
|
||||
lines.append(f"- **{name}** (max sim: {ms:.3f} with {bm}) — {title}")
|
||||
else:
|
||||
lines.append("No drafts with max similarity below 0.70.")
|
||||
|
||||
report = "\n".join(lines)
|
||||
path = self.output_dir / "overlap-matrix.md"
|
||||
path.write_text(report)
|
||||
return str(path)
|
||||
|
||||
def authors_report(self) -> str:
|
||||
"""Generate author/organization network report."""
|
||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
||||
author_count = self.db.author_count()
|
||||
total_drafts = self.db.count_drafts()
|
||||
|
||||
# Build rating lookup for category info
|
||||
pairs_data = self.db.drafts_with_ratings(limit=500)
|
||||
rating_map: dict[str, Rating] = {}
|
||||
for draft, rating in pairs_data:
|
||||
rating_map[draft.name] = rating
|
||||
|
||||
lines = [
|
||||
"# Author & Organization Network",
|
||||
f"*Generated {now} — {author_count} unique authors across {total_drafts} drafts*\n",
|
||||
]
|
||||
|
||||
# Top authors
|
||||
top = self.db.top_authors(limit=30)
|
||||
lines.extend([
|
||||
"## Top Authors by Draft Count\n",
|
||||
"| # | Author | Organization | Drafts | Categories |",
|
||||
"|--:|--------|-------------|-------:|------------|",
|
||||
])
|
||||
for rank, (name, aff, cnt, draft_names) in enumerate(top, 1):
|
||||
cats: set[str] = set()
|
||||
for dn in draft_names:
|
||||
r = rating_map.get(dn)
|
||||
if r:
|
||||
cats.update(r.categories)
|
||||
cat_str = ", ".join(sorted(cats)[:3])
|
||||
lines.append(f"| {rank} | {name} | {aff} | {cnt} | {cat_str} |")
|
||||
|
||||
# Top orgs
|
||||
orgs = self.db.top_orgs(limit=20)
|
||||
lines.extend([
|
||||
"\n## Top Organizations\n",
|
||||
"| # | Organization | Authors | Drafts |",
|
||||
"|--:|-------------|--------:|-------:|",
|
||||
])
|
||||
for rank, (org, authors, drafts) in enumerate(orgs, 1):
|
||||
lines.append(f"| {rank} | {org} | {authors} | {drafts} |")
|
||||
|
||||
# Co-author pairs
|
||||
coauthors = self.db.coauthor_pairs()
|
||||
if coauthors:
|
||||
lines.extend([
|
||||
"\n## Strongest Collaboration Pairs\n",
|
||||
"| Author A | Author B | Shared Drafts |",
|
||||
"|----------|----------|-----:|",
|
||||
])
|
||||
for a, b, shared in coauthors[:20]:
|
||||
lines.append(f"| {a} | {b} | {shared} |")
|
||||
|
||||
# Cross-org
|
||||
cross = self.db.cross_org_collaborations(limit=15)
|
||||
if cross:
|
||||
lines.extend([
|
||||
"\n## Cross-Organization Collaboration\n",
|
||||
"| Org A | Org B | Shared Drafts |",
|
||||
"|-------|-------|-----:|",
|
||||
])
|
||||
for org_a, org_b, shared in cross:
|
||||
lines.append(f"| {org_a} | {org_b} | {shared} |")
|
||||
|
||||
report = "\n".join(lines)
|
||||
path = self.output_dir / "authors.md"
|
||||
path.write_text(report)
|
||||
return str(path)
|
||||
|
||||
def ideas_report(self) -> str:
|
||||
"""Generate report on extracted ideas across all drafts."""
|
||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
||||
all_ideas = self.db.all_ideas()
|
||||
|
||||
# Build rating lookup for category info
|
||||
pairs_data = self.db.drafts_with_ratings(limit=500)
|
||||
rating_map: dict[str, Rating] = {}
|
||||
for draft, rating in pairs_data:
|
||||
rating_map[draft.name] = rating
|
||||
|
||||
# Group ideas by normalized title for frequency analysis
|
||||
from difflib import SequenceMatcher
|
||||
idea_groups: list[dict] = [] # [{canonical, ideas: [idea], drafts: set}]
|
||||
|
||||
for idea in all_ideas:
|
||||
title_lower = idea["title"].lower().strip()
|
||||
matched = False
|
||||
for group in idea_groups:
|
||||
ratio = SequenceMatcher(None, title_lower, group["canonical"]).ratio()
|
||||
if ratio >= 0.75:
|
||||
group["ideas"].append(idea)
|
||||
group["drafts"].add(idea["draft_name"])
|
||||
matched = True
|
||||
break
|
||||
if not matched:
|
||||
idea_groups.append({
|
||||
"canonical": title_lower,
|
||||
"title": idea["title"],
|
||||
"ideas": [idea],
|
||||
"drafts": {idea["draft_name"]},
|
||||
})
|
||||
|
||||
idea_groups.sort(key=lambda g: len(g["drafts"]), reverse=True)
|
||||
drafts_with_ideas = len(set(i["draft_name"] for i in all_ideas))
|
||||
|
||||
lines = [
|
||||
"# Technical Ideas Extracted from IETF AI/Agent Drafts",
|
||||
f"*Generated {now} — {len(all_ideas)} ideas from {drafts_with_ideas} drafts*\n",
|
||||
]
|
||||
|
||||
# Most common ideas (3+ drafts)
|
||||
common = [g for g in idea_groups if len(g["drafts"]) >= 3]
|
||||
if common:
|
||||
lines.extend([
|
||||
"## Most Common Ideas (appearing in 3+ drafts)\n",
|
||||
"| Idea | Appearances | Drafts |",
|
||||
"|------|------------:|--------|",
|
||||
])
|
||||
for g in common:
|
||||
draft_list = ", ".join(sorted(g["drafts"])[:5])
|
||||
if len(g["drafts"]) > 5:
|
||||
draft_list += f" +{len(g['drafts'])-5} more"
|
||||
lines.append(f"| {g['title']} | {len(g['drafts'])} | {draft_list} |")
|
||||
|
||||
# Ideas appearing in 2 drafts
|
||||
two = [g for g in idea_groups if len(g["drafts"]) == 2]
|
||||
if two:
|
||||
lines.append(f"\n## Ideas Appearing in 2 Drafts ({len(two)} ideas)\n")
|
||||
for g in two[:30]:
|
||||
draft_list = ", ".join(sorted(g["drafts"]))
|
||||
lines.append(f"- **{g['title']}** — {draft_list}")
|
||||
|
||||
# Unique ideas (only 1 draft) - just count and top examples
|
||||
unique = [g for g in idea_groups if len(g["drafts"]) == 1]
|
||||
lines.append(f"\n## Unique Ideas ({len(unique)} ideas appearing in only 1 draft)\n")
|
||||
for g in unique[:20]:
|
||||
idea = g["ideas"][0]
|
||||
lines.append(f"- **{g['title']}** ({idea['draft_name']}) — {idea['description'][:100]}")
|
||||
if len(unique) > 20:
|
||||
lines.append(f"\n*...and {len(unique) - 20} more unique ideas*")
|
||||
|
||||
# By type
|
||||
by_type: dict[str, int] = defaultdict(int)
|
||||
for idea in all_ideas:
|
||||
by_type[idea.get("type", "other")] += 1
|
||||
if by_type:
|
||||
lines.extend(["\n## Ideas by Type\n"])
|
||||
for t, count in sorted(by_type.items(), key=lambda x: x[1], reverse=True):
|
||||
lines.append(f"- **{t or 'untyped'}**: {count}")
|
||||
|
||||
report = "\n".join(lines)
|
||||
path = self.output_dir / "ideas.md"
|
||||
path.write_text(report)
|
||||
return str(path)
|
||||
|
||||
def gaps_report(self) -> str:
|
||||
"""Generate gap analysis report."""
|
||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
||||
gaps = self.db.all_gaps()
|
||||
total_drafts = self.db.count_drafts()
|
||||
|
||||
lines = [
|
||||
"# Gap Analysis: IETF AI/Agent Draft Landscape",
|
||||
f"*Generated {now} — analyzing {total_drafts} drafts*\n",
|
||||
]
|
||||
|
||||
# Group by severity
|
||||
severity_order = {"critical": 0, "high": 1, "medium": 2, "low": 3}
|
||||
gaps.sort(key=lambda g: severity_order.get(g["severity"], 4))
|
||||
|
||||
for i, gap in enumerate(gaps, 1):
|
||||
sev = gap["severity"].upper()
|
||||
lines.extend([
|
||||
f"### {i}. {gap['topic']}",
|
||||
f"**Severity:** {sev} ",
|
||||
f"**Category:** {gap['category'] or 'cross-cutting'} ",
|
||||
f"**Description:** {gap['description']} ",
|
||||
f"**Evidence:** {gap['evidence']}\n",
|
||||
])
|
||||
|
||||
# Summary
|
||||
by_sev: dict[str, int] = defaultdict(int)
|
||||
for g in gaps:
|
||||
by_sev[g["severity"]] += 1
|
||||
lines.append("## Summary by Severity\n")
|
||||
for sev in ["critical", "high", "medium", "low"]:
|
||||
if by_sev[sev]:
|
||||
lines.append(f"- **{sev.title()}:** {by_sev[sev]} gaps")
|
||||
|
||||
report = "\n".join(lines)
|
||||
path = self.output_dir / "gaps.md"
|
||||
path.write_text(report)
|
||||
return str(path)
|
||||
|
||||
1071
src/ietf_analyzer/visualize.py
Normal file
1071
src/ietf_analyzer/visualize.py
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user