ietf-draft-analyzer/docs/plans/landscape-survey-paper.md

Verified IETF AI/Agent Landscape Survey — Paper Plan

Decision (2026-05-23): Neutral, citable landscape survey. IETF-only corpus. Full inter-rater verification (Cohen's Kappa) + systematic coverage validation. NO ECT advocacy.

Corpus (verified, deterministic)

• source='ietf': 597 docs. Minus 73 false-positive flags → 524 clean IETF Internet-Drafts.
• All 597 have ISO-parseable time (the date-quality problem was confined to ISO/ETSI/ITU — excluded here).
• All rated. 591 have full_text.

The honest methodological facts (must be in the paper, not hidden)

1. Ratings are abstract-only (analyzer.py:298, abstract[:2000]), not full-text. Documented limitation.
2. Categories / gap-scores are LLM-generated (Sonnet claude-sonnet-4-20250514, cheap path Haiku 4.5).
3. Self-documented limitations already in analyzer.py header (lines 42–147) + data/reports/methodology.md.

Phase 0 — Deterministic foundation (FREE, local) ✅ partly done

• IETF-only corpus size + false-positive removal → 524
• Category distribution (clean): 141 identity, 108 A2A, 64 netops, 48 ML-traffic, 44 data-formats, 35 discovery, 30 policy, 24 safety, 16 model-serving, 9 other, 5 human-agent
• Monthly submission curve from clean ISO dates → honest growth statement (replace stale "36×")
• Working-group / author concentration (who writes the drafts)
• Embedding-based overlap/similarity matrix + clustering (Ollama, local) → "redundancy" finding
• Coverage map: which categories are dense vs sparse (neutral framing, no ECT)

Phase 1 — Inter-rater verification (CHEAP API, ~$3–9; needs explicit go)

• Re-rate all 524 abstracts with a second independent model (Haiku) using the pinned prompt.
• Compare against existing labels (model A) → Cohen's Kappa per dimension + on category assignment.
• Stratified manual spot-check (~50 drafts) by hand → human-vs-LLM agreement.
• Report kappa + confusion matrix honestly. If kappa is low on some categories, that IS a finding.
• Cost optimisations: Batch API (50% off), prompt-cache the rubric (90% off cached input).

Phase 2 — Coverage validation (FREE/cheap)

• For each category, embedding nearest-neighbour + keyword counter-search to confirm sparse areas are genuinely sparse (not a classification artefact). Reframed neutrally (NOT "ECT gap").

Phase 3 — Write-up

• New survey paper (separate from ECT paper). Reproducible: ship queries + corpus snapshot hash.
• Sections: corpus & method, category landscape, temporal trends, author/WG structure, redundancy/overlap, reliability (kappa), limitations, related work.

Cost summary

• Phase 0/2: €0 (local SQLite + Ollama).
• Phase 1: ~$3–9 total (abstract-only keeps it tiny). Quantify exact tokens before running.