feat(survey): add IETF landscape survey (kappa, phase0, rerate), gaps update; bump wimse-ect; gitignore run logs

docs/plans/landscape-survey-paper.md

@@ -0,0 +1,50 @@
+# Verified IETF AI/Agent Landscape Survey — Paper Plan
+
+**Decision (2026-05-23):** Neutral, citable landscape survey. IETF-only corpus. Full
+inter-rater verification (Cohen's Kappa) + systematic coverage validation. NO ECT advocacy.
+
+## Corpus (verified, deterministic)
+
+- `source='ietf'`: 597 docs. Minus 73 false-positive flags → **524 clean IETF Internet-Drafts**.
+- All 597 have ISO-parseable `time` (the date-quality problem was confined to ISO/ETSI/ITU — excluded here).
+- All rated. 591 have full_text.
+
+## The honest methodological facts (must be in the paper, not hidden)
+
+1. **Ratings are abstract-only** (`analyzer.py:298`, `abstract[:2000]`), not full-text. Documented limitation.
+2. **Categories / gap-scores are LLM-generated** (Sonnet `claude-sonnet-4-20250514`, cheap path Haiku 4.5).
+3. **Self-documented limitations** already in `analyzer.py` header (lines 42–147) + `data/reports/methodology.md`.
+
+## Phase 0 — Deterministic foundation (FREE, local) ✅ partly done
+
+- [x] IETF-only corpus size + false-positive removal → 524
+- [x] Category distribution (clean): 141 identity, 108 A2A, 64 netops, 48 ML-traffic, 44 data-formats,
+      35 discovery, 30 policy, 24 safety, 16 model-serving, 9 other, 5 human-agent
+- [ ] Monthly submission curve from clean ISO dates → honest growth statement (replace stale "36×")
+- [ ] Working-group / author concentration (who writes the drafts)
+- [ ] Embedding-based overlap/similarity matrix + clustering (Ollama, local) → "redundancy" finding
+- [ ] Coverage map: which categories are dense vs sparse (neutral framing, no ECT)
+
+## Phase 1 — Inter-rater verification (CHEAP API, ~$3–9; needs explicit go)
+
+- Re-rate all 524 abstracts with a **second independent model** (Haiku) using the pinned prompt.
+- Compare against existing labels (model A) → **Cohen's Kappa** per dimension + on category assignment.
+- Stratified manual spot-check (~50 drafts) by hand → human-vs-LLM agreement.
+- Report kappa + confusion matrix honestly. If kappa is low on some categories, that IS a finding.
+- Cost optimisations: Batch API (50% off), prompt-cache the rubric (90% off cached input).
+
+## Phase 2 — Coverage validation (FREE/cheap)
+
+- For each category, embedding nearest-neighbour + keyword counter-search to confirm sparse areas
+  are genuinely sparse (not a classification artefact). Reframed neutrally (NOT "ECT gap").
+
+## Phase 3 — Write-up
+
+- New survey paper (separate from ECT paper). Reproducible: ship queries + corpus snapshot hash.
+- Sections: corpus & method, category landscape, temporal trends, author/WG structure,
+  redundancy/overlap, reliability (kappa), limitations, related work.
+
+## Cost summary
+
+- Phase 0/2: €0 (local SQLite + Ollama).
+- Phase 1: ~$3–9 total (abstract-only keeps it tiny). Quantify exact tokens before running.