# Verified IETF AI/Agent Landscape Survey — Paper Plan

**Decision (2026-05-23):** Neutral, citable landscape survey. IETF-only corpus. Full
inter-rater verification (Cohen's Kappa) + systematic coverage validation. NO ECT advocacy.

## Corpus (verified, deterministic)

- `source='ietf'`: 597 docs. Minus 73 false-positive flags → **524 clean IETF Internet-Drafts**.
- All 597 have ISO-parseable `time` (the date-quality problem was confined to ISO/ETSI/ITU — excluded here).
- All rated. 591 have full_text.

## The honest methodological facts (must be in the paper, not hidden)

1. **Ratings are abstract-only** (`analyzer.py:298`, `abstract[:2000]`), not full-text. Documented limitation.
2. **Categories / gap-scores are LLM-generated** (Sonnet `claude-sonnet-4-20250514`, cheap path Haiku 4.5).
3. **Self-documented limitations** already in `analyzer.py` header (lines 42–147) + `data/reports/methodology.md`.

## Phase 0 — Deterministic foundation (FREE, local) ✅ partly done

- [x] IETF-only corpus size + false-positive removal → 524
- [x] Category distribution (clean): 141 identity, 108 A2A, 64 netops, 48 ML-traffic, 44 data-formats,
      35 discovery, 30 policy, 24 safety, 16 model-serving, 9 other, 5 human-agent
- [ ] Monthly submission curve from clean ISO dates → honest growth statement (replace stale "36×")
- [ ] Working-group / author concentration (who writes the drafts)
- [ ] Embedding-based overlap/similarity matrix + clustering (Ollama, local) → "redundancy" finding
- [ ] Coverage map: which categories are dense vs sparse (neutral framing, no ECT)

## Phase 1 — Inter-rater verification (CHEAP API, ~$3–9; needs explicit go)

- Re-rate all 524 abstracts with a **second independent model** (Haiku) using the pinned prompt.
- Compare against existing labels (model A) → **Cohen's Kappa** per dimension + on category assignment.
- Stratified manual spot-check (~50 drafts) by hand → human-vs-LLM agreement.
- Report kappa + confusion matrix honestly. If kappa is low on some categories, that IS a finding.
- Cost optimisations: Batch API (50% off), prompt-cache the rubric (90% off cached input).

## Phase 2 — Coverage validation (FREE/cheap)

- For each category, embedding nearest-neighbour + keyword counter-search to confirm sparse areas
  are genuinely sparse (not a classification artefact). Reframed neutrally (NOT "ECT gap").

## Phase 3 — Write-up

- New survey paper (separate from ECT paper). Reproducible: ship queries + corpus snapshot hash.
- Sections: corpus & method, category landscape, temporal trends, author/WG structure,
  redundancy/overlap, reliability (kappa), limitations, related work.

## Cost summary

- Phase 0/2: €0 (local SQLite + Ollama).
- Phase 1: ~$3–9 total (abstract-only keeps it tiny). Quantify exact tokens before running.