51 lines
2.8 KiB
Markdown
51 lines
2.8 KiB
Markdown
# Verified IETF AI/Agent Landscape Survey — Paper Plan
|
||
|
||
**Decision (2026-05-23):** Neutral, citable landscape survey. IETF-only corpus. Full
|
||
inter-rater verification (Cohen's Kappa) + systematic coverage validation. NO ECT advocacy.
|
||
|
||
## Corpus (verified, deterministic)
|
||
|
||
- `source='ietf'`: 597 docs. Minus 73 false-positive flags → **524 clean IETF Internet-Drafts**.
|
||
- All 597 have ISO-parseable `time` (the date-quality problem was confined to ISO/ETSI/ITU — excluded here).
|
||
- All rated. 591 have full_text.
|
||
|
||
## The honest methodological facts (must be in the paper, not hidden)
|
||
|
||
1. **Ratings are abstract-only** (`analyzer.py:298`, `abstract[:2000]`), not full-text. Documented limitation.
|
||
2. **Categories / gap-scores are LLM-generated** (Sonnet `claude-sonnet-4-20250514`, cheap path Haiku 4.5).
|
||
3. **Self-documented limitations** already in `analyzer.py` header (lines 42–147) + `data/reports/methodology.md`.
|
||
|
||
## Phase 0 — Deterministic foundation (FREE, local) ✅ partly done
|
||
|
||
- [x] IETF-only corpus size + false-positive removal → 524
|
||
- [x] Category distribution (clean): 141 identity, 108 A2A, 64 netops, 48 ML-traffic, 44 data-formats,
|
||
35 discovery, 30 policy, 24 safety, 16 model-serving, 9 other, 5 human-agent
|
||
- [ ] Monthly submission curve from clean ISO dates → honest growth statement (replace stale "36×")
|
||
- [ ] Working-group / author concentration (who writes the drafts)
|
||
- [ ] Embedding-based overlap/similarity matrix + clustering (Ollama, local) → "redundancy" finding
|
||
- [ ] Coverage map: which categories are dense vs sparse (neutral framing, no ECT)
|
||
|
||
## Phase 1 — Inter-rater verification (CHEAP API, ~$3–9; needs explicit go)
|
||
|
||
- Re-rate all 524 abstracts with a **second independent model** (Haiku) using the pinned prompt.
|
||
- Compare against existing labels (model A) → **Cohen's Kappa** per dimension + on category assignment.
|
||
- Stratified manual spot-check (~50 drafts) by hand → human-vs-LLM agreement.
|
||
- Report kappa + confusion matrix honestly. If kappa is low on some categories, that IS a finding.
|
||
- Cost optimisations: Batch API (50% off), prompt-cache the rubric (90% off cached input).
|
||
|
||
## Phase 2 — Coverage validation (FREE/cheap)
|
||
|
||
- For each category, embedding nearest-neighbour + keyword counter-search to confirm sparse areas
|
||
are genuinely sparse (not a classification artefact). Reframed neutrally (NOT "ECT gap").
|
||
|
||
## Phase 3 — Write-up
|
||
|
||
- New survey paper (separate from ECT paper). Reproducible: ship queries + corpus snapshot hash.
|
||
- Sections: corpus & method, category landscape, temporal trends, author/WG structure,
|
||
redundancy/overlap, reliability (kappa), limitations, related work.
|
||
|
||
## Cost summary
|
||
|
||
- Phase 0/2: €0 (local SQLite + Ollama).
|
||
- Phase 1: ~$3–9 total (abstract-only keeps it tiny). Quantify exact tokens before running.
|