Verified Findings Ledger — IETF AI/Agent Landscape Survey
Status of every headline claim, its evidence, and its reliability.
Corpus: clean IETF = source='ietf' AND not false-positive = 524 Internet-Drafts.
Snapshot: data/drafts.db as of 2026-05-23. Verification: deterministic recompute
(Phase 0) + two-model Batch re-rating with Cohen's/weighted κ (Phase 1).
Legend: ✅ verified deterministic · 🟡 LLM-derived, reliable enough to report ·
🔴 LLM-derived, NOT reliable — do not use as a finding.
Corpus & scope
| Claim |
Value |
Status |
| Clean IETF corpus |
524 (597 raw − 73 false-positive) |
✅ |
| All dates ISO-parseable within IETF |
597/597 |
✅ (date problems were ISO/ETSI/ITU, excluded) |
| Temporal span |
2024-01 … 2026-05 |
✅ |
Temporal trend (replaces stale "36×")
| Claim |
Value |
Status |
| Monthly avg Jun24–May25 |
3.7 |
✅ |
| Monthly avg since Jun25 |
38.8 (~10×) |
✅ |
| Peak month |
2026-03 = 106 |
✅ |
| Peak vs typical 2024 month (~3) |
~35× |
✅ (state as peak-vs-baseline, NOT "average growth") |
| Tail (2026-04/05) |
provisional (fetch lag) |
✅ caveat required |
Author / WG structure
| Claim |
Value |
Status |
| Distinct authors |
619 |
✅ |
| Author concentration |
top-10 = 10.9% of drafts (LOW) |
✅ |
| Individual (no-WG) submissions |
456/524 (87%) |
✅ — key "pre-standardization" signal |
| Distinct WGs (group_uri) |
28 |
✅ (acronyms unresolved; optional Datatracker backfill) |
Embedding redundancy (768-d nomic, 100% coverage, 137k pairs)
| Claim |
Value |
Status |
| Cosine mean / median |
0.711 / ~0.711 |
✅ |
| p90 / p99 / max |
0.790 / 0.850 / 1.000 |
✅ |
| Near-duplicate pairs (>0.9) |
125 |
✅ |
| Drafts with ≥1 near-dup |
170 (32.4%) |
✅ — redundancy finding |
| Cosine≈1.0 pairs |
individual↔WG-adopted same doc |
✅ legit, not error |
Category landscape (primary category)
| Claim |
Value |
Status |
| Distribution sums to 524 |
yes; 0 uncategorized |
✅ |
| Multi-category drafts |
92.9% carry >1 |
✅ |
| Top categories |
Identity/auth 141, A2A 108, NetOps 64 |
🟡 |
| Sparsest (neutral) |
Human-agent 5, Other 9, Model-serving 16 |
🟡 |
| Inter-rater κ (primary category) |
Sonnet↔Haiku 0.652, Sonnet↔Prod 0.645 (substantial) |
🟡 reliable enough |
| Confusion = semantic neighbours |
A2A↔NetOps, A2A↔Discovery, Identity↔Other |
🟡 report as boundary fuzziness |
LLM ordinal quality scores — DO NOT report as findings
| Dimension |
weighted κ (Sonnet↔Haiku) |
Status |
| relevance |
0.728 (but 0.234 vs prod) |
🔴 inconsistent |
| maturity |
0.592 |
🟡/🔴 borderline |
| momentum |
0.457 |
🔴 |
| novelty |
0.206 |
🔴 |
| overlap |
0.127 (slight = noise) |
🔴 — invalidates any "overlap N/5" claim |
Headline methodological finding: LLM categorical assignment of standards
documents is substantially reproducible (κ≈0.65); LLM ordinal quality ratings
(novelty, overlap) are not (κ≈0.13–0.21). A landscape survey may report the
category distribution; it must NOT rest on the quality scores.
Artifacts (reproducible)
- scripts/survey-phase0.py → data/reports/survey-phase0.md
- scripts/rerate-intercoder.py → data/rerate/{sonnet,haiku}.jsonl (Batch API, $2.41)
- scripts/survey-kappa.py → data/reports/survey-kappa.md
