# Verified Findings Ledger — IETF AI/Agent Landscape Survey

Status of every headline claim, its evidence, and its reliability.
Corpus: **clean IETF** = `source='ietf'` AND not false-positive = **524 Internet-Drafts**.
Snapshot: data/drafts.db as of 2026-05-23. Verification: deterministic recompute
(Phase 0) + two-model Batch re-rating with Cohen's/weighted κ (Phase 1).

Legend: ✅ verified deterministic · 🟡 LLM-derived, reliable enough to report ·
🔴 LLM-derived, NOT reliable — do not use as a finding.

## Corpus & scope
| Claim | Value | Status |
|---|---|---|
| Clean IETF corpus | 524 (597 raw − 73 false-positive) | ✅ |
| All dates ISO-parseable within IETF | 597/597 | ✅ (date problems were ISO/ETSI/ITU, excluded) |
| Temporal span | 2024-01 … 2026-05 | ✅ |

## Temporal trend (replaces stale "36×")
| Claim | Value | Status |
|---|---|---|
| Monthly avg Jun24–May25 | 3.7 | ✅ |
| Monthly avg since Jun25 | 38.8 (~10×) | ✅ |
| Peak month | 2026-03 = 106 | ✅ |
| Peak vs typical 2024 month (~3) | ~35× | ✅ (state as peak-vs-baseline, NOT "average growth") |
| Tail (2026-04/05) | provisional (fetch lag) | ✅ caveat required |

## Author / WG structure
| Claim | Value | Status |
|---|---|---|
| Distinct authors | 619 | ✅ |
| Author concentration | top-10 = 10.9% of drafts (LOW) | ✅ |
| Individual (no-WG) submissions | 456/524 (87%) | ✅ — key "pre-standardization" signal |
| Distinct WGs (group_uri) | 28 | ✅ (acronyms unresolved; optional Datatracker backfill) |

## Embedding redundancy (768-d nomic, 100% coverage, 137k pairs)
| Claim | Value | Status |
|---|---|---|
| Cosine mean / median | 0.711 / ~0.711 | ✅ |
| p90 / p99 / max | 0.790 / 0.850 / 1.000 | ✅ |
| Near-duplicate pairs (>0.9) | 125 | ✅ |
| Drafts with ≥1 near-dup | 170 (32.4%) | ✅ — redundancy finding |
| Cosine≈1.0 pairs | individual↔WG-adopted same doc | ✅ legit, not error |

## Category landscape (primary category)
| Claim | Value | Status |
|---|---|---|
| Distribution sums to 524 | yes; 0 uncategorized | ✅ |
| Multi-category drafts | 92.9% carry >1 | ✅ |
| Top categories | Identity/auth 141, A2A 108, NetOps 64 | 🟡 |
| Sparsest (neutral) | Human-agent 5, Other 9, Model-serving 16 | 🟡 |
| **Inter-rater κ (primary category)** | **Sonnet↔Haiku 0.652, Sonnet↔Prod 0.645 (substantial)** | 🟡 reliable enough |
| Confusion = semantic neighbours | A2A↔NetOps, A2A↔Discovery, Identity↔Other | 🟡 report as boundary fuzziness |

## LLM ordinal quality scores — DO NOT report as findings
| Dimension | weighted κ (Sonnet↔Haiku) | Status |
|---|---|---|
| relevance | 0.728 (but 0.234 vs prod) | 🔴 inconsistent |
| maturity | 0.592 | 🟡/🔴 borderline |
| momentum | 0.457 | 🔴 |
| novelty | 0.206 | 🔴 |
| **overlap** | **0.127 (slight = noise)** | 🔴 — invalidates any "overlap N/5" claim |

**Headline methodological finding:** LLM categorical assignment of standards
documents is substantially reproducible (κ≈0.65); LLM *ordinal quality* ratings
(novelty, overlap) are not (κ≈0.13–0.21). A landscape survey may report the
category distribution; it must NOT rest on the quality scores.

## Artifacts (reproducible)
- scripts/survey-phase0.py → data/reports/survey-phase0.md
- scripts/rerate-intercoder.py → data/rerate/{sonnet,haiku}.jsonl (Batch API, $2.41)
- scripts/survey-kappa.py → data/reports/survey-kappa.md