72 lines
3.4 KiB
Markdown
72 lines
3.4 KiB
Markdown
# Verified Findings Ledger — IETF AI/Agent Landscape Survey
|
||
|
||
Status of every headline claim, its evidence, and its reliability.
|
||
Corpus: **clean IETF** = `source='ietf'` AND not false-positive = **524 Internet-Drafts**.
|
||
Snapshot: data/drafts.db as of 2026-05-23. Verification: deterministic recompute
|
||
(Phase 0) + two-model Batch re-rating with Cohen's/weighted κ (Phase 1).
|
||
|
||
Legend: ✅ verified deterministic · 🟡 LLM-derived, reliable enough to report ·
|
||
🔴 LLM-derived, NOT reliable — do not use as a finding.
|
||
|
||
## Corpus & scope
|
||
| Claim | Value | Status |
|
||
|---|---|---|
|
||
| Clean IETF corpus | 524 (597 raw − 73 false-positive) | ✅ |
|
||
| All dates ISO-parseable within IETF | 597/597 | ✅ (date problems were ISO/ETSI/ITU, excluded) |
|
||
| Temporal span | 2024-01 … 2026-05 | ✅ |
|
||
|
||
## Temporal trend (replaces stale "36×")
|
||
| Claim | Value | Status |
|
||
|---|---|---|
|
||
| Monthly avg Jun24–May25 | 3.7 | ✅ |
|
||
| Monthly avg since Jun25 | 38.8 (~10×) | ✅ |
|
||
| Peak month | 2026-03 = 106 | ✅ |
|
||
| Peak vs typical 2024 month (~3) | ~35× | ✅ (state as peak-vs-baseline, NOT "average growth") |
|
||
| Tail (2026-04/05) | provisional (fetch lag) | ✅ caveat required |
|
||
|
||
## Author / WG structure
|
||
| Claim | Value | Status |
|
||
|---|---|---|
|
||
| Distinct authors | 619 | ✅ |
|
||
| Author concentration | top-10 = 10.9% of drafts (LOW) | ✅ |
|
||
| Individual (no-WG) submissions | 456/524 (87%) | ✅ — key "pre-standardization" signal |
|
||
| Distinct WGs (group_uri) | 28 | ✅ (acronyms unresolved; optional Datatracker backfill) |
|
||
|
||
## Embedding redundancy (768-d nomic, 100% coverage, 137k pairs)
|
||
| Claim | Value | Status |
|
||
|---|---|---|
|
||
| Cosine mean / median | 0.711 / ~0.711 | ✅ |
|
||
| p90 / p99 / max | 0.790 / 0.850 / 1.000 | ✅ |
|
||
| Near-duplicate pairs (>0.9) | 125 | ✅ |
|
||
| Drafts with ≥1 near-dup | 170 (32.4%) | ✅ — redundancy finding |
|
||
| Cosine≈1.0 pairs | individual↔WG-adopted same doc | ✅ legit, not error |
|
||
|
||
## Category landscape (primary category)
|
||
| Claim | Value | Status |
|
||
|---|---|---|
|
||
| Distribution sums to 524 | yes; 0 uncategorized | ✅ |
|
||
| Multi-category drafts | 92.9% carry >1 | ✅ |
|
||
| Top categories | Identity/auth 141, A2A 108, NetOps 64 | 🟡 |
|
||
| Sparsest (neutral) | Human-agent 5, Other 9, Model-serving 16 | 🟡 |
|
||
| **Inter-rater κ (primary category)** | **Sonnet↔Haiku 0.652, Sonnet↔Prod 0.645 (substantial)** | 🟡 reliable enough |
|
||
| Confusion = semantic neighbours | A2A↔NetOps, A2A↔Discovery, Identity↔Other | 🟡 report as boundary fuzziness |
|
||
|
||
## LLM ordinal quality scores — DO NOT report as findings
|
||
| Dimension | weighted κ (Sonnet↔Haiku) | Status |
|
||
|---|---|---|
|
||
| relevance | 0.728 (but 0.234 vs prod) | 🔴 inconsistent |
|
||
| maturity | 0.592 | 🟡/🔴 borderline |
|
||
| momentum | 0.457 | 🔴 |
|
||
| novelty | 0.206 | 🔴 |
|
||
| **overlap** | **0.127 (slight = noise)** | 🔴 — invalidates any "overlap N/5" claim |
|
||
|
||
**Headline methodological finding:** LLM categorical assignment of standards
|
||
documents is substantially reproducible (κ≈0.65); LLM *ordinal quality* ratings
|
||
(novelty, overlap) are not (κ≈0.13–0.21). A landscape survey may report the
|
||
category distribution; it must NOT rest on the quality scores.
|
||
|
||
## Artifacts (reproducible)
|
||
- scripts/survey-phase0.py → data/reports/survey-phase0.md
|
||
- scripts/rerate-intercoder.py → data/rerate/{sonnet,haiku}.jsonl (Batch API, $2.41)
|
||
- scripts/survey-kappa.py → data/reports/survey-kappa.md
|