feat(survey): add IETF landscape survey (kappa, phase0, rerate), gaps update; bump wimse-ect; gitignore run logs

data/reports/survey-findings.md

@@ -0,0 +1,71 @@
+# Verified Findings Ledger — IETF AI/Agent Landscape Survey
+
+Status of every headline claim, its evidence, and its reliability.
+Corpus: **clean IETF** = `source='ietf'` AND not false-positive = **524 Internet-Drafts**.
+Snapshot: data/drafts.db as of 2026-05-23. Verification: deterministic recompute
+(Phase 0) + two-model Batch re-rating with Cohen's/weighted κ (Phase 1).
+
+Legend: ✅ verified deterministic · 🟡 LLM-derived, reliable enough to report ·
+🔴 LLM-derived, NOT reliable — do not use as a finding.
+
+## Corpus & scope
+| Claim | Value | Status |
+|---|---|---|
+| Clean IETF corpus | 524 (597 raw − 73 false-positive) | ✅ |
+| All dates ISO-parseable within IETF | 597/597 | ✅ (date problems were ISO/ETSI/ITU, excluded) |
+| Temporal span | 2024-01 … 2026-05 | ✅ |
+
+## Temporal trend (replaces stale "36×")
+| Claim | Value | Status |
+|---|---|---|
+| Monthly avg Jun24–May25 | 3.7 | ✅ |
+| Monthly avg since Jun25 | 38.8 (~10×) | ✅ |
+| Peak month | 2026-03 = 106 | ✅ |
+| Peak vs typical 2024 month (~3) | ~35× | ✅ (state as peak-vs-baseline, NOT "average growth") |
+| Tail (2026-04/05) | provisional (fetch lag) | ✅ caveat required |
+
+## Author / WG structure
+| Claim | Value | Status |
+|---|---|---|
+| Distinct authors | 619 | ✅ |
+| Author concentration | top-10 = 10.9% of drafts (LOW) | ✅ |
+| Individual (no-WG) submissions | 456/524 (87%) | ✅ — key "pre-standardization" signal |
+| Distinct WGs (group_uri) | 28 | ✅ (acronyms unresolved; optional Datatracker backfill) |
+
+## Embedding redundancy (768-d nomic, 100% coverage, 137k pairs)
+| Claim | Value | Status |
+|---|---|---|
+| Cosine mean / median | 0.711 / ~0.711 | ✅ |
+| p90 / p99 / max | 0.790 / 0.850 / 1.000 | ✅ |
+| Near-duplicate pairs (>0.9) | 125 | ✅ |
+| Drafts with ≥1 near-dup | 170 (32.4%) | ✅ — redundancy finding |
+| Cosine≈1.0 pairs | individual↔WG-adopted same doc | ✅ legit, not error |
+
+## Category landscape (primary category)
+| Claim | Value | Status |
+|---|---|---|
+| Distribution sums to 524 | yes; 0 uncategorized | ✅ |
+| Multi-category drafts | 92.9% carry >1 | ✅ |
+| Top categories | Identity/auth 141, A2A 108, NetOps 64 | 🟡 |
+| Sparsest (neutral) | Human-agent 5, Other 9, Model-serving 16 | 🟡 |
+| **Inter-rater κ (primary category)** | **Sonnet↔Haiku 0.652, Sonnet↔Prod 0.645 (substantial)** | 🟡 reliable enough |
+| Confusion = semantic neighbours | A2A↔NetOps, A2A↔Discovery, Identity↔Other | 🟡 report as boundary fuzziness |
+
+## LLM ordinal quality scores — DO NOT report as findings
+| Dimension | weighted κ (Sonnet↔Haiku) | Status |
+|---|---|---|
+| relevance | 0.728 (but 0.234 vs prod) | 🔴 inconsistent |
+| maturity | 0.592 | 🟡/🔴 borderline |
+| momentum | 0.457 | 🔴 |
+| novelty | 0.206 | 🔴 |
+| **overlap** | **0.127 (slight = noise)** | 🔴 — invalidates any "overlap N/5" claim |
+
+**Headline methodological finding:** LLM categorical assignment of standards
+documents is substantially reproducible (κ≈0.65); LLM *ordinal quality* ratings
+(novelty, overlap) are not (κ≈0.13–0.21). A landscape survey may report the
+category distribution; it must NOT rest on the quality scores.
+
+## Artifacts (reproducible)
+- scripts/survey-phase0.py → data/reports/survey-phase0.md
+- scripts/rerate-intercoder.py → data/rerate/{sonnet,haiku}.jsonl (Batch API, $2.41)
+- scripts/survey-kappa.py → data/reports/survey-kappa.md