c/ietf-draft-analyzer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(survey): add IETF landscape survey (kappa, phase0, rerate), gaps update; bump wimse-ect; gitignore run logs

Browse Source
This commit is contained in:
Christian Nennemann
2026-05-25 12:35:31 +02:00
parent 89df70a6c0
commit 6e6e0489b8
43 changed files with 11956 additions and 1384 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1019
data/reports/gaps.md
View File

File diff suppressed because it is too large Load Diff

71
data/reports/survey-findings.md Normal file
View File

@@ -0,0 +1,71 @@
# Verified Findings Ledger — IETF AI/Agent Landscape Survey
Status of every headline claim, its evidence, and its reliability.
Corpus: **clean IETF** = `source='ietf'` AND not false-positive = **524 Internet-Drafts**.
Snapshot: data/drafts.db as of 2026-05-23. Verification: deterministic recompute
(Phase 0) + two-model Batch re-rating with Cohen's/weighted κ (Phase 1).
Legend: ✅ verified deterministic · 🟡 LLM-derived, reliable enough to report ·
🔴 LLM-derived, NOT reliable — do not use as a finding.
## Corpus & scope
| Claim | Value | Status |
|---|---|---|
| Clean IETF corpus | 524 (597 raw − 73 false-positive) | ✅ |
| All dates ISO-parseable within IETF | 597/597 | ✅ (date problems were ISO/ETSI/ITU, excluded) |
| Temporal span | 2024-01 … 2026-05 | ✅ |
## Temporal trend (replaces stale "36×")
| Claim | Value | Status |
|---|---|---|
| Monthly avg Jun24–May25 | 3.7 | ✅ |
| Monthly avg since Jun25 | 38.8 (~10×) | ✅ |
| Peak month | 2026-03 = 106 | ✅ |
| Peak vs typical 2024 month (~3) | ~35× | ✅ (state as peak-vs-baseline, NOT "average growth") |
| Tail (2026-04/05) | provisional (fetch lag) | ✅ caveat required |
## Author / WG structure
| Claim | Value | Status |
|---|---|---|
| Distinct authors | 619 | ✅ |
| Author concentration | top-10 = 10.9% of drafts (LOW) | ✅ |
| Individual (no-WG) submissions | 456/524 (87%) | ✅ — key "pre-standardization" signal |
| Distinct WGs (group_uri) | 28 | ✅ (acronyms unresolved; optional Datatracker backfill) |
## Embedding redundancy (768-d nomic, 100% coverage, 137k pairs)
| Claim | Value | Status |
|---|---|---|
| Cosine mean / median | 0.711 / ~0.711 | ✅ |
| p90 / p99 / max | 0.790 / 0.850 / 1.000 | ✅ |
| Near-duplicate pairs (>0.9) | 125 | ✅ |
| Drafts with ≥1 near-dup | 170 (32.4%) | ✅ — redundancy finding |
| Cosine≈1.0 pairs | individual↔WG-adopted same doc | ✅ legit, not error |
## Category landscape (primary category)
| Claim | Value | Status |
|---|---|---|
| Distribution sums to 524 | yes; 0 uncategorized | ✅ |
| Multi-category drafts | 92.9% carry >1 | ✅ |
| Top categories | Identity/auth 141, A2A 108, NetOps 64 | 🟡 |
| Sparsest (neutral) | Human-agent 5, Other 9, Model-serving 16 | 🟡 |
| **Inter-rater κ (primary category)** | **Sonnet↔Haiku 0.652, Sonnet↔Prod 0.645 (substantial)** | 🟡 reliable enough |
| Confusion = semantic neighbours | A2A↔NetOps, A2A↔Discovery, Identity↔Other | 🟡 report as boundary fuzziness |
## LLM ordinal quality scores — DO NOT report as findings
| Dimension | weighted κ (Sonnet↔Haiku) | Status |
|---|---|---|
| relevance | 0.728 (but 0.234 vs prod) | 🔴 inconsistent |
| maturity | 0.592 | 🟡/🔴 borderline |
| momentum | 0.457 | 🔴 |
| novelty | 0.206 | 🔴 |
| **overlap** | **0.127 (slight = noise)** | 🔴 — invalidates any "overlap N/5" claim |
**Headline methodological finding:** LLM categorical assignment of standards
documents is substantially reproducible (κ≈0.65); LLM *ordinal quality* ratings
(novelty, overlap) are not (κ≈0.13–0.21). A landscape survey may report the
category distribution; it must NOT rest on the quality scores.
## Artifacts (reproducible)
- scripts/survey-phase0.py → data/reports/survey-phase0.md
- scripts/rerate-intercoder.py → data/rerate/{sonnet,haiku}.jsonl (Batch API, $2.41)
- scripts/survey-kappa.py → data/reports/survey-kappa.md

80
data/reports/survey-kappa.md Normal file
View File

@@ -0,0 +1,80 @@
# Inter-rater reliability — IETF landscape survey
Corpus: clean IETF (n≈524). Sonnet=524, Haiku=523, prod labels=524.
κ interpretation (Landis & Koch): <0.2 slight, 0.2–0.4 fair, 0.4–0.6 moderate, 0.6–0.8 substantial, >0.8 almost perfect.
## Sonnet (re-rate) vs Haiku (re-rate) — controlled inter-coder (n shared = 523)
**Primary category** (Cohen's κ): κ = 0.652 (substantial), raw agreement 70.8%, n = 517
Most-confused category pairs:
| A | B | count |
|---|---|------:|
| A2A protocols | Autonomous netops | 17 |
| A2A protocols | Agent discovery/reg | 16 |
| A2A protocols | Agent identity/auth | 15 |
| Agent identity/auth | Other AI/agent | 14 |
| Data formats/interop | Other AI/agent | 10 |
| A2A protocols | Data formats/interop | 9 |
| Agent identity/auth | Data formats/interop | 8 |
| Autonomous netops | Other AI/agent | 6 |
**Ordinal dimensions** (quadratic-weighted κ):
| dimension | κ_w | raw agree | n |
|---|---:|---:|---:|
| novelty | 0.206 (fair) | 39.4% | 523 |
| maturity | 0.592 (moderate) | 54.1% | 523 |
| overlap | 0.127 (slight) | 26.4% | 523 |
| momentum | 0.457 (moderate) | 33.8% | 467 |
| relevance | 0.728 (substantial) | 37.7% | 523 |
## Sonnet (re-rate) vs Production labels — drift/stability (n shared = 524)
**Primary category** (Cohen's κ): κ = 0.645 (substantial), raw agreement 69.9%, n = 518
Most-confused category pairs:
| A | B | count |
|---|---|------:|
| Agent identity/auth | Other AI/agent | 18 |
| A2A protocols | Autonomous netops | 15 |
| A2A protocols | Agent identity/auth | 13 |
| Data formats/interop | Other AI/agent | 11 |
| A2A protocols | Data formats/interop | 8 |
| A2A protocols | Agent discovery/reg | 8 |
| Autonomous netops | Other AI/agent | 7 |
| Agent identity/auth | Data formats/interop | 7 |
**Ordinal dimensions** (quadratic-weighted κ):
| dimension | κ_w | raw agree | n |
|---|---:|---:|---:|
| novelty | 0.477 (moderate) | 43.1% | 524 |
| maturity | 0.620 (substantial) | 52.1% | 524 |
| overlap | 0.282 (fair) | 41.8% | 524 |
| momentum | 0.247 (fair) | 32.4% | 524 |
| relevance | 0.234 (fair) | 43.7% | 524 |
## Haiku (re-rate) vs Production labels (n shared = 523)
**Primary category** (Cohen's κ): κ = 0.596 (moderate), raw agreement 65.9%, n = 522
Most-confused category pairs:
| A | B | count |
|---|---|------:|
| A2A protocols | Agent discovery/reg | 17 |
| A2A protocols | Autonomous netops | 16 |
| Agent identity/auth | Data formats/interop | 16 |
| A2A protocols | Agent identity/auth | 14 |
| Autonomous netops | Other AI/agent | 10 |
| Agent identity/auth | Other AI/agent | 9 |
| AI safety/alignment | Agent identity/auth | 9 |
| A2A protocols | Data formats/interop | 8 |
**Ordinal dimensions** (quadratic-weighted κ):
| dimension | κ_w | raw agree | n |
|---|---:|---:|---:|
| novelty | 0.097 (slight) | 13.2% | 523 |
| maturity | 0.412 (moderate) | 37.3% | 523 |
| overlap | 0.087 (slight) | 21.2% | 523 |
| momentum | 0.114 (slight) | 10.1% | 467 |
| relevance | 0.220 (fair) | 34.8% | 523 |

140
data/reports/survey-phase0.md Normal file
View File

@@ -0,0 +1,140 @@
# Survey Phase 0 — Quantitative IETF AI/Agent Draft Landscape
_Generated 2026-05-23 20:49 UTC by `scripts/survey-phase0.py` (deterministic, no LLM API calls)._
**Corpus definition ("clean IETF corpus"):** `source='ietf' AND (ratings.false_positive=0 OR NULL)`, joining `drafts d` with `ratings r` on `d.name=r.draft_name`. **N = 524 drafts.**
## 1. Author / Working-Group Concentration
- Distinct authors across the clean corpus: **619**
- Clean drafts with at least one resolved author in `draft_authors`: **396** of 524 (75.6%); the remainder have no rows in `draft_authors`.
- Total (author, draft) authorship memberships: **1193**
### Top 15 authors by clean-IETF draft count
| # | Author | Drafts |
| --- | --- | --- |
| 1 | Bing Liu | 22 |
| 2 | Nan Geng | 21 |
| 3 | Zhenbin Li | 20 |
| 4 | Qiangzhou Gao | 19 |
| 5 | Xiaotong Shang | 18 |
| 6 | Jianwei Mao | 14 |
| 7 | Guanming Zeng | 13 |
| 8 | Cullen Fluffy Jennings | 11 |
| 9 | Jonathan Rosenberg | 11 |
| 10 | Henk Birkholz | 10 |
| 11 | Qin Wu | 10 |
| 12 | Aijun Wang | 9 |
| 13 | Chongfeng Xie | 8 |
| 14 | Göran Selander | 8 |
| 15 | Luis M. Contreras | 8 |
### Concentration (top 10 authors)
- Top-10 authors account for **159 of 1193 authorship memberships (13.3%)**.
- Top-10 authors appear on **57 of 524 distinct clean drafts (10.9%)** (a draft is counted once even if multiple top-10 authors appear on it).
### Working groups
The `drafts.group` text column is effectively unpopulated for this corpus (only 524 drafts resolve to no/empty/'none' group, and the column has no real WG names). The meaningful grouping signal is `drafts.group_uri` (Datatracker group IDs).
- Drafts with no working group (group/1027 "individual" pseudo-group or null `group_uri`): **456** of 524.
- Distinct `group_uri` values: **28**.
Top 15 groups by `group_uri` (clean-IETF draft count):
| # | Group (Datatracker ID / label) | Drafts |
| --- | --- | --- |
| 1 | (individual / no WG) | 456 |
| 2 | group/2440 | 9 |
| 3 | group/2249 | 8 |
| 4 | group/2231 | 7 |
| 5 | group/2164 | 6 |
| 6 | group/1674 | 4 |
| 7 | group/1326 | 4 |
| 8 | group/1452 | 3 |
| 9 | group/38 | 2 |
| 10 | group/2430 | 2 |
| 11 | group/2380 | 2 |
| 12 | group/2317 | 2 |
| 13 | group/1956 | 2 |
| 14 | group/1921 | 2 |
| 15 | group/1718 | 2 |
> Note: ID 1027 is the Datatracker "none" pseudo-group for individually submitted drafts; it is not a real working group. Real WG/RG IDs are the remaining rows.
## 2. Embedding Overlap / Redundancy
- Embedding model: `nomic-embed-text`, dimension **768**, stored as float32 BLOB.
- Drafts embedded (clean corpus): **524** (100% coverage); pairwise comparisons: **137,026**.
### Pairwise cosine similarity (off-diagonal)
| Mean | Median | p90 | p99 | Max |
| --- | --- | --- | --- | --- |
| 0.7106 | 0.7110 | 0.7901 | 0.8497 | 1.0000 |
- Near-duplicate pairs (cosine > 0.9): **125**.
- Drafts with at least one near-duplicate (cosine > 0.9): **170** of 524 (32.4%).
### Top 20 most-similar draft pairs
| # | Cosine | Draft A / Draft B | Title A / Title B |
| --- | --- | --- | --- |
| 1 | 1.0000 | `draft-howard-rats-coserv`<br>`draft-ietf-rats-coserv` | Concise Selector for Endorsements and Reference Values<br>Concise Selector for Endorsements and Reference Values |
| 2 | 1.0000 | `draft-ietf-savnet-inter-domain-architecture`<br>`draft-wu-savnet-inter-domain-architecture` | Inter-domain Source Address Validation (SAVNET) Architecture<br>Inter-domain Source Address Validation (SAVNET) Architecture |
| 3 | 1.0000 | `draft-fv-rats-ear`<br>`draft-ietf-rats-ear` | EAT Attestation Results<br>EAT Attestation Results |
| 4 | 1.0000 | `draft-ietf-rats-evidence-trans`<br>`draft-smith-rats-evidence-trans` | Evidence Transformations<br>Evidence Transformations |
| 5 | 0.9993 | `draft-rosenberg-aiproto`<br>`draft-rosenberg-aiproto-nact` | Normalized API for AI Agents Calling Tools (N-ACT)<br>Normalized API for AI Agents Calling Tools (N-ACT) |
| 6 | 0.9986 | `draft-lake-pocero-authkem-ikr-edhoc`<br>`draft-pocero-authkem-ikr-edhoc` | KEM-based Authentication for EDHOC in Initiator-Known Responder (IKR) <br>KEM-based Authentication for EDHOC in Initiator-Known Responder (IKR) |
| 7 | 0.9976 | `draft-dunbar-onions-ac-te-applicability`<br>`draft-dunbar-onsen-ac-te-applicability` | Applying Attachmet Circuit and Traffic Engineering YANG Data Model to <br>Applying Attachmet Circuit and Traffic Engineering YANG Data Model to |
| 8 | 0.9971 | `draft-dunbar-neotec-ac-pe2pe-ucmp-applicability`<br>`draft-dunbar-onsen-ac-pe2pe-ucmp-applicability` | Applying Attachmet Circuit and PE2PE YANG Data Model to dynamic polici<br>Applying Attachmet Circuit and PE2PE YANG Data Model to dynamic polici |
| 9 | 0.9968 | `draft-ar-emu-hybrid-pqc-eapaka`<br>`draft-ar-emu-pqc-eapaka` | Enhancing Security in EAP-AKA' with Hybrid Post-Quantum Cryptography<br>Enhancing Security in EAP-AKA' with Hybrid Post-Quantum Cryptography |
| 10 | 0.9967 | `draft-aft-ai-traffic`<br>`draft-ai-traffic` | Handling inter-DC/Edge AI-related network traffic: Problem statement<br>Handling inter-DC/Edge AI-related network traffic: Problem statement |
| 11 | 0.9967 | `draft-ahn-nmrg-5g-security-i2nsf-framework`<br>`draft-ahn-opsawg-5g-security-i2nsf-framework` | An Integrated Security Service System for 5G Networks using an I2NSF F<br>An Integrated Security Service System for 5G Networks using an I2NSF F |
| 12 | 0.9961 | `draft-liu-saag-zt-problem-statement`<br>`draft-liu-ztcpp-zt-problem-statement` | Zero trust standards in IETF: use cases and problem statement<br>Zero trust standards in IETF: use cases and problem statement |
| 13 | 0.9954 | `draft-zw-nmrg-mcp-network-mgmt`<br>`draft-zw-rtgwg-mcp-network-mgmt` | Model Context Protocol (MCP) Extensions for Network Equipment Manageme<br>Model Context Protocol (MCP) Extensions for Network Equipment Manageme |
| 14 | 0.9952 | `draft-dunbar-neotec-ac-te-applicability`<br>`draft-dunbar-onsen-ac-te-applicability` | Applying Attachmet Circuit and Traffic Engineering YANG Data Model to <br>Applying Attachmet Circuit and Traffic Engineering YANG Data Model to |
| 15 | 0.9948 | `draft-ietf-emu-pqc-eapaka`<br>`draft-ra-emu-pqc-eapaka` | Post-Quantum Key Encapsulation Mechanisms (PQ KEMs) in EAP-AKA prime<br>Post-Quantum Key Encapsulation Mechanisms (PQ KEMs) in EAP-AKA prime |
| 16 | 0.9946 | `draft-sun-zhang-iaip`<br>`draft-sz-dmsc-iaip` | Intent-based Agent Interconnection Protocol at Agent Gateway<br>Intent-based Agent Interconnection Protocol at Agent Gateway |
| 17 | 0.9944 | `draft-ar-emu-hybrid-pqc-eapaka`<br>`draft-ietf-emu-hybrid-pqc-eapaka` | Enhancing Security in EAP-AKA' with Hybrid Post-Quantum Cryptography<br>Enhancing Security in EAP-AKA' with Hybrid Post-Quantum Cryptography |
| 18 | 0.9942 | `draft-intellinode-cats-in-network-scheduling`<br>`draft-li-cats-intellinode-network-scheduling` | IntelliNode: In-Network Intelligent Scheduling Extensions for CATS<br>IntelliNode: In-Network Intelligent Scheduling Extensions for CATS |
| 19 | 0.9941 | `draft-ar-emu-pqc-eapaka`<br>`draft-ietf-emu-hybrid-pqc-eapaka` | Enhancing Security in EAP-AKA' with Hybrid Post-Quantum Cryptography<br>Enhancing Security in EAP-AKA' with Hybrid Post-Quantum Cryptography |
| 20 | 0.9936 | `draft-rosenberg-aiproto-cheq`<br>`draft-rosenberg-cheq` | CHEQ: A Protocol for Confirmation AI Agent Decisions with Human in the<br>CHEQ: A Protocol for Confirmation AI Agent Decisions with Human in the |
## 3. Category Coverage Map
Primary category = first element of the `ratings.categories` JSON array. Sum check: **524 = 524** (OK).
- Drafts carrying more than one category: **487** of 524 (92.9%).
- Drafts with no category (uncategorized): **0**.
### Primary-category distribution + mean scores
Scores are 1-5 integers from `ratings`. `composite` = mean of the 5 dimensions (novelty, maturity, overlap, momentum, relevance) per draft, averaged over the category.
| Category | N | Share | Relevance | Composite | Novelty | Maturity | Overlap | Momentum |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Agent identity/auth | 141 | 26.9% | 4.26 | 3.38 | 3.64 | 3.28 | 2.43 | 3.29 |
| A2A protocols | 108 | 20.6% | 4.18 | 3.32 | 3.67 | 3.1 | 2.48 | 3.18 |
| Autonomous netops | 64 | 12.2% | 3.86 | 3.13 | 3.23 | 2.73 | 2.83 | 3.02 |
| ML traffic mgmt | 48 | 9.2% | 3.85 | 3.09 | 3.1 | 2.71 | 2.94 | 2.83 |
| Data formats/interop | 44 | 8.4% | 3.66 | 3.14 | 3.18 | 3.55 | 2.11 | 3.2 |
| Agent discovery/reg | 35 | 6.7% | 4.2 | 3.24 | 3.6 | 3.09 | 2.31 | 3 |
| Policy/governance | 30 | 5.7% | 4.1 | 3.25 | 3.37 | 2.97 | 2.53 | 3.27 |
| AI safety/alignment | 24 | 4.6% | 4.5 | 3.38 | 4.17 | 3.17 | 1.75 | 3.33 |
| Model serving/inference | 16 | 3.1% | 4.19 | 3.31 | 3.31 | 3.19 | 2.62 | 3.25 |
| Other AI/agent | 9 | 1.7% | 3.56 | 3.18 | 3.22 | 3.44 | 2.22 | 3.44 |
| Human-agent interaction | 5 | 1.0% | 4.4 | 3.48 | 3.8 | 3.4 | 2 | 3.8 |
### Three sparsest categories (descriptive)
Listed neutrally as the categories with the fewest drafts in this corpus:
| Category | N | Relevance | Composite |
| --- | --- | --- | --- |
| Human-agent interaction | 5 | 4.4 | 3.48 |
| Other AI/agent | 9 | 3.56 | 3.18 |
| Model serving/inference | 16 | 4.19 | 3.31 |