3.0 KiB
3.0 KiB
Inter-rater reliability — IETF landscape survey
Corpus: clean IETF (n≈524). Sonnet=524, Haiku=523, prod labels=524.
κ interpretation (Landis & Koch): <0.2 slight, 0.2–0.4 fair, 0.4–0.6 moderate, 0.6–0.8 substantial, >0.8 almost perfect.
Sonnet (re-rate) vs Haiku (re-rate) — controlled inter-coder (n shared = 523)
Primary category (Cohen's κ): κ = 0.652 (substantial), raw agreement 70.8%, n = 517
Most-confused category pairs:
| A | B | count |
|---|---|---|
| A2A protocols | Autonomous netops | 17 |
| A2A protocols | Agent discovery/reg | 16 |
| A2A protocols | Agent identity/auth | 15 |
| Agent identity/auth | Other AI/agent | 14 |
| Data formats/interop | Other AI/agent | 10 |
| A2A protocols | Data formats/interop | 9 |
| Agent identity/auth | Data formats/interop | 8 |
| Autonomous netops | Other AI/agent | 6 |
Ordinal dimensions (quadratic-weighted κ):
| dimension | κ_w | raw agree | n |
|---|---|---|---|
| novelty | 0.206 (fair) | 39.4% | 523 |
| maturity | 0.592 (moderate) | 54.1% | 523 |
| overlap | 0.127 (slight) | 26.4% | 523 |
| momentum | 0.457 (moderate) | 33.8% | 467 |
| relevance | 0.728 (substantial) | 37.7% | 523 |
Sonnet (re-rate) vs Production labels — drift/stability (n shared = 524)
Primary category (Cohen's κ): κ = 0.645 (substantial), raw agreement 69.9%, n = 518
Most-confused category pairs:
| A | B | count |
|---|---|---|
| Agent identity/auth | Other AI/agent | 18 |
| A2A protocols | Autonomous netops | 15 |
| A2A protocols | Agent identity/auth | 13 |
| Data formats/interop | Other AI/agent | 11 |
| A2A protocols | Data formats/interop | 8 |
| A2A protocols | Agent discovery/reg | 8 |
| Autonomous netops | Other AI/agent | 7 |
| Agent identity/auth | Data formats/interop | 7 |
Ordinal dimensions (quadratic-weighted κ):
| dimension | κ_w | raw agree | n |
|---|---|---|---|
| novelty | 0.477 (moderate) | 43.1% | 524 |
| maturity | 0.620 (substantial) | 52.1% | 524 |
| overlap | 0.282 (fair) | 41.8% | 524 |
| momentum | 0.247 (fair) | 32.4% | 524 |
| relevance | 0.234 (fair) | 43.7% | 524 |
Haiku (re-rate) vs Production labels (n shared = 523)
Primary category (Cohen's κ): κ = 0.596 (moderate), raw agreement 65.9%, n = 522
Most-confused category pairs:
| A | B | count |
|---|---|---|
| A2A protocols | Agent discovery/reg | 17 |
| A2A protocols | Autonomous netops | 16 |
| Agent identity/auth | Data formats/interop | 16 |
| A2A protocols | Agent identity/auth | 14 |
| Autonomous netops | Other AI/agent | 10 |
| Agent identity/auth | Other AI/agent | 9 |
| AI safety/alignment | Agent identity/auth | 9 |
| A2A protocols | Data formats/interop | 8 |
Ordinal dimensions (quadratic-weighted κ):
| dimension | κ_w | raw agree | n |
|---|---|---|---|
| novelty | 0.097 (slight) | 13.2% | 523 |
| maturity | 0.412 (moderate) | 37.3% | 523 |
| overlap | 0.087 (slight) | 21.2% | 523 |
| momentum | 0.114 (slight) | 10.1% | 467 |
| relevance | 0.220 (fair) | 34.8% | 523 |