81 lines
3.0 KiB
Markdown
81 lines
3.0 KiB
Markdown
# Inter-rater reliability — IETF landscape survey
|
||
|
||
Corpus: clean IETF (n≈524). Sonnet=524, Haiku=523, prod labels=524.
|
||
|
||
κ interpretation (Landis & Koch): <0.2 slight, 0.2–0.4 fair, 0.4–0.6 moderate, 0.6–0.8 substantial, >0.8 almost perfect.
|
||
|
||
## Sonnet (re-rate) vs Haiku (re-rate) — controlled inter-coder (n shared = 523)
|
||
**Primary category** (Cohen's κ): κ = 0.652 (substantial), raw agreement 70.8%, n = 517
|
||
|
||
Most-confused category pairs:
|
||
| A | B | count |
|
||
|---|---|------:|
|
||
| A2A protocols | Autonomous netops | 17 |
|
||
| A2A protocols | Agent discovery/reg | 16 |
|
||
| A2A protocols | Agent identity/auth | 15 |
|
||
| Agent identity/auth | Other AI/agent | 14 |
|
||
| Data formats/interop | Other AI/agent | 10 |
|
||
| A2A protocols | Data formats/interop | 9 |
|
||
| Agent identity/auth | Data formats/interop | 8 |
|
||
| Autonomous netops | Other AI/agent | 6 |
|
||
|
||
**Ordinal dimensions** (quadratic-weighted κ):
|
||
|
||
| dimension | κ_w | raw agree | n |
|
||
|---|---:|---:|---:|
|
||
| novelty | 0.206 (fair) | 39.4% | 523 |
|
||
| maturity | 0.592 (moderate) | 54.1% | 523 |
|
||
| overlap | 0.127 (slight) | 26.4% | 523 |
|
||
| momentum | 0.457 (moderate) | 33.8% | 467 |
|
||
| relevance | 0.728 (substantial) | 37.7% | 523 |
|
||
|
||
## Sonnet (re-rate) vs Production labels — drift/stability (n shared = 524)
|
||
**Primary category** (Cohen's κ): κ = 0.645 (substantial), raw agreement 69.9%, n = 518
|
||
|
||
Most-confused category pairs:
|
||
| A | B | count |
|
||
|---|---|------:|
|
||
| Agent identity/auth | Other AI/agent | 18 |
|
||
| A2A protocols | Autonomous netops | 15 |
|
||
| A2A protocols | Agent identity/auth | 13 |
|
||
| Data formats/interop | Other AI/agent | 11 |
|
||
| A2A protocols | Data formats/interop | 8 |
|
||
| A2A protocols | Agent discovery/reg | 8 |
|
||
| Autonomous netops | Other AI/agent | 7 |
|
||
| Agent identity/auth | Data formats/interop | 7 |
|
||
|
||
**Ordinal dimensions** (quadratic-weighted κ):
|
||
|
||
| dimension | κ_w | raw agree | n |
|
||
|---|---:|---:|---:|
|
||
| novelty | 0.477 (moderate) | 43.1% | 524 |
|
||
| maturity | 0.620 (substantial) | 52.1% | 524 |
|
||
| overlap | 0.282 (fair) | 41.8% | 524 |
|
||
| momentum | 0.247 (fair) | 32.4% | 524 |
|
||
| relevance | 0.234 (fair) | 43.7% | 524 |
|
||
|
||
## Haiku (re-rate) vs Production labels (n shared = 523)
|
||
**Primary category** (Cohen's κ): κ = 0.596 (moderate), raw agreement 65.9%, n = 522
|
||
|
||
Most-confused category pairs:
|
||
| A | B | count |
|
||
|---|---|------:|
|
||
| A2A protocols | Agent discovery/reg | 17 |
|
||
| A2A protocols | Autonomous netops | 16 |
|
||
| Agent identity/auth | Data formats/interop | 16 |
|
||
| A2A protocols | Agent identity/auth | 14 |
|
||
| Autonomous netops | Other AI/agent | 10 |
|
||
| Agent identity/auth | Other AI/agent | 9 |
|
||
| AI safety/alignment | Agent identity/auth | 9 |
|
||
| A2A protocols | Data formats/interop | 8 |
|
||
|
||
**Ordinal dimensions** (quadratic-weighted κ):
|
||
|
||
| dimension | κ_w | raw agree | n |
|
||
|---|---:|---:|---:|
|
||
| novelty | 0.097 (slight) | 13.2% | 523 |
|
||
| maturity | 0.412 (moderate) | 37.3% | 523 |
|
||
| overlap | 0.087 (slight) | 21.2% | 523 |
|
||
| momentum | 0.114 (slight) | 10.1% | 467 |
|
||
| relevance | 0.220 (fair) | 34.8% | 523 |
|