# Inter-rater reliability — IETF landscape survey

Corpus: clean IETF (n≈524). Sonnet=524, Haiku=523, prod labels=524.

κ interpretation (Landis & Koch): <0.2 slight, 0.2–0.4 fair, 0.4–0.6 moderate, 0.6–0.8 substantial, >0.8 almost perfect.

## Sonnet (re-rate) vs Haiku (re-rate) — controlled inter-coder  (n shared = 523)
**Primary category** (Cohen's κ): κ = 0.652 (substantial), raw agreement 70.8%, n = 517

Most-confused category pairs:
| A | B | count |
|---|---|------:|
| A2A protocols | Autonomous netops | 17 |
| A2A protocols | Agent discovery/reg | 16 |
| A2A protocols | Agent identity/auth | 15 |
| Agent identity/auth | Other AI/agent | 14 |
| Data formats/interop | Other AI/agent | 10 |
| A2A protocols | Data formats/interop | 9 |
| Agent identity/auth | Data formats/interop | 8 |
| Autonomous netops | Other AI/agent | 6 |

**Ordinal dimensions** (quadratic-weighted κ):

| dimension | κ_w | raw agree | n |
|---|---:|---:|---:|
| novelty | 0.206 (fair) | 39.4% | 523 |
| maturity | 0.592 (moderate) | 54.1% | 523 |
| overlap | 0.127 (slight) | 26.4% | 523 |
| momentum | 0.457 (moderate) | 33.8% | 467 |
| relevance | 0.728 (substantial) | 37.7% | 523 |

## Sonnet (re-rate) vs Production labels — drift/stability  (n shared = 524)
**Primary category** (Cohen's κ): κ = 0.645 (substantial), raw agreement 69.9%, n = 518

Most-confused category pairs:
| A | B | count |
|---|---|------:|
| Agent identity/auth | Other AI/agent | 18 |
| A2A protocols | Autonomous netops | 15 |
| A2A protocols | Agent identity/auth | 13 |
| Data formats/interop | Other AI/agent | 11 |
| A2A protocols | Data formats/interop | 8 |
| A2A protocols | Agent discovery/reg | 8 |
| Autonomous netops | Other AI/agent | 7 |
| Agent identity/auth | Data formats/interop | 7 |

**Ordinal dimensions** (quadratic-weighted κ):

| dimension | κ_w | raw agree | n |
|---|---:|---:|---:|
| novelty | 0.477 (moderate) | 43.1% | 524 |
| maturity | 0.620 (substantial) | 52.1% | 524 |
| overlap | 0.282 (fair) | 41.8% | 524 |
| momentum | 0.247 (fair) | 32.4% | 524 |
| relevance | 0.234 (fair) | 43.7% | 524 |

## Haiku (re-rate) vs Production labels  (n shared = 523)
**Primary category** (Cohen's κ): κ = 0.596 (moderate), raw agreement 65.9%, n = 522

Most-confused category pairs:
| A | B | count |
|---|---|------:|
| A2A protocols | Agent discovery/reg | 17 |
| A2A protocols | Autonomous netops | 16 |
| Agent identity/auth | Data formats/interop | 16 |
| A2A protocols | Agent identity/auth | 14 |
| Autonomous netops | Other AI/agent | 10 |
| Agent identity/auth | Other AI/agent | 9 |
| AI safety/alignment | Agent identity/auth | 9 |
| A2A protocols | Data formats/interop | 8 |

**Ordinal dimensions** (quadratic-weighted κ):

| dimension | κ_w | raw agree | n |
|---|---:|---:|---:|
| novelty | 0.097 (slight) | 13.2% | 523 |
| maturity | 0.412 (moderate) | 37.3% | 523 |
| overlap | 0.087 (slight) | 21.2% | 523 |
| momentum | 0.114 (slight) | 10.1% | 467 |
| relevance | 0.220 (fair) | 34.8% | 523 |