feat(survey): add IETF landscape survey (kappa, phase0, rerate), gaps update; bump wimse-ect; gitignore run logs

workspace/drafts/landscape-survey/sections/05-reliability.tex

@@ -0,0 +1,81 @@
+\section{How Reliable Are the Labels?}
+\label{sec:reliability}
+
+Every category in Section~\ref{sec:categories} is an LLM judgment. Before
+treating the distribution as a finding, we ask how reproducible those judgments
+are. We re-rated all 524 drafts a second time with the same pinned prompt using
+two models---Claude Sonnet (\texttt{claude-sonnet-4-20250514}) and Claude Haiku
+(\texttt{claude-haiku-4-5-20251001})---and compared the assignments. We report
+Cohen's $\kappa$ for the nominal primary-category assignment and
+quadratic-weighted $\kappa$ ($\kappa_w$) for the ordinal $1$--$5$ dimensions,
+interpreting magnitudes by the Landis--Koch convention (\,$<0.2$ slight,
+$0.2$--$0.4$ fair, $0.4$--$0.6$ moderate, $0.6$--$0.8$ substantial, $>0.8$
+almost perfect\,).
+
+\paragraph{Category assignment is substantially reliable.}
+For the primary category, Sonnet and Haiku agree at $\kappa = 0.652$ (raw
+agreement $70.8\%$). Each model independently re-rating also agrees
+substantially with the original production labels (Sonnet $\kappa = 0.645$,
+Haiku $\kappa = 0.596$). The residual disagreements are not random: they
+concentrate on semantically adjacent categories
+(Table~\ref{tab:confusion})---A2A protocols versus autonomous network
+operations, A2A versus agent discovery, and identity versus the residual
+``Other'' bucket. These are boundary cases, not classifier noise, which is why
+we treat the distribution as informative while acknowledging $\pm$ a few points
+of category-boundary uncertainty.
+
+\begin{table}[h]
+\centering
+\begin{tabular}{llr}
+\toprule
+\textbf{Category A} & \textbf{Category B} & \textbf{Disagreements} \\
+\midrule
+A2A protocols        & Autonomous netops     & 17 \\
+A2A protocols        & Agent discovery/reg   & 16 \\
+A2A protocols        & Agent identity/auth   & 15 \\
+Agent identity/auth  & Other AI/agent        & 14 \\
+Data formats/interop & Other AI/agent        & 10 \\
+\bottomrule
+\end{tabular}
+\caption{Most-confused primary-category pairs between the two re-rating models.
+Disagreements concentrate on semantic neighbours.}
+\label{tab:confusion}
+\end{table}
+
+\paragraph{Ordinal quality scores are not reliable.}
+The five $1$--$5$ quality dimensions tell a different and cautionary story
+(Table~\ref{tab:kappa}). Between Sonnet and Haiku, \emph{overlap} reaches only
+$\kappa_w = 0.127$ (slight---effectively no better than chance) and
+\emph{novelty} $\kappa_w = 0.206$. \emph{relevance} appears substantial between
+the two re-rating models ($0.728$) but collapses to $0.234$ against the
+production labels, indicating it is not stable across runs either. Only
+\emph{maturity} is consistently moderate ($0.59$--$0.62$).
+
+\begin{table}[h]
+\centering
+\begin{tabular}{lcc}
+\toprule
+\textbf{Dimension} & \textbf{$\kappa_w$ (Sonnet vs.\ Haiku)} & \textbf{$\kappa_w$ (Sonnet vs.\ prod.)} \\
+\midrule
+relevance  & 0.728 & 0.234 \\
+maturity   & 0.592 & 0.620 \\
+momentum   & 0.457 & 0.247 \\
+novelty    & 0.206 & 0.477 \\
+overlap    & 0.127 & 0.282 \\
+\bottomrule
+\end{tabular}
+\caption{Quadratic-weighted $\kappa$ for the ordinal quality dimensions.
+Low and inconsistent values---especially for \emph{overlap} and
+\emph{novelty}---indicate these scores are not reproducible across raters.}
+\label{tab:kappa}
+\end{table}
+
+\paragraph{Consequence.}
+We therefore report the categorical landscape (Section~\ref{sec:categories}) but
+deliberately exclude the LLM ordinal quality scores from our findings. We regard
+this split as a general caution rather than an artifact of our particular
+pipeline: LLMs can place standards documents into a thematic taxonomy with
+substantial agreement, but asking them to score subjective qualities such as
+``novelty'' or ``overlap'' on a numeric scale produces labels that do not
+survive a change of model. Studies that use LLM-assigned quality scores as
+quantitative evidence should report inter-rater reliability before doing so.