82 lines
3.8 KiB
TeX
82 lines
3.8 KiB
TeX
\section{How Reliable Are the Labels?}
|
|
\label{sec:reliability}
|
|
|
|
Every category in Section~\ref{sec:categories} is an LLM judgment. Before
|
|
treating the distribution as a finding, we ask how reproducible those judgments
|
|
are. We re-rated all 524 drafts a second time with the same pinned prompt using
|
|
two models---Claude Sonnet (\texttt{claude-sonnet-4-20250514}) and Claude Haiku
|
|
(\texttt{claude-haiku-4-5-20251001})---and compared the assignments. We report
|
|
Cohen's $\kappa$ for the nominal primary-category assignment and
|
|
quadratic-weighted $\kappa$ ($\kappa_w$) for the ordinal $1$--$5$ dimensions,
|
|
interpreting magnitudes by the Landis--Koch convention (\,$<0.2$ slight,
|
|
$0.2$--$0.4$ fair, $0.4$--$0.6$ moderate, $0.6$--$0.8$ substantial, $>0.8$
|
|
almost perfect\,).
|
|
|
|
\paragraph{Category assignment is substantially reliable.}
|
|
For the primary category, Sonnet and Haiku agree at $\kappa = 0.652$ (raw
|
|
agreement $70.8\%$). Each model independently re-rating also agrees
|
|
substantially with the original production labels (Sonnet $\kappa = 0.645$,
|
|
Haiku $\kappa = 0.596$). The residual disagreements are not random: they
|
|
concentrate on semantically adjacent categories
|
|
(Table~\ref{tab:confusion})---A2A protocols versus autonomous network
|
|
operations, A2A versus agent discovery, and identity versus the residual
|
|
``Other'' bucket. These are boundary cases, not classifier noise, which is why
|
|
we treat the distribution as informative while acknowledging $\pm$ a few points
|
|
of category-boundary uncertainty.
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\begin{tabular}{llr}
|
|
\toprule
|
|
\textbf{Category A} & \textbf{Category B} & \textbf{Disagreements} \\
|
|
\midrule
|
|
A2A protocols & Autonomous netops & 17 \\
|
|
A2A protocols & Agent discovery/reg & 16 \\
|
|
A2A protocols & Agent identity/auth & 15 \\
|
|
Agent identity/auth & Other AI/agent & 14 \\
|
|
Data formats/interop & Other AI/agent & 10 \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\caption{Most-confused primary-category pairs between the two re-rating models.
|
|
Disagreements concentrate on semantic neighbours.}
|
|
\label{tab:confusion}
|
|
\end{table}
|
|
|
|
\paragraph{Ordinal quality scores are not reliable.}
|
|
The five $1$--$5$ quality dimensions tell a different and cautionary story
|
|
(Table~\ref{tab:kappa}). Between Sonnet and Haiku, \emph{overlap} reaches only
|
|
$\kappa_w = 0.127$ (slight---effectively no better than chance) and
|
|
\emph{novelty} $\kappa_w = 0.206$. \emph{relevance} appears substantial between
|
|
the two re-rating models ($0.728$) but collapses to $0.234$ against the
|
|
production labels, indicating it is not stable across runs either. Only
|
|
\emph{maturity} is consistently moderate ($0.59$--$0.62$).
|
|
|
|
\begin{table}[h]
|
|
\centering
|
|
\begin{tabular}{lcc}
|
|
\toprule
|
|
\textbf{Dimension} & \textbf{$\kappa_w$ (Sonnet vs.\ Haiku)} & \textbf{$\kappa_w$ (Sonnet vs.\ prod.)} \\
|
|
\midrule
|
|
relevance & 0.728 & 0.234 \\
|
|
maturity & 0.592 & 0.620 \\
|
|
momentum & 0.457 & 0.247 \\
|
|
novelty & 0.206 & 0.477 \\
|
|
overlap & 0.127 & 0.282 \\
|
|
\bottomrule
|
|
\end{tabular}
|
|
\caption{Quadratic-weighted $\kappa$ for the ordinal quality dimensions.
|
|
Low and inconsistent values---especially for \emph{overlap} and
|
|
\emph{novelty}---indicate these scores are not reproducible across raters.}
|
|
\label{tab:kappa}
|
|
\end{table}
|
|
|
|
\paragraph{Consequence.}
|
|
We therefore report the categorical landscape (Section~\ref{sec:categories}) but
|
|
deliberately exclude the LLM ordinal quality scores from our findings. We regard
|
|
this split as a general caution rather than an artifact of our particular
|
|
pipeline: LLMs can place standards documents into a thematic taxonomy with
|
|
substantial agreement, but asking them to score subjective qualities such as
|
|
``novelty'' or ``overlap'' on a numeric scale produces labels that do not
|
|
survive a change of model. Studies that use LLM-assigned quality scores as
|
|
quantitative evidence should report inter-rater reliability before doing so.
|