\section{How Reliable Are the Labels?}
\label{sec:reliability}

Every category in Section~\ref{sec:categories} is an LLM judgment. Before
treating the distribution as a finding, we ask how reproducible those judgments
are. We re-rated all 524 drafts a second time with the same pinned prompt using
two models---Claude Sonnet (\texttt{claude-sonnet-4-20250514}) and Claude Haiku
(\texttt{claude-haiku-4-5-20251001})---and compared the assignments. We report
Cohen's $\kappa$ for the nominal primary-category assignment and
quadratic-weighted $\kappa$ ($\kappa_w$) for the ordinal $1$--$5$ dimensions,
interpreting magnitudes by the Landis--Koch convention (\,$<0.2$ slight,
$0.2$--$0.4$ fair, $0.4$--$0.6$ moderate, $0.6$--$0.8$ substantial, $>0.8$
almost perfect\,).

\paragraph{Category assignment is substantially reliable.}
For the primary category, Sonnet and Haiku agree at $\kappa = 0.652$ (raw
agreement $70.8\%$). Each model independently re-rating also agrees
substantially with the original production labels (Sonnet $\kappa = 0.645$,
Haiku $\kappa = 0.596$). The residual disagreements are not random: they
concentrate on semantically adjacent categories
(Table~\ref{tab:confusion})---A2A protocols versus autonomous network
operations, A2A versus agent discovery, and identity versus the residual
``Other'' bucket. These are boundary cases, not classifier noise, which is why
we treat the distribution as informative while acknowledging $\pm$ a few points
of category-boundary uncertainty.

\begin{table}[h]
\centering
\begin{tabular}{llr}
\toprule
\textbf{Category A} & \textbf{Category B} & \textbf{Disagreements} \\
\midrule
A2A protocols        & Autonomous netops     & 17 \\
A2A protocols        & Agent discovery/reg   & 16 \\
A2A protocols        & Agent identity/auth   & 15 \\
Agent identity/auth  & Other AI/agent        & 14 \\
Data formats/interop & Other AI/agent        & 10 \\
\bottomrule
\end{tabular}
\caption{Most-confused primary-category pairs between the two re-rating models.
Disagreements concentrate on semantic neighbours.}
\label{tab:confusion}
\end{table}

\paragraph{Ordinal quality scores are not reliable.}
The five $1$--$5$ quality dimensions tell a different and cautionary story
(Table~\ref{tab:kappa}). Between Sonnet and Haiku, \emph{overlap} reaches only
$\kappa_w = 0.127$ (slight---effectively no better than chance) and
\emph{novelty} $\kappa_w = 0.206$. \emph{relevance} appears substantial between
the two re-rating models ($0.728$) but collapses to $0.234$ against the
production labels, indicating it is not stable across runs either. Only
\emph{maturity} is consistently moderate ($0.59$--$0.62$).

\begin{table}[h]
\centering
\begin{tabular}{lcc}
\toprule
\textbf{Dimension} & \textbf{$\kappa_w$ (Sonnet vs.\ Haiku)} & \textbf{$\kappa_w$ (Sonnet vs.\ prod.)} \\
\midrule
relevance  & 0.728 & 0.234 \\
maturity   & 0.592 & 0.620 \\
momentum   & 0.457 & 0.247 \\
novelty    & 0.206 & 0.477 \\
overlap    & 0.127 & 0.282 \\
\bottomrule
\end{tabular}
\caption{Quadratic-weighted $\kappa$ for the ordinal quality dimensions.
Low and inconsistent values---especially for \emph{overlap} and
\emph{novelty}---indicate these scores are not reproducible across raters.}
\label{tab:kappa}
\end{table}

\paragraph{Consequence.}
We therefore report the categorical landscape (Section~\ref{sec:categories}) but
deliberately exclude the LLM ordinal quality scores from our findings. We regard
this split as a general caution rather than an artifact of our particular
pipeline: LLMs can place standards documents into a thematic taxonomy with
substantial agreement, but asking them to score subjective qualities such as
``novelty'' or ``overlap'' on a numeric scale produces labels that do not
survive a change of model. Studies that use LLM-assigned quality scores as
quantitative evidence should report inter-rater reliability before doing so.