66 lines
4.0 KiB
TeX
66 lines
4.0 KiB
TeX
\section{Discussion and Limitations}
|
|
\label{sec:discussion}
|
|
|
|
\subsection{Implications}
|
|
|
|
Two structural features of the corpus reinforce one another. First, the area is
|
|
\emph{pre-standardization}: $456$ of the $524$ drafts ($87\%$) are individual
|
|
submissions not adopted by any working group (Section~\ref{sec:authors}).
|
|
Second, it is semantically redundant: $170$ drafts ($32.4\%$) have at least one
|
|
near-duplicate (cosine $>0.9$) elsewhere in the corpus
|
|
(Section~\ref{sec:redundancy}). Taken together these point to a young, crowded
|
|
design space in which many authors independently propose overlapping solutions
|
|
to the same problems. Such a configuration is consistent with the early phase of
|
|
a standards effort, where consolidation and competition between proposals have
|
|
yet to resolve into adopted work; it does not, on its own, indicate
|
|
duplication of effort in any pejorative sense, since some redundancy is the
|
|
expected by-product of parallel exploration. We note this dynamic descriptively
|
|
rather than predicting which proposals will prevail.
|
|
|
|
Beyond the IETF, our reliability result carries a general caution for
|
|
LLM-assisted corpus studies. The same pipeline that places documents into a
|
|
thematic taxonomy with substantial agreement (Cohen's $\kappa \approx 0.65$)
|
|
produces ordinal quality scores---``novelty,'' ``overlap''---that do not survive
|
|
a change of model ($\kappa_w$ as low as $0.13$). A study that reported the
|
|
category distribution and the quality scores side by side, without a reliability
|
|
check, would present a reproducible measurement and rater noise with equal
|
|
confidence. The discipline of separating the two, by an explicit inter-rater
|
|
analysis, is what allowed us to keep the former and discard the latter.
|
|
|
|
\subsection{Limitations}
|
|
|
|
Several limitations bound the interpretation of our findings.
|
|
|
|
\begin{itemize}[leftmargin=1.4em,topsep=2pt,itemsep=1pt]
|
|
\item \textbf{Abstract-only classification.} Categories and scores are derived
|
|
from each draft's title, metadata, and abstract, not its full text
|
|
(Section~\ref{sec:method}). Classifications therefore reflect how a draft
|
|
presents itself rather than a full reading of its mechanics, and a draft
|
|
whose abstract understates its technical content may be miscategorised.
|
|
\item \textbf{Single snapshot.} The analysis rests on one corpus snapshot
|
|
(\texttt{data/drafts.db} as of 2026-05-23). All counts, trends, and
|
|
similarities are as of that date and will drift as drafts are added,
|
|
revised, expired, or adopted.
|
|
\item \textbf{IETF-only scope.} We deliberately restrict the corpus to IETF
|
|
Internet-Drafts; documents from ISO, ITU, ETSI, NIST, and W3C are
|
|
excluded because of heterogeneous metadata and a different document
|
|
model. The survey therefore says nothing about AI-agent standardization
|
|
outside the IETF, and the overall scale of the field is correspondingly
|
|
understated.
|
|
\item \textbf{LLM-derived categories.} The thematic taxonomy is produced by a
|
|
large language model. The two-model $\kappa$ check
|
|
(Section~\ref{sec:reliability}) mitigates but does not eliminate this
|
|
dependence: substantial agreement still leaves category-boundary
|
|
uncertainty of a few points, concentrated on semantically adjacent
|
|
categories.
|
|
\item \textbf{Provisional recent tail.} Submission counts for the most recent
|
|
months (April--May 2026) are provisional because of indexing and fetch
|
|
lag; the dip after the March 2026 peak in Figure~\ref{fig:temporal}
|
|
should be read as incomplete data rather than as a downturn.
|
|
\item \textbf{Keyword-based candidate selection.} Candidates were identified by
|
|
keyword and topic matching, which can both miss relevant drafts whose
|
|
abstracts avoid the chosen vocabulary and over-include unrelated
|
|
documents; we observed a $12.2\%$ false-positive rate ($73$ of $597$
|
|
candidates) before filtering, and an unknown false-negative rate remains.
|
|
\end{itemize}
|