62 lines
3.6 KiB
TeX
62 lines
3.6 KiB
TeX
\section{Introduction}
|
|
\label{sec:intro}
|
|
|
|
Standardization activity around AI agents and autonomous systems has surged at
|
|
the Internet Engineering Task Force (IETF). In the corpus studied here, monthly
|
|
submissions of AI/agent-related Internet-Drafts grew from an average of $3.7$
|
|
per month over June 2024--May 2025 to $38.8$ per month since June 2025, peaking
|
|
at $106$ drafts in March 2026---roughly $35\times$ a typical 2024 month. A surge
|
|
of this size, concentrated in barely a year, makes the area difficult to
|
|
navigate: there is no settled taxonomy, the boundaries between proposals are
|
|
fluid, and it is unclear how much of the activity represents distinct technical
|
|
work versus overlapping attempts at the same problems.
|
|
|
|
A natural response is to characterise the space quantitatively, and
|
|
LLM-assisted corpus analyses are an increasingly common instrument for doing
|
|
so---using a large language model to classify documents, score them, and
|
|
summarise trends at a scale that manual reading cannot reach
|
|
\cite{tan2024survey,gilardi2023chatgpt}. Such studies, however, rarely report whether the labels they rely on
|
|
are reproducible. An LLM applied to the same documents under a different model,
|
|
or even a different run, may produce different categories and scores; without a
|
|
reliability check, it is impossible to tell which conclusions rest on stable
|
|
signal and which on rater noise.
|
|
|
|
To our knowledge no verified quantitative survey of the AI-agent
|
|
standardization space at the IETF exists. We address this gap with a
|
|
descriptive survey that is explicit about which of its measurements can be
|
|
trusted. This paper is deliberately neutral: it characterises the landscape
|
|
rather than advocating for any particular proposal or design.
|
|
|
|
\paragraph{Contributions.}
|
|
\begin{itemize}[leftmargin=1.4em,topsep=2pt,itemsep=1pt]
|
|
\item A curated corpus of \textbf{524} AI/agent-related IETF Internet-Drafts
|
|
spanning January 2024 to May 2026, constructed by keyword candidate
|
|
selection followed by explicit false-positive filtering
|
|
(Section~\ref{sec:method}).
|
|
\item A characterization of the corpus along four axes---temporal submission
|
|
dynamics, thematic category distribution, authorship and working-group
|
|
structure, and semantic redundancy measured through text embeddings
|
|
(Section~\ref{sec:landscape}).
|
|
\item An explicit two-model inter-rater reliability check that separates the
|
|
labels we can trust from those we cannot: categorical assignment is
|
|
substantially reproducible (Cohen's $\kappa \approx 0.65$), whereas the
|
|
LLM ordinal \emph{quality} scores are not ($\kappa_w = 0.13$--$0.21$ for
|
|
the least stable dimensions). We report the former and exclude the
|
|
latter (Section~\ref{sec:reliability}).
|
|
\item A full release of the data, queries, and rating artifacts used in the
|
|
analysis, so that every reported number can be recomputed.
|
|
\end{itemize}
|
|
|
|
\paragraph{Roadmap.}
|
|
Section~\ref{sec:method} describes corpus construction, the LLM-assisted
|
|
classification pipeline, and the embedding setup.
|
|
Section~\ref{sec:landscape} presents the landscape across the four axes:
|
|
temporal dynamics (Section~\ref{sec:temporal}), category distribution
|
|
(Section~\ref{sec:categories}), authorship and working-group structure
|
|
(Section~\ref{sec:authors}), and semantic redundancy
|
|
(Section~\ref{sec:redundancy}).
|
|
Section~\ref{sec:reliability} reports the two-model reliability experiment that
|
|
underwrites the category labels and disqualifies the quality scores.
|
|
Section~\ref{sec:discussion} discusses implications and limitations, and
|
|
Section~\ref{sec:conclusion} concludes with directions for future work.
|