77 lines
4.3 KiB
TeX
77 lines
4.3 KiB
TeX
\section{Related Work}
|
|
\label{sec:related}
|
|
|
|
\paragraph{Quantitative analysis of IETF activity.}
|
|
Several studies have examined the IETF corpus empirically. McQuistin
|
|
et al.\ characterised the IETF through the lens of RFC deployment,
|
|
using a dataset of 8,711 RFCs, 4,512 authors, and 2.4 million emails
|
|
to measure shifts in publication rate, author concentration, and
|
|
cross-document dependency over time~\cite{mcquistin2021characterising}.
|
|
Zhang et al.\ extended this line of work with a longitudinal analysis
|
|
of author affiliations across 2001--2023, covering 73,764 individuals
|
|
and finding that organisational diversity peaked and then
|
|
stagnated~\cite{zhang2025affiliations}. The IAB's own workshop on
|
|
Analyzing IETF Data (AID, 2021) surveyed open questions about
|
|
what drives drafts to become RFCs and how community diversity
|
|
evolves~\cite{tenoever2022aid}. More recently, Jim\'enez applied LLMs
|
|
directly to IETF working-group records to automate the generation of
|
|
summary reports~\cite{jimenez2024automating}. Our work complements this
|
|
body of evidence by providing the first quantitative survey focused
|
|
\emph{exclusively on the AI/agent topic area}: we document the growth
|
|
trajectory, authorship structure, and thematic distribution of a
|
|
cohort that did not exist in large numbers before 2025, and we do
|
|
so at the Internet-Draft stage rather than the post-publication RFC
|
|
stage.
|
|
|
|
\paragraph{LLM-assisted text classification and annotation.}
|
|
The use of LLMs as automated annotators is well established.
|
|
Gilardi et al.\ showed that ChatGPT outperforms crowd-workers on
|
|
political-science annotation tasks (stance, relevance, frames)
|
|
across 6,183 tweets, with zero-shot accuracy exceeding MTurk workers
|
|
by roughly 25 percentage points on average~\cite{gilardi2023chatgpt}.
|
|
Tan et al.\ survey the broader landscape of LLMs for data annotation
|
|
and synthesis, covering annotation generation, quality assessment, and
|
|
downstream utilisation~\cite{tan2024survey}. Yang et al.\ survey
|
|
emerging AI-agent communication protocols---the class of specification
|
|
documents our corpus comprises---providing context for the
|
|
standardisation subjects under analysis~\cite{yang2025agentprotocols}.
|
|
Our contribution in this dimension is not classification per se but
|
|
the application of the LLM-annotator pipeline to a \emph{technical
|
|
standards corpus} rather than short social-media texts, and the
|
|
explicit reliability gate described next.
|
|
|
|
\paragraph{Inter-rater reliability of LLM labels.}
|
|
The reproducibility of LLM judgments is contested.
|
|
Reiss identified prompt-sensitivity as a reliability hazard, showing
|
|
that minor wording changes or repeated identical inputs can shift
|
|
ChatGPT's classification output below the thresholds conventionally
|
|
required for scientific use~\cite{reiss2023reliability}. Wang et al.\
|
|
found that ChatGPT correlates well with human judgments on several
|
|
natural-language-generation evaluation dimensions but that the
|
|
strength of correlation varies substantially across
|
|
tasks~\cite{wang2023nlgeval}. These results motivate---but do not
|
|
themselves perform---inter-rater reliability analysis using
|
|
established statistics. We use Cohen's $\kappa$~\cite{cohen1960kappa}
|
|
with the Landis--Koch interpretation scale~\cite{landis1977kappa} to
|
|
report agreement both between two independent LLM re-rating runs and
|
|
between those runs and the production labels. To our knowledge, no
|
|
prior quantitative survey of an IETF (or comparable standards-body)
|
|
corpus has reported this reliability decomposition; the closest
|
|
precedent is Jim\'enez~\cite{jimenez2024automating}, who generates
|
|
LLM summaries of IETF records but does not measure label
|
|
reproducibility.
|
|
|
|
\paragraph{Embedding-based semantic analysis of document corpora.}
|
|
We use cosine similarity over dense text embeddings for the redundancy
|
|
analysis. Reimers and Gurevych established that sentence-level
|
|
embeddings computed with siamese BERT networks provide efficient and
|
|
accurate representations for semantic similarity
|
|
search~\cite{reimers2019sbert}; more recent work by Nussbaum et al.\
|
|
produces the \texttt{nomic-embed-text} model used in our
|
|
pipeline~\cite{nussbaum2024nomic}. The application of embedding-based
|
|
near-duplicate detection to a \emph{standards corpus} is, to our
|
|
knowledge, novel: prior IETF studies (e.g.,
|
|
\cite{mcquistin2021characterising}) rely on structural metadata---author
|
|
lists, citation graphs, working-group membership---rather than
|
|
semantic similarity over document content.
|