c/ietf-draft-analyzer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(survey): add IETF landscape survey (kappa, phase0, rerate), gaps update; bump wimse-ect; gitignore run logs

Browse Source
This commit is contained in:
Christian Nennemann
2026-05-25 12:35:31 +02:00
parent 89df70a6c0
commit 6e6e0489b8
43 changed files with 11956 additions and 1384 deletions
Download Patch File Download Diff File Expand all files Collapse all files

24
workspace/drafts/landscape-survey/sections/00-abstract.tex Normal file
View File

@@ -0,0 +1,24 @@
\begin{abstract}
Between 2024 and 2026 the Internet Engineering Task Force (IETF) saw a sharp
rise in Internet-Drafts addressing AI agents and autonomous systems: monthly
submissions in our corpus grew from an average of 3.7 (June 2024--May 2025) to
38.8 (since June 2025), peaking at 106 in March 2026---roughly $35\times$ a
typical 2024 month. We present a quantitative survey of this emerging area
based on a curated corpus of 524 AI/agent-related IETF Internet-Drafts spanning
January 2024 to May 2026. We characterise the corpus along four axes: temporal
submission dynamics, thematic category distribution, authorship and
working-group structure, and semantic redundancy measured through text
embeddings. Two findings stand out: the area is overwhelmingly
\emph{pre-standardization}---87\% of drafts are individual submissions not
adopted by any working group---and it is semantically dense, with 32\% of
drafts having a near-duplicate (cosine $>0.9$) elsewhere in the corpus.
Because the thematic categories are produced by an LLM-assisted pipeline, we
explicitly quantify their reliability through a two-model re-rating experiment:
categorical assignment is substantially reproducible (Cohen's $\kappa = 0.65$),
whereas LLM ordinal \emph{quality} scores such as novelty and overlap are not
($\kappa_w = 0.13$--$0.21$). We therefore report the category landscape but
deliberately exclude quality scores from our findings, and we argue this
distinction is a general caution for the growing practice of LLM-assisted
corpus analysis. All data, queries, and rating artifacts are released for
reproduction.
\end{abstract}

61
workspace/drafts/landscape-survey/sections/01-intro.tex Normal file
View File

@@ -0,0 +1,61 @@
\section{Introduction}
\label{sec:intro}
Standardization activity around AI agents and autonomous systems has surged at
the Internet Engineering Task Force (IETF). In the corpus studied here, monthly
submissions of AI/agent-related Internet-Drafts grew from an average of $3.7$
per month over June 2024--May 2025 to $38.8$ per month since June 2025, peaking
at $106$ drafts in March 2026---roughly $35\times$ a typical 2024 month. A surge
of this size, concentrated in barely a year, makes the area difficult to
navigate: there is no settled taxonomy, the boundaries between proposals are
fluid, and it is unclear how much of the activity represents distinct technical
work versus overlapping attempts at the same problems.
A natural response is to characterise the space quantitatively, and
LLM-assisted corpus analyses are an increasingly common instrument for doing
so---using a large language model to classify documents, score them, and
summarise trends at a scale that manual reading cannot reach
\cite{tan2024survey,gilardi2023chatgpt}. Such studies, however, rarely report whether the labels they rely on
are reproducible. An LLM applied to the same documents under a different model,
or even a different run, may produce different categories and scores; without a
reliability check, it is impossible to tell which conclusions rest on stable
signal and which on rater noise.
To our knowledge no verified quantitative survey of the AI-agent
standardization space at the IETF exists. We address this gap with a
descriptive survey that is explicit about which of its measurements can be
trusted. This paper is deliberately neutral: it characterises the landscape
rather than advocating for any particular proposal or design.
\paragraph{Contributions.}
\begin{itemize}[leftmargin=1.4em,topsep=2pt,itemsep=1pt]
\item A curated corpus of \textbf{524} AI/agent-related IETF Internet-Drafts
spanning January 2024 to May 2026, constructed by keyword candidate
selection followed by explicit false-positive filtering
(Section~\ref{sec:method}).
\item A characterization of the corpus along four axes---temporal submission
dynamics, thematic category distribution, authorship and working-group
structure, and semantic redundancy measured through text embeddings
(Section~\ref{sec:landscape}).
\item An explicit two-model inter-rater reliability check that separates the
labels we can trust from those we cannot: categorical assignment is
substantially reproducible (Cohen's $\kappa \approx 0.65$), whereas the
LLM ordinal \emph{quality} scores are not ($\kappa_w = 0.13$--$0.21$ for
the least stable dimensions). We report the former and exclude the
latter (Section~\ref{sec:reliability}).
\item A full release of the data, queries, and rating artifacts used in the
analysis, so that every reported number can be recomputed.
\end{itemize}
\paragraph{Roadmap.}
Section~\ref{sec:method} describes corpus construction, the LLM-assisted
classification pipeline, and the embedding setup.
Section~\ref{sec:landscape} presents the landscape across the four axes:
temporal dynamics (Section~\ref{sec:temporal}), category distribution
(Section~\ref{sec:categories}), authorship and working-group structure
(Section~\ref{sec:authors}), and semantic redundancy
(Section~\ref{sec:redundancy}).
Section~\ref{sec:reliability} reports the two-model reliability experiment that
underwrites the category labels and disqualifies the quality scores.
Section~\ref{sec:discussion} discusses implications and limitations, and
Section~\ref{sec:conclusion} concludes with directions for future work.

76
workspace/drafts/landscape-survey/sections/02-related.tex Normal file
View File

@@ -0,0 +1,76 @@
\section{Related Work}
\label{sec:related}
\paragraph{Quantitative analysis of IETF activity.}
Several studies have examined the IETF corpus empirically. McQuistin
et al.\ characterised the IETF through the lens of RFC deployment,
using a dataset of 8,711 RFCs, 4,512 authors, and 2.4 million emails
to measure shifts in publication rate, author concentration, and
cross-document dependency over time~\cite{mcquistin2021characterising}.
Zhang et al.\ extended this line of work with a longitudinal analysis
of author affiliations across 2001--2023, covering 73,764 individuals
and finding that organisational diversity peaked and then
stagnated~\cite{zhang2025affiliations}. The IAB's own workshop on
Analyzing IETF Data (AID, 2021) surveyed open questions about
what drives drafts to become RFCs and how community diversity
evolves~\cite{tenoever2022aid}. More recently, Jim\'enez applied LLMs
directly to IETF working-group records to automate the generation of
summary reports~\cite{jimenez2024automating}. Our work complements this
body of evidence by providing the first quantitative survey focused
\emph{exclusively on the AI/agent topic area}: we document the growth
trajectory, authorship structure, and thematic distribution of a
cohort that did not exist in large numbers before 2025, and we do
so at the Internet-Draft stage rather than the post-publication RFC
stage.
\paragraph{LLM-assisted text classification and annotation.}
The use of LLMs as automated annotators is well established.
Gilardi et al.\ showed that ChatGPT outperforms crowd-workers on
political-science annotation tasks (stance, relevance, frames)
across 6,183 tweets, with zero-shot accuracy exceeding MTurk workers
by roughly 25 percentage points on average~\cite{gilardi2023chatgpt}.
Tan et al.\ survey the broader landscape of LLMs for data annotation
and synthesis, covering annotation generation, quality assessment, and
downstream utilisation~\cite{tan2024survey}. Yang et al.\ survey
emerging AI-agent communication protocols---the class of specification
documents our corpus comprises---providing context for the
standardisation subjects under analysis~\cite{yang2025agentprotocols}.
Our contribution in this dimension is not classification per se but
the application of the LLM-annotator pipeline to a \emph{technical
standards corpus} rather than short social-media texts, and the
explicit reliability gate described next.
\paragraph{Inter-rater reliability of LLM labels.}
The reproducibility of LLM judgments is contested.
Reiss identified prompt-sensitivity as a reliability hazard, showing
that minor wording changes or repeated identical inputs can shift
ChatGPT's classification output below the thresholds conventionally
required for scientific use~\cite{reiss2023reliability}. Wang et al.\
found that ChatGPT correlates well with human judgments on several
natural-language-generation evaluation dimensions but that the
strength of correlation varies substantially across
tasks~\cite{wang2023nlgeval}. These results motivate---but do not
themselves perform---inter-rater reliability analysis using
established statistics. We use Cohen's $\kappa$~\cite{cohen1960kappa}
with the Landis--Koch interpretation scale~\cite{landis1977kappa} to
report agreement both between two independent LLM re-rating runs and
between those runs and the production labels. To our knowledge, no
prior quantitative survey of an IETF (or comparable standards-body)
corpus has reported this reliability decomposition; the closest
precedent is Jim\'enez~\cite{jimenez2024automating}, who generates
LLM summaries of IETF records but does not measure label
reproducibility.
\paragraph{Embedding-based semantic analysis of document corpora.}
We use cosine similarity over dense text embeddings for the redundancy
analysis. Reimers and Gurevych established that sentence-level
embeddings computed with siamese BERT networks provide efficient and
accurate representations for semantic similarity
search~\cite{reimers2019sbert}; more recent work by Nussbaum et al.\
produces the \texttt{nomic-embed-text} model used in our
pipeline~\cite{nussbaum2024nomic}. The application of embedding-based
near-duplicate detection to a \emph{standards corpus} is, to our
knowledge, novel: prior IETF studies (e.g.,
\cite{mcquistin2021characterising}) rely on structural metadata---author
lists, citation graphs, working-group membership---rather than
semantic similarity over document content.

57
workspace/drafts/landscape-survey/sections/03-method.tex Normal file
View File

@@ -0,0 +1,57 @@
\section{Corpus and Method}
\label{sec:method}
\subsection{Corpus construction}
Internet-Drafts were collected from the IETF Datatracker. Candidate documents
were identified by keyword and topic matching for AI-agent and autonomous-system
terminology (e.g.\ ``agent'', ``autonomous'', ``LLM'', ``inference''), then
filtered to remove false positives---documents in which such terms occur in an
unrelated sense (``user agent'', ``autonomous system'' in the BGP sense). Of
597 candidate IETF documents, 73 ($12.2\%$) were flagged as false positives and
excluded, yielding a clean corpus of \textbf{524 IETF Internet-Drafts}.
We restrict this survey to IETF Internet-Drafts. The collection pipeline also
ingests documents from other standards bodies (ISO, ITU, ETSI, NIST, W3C), but
those exhibit heterogeneous and frequently missing publication dates and a
different document model; mixing them would compromise the temporal and
authorship analyses. All 524 IETF drafts carry ISO-8601 submission timestamps,
so the IETF-only scope is also the cleanest with respect to date quality. The
corpus snapshot used throughout is \texttt{data/drafts.db} as of 2026-05-23.
\subsection{Thematic classification}
Each draft is assigned to one or more of eleven thematic categories
(Table~\ref{tab:categories}) by an LLM-assisted rating pipeline. The classifier
is prompted with the draft's title, metadata, and abstract (truncated to the
first 2000 characters), and returns a JSON object containing the category
assignment and five ordinal $1$--$5$ quality dimensions (novelty, maturity,
overlap, momentum, relevance). We emphasise two properties of this pipeline
that bear directly on interpretation:
\begin{itemize}[leftmargin=1.4em,topsep=2pt,itemsep=1pt]
\item \textbf{Abstract-only.} Ratings are derived from the abstract, not the
full document text. This keeps the pipeline cheap and uniform but means
classifications reflect how a draft \emph{presents} itself, not a full
reading of its mechanics.
\item \textbf{Model-derived.} Categories and scores are produced by a large
language model (Claude Sonnet, \texttt{claude-sonnet-4-20250514}).
Section~\ref{sec:reliability} quantifies how reproducible these labels
are; the headline result is that we trust the categories but not the
ordinal scores.
\end{itemize}
\subsection{Semantic embeddings}
For the redundancy analysis (Section~\ref{sec:redundancy}) each draft is
embedded into a 768-dimensional vector using the \texttt{nomic-embed-text}
model run locally via Ollama. Embedding coverage is complete (524/524). We
compute cosine similarity over all $\binom{524}{2}=137{,}026$ document pairs.
\subsection{Reproducibility}
The pipeline, queries, and the two-model re-rating used for the reliability
analysis are released as scripts (\texttt{scripts/survey-phase0.py},
\texttt{scripts/rerate-intercoder.py}, \texttt{scripts/survey-kappa.py}). The
re-rating itself was executed through the Anthropic Batch API at a total cost of
USD~2.41. Raw model outputs are released as line-delimited JSON.

137
workspace/drafts/landscape-survey/sections/04-landscape.tex Normal file
View File

@@ -0,0 +1,137 @@
\section{The Landscape}
\label{sec:landscape}
\subsection{Temporal dynamics}
\label{sec:temporal}
Figure~\ref{fig:temporal} shows monthly submission counts for the clean corpus.
Activity is negligible through most of 2024 (mean 3.7 drafts/month over June
2024--May 2025) and inflects sharply in late 2025: October 2025 jumps to 33
drafts, and monthly counts since June 2025 average 38.8. The peak month is
March 2026 with 106 drafts---roughly $35\times$ a typical 2024 month. We report
this as a peak-to-baseline ratio rather than an ``average growth rate'': the
distribution is dominated by a recent spike, not steady exponential growth. The
final two months (April--May 2026) dip relative to the March peak; because
drafts continue to be submitted and indexed, the tail of the curve should be
read as provisional rather than as a downturn.
\begin{figure}[h]
\centering
\begin{tikzpicture}
\node[left,font=\scriptsize] at (-0.35,2.0) {\rotatebox{90}{Drafts/month}};
\draw[gray!40] (-0.1,0.000) -- (11.60,0.000); \node[left,font=\scriptsize] at (-0.1,0.000) {0};
\draw[gray!40] (-0.1,0.943) -- (11.60,0.943); \node[left,font=\scriptsize] at (-0.1,0.943) {25};
\draw[gray!40] (-0.1,1.887) -- (11.60,1.887); \node[left,font=\scriptsize] at (-0.1,1.887) {50};
\draw[gray!40] (-0.1,2.830) -- (11.60,2.830); \node[left,font=\scriptsize] at (-0.1,2.830) {75};
\draw[gray!40] (-0.1,3.774) -- (11.60,3.774); \node[left,font=\scriptsize] at (-0.1,3.774) {100};
\fill[blue!60!black] (0.00,0) rectangle (0.30,0.151);
\fill[blue!60!black] (0.40,0) rectangle (0.70,0.075);
\fill[blue!60!black] (0.80,0) rectangle (1.10,0.038);
\fill[blue!60!black] (1.20,0) rectangle (1.50,0.189);
\fill[blue!60!black] (1.60,0) rectangle (1.90,0.113);
\fill[blue!60!black] (2.00,0) rectangle (2.30,0.038);
\fill[blue!60!black] (2.40,0) rectangle (2.70,0.075);
\fill[blue!60!black] (2.80,0) rectangle (3.10,0.038);
\fill[blue!60!black] (3.20,0) rectangle (3.50,0.377);
\fill[blue!60!black] (3.60,0) rectangle (3.90,0.038);
\fill[blue!60!black] (4.00,0) rectangle (4.30,0.189);
\fill[blue!60!black] (4.40,0) rectangle (4.70,0.113);
\fill[blue!60!black] (4.80,0) rectangle (5.10,0.189);
\fill[blue!60!black] (5.20,0) rectangle (5.50,0.038);
\fill[blue!60!black] (5.60,0) rectangle (5.90,0.075);
\fill[blue!60!black] (6.00,0) rectangle (6.30,0.264);
\fill[blue!60!black] (6.40,0) rectangle (6.70,0.226);
\fill[blue!60!black] (6.80,0) rectangle (7.10,0.151);
\fill[blue!60!black] (7.20,0) rectangle (7.50,0.226);
\fill[blue!60!black] (7.60,0) rectangle (7.90,0.226);
\fill[blue!60!black] (8.00,0) rectangle (8.30,0.302);
\fill[blue!60!black] (8.40,0) rectangle (8.70,1.245);
\fill[blue!60!black] (8.80,0) rectangle (9.10,0.830);
\fill[blue!60!black] (9.20,0) rectangle (9.50,0.415);
\fill[blue!60!black] (9.60,0) rectangle (9.90,1.925);
\fill[blue!60!black] (10.00,0) rectangle (10.30,2.491);
\fill[blue!60!black] (10.40,0) rectangle (10.70,4.000);
\fill[blue!60!black] (10.80,0) rectangle (11.10,2.943);
\fill[blue!60!black] (11.20,0) rectangle (11.50,2.792);
\node[below,font=\scriptsize] at (0.15,-0.05) {2024-01};
\node[below,font=\scriptsize] at (2.55,-0.05) {2024-07};
\node[below,font=\scriptsize] at (4.95,-0.05) {2025-01};
\node[below,font=\scriptsize] at (7.35,-0.05) {2025-07};
\node[below,font=\scriptsize] at (9.75,-0.05) {2026-01};
\node[below,font=\scriptsize] at (11.35,-0.05) {2026-05};
\end{tikzpicture}
\caption{Monthly counts of AI/agent IETF Internet-Drafts, January 2024--May
2026 ($n=524$). The April--May 2026 tail is provisional (indexing lag).}
\label{fig:temporal}
\end{figure}
\subsection{Category distribution}
\label{sec:categories}
Table~\ref{tab:categories} gives the distribution of drafts by primary
category. The field is concentrated in identity/authentication (141 drafts,
$26.9\%$), agent-to-agent communication protocols (108, $20.6\%$), and
autonomous network operations (64, $12.2\%$). The sparsest areas are
human-agent interaction (5), the residual ``Other'' bucket (9), and model
serving/inference (16); we report these descriptively and draw no normative
conclusion about whether sparse areas \emph{ought} to receive more attention.
Categories are not mutually exclusive: $92.9\%$ of drafts carry more than one
category, reflecting genuine thematic overlap (e.g.\ a draft may concern both
agent identity and authorization). We use the primary (first) category for the
distribution.
\begin{table}[h]
\centering
\begin{tabular}{lr}
\toprule
\textbf{Primary category} & \textbf{Drafts} \\
\midrule
Agent identity / authentication & 141 \\
Agent-to-agent (A2A) protocols & 108 \\
Autonomous network operations & 64 \\
ML traffic management & 48 \\
Data formats / interoperability & 44 \\
Agent discovery / registration & 35 \\
Policy / governance & 30 \\
AI safety / alignment & 24 \\
Model serving / inference & 16 \\
Other AI/agent & 9 \\
Human-agent interaction & 5 \\
\midrule
\textbf{Total} & \textbf{524} \\
\bottomrule
\end{tabular}
\caption{Distribution of the clean IETF corpus by primary category.
$92.9\%$ of drafts also carry one or more secondary categories.}
\label{tab:categories}
\end{table}
\subsection{Authorship and working-group structure}
\label{sec:authors}
The corpus is authored by 619 distinct individuals and is \emph{not}
concentrated: the ten most prolific authors together account for only $10.9\%$
of drafts. The most active contributors---Bing Liu (22 drafts), Nan Geng (21),
and Zhenbin Li (20)---form a cluster around autonomous network operations.
The strongest structural signal is the maturity of the work. Of the 524 drafts,
\textbf{456 ($87\%$) are individual submissions} not adopted by any IETF working
group; only 28 distinct working groups appear across the remainder. The area is
thus overwhelmingly \emph{pre-standardization}: a large volume of individual
proposals competing for attention, with comparatively little that has cleared
the bar of working-group adoption.
\subsection{Semantic redundancy}
\label{sec:redundancy}
Pairwise cosine similarity over the 137{,}026 document pairs has a mean of
$0.711$ (p90 $0.790$, p99 $0.850$, max $1.000$). We define a near-duplicate as a
pair with cosine $>0.9$: there are 125 such pairs, and \textbf{170 drafts
($32.4\%$) have at least one near-duplicate} in the corpus. The four pairs at
cosine $\approx 1.0$ are legitimate---they are individual versus
working-group-adopted versions of the same document (for example
\texttt{draft-fv-rats-ear} and \texttt{draft-ietf-rats-ear})---rather than data
artifacts. Combined with the dominance of individual submissions
(Section~\ref{sec:authors}), the high redundancy is consistent with a young,
crowded design space in which many authors propose overlapping solutions to the
same problems before consolidation occurs.

81
workspace/drafts/landscape-survey/sections/05-reliability.tex Normal file
View File

@@ -0,0 +1,81 @@
\section{How Reliable Are the Labels?}
\label{sec:reliability}
Every category in Section~\ref{sec:categories} is an LLM judgment. Before
treating the distribution as a finding, we ask how reproducible those judgments
are. We re-rated all 524 drafts a second time with the same pinned prompt using
two models---Claude Sonnet (\texttt{claude-sonnet-4-20250514}) and Claude Haiku
(\texttt{claude-haiku-4-5-20251001})---and compared the assignments. We report
Cohen's $\kappa$ for the nominal primary-category assignment and
quadratic-weighted $\kappa$ ($\kappa_w$) for the ordinal $1$--$5$ dimensions,
interpreting magnitudes by the Landis--Koch convention (\,$<0.2$ slight,
$0.2$--$0.4$ fair, $0.4$--$0.6$ moderate, $0.6$--$0.8$ substantial, $>0.8$
almost perfect\,).
\paragraph{Category assignment is substantially reliable.}
For the primary category, Sonnet and Haiku agree at $\kappa = 0.652$ (raw
agreement $70.8\%$). Each model independently re-rating also agrees
substantially with the original production labels (Sonnet $\kappa = 0.645$,
Haiku $\kappa = 0.596$). The residual disagreements are not random: they
concentrate on semantically adjacent categories
(Table~\ref{tab:confusion})---A2A protocols versus autonomous network
operations, A2A versus agent discovery, and identity versus the residual
``Other'' bucket. These are boundary cases, not classifier noise, which is why
we treat the distribution as informative while acknowledging $\pm$ a few points
of category-boundary uncertainty.
\begin{table}[h]
\centering
\begin{tabular}{llr}
\toprule
\textbf{Category A} & \textbf{Category B} & \textbf{Disagreements} \\
\midrule
A2A protocols & Autonomous netops & 17 \\
A2A protocols & Agent discovery/reg & 16 \\
A2A protocols & Agent identity/auth & 15 \\
Agent identity/auth & Other AI/agent & 14 \\
Data formats/interop & Other AI/agent & 10 \\
\bottomrule
\end{tabular}
\caption{Most-confused primary-category pairs between the two re-rating models.
Disagreements concentrate on semantic neighbours.}
\label{tab:confusion}
\end{table}
\paragraph{Ordinal quality scores are not reliable.}
The five $1$--$5$ quality dimensions tell a different and cautionary story
(Table~\ref{tab:kappa}). Between Sonnet and Haiku, \emph{overlap} reaches only
$\kappa_w = 0.127$ (slight---effectively no better than chance) and
\emph{novelty} $\kappa_w = 0.206$. \emph{relevance} appears substantial between
the two re-rating models ($0.728$) but collapses to $0.234$ against the
production labels, indicating it is not stable across runs either. Only
\emph{maturity} is consistently moderate ($0.59$--$0.62$).
\begin{table}[h]
\centering
\begin{tabular}{lcc}
\toprule
\textbf{Dimension} & \textbf{$\kappa_w$ (Sonnet vs.\ Haiku)} & \textbf{$\kappa_w$ (Sonnet vs.\ prod.)} \\
\midrule
relevance & 0.728 & 0.234 \\
maturity & 0.592 & 0.620 \\
momentum & 0.457 & 0.247 \\
novelty & 0.206 & 0.477 \\
overlap & 0.127 & 0.282 \\
\bottomrule
\end{tabular}
\caption{Quadratic-weighted $\kappa$ for the ordinal quality dimensions.
Low and inconsistent values---especially for \emph{overlap} and
\emph{novelty}---indicate these scores are not reproducible across raters.}
\label{tab:kappa}
\end{table}
\paragraph{Consequence.}
We therefore report the categorical landscape (Section~\ref{sec:categories}) but
deliberately exclude the LLM ordinal quality scores from our findings. We regard
this split as a general caution rather than an artifact of our particular
pipeline: LLMs can place standards documents into a thematic taxonomy with
substantial agreement, but asking them to score subjective qualities such as
``novelty'' or ``overlap'' on a numeric scale produces labels that do not
survive a change of model. Studies that use LLM-assigned quality scores as
quantitative evidence should report inter-rater reliability before doing so.

65
workspace/drafts/landscape-survey/sections/06-discussion.tex Normal file
View File

@@ -0,0 +1,65 @@
\section{Discussion and Limitations}
\label{sec:discussion}
\subsection{Implications}
Two structural features of the corpus reinforce one another. First, the area is
\emph{pre-standardization}: $456$ of the $524$ drafts ($87\%$) are individual
submissions not adopted by any working group (Section~\ref{sec:authors}).
Second, it is semantically redundant: $170$ drafts ($32.4\%$) have at least one
near-duplicate (cosine $>0.9$) elsewhere in the corpus
(Section~\ref{sec:redundancy}). Taken together these point to a young, crowded
design space in which many authors independently propose overlapping solutions
to the same problems. Such a configuration is consistent with the early phase of
a standards effort, where consolidation and competition between proposals have
yet to resolve into adopted work; it does not, on its own, indicate
duplication of effort in any pejorative sense, since some redundancy is the
expected by-product of parallel exploration. We note this dynamic descriptively
rather than predicting which proposals will prevail.
Beyond the IETF, our reliability result carries a general caution for
LLM-assisted corpus studies. The same pipeline that places documents into a
thematic taxonomy with substantial agreement (Cohen's $\kappa \approx 0.65$)
produces ordinal quality scores---``novelty,'' ``overlap''---that do not survive
a change of model ($\kappa_w$ as low as $0.13$). A study that reported the
category distribution and the quality scores side by side, without a reliability
check, would present a reproducible measurement and rater noise with equal
confidence. The discipline of separating the two, by an explicit inter-rater
analysis, is what allowed us to keep the former and discard the latter.
\subsection{Limitations}
Several limitations bound the interpretation of our findings.
\begin{itemize}[leftmargin=1.4em,topsep=2pt,itemsep=1pt]
\item \textbf{Abstract-only classification.} Categories and scores are derived
from each draft's title, metadata, and abstract, not its full text
(Section~\ref{sec:method}). Classifications therefore reflect how a draft
presents itself rather than a full reading of its mechanics, and a draft
whose abstract understates its technical content may be miscategorised.
\item \textbf{Single snapshot.} The analysis rests on one corpus snapshot
(\texttt{data/drafts.db} as of 2026-05-23). All counts, trends, and
similarities are as of that date and will drift as drafts are added,
revised, expired, or adopted.
\item \textbf{IETF-only scope.} We deliberately restrict the corpus to IETF
Internet-Drafts; documents from ISO, ITU, ETSI, NIST, and W3C are
excluded because of heterogeneous metadata and a different document
model. The survey therefore says nothing about AI-agent standardization
outside the IETF, and the overall scale of the field is correspondingly
understated.
\item \textbf{LLM-derived categories.} The thematic taxonomy is produced by a
large language model. The two-model $\kappa$ check
(Section~\ref{sec:reliability}) mitigates but does not eliminate this
dependence: substantial agreement still leaves category-boundary
uncertainty of a few points, concentrated on semantically adjacent
categories.
\item \textbf{Provisional recent tail.} Submission counts for the most recent
months (April--May 2026) are provisional because of indexing and fetch
lag; the dip after the March 2026 peak in Figure~\ref{fig:temporal}
should be read as incomplete data rather than as a downturn.
\item \textbf{Keyword-based candidate selection.} Candidates were identified by
keyword and topic matching, which can both miss relevant drafts whose
abstracts avoid the chosen vocabulary and over-include unrelated
documents; we observed a $12.2\%$ false-positive rate ($73$ of $597$
candidates) before filtering, and an unknown false-negative rate remains.
\end{itemize}

27
workspace/drafts/landscape-survey/sections/07-conclusion.tex Normal file
View File

@@ -0,0 +1,27 @@
\section{Conclusion and Future Work}
\label{sec:conclusion}
We have presented a verified quantitative survey of AI/agent-related IETF
Internet-Drafts, based on a curated corpus of $524$ documents spanning January
2024 to May 2026. The corpus exhibits a sharp recent surge---from $3.7$ to
$38.8$ drafts per month, peaking at $106$ in March 2026---and is dominated by
individual submissions ($87\%$ not adopted by any working group) with
substantial semantic redundancy ($32.4\%$ of drafts having a near-duplicate),
the signature of a young, pre-standardization design space. Crucially, we
treated the LLM-assisted labels themselves as objects of measurement: a
two-model re-rating shows that categorical assignment is substantially
reproducible (Cohen's $\kappa \approx 0.65$), while ordinal quality scores are
not ($\kappa_w = 0.13$--$0.21$ for the least stable dimensions), so we report
the category landscape and exclude the quality scores.
Several directions extend this work. \emph{Full-text classification} would
replace the abstract-only pipeline and test whether categories shift when the
classifier reads the complete document. \emph{Longitudinal re-runs} on later
snapshots would turn the single-snapshot picture into a moving record of how the
surge evolves and whether redundancy consolidates over time. \emph{Cross-SDO
extension} to ISO, ITU, ETSI, NIST, and W3C---contingent on reconciling their
heterogeneous metadata---would situate the IETF activity within the broader
standards landscape. Finally, \emph{tracking working-group adoption} of
individual drafts would reveal which of the many competing proposals clear the
bar from individual submission to adopted work, giving an empirical handle on
how the pre-standardization field resolves.