58 lines
3.0 KiB
TeX
58 lines
3.0 KiB
TeX
\section{Corpus and Method}
|
|
\label{sec:method}
|
|
|
|
\subsection{Corpus construction}
|
|
|
|
Internet-Drafts were collected from the IETF Datatracker. Candidate documents
|
|
were identified by keyword and topic matching for AI-agent and autonomous-system
|
|
terminology (e.g.\ ``agent'', ``autonomous'', ``LLM'', ``inference''), then
|
|
filtered to remove false positives---documents in which such terms occur in an
|
|
unrelated sense (``user agent'', ``autonomous system'' in the BGP sense). Of
|
|
597 candidate IETF documents, 73 ($12.2\%$) were flagged as false positives and
|
|
excluded, yielding a clean corpus of \textbf{524 IETF Internet-Drafts}.
|
|
|
|
We restrict this survey to IETF Internet-Drafts. The collection pipeline also
|
|
ingests documents from other standards bodies (ISO, ITU, ETSI, NIST, W3C), but
|
|
those exhibit heterogeneous and frequently missing publication dates and a
|
|
different document model; mixing them would compromise the temporal and
|
|
authorship analyses. All 524 IETF drafts carry ISO-8601 submission timestamps,
|
|
so the IETF-only scope is also the cleanest with respect to date quality. The
|
|
corpus snapshot used throughout is \texttt{data/drafts.db} as of 2026-05-23.
|
|
|
|
\subsection{Thematic classification}
|
|
|
|
Each draft is assigned to one or more of eleven thematic categories
|
|
(Table~\ref{tab:categories}) by an LLM-assisted rating pipeline. The classifier
|
|
is prompted with the draft's title, metadata, and abstract (truncated to the
|
|
first 2000 characters), and returns a JSON object containing the category
|
|
assignment and five ordinal $1$--$5$ quality dimensions (novelty, maturity,
|
|
overlap, momentum, relevance). We emphasise two properties of this pipeline
|
|
that bear directly on interpretation:
|
|
|
|
\begin{itemize}[leftmargin=1.4em,topsep=2pt,itemsep=1pt]
|
|
\item \textbf{Abstract-only.} Ratings are derived from the abstract, not the
|
|
full document text. This keeps the pipeline cheap and uniform but means
|
|
classifications reflect how a draft \emph{presents} itself, not a full
|
|
reading of its mechanics.
|
|
\item \textbf{Model-derived.} Categories and scores are produced by a large
|
|
language model (Claude Sonnet, \texttt{claude-sonnet-4-20250514}).
|
|
Section~\ref{sec:reliability} quantifies how reproducible these labels
|
|
are; the headline result is that we trust the categories but not the
|
|
ordinal scores.
|
|
\end{itemize}
|
|
|
|
\subsection{Semantic embeddings}
|
|
|
|
For the redundancy analysis (Section~\ref{sec:redundancy}) each draft is
|
|
embedded into a 768-dimensional vector using the \texttt{nomic-embed-text}
|
|
model run locally via Ollama. Embedding coverage is complete (524/524). We
|
|
compute cosine similarity over all $\binom{524}{2}=137{,}026$ document pairs.
|
|
|
|
\subsection{Reproducibility}
|
|
|
|
The pipeline, queries, and the two-model re-rating used for the reliability
|
|
analysis are released as scripts (\texttt{scripts/survey-phase0.py},
|
|
\texttt{scripts/rerate-intercoder.py}, \texttt{scripts/survey-kappa.py}). The
|
|
re-rating itself was executed through the Anthropic Batch API at a total cost of
|
|
USD~2.41. Raw model outputs are released as line-delimited JSON.
|