feat(survey): add IETF landscape survey (kappa, phase0, rerate), gaps update; bump wimse-ect; gitignore run logs

workspace/drafts/landscape-survey/sections/03-method.tex

@@ -0,0 +1,57 @@
+\section{Corpus and Method}
+\label{sec:method}
+
+\subsection{Corpus construction}
+
+Internet-Drafts were collected from the IETF Datatracker. Candidate documents
+were identified by keyword and topic matching for AI-agent and autonomous-system
+terminology (e.g.\ ``agent'', ``autonomous'', ``LLM'', ``inference''), then
+filtered to remove false positives---documents in which such terms occur in an
+unrelated sense (``user agent'', ``autonomous system'' in the BGP sense). Of
+597 candidate IETF documents, 73 ($12.2\%$) were flagged as false positives and
+excluded, yielding a clean corpus of \textbf{524 IETF Internet-Drafts}.
+
+We restrict this survey to IETF Internet-Drafts. The collection pipeline also
+ingests documents from other standards bodies (ISO, ITU, ETSI, NIST, W3C), but
+those exhibit heterogeneous and frequently missing publication dates and a
+different document model; mixing them would compromise the temporal and
+authorship analyses. All 524 IETF drafts carry ISO-8601 submission timestamps,
+so the IETF-only scope is also the cleanest with respect to date quality. The
+corpus snapshot used throughout is \texttt{data/drafts.db} as of 2026-05-23.
+
+\subsection{Thematic classification}
+
+Each draft is assigned to one or more of eleven thematic categories
+(Table~\ref{tab:categories}) by an LLM-assisted rating pipeline. The classifier
+is prompted with the draft's title, metadata, and abstract (truncated to the
+first 2000 characters), and returns a JSON object containing the category
+assignment and five ordinal $1$--$5$ quality dimensions (novelty, maturity,
+overlap, momentum, relevance). We emphasise two properties of this pipeline
+that bear directly on interpretation:
+
+\begin{itemize}[leftmargin=1.4em,topsep=2pt,itemsep=1pt]
+  \item \textbf{Abstract-only.} Ratings are derived from the abstract, not the
+        full document text. This keeps the pipeline cheap and uniform but means
+        classifications reflect how a draft \emph{presents} itself, not a full
+        reading of its mechanics.
+  \item \textbf{Model-derived.} Categories and scores are produced by a large
+        language model (Claude Sonnet, \texttt{claude-sonnet-4-20250514}).
+        Section~\ref{sec:reliability} quantifies how reproducible these labels
+        are; the headline result is that we trust the categories but not the
+        ordinal scores.
+\end{itemize}
+
+\subsection{Semantic embeddings}
+
+For the redundancy analysis (Section~\ref{sec:redundancy}) each draft is
+embedded into a 768-dimensional vector using the \texttt{nomic-embed-text}
+model run locally via Ollama. Embedding coverage is complete (524/524). We
+compute cosine similarity over all $\binom{524}{2}=137{,}026$ document pairs.
+
+\subsection{Reproducibility}
+
+The pipeline, queries, and the two-model re-rating used for the reliability
+analysis are released as scripts (\texttt{scripts/survey-phase0.py},
+\texttt{scripts/rerate-intercoder.py}, \texttt{scripts/survey-kappa.py}). The
+re-rating itself was executed through the Anthropic Batch API at a total cost of
+USD~2.41. Raw model outputs are released as line-delimited JSON.