feat(survey): add IETF landscape survey (kappa, phase0, rerate), gaps update; bump wimse-ect; gitignore run logs

workspace/drafts/landscape-survey/sections/06-discussion.tex

@@ -0,0 +1,65 @@
+\section{Discussion and Limitations}
+\label{sec:discussion}
+
+\subsection{Implications}
+
+Two structural features of the corpus reinforce one another. First, the area is
+\emph{pre-standardization}: $456$ of the $524$ drafts ($87\%$) are individual
+submissions not adopted by any working group (Section~\ref{sec:authors}).
+Second, it is semantically redundant: $170$ drafts ($32.4\%$) have at least one
+near-duplicate (cosine $>0.9$) elsewhere in the corpus
+(Section~\ref{sec:redundancy}). Taken together these point to a young, crowded
+design space in which many authors independently propose overlapping solutions
+to the same problems. Such a configuration is consistent with the early phase of
+a standards effort, where consolidation and competition between proposals have
+yet to resolve into adopted work; it does not, on its own, indicate
+duplication of effort in any pejorative sense, since some redundancy is the
+expected by-product of parallel exploration. We note this dynamic descriptively
+rather than predicting which proposals will prevail.
+
+Beyond the IETF, our reliability result carries a general caution for
+LLM-assisted corpus studies. The same pipeline that places documents into a
+thematic taxonomy with substantial agreement (Cohen's $\kappa \approx 0.65$)
+produces ordinal quality scores---``novelty,'' ``overlap''---that do not survive
+a change of model ($\kappa_w$ as low as $0.13$). A study that reported the
+category distribution and the quality scores side by side, without a reliability
+check, would present a reproducible measurement and rater noise with equal
+confidence. The discipline of separating the two, by an explicit inter-rater
+analysis, is what allowed us to keep the former and discard the latter.
+
+\subsection{Limitations}
+
+Several limitations bound the interpretation of our findings.
+
+\begin{itemize}[leftmargin=1.4em,topsep=2pt,itemsep=1pt]
+  \item \textbf{Abstract-only classification.} Categories and scores are derived
+        from each draft's title, metadata, and abstract, not its full text
+        (Section~\ref{sec:method}). Classifications therefore reflect how a draft
+        presents itself rather than a full reading of its mechanics, and a draft
+        whose abstract understates its technical content may be miscategorised.
+  \item \textbf{Single snapshot.} The analysis rests on one corpus snapshot
+        (\texttt{data/drafts.db} as of 2026-05-23). All counts, trends, and
+        similarities are as of that date and will drift as drafts are added,
+        revised, expired, or adopted.
+  \item \textbf{IETF-only scope.} We deliberately restrict the corpus to IETF
+        Internet-Drafts; documents from ISO, ITU, ETSI, NIST, and W3C are
+        excluded because of heterogeneous metadata and a different document
+        model. The survey therefore says nothing about AI-agent standardization
+        outside the IETF, and the overall scale of the field is correspondingly
+        understated.
+  \item \textbf{LLM-derived categories.} The thematic taxonomy is produced by a
+        large language model. The two-model $\kappa$ check
+        (Section~\ref{sec:reliability}) mitigates but does not eliminate this
+        dependence: substantial agreement still leaves category-boundary
+        uncertainty of a few points, concentrated on semantically adjacent
+        categories.
+  \item \textbf{Provisional recent tail.} Submission counts for the most recent
+        months (April--May 2026) are provisional because of indexing and fetch
+        lag; the dip after the March 2026 peak in Figure~\ref{fig:temporal}
+        should be read as incomplete data rather than as a downturn.
+  \item \textbf{Keyword-based candidate selection.} Candidates were identified by
+        keyword and topic matching, which can both miss relevant drafts whose
+        abstracts avoid the chosen vocabulary and over-include unrelated
+        documents; we observed a $12.2\%$ false-positive rate ($73$ of $597$
+        candidates) before filtering, and an unknown false-negative rate remains.
+\end{itemize}