ietf-draft-analyzer/data/reports/methodology.md

# Methodology — IETF Draft Analyzer

*This document describes the data collection, analysis pipeline, and known limitations of the IETF Draft Analyzer project. It is intended to provide transparency for anyone evaluating the findings in the blog series or reports.*

---

## 1. Data Collection

### Source

All data is fetched from the IETF Datatracker API (`https://datatracker.ietf.org/api/v1/doc/document/`). Full draft text is retrieved from `https://www.ietf.org/archive/id/{name}-{rev}.txt`. Author and affiliation data comes from the `/api/v1/doc/documentauthor/` and `/api/v1/person/person/` endpoints.

### Keyword Selection

The corpus is built by searching for drafts matching 12 keywords across both `name__contains` and `abstract__contains` fields:

`agent`, `ai-agent`, `llm`, `autonomous`, `machine-learning`, `artificial-intelligence`, `mcp`, `agentic`, `inference`, `generative`, `intelligent`, `aipref`

Only drafts with `type__slug=draft` and submission date >= 2024-01-01 are included.

### Selection Bias Acknowledgment

Keyword-based selection introduces both false positives and false negatives:

- **False positives**: Keywords like "agent" match "user agent" in HTTP contexts, "autonomous" matches "autonomous systems" (AS) in routing, and "intelligent" matches "intelligent networking" unrelated to AI. We estimate 30-50 false positives remain in the corpus despite relevance filtering. Drafts with relevance score <= 2 are the most obvious, but some false positives receive relevance scores of 3-4 from the LLM judge.
- **False negatives**: Relevant drafts using terminology not in our keyword list (e.g., "cognitive," "self-driving network," or domain-specific terms) are missed entirely.
- **Temporal bias**: The `fetch_since` cutoff of 2024-01-01 excludes earlier foundational work that may inform the current landscape.

### Organization Normalization

Author affiliations are normalized using a hand-curated alias table of 40+ mappings (e.g., "Huawei Technologies Co., Ltd." -> "Huawei") plus automatic suffix stripping for common patterns (", Inc.", " LLC", " AB", etc.). This normalization is essential for cross-org analysis but introduces judgment calls about organizational boundaries.

---

## 2. Analysis Pipeline

The pipeline runs in six stages, each building on the previous:

```
fetch --> analyze --> embed --> ideas --> gaps --> report
  |         |          |        |        |         |
  v         v          v        v        v         v
Datatracker  Claude     Ollama   Claude   Claude   Markdown
  API       Sonnet   nomic-embed  Haiku   Sonnet   + rich
```

### Stage 1: Fetch
Retrieves draft metadata, full text, and author information from the Datatracker API with a 0.5-second polite delay between requests.

### Stage 2: Analyze (Rating)
Each draft is rated by Claude Sonnet on five dimensions using a compact structured prompt that includes the draft's name, title, date, page count, and abstract (truncated to 2000 characters). See "Rating Rubric" below.

### Stage 3: Embed
Vector embeddings are generated using Ollama with the `nomic-embed-text` model. The input combines the draft's title, abstract, and first 4000 characters of full text. Embeddings are 768-dimensional vectors stored as binary blobs in SQLite. See "Embedding Model" below.

### Stage 4: Ideas
Technical ideas are extracted by Claude. Individual extraction uses Sonnet with abstract + first 3000 characters of full text. Batch extraction uses Haiku with abstract only (truncated to 800 characters). The prompt requests 1-4 top-level novel contributions per draft.

### Stage 5: Gaps
A single Claude Sonnet call receives compressed landscape statistics (category counts, top ideas, overlap summary) and identifies 8-15 standardization gaps. See "Gap Analysis" below.

### Stage 6: Report
Markdown reports are generated from database queries. No LLM is involved in report generation.

### Caching

All Claude API calls are cached in an `llm_cache` table keyed by SHA-256 hash of the full prompt. Re-runs return cached results, making the pipeline idempotent. This also means that intra-rater consistency cannot be measured from cached results.

---

## 3. Rating Rubric

Each draft is scored on five dimensions, each on a 1-5 integer scale:

| Dimension | 1 | 2 | 3 | 4 | 5 |
|-----------|---|---|---|---|---|
| **Novelty** | Trivial/obvious extension | Incremental | Useful contribution | Notable originality | Genuinely novel approach |
| **Maturity** | Problem statement only | Early sketch | Defined protocol/mechanism | Detailed spec with examples | Implementation-ready with test vectors |
| **Overlap** | Unique approach | Minor similarities | Shares concepts with 1-2 drafts | Significant overlap | Near-duplicate of existing work |
| **Momentum** | Inactive/abandoned | Single revision | Active development | WG interest/adoption | Strong community momentum |
| **Relevance** | Not about AI/agents (false positive) | Tangentially related | Partially relevant | Directly relevant | Core AI agent topic |

### Composite Score

The composite score used in reports is a 4-dimension average of novelty, maturity, momentum, and relevance (excluding overlap, since overlap measures redundancy rather than quality). Exact decimal values are used; rounding is avoided.

### Scale Interpretation

Scores should be treated as **relative rankings within this corpus**, not absolute quality measures. Key limitations:

- **Abstract-only input**: Ratings are based on the draft's abstract (truncated to 2000 characters), not the full text. Maturity and overlap scores are particularly affected, since the abstract may not convey the full technical depth or specificity of the draft.
- **Single LLM judge**: Claude Sonnet is the sole rater. No human calibration study has been performed. No second-model comparison has been conducted. Even a small calibration set (20-30 drafts rated by domain experts) would substantially strengthen confidence.
- **No consistency measurement**: Each draft is rated once. The caching mechanism prevents re-rating, so Claude's self-consistency on these drafts is untested.
- **Overlap score limitations**: Claude rates each draft independently without access to the full corpus. The overlap dimension reflects Claude's general knowledge of the field, not corpus-specific similarity analysis. For corpus-level overlap, use the embedding-based similarity analysis instead.
- **Relevance inflation**: Keyword-matched drafts tend to score high on relevance by construction. The distribution is right-skewed (most drafts at 4-5).
- **Batch effects**: When rated in batches of 5 (using `BATCH_PROMPT`), position effects and comparison effects between drafts in the same batch are uncontrolled. Abstracts are truncated more aggressively (1500 chars) in batch mode.

---

## 4. Clustering

### Method

Greedy single-linkage clustering on the pairwise cosine similarity matrix of draft embeddings.

**Algorithm**: For each unvisited draft (seed), find all unvisited drafts with cosine similarity >= threshold to the seed. Add them to the seed's cluster and mark them visited. This produces disjoint clusters where every member is similar to the seed, but members are not guaranteed to be similar to each other (single-linkage property).

### Thresholds

| Threshold | Label | Justification |
|-----------|-------|---------------|
| 0.85 | Topically overlapping | Empirical: 0.80 produced too many false groupings; 0.90 missed obvious clusters; 0.85 yielded groups that looked reasonable on manual spot-checking |
| 0.90 | Near-duplicates | Empirical: pairs above 0.90 consistently covered the same topic with similar approaches |
| 0.98 | Functionally identical | Empirical: pairs above 0.98 were essentially the same document under different names |

**None of these thresholds are derived from a principled analysis.** A sensitivity analysis (running clustering at 0.80, 0.85, 0.90 and comparing results) would strengthen confidence. Different embedding models would produce different similarity distributions, potentially requiring different thresholds.

### Limitations

- **Single-linkage chaining**: A chain of pairwise-similar drafts can produce clusters containing semantically distant drafts connected through intermediaries.
- **No comparison to alternatives**: The clustering has not been compared against k-means, DBSCAN, hierarchical agglomerative clustering, or other standard methods.
- **General-purpose embeddings**: The `nomic-embed-text` model was not trained specifically for technical standards document similarity. Domain-specific or fine-tuned embeddings might produce significantly different cluster structures.
- **Inconsistent embedding input**: Drafts with full text available are embedded from title + abstract + 4000 chars of body. Drafts without full text are embedded from title + abstract only. This creates systematic quality differences in embeddings.

---

## 5. Gap Analysis

The gap analysis sends Claude Sonnet a compressed landscape summary containing:
- Category distribution (category name and draft count)
- Top 20 most frequently occurring idea titles
- Overlap summary (top 5 categories by count, labeled "high internal overlap")

Claude is instructed to identify 8-15 gaps with topic, description, category, severity (critical/high/medium/low), and evidence.

### Limitations

- **Single-shot generation**: Gaps are identified in one LLM call, not through systematic comparison against a reference taxonomy.
- **No reference architecture**: A rigorous gap analysis would compare the corpus against an explicit agent ecosystem reference model (e.g., NIST AI RMF, FIPA agent platform model). The current approach relies on Claude's general knowledge.
- **Circular overlap summary**: The overlap information fed to Claude is just category-level counts, not specific technical areas of overlap within categories.
- **Variable evidence quality**: Some gap evidence cites specific data ("only N drafts address X"), while other evidence is based on Claude's inference about what is missing.
- **Ungrounded severity**: The distinction between critical, high, medium, and low severity is assigned by Claude without defined thresholds.

### Strengthening Options

- Ground against a reference architecture (FIPA, NIST AI RMF, or a custom agent ecosystem model)
- Run multiple independent gap analyses and intersect results
- Have domain experts validate and rank gaps
- Cite specific drafts that partially address each gap

---

## 6. Embedding Model

**Model**: `nomic-embed-text` (Nomic AI), run locally via Ollama.

**Properties**:
- 768-dimensional embeddings
- Context window: ~8192 tokens
- General-purpose text embedding model trained on diverse English text
- Not fine-tuned for technical/standards document similarity

**Input**: Title + abstract + first 4000 characters of full text (when available), concatenated with double newlines. Input is truncated to ~32,000 characters before embedding.

**Similarity metric**: Cosine similarity, computed as dot product divided by product of L2 norms.

**Limitations**: As a general-purpose model, nomic-embed-text may not capture domain-specific semantic relationships in standards documents as well as a model fine-tuned on technical/legal/standards text. The embeddings have not been evaluated against a gold-standard similarity judgment set for IETF drafts.

---

## 7. Known Limitations Summary

| Limitation | Impact | Mitigation |
|------------|--------|------------|
| Abstract-only rating | Maturity/overlap scores may be unreliable | Could re-rate with full text for a validation sample |
| No human calibration | Rating validity is unknown | Calibration study with 5 experts on 25 drafts |
| Keyword selection bias | ~30-50 false positives, unknown false negatives | Relevance filtering, manual review of low-scoring drafts |
| Empirical clustering thresholds | Cluster boundaries may be arbitrary | Sensitivity analysis at multiple thresholds |
| Single-shot gap analysis | Gaps may be incomplete or misprioritized | Ground against reference architecture |
| General-purpose embeddings | Domain-specific similarity may be missed | Evaluate against expert similarity judgments |
| Batch vs. individual extraction quality | Idea counts and quality may vary by extraction method | Compare batch (Haiku) vs. individual (Sonnet) on sample |
| Organization normalization | Cross-org analysis depends on alias accuracy | Publish and review normalization table |

---

## 8. Related Work

The IETF's AI agent standardization effort exists within a broader ecosystem of agent-related standards and research. This analysis would benefit from comparison against:

### FIPA (Foundation for Intelligent Physical Agents)

The original agent communication standards body (1996-2005). FIPA's Agent Communication Language (ACL), Agent Management Specification, and Agent Platform specifications are the direct ancestors of modern A2A protocols. Key specifications include:
- **FIPA ACL** (SC00061): Message structure and performatives for agent communication
- **FIPA Agent Management** (SC00023): Agent platform architecture with Agent Management System (AMS), Directory Facilitator (DF), and Message Transport Service (MTS)
- **FIPA Interaction Protocols** (SC00026-SC00036): Request, query, contract net, brokering, and other standard interaction patterns

FIPA's work is relevant because many of the "novel" A2A protocol proposals in the IETF corpus address problems FIPA solved (or attempted to solve) 20+ years ago. The absence of FIPA references in most current drafts suggests a lack of awareness of prior art.

### IEEE P3394 — Standard for Trustworthy Autonomous and Semi-Autonomous Systems

An active IEEE working group developing standards for trustworthy AI agents, addressing trust, transparency, and accountability. Relevant to the IETF's AI safety/alignment and policy/governance categories. The IETF's safety deficit (4:1 capability-to-safety ratio) should be evaluated in the context of IEEE's complementary safety-focused standardization.

### W3C Web of Things (WoT)

The W3C WoT Architecture and Thing Description specifications address agent/device discovery and interoperability in IoT contexts:
- **WoT Architecture** (W3C REC): Defines servients, protocol bindings, and discovery mechanisms applicable to agent systems
- **WoT Thing Description** (W3C REC): Machine-readable metadata for capabilities, interfaces, and security -- analogous to agent capability description proposals in the IETF corpus

Several IETF drafts build on or compete with WoT concepts for agent discovery and description.

### Academic Multi-Agent Systems (MAS) Research

The multi-agent systems research community (AAMAS, JAIR, JAAMAS) has decades of work on problems the IETF drafts are now addressing at the protocol level:
- **Agent coordination**: Consensus, negotiation, auction mechanisms (relevant to IETF gap "Multi-Agent Consensus Protocols")
- **Trust and reputation**: Computational trust models, reputation systems (relevant to agent identity/auth drafts)
- **Agent verification**: Model checking, runtime verification of agent behavior (relevant to IETF gap "Agent Behavioral Verification")
- **MAS security**: Secure agent platforms, malicious agent detection (relevant to AI safety/alignment drafts)

Key survey references:
- Wooldridge, M. (2009). "An Introduction to MultiAgent Systems" -- Standard textbook covering agent architectures, communication, coordination
- Dorri, A., Kanhere, S.S., Jurdak, R. (2018). "Multi-Agent Systems: A Survey" -- IEEE Access, covering modern MAS challenges
- The AAMAS conference proceedings (annual since 2002) -- primary venue for MAS research

### Other Relevant Standards Bodies

- **OASIS**: TOSCA (Topology and Orchestration Specification for Cloud Applications) and prior work on service-oriented agent architectures
- **ITU-T**: Y.3170 series on machine learning in future networks, relevant to the autonomous netops and ML traffic management categories
- **ETSI**: ENI (Experiential Networked Intelligence) and ZSM (Zero-touch network and Service Management), addressing autonomous network management

---

*This document was created 2026-03-08 in response to scientific and statistical review findings. It should be updated as the analysis pipeline evolves.*