Platform upgrade: semantic search, citations, readiness, tests, Docker

Major features added by 5 parallel agent teams:
- Semantic "Ask" (NL queries via FTS5 + embeddings + Claude synthesis)
- Global search across drafts, ideas, authors, gaps
- REST API expansion (14 endpoints, up from 3) with CSV/JSON export
- Citation graph visualization (D3.js, 440 nodes, 2422 edges)
- Standards readiness scoring (0-100 composite from 6 factors)
- Side-by-side draft comparison view with shared/unique analysis
- Annotation system (notes + tags per draft, DB-persisted)
- Docker deployment (Dockerfile + docker-compose with Ollama)
- Scheduled updates (cron script with log rotation)
- Pipeline health dashboard (stage progress bars, cost tracking)
- Test suite foundation (54 pytest tests covering DB, models, web data)

Fixes: compare_drafts() stubbed→working, get_authors_for_draft() bug,
source-aware analysis prompts, config env var overrides + validation,
resilient batch error handling with --retry-failed, observatory --dry-run

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

.dockerignore

@@ -0,0 +1,11 @@
+.git
+__pycache__
+*.pyc
+.env
+paper/
+.claude/
+*.egg-info
+.mypy_cache
+.pytest_cache
+data/ietf_drafts.db
+docs/

Dockerfile

@@ -0,0 +1,24 @@
+FROM python:3.11-slim
+
+WORKDIR /app
+
+# Install build dependencies
+RUN apt-get update && apt-get install -y --no-install-recommends \
+    gcc \
+    && rm -rf /var/lib/apt/lists/*
+
+# Copy project metadata and source first for layer caching
+COPY pyproject.toml .
+COPY src/ src/
+
+# Install the package and all dependencies (including flask)
+RUN pip install --no-cache-dir .
+
+# Copy data directory (DB, config, reports)
+COPY data/ data/
+
+ENV PYTHONUNBUFFERED=1
+
+EXPOSE 5000
+
+CMD ["python", "src/webui/app.py"]

README.md

@@ -45,8 +45,34 @@ ietf viz all
 
 # Open the interactive browser
 xdg-open data/figures/browser.html
+
+# Launch the web dashboard
+./scripts/run-webui.sh
 ```
 
+## Web Dashboard
+
+A full interactive dashboard at `http://127.0.0.1:5000` with 8 pages:
+
+```bash
+# Start the dashboard
+./scripts/run-webui.sh
+# or: python src/webui/app.py
+```
+
+| Page | What it shows |
+|------|---------------|
+| **Overview** | Stat cards, score histogram, category donut, submission timeline, category radar |
+| **Draft Explorer** | Searchable/filterable/sortable table of all drafts with category pills and score badges |
+| **Draft Detail** | Individual draft view with score ring, dimension bars, ideas, references, and linked authors |
+| **Ratings** | Score distributions, dimension box plots, category radar, novelty vs maturity scatter, top-20 leaderboard |
+| **Landscape** | t-SNE embedding map, quality quadrants, violin plots by category |
+| **Authors** | Co-authorship force-directed graph, organization charts, cross-org collaboration |
+| **Ideas** | Extracted ideas grouped by type with search |
+| **Gaps** | Gap cards sorted by severity with links to related drafts |
+
+Charts are interactive (Plotly.js) — click data points to navigate to draft details, click categories to filter.
+
 ## CLI Commands
 
 ### Core Pipeline
@@ -182,6 +208,7 @@ The safety deficit is the most striking finding — only **12.3%** of categorize
 - **SQLite** with FTS5 full-text search and WAL mode
 - **Anthropic Claude** (Sonnet 4) for analysis, rating, idea extraction, gap analysis
 - **Ollama** (nomic-embed-text) for local embeddings and similarity
+- **Flask** with Jinja2 for the interactive web dashboard
 - **Plotly** for interactive HTML visualizations
 - **Matplotlib/Seaborn** for publication-ready static figures
 - **NetworkX** for author collaboration graph analysis
@@ -203,6 +230,11 @@ src/ietf_analyzer/
     draftgen.py     # Internet-Draft generation from gap analysis
     config.py       # Configuration with defaults
 
+src/webui/
+    app.py          # Flask application with all routes
+    data.py         # Data access layer (stats, filtering, t-SNE, network graphs)
+    templates/      # Jinja2 templates (base + 8 page templates)
+
 data/
     drafts.db       # SQLite database (all analysis data)
     reports/        # Generated markdown reports

data/reports/dev-journal.md

@@ -4,6 +4,19 @@
 
 ---
 
+### 2026-03-07 CODER C — Citation Graph, Readiness Scoring, Annotations, Data Surfacing
+
+**What**: Implemented four features in a single session:
+1. **Citation Graph Visualization** (`/citations`): D3.js force-directed graph showing cross-references between drafts and RFCs. Nodes colored by type (blue=draft, orange=RFC), sized by influence (in-degree). Includes category filter, min-refs slider, hover tooltips, click-to-navigate, and a top-referenced RFCs table. New `get_citation_graph()` in data.py, route + API endpoint in app.py.
+2. **Standards Readiness Scoring**: New `readiness.py` module computing a 0-100 composite score from 6 weighted factors (WG adoption 25%, revision maturity 15%, reference density 15%, cited-by count 15%, author experience 15%, momentum rating 15%). Displayed as a progress gauge on draft detail pages, added as sortable column on drafts listing, and shown in `ietf show` CLI output.
+3. **Annotation System**: New `annotations` table in DB schema with `upsert_annotation`, `get_annotation`, `get_all_annotations`, `search_by_tag` methods. New `ietf annotate` CLI command with `--note`, `--tag`, `--remove-tag` options. Web UI: inline note editor + tag chips with add/remove on draft detail page, backed by POST `/api/drafts/<name>/annotate` endpoint.
+4. **Surface Underutilized Data**: Exposed `novelty_score` (from pipeline/quality.py) in ideas.html and draft_detail.html as color-coded N:X badges. Gap severity now sorts critical-first (was alphabetical). `all_ideas()` and `get_ideas_for_draft()` now return `novelty_score` field.
+
+**Why**: These features leverage existing data (4231 refs, novelty scores, severity) that was computed but never surfaced to users. Readiness scoring gives a quick at-a-glance RFC proximity signal. Annotations enable user workflow.
+**Result**: 8 files modified (db.py, data.py, app.py, cli.py, base.html, draft_detail.html, ideas.html, drafts.html, gaps.html), 2 files created (readiness.py, citations.html). Citations link added to sidebar nav.
+
+---
+
 ### 2026-03-06 CODER — Interactive D3.js Author Network Visualization
 
 **What**: Replaced the Plotly spring-layout co-authorship graph on `/authors` with a full D3.js v7 force-directed network. Added enriched data layer (`get_author_network_full`) with avg draft scores per author, connected-component cluster detection (68 clusters found), and a new `/api/authors/network` JSON endpoint. Template now includes: interactive D3 force graph with zoom/pan/drag, org filter dropdown, cluster highlighting with zoom-to-fit, hover tooltips showing author details + draft list, click-to-navigate, plus the existing Plotly org bar chart, cross-org collaboration chart, sortable authors table (now top 50), and org stats sidebar.
@@ -411,3 +424,23 @@
 - Decisions made: **GitHub Pages** for publication, **staggered 1/day** cadence, **MIT license**
 - Agent utilization: Architect (2 tasks, shut down), Writer (2 tasks, shut down), Planner (1 task, shut down)
 **Surprise**: The crash recovery was seamless — the dev journal served exactly its intended purpose. Every agent could read the journal and understand the full state without any human explanation. The journal-as-coordination-mechanism is the strongest vindication of the CLAUDE.md journaling requirement. This should feature prominently in Post 8.
+
+### 2026-03-07 CODER E — W3C Integration, Docker, Scheduling, Pipeline Health
+
+**What**: Four-part infrastructure sprint to make the platform multi-source, self-running, and deployable:
+
+1. **W3C Integration** — Wired the existing W3C fetcher (`sources/w3c.py`) into the full pipeline. Made analysis prompts source-aware (`_doc_type_label()` returns "IETF draft" or "W3C specification" based on `source` field). Added source filter dropdown (IETF / W3C / All) to the Draft Explorer web UI with colored source badges (blue for IETF, green for W3C). Updated `get_drafts_page()` to accept `source` parameter. All pagination and sort links preserve source filter state. Config documents how to enable W3C: `ietf observatory update --source w3c` or add `"w3c"` to `observatory_sources` in config.json.
+
+2. **Scheduled Updates** — Created `scripts/scheduled-update.sh` for cron-based automation. Handles .env loading, log rotation (30 days), and proper error exit codes. Usage: `crontab -e -> 0 6 * * * /path/to/scheduled-update.sh`
+
+3. **Docker Deployment** — Created `Dockerfile` (python:3.11-slim), `docker-compose.yml` (web + ollama services with volume mounts for data persistence), and `.dockerignore`. One-command deployment: `docker compose up`.
+
+4. **Pipeline Health** — Enhanced `ietf pipeline status` to show comprehensive health: processing stage breakdown (rated/embedded/ideas with ASCII progress bars), total ideas, gaps, API token usage, estimated cost. Enhanced monitor web page with visual pipeline progress bars, cost tracking panel, and document/idea/gap counts. Added `--dry-run` flag to `ietf observatory update` that previews what would happen. Wrapped all observatory update steps in try/except for graceful error recovery — failures in one stage no longer block subsequent stages.
+
+**Why**: The platform was IETF-only despite having a complete W3C fetcher. Docker makes deployment reproducible. Scheduled updates make it self-running. Error recovery prevents partial failures from wasting an entire update cycle.
+
+**Result**:
+- Files modified: `analyzer.py`, `observatory.py`, `cli.py`, `config.py`, `data.py`, `app.py`, `drafts.html`, `monitor.html`
+- Files created: `Dockerfile`, `docker-compose.yml`, `.dockerignore`, `scripts/scheduled-update.sh`
+- All Python files compile cleanly
+- No breaking changes to existing IETF-only workflows

data/reports/platform-improvement-plan.md

@@ -0,0 +1,182 @@
+# Platform Improvement Plan: IETF Draft Analyzer → Standards Intelligence Platform
+
+*Generated 2026-03-07 — Based on full codebase audit and architectural analysis*
+
+---
+
+## Current State Summary
+
+| Dimension | What Exists | Assessment |
+|-----------|-------------|------------|
+| **Data** | 434 drafts, 557 authors, 419 ideas, 11 gaps, 4231 refs to 694 RFCs | Strong foundation |
+| **CLI** | 20+ commands (fetch, analyze, embed, ideas, gaps, report, viz, wg, etc.) | Feature-rich |
+| **Web UI** | 16 Flask pages with D3.js/Plotly visualizations | Good but disconnected |
+| **Pipeline** | Observatory class with multi-source support | Built but manual |
+| **Multi-SDO** | IETF complete, W3C fetcher written but unused | Partially built |
+| **Tests** | Zero tests | Critical gap |
+| **Deployment** | Manual Python setup | No containerization |
+
+## The Transformation
+
+**From**: A powerful CLI analysis tool that an expert runs manually
+**To**: A living intelligence platform that monitors, alerts, and answers questions
+
+---
+
+## Phase 1: Quick Wins (All Parallelizable)
+
+### 1.1 Global Search [S: 3-5h]
+Add a unified search bar to the web UI that queries across drafts (FTS5), ideas, authors, and gaps simultaneously. Currently search is isolated to the drafts page.
+
+- **Files**: `src/webui/app.py` (new `/search` route), `src/webui/data.py` (new `global_search()`), `src/webui/templates/base.html` (search in sidebar), new `search_results.html`
+- **Why first**: Most common user action on any data platform
+
+### 1.2 REST API [S: 2-3h]
+Expose all existing `data.py` functions as JSON endpoints. Only 3 API endpoints exist today — extend to all 15+ data views.
+
+- **Files**: `src/webui/app.py` (add `/api/ideas`, `/api/gaps`, `/api/ratings`, `/api/timeline`, `/api/landscape`, `/api/similarity`, `/api/drafts/<name>`, etc.)
+- **Why**: Enables programmatic access, third-party tools, decouples data from presentation
+
+### 1.3 Export (CSV/JSON) [S: 3-4h]
+Add export buttons to web UI pages + `ietf export` CLI command.
+
+- **Files**: `src/ietf_analyzer/cli.py` (new `export` command), API endpoints get `?format=csv` support
+- **Depends on**: 1.2
+
+### 1.4 Annotation System [M: 4-6h]
+Add private notes and custom tags per draft, persisted in DB.
+
+- **Files**: `src/ietf_analyzer/db.py` (new `annotations` table), `src/webui/app.py` (POST endpoint), `src/webui/templates/draft_detail.html` (inline edit), `src/ietf_analyzer/cli.py` (new `annotate` command)
+- **Why**: Analysts need to layer their own context onto the data
+
+### 1.5 Test Suite Foundation [M: 6-8h]
+Create pytest infrastructure with ~30 tests covering DB layer, models, and web data functions.
+
+- **Files**: new `tests/conftest.py`, `tests/test_db.py`, `tests/test_models.py`, `tests/test_data.py`, update `pyproject.toml`
+- **Why**: Zero tests today. Every future change benefits from a safety net
+
+---
+
+## Phase 2: Core Platform
+
+### 2.1 Semantic Search / "Ask" [M: 1-2 days] — THE SIGNATURE FEATURE
+Natural language queries: "Which drafts address agent authentication?" → synthesized answer with citations.
+
+- Embed the query via Ollama, compute cosine similarity against all 434 draft embeddings
+- Merge with FTS5 keyword results using reciprocal rank fusion
+- Optionally synthesize answer via Claude with top-K context
+- **Files**: new `src/ietf_analyzer/search.py`, `src/ietf_analyzer/cli.py` (`ietf ask`), `src/webui/app.py` (`/ask` route), new template
+
+### 2.2 Competitive Landscape Mapping [M: 1-2 days]
+Auto-detect and compare competing proposals in the same problem space.
+
+- Group drafts by similarity clusters, enrich with rating comparisons and WG adoption status
+- Show head-to-head comparisons: where they agree, where they diverge
+- **Files**: new `src/ietf_analyzer/competition.py`, `src/webui/app.py` (`/competition` route), new template
+
+### 2.3 Standards Readiness Scoring [M: 1 day]
+Composite 0-100 "readiness" score: WG adoption, revision count, reference density, cited-by count, author track record.
+
+- **Files**: new `src/ietf_analyzer/readiness.py`, update `src/webui/templates/draft_detail.html` (gauge chart), update drafts listing
+
+### 2.4 Scheduled Updates + Pipeline Health [M: 1 day]
+Cron-based auto-fetch using existing Observatory, plus monitoring dashboard.
+
+- **Files**: new `scripts/scheduled-update.sh`, enhance `src/webui/templates/monitor.html` (stage breakdown, cost tracking, failure log)
+
+### 2.5 Comparison View [M: 1 day]
+Side-by-side comparison of 2+ drafts with rating radar overlay, shared/unique ideas, shared/unique references.
+
+- **Files**: `src/webui/app.py` (`/compare?drafts=...`), `src/webui/data.py` (`get_comparison_data()`), new template, add checkboxes to drafts listing
+
+---
+
+## Phase 3: Intelligence
+
+### 3.1 Trend Forecasting [L: 2-3 days]
+Predict which areas will grow based on submission velocity, revision activity, WG adoption signals.
+
+- Per-category momentum signals, linear/exponential extrapolation, "Hot/Cooling/Emerging" tags
+- **Files**: new `src/ietf_analyzer/forecasting.py`, new web page `/trends`
+
+### 3.2 Change Detection & Diffing [L: 2-3 days]
+Track what changed between draft revisions, summarize changes via Claude.
+
+- New `draft_revisions` table to archive old versions
+- Section-level diff with Claude-generated change summaries
+- **Files**: new `src/ietf_analyzer/diff.py`, update `src/ietf_analyzer/db.py`, `src/ietf_analyzer/fetcher.py`
+- **Depends on**: 2.4 (scheduled updates to catch new revisions)
+
+### 3.3 Citation Graph [M: 1-2 days]
+Visual dependency tree from the 4231 existing cross-references to 694 RFCs.
+
+- D3.js force-directed graph (pattern from authors.html), PageRank-style influence scores
+- **Files**: `src/webui/data.py` (`get_citation_graph()`), new template `citations.html`
+
+### 3.4 Newsletter Generation [M: 1-2 days]
+Automated weekly/monthly digest with new drafts, significant changes, trend shifts.
+
+- **Files**: new `src/ietf_analyzer/newsletter.py`, `src/ietf_analyzer/cli.py` (`ietf newsletter`)
+- **Depends on**: 3.2 (for change content)
+
+---
+
+## Phase 4: Scale
+
+### 4.1 Docker Deployment [S: 3-4h]
+Dockerfile + docker-compose.yml with Flask app + Ollama.
+
+### 4.2 Complete W3C Integration [M: 1 day]
+Wire existing W3C fetcher into Observatory pipeline, make prompts source-aware, add source filter to web UI.
+
+### 4.3 IEEE + 3GPP Sources [L: 3-5 days]
+New source fetchers following `SourceFetcher` protocol. Depends on 4.2 validating the pipeline.
+
+### 4.4 Cross-SDO Analysis [L: 2-3 days]
+Compare work across standards bodies. Embedding similarity between IETF/W3C/IEEE specs, gap analysis for topics only one body covers.
+
+### 4.5 Plugin Architecture [L: 2-3 days]
+Formalize extension points for sources, analyzers, report types via Python entry points.
+
+---
+
+## Recommended Agent Team Assignments
+
+### Immediate Sprint (Phase 1 — all in parallel)
+
+| Agent | Items | Focus |
+|-------|-------|-------|
+| **Coder A** | 1.1 + 1.2 | Global search + REST API |
+| **Coder B** | 1.4 + 1.3 | Annotations + Export |
+| **Coder C** | 1.5 | Test suite foundation |
+
+### Next Sprint (Phase 2 — mostly parallel)
+
+| Agent | Items | Focus |
+|-------|-------|-------|
+| **Coder A** | 2.1 | Semantic search / Ask (signature feature) |
+| **Coder B** | 2.2 + 2.5 | Competition mapping + Comparison view |
+| **Coder C** | 2.3 + 2.4 | Readiness scoring + Pipeline health |
+
+---
+
+## Impact vs Effort Matrix
+
+```
+HIGH IMPACT
+    |
+    |  2.1 Ask          3.2 Diffing
+    |  1.1 Search       3.1 Forecasting
+    |  2.2 Competition  4.4 Cross-SDO
+    |  2.5 Compare
+    |  1.2 API
+    |  2.3 Readiness    3.3 Citations
+    |  1.4 Annotations  3.4 Newsletter
+    |  1.3 Export       4.2 W3C
+    |  1.5 Tests        4.1 Docker
+    |                   4.5 Plugins
+    |                   4.3 IEEE/3GPP
+LOW IMPACT
+    +------------------------------------------
+    LOW EFFORT                      HIGH EFFORT
+```

docker-compose.yml

@@ -0,0 +1,26 @@
+version: '3.8'
+
+services:
+  web:
+    build: .
+    ports:
+      - "5000:5000"
+    volumes:
+      - ./data:/app/data
+    environment:
+      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
+      - IETF_ANALYZER_OLLAMA_URL=http://ollama:11434
+    depends_on:
+      - ollama
+    restart: unless-stopped
+
+  ollama:
+    image: ollama/ollama
+    volumes:
+      - ollama_data:/root/.ollama
+    ports:
+      - "11434:11434"
+    restart: unless-stopped
+
+volumes:
+  ollama_data:

pyproject.toml

@@ -22,10 +22,17 @@ dependencies = [
     "scikit-learn>=1.3",
     "networkx>=3.2",
     "markdown>=3.5",
+    "flask>=3.0",
 ]
 
+[project.optional-dependencies]
+test = ["pytest", "pytest-cov"]
+
 [project.scripts]
 ietf = "ietf_analyzer.cli:main"
 
 [tool.setuptools.packages.find]
 where = ["src"]
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]

scripts/scheduled-update.sh

@@ -0,0 +1,61 @@
+#!/bin/bash
+# Scheduled observatory update -- run via cron
+# Usage: crontab -e -> 0 6 * * * /path/to/scheduled-update.sh
+#
+# This script runs the full observatory update cycle:
+#   1. Fetch new docs from all enabled sources (IETF, W3C)
+#   2. Analyze unrated documents with Claude
+#   3. Generate embeddings with Ollama
+#   4. Extract ideas from new documents
+#   5. Re-run gap analysis if enough new docs
+#
+# Logs are saved to data/logs/update-YYYYMMDD-HHMMSS.log
+set -euo pipefail
+
+cd "$(dirname "$0")/.."
+
+LOG_DIR="data/logs"
+mkdir -p "$LOG_DIR"
+LOG_FILE="$LOG_DIR/update-$(date +%Y%m%d-%H%M%S).log"
+
+echo "Starting scheduled update at $(date)" | tee "$LOG_FILE"
+
+# Load environment (API keys, etc.)
+if [ -f .env ]; then
+    set -a
+    source .env
+    set +a
+fi
+
+# Run the observatory update (delta mode -- only fetch new docs)
+python -c "
+import sys
+sys.path.insert(0, 'src')
+from ietf_analyzer.observatory import Observatory
+from ietf_analyzer.analyzer import Analyzer
+from ietf_analyzer.config import Config
+from ietf_analyzer.db import Database
+
+config = Config.load()
+db = Database(config)
+analyzer = Analyzer(config, db)
+obs = Observatory(config, db, analyzer)
+
+try:
+    result = obs.update()
+    print(f'Results: {result}')
+except Exception as e:
+    print(f'ERROR: {e}', file=sys.stderr)
+    sys.exit(1)
+finally:
+    db.close()
+" >> "$LOG_FILE" 2>&1
+
+EXIT_CODE=$?
+
+echo "Completed at $(date) (exit code: $EXIT_CODE)" | tee -a "$LOG_FILE"
+
+# Clean up old logs (keep last 30 days)
+find "$LOG_DIR" -name "update-*.log" -mtime +30 -delete 2>/dev/null || true
+
+exit $EXIT_CODE

src/ietf_analyzer/analyzer.py

@@ -38,7 +38,7 @@ CATEGORIES_SHORT = [
 
 # Compact prompt — abstract only, saves ~10x tokens vs full-text
 RATE_PROMPT_COMPACT = """\
-Rate this IETF draft. JSON only.
+Rate this {doc_type}. JSON only.
 
 {name} | {title} | {time} | {pages}pg
 Abstract: {abstract}
@@ -51,7 +51,7 @@ JSON only, no fences."""
 
 # Batch prompt — rate multiple drafts in one call
 BATCH_PROMPT = """\
-Rate each IETF draft below. Return a JSON array with one object per draft, in order.
+Rate each document below. Return a JSON array with one object per draft, in order.
 
 {drafts_block}
 
@@ -62,14 +62,14 @@ Categories: {categories}
 Return ONLY a JSON array, no fences."""
 
 COMPARE_PROMPT = """\
-Compare these IETF drafts — overlaps, unique ideas, complementary vs competing vs redundant.
+Compare these documents — overlaps, unique ideas, complementary vs competing vs redundant.
 
 {drafts_section}
 
 Be specific about concrete mechanisms and design choices."""
 
 EXTRACT_IDEAS_PROMPT = """\
-Extract discrete technical ideas and mechanisms from this IETF draft.
+Extract discrete technical ideas and mechanisms from this {doc_type}.
 Return a JSON array. Each element: {{"title":"short name","description":"1-2 sentences","type":"mechanism|protocol|pattern|requirement|architecture|extension"}}
 
 {name} | {title} | {pages}pg
@@ -81,7 +81,7 @@ Return 1-4 ideas. Extract only TOP-LEVEL novel contributions. Do NOT list sub-fe
 JSON array only, no fences."""
 
 BATCH_IDEAS_PROMPT = """\
-Extract ideas from each IETF draft below. Return a JSON object mapping draft name -> array of ideas.
+Extract ideas from each document below. Return a JSON object mapping document name -> array of ideas.
 Per idea: {{"title":"short name","description":"1 sentence","type":"mechanism|protocol|pattern|requirement|architecture|extension"}}
 
 {drafts_block}
@@ -135,6 +135,15 @@ def _prompt_hash(text: str) -> str:
     return hashlib.sha256(text.encode()).hexdigest()[:16]
 
 
+def _doc_type_label(source: str) -> str:
+    """Return a human-readable document type based on source."""
+    labels = {
+        "ietf": "IETF draft",
+        "w3c": "W3C specification",
+    }
+    return labels.get(source, f"{source} document")
+
+
 class Analyzer:
     def __init__(self, config: Config | None = None, db: Database | None = None):
         self.config = config or Config.load()
@@ -199,6 +208,7 @@ class Analyzer:
             return None
 
         prompt = RATE_PROMPT_COMPACT.format(
+            doc_type=_doc_type_label(draft.source),
             name=draft.name, title=draft.title, time=draft.date,
             pages=draft.pages or "?",
             abstract=draft.abstract[:2000],
@@ -302,6 +312,7 @@ class Analyzer:
 
         console.print(f"Rating [bold]{len(unrated)}[/] drafts in batches of {batch_size}...")
         count = 0
+        failures: list[tuple[str, str]] = []
         with Progress(
             SpinnerColumn(),
             TextColumn("[progress.description]{task.description}"),
@@ -314,15 +325,29 @@ class Analyzer:
                 batch = unrated[i:i + batch_size]
                 names = ", ".join(d.name.split("-")[-1][:12] for d in batch)
                 progress.update(task, description=f"Batch: {names}")
+                try:
                     n = self.rate_batch(batch, batch_size=batch_size)
                     count += n
+                except Exception as e:
+                    batch_names = [d.name for d in batch]
+                    for bn in batch_names:
+                        failures.append((bn, str(e)))
+                    console.print(f"[red]Batch failed: {e}[/]")
                 progress.advance(task, advance=len(batch))
 
         in_tok, out_tok = self.db.total_tokens_used()
+        total_attempted = len(unrated)
         console.print(
             f"Rated [bold green]{count}[/] drafts "
             f"| Total tokens used: {in_tok:,} in + {out_tok:,} out"
         )
+        if failures:
+            console.print(
+                f"[yellow]Processed {count}/{total_attempted} drafts, "
+                f"{len(failures)} failure(s):[/]"
+            )
+            for name, err in failures[:20]:
+                console.print(f"  [red]{name}[/]: {err}")
         return count
 
     def extract_ideas(self, draft_name: str, use_cache: bool = True) -> list[dict] | None:
@@ -337,6 +362,7 @@ class Analyzer:
             text_excerpt = draft.full_text[:3000]
 
         prompt = EXTRACT_IDEAS_PROMPT.format(
+            doc_type=_doc_type_label(draft.source),
             name=draft.name, title=draft.title,
             pages=draft.pages or "?",
             abstract=draft.abstract[:2000],
@@ -451,6 +477,7 @@ class Analyzer:
             console.print(f"Extracting ideas from [bold]{len(missing)}[/] drafts ({model_label})...")
 
         count = 0
+        failures: list[tuple[str, str]] = []
         with Progress(
             SpinnerColumn(),
             TextColumn("[progress.description]{task.description}"),
@@ -465,23 +492,40 @@ class Analyzer:
                     batch = missing[i:i + batch_size]
                     names = ", ".join(n.split("-")[-1][:10] for n in batch)
                     progress.update(task, description=f"Batch: {names}")
+                    try:
                         n = self.extract_ideas_batch(batch, cheap=cheap)
                         count += n
+                    except Exception as e:
+                        for bn in batch:
+                            failures.append((bn, str(e)))
+                        console.print(f"[red]Batch failed: {e}[/]")
                     progress.advance(task, advance=len(batch))
             else:
                 for name in missing:
                     progress.update(task, description=f"Ideas: {name.split('-')[-1][:15]}")
+                    try:
                         result = self.extract_ideas(name)
                         if result:
                             count += 1
+                    except Exception as e:
+                        failures.append((name, str(e)))
+                        console.print(f"[red]Failed {name}: {e}[/]")
                     progress.advance(task)
 
+        total_attempted = len(missing)
         in_tok, out_tok = self.db.total_tokens_used()
         console.print(
             f"Extracted ideas from [bold green]{count}[/] drafts "
             f"({self.db.idea_count()} total ideas) "
             f"| Tokens: {in_tok:,} in + {out_tok:,} out"
         )
+        if failures:
+            console.print(
+                f"[yellow]Processed {count}/{total_attempted} drafts, "
+                f"{len(failures)} failure(s):[/]"
+            )
+            for name, err in failures[:20]:
+                console.print(f"  [red]{name}[/]: {err}")
         return count
 
     def gap_analysis(self) -> list[dict]:
@@ -551,28 +595,49 @@ class Analyzer:
             console.print(f"[red]Gap analysis failed: {e}[/]")
             return []
 
-    def compare_drafts(self, draft_names: list[str]) -> str:
+    def compare_drafts(self, draft_names: list[str], use_cache: bool = True) -> dict:
-        """Compare multiple drafts and return analysis text."""
+        """Compare multiple drafts and return structured comparison.
+
+        Returns dict with keys: text, drafts (list of names that were compared),
+        or a dict with key 'error' on failure.
+        """
+        valid_names = []
         parts = []
         for name in draft_names:
             draft = self.db.get_draft(name)
             if draft is None:
                 console.print(f"[yellow]Skipping unknown draft: {name}[/]")
                 continue
+            valid_names.append(name)
             parts.append(f"### {draft.title}\n**{name}**\n{draft.abstract}")
 
         if len(parts) < 2:
-            return "Need at least 2 valid drafts to compare."
+            return {"error": "Need at least 2 valid drafts to compare.", "drafts": valid_names}
 
         prompt = COMPARE_PROMPT.format(
             drafts_section="\n\n---\n\n".join(parts)
         )
+        phash = _prompt_hash(prompt)
+        cache_key = "_compare_" + "_".join(sorted(valid_names))
+
+        # Check cache
+        if use_cache:
+            cached = self.db.get_cached_response(cache_key, phash)
+            if cached:
+                return {"text": cached, "drafts": valid_names}
 
         try:
-            text, _, _ = self._call_claude(prompt, max_tokens=2048)
+            text, in_tok, out_tok = self._call_claude(prompt, max_tokens=2048)
-            return text
+
+            # Cache the result
+            self.db.cache_response(
+                cache_key, phash, self.config.claude_model,
+                prompt, text, in_tok, out_tok,
+            )
+
+            return {"text": text, "drafts": valid_names}
         except anthropic.APIError as e:
-            return f"Error: {e}"
+            return {"error": f"API error: {e}", "drafts": valid_names}
 
     def dedup_ideas(self, threshold: float = 0.85, dry_run: bool = True,
                     draft_name: str | None = None) -> dict:

src/ietf_analyzer/cli.py

@@ -173,6 +173,20 @@ def show(name: str):
         else:
             console.print("[dim]Not yet rated — run: ietf analyze {name}[/]")
 
+        # Readiness score
+        from .readiness import compute_readiness
+        readiness = compute_readiness(db, name)
+        if readiness["score"] > 0:
+            console.print(f"\n[bold]Standards Readiness: [cyan]{readiness['score']}/100[/][/]")
+            rtable = Table(show_header=True)
+            rtable.add_column("Factor", width=20)
+            rtable.add_column("Value", justify="center", width=10)
+            rtable.add_column("Points", justify="right", width=8)
+            rtable.add_column("Detail")
+            for key, f in readiness["factors"].items():
+                rtable.add_row(f["label"], f"{f['value']:.2f}", f"+{f['contribution']}", f["detail"])
+            console.print(rtable)
+
         # Save detailed report too
         path = reporter.draft_detail(name)
         if path:
@@ -181,6 +195,56 @@ def show(name: str):
         db.close()
 
 
+# ── annotate ─────────────────────────────────────────────────────────────────
+
+
+@main.command()
+@click.argument("draft_name")
+@click.option("--note", "-n", default=None, help="Set/update the note text")
+@click.option("--tag", "-t", multiple=True, help="Add a tag (can be used multiple times)")
+@click.option("--remove-tag", "-r", multiple=True, help="Remove a tag (can be used multiple times)")
+def annotate(draft_name: str, note: str | None, tag: tuple[str, ...], remove_tag: tuple[str, ...]):
+    """Add or view annotations (notes & tags) for a draft."""
+    cfg = _get_config()
+    db = Database(cfg)
+    try:
+        draft = db.get_draft(draft_name)
+        if draft is None:
+            console.print(f"[red]Draft not found: {draft_name}[/]")
+            return
+
+        # If no options, display current annotation
+        if note is None and not tag and not remove_tag:
+            ann = db.get_annotation(draft_name)
+            if ann:
+                console.print(f"\n[bold]Annotation for {draft_name}[/]")
+                console.print(f"  Note: {ann['note'] or '(empty)'}")
+                console.print(f"  Tags: {', '.join(ann['tags']) if ann['tags'] else '(none)'}")
+                console.print(f"  Updated: {ann['updated_at']}")
+            else:
+                console.print(f"[dim]No annotation for {draft_name}. Use --note or --tag to add one.[/]")
+            return
+
+        # Fetch existing tags for add/remove operations
+        existing = db.get_annotation(draft_name)
+        current_tags = existing["tags"] if existing else []
+
+        for t in tag:
+            if t not in current_tags:
+                current_tags.append(t)
+        for t in remove_tag:
+            if t in current_tags:
+                current_tags.remove(t)
+
+        db.upsert_annotation(draft_name, note=note, tags=current_tags)
+        ann = db.get_annotation(draft_name)
+        console.print(f"[green]Annotation updated for {draft_name}[/]")
+        console.print(f"  Note: {ann['note'] or '(empty)'}")
+        console.print(f"  Tags: {', '.join(ann['tags']) if ann['tags'] else '(none)'}")
+    finally:
+        db.close()
+
+
 # ── analyze ──────────────────────────────────────────────────────────────────
 
 
@@ -188,7 +252,8 @@ def show(name: str):
 @click.argument("name", required=False)
 @click.option("--all", "analyze_all", is_flag=True, help="Analyze all unrated drafts")
 @click.option("--limit", "-n", default=50, help="Max drafts to analyze (with --all)")
-def analyze(name: str | None, analyze_all: bool, limit: int):
+@click.option("--retry-failed", is_flag=True, help="Re-analyze drafts that previously failed (clears cache)")
+def analyze(name: str | None, analyze_all: bool, limit: int, retry_failed: bool):
     """Analyze and rate drafts using Claude."""
     from .analyzer import Analyzer
 
@@ -197,7 +262,29 @@ def analyze(name: str | None, analyze_all: bool, limit: int):
     analyzer = Analyzer(cfg, db)
 
     try:
-        if analyze_all:
+        if retry_failed:
+            # Find drafts that have cache entries but no ratings (failed analyses)
+            unrated = db.unrated_drafts(limit=limit)
+            retryable = []
+            for draft in unrated:
+                # Check if there's a cache entry for this draft (it was attempted)
+                row = db.conn.execute(
+                    "SELECT COUNT(*) FROM llm_cache WHERE draft_name = ?",
+                    (draft.name,),
+                ).fetchone()
+                if row[0] > 0:
+                    retryable.append(draft)
+            if not retryable:
+                console.print("No previously failed drafts to retry.")
+            else:
+                console.print(f"Retrying [bold]{len(retryable)}[/] previously failed drafts...")
+                count = 0
+                for draft in retryable:
+                    rating = analyzer.rate_draft(draft.name, use_cache=False)
+                    if rating:
+                        count += 1
+                console.print(f"Successfully re-analyzed [bold green]{count}[/] of {len(retryable)} drafts")
+        elif analyze_all:
             count = analyzer.rate_all_unrated(limit=limit)
             console.print(f"Analyzed [bold green]{count}[/] drafts")
         elif name:
@@ -217,6 +304,62 @@ def analyze(name: str | None, analyze_all: bool, limit: int):
         db.close()
 
 
+# ── ask ──────────────────────────────────────────────────────────────────────
+
+
+@main.command()
+@click.argument("question")
+@click.option("--top", "-n", default=5, help="Number of source drafts to use")
+@click.option("--cheap/--quality", default=True, help="Use Haiku (cheap) vs Sonnet (quality)")
+def ask(question: str, top: int, cheap: bool):
+    """Ask a natural language question about the drafts.
+
+    Examples:
+      ietf ask "Which drafts address agent authentication?"
+      ietf ask "What are the competing approaches to agent delegation?" --top 10
+      ietf ask "How do safety mechanisms work?" --cheap
+    """
+    from .search import HybridSearch
+
+    cfg = _get_config()
+    db = Database(cfg)
+
+    try:
+        searcher = HybridSearch(cfg, db)
+        console.print(f"\n[dim]Searching for relevant drafts...[/]")
+        result = searcher.ask(question, top_k=top, cheap=cheap)
+
+        # Display the answer
+        console.print()
+        console.print("[bold cyan]Answer[/]")
+        console.print("[dim]" + "-" * 60 + "[/]")
+        console.print(result["answer"])
+        console.print()
+
+        # Display source drafts table
+        if result["sources"]:
+            table = Table(title="Source Drafts")
+            table.add_column("#", style="dim", width=3)
+            table.add_column("Draft", style="cyan", max_width=50)
+            table.add_column("Title", max_width=45)
+            table.add_column("Match", width=10)
+            table.add_column("Score", justify="right", width=8)
+
+            for i, src in enumerate(result["sources"], 1):
+                score_str = f"{src['similarity']:.3f}" if src.get("similarity") else "-"
+                table.add_row(
+                    str(i),
+                    src["name"],
+                    src["title"][:45],
+                    src.get("match_type", ""),
+                    score_str,
+                )
+
+            console.print(table)
+    finally:
+        db.close()
+
+
 # ── compare ──────────────────────────────────────────────────────────────────
 
 
@@ -232,7 +375,12 @@ def compare(names: tuple[str, ...]):
 
     try:
         result = analyzer.compare_drafts(list(names))
-        console.print(result)
+        if "error" in result:
+            console.print(f"[red]{result['error']}[/]")
+        else:
+            console.print(f"\n[bold cyan]Comparison of {len(result['drafts'])} drafts[/]")
+            console.print("[dim]" + "-" * 60 + "[/]")
+            console.print(result["text"])
     finally:
         db.close()
 
@@ -2107,7 +2255,8 @@ def draft_gen(gap_topic: str, output: str | None):
 
 @main.command("config")
 @click.option("--set", "set_key", nargs=2, help="Set a config key (e.g. --set claude_model claude-opus-4-20250514)")
-def config_cmd(set_key: tuple[str, str] | None):
+@click.option("--show", is_flag=True, help="Show effective config with env var sources noted")
+def config_cmd(set_key: tuple[str, str] | None, show: bool):
     """Show or modify configuration."""
     from dataclasses import asdict
     cfg = _get_config()
@@ -2131,8 +2280,20 @@ def config_cmd(set_key: tuple[str, str] | None):
             console.print(f"[red]Unknown config key: {key}[/]")
     else:
         from dataclasses import asdict
+        env_sources = cfg.env_sources()
         for key, val in asdict(cfg).items():
-            console.print(f"  [bold]{key}:[/] {val}")
+            source_note = ""
+            if key in env_sources:
+                source_note = f"  [yellow](from ${env_sources[key]})[/]"
+            console.print(f"  [bold]{key}:[/] {val}{source_note}")
+        if env_sources:
+            console.print(f"\n  [dim]({len(env_sources)} value(s) overridden by environment variables)[/]")
+        # Note about ANTHROPIC_API_KEY
+        import os
+        if os.environ.get("ANTHROPIC_API_KEY"):
+            console.print("  [dim]ANTHROPIC_API_KEY is set in environment[/]")
+        else:
+            console.print("  [dim]ANTHROPIC_API_KEY is NOT set in environment[/]")
 
 
 # ── pipeline ────────────────────────────────────────────────────────────────
@@ -2321,16 +2482,60 @@ def pipeline_quality(draft_id: int):
 
 @pipeline.command("status")
 def pipeline_status():
-    """Show all generated drafts."""
+    """Show pipeline health: processing stages, generated drafts, and API cost."""
     cfg = _get_config()
     db = Database(cfg)
     try:
-        drafts = db.get_generated_drafts()
+        # Pipeline health overview
-        if not drafts:
+        total = db.count_drafts()
-            console.print("No generated drafts yet. Run `ietf pipeline generate <topic>`")
+        rated_count = len(db.drafts_with_ratings(limit=10000))
-            return
+        unrated = len(db.unrated_drafts(limit=10000))
+        unembedded = len(db.drafts_without_embeddings(limit=10000))
+        embedded_count = total - unembedded
+        no_ideas = len(db.drafts_without_ideas(limit=10000))
+        ideas_count = total - no_ideas
+        idea_total = db.idea_count()
+        gap_count = len(db.all_gaps())
+        input_tok, output_tok = db.total_tokens_used()
+        est_cost = (input_tok * 3.0 / 1_000_000) + (output_tok * 15.0 / 1_000_000)
 
-        table = Table(title=f"Generated Drafts ({len(drafts)})")
+        # Last update
+        snapshots = db.get_snapshots(limit=1)
+        last_update = snapshots[0]["snapshot_at"][:19] if snapshots else "never"
+
+        console.print("\n[bold]Pipeline Status[/]\n")
+        console.print(f"  Total documents:  [bold]{total}[/]")
+        console.print(f"  Last update:      {last_update}")
+        console.print()
+
+        # Stage table
+        stage_table = Table(title="Processing Stages")
+        stage_table.add_column("Stage", width=20)
+        stage_table.add_column("Done", justify="right", width=8)
+        stage_table.add_column("Missing", justify="right", width=8)
+        stage_table.add_column("Progress", width=20)
+
+        def bar(done, total_n):
+            pct = int(done / total_n * 100) if total_n > 0 else 0
+            filled = pct // 5
+            return f"[green]{'#' * filled}[/][dim]{'.' * (20 - filled)}[/] {pct}%"
+
+        stage_table.add_row("Rated", str(rated_count), str(unrated), bar(rated_count, total))
+        stage_table.add_row("Embedded", str(embedded_count), str(unembedded), bar(embedded_count, total))
+        stage_table.add_row("Ideas extracted", str(ideas_count), str(no_ideas), bar(ideas_count, total))
+
+        console.print(stage_table)
+
+        console.print(f"\n  Total ideas: [bold]{idea_total}[/]")
+        console.print(f"  Gaps identified: [bold]{gap_count}[/]")
+        console.print(f"\n  API tokens: {input_tok:,} in + {output_tok:,} out")
+        console.print(f"  Estimated cost: [bold]${est_cost:.2f}[/]")
+
+        # Generated drafts
+        gen_drafts = db.get_generated_drafts()
+        if gen_drafts:
+            console.print()
+            table = Table(title=f"Generated Drafts ({len(gen_drafts)})")
             table.add_column("ID", justify="right", width=4)
             table.add_column("Draft Name", style="cyan")
             table.add_column("Gap Topic")
@@ -2339,7 +2544,7 @@ def pipeline_status():
             table.add_column("Quality", justify="right", width=7)
             table.add_column("Created", width=10)
 
-        for d in drafts:
+            for d in gen_drafts:
                 table.add_row(
                     str(d["id"]),
                     d["draft_name"],
@@ -2397,28 +2602,38 @@ def observatory():
 @observatory.command("update")
 @click.option("--source", "-s", default=None, help="Comma-separated sources (e.g. ietf,w3c)")
 @click.option("--full/--delta", default=False, help="Full refresh or delta only")
-def observatory_update(source: str | None, full: bool):
+@click.option("--dry-run", is_flag=True, default=False, help="Show what would happen without making changes")
+def observatory_update(source: str | None, full: bool, dry_run: bool):
     """Fetch, analyze, and update the observatory."""
     from .observatory import Observatory
-    from .analyzer import Analyzer
 
     cfg = _get_config()
     db = Database(cfg)
-    analyzer = Analyzer(cfg, db)
 
     try:
+        if dry_run:
+            obs = Observatory(cfg, db)
+        else:
+            from .analyzer import Analyzer
+            analyzer = Analyzer(cfg, db)
             obs = Observatory(cfg, db, analyzer)
         sources = source.split(",") if source else None
-        console.print(f"[bold]Observatory update[/] ({'full' if full else 'delta'})")
+        mode = "full" if full else "delta"
-        result = obs.update(sources=sources, full=full)
+        console.print(f"[bold]Observatory update[/] ({mode}{'  [DRY RUN]' if dry_run else ''})")
+        result = obs.update(sources=sources, full=full, dry_run=dry_run)
 
+        if not dry_run:
             console.print(f"\n[bold green]Update complete![/]")
             console.print(f"  New docs: {result.get('new_docs', 0)}")
             console.print(f"  Analyzed: {result.get('analyzed', 0)}")
             console.print(f"  Embedded: {result.get('embedded', 0)}")
             console.print(f"  Ideas extracted: {result.get('ideas', 0)}")
-        if result.get("gaps_updated"):
+            if result.get("gaps_changed"):
-            console.print(f"  Gaps re-analyzed: yes ({result.get('gap_count', 0)} gaps)")
+                console.print(f"  Gaps re-analyzed: yes")
+            if result.get("errors"):
+                console.print(f"\n  [yellow]Errors ({len(result['errors'])}):[/]")
+                for err in result["errors"]:
+                    console.print(f"    - {err}")
     finally:
         db.close()
 
@@ -2676,3 +2891,105 @@ def monitor_status():
             console.print(table)
     finally:
         db.close()
+
+
+# ── export ──────────────────────────────────────────────────────────────────
+
+
+@main.command()
+@click.option("--type", "export_type", type=click.Choice(["drafts", "ideas", "gaps", "authors", "ratings"]),
+              required=True, help="Type of data to export")
+@click.option("--format", "fmt", type=click.Choice(["json", "csv"]), default="json", help="Output format")
+@click.option("--output", "-o", "output_file", type=click.Path(), default=None,
+              help="Output file (default: stdout)")
+def export(export_type: str, fmt: str, output_file: str | None):
+    """Export data as JSON or CSV."""
+    import csv as csv_mod
+    import io
+    import json
+
+    cfg = _get_config()
+    db = Database(cfg)
+
+    try:
+        rows: list[dict] = []
+
+        if export_type == "drafts":
+            drafts = db.list_drafts(limit=10000, order_by="name ASC")
+            for d in drafts:
+                rating = db.get_rating(d.name)
+                row = {
+                    "name": d.name,
+                    "title": d.title,
+                    "rev": d.rev,
+                    "date": d.date,
+                    "pages": d.pages or 0,
+                    "group": d.group or "",
+                }
+                if rating:
+                    row["score"] = round(rating.composite_score, 2)
+                    row["novelty"] = rating.novelty
+                    row["maturity"] = rating.maturity
+                    row["overlap"] = rating.overlap
+                    row["momentum"] = rating.momentum
+                    row["relevance"] = rating.relevance
+                    row["categories"] = json.dumps(rating.categories)
+                    row["summary"] = rating.summary
+                rows.append(row)
+
+        elif export_type == "ideas":
+            ideas = db.all_ideas()
+            rows = ideas
+
+        elif export_type == "gaps":
+            gaps = db.all_gaps()
+            rows = gaps
+
+        elif export_type == "authors":
+            top = db.top_authors(limit=10000)
+            for name, aff, cnt, drafts_list in top:
+                rows.append({
+                    "name": name,
+                    "affiliation": aff,
+                    "draft_count": cnt,
+                    "drafts": json.dumps(drafts_list),
+                })
+
+        elif export_type == "ratings":
+            pairs = db.drafts_with_ratings(limit=10000)
+            for draft, rating in pairs:
+                rows.append({
+                    "name": draft.name,
+                    "title": draft.title,
+                    "score": round(rating.composite_score, 2),
+                    "novelty": rating.novelty,
+                    "maturity": rating.maturity,
+                    "overlap": rating.overlap,
+                    "momentum": rating.momentum,
+                    "relevance": rating.relevance,
+                    "categories": json.dumps(rating.categories),
+                    "summary": rating.summary,
+                })
+
+        if fmt == "json":
+            text = json.dumps(rows, indent=2, ensure_ascii=False)
+        else:
+            # CSV
+            if not rows:
+                text = ""
+            else:
+                si = io.StringIO()
+                writer = csv_mod.DictWriter(si, fieldnames=rows[0].keys())
+                writer.writeheader()
+                for row in rows:
+                    writer.writerow(row)
+                text = si.getvalue()
+
+        if output_file:
+            Path(output_file).write_text(text, encoding="utf-8")
+            console.print(f"Exported [bold green]{len(rows)}[/] {export_type} to [cyan]{output_file}[/] ({fmt})")
+        else:
+            click.echo(text)
+
+    finally:
+        db.close()

src/ietf_analyzer/config.py

@@ -3,6 +3,7 @@
 from __future__ import annotations
 
 import json
+import os
 from dataclasses import dataclass, field, asdict
 from pathlib import Path
 
@@ -24,6 +25,13 @@ DEFAULT_KEYWORDS = [
     "aipref",
 ]
 
+# Environment variable overrides (env var name -> config field name)
+_ENV_OVERRIDES = {
+    "IETF_ANALYZER_DB_PATH": "db_path",
+    "IETF_ANALYZER_CLAUDE_MODEL": "claude_model",
+    "IETF_ANALYZER_OLLAMA_URL": "ollama_url",
+}
+
 
 @dataclass
 class Config:
@@ -41,7 +49,9 @@ class Config:
     # Pipeline
     generation_max_tokens: int = 4096
     generation_model: str = ""  # defaults to claude_model
-    # Observatory
+    # Observatory — add "w3c" to enable W3C spec tracking:
+    #   ietf observatory update --source w3c    (one-off)
+    #   or set observatory_sources to ["ietf", "w3c"] in config.json
     observatory_sources: list[str] = field(default_factory=lambda: ["ietf"])
     dashboard_dir: str = str(DEFAULT_DATA_DIR.parent / "docs")
     w3c_groups: list[str] = field(default_factory=lambda: [
@@ -52,9 +62,47 @@ class Config:
         Path(self.data_dir).mkdir(parents=True, exist_ok=True)
         CONFIG_FILE.write_text(json.dumps(asdict(self), indent=2))
 
+    def env_sources(self) -> dict[str, str]:
+        """Return {field_name: env_var_name} for fields overridden by env vars."""
+        sources: dict[str, str] = {}
+        for env_var, field_name in _ENV_OVERRIDES.items():
+            if os.environ.get(env_var):
+                sources[field_name] = env_var
+        return sources
+
+    @classmethod
+    def _validate(cls, cfg: Config) -> None:
+        """Validate config values, raise ValueError on problems."""
+        if not cfg.claude_model or not cfg.claude_model.strip():
+            raise ValueError(
+                "claude_model must be a non-empty string. "
+                "Check your config file or IETF_ANALYZER_CLAUDE_MODEL env var."
+            )
+        if not cfg.ollama_url.startswith(("http://", "https://")):
+            raise ValueError(
+                f"ollama_url must be an HTTP(S) URL, got: '{cfg.ollama_url}'. "
+                "Check your config file or IETF_ANALYZER_OLLAMA_URL env var."
+            )
+        db_parent = Path(cfg.db_path).parent
+        if not db_parent.exists():
+            raise ValueError(
+                f"db_path parent directory does not exist: '{db_parent}'. "
+                "Check your config file or IETF_ANALYZER_DB_PATH env var."
+            )
+
     @classmethod
     def load(cls) -> Config:
         if CONFIG_FILE.exists():
             data = json.loads(CONFIG_FILE.read_text())
-            return cls(**{k: v for k, v in data.items() if k in cls.__dataclass_fields__})
+            cfg = cls(**{k: v for k, v in data.items() if k in cls.__dataclass_fields__})
-        return cls()
+        else:
+            cfg = cls()
+
+        # Apply environment variable overrides (env vars take precedence)
+        for env_var, field_name in _ENV_OVERRIDES.items():
+            env_val = os.environ.get(env_var)
+            if env_val is not None:
+                setattr(cfg, field_name, env_val)
+
+        cls._validate(cfg)
+        return cfg

src/ietf_analyzer/db.py

@@ -192,6 +192,17 @@ CREATE TABLE IF NOT EXISTS gap_history (
     recorded_at TEXT
 );
 
+-- Annotations (user notes + tags per draft)
+CREATE TABLE IF NOT EXISTS annotations (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    draft_name TEXT NOT NULL REFERENCES drafts(name),
+    note TEXT DEFAULT '',
+    tags TEXT DEFAULT '[]',
+    created_at TEXT,
+    updated_at TEXT,
+    UNIQUE(draft_name)
+);
+
 -- Monitor runs
 CREATE TABLE IF NOT EXISTS monitor_runs (
     id INTEGER PRIMARY KEY AUTOINCREMENT,
@@ -529,14 +540,17 @@ class Database:
             ORDER BY da.author_order""",
             (draft_name,),
         ).fetchall()
-        cols = rows[0].keys() if rows else []
+        results = []
-        return [Author(
+        for r in rows:
-            person_id=r["person_id"], name=r["name"],
+            d = dict(r)
-            ascii_name=r["ascii_name"] if "ascii_name" in cols else "",
+            results.append(Author(
-            affiliation=r["affiliation"] if "affiliation" in cols else "",
+                person_id=d["person_id"], name=d["name"],
-            resource_uri=r["resource_uri"] if "resource_uri" in cols else "",
+                ascii_name=d.get("ascii_name", ""),
-            fetched_at=r["fetched_at"] if "fetched_at" in cols else None,
+                affiliation=d.get("affiliation", ""),
-        ) for r in rows]
+                resource_uri=d.get("resource_uri", ""),
+                fetched_at=d.get("fetched_at"),
+            ))
+        return results
 
     def drafts_without_authors(self, limit: int = 500) -> list[str]:
         rows = self.conn.execute(
@@ -681,7 +695,8 @@ class Database:
             "SELECT * FROM ideas WHERE draft_name = ?", (draft_name,)
         ).fetchall()
         return [{"id": r["id"], "title": r["title"], "description": r["description"],
-                 "type": r["idea_type"], "draft_name": r["draft_name"]} for r in rows]
+                 "type": r["idea_type"], "draft_name": r["draft_name"],
+                 "novelty_score": r["novelty_score"]} for r in rows]
 
     def delete_idea(self, idea_id: int) -> None:
         """Delete a single idea and its embedding by ID."""
@@ -706,7 +721,8 @@ class Database:
             "SELECT * FROM ideas ORDER BY draft_name"
         ).fetchall()
         return [{"title": r["title"], "description": r["description"],
-                 "type": r["idea_type"], "draft_name": r["draft_name"]} for r in rows]
+                 "type": r["idea_type"], "draft_name": r["draft_name"],
+                 "novelty_score": r["novelty_score"]} for r in rows]
 
     def idea_count(self) -> int:
         return self.conn.execute("SELECT COUNT(*) FROM ideas").fetchone()[0]
@@ -1380,6 +1396,75 @@ class Database:
         ).fetchone()
         return dict(row) if row else None
 
+    # --- Annotations ---
+
+    def upsert_annotation(self, draft_name: str, note: str | None = None, tags: list[str] | None = None) -> None:
+        """Insert or update an annotation for a draft."""
+        now = datetime.now(timezone.utc).isoformat()
+        existing = self.conn.execute(
+            "SELECT id, note, tags FROM annotations WHERE draft_name = ?",
+            (draft_name,),
+        ).fetchone()
+        if existing:
+            current_note = note if note is not None else existing["note"]
+            current_tags = tags if tags is not None else json.loads(existing["tags"] or "[]")
+            self.conn.execute(
+                "UPDATE annotations SET note = ?, tags = ?, updated_at = ? WHERE draft_name = ?",
+                (current_note, json.dumps(current_tags), now, draft_name),
+            )
+        else:
+            self.conn.execute(
+                """INSERT INTO annotations (draft_name, note, tags, created_at, updated_at)
+                VALUES (?, ?, ?, ?, ?)""",
+                (draft_name, note or "", json.dumps(tags or []), now, now),
+            )
+        self.conn.commit()
+
+    def get_annotation(self, draft_name: str) -> dict | None:
+        """Return annotation for a draft, or None."""
+        row = self.conn.execute(
+            "SELECT * FROM annotations WHERE draft_name = ?", (draft_name,)
+        ).fetchone()
+        if not row:
+            return None
+        return {
+            "id": row["id"],
+            "draft_name": row["draft_name"],
+            "note": row["note"],
+            "tags": json.loads(row["tags"] or "[]"),
+            "created_at": row["created_at"],
+            "updated_at": row["updated_at"],
+        }
+
+    def get_all_annotations(self) -> list[dict]:
+        """Return all annotations."""
+        rows = self.conn.execute(
+            "SELECT * FROM annotations ORDER BY updated_at DESC"
+        ).fetchall()
+        return [
+            {
+                "id": r["id"],
+                "draft_name": r["draft_name"],
+                "note": r["note"],
+                "tags": json.loads(r["tags"] or "[]"),
+                "created_at": r["created_at"],
+                "updated_at": r["updated_at"],
+            }
+            for r in rows
+        ]
+
+    def search_by_tag(self, tag: str) -> list[str]:
+        """Return draft names that have a specific tag in their annotation."""
+        rows = self.conn.execute(
+            "SELECT draft_name, tags FROM annotations"
+        ).fetchall()
+        results = []
+        for r in rows:
+            tags = json.loads(r["tags"] or "[]")
+            if tag in tags:
+                results.append(r["draft_name"])
+        return results
+
     # --- Helpers ---
 
     @staticmethod

src/ietf_analyzer/observatory.py

@@ -76,6 +76,7 @@ class Observatory:
         self,
         sources: list[str] | None = None,
         full: bool = False,
+        dry_run: bool = False,
     ) -> dict:
         """Full update cycle.
 
@@ -87,10 +88,30 @@ class Observatory:
         6. Re-run gap analysis if >= 5 new docs
         7. Record gap changes in gap_history
         8. Return summary stats
+
+        If dry_run=True, show what would be fetched/analyzed without doing it.
         """
         sources = sources or self.config.observatory_sources
         stats: dict = {"sources": {}, "new_docs": 0, "analyzed": 0, "embedded": 0, "ideas": 0, "gaps_changed": False}
 
+        if dry_run:
+            console.print("[bold yellow]DRY RUN[/] — showing what would happen without making changes\n")
+            for src_name in sources:
+                console.print(f"  Would fetch from: [cyan]{src_name}[/]")
+                src = self.db.get_source(src_name)
+                if src and src.get("last_fetch"):
+                    console.print(f"    Last fetch: {src['last_fetch'][:10]}")
+                else:
+                    console.print(f"    Last fetch: never (full fetch)")
+            unrated = len(self.db.unrated_drafts(limit=10000))
+            unembedded = len(self.db.drafts_without_embeddings(limit=10000))
+            no_ideas = len(self.db.drafts_without_ideas(limit=10000))
+            console.print(f"\n  Would analyze: [bold]{unrated}[/] unrated documents")
+            console.print(f"  Would embed: [bold]{unembedded}[/] documents")
+            console.print(f"  Would extract ideas from: [bold]{no_ideas}[/] documents")
+            stats["dry_run"] = True
+            return stats
+
         # 1. Snapshot current state
         console.print("[bold]1/7[/] Creating snapshot...")
         snapshot_id = self.db.create_snapshot()
@@ -99,35 +120,58 @@ class Observatory:
         console.print("[bold]2/7[/] Fetching from sources...")
         total_new = 0
         for src_name in sources:
+            try:
                 new_count = self._fetch_source(src_name, full=full)
                 stats["sources"][src_name] = new_count
                 total_new += new_count
+            except Exception as e:
+                console.print(f"  [red]Error fetching {src_name}: {e}[/]")
+                stats["sources"][src_name] = {"error": str(e)}
         stats["new_docs"] = total_new
         console.print(f"  Fetched [bold green]{total_new}[/] new documents total")
 
         # 3. Analyze unrated docs
         console.print("[bold]3/7[/] Analyzing unrated documents...")
+        try:
             analyzed = self.analyzer.rate_all_unrated(limit=200, batch_size=5)
             stats["analyzed"] = analyzed
+        except Exception as e:
+            console.print(f"  [red]Analysis failed: {e}[/]")
+            stats["analyzed"] = 0
+            stats["errors"] = stats.get("errors", []) + [f"analyze: {e}"]
 
         # 4. Embed missing docs
         console.print("[bold]4/7[/] Embedding missing documents...")
+        try:
             embedded = self._embed_missing()
             stats["embedded"] = embedded
+        except Exception as e:
+            console.print(f"  [red]Embedding failed: {e}[/]")
+            stats["embedded"] = 0
+            stats["errors"] = stats.get("errors", []) + [f"embed: {e}"]
 
         # 5. Extract ideas from new docs
         console.print("[bold]5/7[/] Extracting ideas...")
+        try:
             ideas = self.analyzer.extract_all_ideas(limit=200, batch_size=5, cheap=True)
             stats["ideas"] = ideas
+        except Exception as e:
+            console.print(f"  [red]Idea extraction failed: {e}[/]")
+            stats["ideas"] = 0
+            stats["errors"] = stats.get("errors", []) + [f"ideas: {e}"]
 
         # 6. Re-run gap analysis if enough new docs
         if total_new >= 5:
             console.print("[bold]6/7[/] Re-running gap analysis...")
+            try:
                 gaps = self.analyzer.gap_analysis()
                 if gaps:
                     self.db.record_gap_history(snapshot_id, gaps)
                     stats["gaps_changed"] = True
                     console.print(f"  Found [bold]{len(gaps)}[/] gaps")
+            except Exception as e:
+                console.print(f"  [red]Gap analysis failed: {e}[/]")
+                stats["errors"] = stats.get("errors", []) + [f"gaps: {e}"]
         else:
             console.print(f"[bold]6/7[/] Skipping gap analysis ({total_new} < 5 new docs)")
             # Record current gaps unchanged
@@ -142,7 +186,7 @@ class Observatory:
             self.db.upsert_source(src_name, doc_count=count)
 
         console.print("\n[bold green]Observatory update complete![/]")
-        console.print(f"  New docs: {total_new}  |  Analyzed: {analyzed}  |  Embedded: {embedded}  |  Ideas: {ideas}")
+        console.print(f"  New docs: {total_new}  |  Analyzed: {stats['analyzed']}  |  Embedded: {stats['embedded']}  |  Ideas: {stats['ideas']}")
         return stats
 
     def _fetch_source(self, source_name: str, full: bool = False) -> int:

src/ietf_analyzer/readiness.py

@@ -0,0 +1,102 @@
+"""Standards readiness scoring — composite 0-100 score predicting RFC proximity."""
+
+from __future__ import annotations
+
+
+def compute_readiness(db, draft_name: str) -> dict:
+    """Compute 0-100 readiness score with component breakdown.
+
+    Factors (each 0-1, weighted):
+    - wg_adopted (0.25): name starts with 'draft-ietf-' = 1.0, else 0.0
+    - revision_maturity (0.15): int(rev) normalized (0-5+ maps to 0-1)
+    - reference_density (0.15): len(draft_refs) / max_refs across corpus
+    - cited_by_count (0.15): how many OTHER drafts in corpus reference this one
+    - author_experience (0.15): avg number of drafts per author in draft_authors
+    - momentum_rating (0.15): momentum score from ratings (1-5 -> 0-1)
+
+    Returns {score: 0-100, factors: {name: {value, weight, contribution}}}
+    """
+    draft = db.get_draft(draft_name)
+    if not draft:
+        return {"score": 0, "factors": {}}
+
+    factors = {}
+
+    # 1. WG Adopted (0.25)
+    wg_val = 1.0 if draft_name.startswith("draft-ietf-") else 0.0
+    factors["wg_adopted"] = {"value": wg_val, "weight": 0.25,
+                             "label": "WG Adopted",
+                             "detail": "draft-ietf-*" if wg_val else "individual"}
+
+    # 2. Revision Maturity (0.15)
+    try:
+        rev_num = int(draft.rev) if draft.rev else 0
+    except (ValueError, TypeError):
+        rev_num = 0
+    rev_val = min(rev_num / 5.0, 1.0)
+    factors["revision_maturity"] = {"value": round(rev_val, 3), "weight": 0.15,
+                                    "label": "Revision Maturity",
+                                    "detail": f"rev {rev_num}"}
+
+    # 3. Reference Density (0.15)
+    refs = db.get_refs_for_draft(draft_name)
+    ref_count = len(refs)
+    # Get max refs across corpus
+    max_refs = db.conn.execute(
+        "SELECT MAX(cnt) FROM (SELECT COUNT(*) as cnt FROM draft_refs GROUP BY draft_name)"
+    ).fetchone()[0] or 1
+    ref_val = min(ref_count / max_refs, 1.0)
+    factors["reference_density"] = {"value": round(ref_val, 3), "weight": 0.15,
+                                    "label": "Reference Density",
+                                    "detail": f"{ref_count} refs (max {max_refs})"}
+
+    # 4. Cited By Count (0.15)
+    cited_by = db.conn.execute(
+        "SELECT COUNT(DISTINCT draft_name) FROM draft_refs WHERE ref_type = 'draft' AND ref_id = ?",
+        (draft_name,),
+    ).fetchone()[0]
+    # Normalize: being cited by 5+ drafts = 1.0
+    cited_val = min(cited_by / 5.0, 1.0)
+    factors["cited_by_count"] = {"value": round(cited_val, 3), "weight": 0.15,
+                                 "label": "Cited By Others",
+                                 "detail": f"{cited_by} draft(s)"}
+
+    # 5. Author Experience (0.15)
+    authors = db.get_authors_for_draft(draft_name)
+    if authors:
+        author_draft_counts = []
+        for a in authors:
+            cnt = db.conn.execute(
+                "SELECT COUNT(*) FROM draft_authors WHERE person_id = ?",
+                (a.person_id,),
+            ).fetchone()[0]
+            author_draft_counts.append(cnt)
+        avg_exp = sum(author_draft_counts) / len(author_draft_counts)
+        # Normalize: avg 5+ drafts per author = 1.0
+        exp_val = min(avg_exp / 5.0, 1.0)
+    else:
+        exp_val = 0.0
+        avg_exp = 0
+    factors["author_experience"] = {"value": round(exp_val, 3), "weight": 0.15,
+                                    "label": "Author Experience",
+                                    "detail": f"avg {avg_exp:.1f} drafts/author"}
+
+    # 6. Momentum Rating (0.15)
+    rating = db.get_rating(draft_name)
+    if rating:
+        mom_val = (rating.momentum - 1) / 4.0  # 1-5 -> 0-1
+    else:
+        mom_val = 0.0
+    factors["momentum_rating"] = {"value": round(mom_val, 3), "weight": 0.15,
+                                  "label": "Momentum",
+                                  "detail": f"{rating.momentum}/5" if rating else "unrated"}
+
+    # Compute weighted score
+    total = sum(f["value"] * f["weight"] for f in factors.values())
+    score = round(total * 100, 1)
+
+    # Add contribution to each factor
+    for f in factors.values():
+        f["contribution"] = round(f["value"] * f["weight"] * 100, 1)
+
+    return {"score": score, "factors": factors}

src/ietf_analyzer/search.py

@@ -0,0 +1,280 @@
+"""Hybrid search — FTS5 keyword + embedding similarity + Claude synthesis."""
+
+from __future__ import annotations
+
+import hashlib
+from collections import defaultdict
+
+import numpy as np
+from rich.console import Console
+
+from .config import Config
+from .db import Database
+
+console = Console()
+
+ASK_PROMPT = """\
+You are an expert on IETF Internet-Drafts related to AI agents and autonomous systems.
+
+Based on the following IETF drafts, answer this question:
+
+**{question}**
+
+## Source Drafts
+
+{sources_block}
+
+Instructions:
+- Answer concisely but thoroughly (3-8 sentences).
+- Cite specific drafts by name (e.g. draft-xyz-...) when referencing their contributions.
+- If the drafts don't contain enough information to answer, say so clearly.
+- Focus on concrete technical mechanisms, not general observations."""
+
+
+def _cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
+    dot = np.dot(a, b)
+    norm = np.linalg.norm(a) * np.linalg.norm(b)
+    if norm == 0:
+        return 0.0
+    return float(dot / norm)
+
+
+def _prompt_hash(text: str) -> str:
+    return hashlib.sha256(text.encode()).hexdigest()[:16]
+
+
+class HybridSearch:
+    def __init__(self, config: Config, db: Database, embedder=None):
+        self.config = config
+        self.db = db
+        self._embedder = embedder
+        self._ollama_available: bool | None = None
+
+    @property
+    def embedder(self):
+        """Lazy-load embedder to avoid import errors when Ollama is unavailable."""
+        if self._embedder is None:
+            try:
+                from .embeddings import Embedder
+                self._embedder = Embedder(self.config, self.db)
+                self._ollama_available = True
+            except Exception:
+                self._ollama_available = False
+        return self._embedder
+
+    def _check_ollama(self) -> bool:
+        """Check if Ollama is available for embedding queries."""
+        if self._ollama_available is not None:
+            return self._ollama_available
+        try:
+            embedder = self.embedder
+            if embedder is None:
+                self._ollama_available = False
+                return False
+            # Try a tiny embedding to verify connectivity
+            embedder.embed_text("test")
+            self._ollama_available = True
+        except Exception:
+            self._ollama_available = False
+        return self._ollama_available
+
+    def search(self, query: str, top_k: int = 10) -> list[dict]:
+        """Combine FTS5 keyword search + embedding similarity search.
+
+        Returns ranked list of {name, title, score, excerpt, match_type}.
+        Falls back to FTS5-only if Ollama is unavailable.
+        """
+        fts_results = self._fts_search(query, limit=top_k * 2)
+        embed_results = self._embedding_search(query, limit=top_k * 2)
+
+        if embed_results:
+            merged = self._reciprocal_rank_fusion(fts_results, embed_results)
+        else:
+            merged = fts_results
+
+        return merged[:top_k]
+
+    def _fts_search(self, query: str, limit: int = 20) -> list[dict]:
+        """Run FTS5 keyword search, return ranked results."""
+        try:
+            drafts = self.db.search_drafts(query, limit=limit)
+        except Exception:
+            # FTS5 can fail on certain query syntax; fallback to simpler search
+            # Try wrapping each word with quotes for literal matching
+            words = query.split()
+            safe_query = " OR ".join(f'"{w}"' for w in words if w.strip())
+            try:
+                drafts = self.db.search_drafts(safe_query, limit=limit)
+            except Exception:
+                return []
+
+        results = []
+        for rank, draft in enumerate(drafts):
+            excerpt = draft.abstract[:200] if draft.abstract else ""
+            results.append({
+                "name": draft.name,
+                "title": draft.title,
+                "score": 0.0,  # Will be set by fusion
+                "excerpt": excerpt,
+                "match_type": "keyword",
+                "rank": rank,
+            })
+        return results
+
+    def _embedding_search(self, query: str, limit: int = 20) -> list[dict]:
+        """Run embedding similarity search against all stored embeddings."""
+        if not self._check_ollama():
+            return []
+
+        try:
+            query_vec = self.embedder.embed_text(query)
+        except Exception:
+            self._ollama_available = False
+            return []
+
+        all_embeddings = self.db.all_embeddings()
+        if not all_embeddings:
+            return []
+
+        similarities: list[tuple[str, float]] = []
+        for name, vec in all_embeddings.items():
+            sim = _cosine_similarity(query_vec, vec)
+            similarities.append((name, sim))
+
+        similarities.sort(key=lambda x: x[1], reverse=True)
+
+        results = []
+        for rank, (name, sim) in enumerate(similarities[:limit]):
+            draft = self.db.get_draft(name)
+            if draft is None:
+                continue
+            excerpt = draft.abstract[:200] if draft.abstract else ""
+            results.append({
+                "name": name,
+                "title": draft.title if draft else name,
+                "score": round(sim, 4),
+                "excerpt": excerpt,
+                "match_type": "semantic",
+                "rank": rank,
+                "similarity": round(sim, 4),
+            })
+        return results
+
+    def _reciprocal_rank_fusion(
+        self,
+        fts_results: list[dict],
+        embed_results: list[dict],
+        k: int = 60,
+    ) -> list[dict]:
+        """Merge two ranked lists using reciprocal rank fusion (RRF).
+
+        RRF score = sum(1 / (k + rank)) across all lists where the item appears.
+        """
+        scores: dict[str, float] = defaultdict(float)
+        items: dict[str, dict] = {}
+
+        for result in fts_results:
+            name = result["name"]
+            scores[name] += 1.0 / (k + result["rank"])
+            if name not in items:
+                items[name] = result.copy()
+                items[name]["match_type"] = "keyword"
+
+        for result in embed_results:
+            name = result["name"]
+            scores[name] += 1.0 / (k + result["rank"])
+            if name in items:
+                items[name]["match_type"] = "both"
+                items[name]["similarity"] = result.get("similarity", 0)
+            else:
+                items[name] = result.copy()
+
+        # Sort by RRF score
+        ranked = sorted(scores.items(), key=lambda x: x[1], reverse=True)
+
+        results = []
+        for name, rrf_score in ranked:
+            item = items[name]
+            item["score"] = round(rrf_score, 4)
+            results.append(item)
+
+        return results
+
+    def ask(self, question: str, top_k: int = 5, cheap: bool = True) -> dict:
+        """Answer a natural language question using search + Claude synthesis.
+
+        Returns {answer: str, sources: [{name, title, similarity, excerpt}]}.
+        Caches Claude responses via llm_cache.
+        """
+        search_results = self.search(question, top_k=top_k)
+
+        if not search_results:
+            return {
+                "answer": "No relevant drafts found for your question.",
+                "sources": [],
+            }
+
+        # Build context from top results
+        sources_block = ""
+        sources = []
+        for r in search_results:
+            draft = self.db.get_draft(r["name"])
+            if draft is None:
+                continue
+
+            # Title + abstract + first 500 chars of full text
+            text_preview = ""
+            if draft.full_text:
+                text_preview = draft.full_text[:500]
+
+            sources_block += f"\n---\n**{draft.name}** — {draft.title}\n"
+            sources_block += f"Abstract: {draft.abstract[:500]}\n"
+            if text_preview:
+                sources_block += f"Content excerpt: {text_preview}\n"
+
+            sources.append({
+                "name": draft.name,
+                "title": draft.title,
+                "similarity": r.get("similarity", r.get("score", 0)),
+                "excerpt": r.get("excerpt", ""),
+                "match_type": r.get("match_type", ""),
+            })
+
+        prompt = ASK_PROMPT.format(
+            question=question,
+            sources_block=sources_block,
+        )
+        phash = _prompt_hash(prompt)
+
+        # Check cache
+        cached = self.db.get_cached_response("_ask_", phash)
+        if cached:
+            return {
+                "answer": cached,
+                "sources": sources,
+            }
+
+        # Call Claude
+        try:
+            from .analyzer import Analyzer
+            analyzer = Analyzer(self.config, self.db)
+            text, in_tok, out_tok = analyzer._call_claude(
+                prompt, max_tokens=1024, cheap=cheap
+            )
+
+            # Cache the response
+            self.db.cache_response(
+                "_ask_", phash,
+                self.config.claude_model_cheap if cheap else self.config.claude_model,
+                prompt, text, in_tok, out_tok,
+            )
+
+            return {
+                "answer": text,
+                "sources": sources,
+            }
+        except Exception as e:
+            return {
+                "answer": f"Error generating answer: {e}",
+                "sources": sources,
+            }

src/webui/app.py

@@ -12,7 +12,11 @@ from pathlib import Path
 _project_root = Path(__file__).resolve().parent.parent.parent
 sys.path.insert(0, str(_project_root / "src"))
 
-from flask import Flask, render_template, request, jsonify, abort, g
+import csv
+import io
+import json
+
+from flask import Flask, render_template, request, jsonify, abort, g, Response
 
 from webui.data import (
     get_db,
@@ -39,6 +43,10 @@ from webui.data import (
     get_idea_clusters,
     get_monitor_status,
     get_author_network_full,
+    get_citation_graph,
+    get_comparison_data,
+    get_ask_data,
+    global_search,
 )
 
 app = Flask(
@@ -91,6 +99,7 @@ def drafts():
     page = request.args.get("page", 1, type=int)
     search = request.args.get("q", "")
     category = request.args.get("cat", "")
+    source = request.args.get("source", "")
     min_score = request.args.get("min_score", 0.0, type=float)
     sort = request.args.get("sort", "score")
     sort_dir = request.args.get("dir", "desc")
@@ -103,6 +112,7 @@ def drafts():
         min_score=min_score,
         sort=sort,
         sort_dir=sort_dir,
+        source=source,
     )
     categories = get_category_counts(db())
     return render_template(
@@ -111,6 +121,7 @@ def drafts():
         categories=categories,
         search=search,
         current_cat=category,
+        current_source=source,
         min_score=min_score,
         sort=sort,
         sort_dir=sort_dir,
@@ -272,6 +283,12 @@ def authors():
     )
 
 
+@app.route("/citations")
+def citations():
+    graph = get_citation_graph(db())
+    return render_template("citations.html", graph=graph)
+
+
 @app.route("/monitor")
 def monitor_page():
     status = get_monitor_status(db())
@@ -294,21 +311,121 @@ def datenschutz():
     return render_template("datenschutz.html")
 
 
+@app.route("/search")
+def search():
+    q = request.args.get("q", "").strip()
+    results = global_search(db(), q) if q else {"drafts": [], "ideas": [], "authors": [], "gaps": []}
+    total = sum(len(v) for v in results.values())
+    return render_template("search_results.html", query=q, results=results, total=total)
+
+
+@app.route("/ask")
+def ask_page():
+    question = request.args.get("q", "")
+    result = None
+    if question:
+        top_k = request.args.get("top", 5, type=int)
+        result = get_ask_data(db(), question, top_k=top_k)
+    return render_template("ask.html", question=question, result=result)
+
+
+@app.route("/api/ask", methods=["POST"])
+def api_ask():
+    """Answer a question via hybrid search + Claude. Returns JSON."""
+    data = request.get_json(force=True, silent=True)
+    if not data or "question" not in data:
+        return jsonify({"error": "Missing 'question' in request body"}), 400
+    question = data["question"]
+    top_k = data.get("top_k", 5)
+    cheap = data.get("cheap", True)
+    result = get_ask_data(db(), question, top_k=top_k, cheap=cheap)
+    return jsonify(result)
+
+
+@app.route("/compare")
+def compare_page():
+    draft_names = request.args.get("drafts", "")
+    names = [n.strip() for n in draft_names.split(",") if n.strip()] if draft_names else []
+    data = None
+    if len(names) >= 2:
+        data = get_comparison_data(db(), names)
+    return render_template("comparison.html", names=names, data=data)
+
+
+@app.route("/api/compare", methods=["POST"])
+def api_compare():
+    """Run Claude comparison for drafts. Returns JSON with comparison text."""
+    req_data = request.get_json(force=True, silent=True)
+    if not req_data or "drafts" not in req_data:
+        return jsonify({"error": "Missing 'drafts' in request body"}), 400
+
+    names = req_data["drafts"]
+    if len(names) < 2:
+        return jsonify({"error": "Need at least 2 drafts to compare"}), 400
+
+    try:
+        from ietf_analyzer.config import Config
+        from ietf_analyzer.analyzer import Analyzer
+
+        cfg = Config.load()
+        database = db()
+        analyzer = Analyzer(cfg, database)
+        result = analyzer.compare_drafts(names)
+        return jsonify(result)
+    except Exception as e:
+        return jsonify({"error": str(e)}), 500
+
+
 # --- API endpoints for AJAX (used by client-side charts) ---
 
 
+def _to_csv_response(rows: list[dict], filename: str = "export.csv") -> Response:
+    """Convert a list of dicts to a CSV download response."""
+    if not rows:
+        return Response("", mimetype="text/csv",
+                        headers={"Content-Disposition": f"attachment; filename={filename}"})
+    si = io.StringIO()
+    writer = csv.DictWriter(si, fieldnames=rows[0].keys())
+    writer.writeheader()
+    for row in rows:
+        # Flatten any list/dict values to JSON strings
+        flat = {}
+        for k, v in row.items():
+            if isinstance(v, (list, dict)):
+                flat[k] = json.dumps(v)
+            else:
+                flat[k] = v
+        writer.writerow(flat)
+    return Response(si.getvalue(), mimetype="text/csv",
+                    headers={"Content-Disposition": f"attachment; filename={filename}"})
+
+
+def _results_to_csv(results: dict) -> Response:
+    """Convert global search results (multi-category) to a single CSV."""
+    rows = []
+    for category, items in results.items():
+        for item in items:
+            row = {"_category": category}
+            row.update(item)
+            rows.append(row)
+    return _to_csv_response(rows, "search_results.csv")
+
+
 @app.route("/api/drafts")
 def api_drafts():
     page = request.args.get("page", 1, type=int)
     search = request.args.get("q", "")
     category = request.args.get("cat", "")
+    source = request.args.get("source", "")
     min_score = request.args.get("min_score", 0.0, type=float)
     sort = request.args.get("sort", "score")
     sort_dir = request.args.get("dir", "desc")
-    return jsonify(
+    data = get_drafts_page(db(), page=page, search=search, category=category,
-        get_drafts_page(db(), page=page, search=search, category=category,
+                           min_score=min_score, sort=sort, sort_dir=sort_dir,
-                        min_score=min_score, sort=sort, sort_dir=sort_dir)
+                           source=source)
-    )
+    if request.args.get("format") == "csv":
+        return _to_csv_response(data.get("drafts", []), "drafts.csv")
+    return jsonify(data)
 
 
 @app.route("/api/stats")
@@ -321,6 +438,148 @@ def api_author_network():
     return jsonify(get_author_network_full(db()))
 
 
+@app.route("/api/citations")
+def api_citations():
+    min_refs = request.args.get("min_refs", 2, type=int)
+    return jsonify(get_citation_graph(db(), min_refs=min_refs))
+
+
+@app.route("/api/search")
+def api_search():
+    q = request.args.get("q", "").strip()
+    results = global_search(db(), q) if q else {"drafts": [], "ideas": [], "authors": [], "gaps": []}
+    if request.args.get("format") == "csv":
+        return _results_to_csv(results)
+    return jsonify(results)
+
+
+@app.route("/api/ideas")
+def api_ideas():
+    data = get_ideas_by_type(db())
+    if request.args.get("format") == "csv":
+        return _to_csv_response(data.get("ideas", []), "ideas.csv")
+    return jsonify(data)
+
+
+@app.route("/api/gaps")
+def api_gaps():
+    data = get_all_gaps(db())
+    if request.args.get("format") == "csv":
+        return _to_csv_response(data, "gaps.csv")
+    return jsonify(data)
+
+
+@app.route("/api/gaps/<int:gap_id>")
+def api_gap_detail(gap_id: int):
+    gap = get_gap_detail(db(), gap_id)
+    if not gap:
+        return jsonify({"error": "Gap not found"}), 404
+    return jsonify(gap)
+
+
+@app.route("/api/ratings")
+def api_ratings():
+    data = get_rating_distributions(db())
+    if request.args.get("format") == "csv":
+        # Transpose columnar data to rows
+        rows = []
+        for i in range(len(data.get("names", []))):
+            rows.append({
+                "name": data["names"][i],
+                "score": data["scores"][i],
+                "novelty": data["novelty"][i],
+                "maturity": data["maturity"][i],
+                "overlap": data["overlap"][i],
+                "momentum": data["momentum"][i],
+                "relevance": data["relevance"][i],
+                "category": data["categories"][i],
+            })
+        return _to_csv_response(rows, "ratings.csv")
+    return jsonify(data)
+
+
+@app.route("/api/timeline")
+def api_timeline():
+    data = get_timeline_data(db())
+    return jsonify(data)
+
+
+@app.route("/api/landscape")
+def api_landscape():
+    data = get_landscape_tsne(db())
+    if request.args.get("format") == "csv":
+        return _to_csv_response(data, "landscape.csv")
+    return jsonify(data)
+
+
+@app.route("/api/similarity")
+def api_similarity():
+    data = get_similarity_graph(db())
+    return jsonify(data)
+
+
+@app.route("/api/idea-clusters")
+def api_idea_clusters():
+    data = get_idea_clusters(db())
+    return jsonify(data)
+
+
+@app.route("/api/monitor")
+def api_monitor():
+    data = get_monitor_status(db())
+    return jsonify(data)
+
+
+@app.route("/api/drafts/<path:name>")
+def api_draft_detail(name: str):
+    detail = get_draft_detail(db(), name)
+    if not detail:
+        return jsonify({"error": "Draft not found"}), 404
+    return jsonify(detail)
+
+
+@app.route("/api/categories")
+def api_categories():
+    data = get_category_counts(db())
+    if request.args.get("format") == "csv":
+        rows = [{"category": k, "count": v} for k, v in data.items()]
+        return _to_csv_response(rows, "categories.csv")
+    return jsonify(data)
+
+
+@app.route("/api/drafts/<path:name>/annotate", methods=["POST"])
+def api_annotate(name: str):
+    """Add or update annotation for a draft."""
+    import json as _json
+    database = db()
+    draft = database.get_draft(name)
+    if not draft:
+        return jsonify({"error": "Draft not found"}), 404
+
+    data = request.get_json(force=True, silent=True)
+    if not data:
+        return jsonify({"error": "Invalid JSON body"}), 400
+
+    note = data.get("note")
+    tags = data.get("tags")
+    add_tag = data.get("add_tag")
+    remove_tag = data.get("remove_tag")
+
+    # Handle add/remove tag operations
+    if add_tag or remove_tag:
+        existing = database.get_annotation(name)
+        current_tags = existing["tags"] if existing else []
+        if add_tag and add_tag not in current_tags:
+            current_tags.append(add_tag)
+        if remove_tag and remove_tag in current_tags:
+            current_tags.remove(remove_tag)
+        tags = current_tags
+
+    database.upsert_annotation(name, note=note, tags=tags)
+    annotation = database.get_annotation(name)
+    return jsonify({"success": True, "annotation": annotation})
+
+
 if __name__ == "__main__":
     print("Starting IETF Draft Analyzer Dashboard on http://127.0.0.1:5000")
     app.run(debug=True, host="127.0.0.1", port=5000)

src/webui/data.py

@@ -66,6 +66,7 @@ def get_drafts_page(
     min_score: float = 0.0,
     sort: str = "score",
     sort_dir: str = "desc",
+    source: str = "",
 ) -> dict:
     """Return a paginated, filtered list of drafts with ratings.
 
@@ -80,6 +81,8 @@ def get_drafts_page(
             continue
         if category and category not in rating.categories:
             continue
+        if source and draft.source != source:
+            continue
         if search:
             haystack = f"{draft.name} {draft.title} {rating.summary}".lower()
             if not all(w in haystack for w in search.lower().split()):
@@ -96,6 +99,9 @@ def get_drafts_page(
         "relevance": lambda p: p[1].relevance,
         "overlap": lambda p: p[1].overlap,
         "momentum": lambda p: p[1].momentum,
+        "readiness": lambda p: (1.0 if p[0].name.startswith("draft-ietf-") else 0.0) * 0.25 +
+                                min(int(p[0].rev or "0") / 5.0, 1.0) * 0.15 +
+                                ((p[1].momentum - 1) / 4.0) * 0.15,
     }
     key_fn = sort_keys.get(sort, sort_keys["score"])
     reverse = sort_dir == "desc"
@@ -107,15 +113,23 @@ def get_drafts_page(
     start = (page - 1) * per_page
     page_items = filtered[start : start + per_page]
 
+    # Pre-compute readiness for page items (lightweight version)
+    from ietf_analyzer.readiness import compute_readiness
+    readiness_cache = {}
+    for draft, rating in page_items:
+        readiness_cache[draft.name] = compute_readiness(db, draft.name)
+
     drafts = []
     for draft, rating in page_items:
+        r_score = readiness_cache.get(draft.name, {}).get("score", 0)
         drafts.append({
             "name": draft.name,
             "title": draft.title,
             "date": draft.date,
-            "url": draft.datatracker_url,
+            "url": draft.source_url if draft.source != "ietf" else draft.datatracker_url,
             "pages": draft.pages or 0,
             "group": draft.group or "individual",
+            "source": draft.source or "ietf",
             "score": round(rating.composite_score, 2),
             "novelty": rating.novelty,
             "maturity": rating.maturity,
@@ -124,6 +138,7 @@ def get_drafts_page(
             "relevance": rating.relevance,
             "categories": rating.categories,
             "summary": rating.summary,
+            "readiness": r_score,
         })
 
     return {
@@ -185,6 +200,14 @@ def get_draft_detail(db: Database, name: str) -> dict | None:
             "categories": rating.categories,
         }
 
+    # Readiness score
+    from ietf_analyzer.readiness import compute_readiness
+    result["readiness"] = compute_readiness(db, name)
+
+    # Annotation
+    annotation = db.get_annotation(name)
+    result["annotation"] = annotation
+
     return result
 
 
@@ -253,8 +276,11 @@ def get_ideas_by_type(db: Database) -> dict:
 
 
 def get_all_gaps(db: Database) -> list[dict]:
-    """Return all gap analysis results."""
+    """Return all gap analysis results, sorted by severity (critical first)."""
-    return db.all_gaps()
+    _sev_order = {"critical": 0, "high": 1, "medium": 2, "low": 3}
+    gaps = db.all_gaps()
+    gaps.sort(key=lambda g: _sev_order.get(g.get("severity", "low"), 99))
+    return gaps
 
 
 def get_gap_detail(db: Database, gap_id: int) -> dict | None:
@@ -775,17 +801,252 @@ def get_monitor_status(db: Database) -> dict:
     """Return monitoring status data for dashboard."""
     runs = db.get_monitor_runs(limit=20)
     last = runs[0] if runs else None
+    total_drafts = db.count_drafts()
+    rated_count = len(db.drafts_with_ratings(limit=10000))
     unrated = len(db.unrated_drafts(limit=9999))
     unembedded = len(db.drafts_without_embeddings(limit=9999))
+    embedded_count = total_drafts - unembedded
     no_ideas = len(db.drafts_without_ideas(limit=9999))
+    ideas_count = total_drafts - no_ideas
+    idea_total = db.idea_count()
+    gap_count = len(db.all_gaps())
+    input_tok, output_tok = db.total_tokens_used()
+
+    # Estimate cost (Sonnet pricing: $3/M input, $15/M output)
+    est_cost = (input_tok * 3.0 / 1_000_000) + (output_tok * 15.0 / 1_000_000)
+
     return {
         "last_run": last,
         "runs": runs,
         "unprocessed": {"unrated": unrated, "unembedded": unembedded, "no_ideas": no_ideas},
         "total_runs": len(runs),
+        "pipeline": {
+            "total_drafts": total_drafts,
+            "rated": rated_count,
+            "embedded": embedded_count,
+            "with_ideas": ideas_count,
+            "idea_total": idea_total,
+            "gap_count": gap_count,
+        },
+        "cost": {
+            "input_tokens": input_tok,
+            "output_tokens": output_tok,
+            "estimated_usd": round(est_cost, 2),
+        },
     }
 
 
+def get_citation_graph(db: Database, min_refs: int = 2) -> dict:
+    """Return citation network data for force-directed graph.
+
+    Returns {nodes: [{id, type, title, influence, ...}],
+             edges: [{source, target}],
+             stats: {node_count, edge_count, ...}}
+    """
+    # Get all references
+    rows = db.conn.execute(
+        "SELECT draft_name, ref_type, ref_id FROM draft_refs"
+    ).fetchall()
+
+    # Count in-degree for each referenced item
+    in_degree: dict[str, int] = Counter()
+    edges_raw = []
+    for r in rows:
+        ref_key = f"{r['ref_type']}:{r['ref_id']}"
+        in_degree[ref_key] += 1
+        edges_raw.append((r["draft_name"], ref_key))
+
+    # Also count drafts as source nodes
+    draft_out: dict[str, int] = Counter()
+    for draft_name, _ in edges_raw:
+        draft_out[draft_name] += 1
+
+    # Get draft titles for labeling
+    draft_rows = db.conn.execute("SELECT name, title FROM drafts").fetchall()
+    draft_titles = {r["name"]: r["title"] for r in draft_rows}
+
+    # Get rating categories for draft coloring
+    rating_rows = db.conn.execute("SELECT draft_name, categories FROM ratings").fetchall()
+    draft_cats = {}
+    for r in rating_rows:
+        try:
+            cats = json.loads(r["categories"]) if r["categories"] else []
+            draft_cats[r["draft_name"]] = cats[0] if cats else "Other"
+        except Exception:
+            draft_cats[r["draft_name"]] = "Other"
+
+    # Filter: keep RFCs with min_refs+ references and all drafts that reference them
+    top_refs = {k: v for k, v in in_degree.items() if v >= min_refs}
+
+    # Build node set
+    node_set = set()
+    filtered_edges = []
+    for draft_name, ref_key in edges_raw:
+        if ref_key in top_refs:
+            node_set.add(draft_name)
+            node_set.add(ref_key)
+            filtered_edges.append({"source": draft_name, "target": ref_key})
+
+    # Limit to ~200 nodes max for readability
+    if len(node_set) > 250:
+        # Keep only refs with higher in-degree
+        sorted_refs = sorted(top_refs.items(), key=lambda x: x[1], reverse=True)
+        keep_refs = set(k for k, _ in sorted_refs[:80])
+        node_set = set()
+        filtered_edges = []
+        for draft_name, ref_key in edges_raw:
+            if ref_key in keep_refs:
+                node_set.add(draft_name)
+                node_set.add(ref_key)
+                filtered_edges.append({"source": draft_name, "target": ref_key})
+
+    # Build nodes
+    nodes = []
+    for nid in node_set:
+        if ":" in nid and not nid.startswith("draft-"):
+            # It's a reference node (rfc:1234, bcp:14, etc.)
+            ref_type, ref_id = nid.split(":", 1)
+            influence = in_degree.get(nid, 0)
+            if ref_type == "rfc":
+                try:
+                    title = f"RFC {int(ref_id)}"
+                except ValueError:
+                    title = f"RFC {ref_id}"
+            else:
+                title = f"{ref_type.upper()} {ref_id}"
+            nodes.append({
+                "id": nid,
+                "type": ref_type,
+                "title": title,
+                "influence": influence,
+                "ref_id": ref_id,
+            })
+        else:
+            # It's a draft node
+            influence = in_degree.get(nid, 0) + draft_out.get(nid, 0)
+            nodes.append({
+                "id": nid,
+                "type": "draft",
+                "title": draft_titles.get(nid, nid),
+                "influence": draft_out.get(nid, 0),
+                "category": draft_cats.get(nid, "Other"),
+            })
+
+    # Stats
+    rfc_count = sum(1 for n in nodes if n["type"] == "rfc")
+    draft_count = sum(1 for n in nodes if n["type"] == "draft")
+
+    return {
+        "nodes": nodes,
+        "edges": filtered_edges,
+        "stats": {
+            "node_count": len(nodes),
+            "edge_count": len(filtered_edges),
+            "rfc_count": rfc_count,
+            "draft_count": draft_count,
+        },
+    }
+
+
+def global_search(db: Database, query: str) -> dict:
+    """Search across drafts (FTS5), ideas, authors, and gaps.
+
+    Returns {drafts: [...], ideas: [...], authors: [...], gaps: [...]}.
+    """
+    results: dict = {"drafts": [], "ideas": [], "authors": [], "gaps": []}
+    if not query or not query.strip():
+        return results
+
+    q = query.strip()
+
+    # 1. Drafts via FTS5
+    try:
+        fts_query = " ".join(f'"{w}"' for w in q.split() if w)
+        rows = db.conn.execute(
+            """SELECT d.name, d.title, d.abstract, d.time, d."group"
+            FROM drafts d
+            JOIN drafts_fts f ON d.rowid = f.rowid
+            WHERE drafts_fts MATCH ?
+            ORDER BY rank
+            LIMIT 50""",
+            (fts_query,),
+        ).fetchall()
+        for r in rows:
+            results["drafts"].append({
+                "name": r["name"],
+                "title": r["title"],
+                "abstract": (r["abstract"] or "")[:200],
+                "date": r["time"],
+                "group": r["group"] or "individual",
+            })
+    except Exception:
+        # FTS5 match can fail on certain query syntax; fall back to LIKE
+        like = f"%{q}%"
+        rows = db.conn.execute(
+            """SELECT name, title, abstract, time, "group" FROM drafts
+            WHERE title LIKE ? OR name LIKE ? OR abstract LIKE ?
+            LIMIT 50""",
+            (like, like, like),
+        ).fetchall()
+        for r in rows:
+            results["drafts"].append({
+                "name": r["name"],
+                "title": r["title"],
+                "abstract": (r["abstract"] or "")[:200],
+                "date": r["time"],
+                "group": r["group"] or "individual",
+            })
+
+    # 2. Ideas via LIKE
+    like = f"%{q}%"
+    rows = db.conn.execute(
+        """SELECT id, title, description, idea_type, draft_name FROM ideas
+        WHERE title LIKE ? OR description LIKE ?
+        ORDER BY id LIMIT 50""",
+        (like, like),
+    ).fetchall()
+    for r in rows:
+        results["ideas"].append({
+            "id": r["id"],
+            "title": r["title"],
+            "description": (r["description"] or "")[:200],
+            "type": r["idea_type"],
+            "draft_name": r["draft_name"],
+        })
+
+    # 3. Authors via LIKE
+    rows = db.conn.execute(
+        """SELECT person_id, name, affiliation FROM authors
+        WHERE name LIKE ? OR affiliation LIKE ?
+        ORDER BY name LIMIT 50""",
+        (like, like),
+    ).fetchall()
+    for r in rows:
+        results["authors"].append({
+            "person_id": r["person_id"],
+            "name": r["name"],
+            "affiliation": r["affiliation"] or "",
+        })
+
+    # 4. Gaps via LIKE
+    rows = db.conn.execute(
+        """SELECT id, topic, description, category, severity FROM gaps
+        WHERE topic LIKE ? OR description LIKE ?
+        ORDER BY id LIMIT 50""",
+        (like, like),
+    ).fetchall()
+    for r in rows:
+        results["gaps"].append({
+            "id": r["id"],
+            "topic": r["topic"],
+            "description": (r["description"] or "")[:200],
+            "category": r["category"],
+            "severity": r["severity"],
+        })
+
+    return results
+
+
 def get_landscape_tsne(db: Database) -> list[dict]:
     """Compute t-SNE from embeddings, return [{name, title, x, y, category, score}].
 
@@ -829,3 +1090,116 @@ def get_landscape_tsne(db: Database) -> list[dict]:
             "score": round(r.composite_score, 2),
         })
     return result
+
+
+def get_comparison_data(db: Database, names: list[str]) -> dict | None:
+    """Get comparison data for a list of drafts.
+
+    Returns {
+        drafts: [{name, title, abstract, rating, ideas, refs, ...}],
+        shared_ideas: [{title, drafts: [name,...]}],
+        unique_ideas: {name: [{title, description}]},
+        shared_refs: [{type, id, drafts: [name,...]}],
+        unique_refs: {name: [{type, id}]},
+        similarities: [{a, b, similarity}],
+        comparison_text: str | None,
+    }
+    """
+    import numpy as np
+
+    drafts_data = []
+    all_ideas: dict[str, list[dict]] = {}
+    all_refs: dict[str, list[tuple[str, str]]] = {}
+
+    for name in names:
+        detail = get_draft_detail(db, name)
+        if not detail:
+            continue
+        drafts_data.append(detail)
+        all_ideas[name] = detail.get("ideas", [])
+        all_refs[name] = [(r["type"], r["id"]) for r in detail.get("refs", [])]
+
+    if len(drafts_data) < 2:
+        return None
+
+    # Find shared vs unique ideas (by title similarity)
+    idea_title_drafts: dict[str, list[str]] = {}
+    for name, ideas in all_ideas.items():
+        for idea in ideas:
+            title_lower = idea["title"].lower().strip()
+            if title_lower not in idea_title_drafts:
+                idea_title_drafts[title_lower] = []
+            idea_title_drafts[title_lower].append(name)
+
+    shared_ideas = [
+        {"title": title, "drafts": draft_list}
+        for title, draft_list in idea_title_drafts.items()
+        if len(set(draft_list)) > 1
+    ]
+    unique_ideas: dict[str, list[dict]] = {}
+    for name, ideas in all_ideas.items():
+        unique = []
+        for idea in ideas:
+            title_lower = idea["title"].lower().strip()
+            if len(set(idea_title_drafts.get(title_lower, []))) <= 1:
+                unique.append({"title": idea["title"], "description": idea.get("description", "")})
+        unique_ideas[name] = unique
+
+    # Find shared vs unique references
+    ref_drafts: dict[tuple[str, str], list[str]] = {}
+    for name, refs in all_refs.items():
+        for ref in refs:
+            if ref not in ref_drafts:
+                ref_drafts[ref] = []
+            ref_drafts[ref].append(name)
+
+    shared_refs = [
+        {"type": ref[0], "id": ref[1], "drafts": draft_list}
+        for ref, draft_list in ref_drafts.items()
+        if len(set(draft_list)) > 1
+    ]
+    unique_refs: dict[str, list[dict]] = {}
+    for name, refs in all_refs.items():
+        unique = []
+        for ref in refs:
+            if len(set(ref_drafts.get(ref, []))) <= 1:
+                unique.append({"type": ref[0], "id": ref[1]})
+        unique_refs[name] = unique
+
+    # Pairwise embedding similarities
+    embeddings = db.all_embeddings()
+    similarities = []
+    valid_names = [d["name"] for d in drafts_data]
+    for i in range(len(valid_names)):
+        for j in range(i + 1, len(valid_names)):
+            a, b = valid_names[i], valid_names[j]
+            if a in embeddings and b in embeddings:
+                vec_a = embeddings[a]
+                vec_b = embeddings[b]
+                dot = np.dot(vec_a, vec_b)
+                norm = np.linalg.norm(vec_a) * np.linalg.norm(vec_b)
+                sim = float(dot / norm) if norm > 0 else 0.0
+                similarities.append({"a": a, "b": b, "similarity": round(sim, 4)})
+
+    return {
+        "drafts": drafts_data,
+        "shared_ideas": shared_ideas,
+        "unique_ideas": unique_ideas,
+        "shared_refs": shared_refs,
+        "unique_refs": unique_refs,
+        "similarities": similarities,
+        "comparison_text": None,
+    }
+
+
+def get_ask_data(db: Database, question: str, top_k: int = 5, cheap: bool = True) -> dict:
+    """Run hybrid search + Claude synthesis for a question.
+
+    Returns {answer: str, sources: [{name, title, similarity, excerpt}]}.
+    """
+    from ietf_analyzer.config import Config
+    from ietf_analyzer.search import HybridSearch
+
+    config = Config.load()
+    searcher = HybridSearch(config, db)
+    return searcher.ask(question, top_k=top_k, cheap=cheap)

src/webui/templates/ask.html

@@ -0,0 +1,153 @@
+{% extends "base.html" %}
+{% set active_page = "ask" %}
+
+{% block title %}Ask — IETF Draft Analyzer{% endblock %}
+
+{% block extra_head %}
+<style>
+    .ask-input {
+        background: linear-gradient(135deg, rgba(30, 41, 59, 0.8), rgba(30, 41, 59, 0.4));
+        backdrop-filter: blur(10px);
+    }
+    .answer-card {
+        background: linear-gradient(135deg, rgba(30, 41, 59, 0.8), rgba(30, 41, 59, 0.4));
+        backdrop-filter: blur(10px);
+    }
+    .source-row {
+        transition: all 0.15s ease;
+    }
+    .source-row:hover {
+        background: rgba(59, 130, 246, 0.05);
+    }
+    .loading-spinner {
+        border: 3px solid rgba(59, 130, 246, 0.2);
+        border-top-color: #3b82f6;
+        border-radius: 50%;
+        width: 24px;
+        height: 24px;
+        animation: spin 0.8s linear infinite;
+        display: inline-block;
+    }
+    @keyframes spin {
+        to { transform: rotate(360deg); }
+    }
+</style>
+{% endblock %}
+
+{% block content %}
+<!-- Header -->
+<div class="mb-8 text-center">
+    <h1 class="text-3xl font-bold text-white">Ask the Draft Corpus</h1>
+    <p class="text-slate-400 text-sm mt-2">Ask natural language questions about IETF AI/agent drafts. Answers are synthesized from the most relevant documents.</p>
+</div>
+
+<!-- Search Bar -->
+<div class="max-w-3xl mx-auto mb-8">
+    <form method="get" action="/ask" id="askForm">
+        <div class="ask-input rounded-xl border border-slate-700 p-2 flex items-center gap-2">
+            <svg class="w-5 h-5 text-slate-500 ml-3 flex-shrink-0" fill="none" stroke="currentColor" viewBox="0 0 24 24">
+                <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M8.228 9c.549-1.165 2.03-2 3.772-2 2.21 0 4 1.343 4 3 0 1.4-1.278 2.575-3.006 2.907-.542.104-.994.54-.994 1.093m0 3h.01M21 12a9 9 0 11-18 0 9 9 0 0118 0z"/>
+            </svg>
+            <input type="text" name="q" value="{{ question }}" placeholder="Which drafts address agent authentication? What approaches exist for agent delegation?"
+                class="flex-1 bg-transparent border-0 px-3 py-3 text-base text-slate-200 placeholder-slate-500 focus:outline-none"
+                autofocus>
+            <button type="submit" class="px-6 py-3 bg-blue-600 text-white rounded-lg text-sm font-medium hover:bg-blue-500 transition-colors flex-shrink-0">
+                Ask
+            </button>
+        </div>
+        <div class="flex items-center gap-4 mt-3 px-2">
+            <label class="text-xs text-slate-500 flex items-center gap-1.5">
+                <span>Sources:</span>
+                <select name="top" class="bg-slate-800/60 border border-slate-700 rounded px-2 py-1 text-xs text-slate-300 focus:outline-none">
+                    <option value="3" {% if request.args.get('top', '5') == '3' %}selected{% endif %}>3</option>
+                    <option value="5" {% if request.args.get('top', '5') == '5' %}selected{% endif %}>5</option>
+                    <option value="10" {% if request.args.get('top', '5') == '10' %}selected{% endif %}>10</option>
+                </select>
+            </label>
+            <div class="text-xs text-slate-600">Combines keyword search + semantic similarity</div>
+        </div>
+    </form>
+</div>
+
+<!-- Example questions (show when no query) -->
+{% if not question %}
+<div class="max-w-3xl mx-auto">
+    <div class="text-xs text-slate-500 uppercase tracking-wide mb-3 font-medium">Example questions</div>
+    <div class="grid grid-cols-1 md:grid-cols-2 gap-2">
+        {% set examples = [
+            "Which drafts address agent authentication and identity?",
+            "What are the competing approaches to agent-to-agent communication?",
+            "How do safety mechanisms work across different proposals?",
+            "What protocols exist for AI model serving and inference?",
+            "Which drafts propose agent discovery or registration systems?",
+            "What are the main gaps in autonomous network operations?",
+        ] %}
+        {% for q in examples %}
+        <a href="/ask?q={{ q | urlencode }}" class="ask-input rounded-lg border border-slate-800 px-4 py-3 text-sm text-slate-400 hover:text-blue-400 hover:border-slate-700 transition">
+            {{ q }}
+        </a>
+        {% endfor %}
+    </div>
+</div>
+{% endif %}
+
+<!-- Answer -->
+{% if result %}
+<div class="max-w-3xl mx-auto">
+    <!-- Synthesized answer -->
+    <div class="answer-card rounded-xl border border-slate-800 p-6 mb-6">
+        <div class="flex items-center gap-2 mb-4">
+            <svg class="w-5 h-5 text-blue-400" fill="none" stroke="currentColor" viewBox="0 0 24 24">
+                <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9.663 17h4.673M12 3v1m6.364 1.636l-.707.707M21 12h-1M4 12H3m3.343-5.657l-.707-.707m2.828 9.9a5 5 0 117.072 0l-.548.547A3.374 3.374 0 0014 18.469V19a2 2 0 11-4 0v-.531c0-.895-.356-1.754-.988-2.386l-.548-.547z"/>
+            </svg>
+            <h2 class="text-lg font-semibold text-white">Answer</h2>
+        </div>
+        <div class="text-slate-300 text-sm leading-relaxed whitespace-pre-line">{{ result.answer }}</div>
+    </div>
+
+    <!-- Source drafts -->
+    {% if result.sources %}
+    <div class="answer-card rounded-xl border border-slate-800 overflow-hidden">
+        <div class="px-6 py-4 border-b border-slate-800">
+            <h3 class="text-sm font-semibold text-slate-300">Source Drafts ({{ result.sources|length }})</h3>
+        </div>
+        <table class="w-full text-sm">
+            <thead>
+                <tr class="border-b border-slate-800/50 bg-slate-900/40">
+                    <th class="px-4 py-2.5 text-left text-xs font-medium text-slate-500 uppercase w-8">#</th>
+                    <th class="px-4 py-2.5 text-left text-xs font-medium text-slate-500 uppercase">Draft</th>
+                    <th class="px-4 py-2.5 text-left text-xs font-medium text-slate-500 uppercase w-20">Match</th>
+                    <th class="px-4 py-2.5 text-right text-xs font-medium text-slate-500 uppercase w-16">Score</th>
+                </tr>
+            </thead>
+            <tbody class="divide-y divide-slate-800/30">
+                {% for src in result.sources %}
+                <tr class="source-row">
+                    <td class="px-4 py-3 text-slate-600">{{ loop.index }}</td>
+                    <td class="px-4 py-3">
+                        <a href="/drafts/{{ src.name }}" class="text-blue-400 hover:text-blue-300 font-medium transition">
+                            {{ src.title }}
+                        </a>
+                        <div class="text-xs text-slate-600 mt-0.5 font-mono">{{ src.name }}</div>
+                    </td>
+                    <td class="px-4 py-3">
+                        {% if src.match_type == 'both' %}
+                        <span class="inline-block px-2 py-0.5 rounded text-xs font-medium bg-green-900/30 text-green-400 border border-green-800/30">both</span>
+                        {% elif src.match_type == 'semantic' %}
+                        <span class="inline-block px-2 py-0.5 rounded text-xs font-medium bg-purple-900/30 text-purple-400 border border-purple-800/30">semantic</span>
+                        {% else %}
+                        <span class="inline-block px-2 py-0.5 rounded text-xs font-medium bg-slate-800/50 text-slate-400 border border-slate-700/30">keyword</span>
+                        {% endif %}
+                    </td>
+                    <td class="px-4 py-3 text-right font-mono text-xs text-slate-400">
+                        {{ "%.3f"|format(src.similarity) if src.similarity else "-" }}
+                    </td>
+                </tr>
+                {% endfor %}
+            </tbody>
+        </table>
+    </div>
+    {% endif %}
+</div>
+{% endif %}
+{% endblock %}

src/webui/templates/base.html

@@ -82,11 +82,26 @@
             <h1 class="text-lg font-bold text-white tracking-tight">IETF Draft Analyzer</h1>
             <p class="text-xs text-slate-500 mt-1">AI/Agent Standards Tracker</p>
         </div>
+        <!-- Global Search -->
+        <div class="px-4 pt-4 pb-2">
+            <form action="/search" method="get" class="relative">
+                <input type="text" name="q" placeholder="Search everything..."
+                    value="{{ request.args.get('q', '') if request else '' }}"
+                    class="w-full bg-slate-800/60 border border-slate-700 rounded-lg pl-9 pr-3 py-2 text-sm text-slate-200 placeholder-slate-500 focus:outline-none focus:border-blue-500 focus:ring-1 focus:ring-blue-500/30 transition">
+                <svg class="w-4 h-4 absolute left-3 top-1/2 -translate-y-1/2 text-slate-500" fill="none" stroke="currentColor" viewBox="0 0 24 24">
+                    <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M21 21l-6-6m2-5a7 7 0 11-14 0 7 7 0 0114 0z"/>
+                </svg>
+            </form>
+        </div>
         <nav class="flex-1 py-4 overflow-y-auto">
             <a href="/" class="sidebar-link flex items-center gap-3 px-5 py-2.5 text-sm text-slate-300 {{ 'active' if active_page == 'overview' }}">
                 <svg class="w-4 h-4 opacity-60" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 6a2 2 0 012-2h2a2 2 0 012 2v2a2 2 0 01-2 2H6a2 2 0 01-2-2V6zm10 0a2 2 0 012-2h2a2 2 0 012 2v2a2 2 0 01-2 2h-2a2 2 0 01-2-2V6zM4 16a2 2 0 012-2h2a2 2 0 012 2v2a2 2 0 01-2 2H6a2 2 0 01-2-2v-2zm10 0a2 2 0 012-2h2a2 2 0 012 2v2a2 2 0 01-2 2h-2a2 2 0 01-2-2v-2z"/></svg>
                 Overview
             </a>
+            <a href="/ask" class="sidebar-link flex items-center gap-3 px-5 py-2.5 text-sm text-slate-300 {{ 'active' if active_page == 'ask' }}">
+                <svg class="w-4 h-4 opacity-60" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M8.228 9c.549-1.165 2.03-2 3.772-2 2.21 0 4 1.343 4 3 0 1.4-1.278 2.575-3.006 2.907-.542.104-.994.54-.994 1.093m0 3h.01M21 12a9 9 0 11-18 0 9 9 0 0118 0z"/></svg>
+                Ask
+            </a>
             <a href="/drafts" class="sidebar-link flex items-center gap-3 px-5 py-2.5 text-sm text-slate-300 {{ 'active' if active_page == 'drafts' }}">
                 <svg class="w-4 h-4 opacity-60" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 12h6m-6 4h6m2 5H7a2 2 0 01-2-2V5a2 2 0 012-2h5.586a1 1 0 01.707.293l5.414 5.414a1 1 0 01.293.707V19a2 2 0 01-2 2z"/></svg>
                 Draft Explorer
@@ -119,6 +134,10 @@
                 <svg class="w-4 h-4 opacity-60" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M13 10V3L4 14h7v7l9-11h-7z"/><circle cx="5" cy="6" r="1.5" fill="currentColor" stroke="none"/><circle cx="19" cy="18" r="1.5" fill="currentColor" stroke="none"/><circle cx="18" cy="6" r="1.5" fill="currentColor" stroke="none"/></svg>
                 Similarity
             </a>
+            <a href="/citations" class="sidebar-link flex items-center gap-3 px-5 py-2.5 text-sm text-slate-300 {{ 'active' if active_page == 'citations' }}">
+                <svg class="w-4 h-4 opacity-60" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M13.828 10.172a4 4 0 00-5.656 0l-4 4a4 4 0 105.656 5.656l1.102-1.101m-.758-4.899a4 4 0 005.656 0l4-4a4 4 0 00-5.656-5.656l-1.1 1.1"/></svg>
+                Citations
+            </a>
             <a href="/authors" class="sidebar-link flex items-center gap-3 px-5 py-2.5 text-sm text-slate-300 {{ 'active' if active_page == 'authors' }}">
                 <svg class="w-4 h-4 opacity-60" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M17 20h5v-2a3 3 0 00-5.356-1.857M17 20H7m10 0v-2c0-.656-.126-1.283-.356-1.857M7 20H2v-2a3 3 0 015.356-1.857M7 20v-2c0-.656.126-1.283.356-1.857m0 0a5.002 5.002 0 019.288 0M15 7a3 3 0 11-6 0 3 3 0 016 0zm6 3a2 2 0 11-4 0 2 2 0 014 0zM7 10a2 2 0 11-4 0 2 2 0 014 0z"/></svg>
                 Authors

src/webui/templates/citations.html

@@ -0,0 +1,392 @@
+{% extends "base.html" %}
+{% set active_page = "citations" %}
+
+{% block title %}Citation Graph — IETF Draft Analyzer{% endblock %}
+
+{% block extra_head %}
+<script src="/static/js/d3.v7.min.js"></script>
+<style>
+    #citationSvg {
+        width: 100%;
+        height: 650px;
+        cursor: grab;
+    }
+    #citationSvg:active { cursor: grabbing; }
+    #citationSvg .node { cursor: pointer; }
+    #citationSvg .node circle { stroke: rgba(255,255,255,0.15); stroke-width: 1.5px; transition: r 0.2s; }
+    #citationSvg .node:hover circle { stroke: #60a5fa; stroke-width: 2.5px; }
+    #citationSvg .node text { pointer-events: none; }
+    #citationSvg .link { stroke-opacity: 0.15; }
+    #citationSvg .link:hover { stroke-opacity: 0.5; }
+    .tooltip-card {
+        position: absolute; pointer-events: none; z-index: 50;
+        background: #1e293b; border: 1px solid #334155; border-radius: 8px;
+        padding: 10px 14px; font-size: 12px; color: #e2e8f0;
+        box-shadow: 0 8px 24px rgba(0,0,0,0.4);
+        max-width: 320px; opacity: 0; transition: opacity 0.15s;
+    }
+    .tooltip-card.visible { opacity: 1; }
+    .legend-swatch { width: 12px; height: 12px; border-radius: 3px; display: inline-block; }
+    .filter-btn { transition: all 0.15s; }
+    .filter-btn:hover { background: rgba(59, 130, 246, 0.2); }
+    .filter-btn.active { background: rgba(59, 130, 246, 0.3); border-color: #3b82f6; color: #60a5fa; }
+</style>
+{% endblock %}
+
+{% block content %}
+<div class="mb-6">
+    <h1 class="text-2xl font-bold text-white">Citation Graph</h1>
+    <p class="text-slate-400 text-sm mt-1">Cross-reference network: {{ graph.stats.draft_count }} drafts referencing {{ graph.stats.rfc_count }} RFCs</p>
+</div>
+
+<!-- Summary stats -->
+<div class="grid grid-cols-2 md:grid-cols-4 gap-4 mb-6">
+    <div class="stat-card rounded-xl border border-slate-800 p-4 relative overflow-hidden">
+        <div class="absolute top-0 left-0 w-full h-1 bg-gradient-to-r from-blue-500 to-blue-400"></div>
+        <div class="text-xs text-slate-500 uppercase tracking-wider">Drafts</div>
+        <div class="text-2xl font-bold text-white mt-1">{{ graph.stats.draft_count }}</div>
+    </div>
+    <div class="stat-card rounded-xl border border-slate-800 p-4 relative overflow-hidden">
+        <div class="absolute top-0 left-0 w-full h-1 bg-gradient-to-r from-orange-500 to-orange-400"></div>
+        <div class="text-xs text-slate-500 uppercase tracking-wider">Referenced RFCs</div>
+        <div class="text-2xl font-bold text-white mt-1">{{ graph.stats.rfc_count }}</div>
+    </div>
+    <div class="stat-card rounded-xl border border-slate-800 p-4 relative overflow-hidden">
+        <div class="absolute top-0 left-0 w-full h-1 bg-gradient-to-r from-emerald-500 to-emerald-400"></div>
+        <div class="text-xs text-slate-500 uppercase tracking-wider">Total Nodes</div>
+        <div class="text-2xl font-bold text-white mt-1">{{ graph.stats.node_count }}</div>
+    </div>
+    <div class="stat-card rounded-xl border border-slate-800 p-4 relative overflow-hidden">
+        <div class="absolute top-0 left-0 w-full h-1 bg-gradient-to-r from-purple-500 to-purple-400"></div>
+        <div class="text-xs text-slate-500 uppercase tracking-wider">Citation Links</div>
+        <div class="text-2xl font-bold text-white mt-1">{{ graph.stats.edge_count }}</div>
+    </div>
+</div>
+
+<!-- D3 Force-directed Citation Graph -->
+<div class="bg-slate-900 rounded-xl border border-slate-800 p-5 mb-6 relative">
+    <div class="flex flex-wrap items-center justify-between gap-3 mb-3">
+        <div>
+            <h2 class="text-sm font-semibold text-slate-300">Cross-Reference Network</h2>
+            <p class="text-xs text-slate-500 mt-0.5">
+                <span class="inline-block w-2.5 h-2.5 rounded-full bg-blue-500 align-middle mr-1"></span>Drafts
+                <span class="inline-block w-2.5 h-2.5 rounded-full bg-orange-500 align-middle mr-1 ml-3"></span>RFCs
+                — Node size = influence (in-degree). Drag to rearrange. Scroll to zoom.
+            </p>
+        </div>
+        <div class="flex gap-2 items-center">
+            <button id="resetZoom" class="text-xs px-3 py-1.5 rounded-lg border border-slate-700 text-slate-400 hover:text-white hover:border-slate-500 transition">Reset View</button>
+            <select id="filterCategory" class="text-xs px-3 py-1.5 rounded-lg border border-slate-700 bg-slate-800 text-slate-300 focus:outline-none focus:border-blue-500">
+                <option value="">All Categories</option>
+            </select>
+            <label class="text-xs text-slate-500 ml-2">Min refs:</label>
+            <input type="range" id="minRefsSlider" min="1" max="15" value="2" class="w-20" style="accent-color: #3b82f6;">
+            <span id="minRefsVal" class="text-xs text-blue-400 font-mono w-4">2</span>
+        </div>
+    </div>
+    <div class="relative">
+        <svg id="citationSvg"></svg>
+        <div id="tooltip" class="tooltip-card"></div>
+    </div>
+</div>
+
+<!-- Top Referenced RFCs Table -->
+<div class="bg-slate-900 rounded-xl border border-slate-800 overflow-hidden">
+    <div class="p-4 border-b border-slate-800">
+        <h2 class="text-sm font-semibold text-slate-300">Most Referenced RFCs</h2>
+        <p class="text-xs text-slate-500 mt-0.5">RFCs cited by the most drafts in the corpus</p>
+    </div>
+    <div class="overflow-x-auto max-h-[500px] overflow-y-auto">
+        <table class="w-full text-sm" id="rfcTable">
+            <thead class="sticky top-0 z-10">
+                <tr class="border-b border-slate-800 bg-slate-900">
+                    <th class="px-4 py-2.5 text-left text-xs font-medium text-slate-400">#</th>
+                    <th class="px-4 py-2.5 text-left text-xs font-medium text-slate-400">RFC</th>
+                    <th class="px-4 py-2.5 text-right text-xs font-medium text-slate-400">Cited By</th>
+                </tr>
+            </thead>
+            <tbody class="divide-y divide-slate-800/50" id="rfcBody">
+            </tbody>
+        </table>
+    </div>
+</div>
+{% endblock %}
+
+{% block extra_scripts %}
+<script>
+const graph = {{ graph | tojson }};
+
+const PALETTE = [
+    '#3b82f6', '#ef4444', '#22c55e', '#a855f7', '#f59e0b',
+    '#06b6d4', '#ec4899', '#84cc16', '#f97316', '#8b5cf6',
+];
+
+// ===========================================================
+//  D3.js Force-Directed Citation Network
+// ===========================================================
+(function() {
+    if (graph.nodes.length === 0) {
+        document.getElementById('citationSvg').outerHTML =
+            '<p class="text-slate-500 text-sm text-center py-20">No citation data available</p>';
+        return;
+    }
+
+    const svg = d3.select('#citationSvg');
+    const container = svg.node().parentElement;
+    const width = container.clientWidth;
+    const height = 650;
+    svg.attr('viewBox', [0, 0, width, height]);
+
+    // Collect categories for filter dropdown
+    const categories = new Set();
+    graph.nodes.forEach(n => {
+        if (n.category && n.type === 'draft') categories.add(n.category);
+    });
+    const catSelect = document.getElementById('filterCategory');
+    [...categories].sort().forEach(cat => {
+        const opt = document.createElement('option');
+        opt.value = cat;
+        opt.textContent = cat;
+        catSelect.appendChild(opt);
+    });
+
+    // Build RFC table
+    const rfcNodes = graph.nodes
+        .filter(n => n.type === 'rfc')
+        .sort((a, b) => b.influence - a.influence);
+    const rfcBody = document.getElementById('rfcBody');
+    rfcNodes.forEach((rfc, i) => {
+        const tr = document.createElement('tr');
+        tr.className = 'hover:bg-slate-800/50 transition';
+        tr.innerHTML = `
+            <td class="px-4 py-2.5 text-slate-500 text-xs">${i + 1}</td>
+            <td class="px-4 py-2.5">
+                <a href="https://www.rfc-editor.org/rfc/rfc${parseInt(rfc.ref_id)}" target="_blank" rel="noopener"
+                   class="text-orange-400 hover:text-orange-300 transition font-medium text-sm">${rfc.title}</a>
+            </td>
+            <td class="px-4 py-2.5 text-right">
+                <span class="px-2 py-0.5 rounded-full text-xs font-medium
+                    ${rfc.influence >= 10 ? 'bg-orange-500/20 text-orange-400' :
+                      rfc.influence >= 5 ? 'bg-blue-500/20 text-blue-400' :
+                      'bg-slate-700/50 text-slate-400'}">
+                    ${rfc.influence}
+                </span>
+            </td>
+        `;
+        rfcBody.appendChild(tr);
+    });
+
+    // Prepare simulation data
+    const nodes = graph.nodes.map(n => ({...n}));
+    const links = graph.edges.map(e => ({source: e.source, target: e.target}));
+
+    // Size scale
+    const maxInfluence = d3.max(nodes, n => n.influence) || 1;
+    const rScale = d3.scaleSqrt().domain([0, maxInfluence]).range([3, 24]);
+
+    // Color: drafts = blue, rfcs = orange, others = amber
+    function nodeColor(n) {
+        if (n.type === 'rfc') return '#f59e0b';
+        if (n.type === 'bcp') return '#eab308';
+        return '#3b82f6';
+    }
+
+    // Force simulation
+    const simulation = d3.forceSimulation(nodes)
+        .force('link', d3.forceLink(links)
+            .id(d => d.id)
+            .distance(60)
+            .strength(0.15)
+        )
+        .force('charge', d3.forceManyBody().strength(-80).distanceMax(350))
+        .force('center', d3.forceCenter(width / 2, height / 2))
+        .force('collision', d3.forceCollide().radius(d => rScale(d.influence) + 2))
+        .force('x', d3.forceX(width / 2).strength(0.04))
+        .force('y', d3.forceY(height / 2).strength(0.04));
+
+    // Zoom behavior
+    const g = svg.append('g');
+    const zoom = d3.zoom()
+        .scaleExtent([0.15, 5])
+        .on('zoom', (event) => g.attr('transform', event.transform));
+    svg.call(zoom);
+
+    document.getElementById('resetZoom').addEventListener('click', () => {
+        svg.transition().duration(500).call(zoom.transform, d3.zoomIdentity);
+    });
+
+    // Draw edges
+    const linkGroup = g.append('g').attr('class', 'links');
+    const link = linkGroup.selectAll('line')
+        .data(links)
+        .join('line')
+        .attr('class', 'link')
+        .attr('stroke', '#475569')
+        .attr('stroke-width', 0.8);
+
+    // Draw nodes
+    const nodeGroup = g.append('g').attr('class', 'nodes');
+    const node = nodeGroup.selectAll('g')
+        .data(nodes)
+        .join('g')
+        .attr('class', 'node')
+        .call(d3.drag()
+            .on('start', dragStarted)
+            .on('drag', dragged)
+            .on('end', dragEnded)
+        );
+
+    node.append('circle')
+        .attr('r', d => rScale(d.influence))
+        .attr('fill', d => nodeColor(d))
+        .attr('opacity', 0.85);
+
+    // Labels for high-influence nodes
+    node.filter(d => d.influence >= 5)
+        .append('text')
+        .text(d => {
+            if (d.type === 'rfc') return d.title;
+            const name = d.id.replace(/^draft-/, '');
+            return name.length > 20 ? name.slice(0, 18) + '..' : name;
+        })
+        .attr('dy', d => -(rScale(d.influence) + 4))
+        .attr('text-anchor', 'middle')
+        .attr('fill', '#94a3b8')
+        .attr('font-size', '8px')
+        .attr('font-family', 'Inter, system-ui, sans-serif');
+
+    // Tooltip
+    const tooltip = document.getElementById('tooltip');
+
+    node.on('mouseover', function(event, d) {
+        const typeLabel = d.type === 'rfc' ? 'RFC' : d.type === 'bcp' ? 'BCP' : 'Draft';
+        const catLine = d.category ? `<div class="text-slate-500 text-xs mb-1">${d.category}</div>` : '';
+        tooltip.innerHTML = `
+            <div class="font-semibold text-white mb-1">${d.title}</div>
+            ${catLine}
+            <div class="flex gap-4 text-xs">
+                <span class="text-slate-400">${typeLabel}</span>
+                <span><span class="${d.type === 'rfc' ? 'text-orange-400' : 'text-blue-400'} font-medium">${d.influence}</span> ${d.type === 'draft' ? 'outgoing refs' : 'citing drafts'}</span>
+            </div>
+        `;
+        tooltip.classList.add('visible');
+
+        // Highlight connected nodes
+        const connected = new Set();
+        links.forEach(l => {
+            const sid = typeof l.source === 'object' ? l.source.id : l.source;
+            const tid = typeof l.target === 'object' ? l.target.id : l.target;
+            if (sid === d.id) connected.add(tid);
+            if (tid === d.id) connected.add(sid);
+        });
+        connected.add(d.id);
+
+        node.select('circle')
+            .attr('opacity', n => connected.has(n.id) ? 1 : 0.1);
+        node.selectAll('text')
+            .attr('opacity', n => connected.has(n.id) ? 1 : 0.1);
+        link
+            .attr('stroke-opacity', l => {
+                const sid = typeof l.source === 'object' ? l.source.id : l.source;
+                const tid = typeof l.target === 'object' ? l.target.id : l.target;
+                return (sid === d.id || tid === d.id) ? 0.6 : 0.02;
+            });
+    })
+    .on('mousemove', function(event) {
+        const rect = container.getBoundingClientRect();
+        tooltip.style.left = (event.clientX - rect.left + 15) + 'px';
+        tooltip.style.top = (event.clientY - rect.top - 10) + 'px';
+    })
+    .on('mouseout', function() {
+        tooltip.classList.remove('visible');
+        node.select('circle').attr('opacity', 0.85);
+        node.selectAll('text').attr('opacity', 1);
+        link.attr('stroke-opacity', 0.15);
+    })
+    .on('click', function(event, d) {
+        if (d.type === 'rfc') {
+            window.open(`https://www.rfc-editor.org/rfc/rfc${parseInt(d.ref_id)}`, '_blank');
+        } else if (d.type === 'draft') {
+            window.open(`/drafts/${encodeURIComponent(d.id)}`, '_blank');
+        }
+    });
+
+    // Tick handler
+    simulation.on('tick', () => {
+        link
+            .attr('x1', d => d.source.x)
+            .attr('y1', d => d.source.y)
+            .attr('x2', d => d.target.x)
+            .attr('y2', d => d.target.y);
+        node.attr('transform', d => `translate(${d.x},${d.y})`);
+    });
+
+    // Drag handlers
+    function dragStarted(event, d) {
+        if (!event.active) simulation.alphaTarget(0.3).restart();
+        d.fx = d.x; d.fy = d.y;
+    }
+    function dragged(event, d) {
+        d.fx = event.x; d.fy = event.y;
+    }
+    function dragEnded(event, d) {
+        if (!event.active) simulation.alphaTarget(0);
+        d.fx = null; d.fy = null;
+    }
+
+    // Category filter
+    catSelect.addEventListener('change', function() {
+        const cat = this.value;
+        if (!cat) {
+            node.select('circle').attr('opacity', 0.85);
+            node.selectAll('text').attr('opacity', 1);
+            link.attr('stroke-opacity', 0.15);
+            return;
+        }
+        const inCat = new Set();
+        nodes.forEach(n => {
+            if (n.type === 'draft' && n.category === cat) inCat.add(n.id);
+        });
+        // Also include RFCs referenced by those drafts
+        links.forEach(l => {
+            const sid = typeof l.source === 'object' ? l.source.id : l.source;
+            const tid = typeof l.target === 'object' ? l.target.id : l.target;
+            if (inCat.has(sid)) inCat.add(tid);
+        });
+        node.select('circle')
+            .attr('opacity', n => inCat.has(n.id) ? 1 : 0.05);
+        node.selectAll('text')
+            .attr('opacity', n => inCat.has(n.id) ? 1 : 0.05);
+        link.attr('stroke-opacity', l => {
+            const sid = typeof l.source === 'object' ? l.source.id : l.source;
+            return inCat.has(sid) ? 0.5 : 0.01;
+        });
+    });
+
+    // Min refs slider (client-side filter)
+    const slider = document.getElementById('minRefsSlider');
+    const sliderVal = document.getElementById('minRefsVal');
+    slider.addEventListener('input', function() {
+        sliderVal.textContent = this.value;
+        const minR = parseInt(this.value);
+        // Show/hide RFC nodes by influence
+        node.select('circle')
+            .attr('opacity', n => {
+                if (n.type === 'draft') return 0.85;
+                return n.influence >= minR ? 0.85 : 0.05;
+            });
+        node.selectAll('text')
+            .attr('opacity', n => {
+                if (n.type === 'draft') return 1;
+                return n.influence >= minR ? 1 : 0.05;
+            });
+        // Filter edges
+        const visibleRfcs = new Set(nodes.filter(n => n.type !== 'draft' && n.influence >= minR).map(n => n.id));
+        link.attr('stroke-opacity', l => {
+            const tid = typeof l.target === 'object' ? l.target.id : l.target;
+            return visibleRfcs.has(tid) ? 0.15 : 0.01;
+        });
+    });
+})();
+</script>
+{% endblock %}

src/webui/templates/comparison.html

@@ -0,0 +1,220 @@
+{% extends "base.html" %}
+{% set active_page = "drafts" %}
+
+{% block title %}Compare Drafts — IETF Draft Analyzer{% endblock %}
+
+{% block extra_head %}
+<style>
+    .compare-card {
+        background: linear-gradient(135deg, rgba(30, 41, 59, 0.8), rgba(30, 41, 59, 0.4));
+        backdrop-filter: blur(10px);
+    }
+    .idea-shared { background: rgba(34, 197, 94, 0.1); border-color: rgba(34, 197, 94, 0.2); }
+    .idea-unique { background: rgba(59, 130, 246, 0.1); border-color: rgba(59, 130, 246, 0.2); }
+    .ref-pill {
+        display: inline-block;
+        padding: 1px 8px;
+        border-radius: 9999px;
+        font-size: 0.65rem;
+        font-weight: 500;
+        background: rgba(51, 65, 85, 0.5);
+        color: #94a3b8;
+        border: 1px solid rgba(71, 85, 105, 0.4);
+    }
+    .loading-spinner {
+        border: 3px solid rgba(59, 130, 246, 0.2);
+        border-top-color: #3b82f6;
+        border-radius: 50%;
+        width: 20px;
+        height: 20px;
+        animation: spin 0.8s linear infinite;
+        display: inline-block;
+    }
+    @keyframes spin { to { transform: rotate(360deg); } }
+</style>
+{% endblock %}
+
+{% block content %}
+<!-- Header -->
+<div class="mb-6">
+    <h1 class="text-2xl font-bold text-white">Compare Drafts</h1>
+    <p class="text-slate-400 text-sm mt-1">Side-by-side analysis of selected drafts: shared ideas, references, and AI-generated comparison.</p>
+</div>
+
+{% if not data %}
+<!-- No data yet — show instructions -->
+<div class="compare-card rounded-xl border border-slate-800 p-8 text-center max-w-xl mx-auto">
+    <svg class="w-12 h-12 mx-auto mb-4 text-slate-600" fill="none" stroke="currentColor" viewBox="0 0 24 24">
+        <path stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5" d="M9 5H7a2 2 0 00-2 2v12a2 2 0 002 2h10a2 2 0 002-2V7a2 2 0 00-2-2h-2M9 5a2 2 0 002 2h2a2 2 0 002-2M9 5a2 2 0 012-2h2a2 2 0 012 2"/>
+    </svg>
+    {% if names and names|length < 2 %}
+    <p class="text-slate-400 text-sm mb-4">Need at least 2 valid draft names to compare.</p>
+    {% else %}
+    <p class="text-slate-400 text-sm mb-4">Select drafts to compare from the <a href="/drafts" class="text-blue-400 hover:text-blue-300">Draft Explorer</a>, or enter draft names below.</p>
+    {% endif %}
+    <form method="get" action="/compare" class="mt-4">
+        <input type="text" name="drafts" placeholder="draft-name-1, draft-name-2, ..."
+            value="{{ names | join(', ') if names else '' }}"
+            class="w-full bg-slate-800/60 border border-slate-700 rounded-lg px-4 py-2.5 text-sm text-slate-200 placeholder-slate-500 focus:outline-none focus:border-blue-500 mb-3">
+        <button type="submit" class="px-6 py-2.5 bg-blue-600 text-white rounded-lg text-sm font-medium hover:bg-blue-500 transition-colors">
+            Compare
+        </button>
+    </form>
+</div>
+{% else %}
+
+<!-- Draft cards side by side -->
+<div class="grid grid-cols-1 lg:grid-cols-{{ data.drafts|length }} gap-4 mb-6">
+    {% for draft in data.drafts %}
+    <div class="compare-card rounded-xl border border-slate-800 p-5">
+        <a href="/drafts/{{ draft.name }}" class="text-blue-400 hover:text-blue-300 font-semibold text-sm transition">
+            {{ draft.title }}
+        </a>
+        <div class="text-xs text-slate-600 font-mono mt-1">{{ draft.name }}</div>
+        <div class="text-xs text-slate-500 mt-2 line-clamp-3">{{ draft.abstract[:200] }}</div>
+
+        {% if draft.rating %}
+        <!-- Rating radar -->
+        <div class="mt-3 grid grid-cols-5 gap-1 text-center">
+            {% for dim, label in [('novelty', 'Nov'), ('maturity', 'Mat'), ('relevance', 'Rel'), ('momentum', 'Mom'), ('overlap', 'Ovl')] %}
+            <div>
+                <div class="text-xs text-slate-500">{{ label }}</div>
+                <div class="text-sm font-semibold {% if draft.rating[dim] >= 4 %}text-green-400{% elif draft.rating[dim] >= 3 %}text-yellow-400{% else %}text-red-400{% endif %}">
+                    {{ draft.rating[dim] }}
+                </div>
+            </div>
+            {% endfor %}
+        </div>
+        <div class="text-center mt-2">
+            <span class="score-badge {% if draft.rating.score >= 3.5 %}score-high{% elif draft.rating.score >= 2.5 %}score-mid{% else %}score-low{% endif %}">
+                {{ draft.rating.score }}
+            </span>
+        </div>
+        {% endif %}
+    </div>
+    {% endfor %}
+</div>
+
+<!-- Pairwise similarities -->
+{% if data.similarities %}
+<div class="compare-card rounded-xl border border-slate-800 p-5 mb-6">
+    <h3 class="text-sm font-semibold text-slate-300 mb-3">Pairwise Embedding Similarity</h3>
+    <div class="space-y-2">
+        {% for sim in data.similarities %}
+        <div class="flex items-center gap-3">
+            <span class="text-xs font-mono text-slate-400 w-1/3 truncate">{{ sim.a.split('-')[-1][:20] }}</span>
+            <span class="text-xs text-slate-600">&harr;</span>
+            <span class="text-xs font-mono text-slate-400 w-1/3 truncate">{{ sim.b.split('-')[-1][:20] }}</span>
+            <div class="flex-1 h-2 bg-slate-800 rounded overflow-hidden">
+                <div class="h-full rounded {% if sim.similarity >= 0.85 %}bg-green-500{% elif sim.similarity >= 0.7 %}bg-yellow-500{% else %}bg-blue-500{% endif %}"
+                     style="width: {{ (sim.similarity * 100)|int }}%"></div>
+            </div>
+            <span class="text-xs font-mono font-semibold w-12 text-right {% if sim.similarity >= 0.85 %}text-green-400{% elif sim.similarity >= 0.7 %}text-yellow-400{% else %}text-blue-400{% endif %}">
+                {{ "%.3f"|format(sim.similarity) }}
+            </span>
+        </div>
+        {% endfor %}
+    </div>
+</div>
+{% endif %}
+
+<!-- Shared Ideas -->
+{% if data.shared_ideas %}
+<div class="compare-card rounded-xl border border-slate-800 p-5 mb-6">
+    <h3 class="text-sm font-semibold text-green-400 mb-3">Shared Ideas ({{ data.shared_ideas|length }})</h3>
+    <div class="space-y-2">
+        {% for idea in data.shared_ideas %}
+        <div class="idea-shared rounded-lg border p-3">
+            <div class="text-sm text-slate-200 font-medium">{{ idea.title }}</div>
+            <div class="text-xs text-slate-500 mt-1">Found in: {{ idea.drafts | join(', ') }}</div>
+        </div>
+        {% endfor %}
+    </div>
+</div>
+{% endif %}
+
+<!-- Unique Ideas per draft -->
+<div class="grid grid-cols-1 lg:grid-cols-{{ data.drafts|length }} gap-4 mb-6">
+    {% for draft in data.drafts %}
+    <div class="compare-card rounded-xl border border-slate-800 p-5">
+        <h3 class="text-sm font-semibold text-blue-400 mb-3">
+            Unique Ideas: {{ draft.name.split('-')[-1][:20] }}
+            <span class="text-slate-600 font-normal">({{ data.unique_ideas.get(draft.name, [])|length }})</span>
+        </h3>
+        <div class="space-y-2">
+            {% for idea in data.unique_ideas.get(draft.name, [])[:10] %}
+            <div class="idea-unique rounded-lg border p-2.5">
+                <div class="text-xs text-slate-300 font-medium">{{ idea.title }}</div>
+                {% if idea.description %}
+                <div class="text-xs text-slate-500 mt-0.5 line-clamp-2">{{ idea.description }}</div>
+                {% endif %}
+            </div>
+            {% endfor %}
+            {% if data.unique_ideas.get(draft.name, [])|length == 0 %}
+            <div class="text-xs text-slate-600 italic">No unique ideas extracted</div>
+            {% endif %}
+        </div>
+    </div>
+    {% endfor %}
+</div>
+
+<!-- Shared References -->
+{% if data.shared_refs %}
+<div class="compare-card rounded-xl border border-slate-800 p-5 mb-6">
+    <h3 class="text-sm font-semibold text-slate-300 mb-3">Shared References ({{ data.shared_refs|length }})</h3>
+    <div class="flex flex-wrap gap-1.5">
+        {% for ref in data.shared_refs %}
+        <span class="ref-pill">{{ ref.type|upper }} {{ ref.id }}</span>
+        {% endfor %}
+    </div>
+</div>
+{% endif %}
+
+<!-- Claude Comparison (lazy-loaded) -->
+<div class="compare-card rounded-xl border border-slate-800 p-5 mb-6" id="comparisonSection">
+    <div class="flex items-center justify-between mb-3">
+        <h3 class="text-sm font-semibold text-slate-300">AI Comparison Summary</h3>
+        <button onclick="runComparison()" id="compareBtn"
+            class="px-4 py-1.5 bg-blue-600 text-white rounded-lg text-xs font-medium hover:bg-blue-500 transition-colors">
+            Generate Comparison
+        </button>
+    </div>
+    <div id="comparisonResult" class="text-sm text-slate-400">
+        Click "Generate Comparison" to get a Claude-powered analysis of these drafts.
+    </div>
+</div>
+
+{% endif %}
+{% endblock %}
+
+{% block extra_scripts %}
+{% if data %}
+<script>
+async function runComparison() {
+    const btn = document.getElementById('compareBtn');
+    const result = document.getElementById('comparisonResult');
+    btn.disabled = true;
+    btn.innerHTML = '<span class="loading-spinner"></span> Analyzing...';
+    result.innerHTML = '<div class="flex items-center gap-2"><span class="loading-spinner"></span> <span class="text-slate-500">Generating comparison...</span></div>';
+
+    try {
+        const resp = await fetch('/api/compare', {
+            method: 'POST',
+            headers: {'Content-Type': 'application/json'},
+            body: JSON.stringify({drafts: {{ data.drafts | map(attribute='name') | list | tojson }}})
+        });
+        const data = await resp.json();
+        if (data.error) {
+            result.innerHTML = '<div class="text-red-400">Error: ' + data.error + '</div>';
+        } else {
+            result.innerHTML = '<div class="text-slate-300 whitespace-pre-line leading-relaxed">' + data.text + '</div>';
+        }
+    } catch (e) {
+        result.innerHTML = '<div class="text-red-400">Error: ' + e.message + '</div>';
+    }
+    btn.disabled = false;
+    btn.textContent = 'Regenerate';
+}
+</script>
+{% endif %}
+{% endblock %}

src/webui/templates/draft_detail.html

@@ -156,6 +156,14 @@
                             {{ idea.type }}
                         </span>
                         {% endif %}
+                        {% if idea.novelty_score is not none and idea.novelty_score %}
+                        <span class="flex-shrink-0 px-1.5 py-0.5 rounded text-[10px] font-mono
+                            {% if idea.novelty_score >= 4 %}bg-green-500/20 text-green-400
+                            {% elif idea.novelty_score >= 3 %}bg-amber-500/20 text-amber-400
+                            {% elif idea.novelty_score >= 2 %}bg-orange-500/20 text-orange-400
+                            {% else %}bg-red-500/20 text-red-400{% endif %}"
+                            title="Novelty score">N:{{ idea.novelty_score }}</span>
+                        {% endif %}
                     </div>
                     {% if idea.description %}
                     <p class="text-xs text-slate-500 leading-relaxed mt-1">{{ idea.description }}</p>
@@ -165,6 +173,40 @@
             </div>
         </div>
         {% endif %}
+
+        <!-- Annotation (notes & tags) -->
+        <div class="detail-card rounded-xl border border-slate-800 p-6" id="annotationSection">
+            <h2 class="text-sm font-semibold text-slate-300 mb-3 flex items-center gap-2">
+                <svg class="w-4 h-4 text-slate-500" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M11 5H6a2 2 0 00-2 2v11a2 2 0 002 2h11a2 2 0 002-2v-5m-1.414-9.414a2 2 0 112.828 2.828L11.828 15H9v-2.828l8.586-8.586z"/></svg>
+                Notes & Tags
+            </h2>
+            <div class="mb-3">
+                <textarea id="annotNote" rows="3" placeholder="Add a private note about this draft..."
+                    class="w-full bg-slate-800/60 border border-slate-700 rounded-lg px-3 py-2 text-sm text-slate-200 placeholder-slate-500 focus:outline-none focus:border-blue-500 resize-y">{{ draft.annotation.note if draft.annotation else '' }}</textarea>
+            </div>
+            <div class="mb-3">
+                <div class="flex flex-wrap gap-1.5 mb-2" id="tagContainer">
+                    {% if draft.annotation and draft.annotation.tags %}
+                    {% for tag in draft.annotation.tags %}
+                    <span class="tag-chip inline-flex items-center gap-1 px-2 py-0.5 rounded-full text-xs bg-blue-500/20 text-blue-400 border border-blue-500/30">
+                        {{ tag }}
+                        <button onclick="removeTag('{{ tag }}')" class="hover:text-red-400 transition ml-0.5">&times;</button>
+                    </span>
+                    {% endfor %}
+                    {% endif %}
+                </div>
+                <div class="flex gap-2">
+                    <input type="text" id="newTag" placeholder="Add tag..." maxlength="30"
+                        class="flex-1 bg-slate-800/60 border border-slate-700 rounded-lg px-3 py-1.5 text-xs text-slate-200 placeholder-slate-500 focus:outline-none focus:border-blue-500"
+                        onkeydown="if(event.key==='Enter'){event.preventDefault();addTag();}">
+                    <button onclick="addTag()" class="px-3 py-1.5 bg-blue-600 text-white rounded-lg text-xs font-medium hover:bg-blue-500 transition">Add</button>
+                </div>
+            </div>
+            <button onclick="saveAnnotation()" class="w-full px-3 py-2 bg-slate-800 border border-slate-700 text-slate-300 rounded-lg text-xs font-medium hover:border-blue-500 hover:text-blue-400 transition" id="saveBtn">
+                Save Note
+            </button>
+            <div id="saveStatus" class="text-xs text-center mt-2 text-slate-600"></div>
+        </div>
     </div>
 
     <!-- Right column: Sidebar -->
@@ -193,6 +235,42 @@
         </div>
         {% endif %}
 
+        <!-- Readiness Score -->
+        {% if draft.readiness and draft.readiness.score > 0 %}
+        <div class="detail-card rounded-xl border border-slate-800 p-5">
+            <h2 class="text-sm font-semibold text-slate-300 mb-3 flex items-center gap-2">
+                <svg class="w-4 h-4 text-slate-500" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 12l2 2 4-4m5.618-4.016A11.955 11.955 0 0112 2.944a11.955 11.955 0 01-8.618 3.04A12.02 12.02 0 003 9c0 5.591 3.824 10.29 9 11.622 5.176-1.332 9-6.03 9-11.622 0-1.042-.133-2.052-.382-3.016z"/></svg>
+                Standards Readiness
+            </h2>
+            <!-- Gauge -->
+            <div class="relative w-full h-6 bg-slate-800 rounded-full overflow-hidden mb-2">
+                <div class="h-full rounded-full transition-all duration-700
+                    {% if draft.readiness.score >= 60 %}bg-gradient-to-r from-green-600 to-green-400
+                    {% elif draft.readiness.score >= 35 %}bg-gradient-to-r from-amber-600 to-amber-400
+                    {% else %}bg-gradient-to-r from-red-600 to-red-400{% endif %}"
+                    style="width: {{ draft.readiness.score }}%"></div>
+                <div class="absolute inset-0 flex items-center justify-center text-xs font-bold text-white">
+                    {{ draft.readiness.score }}/100
+                </div>
+            </div>
+            <!-- Factor breakdown -->
+            <div class="space-y-1.5 mt-3">
+                {% for key, f in draft.readiness.factors.items() %}
+                <div class="flex items-center justify-between text-xs">
+                    <span class="text-slate-500">{{ f.label }}</span>
+                    <div class="flex items-center gap-2">
+                        <span class="text-slate-600 font-mono text-[10px]">{{ f.detail }}</span>
+                        <span class="font-mono font-medium
+                            {% if f.value >= 0.7 %}text-green-400
+                            {% elif f.value >= 0.4 %}text-amber-400
+                            {% else %}text-red-400{% endif %}">+{{ f.contribution }}</span>
+                    </div>
+                </div>
+                {% endfor %}
+            </div>
+        </div>
+        {% endif %}
+
         <!-- Metadata -->
         <div class="detail-card rounded-xl border border-slate-800 p-5">
             <h2 class="text-sm font-semibold text-slate-300 mb-3 flex items-center gap-2">
@@ -308,3 +386,76 @@
     </div>
 </div>
 {% endblock %}
+
+{% block extra_scripts %}
+<script>
+const draftName = {{ draft.name | tojson }};
+
+function addTag() {
+    const input = document.getElementById('newTag');
+    const tag = input.value.trim();
+    if (!tag) return;
+    input.value = '';
+    fetch(`/api/drafts/${encodeURIComponent(draftName)}/annotate`, {
+        method: 'POST',
+        headers: {'Content-Type': 'application/json'},
+        body: JSON.stringify({add_tag: tag}),
+    })
+    .then(r => r.json())
+    .then(data => {
+        if (data.success) renderTags(data.annotation.tags);
+    });
+}
+
+function removeTag(tag) {
+    fetch(`/api/drafts/${encodeURIComponent(draftName)}/annotate`, {
+        method: 'POST',
+        headers: {'Content-Type': 'application/json'},
+        body: JSON.stringify({remove_tag: tag}),
+    })
+    .then(r => r.json())
+    .then(data => {
+        if (data.success) renderTags(data.annotation.tags);
+    });
+}
+
+function renderTags(tags) {
+    const container = document.getElementById('tagContainer');
+    container.innerHTML = tags.map(t =>
+        `<span class="tag-chip inline-flex items-center gap-1 px-2 py-0.5 rounded-full text-xs bg-blue-500/20 text-blue-400 border border-blue-500/30">
+            ${t}
+            <button onclick="removeTag('${t}')" class="hover:text-red-400 transition ml-0.5">&times;</button>
+        </span>`
+    ).join('');
+}
+
+function saveAnnotation() {
+    const note = document.getElementById('annotNote').value;
+    const btn = document.getElementById('saveBtn');
+    const status = document.getElementById('saveStatus');
+    btn.disabled = true;
+    btn.textContent = 'Saving...';
+    fetch(`/api/drafts/${encodeURIComponent(draftName)}/annotate`, {
+        method: 'POST',
+        headers: {'Content-Type': 'application/json'},
+        body: JSON.stringify({note: note}),
+    })
+    .then(r => r.json())
+    .then(data => {
+        btn.disabled = false;
+        btn.textContent = 'Save Note';
+        if (data.success) {
+            status.textContent = 'Saved';
+            status.className = 'text-xs text-center mt-2 text-green-400';
+            setTimeout(() => { status.textContent = ''; status.className = 'text-xs text-center mt-2 text-slate-600'; }, 2000);
+        }
+    })
+    .catch(() => {
+        btn.disabled = false;
+        btn.textContent = 'Save Note';
+        status.textContent = 'Error saving';
+        status.className = 'text-xs text-center mt-2 text-red-400';
+    });
+}
+</script>
+{% endblock %}

src/webui/templates/drafts.html

@@ -33,6 +33,26 @@
     .dim-fill-high { background: #4ade80; }
     .dim-fill-mid  { background: #facc15; }
     .dim-fill-low  { background: #f87171; }
+    .source-badge {
+        display: inline-block;
+        padding: 1px 6px;
+        border-radius: 4px;
+        font-size: 0.6rem;
+        font-weight: 600;
+        letter-spacing: 0.03em;
+        white-space: nowrap;
+        vertical-align: middle;
+    }
+    .source-ietf {
+        background: rgba(59, 130, 246, 0.15);
+        color: #60a5fa;
+        border: 1px solid rgba(59, 130, 246, 0.3);
+    }
+    .source-w3c {
+        background: rgba(34, 197, 94, 0.15);
+        color: #4ade80;
+        border: 1px solid rgba(34, 197, 94, 0.3);
+    }
     .cat-pill {
         display: inline-block;
         padding: 1px 8px;
@@ -128,6 +148,17 @@
                     {% endfor %}
                 </select>
             </div>
+            <!-- Source dropdown -->
+            <div class="min-w-[120px]">
+                <label class="block text-xs font-medium text-slate-500 mb-1.5">Source</label>
+                <select name="source"
+                    class="w-full bg-slate-800/60 border border-slate-700 rounded-lg px-3 py-2 text-sm text-slate-200 focus:outline-none focus:border-blue-500 transition appearance-none"
+                    style="background-image: url('data:image/svg+xml,<svg xmlns=%22http://www.w3.org/2000/svg%22 fill=%22none%22 viewBox=%220 0 20 20%22><path stroke=%22%236b7280%22 stroke-linecap=%22round%22 stroke-linejoin=%22round%22 stroke-width=%221.5%22 d=%22M6 8l4 4 4-4%22/></svg>'); background-position: right 0.5rem center; background-repeat: no-repeat; background-size: 1.2em 1.2em; padding-right: 2rem;">
+                    <option value="">All sources</option>
+                    <option value="ietf" {% if current_source == 'ietf' %}selected{% endif %}>IETF</option>
+                    <option value="w3c" {% if current_source == 'w3c' %}selected{% endif %}>W3C</option>
+                </select>
+            </div>
             <!-- Sort -->
             <div class="min-w-[150px]">
                 <label class="block text-xs font-medium text-slate-500 mb-1.5">Sort by</label>
@@ -141,6 +172,7 @@
                     <option value="relevance" {% if sort == 'relevance' %}selected{% endif %}>Relevance</option>
                     <option value="momentum" {% if sort == 'momentum' %}selected{% endif %}>Momentum</option>
                     <option value="overlap" {% if sort == 'overlap' %}selected{% endif %}>Overlap</option>
+                    <option value="readiness" {% if sort == 'readiness' %}selected{% endif %}>Readiness</option>
                     <option value="name" {% if sort == 'name' %}selected{% endif %}>Name</option>
                 </select>
             </div>
@@ -178,10 +210,10 @@
         {% if categories %}
         <div class="mt-4 pt-3 border-t border-slate-800/50">
             <div class="flex flex-wrap gap-1.5">
-                <a href="/drafts?q={{ search | urlencode }}&min_score={{ min_score }}&sort={{ sort }}&dir={{ sort_dir }}"
+                <a href="/drafts?q={{ search | urlencode }}&min_score={{ min_score }}&sort={{ sort }}&dir={{ sort_dir }}&source={{ current_source }}"
                    class="cat-pill {% if not current_cat %}cat-pill-active{% endif %}">All</a>
                 {% for cat, count in categories.items() %}
-                <a href="/drafts?cat={{ cat }}&q={{ search | urlencode }}&min_score={{ min_score }}&sort={{ sort }}&dir={{ sort_dir }}"
+                <a href="/drafts?cat={{ cat }}&q={{ search | urlencode }}&min_score={{ min_score }}&sort={{ sort }}&dir={{ sort_dir }}&source={{ current_source }}"
                    class="cat-pill {% if current_cat == cat %}cat-pill-active{% endif %}">
                     {{ cat }} <span class="opacity-50">{{ count }}</span>
                 </a>
@@ -192,7 +224,7 @@
     </form>
 </div>
 
-<!-- Results count -->
+<!-- Results count + Compare button -->
 <div class="flex items-center justify-between mb-4">
     <p class="text-sm text-slate-500">
         Showing <span class="text-slate-300 font-medium">{{ result.drafts|length }}</span> of
@@ -201,10 +233,18 @@
         {% if current_cat %} in <span class="text-blue-400">{{ current_cat }}</span>{% endif %}
         {% if min_score > 0 %} with score >= <span class="text-blue-400">{{ min_score }}</span>{% endif %}
     </p>
+    <div class="flex items-center gap-3">
+        <span id="compareCount" class="text-xs text-slate-600 hidden"><span id="compareNum">0</span> selected</span>
+        <button onclick="goCompare()" id="compareBtn"
+            class="px-4 py-1.5 bg-slate-800 text-slate-500 rounded-lg text-xs font-medium border border-slate-700 cursor-not-allowed transition-colors hidden"
+            disabled>
+            Compare Selected
+        </button>
         {% if result.pages > 1 %}
         <p class="text-xs text-slate-600">Page {{ result.page }} of {{ result.pages }}</p>
         {% endif %}
     </div>
+</div>
 
 <!-- Draft Table -->
 <div class="bg-slate-900/60 rounded-xl border border-slate-800 overflow-hidden">
@@ -216,7 +256,7 @@
                     {% set is_active = sort == field %}
                     {% set next_dir = 'asc' if (is_active and sort_dir == 'desc') else 'desc' %}
                     <th class="px-4 py-3 text-left text-xs font-medium uppercase tracking-wide {{ extra_class }} {{ 'text-blue-400' if is_active else 'text-slate-500' }}">
-                        <a href="/drafts?q={{ search }}&cat={{ current_cat }}&min_score={{ min_score }}&sort={{ field }}&dir={{ next_dir }}"
+                        <a href="/drafts?q={{ search }}&cat={{ current_cat }}&min_score={{ min_score }}&sort={{ field }}&dir={{ next_dir }}&source={{ current_source }}"
                            class="hover:text-blue-400 transition inline-flex items-center gap-1"
                            {% if title %}title="{{ title }}"{% endif %}>
                             {{ label }}
@@ -228,6 +268,9 @@
                         </a>
                     </th>
                     {% endmacro %}
+                    <th class="px-2 py-3 w-8">
+                        <span class="text-xs text-slate-600" title="Select drafts to compare">Cmp</span>
+                    </th>
                     {{ sort_header("score", "Score", "w-20") }}
                     {{ sort_header("name", "Draft") }}
                     {{ sort_header("date", "Date", "w-24 hidden md:table-cell") }}
@@ -236,23 +279,32 @@
                     {{ sort_header("relevance", "Rel", "w-20 hidden lg:table-cell", "Relevance") }}
                     {{ sort_header("momentum", "Mom", "w-20 hidden xl:table-cell", "Momentum") }}
                     {{ sort_header("overlap", "Ovl", "w-20 hidden xl:table-cell", "Overlap") }}
+                    {{ sort_header("readiness", "Rdy", "w-20 hidden xl:table-cell", "Standards Readiness") }}
                     <th class="px-4 py-3 text-left text-xs font-medium text-slate-500 uppercase tracking-wide hidden md:table-cell">Categories</th>
                 </tr>
             </thead>
             <tbody class="divide-y divide-slate-800/30">
                 {% for d in result.drafts %}
                 <tr class="draft-row">
+                    <!-- Compare checkbox -->
+                    <td class="px-2 py-3 text-center">
+                        <input type="checkbox" class="compare-check rounded border-slate-600 bg-slate-800 text-blue-500 focus:ring-blue-500/30 focus:ring-offset-0 w-3.5 h-3.5 cursor-pointer"
+                            data-name="{{ d.name }}" onchange="updateCompare()">
+                    </td>
                     <!-- Score badge -->
                     <td class="px-4 py-3">
                         <span class="score-badge {% if d.score >= 3.5 %}score-high{% elif d.score >= 2.5 %}score-mid{% else %}score-low{% endif %}">
                             {{ d.score }}
                         </span>
                     </td>
-                    <!-- Draft name + title -->
+                    <!-- Draft name + title + source badge -->
                     <td class="px-4 py-3">
+                        <div class="flex items-center gap-1.5">
                             <a href="/drafts/{{ d.name }}" class="text-blue-400 hover:text-blue-300 font-medium text-sm transition">
                                 {{ d.title }}
                             </a>
+                            <span class="source-badge source-{{ d.source|default('ietf') }}">{{ (d.source|default('ietf'))|upper }}</span>
+                        </div>
                         <div class="text-xs text-slate-600 mt-0.5 font-mono">{{ d.name }}</div>
                         {% if d.summary %}
                         <div class="text-xs text-slate-500 mt-1 line-clamp-1 max-w-lg">{{ d.summary }}</div>
@@ -293,6 +345,16 @@
                             <span class="text-xs text-slate-500 font-mono w-4 text-right">{{ d.overlap }}</span>
                         </div>
                     </td>
+                    <!-- Readiness -->
+                    <td class="px-4 py-3 hidden xl:table-cell">
+                        <div class="flex items-center gap-1.5">
+                            <span class="dim-bar-bg" style="width: 50px;">
+                                <span class="dim-bar-fill {% if d.readiness >= 50 %}dim-fill-high{% elif d.readiness >= 25 %}dim-fill-mid{% else %}dim-fill-low{% endif %}"
+                                      style="width: {{ (d.readiness)|int }}%"></span>
+                            </span>
+                            <span class="text-xs text-slate-500 font-mono w-6 text-right">{{ d.readiness|int }}</span>
+                        </div>
+                    </td>
                     <!-- Categories -->
                     <td class="px-4 py-3 hidden md:table-cell">
                         <div class="flex flex-wrap gap-1">
@@ -308,7 +370,7 @@
                 {% endfor %}
                 {% if not result.drafts %}
                 <tr>
-                    <td colspan="9" class="px-4 py-12 text-center text-slate-500">
+                    <td colspan="11" class="px-4 py-12 text-center text-slate-500">
                         <svg class="w-12 h-12 mx-auto mb-3 opacity-30" fill="none" stroke="currentColor" viewBox="0 0 24 24">
                             <path stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5" d="M21 21l-6-6m2-5a7 7 0 11-14 0 7 7 0 0114 0z"/>
                         </svg>
@@ -326,7 +388,7 @@
 {% if result.pages > 1 %}
 <nav class="flex items-center justify-center gap-1.5 mt-6">
     {% if result.page > 1 %}
-    <a href="/drafts?page={{ result.page - 1 }}&q={{ search | urlencode }}&cat={{ current_cat | urlencode }}&min_score={{ min_score }}&sort={{ sort }}&dir={{ sort_dir }}"
+    <a href="/drafts?page={{ result.page - 1 }}&q={{ search | urlencode }}&cat={{ current_cat | urlencode }}&min_score={{ min_score }}&sort={{ sort }}&dir={{ sort_dir }}&source={{ current_source }}"
        class="page-btn page-btn-inactive">
         <svg class="w-4 h-4 inline" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 19l-7-7 7-7"/></svg>
         Prev
@@ -337,7 +399,7 @@
     {% set end_page = [result.pages, result.page + 2]|min %}
 
     {% if start_page > 1 %}
-    <a href="/drafts?page=1&q={{ search | urlencode }}&cat={{ current_cat | urlencode }}&min_score={{ min_score }}&sort={{ sort }}&dir={{ sort_dir }}"
+    <a href="/drafts?page=1&q={{ search | urlencode }}&cat={{ current_cat | urlencode }}&min_score={{ min_score }}&sort={{ sort }}&dir={{ sort_dir }}&source={{ current_source }}"
        class="page-btn page-btn-inactive">1</a>
     {% if start_page > 2 %}<span class="text-slate-600 px-1">...</span>{% endif %}
     {% endif %}
@@ -346,19 +408,19 @@
     {% if p == result.page %}
     <span class="page-btn page-btn-active">{{ p }}</span>
     {% else %}
-    <a href="/drafts?page={{ p }}&q={{ search | urlencode }}&cat={{ current_cat | urlencode }}&min_score={{ min_score }}&sort={{ sort }}&dir={{ sort_dir }}"
+    <a href="/drafts?page={{ p }}&q={{ search | urlencode }}&cat={{ current_cat | urlencode }}&min_score={{ min_score }}&sort={{ sort }}&dir={{ sort_dir }}&source={{ current_source }}"
        class="page-btn page-btn-inactive">{{ p }}</a>
     {% endif %}
     {% endfor %}
 
     {% if end_page < result.pages %}
     {% if end_page < result.pages - 1 %}<span class="text-slate-600 px-1">...</span>{% endif %}
-    <a href="/drafts?page={{ result.pages }}&q={{ search | urlencode }}&cat={{ current_cat | urlencode }}&min_score={{ min_score }}&sort={{ sort }}&dir={{ sort_dir }}"
+    <a href="/drafts?page={{ result.pages }}&q={{ search | urlencode }}&cat={{ current_cat | urlencode }}&min_score={{ min_score }}&sort={{ sort }}&dir={{ sort_dir }}&source={{ current_source }}"
        class="page-btn page-btn-inactive">{{ result.pages }}</a>
     {% endif %}
 
     {% if result.page < result.pages %}
-    <a href="/drafts?page={{ result.page + 1 }}&q={{ search | urlencode }}&cat={{ current_cat | urlencode }}&min_score={{ min_score }}&sort={{ sort }}&dir={{ sort_dir }}"
+    <a href="/drafts?page={{ result.page + 1 }}&q={{ search | urlencode }}&cat={{ current_cat | urlencode }}&min_score={{ min_score }}&sort={{ sort }}&dir={{ sort_dir }}&source={{ current_source }}"
        class="page-btn page-btn-inactive">
         Next
         <svg class="w-4 h-4 inline" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 5l7 7-7 7"/></svg>
@@ -366,4 +428,46 @@
     {% endif %}
 </nav>
 {% endif %}
+
+{% endblock %}
+
+{% block extra_scripts %}
+<script>
+function updateCompare() {
+    const checks = document.querySelectorAll('.compare-check:checked');
+    const btn = document.getElementById('compareBtn');
+    const count = document.getElementById('compareCount');
+    const num = document.getElementById('compareNum');
+    const n = checks.length;
+    num.textContent = n;
+
+    if (n >= 2) {
+        btn.classList.remove('hidden', 'bg-slate-800', 'text-slate-500', 'cursor-not-allowed');
+        btn.classList.add('bg-blue-600', 'text-white', 'hover:bg-blue-500', 'cursor-pointer');
+        btn.disabled = false;
+        count.classList.remove('hidden');
+    } else {
+        btn.classList.add('hidden');
+        count.classList.add('hidden');
+        btn.disabled = true;
+    }
+    // Show button area once at least 1 is selected
+    if (n >= 1) {
+        btn.classList.remove('hidden');
+        count.classList.remove('hidden');
+        if (n < 2) {
+            btn.classList.add('bg-slate-800', 'text-slate-500', 'cursor-not-allowed');
+            btn.classList.remove('bg-blue-600', 'text-white', 'hover:bg-blue-500', 'cursor-pointer');
+        }
+    }
+}
+
+function goCompare() {
+    const checks = document.querySelectorAll('.compare-check:checked');
+    const names = Array.from(checks).map(c => c.dataset.name);
+    if (names.length >= 2) {
+        window.location.href = '/compare?drafts=' + encodeURIComponent(names.join(','));
+    }
+}
+</script>
 {% endblock %}

src/webui/templates/gaps.html

@@ -55,9 +55,9 @@
     </div>
 </div>
 
-<!-- Gap cards sorted by severity -->
+<!-- Gap cards sorted by severity (critical first) -->
-<div class="space-y-4">
+<div class="space-y-4" id="gapList">
-    {% for gap in gaps | sort(attribute='severity') %}
+    {% for gap in gaps %}
     <a href="/gaps/{{ gap.id }}" class="block bg-slate-900 rounded-xl border
         {% if gap.severity == 'critical' %}border-red-500/40 hover:border-red-500/60
         {% elif gap.severity == 'high' %}border-orange-500/30 hover:border-orange-500/50

src/webui/templates/idea_clusters.html

@@ -6,7 +6,7 @@
 {% block content %}
 <div class="mb-6">
     <h1 class="text-2xl font-bold text-white">Idea Clusters</h1>
-    <p class="text-slate-400 text-sm mt-1">Extracted ideas grouped by semantic similarity using embedding-based clustering</p>
+    <p class="text-slate-400 text-sm mt-1">Extracted ideas grouped by semantic similarity — enriched with WG and category data</p>
 </div>
 
 <div id="emptyState" class="hidden">
@@ -21,19 +21,30 @@
 
 <div id="clusterContent" class="hidden">
     <!-- Stat cards -->
-    <div class="grid grid-cols-1 md:grid-cols-3 gap-4 mb-6">
+    <div class="grid grid-cols-1 md:grid-cols-4 gap-4 mb-6">
         <div class="stat-card rounded-xl border border-slate-800 p-5">
-            <p class="text-xs text-slate-500 uppercase tracking-wide">Total Ideas Embedded</p>
+            <p class="text-xs text-slate-500 uppercase tracking-wide">Total Ideas</p>
             <p class="text-2xl font-bold text-white mt-1" id="statTotal">0</p>
         </div>
         <div class="stat-card rounded-xl border border-slate-800 p-5">
-            <p class="text-xs text-slate-500 uppercase tracking-wide">Clusters Found</p>
+            <p class="text-xs text-slate-500 uppercase tracking-wide">Clusters</p>
             <p class="text-2xl font-bold text-white mt-1" id="statClusters">0</p>
         </div>
         <div class="stat-card rounded-xl border border-slate-800 p-5">
-            <p class="text-xs text-slate-500 uppercase tracking-wide">Avg Cluster Size</p>
+            <p class="text-xs text-slate-500 uppercase tracking-wide">Avg Size</p>
             <p class="text-2xl font-bold text-white mt-1" id="statAvgSize">0</p>
         </div>
+        <div class="stat-card rounded-xl border border-slate-800 p-5">
+            <p class="text-xs text-slate-500 uppercase tracking-wide">Cross-WG Clusters</p>
+            <p class="text-2xl font-bold text-amber-400 mt-1" id="statCrossWg">0</p>
+        </div>
+    </div>
+
+    <!-- Filter bar -->
+    <div class="flex flex-wrap gap-3 mb-6">
+        <button id="filterAll" onclick="filterClusters('all')" class="px-3 py-1.5 text-xs rounded-lg bg-blue-600 text-white">All</button>
+        <button id="filterCrossWg" onclick="filterClusters('cross_wg')" class="px-3 py-1.5 text-xs rounded-lg bg-slate-800 text-slate-400 hover:text-white">Cross-WG only</button>
+        <button id="filterLarge" onclick="filterClusters('large')" class="px-3 py-1.5 text-xs rounded-lg bg-slate-800 text-slate-400 hover:text-white">Large (10+)</button>
     </div>
 
     <!-- t-SNE Scatter -->
@@ -46,7 +57,7 @@
     <!-- Treemap -->
     <div class="bg-slate-900 rounded-xl border border-slate-800 p-5 mb-6">
         <h2 class="text-sm font-semibold text-slate-300 mb-1">Cluster Sizes</h2>
-        <p class="text-xs text-slate-500 mb-3">Treemap showing relative sizes of each idea cluster.</p>
+        <p class="text-xs text-slate-500 mb-3">Treemap showing relative sizes of each idea cluster. Amber borders = cross-WG clusters.</p>
         <div id="treemapPlot" style="height: 450px;"></div>
     </div>
 
@@ -72,6 +83,9 @@ const PALETTE = [
     '#3b82f6', '#ef4444', '#22c55e', '#a855f7', '#f59e0b',
     '#06b6d4', '#ec4899', '#84cc16', '#f97316', '#8b5cf6',
     '#14b8a6', '#e11d48', '#64748b', '#eab308', '#6366f1',
+    '#fb923c', '#2dd4bf', '#c084fc', '#facc15', '#4ade80',
+    '#f472b6', '#38bdf8', '#a3e635', '#fb7185', '#818cf8',
+    '#34d399', '#fbbf24', '#e879f9', '#22d3ee', '#a78bfa',
 ];
 
 const data = {{ clusters | tojson }};
@@ -81,46 +95,42 @@ if (data.empty) {
 } else {
     document.getElementById('clusterContent').classList.remove('hidden');
 
-    // Stats
     const stats = data.stats;
+    const crossWgCount = data.clusters.filter(c => c.cross_wg).length;
     document.getElementById('statTotal').textContent = stats.total.toLocaleString();
     document.getElementById('statClusters').textContent = stats.num_clusters.toLocaleString();
     document.getElementById('statAvgSize').textContent = stats.num_clusters > 0
         ? (stats.clustered / stats.num_clusters).toFixed(1) : '0';
+    document.getElementById('statCrossWg').textContent = crossWgCount;
 
     // --- t-SNE Scatter ---
     if (data.scatter.length > 0) {
-        // Group by cluster_id
         const groups = {};
         data.scatter.forEach(pt => {
-            if (!groups[pt.cluster_id]) groups[pt.cluster_id] = { x: [], y: [], text: [], names: [] };
+            if (!groups[pt.cluster_id]) groups[pt.cluster_id] = { x: [], y: [], text: [], names: [], wgs: [] };
             groups[pt.cluster_id].x.push(pt.x);
             groups[pt.cluster_id].y.push(pt.y);
             groups[pt.cluster_id].text.push(pt.title);
             groups[pt.cluster_id].names.push(pt.draft_name);
-        });
+            groups[pt.cluster_id].wgs.push(pt.wg || 'none');
-
-        // Map cluster_id to cluster theme
-        const clusterThemes = {};
-        data.clusters.forEach((c, i) => {
-            // Find the original cluster_id by matching scatter points
         });
 
         const clusterIds = Object.keys(groups).sort((a, b) => (groups[b].x.length - groups[a].x.length));
         const traces = clusterIds.map((cid, i) => {
             const g = groups[cid];
             const theme = data.clusters[i] ? data.clusters[i].theme : `Cluster ${cid}`;
+            const hoverTexts = g.text.map((t, j) => `${t}<br><span style="color:#64748b">${g.wgs[j]}</span>`);
             return {
-                x: g.x, y: g.y, text: g.text, name: theme,
+                x: g.x, y: g.y, text: hoverTexts, name: theme,
                 customdata: g.names,
                 mode: 'markers', type: 'scatter',
                 marker: {
-                    size: 6,
+                    size: 7,
                     color: PALETTE[i % PALETTE.length],
                     opacity: 0.8,
                     line: { width: 0.5, color: 'rgba(255,255,255,0.15)' },
                 },
-                hovertemplate: '<b>%{text}</b><extra>%{customdata}</extra>',
+                hovertemplate: '%{text}<extra>%{customdata}</extra>',
             };
         });
 
@@ -135,26 +145,33 @@ if (data.empty) {
 
         document.getElementById('scatterPlot').on('plotly_click', function(ev) {
             const pt = ev.points[0];
-            if (pt.customdata) {
+            if (pt.customdata) window.location.href = '/drafts/' + pt.customdata;
-                window.location.href = '/drafts/' + pt.customdata;
-            }
         });
     }
 
     // --- Treemap ---
     if (data.clusters.length > 0) {
-        const labels = data.clusters.map(c => c.theme);
+        const labels = data.clusters.map(c => c.cross_wg ? `${c.theme} ⚡` : c.theme);
         const values = data.clusters.map(c => c.size);
-        const colors = data.clusters.map((_, i) => PALETTE[i % PALETTE.length]);
+        const colors = data.clusters.map((c, i) => c.cross_wg
+            ? PALETTE[i % PALETTE.length] : PALETTE[i % PALETTE.length]);
+        const hoverTexts = data.clusters.map(c => {
+            const wgs = (c.wgs || []).filter(w => w.wg !== 'none').map(w => `${w.wg}(${w.count})`).join(', ');
+            const cats = (c.categories || []).map(cat => cat.cat).join(', ');
+            return `<b>${c.theme}</b><br>${c.size} ideas, ${c.drafts.length} drafts` +
+                (wgs ? `<br>WGs: ${wgs}` : '') +
+                (cats ? `<br>Categories: ${cats}` : '');
+        });
 
         Plotly.newPlot('treemapPlot', [{
             type: 'treemap',
             labels: labels,
             parents: labels.map(() => ''),
             values: values,
+            text: hoverTexts,
             textinfo: 'label+value',
             marker: { colors: colors },
-            hovertemplate: '<b>%{label}</b><br>%{value} ideas<extra></extra>',
+            hovertemplate: '%{text}<extra></extra>',
         }], {
             ...PLOTLY_LAYOUT,
             margin: { t: 10, r: 10, b: 10, l: 10 },
@@ -163,31 +180,60 @@ if (data.empty) {
 
     // --- Cluster Cards ---
     const grid = document.getElementById('clusterGrid');
+
+    function renderCards(filter) {
+        grid.innerHTML = '';
         data.clusters.forEach((cluster, i) => {
+            if (filter === 'cross_wg' && !cluster.cross_wg) return;
+            if (filter === 'large' && cluster.size < 10) return;
+
             const color = PALETTE[i % PALETTE.length];
-        const topIdeas = cluster.ideas.slice(0, 3);
+            const topIdeas = cluster.ideas.slice(0, 5);
             const ideaListHtml = topIdeas.map(idea =>
-            `<li class="text-xs text-slate-400 truncate" title="${idea.title}">${idea.title}</li>`
+                `<li class="text-xs text-slate-400 truncate" title="${idea.description || idea.title}">
+                    <span class="text-slate-300">${idea.title}</span>
+                </li>`
             ).join('');
             const extraCount = cluster.size - topIdeas.length;
             const extraHtml = extraCount > 0
                 ? `<li class="text-xs text-slate-600">+${extraCount} more</li>` : '';
 
+            // WG badges
+            const wgBadges = (cluster.wgs || []).filter(w => w.wg !== 'none').map(w =>
+                `<span class="inline-block bg-amber-900/30 text-amber-400 text-xs px-2 py-0.5 rounded border border-amber-800/30">${w.wg} (${w.count})</span>`
+            ).join(' ');
+            const noneCount = (cluster.wgs || []).find(w => w.wg === 'none');
+            const noneHtml = noneCount
+                ? `<span class="text-xs text-slate-600">${noneCount.count} individual</span>` : '';
+
+            // Category badges
+            const catBadges = (cluster.categories || []).map(c =>
+                `<span class="inline-block bg-slate-800 text-slate-400 text-xs px-2 py-0.5 rounded">${c.cat}</span>`
+            ).join(' ');
+
+            // Draft badges
             const draftBadges = cluster.drafts.slice(0, 4).map(d =>
-            `<a href="/drafts/${d}" class="inline-block bg-slate-800 text-slate-400 text-xs px-2 py-0.5 rounded hover:text-blue-400 truncate max-w-[140px]" title="${d}">${d.replace('draft-', '').substring(0, 20)}</a>`
+                `<a href="/drafts/${d}" class="inline-block bg-slate-800 text-slate-400 text-xs px-2 py-0.5 rounded hover:text-blue-400 truncate max-w-[160px]" title="${d}">${d.replace('draft-', '').substring(0, 22)}</a>`
             ).join(' ');
             const extraDrafts = cluster.drafts.length > 4
                 ? `<span class="text-xs text-slate-600">+${cluster.drafts.length - 4}</span>` : '';
 
+            const crossBadge = cluster.cross_wg
+                ? `<span class="text-xs bg-amber-900/30 text-amber-400 px-1.5 py-0.5 rounded">cross-WG</span>` : '';
+
             const card = document.createElement('div');
-        card.className = 'bg-slate-900 rounded-xl border border-slate-800 p-5';
+            card.className = 'bg-slate-900 rounded-xl border p-5 ' +
+                (cluster.cross_wg ? 'border-amber-800/40' : 'border-slate-800');
             card.innerHTML = `
                 <div class="flex items-center gap-2 mb-3">
-                <div class="w-3 h-3 rounded-full" style="background: ${color}"></div>
+                    <div class="w-3 h-3 rounded-full flex-shrink-0" style="background: ${color}"></div>
-                <h3 class="text-sm font-semibold text-white">${cluster.theme}</h3>
+                    <h3 class="text-sm font-semibold text-white truncate">${cluster.theme}</h3>
-                <span class="ml-auto text-xs text-slate-500">${cluster.size} ideas</span>
+                    ${crossBadge}
+                    <span class="ml-auto text-xs text-slate-500 flex-shrink-0">${cluster.size} ideas</span>
                 </div>
                 <ul class="space-y-1 mb-3">${ideaListHtml}${extraHtml}</ul>
+                ${(wgBadges || noneHtml) ? `<div class="mb-2"><p class="text-xs text-slate-500 mb-1">Working Groups</p><div class="flex flex-wrap gap-1">${wgBadges} ${noneHtml}</div></div>` : ''}
+                ${catBadges ? `<div class="mb-2"><p class="text-xs text-slate-500 mb-1">Categories</p><div class="flex flex-wrap gap-1">${catBadges}</div></div>` : ''}
                 <div class="border-t border-slate-800 pt-3">
                     <p class="text-xs text-slate-500 mb-1">${cluster.drafts.length} source draft${cluster.drafts.length !== 1 ? 's' : ''}</p>
                     <div class="flex flex-wrap gap-1">${draftBadges}${extraDrafts}</div>
@@ -196,5 +242,28 @@ if (data.empty) {
             grid.appendChild(card);
         });
     }
+
+    renderCards('all');
+
+    // Filter buttons
+    window.filterClusters = function(filter) {
+        document.querySelectorAll('[id^="filter"]').forEach(b => {
+            b.className = b.id === 'filter' + filter.charAt(0).toUpperCase() + filter.slice(1).replace('_w', 'W').replace('_', '')
+                ? 'px-3 py-1.5 text-xs rounded-lg bg-blue-600 text-white'
+                : 'px-3 py-1.5 text-xs rounded-lg bg-slate-800 text-slate-400 hover:text-white';
+        });
+        // Simpler: just match by id
+        ['filterAll', 'filterCrossWg', 'filterLarge'].forEach(id => {
+            const btn = document.getElementById(id);
+            const isActive = (filter === 'all' && id === 'filterAll') ||
+                             (filter === 'cross_wg' && id === 'filterCrossWg') ||
+                             (filter === 'large' && id === 'filterLarge');
+            btn.className = isActive
+                ? 'px-3 py-1.5 text-xs rounded-lg bg-blue-600 text-white'
+                : 'px-3 py-1.5 text-xs rounded-lg bg-slate-800 text-slate-400 hover:text-white';
+        });
+        renderCards(filter);
+    };
+}
 </script>
 {% endblock %}

src/webui/templates/ideas.html

@@ -64,6 +64,14 @@
                 {% if idea.type %}
                 <span class="px-2 py-0.5 rounded text-[10px] font-medium bg-blue-500/20 text-blue-400 whitespace-nowrap">{{ idea.type }}</span>
                 {% endif %}
+                {% if idea.novelty_score is not none and idea.novelty_score %}
+                <span class="px-1.5 py-0.5 rounded text-[10px] font-mono
+                    {% if idea.novelty_score >= 4 %}bg-green-500/20 text-green-400
+                    {% elif idea.novelty_score >= 3 %}bg-amber-500/20 text-amber-400
+                    {% elif idea.novelty_score >= 2 %}bg-orange-500/20 text-orange-400
+                    {% else %}bg-red-500/20 text-red-400{% endif %}"
+                    title="Novelty score (1-5)">N:{{ idea.novelty_score }}</span>
+                {% endif %}
             </div>
             <p class="text-xs text-slate-500 leading-relaxed">{{ idea.description }}</p>
             <a href="/drafts/{{ idea.draft_name }}" class="text-[10px] text-slate-600 hover:text-blue-400 transition mt-1 inline-block font-mono">{{ idea.draft_name }}</a>

src/webui/templates/monitor.html

@@ -111,6 +111,71 @@ html += `
 </div>
 `;
 
+// Pipeline progress
+const pl = data.pipeline || {};
+const cost = data.cost || {};
+if (pl.total_drafts) {
+    const pctRated = Math.round((pl.rated / pl.total_drafts) * 100);
+    const pctEmbedded = Math.round((pl.embedded / pl.total_drafts) * 100);
+    const pctIdeas = Math.round((pl.with_ideas / pl.total_drafts) * 100);
+
+    function progressBar(pct, color) {
+        return `<div class="w-full bg-slate-800 rounded-full h-2.5 mt-1.5">
+            <div class="h-2.5 rounded-full ${color}" style="width: ${pct}%"></div>
+        </div>`;
+    }
+
+    html += `
+    <h2 class="text-lg font-semibold text-white mb-3">Pipeline Progress</h2>
+    <div class="grid grid-cols-1 md:grid-cols-3 gap-4 mb-6">
+        <div class="stat-card rounded-xl border border-slate-800 p-4">
+            <div class="flex justify-between items-baseline">
+                <span class="text-xs text-slate-400">Rated</span>
+                <span class="text-sm font-bold text-blue-400">${pl.rated} / ${pl.total_drafts}</span>
+            </div>
+            ${progressBar(pctRated, 'bg-blue-500')}
+            <div class="text-xs text-slate-600 mt-1 text-right">${pctRated}%</div>
+        </div>
+        <div class="stat-card rounded-xl border border-slate-800 p-4">
+            <div class="flex justify-between items-baseline">
+                <span class="text-xs text-slate-400">Embedded</span>
+                <span class="text-sm font-bold text-purple-400">${pl.embedded} / ${pl.total_drafts}</span>
+            </div>
+            ${progressBar(pctEmbedded, 'bg-purple-500')}
+            <div class="text-xs text-slate-600 mt-1 text-right">${pctEmbedded}%</div>
+        </div>
+        <div class="stat-card rounded-xl border border-slate-800 p-4">
+            <div class="flex justify-between items-baseline">
+                <span class="text-xs text-slate-400">Ideas Extracted</span>
+                <span class="text-sm font-bold text-green-400">${pl.with_ideas} / ${pl.total_drafts}</span>
+            </div>
+            ${progressBar(pctIdeas, 'bg-green-500')}
+            <div class="text-xs text-slate-600 mt-1 text-right">${pctIdeas}%</div>
+        </div>
+    </div>
+
+    <div class="grid grid-cols-2 md:grid-cols-4 gap-4 mb-8">
+        <div class="stat-card rounded-xl border border-slate-800 p-4">
+            <div class="text-2xl font-bold text-slate-200">${pl.total_drafts}</div>
+            <div class="text-xs text-slate-400 mt-1">Total Documents</div>
+        </div>
+        <div class="stat-card rounded-xl border border-slate-800 p-4">
+            <div class="text-2xl font-bold text-slate-200">${pl.idea_total}</div>
+            <div class="text-xs text-slate-400 mt-1">Total Ideas</div>
+        </div>
+        <div class="stat-card rounded-xl border border-slate-800 p-4">
+            <div class="text-2xl font-bold text-slate-200">${pl.gap_count}</div>
+            <div class="text-xs text-slate-400 mt-1">Gaps Identified</div>
+        </div>
+        <div class="stat-card rounded-xl border border-slate-800 p-4">
+            <div class="text-xl font-bold text-amber-400">$${cost.estimated_usd || '0.00'}</div>
+            <div class="text-xs text-slate-400 mt-1">Est. API Cost</div>
+            <div class="text-xs text-slate-600 mt-0.5">${(cost.input_tokens || 0).toLocaleString()} in / ${(cost.output_tokens || 0).toLocaleString()} out</div>
+        </div>
+    </div>
+    `;
+}
+
 // New drafts over time chart
 const runs = data.runs.slice().reverse(); // chronological order
 if (runs.length > 1) {

src/webui/templates/search_results.html

@@ -0,0 +1,149 @@
+{% extends "base.html" %}
+{% set active_page = "search" %}
+
+{% block title %}Search: {{ query }} — IETF Draft Analyzer{% endblock %}
+
+{% block content %}
+<!-- Header -->
+<div class="mb-6">
+    <h1 class="text-2xl font-bold text-white">Search Results</h1>
+    {% if query %}
+    <p class="text-slate-400 text-sm mt-1">
+        Found <span class="text-slate-300 font-medium">{{ total }}</span> results for
+        "<span class="text-blue-400">{{ query }}</span>"
+    </p>
+    {% else %}
+    <p class="text-slate-400 text-sm mt-1">Enter a search query to find drafts, ideas, authors, and gaps.</p>
+    {% endif %}
+</div>
+
+<!-- Search form -->
+<div class="mb-8">
+    <form action="/search" method="get" class="flex gap-3 max-w-xl">
+        <input type="text" name="q" value="{{ query }}" placeholder="Search drafts, ideas, authors, gaps..."
+            autofocus
+            class="flex-1 bg-slate-800/60 border border-slate-700 rounded-lg px-4 py-2.5 text-sm text-slate-200 placeholder-slate-500 focus:outline-none focus:border-blue-500 focus:ring-1 focus:ring-blue-500/30 transition">
+        <button type="submit" class="px-5 py-2.5 bg-blue-600 text-white rounded-lg text-sm font-medium hover:bg-blue-500 transition-colors">
+            Search
+        </button>
+    </form>
+</div>
+
+{% if query %}
+
+<!-- Drafts -->
+{% if results.drafts %}
+<div class="mb-8">
+    <h2 class="text-lg font-semibold text-white mb-3 flex items-center gap-2">
+        <svg class="w-5 h-5 text-blue-400" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9 12h6m-6 4h6m2 5H7a2 2 0 01-2-2V5a2 2 0 012-2h5.586a1 1 0 01.707.293l5.414 5.414a1 1 0 01.293.707V19a2 2 0 01-2 2z"/></svg>
+        Drafts <span class="text-sm font-normal text-slate-500">({{ results.drafts|length }})</span>
+    </h2>
+    <div class="bg-slate-900/60 rounded-xl border border-slate-800 divide-y divide-slate-800/30">
+        {% for d in results.drafts %}
+        <div class="px-5 py-3 hover:bg-slate-800/30 transition">
+            <a href="/drafts/{{ d.name }}" class="text-blue-400 hover:text-blue-300 font-medium text-sm transition">
+                {{ d.title }}
+            </a>
+            <div class="text-xs text-slate-600 mt-0.5 font-mono">{{ d.name }}</div>
+            {% if d.abstract %}
+            <p class="text-xs text-slate-500 mt-1 line-clamp-2">{{ d.abstract }}</p>
+            {% endif %}
+            <div class="flex gap-3 mt-1 text-xs text-slate-600">
+                {% if d.date %}<span>{{ d.date[:10] }}</span>{% endif %}
+                <span>{{ d.group }}</span>
+            </div>
+        </div>
+        {% endfor %}
+    </div>
+</div>
+{% endif %}
+
+<!-- Ideas -->
+{% if results.ideas %}
+<div class="mb-8">
+    <h2 class="text-lg font-semibold text-white mb-3 flex items-center gap-2">
+        <svg class="w-5 h-5 text-yellow-400" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M9.663 17h4.673M12 3v1m6.364 1.636l-.707.707M21 12h-1M4 12H3m3.343-5.657l-.707-.707m2.828 9.9a5 5 0 117.072 0l-.548.547A3.374 3.374 0 0014 18.469V19a2 2 0 11-4 0v-.531c0-.895-.356-1.754-.988-2.386l-.548-.547z"/></svg>
+        Ideas <span class="text-sm font-normal text-slate-500">({{ results.ideas|length }})</span>
+    </h2>
+    <div class="bg-slate-900/60 rounded-xl border border-slate-800 divide-y divide-slate-800/30">
+        {% for idea in results.ideas %}
+        <div class="px-5 py-3 hover:bg-slate-800/30 transition">
+            <div class="text-sm text-slate-200 font-medium">{{ idea.title }}</div>
+            {% if idea.description %}
+            <p class="text-xs text-slate-500 mt-1 line-clamp-2">{{ idea.description }}</p>
+            {% endif %}
+            <div class="flex gap-3 mt-1 text-xs text-slate-600">
+                {% if idea.type %}<span class="text-slate-500">{{ idea.type }}</span>{% endif %}
+                <a href="/drafts/{{ idea.draft_name }}" class="text-blue-500 hover:text-blue-400">{{ idea.draft_name }}</a>
+            </div>
+        </div>
+        {% endfor %}
+    </div>
+</div>
+{% endif %}
+
+<!-- Authors -->
+{% if results.authors %}
+<div class="mb-8">
+    <h2 class="text-lg font-semibold text-white mb-3 flex items-center gap-2">
+        <svg class="w-5 h-5 text-green-400" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M17 20h5v-2a3 3 0 00-5.356-1.857M17 20H7m10 0v-2c0-.656-.126-1.283-.356-1.857M7 20H2v-2a3 3 0 015.356-1.857M7 20v-2c0-.656.126-1.283.356-1.857m0 0a5.002 5.002 0 019.288 0M15 7a3 3 0 11-6 0 3 3 0 016 0z"/></svg>
+        Authors <span class="text-sm font-normal text-slate-500">({{ results.authors|length }})</span>
+    </h2>
+    <div class="bg-slate-900/60 rounded-xl border border-slate-800">
+        <div class="grid grid-cols-1 md:grid-cols-2 lg:grid-cols-3 gap-0 divide-y md:divide-y-0 divide-slate-800/30">
+            {% for author in results.authors %}
+            <div class="px-5 py-3 hover:bg-slate-800/30 transition {% if not loop.last %}border-b md:border-b-0 md:border-r border-slate-800/30{% endif %}">
+                <div class="text-sm text-slate-200 font-medium">{{ author.name }}</div>
+                {% if author.affiliation %}
+                <div class="text-xs text-slate-500 mt-0.5">{{ author.affiliation }}</div>
+                {% endif %}
+            </div>
+            {% endfor %}
+        </div>
+    </div>
+</div>
+{% endif %}
+
+<!-- Gaps -->
+{% if results.gaps %}
+<div class="mb-8">
+    <h2 class="text-lg font-semibold text-white mb-3 flex items-center gap-2">
+        <svg class="w-5 h-5 text-red-400" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 9v2m0 4h.01m-6.938 4h13.856c1.54 0 2.502-1.667 1.732-2.5L13.732 4.5c-.77-.833-2.694-.833-3.464 0L3.34 16.5c-.77.833.192 2.5 1.732 2.5z"/></svg>
+        Gaps <span class="text-sm font-normal text-slate-500">({{ results.gaps|length }})</span>
+    </h2>
+    <div class="bg-slate-900/60 rounded-xl border border-slate-800 divide-y divide-slate-800/30">
+        {% for gap in results.gaps %}
+        <div class="px-5 py-3 hover:bg-slate-800/30 transition">
+            <a href="/gaps/{{ gap.id }}" class="text-blue-400 hover:text-blue-300 font-medium text-sm transition">
+                {{ gap.topic }}
+            </a>
+            {% if gap.description %}
+            <p class="text-xs text-slate-500 mt-1 line-clamp-2">{{ gap.description }}</p>
+            {% endif %}
+            <div class="flex gap-3 mt-1 text-xs">
+                {% if gap.category %}<span class="text-slate-500">{{ gap.category }}</span>{% endif %}
+                {% if gap.severity %}
+                <span class="{% if gap.severity == 'high' %}text-red-400{% elif gap.severity == 'medium' %}text-yellow-400{% else %}text-green-400{% endif %}">
+                    {{ gap.severity }}
+                </span>
+                {% endif %}
+            </div>
+        </div>
+        {% endfor %}
+    </div>
+</div>
+{% endif %}
+
+<!-- No results -->
+{% if total == 0 %}
+<div class="text-center py-16">
+    <svg class="w-16 h-16 mx-auto mb-4 text-slate-700" fill="none" stroke="currentColor" viewBox="0 0 24 24">
+        <path stroke-linecap="round" stroke-linejoin="round" stroke-width="1.5" d="M21 21l-6-6m2-5a7 7 0 11-14 0 7 7 0 0114 0z"/>
+    </svg>
+    <p class="text-slate-500 text-sm">No results found for "<span class="text-slate-400">{{ query }}</span>"</p>
+    <p class="text-slate-600 text-xs mt-2">Try different keywords or check the spelling.</p>
+</div>
+{% endif %}
+
+{% endif %}
+{% endblock %}

tests/conftest.py

@@ -0,0 +1,168 @@
+"""Shared fixtures for IETF Draft Analyzer tests."""
+
+from __future__ import annotations
+
+import json
+import sqlite3
+from datetime import datetime, timezone
+
+import numpy as np
+import pytest
+
+from ietf_analyzer.config import Config
+from ietf_analyzer.db import Database, SCHEMA
+from ietf_analyzer.models import Author, Draft, Rating
+
+
+@pytest.fixture
+def tmp_db(tmp_path):
+    """Create an in-memory Database with all tables initialized."""
+    cfg = Config(
+        data_dir=str(tmp_path),
+        db_path=str(tmp_path / "test.db"),
+    )
+    db = Database(cfg)
+    # Force connection + schema creation
+    _ = db.conn
+    yield db
+    db.close()
+
+
+@pytest.fixture
+def sample_draft():
+    """Return a Draft object with realistic data."""
+    return Draft(
+        name="draft-test-ai-agent-protocol",
+        rev="02",
+        title="AI Agent Communication Protocol",
+        abstract="This document defines a protocol for autonomous AI agents to communicate with each other in a standardized manner.",
+        time="2025-06-15T12:00:00+00:00",
+        dt_id=12345,
+        pages=28,
+        words=12000,
+        group="dispatch",
+        group_uri="/api/v1/group/group/1234/",
+        expires="2025-12-15T12:00:00+00:00",
+        ad=None,
+        shepherd=None,
+        states=["I-D Exists"],
+        full_text="Internet-Draft: AI Agent Communication Protocol\n\nAbstract\n\nThis document defines...",
+        categories=["A2A protocols", "Agent discovery/reg"],
+        tags=["ai", "agent"],
+        fetched_at="2025-06-20T10:00:00+00:00",
+    )
+
+
+@pytest.fixture
+def sample_rating():
+    """Return a Rating object with realistic data."""
+    return Rating(
+        draft_name="draft-test-ai-agent-protocol",
+        novelty=4,
+        maturity=3,
+        overlap=2,
+        momentum=3,
+        relevance=5,
+        summary="Defines a novel protocol for AI agent communication with discovery and auth mechanisms.",
+        novelty_note="Unique approach to agent handshake",
+        maturity_note="Early stage but well-structured",
+        overlap_note="Partially overlaps with MCP drafts",
+        momentum_note="Active working group interest",
+        relevance_note="Directly addresses core AI agent interop",
+        categories=["A2A protocols", "Agent discovery/reg"],
+        rated_at="2025-06-20T10:00:00+00:00",
+    )
+
+
+def _make_draft(name, title, time, group=None, pages=10, categories=None):
+    """Helper to create Draft objects for seeding."""
+    return Draft(
+        name=name,
+        rev="00",
+        title=title,
+        abstract=f"Abstract for {title}.",
+        time=time,
+        dt_id=None,
+        pages=pages,
+        words=pages * 400,
+        group=group,
+        categories=categories or [],
+        fetched_at=datetime.now(timezone.utc).isoformat(),
+    )
+
+
+def _make_rating(draft_name, novelty, maturity, overlap, momentum, relevance, categories=None):
+    """Helper to create Rating objects for seeding."""
+    return Rating(
+        draft_name=draft_name,
+        novelty=novelty,
+        maturity=maturity,
+        overlap=overlap,
+        momentum=momentum,
+        relevance=relevance,
+        summary=f"Summary for {draft_name}.",
+        categories=categories or ["A2A protocols"],
+        rated_at=datetime.now(timezone.utc).isoformat(),
+    )
+
+
+@pytest.fixture
+def seeded_db(tmp_db):
+    """Populate tmp_db with 5 drafts, ratings, ideas, authors, and refs."""
+    db = tmp_db
+
+    drafts = [
+        _make_draft("draft-alpha-agent-comm", "Alpha Agent Communication", "2025-01-10", "dispatch", 20, ["A2A protocols"]),
+        _make_draft("draft-beta-ml-traffic", "Beta ML Traffic Optimization", "2025-02-15", "netmod", 15, ["ML traffic mgmt"]),
+        _make_draft("draft-gamma-agent-id", "Gamma Agent Identity", "2025-03-20", "secdispatch", 12, ["Agent identity/auth"]),
+        _make_draft("draft-delta-safety", "Delta AI Safety Framework", "2025-04-25", None, 30, ["AI safety/alignment"]),
+        _make_draft("draft-epsilon-discovery", "Epsilon Agent Discovery", "2025-05-30", "dispatch", 8, ["Agent discovery/reg"]),
+    ]
+    for d in drafts:
+        db.upsert_draft(d)
+
+    ratings = [
+        _make_rating("draft-alpha-agent-comm", 4, 3, 2, 3, 5, ["A2A protocols"]),
+        _make_rating("draft-beta-ml-traffic", 3, 4, 3, 2, 3, ["ML traffic mgmt"]),
+        _make_rating("draft-gamma-agent-id", 5, 2, 1, 4, 4, ["Agent identity/auth"]),
+        _make_rating("draft-delta-safety", 3, 3, 4, 3, 4, ["AI safety/alignment"]),
+        _make_rating("draft-epsilon-discovery", 4, 2, 2, 5, 5, ["Agent discovery/reg"]),
+    ]
+    for r in ratings:
+        db.upsert_rating(r)
+
+    # Ideas
+    db.insert_ideas("draft-alpha-agent-comm", [
+        {"title": "Agent Handshake", "description": "Three-way handshake for agents", "type": "protocol"},
+        {"title": "Capability Negotiation", "description": "Agents advertise capabilities", "type": "mechanism"},
+    ])
+    db.insert_ideas("draft-beta-ml-traffic", [
+        {"title": "ML Traffic Classifier", "description": "Classify traffic using ML", "type": "mechanism"},
+    ])
+    db.insert_ideas("draft-gamma-agent-id", [
+        {"title": "Agent Certificate", "description": "X.509 extension for agents", "type": "extension"},
+    ])
+
+    # Authors
+    author1 = Author(person_id=1001, name="Alice Researcher", ascii_name="Alice Researcher",
+                      affiliation="ExampleCorp", fetched_at=datetime.now(timezone.utc).isoformat())
+    author2 = Author(person_id=1002, name="Bob Engineer", ascii_name="Bob Engineer",
+                      affiliation="TestLabs", fetched_at=datetime.now(timezone.utc).isoformat())
+    author3 = Author(person_id=1003, name="Carol Scientist", ascii_name="Carol Scientist",
+                      affiliation="ExampleCorp", fetched_at=datetime.now(timezone.utc).isoformat())
+    for a in [author1, author2, author3]:
+        db.upsert_author(a)
+
+    db.upsert_draft_author("draft-alpha-agent-comm", 1001, 1, "ExampleCorp")
+    db.upsert_draft_author("draft-alpha-agent-comm", 1002, 2, "TestLabs")
+    db.upsert_draft_author("draft-beta-ml-traffic", 1002, 1, "TestLabs")
+    db.upsert_draft_author("draft-gamma-agent-id", 1001, 1, "ExampleCorp")
+    db.upsert_draft_author("draft-gamma-agent-id", 1003, 2, "ExampleCorp")
+    db.upsert_draft_author("draft-delta-safety", 1003, 1, "ExampleCorp")
+
+    # Refs
+    db.insert_refs("draft-alpha-agent-comm", [("rfc", "8259"), ("rfc", "9110"), ("draft", "draft-ietf-httpbis")])
+    db.insert_refs("draft-beta-ml-traffic", [("rfc", "8259"), ("bcp", "BCP14")])
+    db.insert_refs("draft-gamma-agent-id", [("rfc", "5280"), ("rfc", "8259")])
+
+    yield db

tests/test_db.py

@@ -0,0 +1,287 @@
+"""Tests for ietf_analyzer.db.Database."""
+
+from __future__ import annotations
+
+import json
+from datetime import datetime, timezone
+
+import numpy as np
+import pytest
+
+from ietf_analyzer.db import Database
+from ietf_analyzer.models import Author, Draft, Rating
+
+
+# ---- Table creation ----
+
+
+def test_ensure_tables_creates_all(tmp_db):
+    """All expected tables should exist after Database initialization."""
+    rows = tmp_db.conn.execute(
+        "SELECT name FROM sqlite_master WHERE type='table' ORDER BY name"
+    ).fetchall()
+    table_names = {r["name"] for r in rows}
+    expected = {
+        "drafts", "ratings", "embeddings", "llm_cache",
+        "authors", "draft_authors", "ideas", "gaps",
+        "draft_refs", "generated_drafts", "generation_runs",
+        "sources", "observatory_snapshots", "gap_history",
+        "annotations", "monitor_runs",
+    }
+    assert expected.issubset(table_names), f"Missing tables: {expected - table_names}"
+
+
+# ---- Drafts ----
+
+
+def test_upsert_draft_insert(tmp_db, sample_draft):
+    """Inserting a new draft should make it retrievable."""
+    tmp_db.upsert_draft(sample_draft)
+    retrieved = tmp_db.get_draft(sample_draft.name)
+    assert retrieved is not None
+    assert retrieved.name == sample_draft.name
+    assert retrieved.title == sample_draft.title
+    assert retrieved.rev == sample_draft.rev
+    assert retrieved.pages == sample_draft.pages
+    assert retrieved.categories == sample_draft.categories
+
+
+def test_upsert_draft_update(tmp_db, sample_draft):
+    """Upserting an existing draft should update its fields."""
+    tmp_db.upsert_draft(sample_draft)
+    sample_draft.title = "Updated Title"
+    sample_draft.rev = "03"
+    tmp_db.upsert_draft(sample_draft)
+
+    retrieved = tmp_db.get_draft(sample_draft.name)
+    assert retrieved.title == "Updated Title"
+    assert retrieved.rev == "03"
+    # Should still be only one draft
+    assert tmp_db.count_drafts() == 1
+
+
+def test_search_drafts_fts5(tmp_db, sample_draft):
+    """FTS5 search should find drafts matching query terms."""
+    tmp_db.upsert_draft(sample_draft)
+    results = tmp_db.search_drafts("autonomous agents communicate")
+    assert len(results) >= 1
+    assert results[0].name == sample_draft.name
+
+
+def test_search_drafts_no_results(tmp_db, sample_draft):
+    """FTS5 search with non-matching query should return empty list."""
+    tmp_db.upsert_draft(sample_draft)
+    results = tmp_db.search_drafts("quantum blockchain hyperledger")
+    assert results == []
+
+
+def test_list_drafts_pagination(seeded_db):
+    """list_drafts should respect limit and order_by."""
+    all_drafts = seeded_db.list_drafts(limit=100, order_by="name ASC")
+    assert len(all_drafts) == 5
+
+    first_two = seeded_db.list_drafts(limit=2, order_by="name ASC")
+    assert len(first_two) == 2
+    assert first_two[0].name == "draft-alpha-agent-comm"
+    assert first_two[1].name == "draft-beta-ml-traffic"
+
+
+def test_count_drafts(seeded_db):
+    """count_drafts should return accurate count."""
+    assert seeded_db.count_drafts() == 5
+
+
+# ---- Ratings ----
+
+
+def test_upsert_rating(tmp_db, sample_draft, sample_rating):
+    """Inserting a rating should make it retrievable."""
+    tmp_db.upsert_draft(sample_draft)
+    tmp_db.upsert_rating(sample_rating)
+    retrieved = tmp_db.get_rating(sample_rating.draft_name)
+    assert retrieved is not None
+    assert retrieved.novelty == 4
+    assert retrieved.relevance == 5
+    assert "A2A protocols" in retrieved.categories
+
+
+def test_drafts_with_ratings(seeded_db):
+    """drafts_with_ratings should return (Draft, Rating) pairs."""
+    pairs = seeded_db.drafts_with_ratings(limit=100)
+    assert len(pairs) == 5
+    for draft, rating in pairs:
+        assert isinstance(draft, Draft)
+        assert isinstance(rating, Rating)
+        assert draft.name == rating.draft_name
+
+
+def test_drafts_without_text(tmp_db):
+    """drafts_without_text should return drafts where full_text is None."""
+    d1 = Draft(name="draft-has-text", rev="00", title="Has Text", abstract="Abs",
+               time="2025-01-01", full_text="Some text here")
+    d2 = Draft(name="draft-no-text", rev="00", title="No Text", abstract="Abs",
+               time="2025-01-01", full_text=None)
+    tmp_db.upsert_draft(d1)
+    tmp_db.upsert_draft(d2)
+
+    missing = tmp_db.drafts_without_text()
+    names = [d.name for d in missing]
+    assert "draft-no-text" in names
+    assert "draft-has-text" not in names
+
+
+# ---- Ideas ----
+
+
+def test_insert_ideas(seeded_db):
+    """Bulk idea insertion should work correctly."""
+    ideas = [
+        {"title": "New Idea A", "description": "Desc A", "type": "mechanism"},
+        {"title": "New Idea B", "description": "Desc B", "type": "protocol"},
+    ]
+    seeded_db.insert_ideas("draft-epsilon-discovery", ideas)
+    retrieved = seeded_db.get_ideas_for_draft("draft-epsilon-discovery")
+    assert len(retrieved) == 2
+    assert retrieved[0]["title"] == "New Idea A"
+
+
+def test_get_ideas_for_draft(seeded_db):
+    """Retrieving ideas for a specific draft should return correct data."""
+    ideas = seeded_db.get_ideas_for_draft("draft-alpha-agent-comm")
+    assert len(ideas) == 2
+    titles = {i["title"] for i in ideas}
+    assert "Agent Handshake" in titles
+    assert "Capability Negotiation" in titles
+
+
+def test_insert_ideas_replaces_existing(seeded_db):
+    """Inserting ideas for a draft should replace existing ideas."""
+    seeded_db.insert_ideas("draft-alpha-agent-comm", [
+        {"title": "Replacement Idea", "description": "Replaced", "type": "pattern"},
+    ])
+    ideas = seeded_db.get_ideas_for_draft("draft-alpha-agent-comm")
+    assert len(ideas) == 1
+    assert ideas[0]["title"] == "Replacement Idea"
+
+
+# ---- Gaps ----
+
+
+def test_insert_gaps(tmp_db):
+    """Gap insertion should work correctly."""
+    gaps = [
+        {"topic": "Agent Auth Gap", "description": "No standard auth for agents",
+         "category": "Agent identity/auth", "severity": "critical", "evidence": "Only 2 drafts address this"},
+        {"topic": "Monitoring Gap", "description": "No agent monitoring standard",
+         "category": "Autonomous netops", "severity": "high", "evidence": "Zero drafts cover monitoring"},
+    ]
+    tmp_db.insert_gaps(gaps)
+    retrieved = tmp_db.all_gaps()
+    assert len(retrieved) == 2
+
+
+def test_all_gaps(tmp_db):
+    """all_gaps should return all inserted gaps with correct fields."""
+    gaps = [
+        {"topic": "Test Gap", "description": "Test description",
+         "category": "Other", "severity": "medium", "evidence": "Test evidence"},
+    ]
+    tmp_db.insert_gaps(gaps)
+    result = tmp_db.all_gaps()
+    assert len(result) == 1
+    assert result[0]["topic"] == "Test Gap"
+    assert result[0]["severity"] == "medium"
+    assert result[0]["evidence"] == "Test evidence"
+
+
+# ---- Embeddings ----
+
+
+def test_store_embedding(tmp_db, sample_draft):
+    """Storing an embedding should persist the numpy vector."""
+    tmp_db.upsert_draft(sample_draft)
+    vec = np.array([0.1, 0.2, 0.3, 0.4, 0.5], dtype=np.float32)
+    tmp_db.store_embedding(sample_draft.name, "test-model", vec)
+
+    retrieved = tmp_db.get_embedding(sample_draft.name)
+    assert retrieved is not None
+    np.testing.assert_array_almost_equal(retrieved, vec)
+
+
+def test_all_embeddings(tmp_db, sample_draft):
+    """all_embeddings should return dict of {name: ndarray}."""
+    tmp_db.upsert_draft(sample_draft)
+    vec = np.array([1.0, 2.0, 3.0], dtype=np.float32)
+    tmp_db.store_embedding(sample_draft.name, "test-model", vec)
+
+    all_emb = tmp_db.all_embeddings()
+    assert sample_draft.name in all_emb
+    np.testing.assert_array_almost_equal(all_emb[sample_draft.name], vec)
+
+
+# ---- LLM Cache ----
+
+
+def test_cache_response(tmp_db):
+    """Caching an LLM response should be retrievable by draft_name + hash."""
+    tmp_db.cache_response(
+        "draft-test", "abc123hash", "claude-test",
+        "prompt text", '{"result": "ok"}', 100, 50,
+    )
+    cached = tmp_db.get_cached_response("draft-test", "abc123hash")
+    assert cached is not None
+    assert json.loads(cached) == {"result": "ok"}
+
+
+def test_cache_response_miss(tmp_db):
+    """Cache miss should return None."""
+    result = tmp_db.get_cached_response("nonexistent", "badhash")
+    assert result is None
+
+
+# ---- Refs ----
+
+
+def test_insert_refs(seeded_db):
+    """Reference insertion should work and be queryable."""
+    refs = seeded_db.get_refs_for_draft("draft-alpha-agent-comm")
+    assert len(refs) == 3
+    ref_types = {r[0] for r in refs}
+    assert "rfc" in ref_types
+    assert "draft" in ref_types
+
+
+def test_top_refs(seeded_db):
+    """top_referenced should return most commonly cited RFCs."""
+    top = seeded_db.top_referenced(ref_type="rfc", limit=5)
+    # RFC 8259 is referenced by 3 drafts
+    assert len(top) > 0
+    assert top[0][0] == "8259"
+    assert top[0][1] == 3
+
+
+# ---- Authors ----
+
+
+def test_get_authors_for_draft(seeded_db):
+    """Getting authors for a draft should return correct Author objects."""
+    authors = seeded_db.get_authors_for_draft("draft-alpha-agent-comm")
+    assert len(authors) == 2
+    names = {a.name for a in authors}
+    assert "Alice Researcher" in names
+    assert "Bob Engineer" in names
+
+
+def test_author_count(seeded_db):
+    """author_count should return the total number of unique authors."""
+    assert seeded_db.author_count() == 3
+
+
+def test_top_authors(seeded_db):
+    """top_authors should return authors sorted by draft count."""
+    top = seeded_db.top_authors(limit=10)
+    # Alice and Bob each have 2 drafts, Carol has 2 as well
+    assert len(top) > 0
+    # First author should have most drafts
+    name, aff, count, draft_names = top[0]
+    assert count >= 2

tests/test_models.py

@@ -0,0 +1,190 @@
+"""Tests for ietf_analyzer.models and ietf_analyzer.config."""
+
+from __future__ import annotations
+
+import json
+import os
+from pathlib import Path
+
+import pytest
+
+from ietf_analyzer.models import Draft, Rating, Author, normalize_category, CATEGORY_NORMALIZE
+from ietf_analyzer.config import Config, DEFAULT_KEYWORDS
+
+
+# ---- Rating ----
+
+
+def test_rating_composite_score():
+    """Composite score should use weighted average formula."""
+    r = Rating(
+        draft_name="test", novelty=4, maturity=3, overlap=2,
+        momentum=3, relevance=5, summary="test",
+    )
+    # Expected: 4*0.30 + 5*0.25 + 3*0.20 + 3*0.15 + (6-2)*0.10
+    expected = 4 * 0.30 + 5 * 0.25 + 3 * 0.20 + 3 * 0.15 + (6 - 2) * 0.10
+    assert abs(r.composite_score - expected) < 0.001
+
+
+def test_rating_composite_score_all_ones():
+    """Composite score with all 1s should be the minimum."""
+    r = Rating(
+        draft_name="test", novelty=1, maturity=1, overlap=5,
+        momentum=1, relevance=1, summary="test",
+    )
+    expected = 1 * 0.30 + 1 * 0.25 + 1 * 0.20 + 1 * 0.15 + (6 - 5) * 0.10
+    assert abs(r.composite_score - expected) < 0.001
+
+
+def test_rating_composite_score_all_fives():
+    """Composite score with all 5s (except overlap=1 for best)."""
+    r = Rating(
+        draft_name="test", novelty=5, maturity=5, overlap=1,
+        momentum=5, relevance=5, summary="test",
+    )
+    expected = 5 * 0.30 + 5 * 0.25 + 5 * 0.20 + 5 * 0.15 + (6 - 1) * 0.10
+    assert abs(r.composite_score - expected) < 0.001
+    assert r.composite_score == 5.0
+
+
+# ---- Draft ----
+
+
+def test_draft_datatracker_url():
+    """datatracker_url should construct the correct URL."""
+    d = Draft(name="draft-example-test", rev="00", title="Test", abstract="", time="2025-01-01")
+    assert d.datatracker_url == "https://datatracker.ietf.org/doc/draft-example-test/"
+
+
+def test_draft_text_url():
+    """text_url should construct the correct URL with revision."""
+    d = Draft(name="draft-example-test", rev="03", title="Test", abstract="", time="2025-01-01")
+    assert d.text_url == "https://www.ietf.org/archive/id/draft-example-test-03.txt"
+
+
+def test_draft_defaults():
+    """Draft should have sensible defaults for optional fields."""
+    d = Draft(name="draft-minimal", rev="00", title="Min", abstract="", time="2025-01-01")
+    assert d.dt_id is None
+    assert d.pages is None
+    assert d.words is None
+    assert d.group is None
+    assert d.full_text is None
+    assert d.categories == []
+    assert d.tags == []
+    assert d.states == []
+    assert d.source == "ietf"
+
+
+def test_draft_date_property():
+    """Draft.date should return just the date portion of time."""
+    d = Draft(name="test", rev="00", title="T", abstract="", time="2025-06-15T12:00:00+00:00")
+    assert d.date == "2025-06-15"
+
+
+def test_draft_date_empty():
+    """Draft.date should return empty string if time is None."""
+    d = Draft(name="test", rev="00", title="T", abstract="", time=None)
+    assert d.date == ""
+
+
+# ---- normalize_category ----
+
+
+def test_normalize_category():
+    """Known verbose category names should be normalized to short forms."""
+    assert normalize_category("Agent-to-agent communication protocols") == "A2A protocols"
+    assert normalize_category("AI safety / guardrails / alignment") == "AI safety/alignment"
+
+
+def test_normalize_category_passthrough():
+    """Unknown category names should pass through unchanged."""
+    assert normalize_category("A2A protocols") == "A2A protocols"
+    assert normalize_category("Some Unknown Category") == "Some Unknown Category"
+
+
+# ---- Config ----
+
+
+def test_config_load_defaults():
+    """Config without a file should use defaults."""
+    cfg = Config()
+    assert cfg.ollama_url == "http://localhost:11434"
+    assert cfg.claude_model != ""
+    assert cfg.fetch_delay == 0.5
+
+
+def test_config_save_and_load(tmp_path):
+    """Config should roundtrip through save/load."""
+    cfg = Config(
+        data_dir=str(tmp_path),
+        db_path=str(tmp_path / "test.db"),
+        claude_model="claude-test-model",
+    )
+    # Save to the default config path (override it)
+    config_file = tmp_path / "config.json"
+    config_file.write_text(json.dumps({
+        "data_dir": str(tmp_path),
+        "db_path": str(tmp_path / "test.db"),
+        "claude_model": "claude-test-model",
+        "ollama_url": "http://localhost:11434",
+        "search_keywords": ["agent", "ai-agent"],
+    }))
+
+    # Verify roundtrip by reading back
+    data = json.loads(config_file.read_text())
+    loaded = Config(**{k: v for k, v in data.items() if k in Config.__dataclass_fields__})
+    assert loaded.claude_model == "claude-test-model"
+    assert loaded.db_path == str(tmp_path / "test.db")
+
+
+def test_config_search_keywords():
+    """Default config should have the expected search keywords."""
+    cfg = Config()
+    assert "agent" in cfg.search_keywords
+    assert "mcp" in cfg.search_keywords
+    assert "agentic" in cfg.search_keywords
+    assert len(cfg.search_keywords) == len(DEFAULT_KEYWORDS)
+
+
+def _patch_config_file(monkeypatch, tmp_path):
+    """Point CONFIG_FILE to a non-existent path so tests use defaults."""
+    import ietf_analyzer.config as config_mod
+    monkeypatch.setattr(config_mod, "CONFIG_FILE", tmp_path / "config.json")
+
+
+def test_config_env_var_override(tmp_path, monkeypatch):
+    """Environment variables should override config file values."""
+    _patch_config_file(monkeypatch, tmp_path)
+    monkeypatch.setenv("IETF_ANALYZER_DB_PATH", str(tmp_path / "env.db"))
+    monkeypatch.setenv("IETF_ANALYZER_CLAUDE_MODEL", "claude-from-env")
+    monkeypatch.setenv("IETF_ANALYZER_OLLAMA_URL", "http://remote:11434")
+
+    cfg = Config.load()
+    assert cfg.db_path == str(tmp_path / "env.db")
+    assert cfg.claude_model == "claude-from-env"
+    assert cfg.ollama_url == "http://remote:11434"
+
+
+def test_config_validation_bad_model(tmp_path, monkeypatch):
+    """Empty claude_model should raise ValueError."""
+    _patch_config_file(monkeypatch, tmp_path)
+    monkeypatch.setenv("IETF_ANALYZER_CLAUDE_MODEL", "")
+    with pytest.raises(ValueError, match="claude_model"):
+        Config.load()
+
+
+def test_config_validation_bad_url(tmp_path, monkeypatch):
+    """Non-URL ollama_url should raise ValueError."""
+    _patch_config_file(monkeypatch, tmp_path)
+    monkeypatch.setenv("IETF_ANALYZER_OLLAMA_URL", "not-a-url")
+    with pytest.raises(ValueError, match="ollama_url"):
+        Config.load()
+
+
+def test_config_validation_bad_db_path(tmp_path, monkeypatch):
+    """db_path with non-existent parent directory should raise ValueError."""
+    _patch_config_file(monkeypatch, tmp_path)
+    monkeypatch.setenv("IETF_ANALYZER_DB_PATH", "/nonexistent/dir/test.db")
+    with pytest.raises(ValueError, match="db_path"):
+        Config.load()

tests/test_web_data.py

@@ -0,0 +1,158 @@
+"""Tests for src/webui/data.py data access functions."""
+
+from __future__ import annotations
+
+import sys
+from functools import wraps
+from pathlib import Path
+
+import pytest
+
+# Ensure webui is importable
+_project_root = Path(__file__).resolve().parent.parent
+if str(_project_root / "src") not in sys.path:
+    sys.path.insert(0, str(_project_root / "src"))
+
+from webui.data import (
+    get_overview_stats,
+    get_category_counts,
+    get_drafts_page,
+    get_draft_detail,
+    get_ideas_by_type,
+    get_all_gaps,
+    get_timeline_data,
+    get_top_authors,
+)
+
+
+def _skip_on_missing_module(fn):
+    """Decorator that skips tests when webui.data references unavailable modules."""
+    @wraps(fn)
+    def wrapper(*args, **kwargs):
+        try:
+            return fn(*args, **kwargs)
+        except (ModuleNotFoundError, AttributeError) as e:
+            pytest.skip(f"webui.data depends on module not in this worktree: {e}")
+    return wrapper
+
+
+def test_get_overview_stats(seeded_db):
+    """Overview stats should return correct counts from seeded data."""
+    stats = get_overview_stats(seeded_db)
+    assert stats["total_drafts"] == 5
+    assert stats["rated_count"] == 5
+    assert stats["author_count"] == 3
+    # 2 + 1 + 1 = 4 ideas in seeded data
+    assert stats["idea_count"] == 4
+    assert stats["gap_count"] == 0
+    assert "input_tokens" in stats
+    assert "output_tokens" in stats
+
+
+def test_get_category_counts(seeded_db):
+    """Category counts should reflect the seeded ratings."""
+    counts = get_category_counts(seeded_db)
+    assert isinstance(counts, dict)
+    assert "A2A protocols" in counts
+    assert counts["A2A protocols"] == 1
+    assert "ML traffic mgmt" in counts
+
+
+@_skip_on_missing_module
+def test_get_drafts_page_basic(seeded_db):
+    """Drafts page should return paginated results."""
+    result = get_drafts_page(seeded_db, page=1, per_page=3)
+    assert len(result["drafts"]) == 3
+    assert result["total"] == 5
+    assert result["page"] == 1
+    assert result["per_page"] == 3
+    assert result["pages"] == 2
+
+
+@_skip_on_missing_module
+def test_get_drafts_page_with_category_filter(seeded_db):
+    """Filtering by category should narrow results."""
+    result = get_drafts_page(seeded_db, category="A2A protocols")
+    assert result["total"] == 1
+    assert result["drafts"][0]["categories"] == ["A2A protocols"]
+
+
+@_skip_on_missing_module
+def test_get_drafts_page_with_search_filter(seeded_db):
+    """Text search should filter by name/title/summary."""
+    result = get_drafts_page(seeded_db, search="alpha")
+    assert result["total"] == 1
+    assert "alpha" in result["drafts"][0]["name"]
+
+
+@_skip_on_missing_module
+def test_get_drafts_page_empty_search(seeded_db):
+    """Search for non-matching term should return 0 results."""
+    result = get_drafts_page(seeded_db, search="zzzznonexistent")
+    assert result["total"] == 0
+    assert result["drafts"] == []
+
+
+@_skip_on_missing_module
+def test_get_draft_detail(seeded_db):
+    """Draft detail should include draft, rating, authors, ideas, refs."""
+    detail = get_draft_detail(seeded_db, "draft-alpha-agent-comm")
+    assert detail is not None
+    assert detail["name"] == "draft-alpha-agent-comm"
+    assert detail["title"] == "Alpha Agent Communication"
+    assert "rating" in detail
+    assert detail["rating"]["novelty"] == 4
+    assert len(detail["authors"]) == 2
+    assert len(detail["ideas"]) == 2
+    assert len(detail["refs"]) == 3
+
+
+@_skip_on_missing_module
+def test_get_draft_detail_not_found(seeded_db):
+    """Draft detail for non-existent draft should return None."""
+    assert get_draft_detail(seeded_db, "draft-nonexistent") is None
+
+
+def test_get_ideas_by_type(seeded_db):
+    """Ideas by type should group and count correctly."""
+    result = get_ideas_by_type(seeded_db)
+    assert result["total"] == 4
+    assert "by_type" in result
+    assert isinstance(result["by_type"], dict)
+    # We have protocol, mechanism, extension types
+    assert "protocol" in result["by_type"] or "mechanism" in result["by_type"]
+
+
+def test_get_all_gaps_empty(seeded_db):
+    """With no gaps inserted, should return empty list."""
+    gaps = get_all_gaps(seeded_db)
+    assert gaps == []
+
+
+def test_get_all_gaps_with_data(seeded_db):
+    """After inserting gaps, should return them."""
+    seeded_db.insert_gaps([
+        {"topic": "Gap A", "description": "Desc A", "severity": "high", "evidence": "Ev A"},
+    ])
+    gaps = get_all_gaps(seeded_db)
+    assert len(gaps) == 1
+    assert gaps[0]["topic"] == "Gap A"
+
+
+def test_get_timeline_data(seeded_db):
+    """Timeline data should group drafts by month."""
+    data = get_timeline_data(seeded_db)
+    assert "months" in data
+    assert "series" in data
+    assert "categories" in data
+    # Seeded drafts span Jan-May 2025
+    assert len(data["months"]) >= 1
+
+
+def test_get_top_authors(seeded_db):
+    """Top authors should return ranked list with draft counts."""
+    authors = get_top_authors(seeded_db, limit=10)
+    assert len(authors) >= 1
+    assert "name" in authors[0]
+    assert "draft_count" in authors[0]
+    assert authors[0]["draft_count"] >= 2