Platform upgrade: semantic search, citations, readiness, tests, Docker

Major features added by 5 parallel agent teams:
- Semantic "Ask" (NL queries via FTS5 + embeddings + Claude synthesis)
- Global search across drafts, ideas, authors, gaps
- REST API expansion (14 endpoints, up from 3) with CSV/JSON export
- Citation graph visualization (D3.js, 440 nodes, 2422 edges)
- Standards readiness scoring (0-100 composite from 6 factors)
- Side-by-side draft comparison view with shared/unique analysis
- Annotation system (notes + tags per draft, DB-persisted)
- Docker deployment (Dockerfile + docker-compose with Ollama)
- Scheduled updates (cron script with log rotation)
- Pipeline health dashboard (stage progress bars, cost tracking)
- Test suite foundation (54 pytest tests covering DB, models, web data)

Fixes: compare_drafts() stubbed→working, get_authors_for_draft() bug,
source-aware analysis prompts, config env var overrides + validation,
resilient batch error handling with --retry-failed, observatory --dry-run

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

data/reports/dev-journal.md

@@ -4,6 +4,19 @@
 
 ---
 
+### 2026-03-07 CODER C — Citation Graph, Readiness Scoring, Annotations, Data Surfacing
+
+**What**: Implemented four features in a single session:
+1. **Citation Graph Visualization** (`/citations`): D3.js force-directed graph showing cross-references between drafts and RFCs. Nodes colored by type (blue=draft, orange=RFC), sized by influence (in-degree). Includes category filter, min-refs slider, hover tooltips, click-to-navigate, and a top-referenced RFCs table. New `get_citation_graph()` in data.py, route + API endpoint in app.py.
+2. **Standards Readiness Scoring**: New `readiness.py` module computing a 0-100 composite score from 6 weighted factors (WG adoption 25%, revision maturity 15%, reference density 15%, cited-by count 15%, author experience 15%, momentum rating 15%). Displayed as a progress gauge on draft detail pages, added as sortable column on drafts listing, and shown in `ietf show` CLI output.
+3. **Annotation System**: New `annotations` table in DB schema with `upsert_annotation`, `get_annotation`, `get_all_annotations`, `search_by_tag` methods. New `ietf annotate` CLI command with `--note`, `--tag`, `--remove-tag` options. Web UI: inline note editor + tag chips with add/remove on draft detail page, backed by POST `/api/drafts/<name>/annotate` endpoint.
+4. **Surface Underutilized Data**: Exposed `novelty_score` (from pipeline/quality.py) in ideas.html and draft_detail.html as color-coded N:X badges. Gap severity now sorts critical-first (was alphabetical). `all_ideas()` and `get_ideas_for_draft()` now return `novelty_score` field.
+
+**Why**: These features leverage existing data (4231 refs, novelty scores, severity) that was computed but never surfaced to users. Readiness scoring gives a quick at-a-glance RFC proximity signal. Annotations enable user workflow.
+**Result**: 8 files modified (db.py, data.py, app.py, cli.py, base.html, draft_detail.html, ideas.html, drafts.html, gaps.html), 2 files created (readiness.py, citations.html). Citations link added to sidebar nav.
+
+---
+
 ### 2026-03-06 CODER — Interactive D3.js Author Network Visualization
 
 **What**: Replaced the Plotly spring-layout co-authorship graph on `/authors` with a full D3.js v7 force-directed network. Added enriched data layer (`get_author_network_full`) with avg draft scores per author, connected-component cluster detection (68 clusters found), and a new `/api/authors/network` JSON endpoint. Template now includes: interactive D3 force graph with zoom/pan/drag, org filter dropdown, cluster highlighting with zoom-to-fit, hover tooltips showing author details + draft list, click-to-navigate, plus the existing Plotly org bar chart, cross-org collaboration chart, sortable authors table (now top 50), and org stats sidebar.
@@ -411,3 +424,23 @@
 - Decisions made: **GitHub Pages** for publication, **staggered 1/day** cadence, **MIT license**
 - Agent utilization: Architect (2 tasks, shut down), Writer (2 tasks, shut down), Planner (1 task, shut down)
 **Surprise**: The crash recovery was seamless — the dev journal served exactly its intended purpose. Every agent could read the journal and understand the full state without any human explanation. The journal-as-coordination-mechanism is the strongest vindication of the CLAUDE.md journaling requirement. This should feature prominently in Post 8.
+
+### 2026-03-07 CODER E — W3C Integration, Docker, Scheduling, Pipeline Health
+
+**What**: Four-part infrastructure sprint to make the platform multi-source, self-running, and deployable:
+
+1. **W3C Integration** — Wired the existing W3C fetcher (`sources/w3c.py`) into the full pipeline. Made analysis prompts source-aware (`_doc_type_label()` returns "IETF draft" or "W3C specification" based on `source` field). Added source filter dropdown (IETF / W3C / All) to the Draft Explorer web UI with colored source badges (blue for IETF, green for W3C). Updated `get_drafts_page()` to accept `source` parameter. All pagination and sort links preserve source filter state. Config documents how to enable W3C: `ietf observatory update --source w3c` or add `"w3c"` to `observatory_sources` in config.json.
+
+2. **Scheduled Updates** — Created `scripts/scheduled-update.sh` for cron-based automation. Handles .env loading, log rotation (30 days), and proper error exit codes. Usage: `crontab -e -> 0 6 * * * /path/to/scheduled-update.sh`
+
+3. **Docker Deployment** — Created `Dockerfile` (python:3.11-slim), `docker-compose.yml` (web + ollama services with volume mounts for data persistence), and `.dockerignore`. One-command deployment: `docker compose up`.
+
+4. **Pipeline Health** — Enhanced `ietf pipeline status` to show comprehensive health: processing stage breakdown (rated/embedded/ideas with ASCII progress bars), total ideas, gaps, API token usage, estimated cost. Enhanced monitor web page with visual pipeline progress bars, cost tracking panel, and document/idea/gap counts. Added `--dry-run` flag to `ietf observatory update` that previews what would happen. Wrapped all observatory update steps in try/except for graceful error recovery — failures in one stage no longer block subsequent stages.
+
+**Why**: The platform was IETF-only despite having a complete W3C fetcher. Docker makes deployment reproducible. Scheduled updates make it self-running. Error recovery prevents partial failures from wasting an entire update cycle.
+
+**Result**:
+- Files modified: `analyzer.py`, `observatory.py`, `cli.py`, `config.py`, `data.py`, `app.py`, `drafts.html`, `monitor.html`
+- Files created: `Dockerfile`, `docker-compose.yml`, `.dockerignore`, `scripts/scheduled-update.sh`
+- All Python files compile cleanly
+- No breaking changes to existing IETF-only workflows