docs(status): log 2026-05-22 deployment + data refresh

insert_gaps did a blanket 'DELETE FROM gaps', which fails with
'FOREIGN KEY constraint failed' whenever proposal_gaps references a gap
(generated proposals). Delete only gaps not referenced by a proposal so
the refresh preserves proposal linkage and never trips the FK.

Also logs the 2026-05-22 data refresh (761->889 drafts) in dev-journal.

data/reports/dev-journal.md

@@ -4,6 +4,23 @@
 
 ---
 
+### 2026-05-22 SESSION — Data refresh as of today: 761 → 889 drafts (full Sonnet pipeline)
+
+**What**: First full corpus refresh since 2026-03-08. Fetched the delta from Datatracker (128 new drafts, all agent/identity/oauth-token topics), backfilled their full text, then ran the whole pipeline on Sonnet: rate → embed → extract ideas → score novelty → gap analysis → idea embeddings → convergence. Synced the fresh `drafts.db` to the server and restarted the `ietf` container so ietf.nennemann.de serves it.
+
+**Why**: The deployed site was showing March data; the user wanted it current.
+
+**Result** (live API stats): 816 relevant drafts (889 total, 73 false-positives), 722 authors, 973 ideas (avg novelty 2.8), 18 gaps, 170 cross-org convergent ideas (was 132). Tracked token usage ~1.0M in / 472K out on Sonnet.
+
+**Surprise / lessons**:
+- The fetch pipeline inserted the 128 new drafts but left all of them **without full text** (the text URL needs the `-NN` revision suffix; the per-source download skipped them). Wrote `scripts/backfill-unrated-text.py` (rev-fallback) to fix — analysis quality depends on full text.
+- The shell `ANTHROPIC_API_KEY` env var was **stale (401)**; the valid key was in `.env`. python-dotenv doesn't override an existing env var, so the CLI silently used the bad one. Had to pass the `.env` key explicitly.
+- **Bug fixed**: `db.insert_gaps()` did a blanket `DELETE FROM gaps`, which trips the `proposal_gaps.gap_id` FK whenever generated proposals exist (it did — 3 proposals / 7 links). Changed it to delete only gaps not referenced by a proposal, so `gaps --refresh` is non-destructive.
+
+**Cost**: ~$10 tracked (Sonnet). ideas/gaps are dev-only pages, not shown on the production site, but refreshed anyway per user request.
+
+---
+
 ### 2026-05-22 SESSION — Deployed the web dashboard at ietf.nennemann.de
 
 **What**: Brought the Flask dashboard online on the nennemann-dev server (Hetzner CAX21) behind Caddy at `https://ietf.nennemann.de`, basic_auth gated (shared `vorschau` preview password), `noindex`. Added an `ietf` Docker service to `nennemann-biz/infra/dev/docker-compose.yml` (build context `/home/dev/repos/research.ietf`, host :8082 -> container :5000, data dir mounted read-write so pageview analytics persist). Container runs in PRODUCTION mode (admin routes 404).

docs/status.md

@@ -1,5 +1,21 @@
 # research.ietf — Status
 
+## 2026-05-22 — Deployed at ietf.nennemann.de + full data refresh
+
+### What was done
+- **Deployed** the Flask dashboard to nennemann-dev behind Caddy: `https://ietf.nennemann.de` (basic_auth `vorschau`, `noindex`, production mode). Docker `dev-ietf-1` on `:8082`→`:5000`, data dir mounted read-write. Dockerfile fixed to bind `0.0.0.0`. Infra in `nennemann-biz/infra/dev` (compose + Caddy vhost).
+- **Data refresh** (first since 2026-03-08): 761→889 drafts. Full Sonnet pipeline (rate/embed/ideas/score/gaps/convergence). Live stats: 816 relevant, 722 authors, 973 ideas, 18 gaps, 170 convergent. Fresh `drafts.db` synced to server.
+- New tool: `scripts/backfill-unrated-text.py` (fetch leaves new drafts without full text — rev-suffix issue).
+- Bug fix: `db.insert_gaps()` now FK-safe (was a blanket `DELETE FROM gaps`).
+
+### Next / open
+- Refresh procedure documented in workspace `docs/services.md` (sync **only** drafts.db; analytics.db is server-owned).
+- `ANTHROPIC_API_KEY` shell env was stale — use the key in `.env`.
+- Optional: schedule recurring refresh (`scripts/scheduled-update.sh`) if desired.
+- 16 drafts yielded no extractable ideas (short/non-technical) — expected, left as-is.
+
+---
+
 ## 2026-04-11 — Refimpl -01 hash format fix + draft rebuild
 
 ### What was done

scripts/backfill-unrated-text.py

@@ -0,0 +1,95 @@
+#!/usr/bin/env python3
+"""Backfill full_text for unrated drafts that are missing it.
+
+The fetch pipeline sometimes leaves new IETF drafts without full text (the per-source
+text download can skip them). Analysis quality depends on full text, so this script
+downloads it directly from the IETF archive using the draft's recorded revision, with
+a fallback that probes nearby revisions if the recorded one 404s.
+
+Usage:
+    PYTHONPATH=src python3 scripts/backfill-unrated-text.py [--all] [--limit N]
+
+    --all    Backfill ALL drafts missing full_text (not just unrated ones).
+    --limit  Cap the number of drafts processed.
+"""
+from __future__ import annotations
+
+import argparse
+import sqlite3
+import sys
+import time
+
+sys.path.insert(0, "src")
+
+import httpx
+
+from ietf_analyzer.config import Config
+
+TEXT_BASE = "https://www.ietf.org/archive/id"
+
+
+def candidate_urls(name: str, rev: str) -> list[str]:
+    """URLs to try, in order: recorded rev, then 00..09, then bare name."""
+    urls = [f"{TEXT_BASE}/{name}-{rev}.txt"] if rev else []
+    for r in (f"{i:02d}" for i in range(10)):
+        if r != rev:
+            urls.append(f"{TEXT_BASE}/{name}-{r}.txt")
+    urls.append(f"{TEXT_BASE}/{name}.txt")
+    return urls
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--all", action="store_true", help="all drafts missing text, not just unrated")
+    ap.add_argument("--limit", type=int, default=0)
+    args = ap.parse_args()
+
+    cfg = Config.load()
+    conn = sqlite3.connect(cfg.db_path)
+    conn.row_factory = sqlite3.Row
+
+    where_unrated = "" if args.all else (
+        "AND NOT EXISTS (SELECT 1 FROM ratings r WHERE r.draft_name = d.name)"
+    )
+    sql = f"""
+        SELECT name, rev FROM drafts d
+        WHERE (full_text IS NULL OR full_text = '')
+          AND source = 'ietf'
+          {where_unrated}
+        ORDER BY name
+    """
+    rows = conn.execute(sql).fetchall()
+    if args.limit:
+        rows = rows[: args.limit]
+
+    print(f"Backfilling text for {len(rows)} drafts...")
+    client = httpx.Client(timeout=30, follow_redirects=True)
+    ok = fail = 0
+    for i, row in enumerate(rows, 1):
+        name, rev = row["name"], row["rev"] or "00"
+        text = None
+        for url in candidate_urls(name, rev):
+            try:
+                resp = client.get(url)
+                if resp.status_code == 200 and len(resp.text) > 200:
+                    text = resp.text[:500_000]  # cap 500K
+                    break
+            except httpx.HTTPError:
+                continue
+        if text:
+            conn.execute("UPDATE drafts SET full_text = ? WHERE name = ?", (text, name))
+            conn.commit()
+            ok += 1
+            print(f"  [{i}/{len(rows)}] OK   {name} ({len(text)} chars)")
+        else:
+            fail += 1
+            print(f"  [{i}/{len(rows)}] FAIL {name}")
+        time.sleep(0.3)
+
+    print(f"\nDone. {ok} downloaded, {fail} failed.")
+    conn.close()
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

src/ietf_analyzer/db.py

@@ -975,7 +975,12 @@ class Database:
     # --- Gaps ---
 
     def insert_gaps(self, gaps: list[dict]) -> None:
-        self.conn.execute("DELETE FROM gaps")  # Replace old analysis
+        # Replace old analysis, but keep any gap still referenced by a generated
+        # proposal (proposal_gaps.gap_id FK) so a refresh never destroys proposal
+        # linkage or trips the foreign-key constraint.
+        self.conn.execute(
+            "DELETE FROM gaps WHERE id NOT IN (SELECT gap_id FROM proposal_gaps)"
+        )
         now = datetime.now(timezone.utc).isoformat()
         for g in gaps:
             self.conn.execute(