fix(webui): timeline opens on full landscape, drop undated points

After the chronological-order fix the page opened on the earliest month
(1995-12), showing only a handful of same-colored dots while the legend
listed every category — looked broken.

- Initialise the plot on the LAST frame (full landscape); Play now replays
  the build-up from the start (fromcurrent: false), slider starts at the end.
- Make _extract_month robust: year-only / junk dates (ISO/ETSI 'time' like
  '2015/CD Amd 2', bare '2023', '') no longer yield malformed month labels
  ('2015-/C') or a garbled 'unknown' frame badge.
- Drop undated docs from the temporal animation (they remain on /landscape).

At the full initial frame every category has points, so the legend matches
what is drawn.

data/reports/dev-journal.md

@@ -4,6 +4,23 @@
 
 ---
 
+### 2026-05-22 SESSION — Data refresh as of today: 761 → 889 drafts (full Sonnet pipeline)
+
+**What**: First full corpus refresh since 2026-03-08. Fetched the delta from Datatracker (128 new drafts, all agent/identity/oauth-token topics), backfilled their full text, then ran the whole pipeline on Sonnet: rate → embed → extract ideas → score novelty → gap analysis → idea embeddings → convergence. Synced the fresh `drafts.db` to the server and restarted the `ietf` container so ietf.nennemann.de serves it.
+
+**Why**: The deployed site was showing March data; the user wanted it current.
+
+**Result** (live API stats): 816 relevant drafts (889 total, 73 false-positives), 722 authors, 973 ideas (avg novelty 2.8), 18 gaps, 170 cross-org convergent ideas (was 132). Tracked token usage ~1.0M in / 472K out on Sonnet.
+
+**Surprise / lessons**:
+- The fetch pipeline inserted the 128 new drafts but left all of them **without full text** (the text URL needs the `-NN` revision suffix; the per-source download skipped them). Wrote `scripts/backfill-unrated-text.py` (rev-fallback) to fix — analysis quality depends on full text.
+- The shell `ANTHROPIC_API_KEY` env var was **stale (401)**; the valid key was in `.env`. python-dotenv doesn't override an existing env var, so the CLI silently used the bad one. Had to pass the `.env` key explicitly.
+- **Bug fixed**: `db.insert_gaps()` did a blanket `DELETE FROM gaps`, which trips the `proposal_gaps.gap_id` FK whenever generated proposals exist (it did — 3 proposals / 7 links). Changed it to delete only gaps not referenced by a proposal, so `gaps --refresh` is non-destructive.
+
+**Cost**: ~$10 tracked (Sonnet). ideas/gaps are dev-only pages, not shown on the production site, but refreshed anyway per user request.
+
+---
+
 ### 2026-05-22 SESSION — Deployed the web dashboard at ietf.nennemann.de
 
 **What**: Brought the Flask dashboard online on the nennemann-dev server (Hetzner CAX21) behind Caddy at `https://ietf.nennemann.de`, basic_auth gated (shared `vorschau` preview password), `noindex`. Added an `ietf` Docker service to `nennemann-biz/infra/dev/docker-compose.yml` (build context `/home/dev/repos/research.ietf`, host :8082 -> container :5000, data dir mounted read-write so pageview analytics persist). Container runs in PRODUCTION mode (admin routes 404).

docs/status.md

@@ -1,5 +1,21 @@
 # research.ietf — Status
 
+## 2026-05-22 — Deployed at ietf.nennemann.de + full data refresh
+
+### What was done
+- **Deployed** the Flask dashboard to nennemann-dev behind Caddy: `https://ietf.nennemann.de` (basic_auth `vorschau`, `noindex`, production mode). Docker `dev-ietf-1` on `:8082`→`:5000`, data dir mounted read-write. Dockerfile fixed to bind `0.0.0.0`. Infra in `nennemann-biz/infra/dev` (compose + Caddy vhost).
+- **Data refresh** (first since 2026-03-08): 761→889 drafts. Full Sonnet pipeline (rate/embed/ideas/score/gaps/convergence). Live stats: 816 relevant, 722 authors, 973 ideas, 18 gaps, 170 convergent. Fresh `drafts.db` synced to server.
+- New tool: `scripts/backfill-unrated-text.py` (fetch leaves new drafts without full text — rev-suffix issue).
+- Bug fix: `db.insert_gaps()` now FK-safe (was a blanket `DELETE FROM gaps`).
+
+### Next / open
+- Refresh procedure documented in workspace `docs/services.md` (sync **only** drafts.db; analytics.db is server-owned).
+- `ANTHROPIC_API_KEY` shell env was stale — use the key in `.env`.
+- Optional: schedule recurring refresh (`scripts/scheduled-update.sh`) if desired.
+- 16 drafts yielded no extractable ideas (short/non-technical) — expected, left as-is.
+
+---
+
 ## 2026-04-11 — Refimpl -01 hash format fix + draft rebuild
 
 ### What was done

scripts/backfill-unrated-text.py

@@ -0,0 +1,95 @@
+#!/usr/bin/env python3
+"""Backfill full_text for unrated drafts that are missing it.
+
+The fetch pipeline sometimes leaves new IETF drafts without full text (the per-source
+text download can skip them). Analysis quality depends on full text, so this script
+downloads it directly from the IETF archive using the draft's recorded revision, with
+a fallback that probes nearby revisions if the recorded one 404s.
+
+Usage:
+    PYTHONPATH=src python3 scripts/backfill-unrated-text.py [--all] [--limit N]
+
+    --all    Backfill ALL drafts missing full_text (not just unrated ones).
+    --limit  Cap the number of drafts processed.
+"""
+from __future__ import annotations
+
+import argparse
+import sqlite3
+import sys
+import time
+
+sys.path.insert(0, "src")
+
+import httpx
+
+from ietf_analyzer.config import Config
+
+TEXT_BASE = "https://www.ietf.org/archive/id"
+
+
+def candidate_urls(name: str, rev: str) -> list[str]:
+    """URLs to try, in order: recorded rev, then 00..09, then bare name."""
+    urls = [f"{TEXT_BASE}/{name}-{rev}.txt"] if rev else []
+    for r in (f"{i:02d}" for i in range(10)):
+        if r != rev:
+            urls.append(f"{TEXT_BASE}/{name}-{r}.txt")
+    urls.append(f"{TEXT_BASE}/{name}.txt")
+    return urls
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--all", action="store_true", help="all drafts missing text, not just unrated")
+    ap.add_argument("--limit", type=int, default=0)
+    args = ap.parse_args()
+
+    cfg = Config.load()
+    conn = sqlite3.connect(cfg.db_path)
+    conn.row_factory = sqlite3.Row
+
+    where_unrated = "" if args.all else (
+        "AND NOT EXISTS (SELECT 1 FROM ratings r WHERE r.draft_name = d.name)"
+    )
+    sql = f"""
+        SELECT name, rev FROM drafts d
+        WHERE (full_text IS NULL OR full_text = '')
+          AND source = 'ietf'
+          {where_unrated}
+        ORDER BY name
+    """
+    rows = conn.execute(sql).fetchall()
+    if args.limit:
+        rows = rows[: args.limit]
+
+    print(f"Backfilling text for {len(rows)} drafts...")
+    client = httpx.Client(timeout=30, follow_redirects=True)
+    ok = fail = 0
+    for i, row in enumerate(rows, 1):
+        name, rev = row["name"], row["rev"] or "00"
+        text = None
+        for url in candidate_urls(name, rev):
+            try:
+                resp = client.get(url)
+                if resp.status_code == 200 and len(resp.text) > 200:
+                    text = resp.text[:500_000]  # cap 500K
+                    break
+            except httpx.HTTPError:
+                continue
+        if text:
+            conn.execute("UPDATE drafts SET full_text = ? WHERE name = ?", (text, name))
+            conn.commit()
+            ok += 1
+            print(f"  [{i}/{len(rows)}] OK   {name} ({len(text)} chars)")
+        else:
+            fail += 1
+            print(f"  [{i}/{len(rows)}] FAIL {name}")
+        time.sleep(0.3)
+
+    print(f"\nDone. {ok} downloaded, {fail} failed.")
+    conn.close()
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

src/ietf_analyzer/db.py

@@ -975,7 +975,12 @@ class Database:
     # --- Gaps ---
 
     def insert_gaps(self, gaps: list[dict]) -> None:
-        self.conn.execute("DELETE FROM gaps")  # Replace old analysis
+        # Replace old analysis, but keep any gap still referenced by a generated
+        # proposal (proposal_gaps.gap_id FK) so a refresh never destroys proposal
+        # linkage or trips the foreign-key constraint.
+        self.conn.execute(
+            "DELETE FROM gaps WHERE id NOT IN (SELECT gap_id FROM proposal_gaps)"
+        )
         now = datetime.now(timezone.utc).isoformat()
         for g in gaps:
             self.conn.execute(

src/webui/data/_shared.py

@@ -20,14 +20,17 @@ _CACHE_TTL = 300  # 5 minutes
 
 
 def _extract_month(time_str: str | None) -> str:
-    """Normalize a date string to YYYY-MM format."""
+    """Normalize a date string to YYYY-MM (or "unknown" if unparseable)."""
     if not time_str:
         return "unknown"
-    if len(time_str) >= 7 and time_str[4] == '-':
+    s = time_str.strip()
-        return time_str[:7]  # Already YYYY-MM-DD
+    if len(s) >= 7 and s[4] == '-' and s[5:7].isdigit():
-    if len(time_str) >= 6 and time_str[:4].isdigit():
+        return s[:7]  # ISO 8601: YYYY-MM-DD...
-        return time_str[:4] + '-' + time_str[4:6]  # YYYYMMDD → YYYY-MM
+    if len(s) >= 6 and s[:6].isdigit():
-    return time_str[:7]
+        return s[:4] + '-' + s[4:6]  # Compact: YYYYMMDD → YYYY-MM
+    if len(s) >= 4 and s[:4].isdigit():
+        return s[:4] + '-01'  # Year-only / year-prefixed junk (e.g. "2015/CD Amd 2")
+    return "unknown"
 
 def _cached(key: str, fn, ttl: float = _CACHE_TTL):
     """Return cached result or compute and cache it."""

src/webui/data/analysis.py

@@ -546,6 +546,8 @@ def _compute_timeline_animation_data(db: Database) -> dict:
         r = rating_map[name]
         d = draft_map.get(name)
         month = _extract_month(d.time if d else None)
+        if month == "unknown":
+            continue  # Undated docs (e.g. ISO/ETSI) can't be placed on a temporal animation
         cat = r.categories[0] if r.categories else "Other"
         month_set.add(month)
         category_monthly[month][cat] += 1
@@ -559,6 +561,12 @@ def _compute_timeline_animation_data(db: Database) -> dict:
             "month": month,
         })
 
+    # Deliver points in chronological order so the front-end's cumulative
+    # filter (p.month <= frame) is append-only.  Otherwise new points get
+    # inserted mid-array and Plotly's index-based frame transition animates
+    # existing markers flying to other drafts' coordinates ("jumping points").
+    points.sort(key=lambda p: (p["month"], p["name"]))
+
     months = sorted(month_set)
     # Convert defaultdict to plain dict for JSON
     cat_monthly_plain = {m: dict(cats) for m, cats in category_monthly.items()}

src/webui/templates/timeline.html

@@ -141,9 +141,12 @@ if (points.length > 0 && months.length > 0) {
         };
     });
 
-    // --- Initial plot (first month) ---
+    // --- Initial plot: show the FULL landscape (last frame) so the page
-    const firstTraces = buildTraces(months[0]);
+    //     doesn't open on the near-empty earliest month. Play replays from
-    const firstCount = points.filter(p => p.month <= months[0]).length;
+    //     the start to animate the build-up. ---
+    const initMonth = months[months.length - 1];
+    const initTraces = buildTraces(initMonth);
+    const initCount = points.length;
 
     // Slider steps
     const sliderSteps = months.map(month => ({
@@ -165,7 +168,7 @@ if (points.length > 0 && months.length > 0) {
                 {
                     label: '&#9654; Play',
                     method: 'animate',
-                    args: [null, { frame: { duration: 500, redraw: true }, transition: { duration: 300 }, fromcurrent: true }]
+                    args: [null, { frame: { duration: 500, redraw: true }, transition: { duration: 300 }, fromcurrent: false }]
                 },
                 {
                     label: '&#9724; Pause',
@@ -175,7 +178,7 @@ if (points.length > 0 && months.length > 0) {
             ]
         }],
         sliders: [{
-            active: 0,
+            active: months.length - 1,
             steps: sliderSteps,
             x: 0.05, len: 0.9,
             xanchor: 'left',
@@ -194,13 +197,13 @@ if (points.length > 0 && months.length > 0) {
         }],
     };
 
-    Plotly.newPlot('tsneAnim', firstTraces, layout, CFG).then(() => {
+    Plotly.newPlot('tsneAnim', initTraces, layout, CFG).then(() => {
         Plotly.addFrames('tsneAnim', frames);
     });
 
     // Update badge on animation frame
     const badge = document.querySelector('#monthBadge span');
-    badge.textContent = `${fmtMonth(months[0])} — ${firstCount} drafts`;
+    badge.textContent = `${fmtMonth(initMonth)} — ${initCount} drafts`;
 
     document.getElementById('tsneAnim').on('plotly_animatingframe', function(ev) {
         const month = ev.name;