Compare commits

6 Commits

Author SHA1 Message Date
9d4b70919f chore(webui): remove timeline animation page
Some checks failed
CI / test (3.11) (push) Failing after 9s
CI / test (3.12) (push) Failing after 9s
Observatory Update / update (push) Failing after 2m0s
The animated t-SNE 'embedding landscape' added no real analytical value and
its cumulative-by-month logic was inherently confusing (ancient drafts
appearing late in the animation). Removed entirely rather than maintained.

Drops the /timeline route, template, nav link, data builder
(get_timeline_animation_data / _compute_timeline_animation_data) and its test.
The Overview mini-timeline and /api/timeline (separate features) are untouched.
2026-05-23 21:52:54 +02:00
d11e980a6a fix(webui): timeline opens on full landscape, drop undated points
Some checks failed
CI / test (3.11) (push) Failing after 9s
CI / test (3.12) (push) Failing after 9s
After the chronological-order fix the page opened on the earliest month
(1995-12), showing only a handful of same-colored dots while the legend
listed every category — looked broken.

- Initialise the plot on the LAST frame (full landscape); Play now replays
  the build-up from the start (fromcurrent: false), slider starts at the end.
- Make _extract_month robust: year-only / junk dates (ISO/ETSI 'time' like
  '2015/CD Amd 2', bare '2023', '') no longer yield malformed month labels
  ('2015-/C') or a garbled 'unknown' frame badge.
- Drop undated docs from the temporal animation (they remain on /landscape).

At the full initial frame every category has points, so the legend matches
what is drawn.
2026-05-23 12:41:06 +02:00
c3af38e0f9 fix(webui): timeline embedding landscape — sort points chronologically
The animated t-SNE landscape delivered points in embedding-dict order, not
chronological order. The front-end cumulative filter (p.month <= frame) then
inserted new points mid-array between frames, so Plotly's index-based frame
transition animated existing markers flying to other drafts' coordinates.
Visible symptom: a couple of points jumping around instead of a growing map.

Sorting points by (month, name) makes each frame's per-category marker list
an append-only prefix of the next, so transitions only add markers.
Verified on the live DB: 188 non-append-only frame transitions before, 0 after.
2026-05-23 12:41:06 +02:00
6538cbebaf docs(status): log 2026-05-22 deployment + data refresh
Some checks failed
CI / test (3.11) (push) Failing after 9s
CI / test (3.12) (push) Failing after 9s
2026-05-22 12:28:53 +02:00
b92a756586 fix(db): make gaps --refresh non-destructive (FK-safe insert_gaps)
Some checks failed
CI / test (3.11) (push) Failing after 9s
CI / test (3.12) (push) Failing after 9s
insert_gaps did a blanket 'DELETE FROM gaps', which fails with
'FOREIGN KEY constraint failed' whenever proposal_gaps references a gap
(generated proposals). Delete only gaps not referenced by a proposal so
the refresh preserves proposal linkage and never trips the FK.

Also logs the 2026-05-22 data refresh (761->889 drafts) in dev-journal.
2026-05-22 12:27:47 +02:00
3e8e52ffe3 feat(scripts): backfill full_text for unrated drafts with rev fallback
Some checks failed
CI / test (3.11) (push) Failing after 9s
CI / test (3.12) (push) Failing after 9s
The fetch pipeline left the 128 new drafts (2026-05 delta) without full
text — the per-source text download skipped them. This script downloads
text from the IETF archive using the recorded revision, probing 00-09 and
the bare name as fallback. Run before 'analyze' so rating uses full text.
2026-05-22 11:57:29 +02:00
11 changed files with 143 additions and 351 deletions

View File

@@ -4,6 +4,23 @@
--- ---
### 2026-05-22 SESSION — Data refresh as of today: 761 → 889 drafts (full Sonnet pipeline)
**What**: First full corpus refresh since 2026-03-08. Fetched the delta from Datatracker (128 new drafts, all agent/identity/oauth-token topics), backfilled their full text, then ran the whole pipeline on Sonnet: rate → embed → extract ideas → score novelty → gap analysis → idea embeddings → convergence. Synced the fresh `drafts.db` to the server and restarted the `ietf` container so ietf.nennemann.de serves it.
**Why**: The deployed site was showing March data; the user wanted it current.
**Result** (live API stats): 816 relevant drafts (889 total, 73 false-positives), 722 authors, 973 ideas (avg novelty 2.8), 18 gaps, 170 cross-org convergent ideas (was 132). Tracked token usage ~1.0M in / 472K out on Sonnet.
**Surprise / lessons**:
- The fetch pipeline inserted the 128 new drafts but left all of them **without full text** (the text URL needs the `-NN` revision suffix; the per-source download skipped them). Wrote `scripts/backfill-unrated-text.py` (rev-fallback) to fix — analysis quality depends on full text.
- The shell `ANTHROPIC_API_KEY` env var was **stale (401)**; the valid key was in `.env`. python-dotenv doesn't override an existing env var, so the CLI silently used the bad one. Had to pass the `.env` key explicitly.
- **Bug fixed**: `db.insert_gaps()` did a blanket `DELETE FROM gaps`, which trips the `proposal_gaps.gap_id` FK whenever generated proposals exist (it did — 3 proposals / 7 links). Changed it to delete only gaps not referenced by a proposal, so `gaps --refresh` is non-destructive.
**Cost**: ~$10 tracked (Sonnet). ideas/gaps are dev-only pages, not shown on the production site, but refreshed anyway per user request.
---
### 2026-05-22 SESSION — Deployed the web dashboard at ietf.nennemann.de ### 2026-05-22 SESSION — Deployed the web dashboard at ietf.nennemann.de
**What**: Brought the Flask dashboard online on the nennemann-dev server (Hetzner CAX21) behind Caddy at `https://ietf.nennemann.de`, basic_auth gated (shared `vorschau` preview password), `noindex`. Added an `ietf` Docker service to `nennemann-biz/infra/dev/docker-compose.yml` (build context `/home/dev/repos/research.ietf`, host :8082 -> container :5000, data dir mounted read-write so pageview analytics persist). Container runs in PRODUCTION mode (admin routes 404). **What**: Brought the Flask dashboard online on the nennemann-dev server (Hetzner CAX21) behind Caddy at `https://ietf.nennemann.de`, basic_auth gated (shared `vorschau` preview password), `noindex`. Added an `ietf` Docker service to `nennemann-biz/infra/dev/docker-compose.yml` (build context `/home/dev/repos/research.ietf`, host :8082 -> container :5000, data dir mounted read-write so pageview analytics persist). Container runs in PRODUCTION mode (admin routes 404).

View File

@@ -1,5 +1,21 @@
# research.ietf — Status # research.ietf — Status
## 2026-05-22 — Deployed at ietf.nennemann.de + full data refresh
### What was done
- **Deployed** the Flask dashboard to nennemann-dev behind Caddy: `https://ietf.nennemann.de` (basic_auth `vorschau`, `noindex`, production mode). Docker `dev-ietf-1` on `:8082``:5000`, data dir mounted read-write. Dockerfile fixed to bind `0.0.0.0`. Infra in `nennemann-biz/infra/dev` (compose + Caddy vhost).
- **Data refresh** (first since 2026-03-08): 761→889 drafts. Full Sonnet pipeline (rate/embed/ideas/score/gaps/convergence). Live stats: 816 relevant, 722 authors, 973 ideas, 18 gaps, 170 convergent. Fresh `drafts.db` synced to server.
- New tool: `scripts/backfill-unrated-text.py` (fetch leaves new drafts without full text — rev-suffix issue).
- Bug fix: `db.insert_gaps()` now FK-safe (was a blanket `DELETE FROM gaps`).
### Next / open
- Refresh procedure documented in workspace `docs/services.md` (sync **only** drafts.db; analytics.db is server-owned).
- `ANTHROPIC_API_KEY` shell env was stale — use the key in `.env`.
- Optional: schedule recurring refresh (`scripts/scheduled-update.sh`) if desired.
- 16 drafts yielded no extractable ideas (short/non-technical) — expected, left as-is.
---
## 2026-04-11 — Refimpl -01 hash format fix + draft rebuild ## 2026-04-11 — Refimpl -01 hash format fix + draft rebuild
### What was done ### What was done

View File

@@ -0,0 +1,95 @@
#!/usr/bin/env python3
"""Backfill full_text for unrated drafts that are missing it.
The fetch pipeline sometimes leaves new IETF drafts without full text (the per-source
text download can skip them). Analysis quality depends on full text, so this script
downloads it directly from the IETF archive using the draft's recorded revision, with
a fallback that probes nearby revisions if the recorded one 404s.
Usage:
PYTHONPATH=src python3 scripts/backfill-unrated-text.py [--all] [--limit N]
--all Backfill ALL drafts missing full_text (not just unrated ones).
--limit Cap the number of drafts processed.
"""
from __future__ import annotations
import argparse
import sqlite3
import sys
import time
sys.path.insert(0, "src")
import httpx
from ietf_analyzer.config import Config
TEXT_BASE = "https://www.ietf.org/archive/id"
def candidate_urls(name: str, rev: str) -> list[str]:
"""URLs to try, in order: recorded rev, then 00..09, then bare name."""
urls = [f"{TEXT_BASE}/{name}-{rev}.txt"] if rev else []
for r in (f"{i:02d}" for i in range(10)):
if r != rev:
urls.append(f"{TEXT_BASE}/{name}-{r}.txt")
urls.append(f"{TEXT_BASE}/{name}.txt")
return urls
def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--all", action="store_true", help="all drafts missing text, not just unrated")
ap.add_argument("--limit", type=int, default=0)
args = ap.parse_args()
cfg = Config.load()
conn = sqlite3.connect(cfg.db_path)
conn.row_factory = sqlite3.Row
where_unrated = "" if args.all else (
"AND NOT EXISTS (SELECT 1 FROM ratings r WHERE r.draft_name = d.name)"
)
sql = f"""
SELECT name, rev FROM drafts d
WHERE (full_text IS NULL OR full_text = '')
AND source = 'ietf'
{where_unrated}
ORDER BY name
"""
rows = conn.execute(sql).fetchall()
if args.limit:
rows = rows[: args.limit]
print(f"Backfilling text for {len(rows)} drafts...")
client = httpx.Client(timeout=30, follow_redirects=True)
ok = fail = 0
for i, row in enumerate(rows, 1):
name, rev = row["name"], row["rev"] or "00"
text = None
for url in candidate_urls(name, rev):
try:
resp = client.get(url)
if resp.status_code == 200 and len(resp.text) > 200:
text = resp.text[:500_000] # cap 500K
break
except httpx.HTTPError:
continue
if text:
conn.execute("UPDATE drafts SET full_text = ? WHERE name = ?", (text, name))
conn.commit()
ok += 1
print(f" [{i}/{len(rows)}] OK {name} ({len(text)} chars)")
else:
fail += 1
print(f" [{i}/{len(rows)}] FAIL {name}")
time.sleep(0.3)
print(f"\nDone. {ok} downloaded, {fail} failed.")
conn.close()
return 0
if __name__ == "__main__":
raise SystemExit(main())

View File

@@ -975,7 +975,12 @@ class Database:
# --- Gaps --- # --- Gaps ---
def insert_gaps(self, gaps: list[dict]) -> None: def insert_gaps(self, gaps: list[dict]) -> None:
self.conn.execute("DELETE FROM gaps") # Replace old analysis # Replace old analysis, but keep any gap still referenced by a generated
# proposal (proposal_gaps.gap_id FK) so a refresh never destroys proposal
# linkage or trips the foreign-key constraint.
self.conn.execute(
"DELETE FROM gaps WHERE id NOT IN (SELECT gap_id FROM proposal_gaps)"
)
now = datetime.now(timezone.utc).isoformat() now = datetime.now(timezone.utc).isoformat()
for g in gaps: for g in gaps:
self.conn.execute( self.conn.execute(

View File

@@ -11,7 +11,6 @@ from webui.data import (
get_draft_detail, get_draft_detail,
get_rating_distributions, get_rating_distributions,
get_timeline_data, get_timeline_data,
get_timeline_animation_data,
get_ideas_by_type, get_ideas_by_type,
get_top_authors, get_top_authors,
get_org_data, get_org_data,
@@ -127,12 +126,6 @@ def ratings():
) )
@pages_bp.route("/timeline")
def timeline_animation():
data = get_timeline_animation_data(db())
return render_template("timeline.html", animation=data)
@pages_bp.route("/idea-clusters") @pages_bp.route("/idea-clusters")
def idea_clusters(): def idea_clusters():
data = get_idea_clusters(db()) data = get_idea_clusters(db())

View File

@@ -69,7 +69,6 @@ from webui.data.analysis import ( # noqa: F401
get_timeline_data, get_timeline_data,
get_similarity_graph, get_similarity_graph,
get_idea_clusters, get_idea_clusters,
get_timeline_animation_data,
get_monitor_status, get_monitor_status,
get_citation_graph, get_citation_graph,
get_landscape_tsne, get_landscape_tsne,

View File

@@ -20,14 +20,17 @@ _CACHE_TTL = 300 # 5 minutes
def _extract_month(time_str: str | None) -> str: def _extract_month(time_str: str | None) -> str:
"""Normalize a date string to YYYY-MM format.""" """Normalize a date string to YYYY-MM (or "unknown" if unparseable)."""
if not time_str: if not time_str:
return "unknown" return "unknown"
if len(time_str) >= 7 and time_str[4] == '-': s = time_str.strip()
return time_str[:7] # Already YYYY-MM-DD if len(s) >= 7 and s[4] == '-' and s[5:7].isdigit():
if len(time_str) >= 6 and time_str[:4].isdigit(): return s[:7] # ISO 8601: YYYY-MM-DD...
return time_str[:4] + '-' + time_str[4:6] # YYYYMMDD → YYYY-MM if len(s) >= 6 and s[:6].isdigit():
return time_str[:7] return s[:4] + '-' + s[4:6] # Compact: YYYYMMDD → YYYY-MM
if len(s) >= 4 and s[:4].isdigit():
return s[:4] + '-01' # Year-only / year-prefixed junk (e.g. "2015/CD Amd 2")
return "unknown"
def _cached(key: str, fn, ttl: float = _CACHE_TTL): def _cached(key: str, fn, ttl: float = _CACHE_TTL):
"""Return cached result or compute and cache it.""" """Return cached result or compute and cache it."""

View File

@@ -502,73 +502,6 @@ def _compute_idea_clusters(db: Database) -> dict:
"empty": False, "empty": False,
} }
def get_timeline_animation_data(db: Database) -> dict:
"""Timeline animation (cached for 5 min)."""
return _cached("timeline_animation", lambda: _compute_timeline_animation_data(db))
def _compute_timeline_animation_data(db: Database) -> dict:
"""Compute t-SNE on all drafts, return points with month info + category_monthly.
t-SNE is computed once on ALL drafts so coordinates are stable across
animation frames. Each point carries a ``month`` field (YYYY-MM) so the
front-end can build cumulative animation frames.
"""
embeddings = db.all_embeddings()
if len(embeddings) < 5:
return {"points": [], "months": [], "category_monthly": {}}
pairs = db.drafts_with_ratings(limit=1000)
rating_map = {d.name: r for d, r in pairs}
draft_map = {d.name: d for d, _ in pairs}
# Filter to drafts that have both embeddings and ratings
names = [n for n in embeddings if n in rating_map]
if len(names) < 5:
return {"points": [], "months": [], "category_monthly": {}}
matrix = np.array([embeddings[n] for n in names])
try:
tsne = TSNE(n_components=2, perplexity=min(30, len(names) - 1),
random_state=42, max_iter=500)
coords = tsne.fit_transform(matrix)
except Exception:
return {"points": [], "months": [], "category_monthly": {}}
# Build points with month
points = []
month_set: set[str] = set()
category_monthly: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int))
for i, name in enumerate(names):
r = rating_map[name]
d = draft_map.get(name)
month = _extract_month(d.time if d else None)
cat = r.categories[0] if r.categories else "Other"
month_set.add(month)
category_monthly[month][cat] += 1
points.append({
"name": name,
"title": d.title if d else name,
"x": round(float(coords[i, 0]), 3),
"y": round(float(coords[i, 1]), 3),
"category": cat,
"score": round(r.composite_score, 2),
"month": month,
})
months = sorted(month_set)
# Convert defaultdict to plain dict for JSON
cat_monthly_plain = {m: dict(cats) for m, cats in category_monthly.items()}
return {
"points": points,
"months": months,
"category_monthly": cat_monthly_plain,
}
def get_monitor_status(db: Database) -> MonitorStatus: def get_monitor_status(db: Database) -> MonitorStatus:
"""Return monitoring status data for dashboard.""" """Return monitoring status data for dashboard."""
runs = db.get_monitor_runs(limit=20) runs = db.get_monitor_runs(limit=20)

View File

@@ -156,10 +156,6 @@
Blog Drafts Blog Drafts
</a> </a>
{% endif %} {% endif %}
<a href="/timeline" class="sidebar-link flex items-center gap-3 px-5 py-2.5 text-sm text-slate-300 {{ 'active' if active_page == 'timeline' }}">
<svg class="w-4 h-4 opacity-60" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 8v4l3 3m6-3a9 9 0 11-18 0 9 9 0 0118 0z"/></svg>
Timeline
</a>
{% if is_admin %} {% if is_admin %}
<a href="/trends" class="sidebar-link flex items-center gap-3 px-5 py-2.5 text-sm text-slate-300 {{ 'active' if active_page == 'trends' }}"> <a href="/trends" class="sidebar-link flex items-center gap-3 px-5 py-2.5 text-sm text-slate-300 {{ 'active' if active_page == 'trends' }}">
<svg class="w-4 h-4 opacity-60" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M13 7h8m0 0v8m0-8l-8 8-4-4-6 6"/></svg> <svg class="w-4 h-4 opacity-60" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M13 7h8m0 0v8m0-8l-8 8-4-4-6 6"/></svg>

View File

@@ -1,260 +0,0 @@
{% extends "base.html" %}
{% set active_page = "timeline" %}
{% block title %}Timeline — IETF Draft Analyzer{% endblock %}
{% block extra_head %}<script src="/static/js/plotly.min.js"></script>{% endblock %}
{% block content %}
<div class="mb-6">
<h1 class="text-2xl font-bold text-white">Timeline Animation</h1>
<p class="text-slate-400 text-sm mt-1">Watch the AI/agent draft landscape evolve month by month</p>
</div>
<!-- Stats summary -->
<div class="grid grid-cols-3 gap-4 mb-6" id="statCards">
</div>
<!-- Animated t-SNE map -->
<div class="bg-slate-900 rounded-xl border border-slate-800 p-5 mb-6" id="tsneSection">
<h2 class="text-sm font-semibold text-slate-300 mb-1">Animated Embedding Landscape</h2>
<p class="text-xs text-slate-500 mb-3">t-SNE projection with cumulative drafts per month. Color = category, size = composite score. Press Play to animate.</p>
<div id="monthBadge" class="text-center mb-2">
<span class="inline-block bg-slate-800 border border-slate-700 rounded-lg px-4 py-1.5 text-sm font-mono text-blue-400"></span>
</div>
<div id="tsneAnim" style="height: 560px;"></div>
</div>
<!-- Stacked area chart -->
<div class="bg-slate-900 rounded-xl border border-slate-800 p-5 mb-6">
<h2 class="text-sm font-semibold text-slate-300 mb-1">Category Submissions Over Time</h2>
<p class="text-xs text-slate-500 mb-3">Stacked area chart showing draft submissions by category per month.</p>
<div id="stackedArea" style="height: 400px;"></div>
</div>
{% endblock %}
{% block extra_scripts %}
<script>
const PLOTLY_LAYOUT = {
paper_bgcolor: 'transparent', plot_bgcolor: 'rgba(15,23,42,0.5)',
font: { color: '#94a3b8', family: 'Inter, system-ui, sans-serif', size: 12 },
margin: { t: 20, r: 20, b: 50, l: 50 },
xaxis: { gridcolor: '#1e293b', zerolinecolor: '#334155' },
yaxis: { gridcolor: '#1e293b', zerolinecolor: '#334155' },
};
const CFG = { responsive: true, displayModeBar: false };
const PALETTE = [
'#3b82f6', '#ef4444', '#22c55e', '#a855f7', '#f59e0b',
'#06b6d4', '#ec4899', '#84cc16', '#f97316', '#8b5cf6',
'#14b8a6', '#e11d48', '#64748b', '#eab308', '#6366f1',
];
const animData = {{ animation | tojson }};
const points = animData.points;
const months = animData.months;
const catMonthly = animData.category_monthly;
const MONTH_NAMES = ['Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec'];
function fmtMonth(ym) {
if (!ym) return ym;
let y, m;
if (ym.includes('-')) {
[y, m] = ym.split('-');
} else if (ym.length >= 6) {
y = ym.slice(0, 4);
m = ym.slice(4, 6);
} else {
return ym;
}
const mi = parseInt(m, 10) - 1;
return (MONTH_NAMES[mi] || m) + ' ' + y;
}
if (points.length > 0 && months.length > 0) {
// --- Stat cards ---
const firstMonth = months[0];
const lastMonth = months[months.length - 1];
const allCats = [...new Set(points.map(p => p.category))];
document.getElementById('statCards').innerHTML = `
<div class="stat-card rounded-xl border border-slate-800 p-5 relative overflow-hidden">
<div class="absolute top-0 left-0 w-full h-1 bg-gradient-to-r from-blue-500 to-blue-400"></div>
<div class="text-3xl font-bold text-blue-400">${months.length}</div>
<div class="text-xs text-slate-400 mt-1 uppercase tracking-wider">Months Span</div>
<div class="text-xs text-slate-500 mt-0.5">${fmtMonth(firstMonth)} ${fmtMonth(lastMonth)}</div>
</div>
<div class="stat-card rounded-xl border border-slate-800 p-5 relative overflow-hidden">
<div class="absolute top-0 left-0 w-full h-1 bg-gradient-to-r from-emerald-500 to-emerald-400"></div>
<div class="text-3xl font-bold text-emerald-400">${points.length}</div>
<div class="text-xs text-slate-400 mt-1 uppercase tracking-wider">Total Drafts</div>
</div>
<div class="stat-card rounded-xl border border-slate-800 p-5 relative overflow-hidden">
<div class="absolute top-0 left-0 w-full h-1 bg-gradient-to-r from-purple-500 to-purple-400"></div>
<div class="text-3xl font-bold text-purple-400">${allCats.length}</div>
<div class="text-xs text-slate-400 mt-1 uppercase tracking-wider">Categories</div>
</div>
`;
// --- Build category list sorted by frequency ---
const catCounts = {};
points.forEach(p => { catCounts[p.category] = (catCounts[p.category] || 0) + 1; });
const catList = Object.keys(catCounts).sort((a, b) => catCounts[b] - catCounts[a]);
const catColor = {};
catList.forEach((c, i) => { catColor[c] = PALETTE[i % PALETTE.length]; });
// --- Helper: build traces for points up to a given month ---
function buildTraces(upToMonth) {
const filtered = points.filter(p => p.month <= upToMonth);
const groups = {};
filtered.forEach(p => {
if (!groups[p.category]) groups[p.category] = { x: [], y: [], size: [], text: [], names: [] };
groups[p.category].x.push(p.x);
groups[p.category].y.push(p.y);
groups[p.category].size.push(Math.max(p.score * 4, 6));
groups[p.category].text.push(p.title);
groups[p.category].names.push(p.name);
});
return catList.map(cat => {
const g = groups[cat] || { x: [], y: [], size: [], text: [], names: [] };
return {
x: g.x, y: g.y, text: g.text, name: cat,
customdata: g.names,
mode: 'markers', type: 'scatter',
marker: {
size: g.size,
color: catColor[cat],
opacity: 0.8,
line: { width: 0.5, color: 'rgba(255,255,255,0.15)' },
},
hovertemplate: '<b>%{text}</b><extra>' + cat + '</extra>',
};
});
}
// --- Build frames ---
const frames = months.map(month => {
const cumCount = points.filter(p => p.month <= month).length;
return {
name: month,
data: buildTraces(month),
};
});
// --- Initial plot (first month) ---
const firstTraces = buildTraces(months[0]);
const firstCount = points.filter(p => p.month <= months[0]).length;
// Slider steps
const sliderSteps = months.map(month => ({
method: 'animate',
label: fmtMonth(month),
args: [[month], { frame: { duration: 500, redraw: true }, transition: { duration: 300 }, mode: 'immediate' }],
}));
const layout = {
...PLOTLY_LAYOUT,
xaxis: { visible: false, showgrid: false, zeroline: false },
yaxis: { visible: false, showgrid: false, zeroline: false },
legend: { font: { size: 10, color: '#94a3b8' }, bgcolor: 'transparent' },
hovermode: 'closest',
margin: { t: 40, r: 20, b: 60, l: 20 },
updatemenus: [{
type: 'buttons', showactive: false, x: 0.05, y: 1.08,
buttons: [
{
label: '&#9654; Play',
method: 'animate',
args: [null, { frame: { duration: 500, redraw: true }, transition: { duration: 300 }, fromcurrent: true }]
},
{
label: '&#9724; Pause',
method: 'animate',
args: [[null], { frame: { duration: 0, redraw: true }, mode: 'immediate' }]
}
]
}],
sliders: [{
active: 0,
steps: sliderSteps,
x: 0.05, len: 0.9,
xanchor: 'left',
y: -0.02,
yanchor: 'top',
pad: { t: 30, b: 10 },
currentvalue: { visible: false },
transition: { duration: 300 },
font: { size: 9, color: '#64748b' },
bgcolor: '#1e293b',
activebgcolor: '#3b82f6',
bordercolor: '#334155',
borderwidth: 1,
ticklen: 4,
tickcolor: '#475569',
}],
};
Plotly.newPlot('tsneAnim', firstTraces, layout, CFG).then(() => {
Plotly.addFrames('tsneAnim', frames);
});
// Update badge on animation frame
const badge = document.querySelector('#monthBadge span');
badge.textContent = `${fmtMonth(months[0])}${firstCount} drafts`;
document.getElementById('tsneAnim').on('plotly_animatingframe', function(ev) {
const month = ev.name;
const cumCount = points.filter(p => p.month <= month).length;
badge.textContent = `${fmtMonth(month)}${cumCount} drafts`;
});
// Click to navigate
document.getElementById('tsneAnim').on('plotly_click', function(data) {
const pt = data.points[0];
if (pt.customdata) {
window.location.href = '/drafts/' + pt.customdata;
}
});
// --- Stacked area chart ---
// Collect all categories across all months
const areaCats = {};
Object.values(catMonthly).forEach(mc => {
Object.keys(mc).forEach(c => { areaCats[c] = true; });
});
// Sort by total count
const areaCatList = Object.keys(areaCats).sort((a, b) => {
const totalA = months.reduce((s, m) => s + ((catMonthly[m] || {})[a] || 0), 0);
const totalB = months.reduce((s, m) => s + ((catMonthly[m] || {})[b] || 0), 0);
return totalB - totalA;
});
const monthLabels = months.map(fmtMonth);
const areaTraces = areaCatList.map((cat, i) => ({
x: monthLabels,
y: months.map(m => (catMonthly[m] || {})[cat] || 0),
name: cat,
type: 'scatter',
mode: 'lines',
stackgroup: 'one',
line: { width: 0.5, color: catColor[cat] || PALETTE[i % PALETTE.length] },
fillcolor: (catColor[cat] || PALETTE[i % PALETTE.length]) + '80',
hovertemplate: '%{x}<br>' + cat + ': %{y}<extra></extra>',
}));
Plotly.newPlot('stackedArea', areaTraces, {
...PLOTLY_LAYOUT,
xaxis: { ...PLOTLY_LAYOUT.xaxis, title: { text: 'Month', font: { size: 11 } } },
yaxis: { ...PLOTLY_LAYOUT.yaxis, title: { text: 'Drafts', font: { size: 11 } } },
legend: { font: { size: 10, color: '#94a3b8' }, orientation: 'h', y: -0.25, x: 0.5, xanchor: 'center' },
hovermode: 'x unified',
margin: { t: 20, r: 20, b: 80, l: 50 },
}, CFG);
} else {
document.getElementById('tsneSection').innerHTML = '<p class="text-slate-500 text-sm text-center py-20">No timeline animation data available. Run the analysis pipeline first.</p>';
document.getElementById('stackedArea').innerHTML = '<p class="text-slate-500 text-sm text-center py-20">No data available.</p>';
document.getElementById('statCards').style.display = 'none';
}
</script>
{% endblock %}

View File

@@ -61,11 +61,6 @@ def test_authors_page(client):
assert resp.status_code == 200 assert resp.status_code == 200
def test_timeline_page(client):
resp = client.get("/timeline")
assert resp.status_code == 200
def test_search_page_empty(client): def test_search_page_empty(client):
resp = client.get("/search") resp = client.get("/search")
assert resp.status_code == 200 assert resp.status_code == 200