Compare commits
6 Commits
fix/timeli
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 9d4b70919f | |||
| d11e980a6a | |||
| c3af38e0f9 | |||
| 6538cbebaf | |||
| b92a756586 | |||
| 3e8e52ffe3 |
@@ -4,6 +4,23 @@
|
||||
|
||||
---
|
||||
|
||||
### 2026-05-22 SESSION — Data refresh as of today: 761 → 889 drafts (full Sonnet pipeline)
|
||||
|
||||
**What**: First full corpus refresh since 2026-03-08. Fetched the delta from Datatracker (128 new drafts, all agent/identity/oauth-token topics), backfilled their full text, then ran the whole pipeline on Sonnet: rate → embed → extract ideas → score novelty → gap analysis → idea embeddings → convergence. Synced the fresh `drafts.db` to the server and restarted the `ietf` container so ietf.nennemann.de serves it.
|
||||
|
||||
**Why**: The deployed site was showing March data; the user wanted it current.
|
||||
|
||||
**Result** (live API stats): 816 relevant drafts (889 total, 73 false-positives), 722 authors, 973 ideas (avg novelty 2.8), 18 gaps, 170 cross-org convergent ideas (was 132). Tracked token usage ~1.0M in / 472K out on Sonnet.
|
||||
|
||||
**Surprise / lessons**:
|
||||
- The fetch pipeline inserted the 128 new drafts but left all of them **without full text** (the text URL needs the `-NN` revision suffix; the per-source download skipped them). Wrote `scripts/backfill-unrated-text.py` (rev-fallback) to fix — analysis quality depends on full text.
|
||||
- The shell `ANTHROPIC_API_KEY` env var was **stale (401)**; the valid key was in `.env`. python-dotenv doesn't override an existing env var, so the CLI silently used the bad one. Had to pass the `.env` key explicitly.
|
||||
- **Bug fixed**: `db.insert_gaps()` did a blanket `DELETE FROM gaps`, which trips the `proposal_gaps.gap_id` FK whenever generated proposals exist (it did — 3 proposals / 7 links). Changed it to delete only gaps not referenced by a proposal, so `gaps --refresh` is non-destructive.
|
||||
|
||||
**Cost**: ~$10 tracked (Sonnet). ideas/gaps are dev-only pages, not shown on the production site, but refreshed anyway per user request.
|
||||
|
||||
---
|
||||
|
||||
### 2026-05-22 SESSION — Deployed the web dashboard at ietf.nennemann.de
|
||||
|
||||
**What**: Brought the Flask dashboard online on the nennemann-dev server (Hetzner CAX21) behind Caddy at `https://ietf.nennemann.de`, basic_auth gated (shared `vorschau` preview password), `noindex`. Added an `ietf` Docker service to `nennemann-biz/infra/dev/docker-compose.yml` (build context `/home/dev/repos/research.ietf`, host :8082 -> container :5000, data dir mounted read-write so pageview analytics persist). Container runs in PRODUCTION mode (admin routes 404).
|
||||
|
||||
@@ -1,5 +1,21 @@
|
||||
# research.ietf — Status
|
||||
|
||||
## 2026-05-22 — Deployed at ietf.nennemann.de + full data refresh
|
||||
|
||||
### What was done
|
||||
- **Deployed** the Flask dashboard to nennemann-dev behind Caddy: `https://ietf.nennemann.de` (basic_auth `vorschau`, `noindex`, production mode). Docker `dev-ietf-1` on `:8082`→`:5000`, data dir mounted read-write. Dockerfile fixed to bind `0.0.0.0`. Infra in `nennemann-biz/infra/dev` (compose + Caddy vhost).
|
||||
- **Data refresh** (first since 2026-03-08): 761→889 drafts. Full Sonnet pipeline (rate/embed/ideas/score/gaps/convergence). Live stats: 816 relevant, 722 authors, 973 ideas, 18 gaps, 170 convergent. Fresh `drafts.db` synced to server.
|
||||
- New tool: `scripts/backfill-unrated-text.py` (fetch leaves new drafts without full text — rev-suffix issue).
|
||||
- Bug fix: `db.insert_gaps()` now FK-safe (was a blanket `DELETE FROM gaps`).
|
||||
|
||||
### Next / open
|
||||
- Refresh procedure documented in workspace `docs/services.md` (sync **only** drafts.db; analytics.db is server-owned).
|
||||
- `ANTHROPIC_API_KEY` shell env was stale — use the key in `.env`.
|
||||
- Optional: schedule recurring refresh (`scripts/scheduled-update.sh`) if desired.
|
||||
- 16 drafts yielded no extractable ideas (short/non-technical) — expected, left as-is.
|
||||
|
||||
---
|
||||
|
||||
## 2026-04-11 — Refimpl -01 hash format fix + draft rebuild
|
||||
|
||||
### What was done
|
||||
|
||||
95
scripts/backfill-unrated-text.py
Normal file
95
scripts/backfill-unrated-text.py
Normal file
@@ -0,0 +1,95 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Backfill full_text for unrated drafts that are missing it.
|
||||
|
||||
The fetch pipeline sometimes leaves new IETF drafts without full text (the per-source
|
||||
text download can skip them). Analysis quality depends on full text, so this script
|
||||
downloads it directly from the IETF archive using the draft's recorded revision, with
|
||||
a fallback that probes nearby revisions if the recorded one 404s.
|
||||
|
||||
Usage:
|
||||
PYTHONPATH=src python3 scripts/backfill-unrated-text.py [--all] [--limit N]
|
||||
|
||||
--all Backfill ALL drafts missing full_text (not just unrated ones).
|
||||
--limit Cap the number of drafts processed.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import sqlite3
|
||||
import sys
|
||||
import time
|
||||
|
||||
sys.path.insert(0, "src")
|
||||
|
||||
import httpx
|
||||
|
||||
from ietf_analyzer.config import Config
|
||||
|
||||
TEXT_BASE = "https://www.ietf.org/archive/id"
|
||||
|
||||
|
||||
def candidate_urls(name: str, rev: str) -> list[str]:
|
||||
"""URLs to try, in order: recorded rev, then 00..09, then bare name."""
|
||||
urls = [f"{TEXT_BASE}/{name}-{rev}.txt"] if rev else []
|
||||
for r in (f"{i:02d}" for i in range(10)):
|
||||
if r != rev:
|
||||
urls.append(f"{TEXT_BASE}/{name}-{r}.txt")
|
||||
urls.append(f"{TEXT_BASE}/{name}.txt")
|
||||
return urls
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--all", action="store_true", help="all drafts missing text, not just unrated")
|
||||
ap.add_argument("--limit", type=int, default=0)
|
||||
args = ap.parse_args()
|
||||
|
||||
cfg = Config.load()
|
||||
conn = sqlite3.connect(cfg.db_path)
|
||||
conn.row_factory = sqlite3.Row
|
||||
|
||||
where_unrated = "" if args.all else (
|
||||
"AND NOT EXISTS (SELECT 1 FROM ratings r WHERE r.draft_name = d.name)"
|
||||
)
|
||||
sql = f"""
|
||||
SELECT name, rev FROM drafts d
|
||||
WHERE (full_text IS NULL OR full_text = '')
|
||||
AND source = 'ietf'
|
||||
{where_unrated}
|
||||
ORDER BY name
|
||||
"""
|
||||
rows = conn.execute(sql).fetchall()
|
||||
if args.limit:
|
||||
rows = rows[: args.limit]
|
||||
|
||||
print(f"Backfilling text for {len(rows)} drafts...")
|
||||
client = httpx.Client(timeout=30, follow_redirects=True)
|
||||
ok = fail = 0
|
||||
for i, row in enumerate(rows, 1):
|
||||
name, rev = row["name"], row["rev"] or "00"
|
||||
text = None
|
||||
for url in candidate_urls(name, rev):
|
||||
try:
|
||||
resp = client.get(url)
|
||||
if resp.status_code == 200 and len(resp.text) > 200:
|
||||
text = resp.text[:500_000] # cap 500K
|
||||
break
|
||||
except httpx.HTTPError:
|
||||
continue
|
||||
if text:
|
||||
conn.execute("UPDATE drafts SET full_text = ? WHERE name = ?", (text, name))
|
||||
conn.commit()
|
||||
ok += 1
|
||||
print(f" [{i}/{len(rows)}] OK {name} ({len(text)} chars)")
|
||||
else:
|
||||
fail += 1
|
||||
print(f" [{i}/{len(rows)}] FAIL {name}")
|
||||
time.sleep(0.3)
|
||||
|
||||
print(f"\nDone. {ok} downloaded, {fail} failed.")
|
||||
conn.close()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -975,7 +975,12 @@ class Database:
|
||||
# --- Gaps ---
|
||||
|
||||
def insert_gaps(self, gaps: list[dict]) -> None:
|
||||
self.conn.execute("DELETE FROM gaps") # Replace old analysis
|
||||
# Replace old analysis, but keep any gap still referenced by a generated
|
||||
# proposal (proposal_gaps.gap_id FK) so a refresh never destroys proposal
|
||||
# linkage or trips the foreign-key constraint.
|
||||
self.conn.execute(
|
||||
"DELETE FROM gaps WHERE id NOT IN (SELECT gap_id FROM proposal_gaps)"
|
||||
)
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
for g in gaps:
|
||||
self.conn.execute(
|
||||
|
||||
@@ -11,7 +11,6 @@ from webui.data import (
|
||||
get_draft_detail,
|
||||
get_rating_distributions,
|
||||
get_timeline_data,
|
||||
get_timeline_animation_data,
|
||||
get_ideas_by_type,
|
||||
get_top_authors,
|
||||
get_org_data,
|
||||
@@ -127,12 +126,6 @@ def ratings():
|
||||
)
|
||||
|
||||
|
||||
@pages_bp.route("/timeline")
|
||||
def timeline_animation():
|
||||
data = get_timeline_animation_data(db())
|
||||
return render_template("timeline.html", animation=data)
|
||||
|
||||
|
||||
@pages_bp.route("/idea-clusters")
|
||||
def idea_clusters():
|
||||
data = get_idea_clusters(db())
|
||||
|
||||
@@ -69,7 +69,6 @@ from webui.data.analysis import ( # noqa: F401
|
||||
get_timeline_data,
|
||||
get_similarity_graph,
|
||||
get_idea_clusters,
|
||||
get_timeline_animation_data,
|
||||
get_monitor_status,
|
||||
get_citation_graph,
|
||||
get_landscape_tsne,
|
||||
|
||||
@@ -20,14 +20,17 @@ _CACHE_TTL = 300 # 5 minutes
|
||||
|
||||
|
||||
def _extract_month(time_str: str | None) -> str:
|
||||
"""Normalize a date string to YYYY-MM format."""
|
||||
"""Normalize a date string to YYYY-MM (or "unknown" if unparseable)."""
|
||||
if not time_str:
|
||||
return "unknown"
|
||||
if len(time_str) >= 7 and time_str[4] == '-':
|
||||
return time_str[:7] # Already YYYY-MM-DD
|
||||
if len(time_str) >= 6 and time_str[:4].isdigit():
|
||||
return time_str[:4] + '-' + time_str[4:6] # YYYYMMDD → YYYY-MM
|
||||
return time_str[:7]
|
||||
s = time_str.strip()
|
||||
if len(s) >= 7 and s[4] == '-' and s[5:7].isdigit():
|
||||
return s[:7] # ISO 8601: YYYY-MM-DD...
|
||||
if len(s) >= 6 and s[:6].isdigit():
|
||||
return s[:4] + '-' + s[4:6] # Compact: YYYYMMDD → YYYY-MM
|
||||
if len(s) >= 4 and s[:4].isdigit():
|
||||
return s[:4] + '-01' # Year-only / year-prefixed junk (e.g. "2015/CD Amd 2")
|
||||
return "unknown"
|
||||
|
||||
def _cached(key: str, fn, ttl: float = _CACHE_TTL):
|
||||
"""Return cached result or compute and cache it."""
|
||||
|
||||
@@ -502,73 +502,6 @@ def _compute_idea_clusters(db: Database) -> dict:
|
||||
"empty": False,
|
||||
}
|
||||
|
||||
def get_timeline_animation_data(db: Database) -> dict:
|
||||
"""Timeline animation (cached for 5 min)."""
|
||||
return _cached("timeline_animation", lambda: _compute_timeline_animation_data(db))
|
||||
|
||||
def _compute_timeline_animation_data(db: Database) -> dict:
|
||||
"""Compute t-SNE on all drafts, return points with month info + category_monthly.
|
||||
|
||||
t-SNE is computed once on ALL drafts so coordinates are stable across
|
||||
animation frames. Each point carries a ``month`` field (YYYY-MM) so the
|
||||
front-end can build cumulative animation frames.
|
||||
"""
|
||||
|
||||
|
||||
embeddings = db.all_embeddings()
|
||||
if len(embeddings) < 5:
|
||||
return {"points": [], "months": [], "category_monthly": {}}
|
||||
|
||||
pairs = db.drafts_with_ratings(limit=1000)
|
||||
rating_map = {d.name: r for d, r in pairs}
|
||||
draft_map = {d.name: d for d, _ in pairs}
|
||||
|
||||
# Filter to drafts that have both embeddings and ratings
|
||||
names = [n for n in embeddings if n in rating_map]
|
||||
if len(names) < 5:
|
||||
return {"points": [], "months": [], "category_monthly": {}}
|
||||
|
||||
matrix = np.array([embeddings[n] for n in names])
|
||||
|
||||
try:
|
||||
tsne = TSNE(n_components=2, perplexity=min(30, len(names) - 1),
|
||||
random_state=42, max_iter=500)
|
||||
coords = tsne.fit_transform(matrix)
|
||||
except Exception:
|
||||
return {"points": [], "months": [], "category_monthly": {}}
|
||||
|
||||
# Build points with month
|
||||
points = []
|
||||
month_set: set[str] = set()
|
||||
category_monthly: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int))
|
||||
|
||||
for i, name in enumerate(names):
|
||||
r = rating_map[name]
|
||||
d = draft_map.get(name)
|
||||
month = _extract_month(d.time if d else None)
|
||||
cat = r.categories[0] if r.categories else "Other"
|
||||
month_set.add(month)
|
||||
category_monthly[month][cat] += 1
|
||||
points.append({
|
||||
"name": name,
|
||||
"title": d.title if d else name,
|
||||
"x": round(float(coords[i, 0]), 3),
|
||||
"y": round(float(coords[i, 1]), 3),
|
||||
"category": cat,
|
||||
"score": round(r.composite_score, 2),
|
||||
"month": month,
|
||||
})
|
||||
|
||||
months = sorted(month_set)
|
||||
# Convert defaultdict to plain dict for JSON
|
||||
cat_monthly_plain = {m: dict(cats) for m, cats in category_monthly.items()}
|
||||
|
||||
return {
|
||||
"points": points,
|
||||
"months": months,
|
||||
"category_monthly": cat_monthly_plain,
|
||||
}
|
||||
|
||||
def get_monitor_status(db: Database) -> MonitorStatus:
|
||||
"""Return monitoring status data for dashboard."""
|
||||
runs = db.get_monitor_runs(limit=20)
|
||||
|
||||
@@ -156,10 +156,6 @@
|
||||
Blog Drafts
|
||||
</a>
|
||||
{% endif %}
|
||||
<a href="/timeline" class="sidebar-link flex items-center gap-3 px-5 py-2.5 text-sm text-slate-300 {{ 'active' if active_page == 'timeline' }}">
|
||||
<svg class="w-4 h-4 opacity-60" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 8v4l3 3m6-3a9 9 0 11-18 0 9 9 0 0118 0z"/></svg>
|
||||
Timeline
|
||||
</a>
|
||||
{% if is_admin %}
|
||||
<a href="/trends" class="sidebar-link flex items-center gap-3 px-5 py-2.5 text-sm text-slate-300 {{ 'active' if active_page == 'trends' }}">
|
||||
<svg class="w-4 h-4 opacity-60" fill="none" stroke="currentColor" viewBox="0 0 24 24"><path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M13 7h8m0 0v8m0-8l-8 8-4-4-6 6"/></svg>
|
||||
|
||||
@@ -1,260 +0,0 @@
|
||||
{% extends "base.html" %}
|
||||
{% set active_page = "timeline" %}
|
||||
|
||||
{% block title %}Timeline — IETF Draft Analyzer{% endblock %}
|
||||
|
||||
{% block extra_head %}<script src="/static/js/plotly.min.js"></script>{% endblock %}
|
||||
|
||||
{% block content %}
|
||||
<div class="mb-6">
|
||||
<h1 class="text-2xl font-bold text-white">Timeline Animation</h1>
|
||||
<p class="text-slate-400 text-sm mt-1">Watch the AI/agent draft landscape evolve month by month</p>
|
||||
</div>
|
||||
|
||||
<!-- Stats summary -->
|
||||
<div class="grid grid-cols-3 gap-4 mb-6" id="statCards">
|
||||
</div>
|
||||
|
||||
<!-- Animated t-SNE map -->
|
||||
<div class="bg-slate-900 rounded-xl border border-slate-800 p-5 mb-6" id="tsneSection">
|
||||
<h2 class="text-sm font-semibold text-slate-300 mb-1">Animated Embedding Landscape</h2>
|
||||
<p class="text-xs text-slate-500 mb-3">t-SNE projection with cumulative drafts per month. Color = category, size = composite score. Press Play to animate.</p>
|
||||
<div id="monthBadge" class="text-center mb-2">
|
||||
<span class="inline-block bg-slate-800 border border-slate-700 rounded-lg px-4 py-1.5 text-sm font-mono text-blue-400"></span>
|
||||
</div>
|
||||
<div id="tsneAnim" style="height: 560px;"></div>
|
||||
</div>
|
||||
|
||||
<!-- Stacked area chart -->
|
||||
<div class="bg-slate-900 rounded-xl border border-slate-800 p-5 mb-6">
|
||||
<h2 class="text-sm font-semibold text-slate-300 mb-1">Category Submissions Over Time</h2>
|
||||
<p class="text-xs text-slate-500 mb-3">Stacked area chart showing draft submissions by category per month.</p>
|
||||
<div id="stackedArea" style="height: 400px;"></div>
|
||||
</div>
|
||||
{% endblock %}
|
||||
|
||||
{% block extra_scripts %}
|
||||
<script>
|
||||
const PLOTLY_LAYOUT = {
|
||||
paper_bgcolor: 'transparent', plot_bgcolor: 'rgba(15,23,42,0.5)',
|
||||
font: { color: '#94a3b8', family: 'Inter, system-ui, sans-serif', size: 12 },
|
||||
margin: { t: 20, r: 20, b: 50, l: 50 },
|
||||
xaxis: { gridcolor: '#1e293b', zerolinecolor: '#334155' },
|
||||
yaxis: { gridcolor: '#1e293b', zerolinecolor: '#334155' },
|
||||
};
|
||||
const CFG = { responsive: true, displayModeBar: false };
|
||||
|
||||
const PALETTE = [
|
||||
'#3b82f6', '#ef4444', '#22c55e', '#a855f7', '#f59e0b',
|
||||
'#06b6d4', '#ec4899', '#84cc16', '#f97316', '#8b5cf6',
|
||||
'#14b8a6', '#e11d48', '#64748b', '#eab308', '#6366f1',
|
||||
];
|
||||
|
||||
const animData = {{ animation | tojson }};
|
||||
const points = animData.points;
|
||||
const months = animData.months;
|
||||
const catMonthly = animData.category_monthly;
|
||||
|
||||
const MONTH_NAMES = ['Jan','Feb','Mar','Apr','May','Jun','Jul','Aug','Sep','Oct','Nov','Dec'];
|
||||
function fmtMonth(ym) {
|
||||
if (!ym) return ym;
|
||||
let y, m;
|
||||
if (ym.includes('-')) {
|
||||
[y, m] = ym.split('-');
|
||||
} else if (ym.length >= 6) {
|
||||
y = ym.slice(0, 4);
|
||||
m = ym.slice(4, 6);
|
||||
} else {
|
||||
return ym;
|
||||
}
|
||||
const mi = parseInt(m, 10) - 1;
|
||||
return (MONTH_NAMES[mi] || m) + ' ' + y;
|
||||
}
|
||||
|
||||
if (points.length > 0 && months.length > 0) {
|
||||
|
||||
// --- Stat cards ---
|
||||
const firstMonth = months[0];
|
||||
const lastMonth = months[months.length - 1];
|
||||
const allCats = [...new Set(points.map(p => p.category))];
|
||||
document.getElementById('statCards').innerHTML = `
|
||||
<div class="stat-card rounded-xl border border-slate-800 p-5 relative overflow-hidden">
|
||||
<div class="absolute top-0 left-0 w-full h-1 bg-gradient-to-r from-blue-500 to-blue-400"></div>
|
||||
<div class="text-3xl font-bold text-blue-400">${months.length}</div>
|
||||
<div class="text-xs text-slate-400 mt-1 uppercase tracking-wider">Months Span</div>
|
||||
<div class="text-xs text-slate-500 mt-0.5">${fmtMonth(firstMonth)} – ${fmtMonth(lastMonth)}</div>
|
||||
</div>
|
||||
<div class="stat-card rounded-xl border border-slate-800 p-5 relative overflow-hidden">
|
||||
<div class="absolute top-0 left-0 w-full h-1 bg-gradient-to-r from-emerald-500 to-emerald-400"></div>
|
||||
<div class="text-3xl font-bold text-emerald-400">${points.length}</div>
|
||||
<div class="text-xs text-slate-400 mt-1 uppercase tracking-wider">Total Drafts</div>
|
||||
</div>
|
||||
<div class="stat-card rounded-xl border border-slate-800 p-5 relative overflow-hidden">
|
||||
<div class="absolute top-0 left-0 w-full h-1 bg-gradient-to-r from-purple-500 to-purple-400"></div>
|
||||
<div class="text-3xl font-bold text-purple-400">${allCats.length}</div>
|
||||
<div class="text-xs text-slate-400 mt-1 uppercase tracking-wider">Categories</div>
|
||||
</div>
|
||||
`;
|
||||
|
||||
// --- Build category list sorted by frequency ---
|
||||
const catCounts = {};
|
||||
points.forEach(p => { catCounts[p.category] = (catCounts[p.category] || 0) + 1; });
|
||||
const catList = Object.keys(catCounts).sort((a, b) => catCounts[b] - catCounts[a]);
|
||||
const catColor = {};
|
||||
catList.forEach((c, i) => { catColor[c] = PALETTE[i % PALETTE.length]; });
|
||||
|
||||
// --- Helper: build traces for points up to a given month ---
|
||||
function buildTraces(upToMonth) {
|
||||
const filtered = points.filter(p => p.month <= upToMonth);
|
||||
const groups = {};
|
||||
filtered.forEach(p => {
|
||||
if (!groups[p.category]) groups[p.category] = { x: [], y: [], size: [], text: [], names: [] };
|
||||
groups[p.category].x.push(p.x);
|
||||
groups[p.category].y.push(p.y);
|
||||
groups[p.category].size.push(Math.max(p.score * 4, 6));
|
||||
groups[p.category].text.push(p.title);
|
||||
groups[p.category].names.push(p.name);
|
||||
});
|
||||
return catList.map(cat => {
|
||||
const g = groups[cat] || { x: [], y: [], size: [], text: [], names: [] };
|
||||
return {
|
||||
x: g.x, y: g.y, text: g.text, name: cat,
|
||||
customdata: g.names,
|
||||
mode: 'markers', type: 'scatter',
|
||||
marker: {
|
||||
size: g.size,
|
||||
color: catColor[cat],
|
||||
opacity: 0.8,
|
||||
line: { width: 0.5, color: 'rgba(255,255,255,0.15)' },
|
||||
},
|
||||
hovertemplate: '<b>%{text}</b><extra>' + cat + '</extra>',
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
// --- Build frames ---
|
||||
const frames = months.map(month => {
|
||||
const cumCount = points.filter(p => p.month <= month).length;
|
||||
return {
|
||||
name: month,
|
||||
data: buildTraces(month),
|
||||
};
|
||||
});
|
||||
|
||||
// --- Initial plot (first month) ---
|
||||
const firstTraces = buildTraces(months[0]);
|
||||
const firstCount = points.filter(p => p.month <= months[0]).length;
|
||||
|
||||
// Slider steps
|
||||
const sliderSteps = months.map(month => ({
|
||||
method: 'animate',
|
||||
label: fmtMonth(month),
|
||||
args: [[month], { frame: { duration: 500, redraw: true }, transition: { duration: 300 }, mode: 'immediate' }],
|
||||
}));
|
||||
|
||||
const layout = {
|
||||
...PLOTLY_LAYOUT,
|
||||
xaxis: { visible: false, showgrid: false, zeroline: false },
|
||||
yaxis: { visible: false, showgrid: false, zeroline: false },
|
||||
legend: { font: { size: 10, color: '#94a3b8' }, bgcolor: 'transparent' },
|
||||
hovermode: 'closest',
|
||||
margin: { t: 40, r: 20, b: 60, l: 20 },
|
||||
updatemenus: [{
|
||||
type: 'buttons', showactive: false, x: 0.05, y: 1.08,
|
||||
buttons: [
|
||||
{
|
||||
label: '▶ Play',
|
||||
method: 'animate',
|
||||
args: [null, { frame: { duration: 500, redraw: true }, transition: { duration: 300 }, fromcurrent: true }]
|
||||
},
|
||||
{
|
||||
label: '◼ Pause',
|
||||
method: 'animate',
|
||||
args: [[null], { frame: { duration: 0, redraw: true }, mode: 'immediate' }]
|
||||
}
|
||||
]
|
||||
}],
|
||||
sliders: [{
|
||||
active: 0,
|
||||
steps: sliderSteps,
|
||||
x: 0.05, len: 0.9,
|
||||
xanchor: 'left',
|
||||
y: -0.02,
|
||||
yanchor: 'top',
|
||||
pad: { t: 30, b: 10 },
|
||||
currentvalue: { visible: false },
|
||||
transition: { duration: 300 },
|
||||
font: { size: 9, color: '#64748b' },
|
||||
bgcolor: '#1e293b',
|
||||
activebgcolor: '#3b82f6',
|
||||
bordercolor: '#334155',
|
||||
borderwidth: 1,
|
||||
ticklen: 4,
|
||||
tickcolor: '#475569',
|
||||
}],
|
||||
};
|
||||
|
||||
Plotly.newPlot('tsneAnim', firstTraces, layout, CFG).then(() => {
|
||||
Plotly.addFrames('tsneAnim', frames);
|
||||
});
|
||||
|
||||
// Update badge on animation frame
|
||||
const badge = document.querySelector('#monthBadge span');
|
||||
badge.textContent = `${fmtMonth(months[0])} — ${firstCount} drafts`;
|
||||
|
||||
document.getElementById('tsneAnim').on('plotly_animatingframe', function(ev) {
|
||||
const month = ev.name;
|
||||
const cumCount = points.filter(p => p.month <= month).length;
|
||||
badge.textContent = `${fmtMonth(month)} — ${cumCount} drafts`;
|
||||
});
|
||||
|
||||
// Click to navigate
|
||||
document.getElementById('tsneAnim').on('plotly_click', function(data) {
|
||||
const pt = data.points[0];
|
||||
if (pt.customdata) {
|
||||
window.location.href = '/drafts/' + pt.customdata;
|
||||
}
|
||||
});
|
||||
|
||||
// --- Stacked area chart ---
|
||||
// Collect all categories across all months
|
||||
const areaCats = {};
|
||||
Object.values(catMonthly).forEach(mc => {
|
||||
Object.keys(mc).forEach(c => { areaCats[c] = true; });
|
||||
});
|
||||
// Sort by total count
|
||||
const areaCatList = Object.keys(areaCats).sort((a, b) => {
|
||||
const totalA = months.reduce((s, m) => s + ((catMonthly[m] || {})[a] || 0), 0);
|
||||
const totalB = months.reduce((s, m) => s + ((catMonthly[m] || {})[b] || 0), 0);
|
||||
return totalB - totalA;
|
||||
});
|
||||
|
||||
const monthLabels = months.map(fmtMonth);
|
||||
const areaTraces = areaCatList.map((cat, i) => ({
|
||||
x: monthLabels,
|
||||
y: months.map(m => (catMonthly[m] || {})[cat] || 0),
|
||||
name: cat,
|
||||
type: 'scatter',
|
||||
mode: 'lines',
|
||||
stackgroup: 'one',
|
||||
line: { width: 0.5, color: catColor[cat] || PALETTE[i % PALETTE.length] },
|
||||
fillcolor: (catColor[cat] || PALETTE[i % PALETTE.length]) + '80',
|
||||
hovertemplate: '%{x}<br>' + cat + ': %{y}<extra></extra>',
|
||||
}));
|
||||
|
||||
Plotly.newPlot('stackedArea', areaTraces, {
|
||||
...PLOTLY_LAYOUT,
|
||||
xaxis: { ...PLOTLY_LAYOUT.xaxis, title: { text: 'Month', font: { size: 11 } } },
|
||||
yaxis: { ...PLOTLY_LAYOUT.yaxis, title: { text: 'Drafts', font: { size: 11 } } },
|
||||
legend: { font: { size: 10, color: '#94a3b8' }, orientation: 'h', y: -0.25, x: 0.5, xanchor: 'center' },
|
||||
hovermode: 'x unified',
|
||||
margin: { t: 20, r: 20, b: 80, l: 50 },
|
||||
}, CFG);
|
||||
|
||||
} else {
|
||||
document.getElementById('tsneSection').innerHTML = '<p class="text-slate-500 text-sm text-center py-20">No timeline animation data available. Run the analysis pipeline first.</p>';
|
||||
document.getElementById('stackedArea').innerHTML = '<p class="text-slate-500 text-sm text-center py-20">No data available.</p>';
|
||||
document.getElementById('statCards').style.display = 'none';
|
||||
}
|
||||
</script>
|
||||
{% endblock %}
|
||||
@@ -61,11 +61,6 @@ def test_authors_page(client):
|
||||
assert resp.status_code == 200
|
||||
|
||||
|
||||
def test_timeline_page(client):
|
||||
resp = client.get("/timeline")
|
||||
assert resp.status_code == 200
|
||||
|
||||
|
||||
def test_search_page_empty(client):
|
||||
resp = client.get("/search")
|
||||
assert resp.status_code == 200
|
||||
|
||||
Reference in New Issue
Block a user