Commit Graph

19 Commits

Author SHA1 Message Date
42b4546ded feat: refresh pipeline, reorganize draft generation, polish public pages
Some checks failed
CI / test (3.11) (push) Failing after 1m39s
CI / test (3.12) (push) Failing after 1m1s
Pipeline refresh:
- Extract ideas from 46 remaining drafts (844 total ideas now)
- Clear stale llm_cache entries blocking re-extraction
- Re-run gap analysis with expanded corpus: 10 gaps (was 12), fresh IDs #1-#10
- Re-link 3 proposals to new gap IDs
- Add scripts/pipeline-refresh.sh for reproducible runs

Draft generation moved from gaps to proposals:
- Remove "Generate Internet-Draft" section from gap detail page
- Add it to proposal detail page instead (proposals → generate I-D flow)
- New route POST /proposals/<id>/generate with richer context
  (proposal title + description + linked gap topics)
- Remove misleading "Search related drafts" link from gap page
  (related drafts already shown inline)

Public page polish:
- Overview: update subtitle to mention all 6 standards bodies
- About: describe multi-source scope (IETF, ISO, ITU-T, ETSI, NIST, W3C)
- About: add guiding question ("Where is the AI agent standards race heading?")
- Obsidian export button hidden in production mode (prior commit)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 05:39:13 +01:00
f8ed2b83e9 fix: security hardening — self-hosted JS, XSS protection, SSRF blocking
- Replace all CDN script tags (marked, plotly) with self-hosted static files
- Add DOMPurify for sanitizing markdown-rendered HTML
- Add escapeHtml() helper to base.html for all innerHTML operations
- Sanitize dynamic data in innerHTML across 13 templates
- Add security headers (X-Content-Type-Options, X-Frame-Options, Referrer-Policy)
- Add SSRF protection to proposal intake URL fetcher (block private/loopback IPs)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 04:47:32 +01:00
d1a20fa02e feat: add blog draft generator and fix broken routes
Blog drafting section (dev-only):
- BlogDraftGenerator gathers project data (gaps, proposals, stats) as
  context and calls Claude to produce Medium-style blog posts
- DB schema: blog_drafts table with title, content, tags, cost tracking
- Web UI: list, generate (async with live preview), detail (rendered +
  source toggle), edit, and export routes
- 6 writing styles: deep-dive, overview, opinion, listicle, comparison,
  series-post
- Nav link added to sidebar under Proposals

Bug fixes found via route testing (scripts/test_all_routes.py):
- /authors/<id>: Draft.status → Draft.states (correct attribute name)
- /false-positives: add missing `import re` in ratings.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 04:39:16 +01:00
c755b2bbf3 Add author detail, idea detail, and gap-draft reverse link pages
- Author detail page (/authors/<person_id>): shows author info, all drafts
  with ratings, and co-authors with shared draft counts. Public route.
- Idea detail page (/ideas/<idea_id>): shows idea metadata, source draft,
  and top-5 most similar ideas via embedding cosine similarity. Admin route.
- Gap detail page: added "Related Drafts" section that finds drafts by
  extracting draft names from evidence text and searching by topic keywords.
- Updated author links across templates to use /authors/<person_id> URLs.
- Added DB methods: get_author_by_id, get_author_drafts, get_coauthors.
- Extended top_authors to include person_id (5th tuple element).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 03:45:00 +01:00
3fb17100d7 Split webui into Flask blueprints and data domain modules
- Split app.py (66 routes) into 3 blueprints: pages (public), api (JSON), admin (@admin_required)
- Split data.py (4,360 LOC) into 7 domain modules: drafts, authors, ratings, gaps, analysis, search, proposals
- Add data/__init__.py re-exporting all public functions for backward compatibility
- Add custom 404/500 error pages matching dark theme
- Add request timing logging via before_request/after_request hooks
- Refactor app.py into create_app() factory pattern
- All 106 tests pass, all 66 routes preserved

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 03:37:15 +01:00
5ec7410b89 feat: proposal intake pipeline with AI-powered generation on /proposals/new
Add full proposal system: DB schema (proposals + proposal_gaps tables),
CLI `ietf intake` command, and web UI with Quick Generate on /proposals/new.
The new page merges AI intake (paste URL/text → Haiku generates multiple
proposals auto-linked to gaps) with manual form entry. Generated proposals
are clickable cards that fill the editor below for refinement.

Uses claude_model_cheap (Haiku) for cost-efficient web intake. Includes
CaML-inspired draft proposals from arXiv:2503.18813 analysis.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 03:15:11 +01:00
ae5e5f8cbf Enforce public/private visibility for web UI pages
Dev-only pages (sources, trends, complexity, idea-analysis, false-positives,
similarity, landscape, export) now require @admin_required and are hidden
from nav in production mode. Citations page keeps the graph public but
hides influence/BCP tabs behind --dev flag.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 20:52:43 +01:00
8515e46d5d Architecture designer, author cluster names, FP filtering, new pages
- Add /architecture page: system-of-systems view with 8 layers, component
  cards, gap markers, source coverage chart, and clickable detail sidebar
- Give author clusters meaningful names from orgs + draft topic keywords
- Filter false positives (73 drafts, 54 ideas) from idea clusters,
  architecture, ideas listing, and search results
- Add NIST source fetcher with curated catalog of 11 AI publications
- New pages: trends, complexity, sources, false positives, idea analysis
- Clickable gap cards with full details (evidence, priority, nearby work)
- Component detail panel with linked drafts and top ideas

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 19:58:40 +01:00
a46a01bd8c Add auto-heal pipeline command and fix multi-source draft processing
- Add `ietf auto` command: fetches, analyzes, embeds, extracts ideas,
  and refreshes gaps across all sources with cost-based auto-approval
- Fix SourceDocument→Draft conversion in auto fetch step
- Fix gap_analysis method name in auto command
- Process all 270 unrated ETSI/ISO/ITU/NIST drafts (761 total, all rated)
- Update web UI templates and data layer for multi-source support

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 18:41:42 +01:00
1ec1f69bee v0.3.0: Publication-ready release with blog site, paper update, and polish
Release prep:
- Version bump to 0.3.0 (pyproject.toml, cli.py)
- Rewrite README.md with current stats (475 drafts, 713 authors, 501 ideas)
- Add CONTRIBUTING.md with dev setup and code conventions

Blog site:
- Add scripts/build-site.py (markdown → HTML with clean CSS, dark mode, nav)
- Generate static site in docs/blog/ (10 pages)
- Ready for GitHub Pages deployment

Academic paper (paper/main.tex):
- Update all counts: 474→475 drafts, 557→710 authors, 1907→462 ideas, 11→12 gaps
- Add false-positive filtering methodology (113 excluded, 361 relevant)
- Add cross-org convergence analysis (132 ideas, 33% rate)
- Add GDPR compliance gap to gap table
- Add LLM-as-judge caveats to rating methodology and limitations
- Add FIPA, IEEE P3394, W3C WoT to related work with bibliography entries
- Fix safety ratio to show monthly variation (1.5:1 to 21:1)

Pipeline:
- Fetch 1 new draft (475 total), 3 new authors (713 total)
- Fix 16 ruff lint errors across test files
- All 106 tests pass

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 17:54:43 +01:00
e7527ad68e Fix remaining critical, high, and medium issues from 4-perspective review
Critical fixes:
- Fix rating clamp range 1-10 → 1-5 (actual scale)
- Add `ietf ideas convergence` command (SequenceMatcher at 0.75 threshold)
- Fix "628 cross-org ideas" → 130 (verified from current DB) across 8 files

Security fixes:
- Sanitize FTS5 query input (strip special chars + boolean operators)
- Add rate limiting (10 req/min/IP) on Claude-calling endpoints
- Change <path:name> → <string:name> on draft routes

Codebase fixes:
- Add Database context manager (__enter__/__exit__)
- Wire false_positive filtering into queries (exclude by default in web UI)
- Fix Post 3 arithmetic ("~300" → "~409" distinct proposals)

Content & licensing:
- Add MIT LICENSE file
- Add IPR/FRAND notes (BCP 79, RFC 8179) to Posts 03 and 07
- Qualify "4:1 safety ratio" with monthly variation in 6 remaining files
- Add "Data as of March 2026" freeze-date headers to all 10 blog posts
- Hedge causal language in Post 04

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 12:47:47 +01:00
a386d0bb1a Add explanation of how author clusters are formed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 21:27:23 +01:00
02049c37a8 Show related drafts in author cluster cards
Each cluster card now shows draft count badge and up to 5 linked draft
titles with clickable links to draft detail pages. Data collected from
member nodes' draft lists during cluster detection.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 21:23:13 +01:00
34c36f81f1 Make /ask free by default, Claude synthesis is opt-in
Search results (FTS5 + Ollama embeddings) are shown immediately at no
cost. AI synthesis via Claude is behind a "Synthesize" button that the
user must explicitly click. Results are cached permanently so repeat
visitors never trigger API calls.

- Split ask into search_only() (free) and ask() (paid, cached)
- GET /ask now uses search_only — no Claude tokens spent
- POST /api/ask/synthesize triggers Claude (Haiku, ~$0.001)
- Cached answers shown with "cached" badge, no re-generation
- Template shows sources immediately + optional synthesize button

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 21:15:07 +01:00
3e36802500 Fix author network graph: tune force sim, reduce node count
- Raise co-authorship threshold from 1 to 2 shared drafts (498→156 nodes)
- Tune D3 force parameters for large graphs: capped link strength,
  wider distance, adaptive charge, lower velocity decay
- Add initial circular layout to prevent explosion on load
- Fix cluster highlighting with fallback name matching and
  position validation before zoom-to-fit

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 21:04:18 +01:00
757b781c67 Platform upgrade: semantic search, citations, readiness, tests, Docker
Major features added by 5 parallel agent teams:
- Semantic "Ask" (NL queries via FTS5 + embeddings + Claude synthesis)
- Global search across drafts, ideas, authors, gaps
- REST API expansion (14 endpoints, up from 3) with CSV/JSON export
- Citation graph visualization (D3.js, 440 nodes, 2422 edges)
- Standards readiness scoring (0-100 composite from 6 factors)
- Side-by-side draft comparison view with shared/unique analysis
- Annotation system (notes + tags per draft, DB-persisted)
- Docker deployment (Dockerfile + docker-compose with Ollama)
- Scheduled updates (cron script with log rotation)
- Pipeline health dashboard (stage progress bars, cost tracking)
- Test suite foundation (54 pytest tests covering DB, models, web data)

Fixes: compare_drafts() stubbed→working, get_authors_for_draft() bug,
source-aware analysis prompts, config env var overrides + validation,
resilient batch error handling with --retry-failed, observatory --dry-run

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 20:52:56 +01:00
da2a989744 GDPR compliance: self-host all assets, add Impressum + Datenschutz
- Self-host Tailwind, Plotly, D3.js (no more CDN requests)
- Self-host Inter font (no more Google Fonts requests)
- Replace JetBrains Mono with system monospace font stack
- Zero external requests when visiting the site (GDPR-safe)
- Add /impressum page (§5 TMG, §18 MStV)
- Add /datenschutz page (DSGVO Art. 13/14)
- Hosting: Hetzner, no cookies, no tracking, no analytics
- Add Impressum/Datenschutz links to sidebar footer
- Configure Flask static_folder for local asset serving

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 08:20:52 +01:00
75c4da72e0 Fix broken reference links and web UI bugs
- Fix RFC URLs with leading zeros (rfc0020 -> rfc20) via int filter
- Draft refs: internal link for drafts in our DB, Datatracker for external
- BCP refs: link to rfc-editor.org/info/bcpN
- Add DB connection teardown (@app.teardown_appcontext)
- Fix JS syntax error in gap_demo.html (HTML-escaped string in script tag)
- Add URL encoding to all query params in drafts.html and draft_detail.html
- Fix variable shadowing of Flask's g import in gaps_demo()
- Add None safety for ideas search data attribute

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-07 07:59:45 +01:00
6e3a387778 Idea quality pipeline, web UI features, academic paper
- Tighten idea extraction prompts (1-4 ideas, no sub-features) reducing
  1,907 ideas to 468 across 434 drafts (78% reduction)
- Add embedding-based dedup (ietf dedup-ideas) for same-draft similarity
- Add novelty scoring (ietf ideas score) and filtering (ietf ideas filter)
  using Claude to rate ideas 1-5, removing 49 generic building blocks
- Final count: 419 high-quality ideas (avg 1.1/draft)
- Web UI: gap explorer with live draft generation and pre-generated demos
- Web UI: D3.js author collaboration network (498 nodes, 1142 edges,
  68 clusters, org filtering, interactive zoom/pan)
- Academic paper: 15-page LaTeX workshop paper analyzing the 434-draft
  AI agent standards landscape
- Save improvement ideas backlog to data/reports/improvement-ideas.md

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 22:17:57 +01:00