Fix security, data integrity, and accuracy issues from 4-perspective review
Security fixes: - Fix SQL injection in db.py:update_generation_run (column name whitelist) - Flask SECRET_KEY from env var instead of hardcoded - Add LLM rating bounds validation (_clamp_rating, 1-10) - Fix JSON extraction trailing whitespace handling Data integrity: - Normalize 21 legacy category names to 11 canonical short forms - Add false_positive column, flag 73 non-AI drafts (361 relevant remain) - Document verified counts: 434 total/361 relevant drafts, 557 authors, 419 ideas, 11 gaps Code quality: - Fix version string 0.1.0 → 0.2.0 - Add close()/context manager to Embedder class - Dynamic matrix size instead of hardcoded "260x260" Blog accuracy: - Fix EU AI Act timeline (enforcement Aug 2026, not "18 months") - Distinguish OAuth consent from GDPR Einwilligung - Add EU AI Act Annex III context to hospital scenario - Add FIPA, eIDAS 2.0 references where relevant Methodology: - Add methodology.md documenting pipeline, limitations, rating rubric - Add LLM-as-judge caveats to analyzer.py - Document clustering threshold rationale Reviews from: legal (German/EU law), statistics, development, science perspectives. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -4,6 +4,98 @@
|
||||
|
||||
---
|
||||
|
||||
### 2026-03-08 CODER — Data Integrity Fixes from Statistical & Scientific Reviews
|
||||
|
||||
**What**: Fixed data integrity issues identified in `review-statistics.md` and `review-science.md`:
|
||||
1. **Category normalization**: Updated 21 ratings rows with legacy long-form category names (e.g., "Agent-to-agent communication protocols") to canonical short forms (e.g., "A2A protocols"). All 11 categories now consistent in the database.
|
||||
2. **False positive flagging**: Added `false_positive` column to ratings table. Flagged 73 drafts as false positives (38 with relevance <= 2, 35 manually reviewed at relevance 3+ that are clearly not AI-agent related — e.g., HPKE, cookies, BGP, EDHOC). Notable: excluding false positives yields exactly 361 relevant drafts.
|
||||
3. **Schema update**: Updated `db.py` schema definition and migration code to include `false_positive` column.
|
||||
4. **Verified counts document**: Created `data/reports/reviews/verified-counts.md` as single source of truth — documents all actual counts (434 drafts, 419 ideas, 11 gaps, 557 authors) with explanations for discrepancies.
|
||||
5. **Gap count confirmed**: 11 gaps in DB, not 12. Blog posts use an editorially rewritten gap list with different names and an extra gap.
|
||||
6. **Ideas count explained**: DB has 419 (post-dedup, 89% of drafts have exactly 1 idea). The 1,780 figure was pre-dedup. The 1,262 figure was from a smaller corpus.
|
||||
|
||||
**Why**: Reviews identified critical data integrity issues that would undermine credibility if published — inconsistent category names affecting counts by 5-15%, no mechanism to exclude false positives, and conflicting counts across all reports.
|
||||
**Result**: Database now has clean categories, false positive flags, and a verified-counts reference document. The coincidence that 434 - 73 false positives = 361 exactly matches the original blog series count.
|
||||
|
||||
---
|
||||
|
||||
### 2026-03-08 CODER — Fix Security & Code Quality Issues from Dev Review
|
||||
|
||||
**What**: Applied 7 targeted fixes from `data/reports/reviews/review-dev.md`:
|
||||
1. SQL injection in `db.py:update_generation_run` — added column name whitelist validation
|
||||
2. Flask SECRET_KEY — changed from hardcoded string to `os.environ.get('FLASK_SECRET_KEY', os.urandom(24).hex())`
|
||||
3. Version string — updated CLI from "0.1.0" to "0.2.0"
|
||||
4. JSON extraction — `_extract_json` now handles trailing whitespace after code fences via `.rstrip()`
|
||||
5. Ollama client lifecycle — added `close()`, `__enter__`, `__exit__` to `Embedder` class
|
||||
6. LLM rating bounds — added `_clamp_rating()` method clamping all rating fields to 1-10 integers in `_parse_rating`
|
||||
7. Hardcoded matrix size — replaced "260x260" with dynamic `{n_drafts}x{n_drafts}` from actual DB count
|
||||
**Why**: Dev reviewer flagged these as critical (SQL injection), high (SECRET_KEY), and medium priority issues
|
||||
**Result**: All 7 fixes applied with minimal targeted edits. No refactoring beyond what was needed.
|
||||
|
||||
---
|
||||
|
||||
### 2026-03-08 CODER — Methodology Documentation and Scientific Rigor Fixes
|
||||
|
||||
**What**: Addressed methodology and scientific rigor issues raised by the science and statistics reviews. Five deliverables:
|
||||
1. Added 35-line methodology comment block to `analyzer.py` documenting LLM-as-judge limitations (abstract-only, no calibration, no consistency check, overlap score limitation, batch effects, relevance inflation). Updated the rating prompt (`RATE_PROMPT_COMPACT`) with an explicit rubric defining what each score level means for each dimension.
|
||||
2. Created `data/reports/methodology.md` — comprehensive methodology document covering data collection (keywords, API, selection bias), analysis pipeline (all 6 stages), rating rubric with scale interpretation, clustering method and threshold justification, gap analysis limitations, embedding model properties, known limitations table, and related work references.
|
||||
3. Added 20-line docstring to `find_clusters()` in `embeddings.py` documenting the 0.85 threshold as an empirical choice with manual inspection rationale, noting that sensitivity analysis would strengthen confidence.
|
||||
4. Added 22-line comment block above `GAP_ANALYSIS_PROMPT` in `analyzer.py` documenting it as single-shot LLM analysis, noting the absence of reference architecture grounding, and listing strengthening options.
|
||||
5. Added methodology caveat notes to blog posts 01 (gold-rush), 03 (oauth-wars), 06 (big-picture), and 07 (how-we-built-this, full Limitations section added). Each note explains ratings are LLM-generated from abstracts without human calibration.
|
||||
6. Added related work section to methodology.md covering FIPA, IEEE P3394, W3C WoT, academic MAS research (AAMAS/JAIR/JAAMAS), and other standards bodies (OASIS, ITU-T, ETSI).
|
||||
|
||||
**Why**: Scientific and statistical reviews identified LLM-as-judge limitations, unjustified thresholds, missing related work, and ungrounded gap analysis as the top methodological weaknesses. These caveats are needed before publication.
|
||||
**Result**: 6 files modified (`analyzer.py`, `embeddings.py`, 4 blog posts), 1 file created (`methodology.md`). All changes are documentation/caveats — no pipeline restructuring.
|
||||
|
||||
---
|
||||
|
||||
### 2026-03-08 STATISTICS REVIEWER — Full Statistical Audit of Blog Series
|
||||
|
||||
**What**: Audited all 10 blog posts, 9 data packages, master stats, and key reports against the actual database (`data/drafts.db`) using sqlite3 queries. Produced comprehensive statistical review at `data/reports/reviews/review-statistics.md`.
|
||||
**Why**: The blog series makes extensive quantitative claims (361 drafts, 1,780 ideas, 12 gaps, 4:1 safety ratio, 36x growth, etc.) that needed cross-checking against the ground truth database before publication.
|
||||
**Result**: Found 3 critical issues, 4 important issues, and 4 minor issues. Most serious: the ideas table has 419 rows (not 1,780 as claimed), the database now has 434 drafts (not 361), gaps are 11 (not 12), and composite scores are inflated by 0.05-0.10 through rounding. The 4:1 safety ratio varies from 1.5:1 to 21:1 by month. The "36x growth" figure cherry-picks endpoints. Qualitative patterns (Huawei dominance, safety deficit, fragmentation) hold directionally. RFC cross-refs (4,231), author count (557), and draft-author links (1,057) match exactly.
|
||||
**Surprise**: The ideas count mismatch (419 vs 1,780) is the most serious finding -- Post 5's entire thesis about "96% of ideas in one draft" and "628 cross-org convergent ideas" is not reproducible from the current database. The pipeline may have been re-run with different parameters, overwriting the original idea extraction.
|
||||
|
||||
---
|
||||
|
||||
### 2026-03-08 LEGAL REVIEWER — Full Legal Review of Blog Series and Reports
|
||||
|
||||
**What**: Reviewed all 10 blog series files (Posts 00-08 plus state-of-ecosystem) and key reports (gaps.md, overview.md) through a German/EU internet law lens. Produced comprehensive legal review covering GDPR, EU AI Act, eIDAS 2.0, NIS2, CRA, product liability, and IETF IPR policy.
|
||||
**Why**: The series makes claims about safety gaps, identity/auth protocols, and regulatory predictions without adequately engaging the EU regulatory framework -- which is not future speculation but current law with imminent enforcement deadlines (AI Act fully applicable August 2026).
|
||||
**Result**: Review written to `data/reports/reviews/review-legal.md`. Found 3 critical issues (consent terminology conflation, hospital scenario understating regulatory reality, GDPR omission from gap analysis), 5 regulatory gaps (AI Act needs structural treatment not just a prediction, eIDAS 2.0 missing from identity discussion, NIS2/CRA unaddressed, German TKG context absent), 5 improvement suggestions, and per-post notes for all 10 files. Top priority: Post 6's AI Act enforcement timeline is wrong (says "18 months" but enforcement begins in 5 months).
|
||||
**Surprise**: The series' best architectural proposal -- assurance profiles L0-L3 -- maps remarkably well to the AI Act's risk-based approach, but the connection is never made explicit. Making it explicit would strengthen both the regulatory argument and the technical proposal.
|
||||
|
||||
---
|
||||
|
||||
### 2026-03-08 REVIEWER-DEV — Full Codebase Engineering Review
|
||||
|
||||
**What**: Comprehensive code review of all core modules (`db.py`, `analyzer.py`, `cli.py`, `fetcher.py`, `embeddings.py`, `authors.py`, `models.py`, `config.py`, `draftgen.py`, `search.py`, `readiness.py`), web UI (`app.py`, `data.py`, `auth.py`), and scripts. Reviewed ~5000 lines of application code and ~2000 lines of web data layer.
|
||||
**Why**: Pre-deployment quality gate. The tool has grown from a simple CLI to a full web dashboard with API endpoints, and the security/quality bar needs to rise accordingly.
|
||||
**Result**: Review written to `data/reports/reviews/review-dev.md`. Found 1 critical issue (SQL injection in `update_generation_run`), 1 high issue (hardcoded Flask SECRET_KEY), 5 bugs, 6 performance concerns, and 14 improvement suggestions. Overall grade: B+ -- solid architecture, needs hardening. Key positives: clean separation of concerns, effective LLM caching, good auth design, proper FTS5 sync triggers.
|
||||
**Surprise**: The `cli.py` file has grown to 2995 lines with ~40 repetitions of the same config/db boilerplate pattern. Also, zero test coverage for the analysis pipeline (`analyzer.py`, `embeddings.py`, `fetcher.py`) despite it being the core of the tool.
|
||||
|
||||
---
|
||||
|
||||
### 2026-03-08 REVIEWER (Science) — Full Scientific Review of Methodology and Outputs
|
||||
|
||||
**What**: Conducted comprehensive scientific review of the entire analysis pipeline, database integrity, reports, and blog posts. Reviewed analyzer.py (rating/idea/gap prompts), embeddings.py (clustering), fetcher.py (data collection), config.py, and all reports/blog posts. Queried database directly for integrity checks.
|
||||
|
||||
**Why**: The analysis makes strong claims (4:1 safety deficit, 12 gaps, 1262 ideas, 9.3% of IETF submissions) that need to withstand scrutiny from IETF participants, academic reviewers, and standards experts. Several methodological weaknesses and data inconsistencies were found that could undermine credibility if not addressed.
|
||||
|
||||
**Result**: Wrote detailed review to `data/reports/reviews/review-science.md` with 8 sections covering methodology, unsupported claims, missing context, data integrity, improvements, taxonomy, and post-by-post notes. Key findings:
|
||||
- **CRITICAL**: Ideas database has 419 entries but blog posts reference 1,262-1,780. Major data inconsistency.
|
||||
- **CRITICAL**: LLM ratings have no human calibration. No inter-rater reliability measurement.
|
||||
- **HIGH**: 55 non-canonical category names in ratings table (normalization not applied to stored data).
|
||||
- **HIGH**: ~30-50 false positive drafts in corpus (e.g., HPKE, PIE bufferbloat rated relevance 5 and 3).
|
||||
- **HIGH**: Missing related work context (FIPA, IEEE P3394, academic MAS research).
|
||||
- **MEDIUM**: Greedy single-linkage clustering at unjustified 0.85 threshold.
|
||||
- Database grew from 361 to 434 drafts but all reports/blogs still cite 361.
|
||||
- 10 prioritized recommendations provided, from calibration study to reference architecture.
|
||||
|
||||
**Surprise**: The ideas count discrepancy (419 vs 1,780) is dramatic -- either mass dedup removed 75%+ of ideas, or the database was regenerated. Either way, Post 05 ("1,262 Ideas") needs a full rewrite. Also, `draft-ietf-hpke-hpke` (generic public key encryption, nothing to do with AI agents) is rated relevance=5, showing the LLM judge is too generous with keyword-matched drafts.
|
||||
|
||||
**Cost**: Zero API cost (review only, no pipeline runs). Approximately 90 minutes of analysis time.
|
||||
|
||||
### 2026-03-07 CODER C — Citation Graph, Readiness Scoring, Annotations, Data Surfacing
|
||||
|
||||
**What**: Implemented four features in a single session:
|
||||
|
||||
Reference in New Issue
Block a user