c/ietf-draft-analyzer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix security, data integrity, and accuracy issues from 4-perspective review

Browse Source 
Security fixes:
- Fix SQL injection in db.py:update_generation_run (column name whitelist)
- Flask SECRET_KEY from env var instead of hardcoded
- Add LLM rating bounds validation (_clamp_rating, 1-10)
- Fix JSON extraction trailing whitespace handling

Data integrity:
- Normalize 21 legacy category names to 11 canonical short forms
- Add false_positive column, flag 73 non-AI drafts (361 relevant remain)
- Document verified counts: 434 total/361 relevant drafts, 557 authors, 419 ideas, 11 gaps

Code quality:
- Fix version string 0.1.0 → 0.2.0
- Add close()/context manager to Embedder class
- Dynamic matrix size instead of hardcoded "260x260"

Blog accuracy:
- Fix EU AI Act timeline (enforcement Aug 2026, not "18 months")
- Distinguish OAuth consent from GDPR Einwilligung
- Add EU AI Act Annex III context to hospital scenario
- Add FIPA, eIDAS 2.0 references where relevant

Methodology:
- Add methodology.md documenting pipeline, limitations, rating rubric
- Add LLM-as-judge caveats to analyzer.py
- Document clustering threshold rationale

Reviews from: legal (German/EU law), statistics, development, science perspectives.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Christian Nennemann
2026-03-08 10:52:33 +01:00
parent a386d0bb1a
commit 439424bd04
19 changed files with 1745 additions and 126 deletions
Download Patch File Download Diff File Expand all files Collapse all files

186
data/reports/reviews/review-dev.md Normal file
View File

@@ -0,0 +1,186 @@
# Development & Engineering Review
**Reviewer**: Development & Engineering Reviewer (Opus 4.6)
**Date**: 2026-03-08
**Scope**: Full codebase review — `src/ietf_analyzer/`, `src/webui/`, `scripts/`, `data/reports/`
---
## Summary Verdict
The codebase is well-structured for a research/analysis tool. Architecture is clean: Click CLI, SQLite with FTS5, Claude for analysis, Ollama for embeddings, Flask web UI. Code is readable and follows consistent patterns. However, there are several security issues (one critical), a few bugs, and significant testing gaps that should be addressed before any public deployment.
**Overall grade: B+** -- solid for a personal research tool, needs hardening for production.
---
## Critical Issues
### 1. SQL Injection in `db.py:update_generation_run` (CRITICAL)
```python
def update_generation_run(self, run_id: int, **kwargs) -> None:
sets = []
for k, v in kwargs.items():
sets.append(f"{k} = ?") # <-- column name from **kwargs, unvalidated
```
The column names come directly from `**kwargs` and are interpolated into the SQL string without validation. While this is only called internally today, any future caller passing user-controlled keyword arguments creates a SQL injection vector. **Fix**: Whitelist allowed column names against the table schema.
### 2. Hardcoded Flask SECRET_KEY (HIGH)
```python
app.config["SECRET_KEY"] = "ietf-dashboard-dev"
```
In `src/webui/app.py:61`. This is a static, publicly visible secret. While the app currently uses no sessions that depend on signing, Flask's session cookie is signed with this key. If any session-based feature is added (and there's already an auth module), cookies can be forged. **Fix**: Generate from environment variable or `secrets.token_hex()` at startup.
### 3. No Rate Limiting on API Endpoints (MEDIUM)
The `/api/ask/synthesize` and `/api/compare` POST endpoints trigger Claude API calls that cost real money. Even with `@admin_required`, in dev mode (`--dev`), any client can trigger unlimited API calls. **Fix**: Add per-IP or per-session rate limiting, at minimum on the Claude-calling endpoints.
---
## Code Issues
### Bugs
1. **`_extract_json` mishandles nested fences** (`analyzer.py:196-201`): If Claude returns code fences with a language tag (e.g., ` ```json\n{...}\n``` `), the first `split("\n", 1)` correctly strips the opening line, but the trailing ```` ` check uses `endswith` which fails if there's trailing whitespace. Minor but can cause silent JSON parse failures.
2. **Version string mismatch** (`cli.py:24`): `@click.version_option(version="0.1.0")` but the project is at v0.2.0 per `CLAUDE.md` and memory. Should be kept in sync, ideally from a single source (`__init__.py` or `pyproject.toml`).
3. **`embed_all_missing` never closes Ollama client**: The `Embedder` class creates an `ollama.Client` but has no `close()` method, unlike `Fetcher` and `AuthorNetwork` which properly close their `httpx.Client`. Not a major issue since Ollama connections are typically local, but inconsistent.
4. **`similarity_matrix` is O(n^2) with no caching**: `embeddings.py:102-113` computes the full pairwise matrix every time. For 361 drafts this is ~65K comparisons per call, and this is called by `find_clusters` and the web UI. The web data layer adds a 5-minute TTL cache, but the CLI path has none.
5. **`overlap_matrix` report hardcodes "260x260"**: `cli.py:603` prints `"Computing 260x260 similarity matrix..."` but the actual corpus is 361 drafts. Cosmetic but suggests stale code.
### Security
6. **`read_generated_draft` path traversal check is good** (`data.py:369-371`): The `resolve()` + `startswith()` guard against directory traversal is correctly implemented. Well done.
7. **FTS5 query injection** (`search.py:97-109`): FTS5 MATCH queries can fail on special characters. The fallback wrapping words in quotes is a reasonable mitigation, but untrusted input containing double quotes could still cause issues. Consider sanitizing with `re.sub(r'[^\w\s]', '', query)` before passing to FTS5.
8. **`draft_detail` route uses `<path:name>` converter** (`app.py:137`): This allows slashes in the draft name URL segment. While the DB lookup parameterizes correctly, the `<path:>` converter should be `<string:>` since draft names don't contain slashes. Using `<path:>` is unnecessarily permissive.
### Performance
9. **`get_drafts_page` loads all rated drafts every time** (`data.py:104`): `db.drafts_with_ratings(limit=1000)` fetches up to 1000 draft+rating pairs into memory, then filters in Python. For 361 drafts this is fine, but the pattern won't scale. More importantly, `compute_readiness` is called per draft on the page (line 161), and each call makes 3-4 separate DB queries. For a page of 50 drafts, that's ~200 DB queries per page load.
10. **`all_embeddings()` loads all vectors into memory** (`db.py:455-460`): For 361 drafts with 768-dim embeddings, this is ~1.1MB -- acceptable. But `_embedding_search` in `search.py:135` calls this on every search query. Should be cached or use a vector similarity index.
11. **`_compute_author_network_full` calls `db.get_draft(dn)` in a loop** (`data.py:621`): For cluster draft lookup, each call is a separate DB query. With 15 drafts per cluster across multiple clusters, this is an N+1 query pattern. Should batch-fetch.
### Code Quality
12. **Excessive boilerplate in CLI commands**: The pattern of `cfg = _get_config(); db = Database(cfg); try: ... finally: db.close()` is repeated ~40 times across the 2995-line `cli.py`. This should be a context manager or Click callback. Example:
```python
@contextmanager
def get_db_context():
cfg = _get_config()
db = Database(cfg)
try:
yield cfg, db
finally:
db.close()
```
13. **`Database` class should be a context manager**: Adding `__enter__`/`__exit__` would eliminate all the `try/finally/close` blocks.
14. **No type hints on `Database` return types for dicts**: Methods like `all_gaps()`, `all_ideas()`, `wg_summary()` return `list[dict]` but the dict structure is undocumented. TypedDict or dataclasses would improve maintainability.
15. **`data.py` imports inside functions**: `compute_readiness` is imported inside `get_drafts_page` and `get_draft_detail` (lines 158, 245). This works but is unusual for a data access layer.
---
## Missing Developer Value
1. **No test coverage for the analysis pipeline**: Tests exist for `db.py`, `models.py`, `web_data.py`, and `obsidian_export.py`, but none for `analyzer.py`, `embeddings.py`, `fetcher.py`, `search.py`, `readiness.py`, or `draftgen.py`. The analysis pipeline is the core of the tool and is completely untested.
2. **No CI/CD configuration**: No GitHub Actions, no `Makefile`, no `tox.ini`. For a tool that generates research outputs, reproducibility matters.
3. **No `pyproject.toml` or `setup.py` visible**: The `ietf` CLI entry point is referenced but the packaging config isn't in the reviewed files. The install path is unclear.
4. **No data validation on LLM outputs**: Rating values from Claude are cast with `int(data.get("n", 3))` but never bounds-checked (except in `score_idea_novelty` where `max(1, min(5, int(v)))` is used). Claude could return 0, 6, or -1 and it would be stored.
5. **No error recovery for partial pipeline runs**: If `rate_all_unrated` fails halfway through, there's no way to resume from where it stopped without re-processing already-rated drafts (the cache helps, but isn't guaranteed to hit if prompts change).
---
## Improvement Suggestions
### High Priority
1. **Validate LLM output bounds**: Add `max(1, min(5, ...))` clamping in `_parse_rating` for all rating fields, not just in `score_idea_novelty`.
2. **Whitelist columns in `update_generation_run`**: Replace dynamic column interpolation with an allowed-columns set.
3. **Generate Flask SECRET_KEY at startup**: `app.config["SECRET_KEY"] = os.environ.get("FLASK_SECRET_KEY", secrets.token_hex(32))`.
4. **Add Database context manager**: `def __enter__(self): return self` / `def __exit__(...): self.close()`.
5. **Add tests for analyzer.py**: Mock the Anthropic client and test JSON parsing, rating bounds, cache hit/miss, batch processing.
### Medium Priority
6. **Deduplicate CLI boilerplate**: Use a Click group callback or context manager to handle config/db lifecycle.
7. **Add rate limiting**: Use `flask-limiter` or a simple token bucket for Claude-calling endpoints.
8. **Batch readiness computation**: Instead of N+1 queries per page, compute readiness factors in bulk SQL queries.
9. **Cache similarity matrix**: Store precomputed matrix in DB or pickle file, invalidate when embeddings change.
10. **Fix version string**: Single source of truth for version number.
### Low Priority
11. **Add TypedDict for common dict shapes**: `IdeaDict`, `GapDict`, `RatingDict` etc.
12. **Add `--dry-run` to more CLI commands**: Currently only `ideas dedup` supports it.
13. **Add OpenAPI/Swagger docs**: The API endpoints are well-structured and would benefit from auto-generated docs.
14. **Consider async for web UI**: The t-SNE and clustering computations block the Flask request thread. Consider `flask[async]` or background tasks.
---
## Architecture Notes
### What Works Well
- **Separation of concerns**: CLI, DB, analysis, embedding, reporting, and web UI are cleanly separated into modules.
- **LLM caching**: The `llm_cache` table with SHA256 prompt hashing is well-designed and saves significant API costs.
- **Graceful degradation**: The search system falls back from semantic+keyword to keyword-only when Ollama is unavailable.
- **Auth design**: The dev/production mode split is simple and effective. Admin routes return 404 in production (not 403), which is security-correct.
- **FTS5 triggers**: The auto-sync triggers for the full-text search index are correctly implemented and handle INSERT/UPDATE/DELETE.
- **UPSERT patterns**: Consistent use of `INSERT ... ON CONFLICT DO UPDATE` throughout the DB layer.
### What Could Be Better
- **3000-line `cli.py`**: This single file has grown large. Consider splitting into `cli/fetch.py`, `cli/analyze.py`, `cli/report.py`, etc.
- **Web data layer fetches everything**: Most endpoints call `db.drafts_with_ratings(limit=1000)` and filter in Python rather than using SQL WHERE clauses. This is a pattern that won't scale.
- **No migration system**: Schema changes rely on additive `ALTER TABLE` in `_migrate_schema`. This works for column additions but can't handle schema changes, index additions, or data migrations. A lightweight migration framework (even just numbered SQL files) would be more robust.
---
## Post-by-Post Notes
*(Blog posts in `data/reports/blog-series/` were not the primary focus of this review. Quick technical accuracy check:)*
No blog posts appear to be written yet based on the git status and project memory. The blog series infrastructure is in place but content generation has not started.
---
## Methodology Assessment
The analysis methodology is defensible for exploratory research:
- **Rating**: Using Claude to rate drafts on 5 dimensions is reasonable for landscape analysis. The compact prompt design saves tokens while capturing key attributes. However, the ratings should be presented as "AI-assessed" with appropriate caveats, since a single LLM pass on abstracts may not capture implementation quality.
- **Embeddings**: Using nomic-embed-text for similarity is appropriate. The 0.85 threshold for clustering seems reasonable. The greedy clustering algorithm in `embeddings.py` is simple but may miss transitive similarities (draft A similar to B, B similar to C, but A not directly similar to C).
- **Gap analysis**: The gap identification prompt uses category distributions and idea frequencies as evidence, which is sound. However, the prompt feeds the LLM its own previous outputs (categories, ideas), creating a feedback loop that could amplify biases.
- **Readiness scoring**: The 6-factor composite score in `readiness.py` is well-designed with reasonable weights. The normalization (rev/5, cited/5, etc.) is transparent and defensible.

175
data/reports/reviews/review-legal.md Normal file
View File

@@ -0,0 +1,175 @@
# Legal Review -- German/EU Internet Law Perspective
*Reviewer: Legal Reviewer Agent | Date: 2026-03-08*
*Scope: Blog posts 00-08 in `data/reports/blog-series/`, key reports in `data/reports/`*
---
## Critical Issues
### 1. "Consent" terminology conflation (Posts 3, 6)
The series uses "consent" interchangeably across OAuth authorization flows, GDPR consent (Art. 6(1)(a) GDPR), and human-in-the-loop approval. These are legally distinct concepts:
- **OAuth consent** is a technical authorization flow where a user delegates access scopes to a client.
- **GDPR consent** (Einwilligung) is a legal basis for data processing that must be freely given, specific, informed, and unambiguous (Art. 4(11) GDPR) and is revocable at any time (Art. 7(3) GDPR).
- **HITL approval gates** (as proposed in Post 6) are operational control mechanisms, not consent under any legal framework.
Post 3 discusses 14 OAuth-for-agents proposals without noting that delegated agent authorization raises fundamental GDPR consent validity questions. Under CJEU case law (Planet49, C-673/17), consent requires a clear affirmative act by the data subject. When an AI agent further delegates to sub-agents, the chain of consent may break entirely. None of the blog posts flag this.
**Recommendation**: Add a clarifying footnote in Post 3 that distinguishes OAuth "consent" from GDPR consent, and note that chained delegation in multi-agent systems raises unresolved consent propagation questions under EU data protection law.
### 2. The hospital scenario in Post 4 understates regulatory reality
The opening scenario -- an AI agent managing a hospital drug-dispensing system where a hallucinated dosage cascades without oversight -- is presented as a gap-analysis illustration. Under EU law, this is not merely an engineering gap; it is a regulatory compliance failure in multiple dimensions:
- **EU AI Act (Regulation 2024/1689)**: A drug-dispensing AI agent is a **high-risk AI system** under Annex III, Section 5(b) (AI systems intended to be used as safety components in the management and operation of critical digital infrastructure, road traffic and the supply of water, gas, heating and electricity) and potentially under the Medical Devices Regulation (MDR 2017/745). High-risk systems require conformity assessment, risk management systems, data governance, and human oversight (Arts. 9-14 AI Act).
- **Product Liability Directive (2024/2853)**: The revised PLD explicitly covers software and AI systems. The cascading failure scenario would trigger strict product liability for the AI system provider.
- **German Patientenrechtegesetz / BGB SS 630a ff.**: The treatment contract (Behandlungsvertrag) places duty of care on the healthcare provider. Automated dispensing without adequate safeguards violates the standard of care.
The blog post frames this as "what goes wrong if this is never addressed" at the standards level. Legally, it is already addressed at the regulatory level -- the gap is in technical implementation, not in the existence of liability. This distinction matters because readers might infer that absent IETF standards mean absent accountability, which is incorrect under EU law.
**Recommendation**: Add a sentence acknowledging that the EU AI Act already classifies such systems as high-risk and imposes mandatory requirements. The IETF gap is in providing the technical mechanisms to *implement* what the regulation *requires*.
### 3. The gap analysis omits GDPR-mandated requirements entirely
The 12 gaps identified across the series and the `gaps.md` report include "Agent Privacy Preservation" (HIGH severity in the report, mentioned as "privacy-preserving discovery" in Post 5) but do not engage with GDPR as a legally binding framework. The gaps are framed as technical desiderata, not regulatory compliance requirements.
Specific GDPR-mandated capabilities that should appear in the gap analysis but do not:
- **Data Protection Impact Assessment (DPIA) support** (Art. 35 GDPR): High-risk agent processing requires DPIAs. No draft or gap addresses machine-readable DPIA tooling.
- **Right to erasure** (Art. 17 GDPR): When agents process personal data across multi-agent chains, the right to erasure must propagate. The ECT-based DAG model proposed in Post 6 records execution evidence but does not address how to *delete* that evidence when legally required.
- **Data portability** (Art. 20 GDPR): Agent-generated data about individuals must be portable. No gap addresses this.
- **Purpose limitation** (Art. 5(1)(b) GDPR): Agents authorized for one purpose must not repurpose data. The "scope aggregation" OAuth proposals (Post 3) could facilitate purpose creep if not constrained.
**Recommendation**: Add GDPR compliance as a cross-cutting regulatory dimension in Post 6 or the gap analysis. The ECT/DAG model is architecturally promising but needs to account for data deletion, purpose limitation, and DPIA requirements.
---
## Regulatory Gaps
### 1. EU AI Act is mentioned once, in a prediction -- it deserves structural treatment
Post 6 predicts that "within 18 months, the safety deficit will begin to close -- not from IETF drafts but from regulatory pressure. The EU AI Act's requirements for high-risk AI systems will drive demand for behavior verification, human override, and audit standards." This is the only substantive mention of the EU AI Act across 8 blog posts and all reports.
The EU AI Act (Regulation 2024/1689) entered into force on 1 August 2024 and will be fully applicable from 2 August 2026. It is not a future event; it is current law with imminent enforcement deadlines. Its requirements map directly to several of the series' key findings:
| AI Act Requirement | Corresponding IETF Gap | Blog Post |
|---|---|---|
| Art. 9: Risk management system | Behavior Verification (Critical) | Post 4 |
| Art. 14: Human oversight | Human Override (High) | Posts 4, 6 |
| Art. 12: Record-keeping / logging | Error Recovery, Data Provenance | Posts 4, 5 |
| Art. 13: Transparency | Explainability (Medium) | Post 5 |
| Art. 15: Accuracy, robustness, cybersecurity | Agent Capability Degradation | Report |
| Art. 17: Quality management system | Lifecycle Management (High) | Post 5 |
The series would be significantly strengthened by treating the AI Act not as a future prediction but as a current regulatory driver that makes several of the identified gaps not just technically desirable but legally mandatory.
### 2. eIDAS 2.0 and the European Digital Identity Wallet
The series discusses agent identity extensively (108 drafts, 14 OAuth proposals) but does not mention eIDAS 2.0 (Regulation 2024/1183). The revised eIDAS framework introduces the European Digital Identity Wallet (EUDI Wallet), which will become available to all EU citizens by 2026-2027.
Implications for agent identity standards:
- eIDAS 2.0 defines **electronic attestations of attributes** that could extend to agent attributes and capabilities.
- The **trust framework** in eIDAS 2.0 (qualified trust services, qualified electronic signatures) provides a mature model for the "dynamic trust and reputation" gap identified in Post 4.
- The legal effect of electronic identification under eIDAS (mutual recognition across EU member states) is relevant to "cross-domain security boundaries" -- a problem the IETF drafts approach from a purely technical angle.
The Ericsson/EDHOC work mentioned in Posts 2 and 3 is architecturally adjacent to eIDAS requirements but is never connected to it.
### 3. NIS2 Directive and critical infrastructure
The NIS2 Directive (Directive 2022/2555), applicable from 18 October 2024, imposes cybersecurity risk-management measures and incident reporting obligations on entities in critical sectors. The series discusses autonomous network operations (93 drafts) and telecom agent deployments without mentioning that telecom operators deploying AI agents are NIS2-obligated entities.
The gap analysis scenario of AI agents managing telecom infrastructure during a major outage (Post 4) directly involves NIS2-covered operations. Incident reporting timelines under NIS2 (24-hour early warning, 72-hour notification) interact with the error recovery gap -- if an agent causes or extends an outage, the NIS2 clock starts.
### 4. Cyber Resilience Act (CRA)
The CRA (Regulation 2024/2847) imposes cybersecurity requirements on products with digital elements, including software. Agent protocols and their implementations will fall under CRA obligations regarding vulnerability handling, security updates, and conformity assessment. The series' discussion of "Agent Firmware/Model Update Security" (HIGH gap) maps to CRA requirements but is not framed as a regulatory obligation.
### 5. German Telecom Law (TKG) and AI in network management
The series highlights that Chinese telecom organizations focus heavily on autonomous network operations. For German/EU telecom operators deploying such agent-based network management, SS 165 TKG (technical protective measures) and SS 168 TKG (incident reporting) impose domestic obligations beyond NIS2. The Bundesnetzagentur has authority to require specific security measures. This is relevant context for the "European telecoms as bridge-builders" narrative in Posts 2 and 5.
---
## Improvement Suggestions
### 1. Add a regulatory context paragraph to Post 1 or Post 4
The series positions the safety deficit as its signature finding. A brief paragraph contextualizing this within the EU regulatory landscape (AI Act, NIS2, CRA, product liability) would make the analysis more actionable for EU-based readers and more legally accurate. The 4:1 safety ratio is not just a community choice; for EU-deployed systems, it is a compliance risk.
### 2. Distinguish IETF IPR policy from open standards
Post 7 describes the tool as "open source" and the database as available. The series discusses IETF drafts without noting the IETF's IPR policy (BCP 79, RFC 8179). IETF participants are required to disclose known IPR claims. For a series advising builders to "watch these drafts" and "design for the DAG," a note about IPR and FRAND licensing would be prudent. Some of the proposed protocols may carry patent claims that affect implementation freedom.
### 3. Frame the geopolitics discussion with care
Post 2 discusses Chinese institutional dominance and "Western absence" in terms that could be read as geopolitical advocacy rather than data-driven observation. Statements like "the standards that will govern how AI agents identify, authenticate, and communicate on the internet are being written by a remarkably narrow group" carry implications.
From a German/EU legal perspective, EU competition law and the EU Foreign Subsidies Regulation (Regulation 2022/2560) provide frameworks for assessing foreign influence in standard-setting. The series would benefit from a brief note that the IETF process is open, consensus-based, and has mechanisms (rough consensus, running code) to mitigate undue influence -- even if the authorship concentration data is concerning.
### 4. Address GDPR implications of agent discovery
Post 5 notes the absence of "privacy-preserving agent discovery" -- that querying for "a medical diagnosis agent" reveals sensitive information. Under GDPR, the query itself could constitute processing of special category data (Art. 9 GDPR, health data). This is not just a gap; it is a legal obstacle to deployment in the EU without privacy-by-design measures. Strengthening this point with a GDPR reference would make it more compelling.
### 5. The "assurance profiles" model should reference EU conformity assessment
Post 6's proposed assurance profiles (L0 through L3) closely parallel the EU AI Act's risk-based approach. Explicitly connecting L2/L3 to EU high-risk AI system requirements would make the architectural proposal more concrete for European audiences and demonstrate that the technical design accounts for regulatory reality.
---
## Post-by-Post Notes
### Post 00 (Series Overview)
- No legal issues. Internal document.
### Post 01 (Gold Rush)
- The claim "AI agents communicating over the internet without agreed-upon identity, security, and interoperability standards is a problem that gets worse every month" is stated as a technical observation. Under EU law, it is also a regulatory compliance problem (AI Act Art. 15, NIS2). Adding this dimension strengthens the claim.
- The 4:1 safety ratio should note that for EU-deployed high-risk systems, this ratio represents potential non-compliance, not merely a community preference.
### Post 02 (Who Writes the Rules)
- The Huawei analysis is data-driven and factual. No legal issues with the presentation.
- The "volume over iteration" section (65% rev-00, pre-meeting submission campaigns) is a legitimate observation about IETF process dynamics. It avoids making claims about intent, which is the correct editorial approach.
- The "Chinese institutional ecosystem" framing is factual but should not be read as implying coordination in the competition-law sense. The IETF is an open forum; coordinated standards participation by companies within a country is normal and lawful.
### Post 03 (OAuth Wars)
- The OAuth cluster analysis is the post most in need of GDPR context. The 14 proposals all address agent authorization, but none addresses the GDPR-specific question: when an agent processes personal data on behalf of a user, what is the legal basis? OAuth delegation is not automatically GDPR-compliant delegation. The controller-processor relationship (Art. 28 GDPR) requires a data processing agreement. None of the drafts described appear to address this.
- The "chained delegation" gap is a GDPR problem as well as a technical one: sub-processors under Art. 28(2)/(4) GDPR require specific or general written authorization from the controller.
### Post 04 (What Nobody Builds)
- The strongest post from a regulatory perspective. The three critical gaps (behavior verification, resource management, error recovery) all map to EU AI Act requirements for high-risk systems.
- The hospital scenario should note that the Medical Devices Regulation (MDR) and the AI Act both apply, and that CE marking for the AI system would require addressing these gaps *before* deployment, not after standards emerge.
- The "4:1 ratio revisited" structural analysis is legally significant: it suggests that the current standards development process may not produce the technical mechanisms needed for EU regulatory compliance within the enforcement timeline (August 2026).
### Post 05 (1262 Ideas / Convergence)
- "Privacy-preserving agent discovery" is identified as absent. This should reference Art. 25 GDPR (data protection by design and by default) as a legal requirement, not just a nice-to-have.
- "Agent cost and billing" -- absent from the corpus -- has implications under the EU's Payment Services Directive (PSD2) and the upcoming PSD3 if agents handle financial transactions.
### Post 06 (Big Picture)
- The "dual regime" (relaxed vs. regulated) framing is excellent and maps well to the AI Act's risk-based approach. The post should make this mapping explicit rather than leaving it implicit.
- The "assurance profiles" proposal (L0-L3) should note that L2/L3 may not be optional for EU deployments -- the AI Act mandates specific technical documentation, logging, and human oversight for high-risk systems. "Dial up" is the wrong metaphor if the law requires maximum assurance.
- The prediction "within 18 months, the safety deficit will begin to close -- not from IETF drafts but from regulatory pressure" should be updated to reflect that the AI Act is already in force and enforcement begins August 2026 -- this is not 18 months away; it is 5 months away at publication time.
- The EU AI Act is not merely "regulatory pressure"; it imposes specific technical requirements with significant penalties (up to 35 million EUR or 7% of global annual turnover under Art. 99).
### Post 07 (How We Built This)
- The description of the analysis pipeline (Claude for analysis, Ollama for embeddings) raises no legal issues, but should note that sending full draft texts to the Claude API involves transmitting potentially IPR-encumbered content to a third-party processor. Under GDPR, this is likely non-personal-data processing and not regulated, but IETF IPR policies (Note Well) could be relevant.
- The "open source" claim for the tool should be paired with a license reference. Under German law (UrhG), software is protected by copyright. Without a stated license, the default is "all rights reserved."
### Post 08 (Agents Building the Analysis)
- The meta-irony section mapping the team's coordination needs to IETF gaps is clever and legally unproblematic.
- The "silent failure" anecdote (Writer's revisions not persisting) is a useful illustration. In a regulated context, this would constitute a failure of the AI Act's Art. 12 logging requirement -- the system reported success while the output was wrong. This parallel could be made explicit.
### State of Ecosystem (Vision Document)
- The three 2027 scenarios and two 2028 equilibria are well-constructed. Scenario A ("fragmentation wins") would be particularly problematic under EU law, as fragmented standards make conformity assessment more expensive and less reliable.
- The "what builders should do today" section advises building human oversight "now, not later." Under the AI Act, this is a legal requirement for high-risk systems, not just engineering advice. Framing it as such would strengthen the recommendation.
---
## Summary of Priority Actions
1. **Post 3**: Add GDPR-aware footnote on OAuth "consent" vs. GDPR consent; note controller-processor implications of chained agent delegation.
2. **Post 4**: Acknowledge that the hospital scenario is already regulated under the AI Act and MDR; the gap is technical implementation, not legal accountability.
3. **Post 6**: Make the AI Act mapping explicit (assurance profiles to conformity assessment); correct the timeline (enforcement begins August 2026, not "18 months").
4. **Cross-series**: Add a brief regulatory context paragraph (1-2 sentences) to Post 1 establishing that the safety deficit has legal implications under EU law, not just engineering ones.
5. **Post 7**: Add open-source license reference; note IETF IPR context for the "watch these drafts" advice.

278
data/reports/reviews/review-science.md Normal file
View File

@@ -0,0 +1,278 @@
# Scientific Review -- IETF Draft Analyzer
*Reviewed 2026-03-08 by Scientific Reviewer agent*
---
## Executive Summary
The IETF Draft Analyzer is an ambitious and largely well-executed landscape analysis. The core findings -- the 4:1 capability-to-safety ratio, the fragmentation across 120+ A2A protocols, the dominance of Chinese technology companies -- are supported by the data and would withstand scrutiny from IETF participants. However, the methodology has several significant weaknesses that should be disclosed transparently, and several claims in the blog posts overstate what the data can actually support.
**Overall assessment**: Publishable with revisions. The research is directionally sound but needs (a) clearer methodological caveats, (b) correction of data inconsistencies, and (c) hedging of several definitive claims.
---
## 1. Methodological Issues
### 1.1 LLM-as-Judge Without Calibration (CRITICAL)
The entire rating system relies on Claude (Sonnet) as the sole judge for five dimensions (novelty, maturity, overlap, momentum, relevance) on a 1-5 scale. This is the central methodological weakness.
**Problems:**
- **No inter-rater reliability**: There is no comparison against human expert ratings. Even a small calibration set (20-30 drafts rated by an IETF participant) would substantially strengthen the methodology.
- **No intra-rater consistency check**: The same draft is never rated twice to measure Claude's self-consistency. Prompt hash caching means re-runs return cached results, so actual consistency is untested.
- **Rating prompt is minimal**: The `RATE_PROMPT_COMPACT` gives Claude a draft's abstract (truncated to 2000 chars), name, date, and page count -- but no access to the full text for rating purposes. This means ratings are abstract-based, not content-based. For maturity and overlap scores in particular, the abstract is insufficient.
- **Batch effects**: Batch rating (`BATCH_PROMPT`) processes 5 drafts together. Position effects (first vs. last in batch) and comparison effects (a mediocre draft looks better next to weak ones) are uncontrolled. Abstracts are also truncated more aggressively (1500 chars vs. 2000) in batch mode.
- **Relevance inflation**: The relevance distribution is heavily right-skewed (196 drafts at 4, 98 at 5, only 38 at 1-2). This suggests Claude is generous with relevance for keyword-matched drafts, making the metric less discriminating than it should be. Only 38 of 434 drafts are rated relevance <= 2, despite clear false positives in the corpus (see Section 3.1).
**Recommendation:** Add a "Limitations" section to the methodology post (Post 7) that explicitly states: ratings are LLM-generated from abstracts only, without human calibration. Consider running a calibration study with 5 domain experts rating 25 drafts each.
### 1.2 Idea Extraction Quality is Unknown
The pipeline extracts "1-4 ideas" per draft via LLM, but there is no precision/recall measurement.
**Current state of the data:**
- The database now contains only **419 ideas** across **377 drafts** (1.1 ideas/draft average), with 337 drafts having exactly 1 idea, 38 having 2, and 2 having 3.
- The blog posts reference "1,262 ideas" and "1,780 ideas" -- these numbers are stale and do not match the current database (419).
- The near-uniform "1 idea per draft" distribution (80% of drafts) suggests the extraction prompt may be over-aggressive in merging or the dedup step removed too many.
**Problems:**
- **Recall**: Many substantial drafts probably define more than one novel contribution. A 1-idea-per-draft average is suspiciously low.
- **Precision**: Without ground truth, we cannot know how many extracted "ideas" are restatements of the abstract vs. genuine technical contributions.
- **Batch vs. individual quality**: Batch extraction (using Haiku, abstract-only at 800 chars) produces different results than individual extraction (Sonnet, abstract + 3000 chars of full text). The quality difference is unquantified.
- **Data staleness**: Blog post 5 ("Where 361 Drafts Converge") cites 1,692 unique ideas. The current database has 419. Either the ideas were mass-deleted (via dedup) or regenerated. This needs reconciliation.
**Recommendation:** Run individual extraction on a sample of 30 drafts and compare to batch results. Establish expected ideas-per-draft range by manually analyzing 10 drafts.
### 1.3 Gap Analysis is Single-Shot LLM Generation
The gap analysis is generated by a single Claude call (`GAP_ANALYSIS_PROMPT`) that receives compressed statistics about the landscape (category counts, top ideas, overlap summary). This is essentially asking Claude to brainstorm gaps based on metadata.
**Problems:**
- **No systematic coverage analysis**: A rigorous gap analysis would compare the corpus against a reference taxonomy of what a complete AI agent ecosystem requires. The current approach relies on Claude's general knowledge rather than a structured framework.
- **Overlap summary is circular**: The "overlap_summary" fed to the gap prompt is just the top-5 categories by count with a generic "high internal overlap" label. This does not tell Claude which specific technical areas overlap -- it just restates what the categories are.
- **Evidence quality varies**: Some gap evidence is specific ("only 44 safety/alignment drafts") while others are vague ("lack agent-specific resource protection mechanisms"). The evidence field should cite specific drafts that partially address each gap.
- **Blog post gap list diverges from database**: The gaps.md report lists 12 gaps (from the database), but blog post 04 lists a different set of 12 gaps with different names and severities. It is unclear which gap analysis is canonical.
**Recommendation:** Ground the gap analysis in a reference architecture (e.g., NIST AI RMF, or an explicit agent ecosystem reference model). Cite specific drafts that partially address each gap rather than category-level statistics.
### 1.4 Clustering Methodology is Naive
The `find_clusters` method uses greedy single-linkage clustering at a fixed 0.85 cosine similarity threshold.
**Problems:**
- **Single-linkage effect**: Once a draft joins a cluster, all drafts similar to it (but not necessarily to the seed) join too. This can create "chaining" where semantically distant drafts end up in the same cluster.
- **Threshold not justified**: The 0.85 threshold for "topically overlapping" and 0.90 for "near-duplicates" are not empirically validated. Different embedding models and text representations would produce different similarity distributions.
- **No comparison to baselines**: How does the 42-cluster result at 0.85 compare to, say, k-means or DBSCAN? The absence of comparison makes it impossible to assess whether 42 is "right."
- **Embedding model limitations**: nomic-embed-text is a competent general-purpose embedding model, but it was not trained specifically for technical/standards document similarity. Domain-specific models or fine-tuned embeddings might produce quite different clusters.
**Recommendation:** Report the similarity score distribution (histogram) and explain why 0.85 was chosen. Consider running DBSCAN as a comparison method.
### 1.5 Embedding Input is Inconsistent
`embed_draft` combines title + abstract + first 4000 chars of full text. But 57 drafts in the database have no ideas extracted, and it is unclear whether all drafts have full text downloaded. Drafts embedded with vs. without full text will have systematically different embedding quality, which affects similarity comparisons.
---
## 2. Unsupported Claims
### 2.1 Blog Post 01 ("Gold Rush")
- **"Nearly 1 in 10 new Internet-Drafts is about AI agents"**: The 9.3% figure for Q1 2026 needs a denominator source. Where does the "1,748 total IETF drafts in Q1 2026" come from? This is not from the analyzer's data; it appears to be external. If the figure is correct it is a strong finding, but the source must be cited.
- **"4,231 cross-references"**: This citation analysis is mentioned but the methodology for extracting citations is not described anywhere in the codebase. How were references parsed? Was this a separate analysis?
- **"The acceleration is not gradual. It is a step function that began in mid-2025"**: This is a strong mathematical claim. A step function implies discontinuity. The data shown (9 drafts in 2024, 190 in 2025) is more consistent with exponential growth than a step function. The framing should be: "rapid acceleration" not "step function."
### 2.2 Blog Post 04 ("What Nobody's Building")
- **The hospital drug-dispensing scenario**: This is vivid but ungrounded. No IETF draft addresses medical device agent systems, and the scenario implies current standards failures that have not occurred. The framing should clarify this is a thought experiment about future risks, not a description of current failures.
- **"0 ideas addressing cross-protocol translation"**: This claim depends entirely on the idea extraction quality. If extraction produces only 1 idea per draft (as current data suggests), many relevant technical contributions may simply not be captured.
### 2.3 Blog Post 05 ("1,262 Ideas")
- **The entire post's numbers are stale**: It references 1,692 unique ideas and 1,780 total. The database now has 419. The convergence analysis ("96% appear in exactly one draft") and cross-org analysis ("628 ideas with cross-org validation") need to be re-verified against the current database.
- **"SequenceMatcher at 0.75 threshold"**: This fuzzy matching methodology is mentioned in the blog post but does not appear in the codebase. Where was this analysis performed? If it was a one-off script, it is not reproducible.
### 2.4 Category Counts Are Inflated by Multi-Assignment
The blog post reports "Data formats and interoperability: 145 drafts (40%)" and "A2A protocols: 120 drafts (33%)." Since drafts average 2.37 categories each, many drafts appear in multiple categories. The post does disclose this ("percentages exceed 100%") but the visual effect of listing 10 categories that sum to >> 100% can mislead. The actual number of truly unique-to-category drafts is not reported.
---
## 3. Missing Context
### 3.1 False Positives in the Corpus
The keyword-based search strategy produces false positives that inflate the corpus. Examples confirmed in the database:
- `draft-pan-tsvwg-pie` (PIE bufferbloat algorithm) -- rated relevance 3, which is too high
- `draft-ietf-hpke-hpke` (Hybrid Public Key Encryption) -- rated relevance 5, clearly wrong for an AI/agent analysis
- `draft-ietf-suit-firmware-encryption` (SUIT manifests) -- rated relevance 4
- `draft-eggert-mailmaint-uaautoconf` (email autoconfiguration) -- rated relevance 4
These drafts match keywords like "agent" (in "user agent"), "autonomous," or "intelligent" in ways unrelated to AI agents. The corpus likely contains 30-50 such false positives (the 38 drafts rated relevance <= 2 are the obvious ones, but many false positives are rated 3-4 by the generous LLM judge).
**Impact:** A ~10% false positive rate in the corpus affects all derived statistics. The "361 drafts" (or now 434) figure should be qualified.
**Recommendation:** Implement a relevance filter. Exclude drafts with relevance <= 2 from all analyses. Better yet, manually review the 50 lowest-scored drafts and create an exclusion list.
### 3.2 Missing Literature Context
The analysis would benefit from referencing:
- **FIPA (Foundation for Intelligent Physical Agents)**: The original agent communication standards body. Their ACL (Agent Communication Language) and Agent Platform specifications from 1997-2004 are the direct ancestors of modern A2A protocols. The absence of FIPA from the analysis is a significant gap -- an IETF participant familiar with agent standards history would notice immediately.
- **W3C Web of Things (WoT)**: The WoT Architecture and Thing Description specifications address agent discovery and interoperability in IoT contexts. Several IETF drafts build on or compete with WoT concepts.
- **IEEE P2048 (Standard for VR/AR Agent Interoperability)** and **IEEE P3394 (Standard for Trustworthy AI Agents)**: These are concurrent standardization efforts that the IETF landscape should be compared against.
- **OASIS TOSCA, Open Agent Architecture (OAA)**: Prior art in agent orchestration and service composition.
- **Academic MAS research**: The multi-agent systems community (AAMAS, JAIR, JAAMAS) has decades of work on agent coordination, trust, and verification. The analysis should at minimum reference survey papers on MAS challenges.
### 3.3 Temporal Analysis Gaps
The growth rate claims in Post 01 would be stronger with:
- Comparison to other fast-growing IETF topics (e.g., QUIC, post-quantum crypto)
- Month-by-month submission data rather than annual/quarterly aggregates
- Distinction between individual drafts and WG-adopted drafts (which indicate greater organizational commitment)
### 3.4 Geographic and Organizational Bias
The author analysis reveals Chinese companies (Huawei: 66 drafts, China Mobile: 35, China Telecom: 24, China Unicom: 21) collectively account for ~34% of all drafts. This concentration is noted but its implications are underexplored:
- Is this ratio typical for the IETF, or unusual for this topic area?
- Does this concentration affect which problems get standardized?
- Are there language/translation barriers affecting the quality assessment?
---
## 4. Data Integrity Issues
### 4.1 Category Normalization Incomplete
The database contains both canonical short names and legacy long names for the same categories:
- "A2A protocols" (139 drafts) vs. "Agent-to-agent communication protocols" (16 drafts) -- these should be the same
- "Agent discovery/reg" (75 drafts) vs. "Agent discovery / registration" (14 drafts)
- "Agent identity/auth" (139 drafts) vs. "Identity / authentication for AI agents" (13 drafts)
The `normalize_category` function exists in the code and is applied on read in many places, but the raw database values were never migrated. This means raw SQL queries (like those in reports) may produce incorrect category counts unless normalization is applied.
**Impact:** Category counts cited in reports and blog posts may be inaccurate by 5-15% depending on which code path generated them.
**Recommendation:** Run a one-time migration to normalize all category values in the `ratings` table.
### 4.2 Ideas Count Discrepancy
The database has 419 ideas (as of this review). Reports reference 1,262 or 1,780. Either:
- Ideas were mass-deleted via dedup (the `dedup_ideas` function exists with 0.85 threshold)
- The database was regenerated with different parameters
- Multiple idea extraction runs produced different results
This needs to be resolved. If the current 419 ideas are correct (post-dedup), then all blog post statistics about idea counts, convergence, and fragmentation must be updated.
### 4.3 57 Drafts Have No Ideas
57 of 434 drafts have no extracted ideas. If these are legitimately off-topic (false positives that should return empty arrays), this is correct. If they are processing failures, they represent missing data.
### 4.4 Database Grew from 361 to 434
The reports and blog posts reference 361 drafts. The database now contains 434. All published statistics are stale. This is not a methodology issue per se, but any publication should use consistent numbers.
---
## 5. Improvement Suggestions
### 5.1 Add a Calibration Study (HIGH PRIORITY)
Select 25 representative drafts spanning all categories. Have 3-5 domain experts rate them on the same 5 dimensions. Compare against Claude's ratings. Report Spearman correlation, Cohen's kappa, or similar inter-rater metrics. This single addition would transform the methodology from "interesting exploratory analysis" to "validated automated assessment."
### 5.2 Define a Reference Architecture
Create an explicit "ideal agent ecosystem" reference model (identity, discovery, communication, authorization, monitoring, safety, governance, lifecycle). Map every draft and gap against this model. This makes the gap analysis systematic rather than ad hoc.
### 5.3 Report Confidence Intervals
For key statistics (category counts, idea counts, similarity thresholds), report sensitivity analyses. What happens to the gap analysis if the similarity threshold is 0.80 or 0.90 instead of 0.85? What if relevance < 3 drafts are excluded?
### 5.4 Version the Analysis
Timestamp all statistics. When the corpus grows from 361 to 434, make it clear which numbers apply to which version of the analysis. Consider a "snapshot" system: v1 = 260 drafts (Feb 2026), v2 = 361 drafts (Mar 2026), v3 = 434 drafts (current).
### 5.5 Publish the Methodology as Reproducible
The blog posts describe methodology in prose but do not provide enough detail for replication. Consider publishing the prompts, thresholds, and pipeline configuration as a supplementary appendix.
### 5.6 Address Ethical Dimensions
The analysis identifies gaps in safety and governance but does not engage with the ethical dimensions of autonomous agent standardization. Questions worth addressing:
- Should the IETF standardize capabilities before safety mechanisms exist?
- What are the risks of the 4:1 capability-to-safety ratio becoming embedded in standards?
- How does geographic concentration in standards development affect global equity?
---
## 6. Taxonomy & Categorization Assessment
### 6.1 Category Scheme
The 11 categories (`CATEGORIES_SHORT` in analyzer.py) are reasonable but have issues:
- **"Other AI/agent"** is a catch-all that weakens analysis. 34 drafts in this category deserve better classification.
- **"Data formats/interop"** is too broad. At 171 drafts (after normalization), it is the largest category but encompasses everything from YANG models to JSON schemas to COSE signing. Sub-categorization would be more informative.
- **Multi-assignment without weighting**: Drafts receive 2.37 categories on average. A primary/secondary distinction would improve precision.
- **No negative categories**: The system cannot mark a draft as "not about AI agents" -- it can only assign categories from the fixed list. A "false positive / tangentially related" category would help.
### 6.2 Gap Classification
The 4-level severity scale (critical, high, medium, low) is reasonable but the threshold between levels is not defined. What makes a gap "critical" vs. "high"? The current distinction appears to be: critical = safety-related, high = functionality-related, medium = optimization-related. This should be stated explicitly.
---
## 7. Post-by-Post Notes
### Post 00 (Series Overview)
Not reviewed (meta-navigation page).
### Post 01 (Gold Rush)
- Strongest post. Claims are mostly well-supported by data.
- Growth rate table needs source citation for total IETF draft counts.
- "Step function" language is too strong; use "rapid acceleration."
- The 4:1 safety deficit framing is the most compelling finding.
### Post 02 (Who Writes the Rules)
Not reviewed in detail.
### Post 03 (OAuth Wars)
Not reviewed in detail.
### Post 04 (What Nobody's Building)
- Hypothetical scenarios are effective but should be explicitly labeled as projections, not current failures.
- Gap list should match the database gap list. Currently there are discrepancies.
- The "0 ideas addressing cross-protocol translation" claim depends on extraction quality now in question.
### Post 05 (1,262 Ideas)
- **Needs full rewrite with current data.** The idea counts (1,262/1,692/1,780 referenced at various points) do not match the database (419). All convergence and fragmentation statistics derived from idea data are unreliable until reconciled.
- The fuzzy matching methodology (SequenceMatcher at 0.75) is not in the codebase and cannot be verified.
### Post 06 (Big Picture)
Not reviewed in detail.
### Post 07 (How We Built This)
- Should contain the "Limitations" section that currently does not exist anywhere.
- Should document all thresholds and their justifications.
### Post 08 (Meta Post)
Not reviewed in detail.
---
## 8. Summary of Recommendations by Priority
| Priority | Issue | Action |
|----------|-------|--------|
| CRITICAL | Ideas data inconsistency (419 vs 1,262+) | Reconcile database and blog post numbers |
| CRITICAL | No LLM rating calibration | Add calibration study or prominent caveat |
| HIGH | Category normalization incomplete in DB | Run migration script |
| HIGH | False positives in corpus (~30-50 drafts) | Implement relevance filter, manual review |
| HIGH | Missing FIPA/W3C/IEEE context | Add related work section |
| MEDIUM | Clustering methodology naive | Report similarity distribution, compare methods |
| MEDIUM | Gap analysis not grounded in reference arch | Define explicit reference model |
| MEDIUM | Stale numbers (361 vs 434 drafts) | Version all statistics |
| LOW | Ethical dimensions unaddressed | Add section in final post |
| LOW | Batch vs individual extraction quality | Run comparison study |
---
*This review was generated by reading all source code (analyzer.py, embeddings.py, fetcher.py, config.py, db.py, models.py), querying the database directly, and reviewing all reports and blog posts. The goal is to strengthen the analysis for publication, not to diminish the substantial work already done.*

333
data/reports/reviews/review-statistics.md Normal file
View File

@@ -0,0 +1,333 @@
# Statistical Review
Reviewed: 2026-03-08
Reviewer: Statistics & Data Analysis Agent (Claude Opus 4.6)
Scope: All blog posts (00-08), data packages (00-06), master stats, and key reports -- cross-checked against `data/drafts.db` via sqlite3 queries.
---
## Data Integrity Issues
### CRITICAL: Database Has Grown Beyond Blog Series Claims
The blog series consistently claims **361 drafts, 557 authors, 1,780 ideas, and 12 gaps**. The current database contains:
| Metric | Claimed | Actual (DB) | Delta |
|--------|---------|-------------|-------|
| Total drafts | 361 | **434** | +73 (20% more) |
| Total authors | 557 | **557** | Match |
| Total ideas | 1,780 | **419** | **-1,361 (76% fewer)** |
| Total gaps | 12 | **11** | -1 |
| Total ratings | 361 | **434** | +73 |
| Total embeddings | 361 | **434** | +73 |
| Draft-author links | 1,057 | **1,057** | Match |
| LLM cache entries | 703 | **1,397** | +694 |
**Root cause**: The database was updated on 2026-03-07 with a new fetch of 431 drafts, bringing the total to 434. The blog series was written against a snapshot taken around 2026-03-03. The master stats file (`00-master-stats.md`) is dated 2026-03-03 and reflects the 361-draft corpus. However, the blog posts do not carry a "data frozen as of" disclaimer -- they state numbers as absolute facts.
**Recommendation**: Add a clear data freeze date to each blog post header (e.g., "Data current as of 2026-03-03, reflecting 361 of 434 drafts now in the database"). Alternatively, update all posts to reflect the 434-draft corpus.
### CRITICAL: Ideas Count Mismatch (1,780 Claimed vs 419 in DB)
The most serious discrepancy. The `ideas` table contains only **419 rows**, not 1,780. The idea type distribution also diverges sharply:
| Type | Claimed | Actual (DB) |
|------|---------|-------------|
| mechanism | 663 | 68 |
| architecture | 280 | 95 |
| pattern | 251 | 35 |
| protocol | 228 | 96 |
| requirement | 171 | 42 |
| extension | 168 | 79 |
| framework | 9 | 3 |
| format | -- | 1 |
| other | 10 | -- |
Only 377 of 434 drafts have any ideas extracted. The 1,780 figure may come from a prior pipeline run whose results were overwritten, or from an in-memory analysis that was not persisted. Either way, the blog series' core claims about "1,780 ideas," "96% appear in only one draft," "628 cross-org convergent ideas (43% of 1,467 clusters)," and the entire idea taxonomy are **not reproducible from the current database**.
**Recommendation**: Re-run idea extraction to populate the database, or clearly note that the 1,780 figure comes from a specific pipeline run that is no longer reflected in the DB. This is the single most important data integrity issue -- Post 5's entire thesis rests on these numbers.
### HIGH: Gap Count and Topics Differ
The DB has **11 gaps**, not 12. The gap topics in the database are:
1. Multi-Agent Consensus Protocols
2. Agent Behavioral Verification
3. Cross-Protocol Agent Migration
4. Real-Time Agent Rollback Mechanisms
5. Agent Resource Accounting and Billing
6. Federated Agent Learning Privacy
7. Agent Capability Negotiation
8. Cross-Domain Agent Audit Trails
9. Agent Failure Cascade Prevention
10. Human Override Standardization
11. Agent Performance Benchmarking
The blog series lists different gap topics (e.g., "Agent Resource Exhaustion Protection" vs DB's "Agent Resource Accounting and Billing"; "Agent Error Recovery and Rollback" vs "Real-Time Agent Rollback Mechanisms"). Post 4's gap list appears to be a curated/rewritten version. The blog's 12-gap list includes "Cross-Protocol Translation" and "Agent Data Provenance" which do not appear as named gaps in the DB.
**Recommendation**: Reconcile the gap list. Either the DB was re-run and lost a gap, or the blog presents an edited version. If the latter, this should be acknowledged as editorial synthesis rather than raw pipeline output.
### HIGH: Composite Rating Calculations Inconsistent
Multiple scoring methodologies are used without disclosure:
| Draft | Blog Score | 5-dim Composite (DB) | 4-dim (excl overlap) |
|-------|-----------|----------------------|----------------------|
| draft-aylward-daap-v2 | 4.8 (Post 1) | 4.0 | 4.75 |
| draft-cowles-volt | 4.8 (Post 1) | 4.0 | 4.75 |
| draft-guy-bary-stamp-protocol | 4.6 (Post 1) | 3.8 | 4.5 |
| draft-drake-email-tpm-attestation | 4.6 (Post 1) | 3.8 | 4.5 |
Post 1 claims DAAP and VOLT scored "4.8" -- this matches neither the 5-dimension composite (4.0) nor the 4-dimension composite excluding overlap (4.75). The master stats correctly uses 4.75 for the same drafts. Post 1 appears to round up (4.75 -> 4.8, 4.5 -> 4.6), which inflates perceived quality.
The "average score" also varies: Post 1 says "3.38/5.0", the master stats say "3.32" (novelty average), the DB 5-dim average is 3.13, and the 4-dim average is 3.27.
**Recommendation**: Pick one composite calculation, document it, and use it consistently. The 4-dim composite (excluding overlap, since overlap measures redundancy rather than quality) is defensible, but the rounding from 4.75 to 4.8 is not. Use exact values.
### MEDIUM: Monthly Draft Counts Differ Between Sources
The master stats growth curve and the actual DB monthly counts diverge:
| Month | Master Stats | Actual DB |
|-------|-------------|-----------|
| 2024-01 | 3 | **7** |
| 2024-02 | 1 | **3** |
| 2024-04 | 1 | **6** |
| 2024-09 | 2 | **11** |
| 2025-10 | 67 | **61** |
| 2025-11 | 61 | **53** |
| 2026-01 | 54 | **51** |
| 2026-02 | 86 | **85** |
| 2026-03 | 22 | **56** |
The master stats show a total of 361 across all months; the DB shows 434. Some of this is explained by the 73 new drafts fetched after the data freeze, but the per-month figures for 2024 are also significantly different (suggesting earlier months got new drafts from the keyword expansion that are counted differently).
The "43x acceleration" claim (from ~2/mo to 86/mo) uses the lowest trough and highest peak, which is cherry-picking. A more honest measure would compare rolling averages.
### MEDIUM: Huawei Draft Counts Vary Across Posts
| Source | Huawei Drafts | Huawei Authors |
|--------|-------------|----------------|
| Post 1 | 66 | 53 |
| Post 2 | 66 | 53 |
| Data Package 02 | "~60+ unique" | "~40+ unique" |
| Master Stats | 57+ | 28+ |
| Actual DB (all Huawei entities) | **69 unique drafts** | multiple entities |
| DB "Huawei" entity only | 39 | 32 |
The consolidation of Huawei sub-entities (Huawei, Huawei Technologies, Huawei Canada, Huawei Singapore, etc.) is done informally. The blog confidently states "53 authors, 66 drafts" but the data package says "~60+ unique drafts, ~40+ unique authors (some overlap)." The actual DB shows 69 unique drafts across all Huawei-named affiliations. The author count depends entirely on deduplication, which is described as "hand-curated" with "40+ mappings."
**Recommendation**: Document the exact normalization rules used to arrive at "53 authors, 66 drafts" and make them reproducible.
---
## Methodological Concerns
### Sampling Bias
The dataset is keyword-filtered (12 keywords across draft names and abstracts). Multiple posts draw sweeping conclusions about "the IETF's AI agent landscape" without sufficient caveats about what this filter captures and misses.
Specific concerns:
- Post 1 claims "nearly 1 in 10 new Internet-Drafts is about AI agents" (9.3%). This figure depends on the denominator (total IETF drafts per year) which is stated but not sourced. Where do the numbers 1,651 (2024) and 2,696 (2025) come from? Are they verifiable?
- The keyword "intelligent" likely captures many non-agent-related drafts about intelligent networking, QoS, etc. The keyword "autonomous" captures autonomous systems (AS) networking drafts. No false-positive analysis is presented.
- Post 7 mentions "~90% accuracy" from spot-checking 50 drafts but provides no breakdown of error types, no inter-rater reliability, and no details on the spot-check methodology.
### Rating Methodology (LLM-as-Judge)
The 1-5 rating scale scored by Claude is presented with minimal caveats in the blog posts. Key issues:
1. **No inter-rater reliability**: The same LLM rated all drafts. No human baseline or second-model comparison is provided.
2. **Abstract-only analysis**: Post 7 acknowledges switching from full-text to abstract-only analysis for ratings, claiming "equivalent ratings." No evidence is presented for this equivalence claim.
3. **Overlap dimension ambiguity**: The "overlap" dimension measures redundancy with other drafts, but since the LLM rates each draft independently, it cannot know the full corpus. The overlap score likely reflects the LLM's general knowledge of the field, not corpus-specific similarity.
4. **Score compression**: All ratings are on a 1-5 scale with integer values only. The max composite (5-dim) is 4.2 and the min is 1.8. The effective range is narrow, making distinctions between drafts less meaningful than the precise decimal composites suggest.
### Clustering and Similarity
- The 0.85 and 0.90 cosine similarity thresholds for overlap clusters are stated but not justified. What threshold sensitivity analysis was performed?
- The "25+ near-duplicate pairs at 0.98" claim is used to argue for deduplication to "roughly 300 distinct proposals" -- but 25 duplicate pairs would reduce the count by at most 25, not 61.
- The SequenceMatcher threshold (0.75) for fuzzy idea matching is stated but not validated. How many false positives does this produce?
### Cross-Org Convergence (628 Ideas)
The 628 cross-org convergent ideas figure is the blog series' lead metric for Post 5. However:
- The methodology (SequenceMatcher at 0.75 threshold across organizational boundaries) is described but the underlying data is not in the DB (only 419 ideas exist).
- No precision/recall analysis is presented. At a 0.75 sequence match threshold, generic titles like "Agent Communication Framework" will match across many drafts regardless of actual technical similarity.
- The claim "43% of unique idea clusters have cross-org validation" depends on the denominator (1,467 unique clusters), which itself depends on the 1,780 raw count that is not reproducible from the DB.
---
## Misleading Claims
### 1. "4:1 Safety Deficit" Ratio
This ratio is presented as the series' signature metric, but its calculation shifts:
- Master stats says "~8:1 capability-to-safety" (after keyword expansion)
- Data package 01 says the safety ratio "improved from 4:1 due to keyword expansion"
- Posts 1-6 consistently use "4:1" as the headline
- Data package 06 says "45 safety drafts vs 316 capability drafts = 7:1"
- The deep analysis shows monthly ratios from 1.5:1 to 21:1
The blog presents "4:1" as a stable finding when the data shows it varies from 1.5:1 to 21:1 depending on the time period and from 4:1 to 8:1 depending on whether keyword-expansion drafts are included. The ratio also depends on multi-labeling: a draft tagged as both "A2A protocols" and "AI safety" counts as both capability and safety.
**Recommendation**: Present the ratio with ranges and context, not as a single stable number. The monthly trend data (Task #24) is more informative than any single ratio.
### 2. "36x Growth"
Post 1 claims "36x growth: 2 drafts/month (Jun 2025) to 72 drafts/month (Feb 2026)." The series overview says the same. But:
- Jun 2025 actually had 5 drafts (per DB), not 2
- Feb 2026 had 85 (per DB), not 72 or 86
- Picking the lowest month and highest month inflates the multiplier
- A rolling 3-month average would show more modest but still impressive growth
### 3. "96% of Ideas Appear in Exactly One Draft"
This is presented as evidence of extreme fragmentation. However:
- The idea extraction pipeline produces ~5 ideas per draft by design
- Many extracted "ideas" are draft-specific component descriptions, not standalone proposals
- Post 5 acknowledges this ("most are draft-specific component descriptions") but still leads with the 96% figure as a shock stat
- The true fragmentation question is whether the *problems being solved* are unique, not whether the *component labels* are unique
### 4. "120 A2A Protocol Drafts"
The category count depends on how "A2A protocols" is defined. The master stats says 136 A2A protocol drafts, but the blog posts use 120. Some posts say "136" (Post 4's gap data package), while others say "120" (Posts 1, 3, 4 text). The inconsistency appears to stem from the category count changing between pipeline runs.
### 5. Causal Language
Several claims use causal framing where only correlation exists:
- "The safety deficit is structural, not attitudinal" (Post 4) -- this is an interpretation, not a finding
- "Gap severity correlates with coordination difficulty" (Post 4) -- stated as found, but the correlation is between two human-assigned ordinal variables (severity levels assigned by Claude, coordination difficulty assessed by the Architect) with N=12 data points
- "The organizations doing the most drafting are focused on capability; the organizations doing the best safety work are doing the least drafting" (Post 2) -- the causal implication is that volume and safety focus are inversely related, but this could simply reflect different organizational missions
---
## Improvement Suggestions
### 1. Add a Data Provenance Section
Each blog post should include a brief provenance note: data freeze date, pipeline version, exact query or command used to generate each key number. This would make claims verifiable.
### 2. Standardize the Composite Score
Choose one formula (recommend: 4-dimension excluding overlap, or 5-dimension with clear labeling) and use exact values (not rounded). Document the formula in Post 7 and use it consistently.
### 3. Validate Idea Extraction
Re-run idea extraction to ensure the DB reflects the claimed 1,780 ideas. If the pipeline was run differently (e.g., with a different prompt or batching strategy), document the exact parameters.
### 4. Add Confidence Intervals
For claims like "4:1 ratio," show the range across different time periods and calculation methods. For trend claims, show the underlying monthly data rather than cherry-picked endpoints.
### 5. Acknowledge LLM-as-Judge Limitations Prominently
Post 7 mentions LLM validation briefly. The rating methodology should include:
- A caveat in every post that uses ratings
- A note that overlap scores are based on LLM general knowledge, not corpus comparison
- Acknowledgment that abstract-only analysis may miss important content
### 6. De-duplicate Before Counting
The "361 drafts" count includes known near-duplicates. The blog acknowledges "probably closer to 300 distinct proposals" (Post 3) but continues using 361 everywhere. Either de-duplicate and use the lower number, or present both with context.
---
## Post-by-Post Notes
### Post 00 (Series Overview)
- Internal architecture document; numbers are consistent with master stats (361/557/628/12). No issues as an internal document.
### Post 01 (Gold Rush)
- **Score inflation**: DAAP cited as 4.8, actual 4-dim composite is 4.75, 5-dim is 4.0. STAMP cited as 4.6, actual is 4.5/3.8. VOLT cited as 4.8, actual is 4.75/4.0.
- **Category table inconsistency**: The post lists "Data formats and interoperability: 145" as the top category, but the master stats shows "A2A protocols: 136" as the top. The post appears to use a different category set than the master stats.
- **Growth figure**: "36x growth" -- cherry-picked from lowest to highest month.
- **"0.5% to 9.3%"**: The denominator (total IETF drafts) is stated but unsourced. The 9.3% figure assumes 1,748 total drafts in Q1 2026 -- where does this come from?
- Average score stated as "3.38" -- does not match any DB calculation (5-dim avg: 3.13; 4-dim avg: 3.27; novelty avg: 3.27).
- **"~1,700 technical ideas"**: Post says "roughly 1,700" in one place; DB has 419.
### Post 02 (Who Writes the Rules)
- Huawei "53 authors, 66 drafts" is stated with confidence but data package says "~60+" with caveats about entity dedup. DB shows 69 unique drafts across Huawei entities.
- "65% are at rev-00" for Huawei -- this figure is for "Huawei" entity only (57 drafts), not the combined 66/69. The denominator matters.
- "43 were submitted in the four weeks before IETF 121" -- data package says "43 of 69 across all entities." The blog says "43" out of Huawei's "66" implying 65%, vs data package's "62% of 69."
- "115 (23%) co-author with people from both Chinese and Western organizations" -- not verifiable from current DB without running the centrality analysis.
- Ericsson "4.8 average revision" claim (line 149) is inconsistent with data package showing Ericsson avg rev as 4.8 -- this appears correct.
### Post 03 (OAuth Wars)
- The 14-draft OAuth list is well-documented with individual scores.
- Score for DAAP is listed as 4.8 but 4-dim composite is 4.75. Other scores in the table appear to be individual dimension values or different calculations (e.g., STAMP at 4.6 vs 4.5 4-dim).
- The data package actually lists 15 OAuth-related drafts (including draft-mw-spice-actor-chain and draft-gaikwad-south-authorization), but the blog says 14. The blog's list of 14 differs slightly from the data package's 15.
- "25+ near-duplicate pairs" leading to "roughly 300 distinct proposals" is a logical leap. 25 duplicate pairs reduce the count by 25 (one from each pair), yielding 336, not "roughly 300."
### Post 04 (What Nobody Builds)
- Gap count: 12 in blog vs 11 in DB. Gap names differ from DB.
- "Ideas Addressing It" column (52, 117, 6, 0, 90, 5, 4, 10, 5, 26, 5, 79) -- these numbers cannot be verified because the ideas table has only 419 rows, not 1,780. With 419 ideas, these per-gap counts are implausible (they sum to 399, nearly the entire ideas table).
- "Only 6 extracted ideas address [error recovery], and all come from a single draft" -- this is a strong claim. With only 419 ideas in the DB, 6 ideas from one draft is plausible, but the DB has no gap-to-idea mapping table to verify.
- "12 (8.8%) of 136 A2A drafts also address safety" -- this requires the categories JSON field in the drafts table. Not independently verified but plausible.
- "Safety has zero co-occurrence with agent discovery/registration and zero co-occurrence with model serving/inference" -- sourced from deep analysis task #27, which is plausible but not verifiable from current DB without re-running the co-occurrence analysis.
### Post 05 (1,262 Ideas / Where Drafts Converge)
- Title references "1262" in the filename but post content uses 1,780, 1,692, and 628. The filename appears to be from the pre-expansion dataset.
- "1,692 unique technical ideas" -- the DB has 419 ideas. This is the largest disconnect in the entire series.
- "Only 75 show up in two or more drafts" -- not verifiable from current DB.
- "628 ideas where different organizations are working on recognizably similar problems" -- the central claim of the post, not verifiable from current DB.
- The idea taxonomy table (mechanism: 663, architecture: 280, etc.) does not match DB (mechanism: 68, architecture: 95, etc.). Both the counts and the rank order differ.
- The convergence table (A2A Communication Paradigm: 8 orgs, etc.) is not verifiable.
### Post 06 (Big Picture)
- Synthesis post; numbers are drawn from prior posts. Inherits all issues.
- "36 (10%) have been adopted by IETF working groups" -- based on naming convention (`draft-ietf-*`). This could be verified with a query but depends on the 361-draft corpus.
- "WG-adopted drafts score higher on average (3.54 vs. 3.31)" -- this uses 4-dim composite, which is consistent with the rest of the 4-dim usage but not labeled as such.
- "75 cross-draft convergent ideas (628 via fuzzy matching)" -- the parenthetical mixing of two very different numbers is confusing. 75 is exact-title matches; 628 is fuzzy cross-org. These are different metrics measuring different things.
### Post 07 (How We Built This)
- **Database table sizes**: Claims 361 drafts, 1,780 ideas, 557 authors, 1,057 draft_authors, 4,231 draft_refs, 12 gaps, 703 llm_cache. DB now shows 434/419/557/1,057/4,231/11/1,397. Only draft_authors and draft_refs match.
- **"43 CLI commands"**: Not verified but seems high. The source code would need to be checked.
- **Cost figures**: "$3.16 for 260 drafts" and total "~$9" are stated without supporting evidence (no token count logs in the DB). Not falsifiable but also not verifiable.
- **"15 report types"**: Not verified.
- Describes rating as "1-5 scale" which matches the DB (max 5, not 10 as the reviewer checklist suggests).
### Post 08 (Agents Building the Analysis)
- Meta post about the process. Numbers reference those from other posts, inheriting their issues.
- "20+ SQL queries" and "7 data packages" -- plausible but not independently verifiable.
- "30 dev-journal entries" -- could be verified by reading dev-journal.md.
- The cost table sums to "~$9" but the individual line items sum to ~$9.00 (2.50+5.50+0.80+0.20 = 9.00). Consistent.
### State of Ecosystem (Vision Document)
- "36x increase" -- same cherry-picking issue as Post 1.
- Uses "72" drafts/month for Feb 2026 (differs from other sources: 86 in master stats, 85 in DB).
- Otherwise consistent with other posts.
### Master Stats (00-master-stats.md)
- **Gap count**: Lists 12 gaps with different names than DB's 11.
- **Idea count**: 1,780 -- does not match DB's 419.
- **Draft count**: 361 -- does not match DB's 434 (but was correct at data freeze date).
- **Composite scores**: Uses 4-dim composite and gets 4.75 for top drafts -- correct for 4-dim, but unlabeled as such.
- **Category distribution**: Uses different category names/counts than the blog posts in some cases (e.g., master stats: "A2A protocols: 136" vs Post 1: "A2A protocols: 120").
---
## Summary of Findings
**Most Serious Issues** (would undermine credibility if published):
1. Ideas count (1,780 claimed, 419 in DB) -- the foundation for Post 5's thesis is not reproducible
2. Composite score inflation (4.75 rounded to 4.8) and inconsistent calculation methods
3. Gap count (12 vs 11) and topic naming mismatches
**Important Issues** (should be fixed before publication):
4. Draft count stale (361 vs 434)
5. "4:1 ratio" is not stable -- varies 1.5:1 to 21:1 by month
6. "36x growth" cherry-picks endpoints
7. Category counts inconsistent between posts and master stats
**Minor Issues** (polish):
8. Huawei entity deduplication is informal
9. LLM-as-judge caveats are insufficient
10. No false-positive analysis for keyword filtering
11. The "25 duplicate pairs -> roughly 300" arithmetic does not work
**What Holds Up Well**:
- RFC cross-reference counts (4,231) match exactly
- Draft-author link count (1,057) matches exactly
- Author count (557) matches exactly
- The qualitative patterns (Huawei dominance, safety deficit, fragmentation) are directionally sound even if specific numbers vary
- The geopolitical analysis and team bloc detection methodology are well-described
- The cost analysis (~$9 total) is internally consistent

121
data/reports/reviews/verified-counts.md Normal file
View File

@@ -0,0 +1,121 @@
# Verified Database Counts
**Source**: `data/drafts.db` -- queried 2026-03-08
**Purpose**: Single source of truth for all counts, replacing inconsistent numbers across blog posts and reports.
---
## Core Tables
| Table | Count | Notes |
|-------|-------|-------|
| drafts | 434 | Up from 361 after 2026-03-07 fetch |
| ratings | 434 | 1:1 with drafts |
| authors | 557 | Unique persons from Datatracker |
| ideas | 419 | See "Ideas Count History" below |
| gaps | 11 | Not 12 -- see gap list below |
| embeddings | 434 | 1:1 with drafts |
| draft_authors | 1,057 | Draft-author links |
| llm_cache | 1,397 | Cached API calls |
## False Positive Analysis
73 drafts flagged as `false_positive = 1` in ratings table (new column added 2026-03-08).
| Criteria | Count |
|----------|-------|
| Relevance <= 2 (auto-flagged) | 38 |
| Relevance 3+ but clearly not AI-agent (manually reviewed) | 35 |
| **Total false positives** | **73** |
| **Drafts excluding false positives** | **361** |
### Relevance Score Distribution (all 434 drafts)
| Relevance | Count |
|-----------|-------|
| 1 | 2 |
| 2 | 36 |
| 3 | 102 |
| 4 | 196 |
| 5 | 98 |
## Category Counts (excluding false positives)
All categories normalized to short-form names (21 legacy long-form entries migrated 2026-03-08).
| Category | Count |
|----------|-------|
| Data formats/interop | 146 |
| A2A protocols | 146 |
| Agent identity/auth | 127 |
| Autonomous netops | 103 |
| Policy/governance | 97 |
| Agent discovery/reg | 82 |
| ML traffic mgmt | 77 |
| AI safety/alignment | 44 |
| Model serving/inference | 42 |
| Human-agent interaction | 33 |
| Other AI/agent | 18 |
Note: Drafts average ~2.4 categories each, so these sum to more than 361.
## Gap List (11 gaps, not 12)
| ID | Topic | Severity | Category |
|----|-------|----------|----------|
| 37 | Multi-Agent Consensus Protocols | high | A2A protocols |
| 38 | Agent Behavioral Verification | critical | AI safety/alignment |
| 39 | Cross-Protocol Agent Migration | medium | Agent discovery/reg |
| 40 | Real-Time Agent Rollback Mechanisms | high | Autonomous netops |
| 41 | Agent Resource Accounting and Billing | medium | new |
| 42 | Federated Agent Learning Privacy | high | Policy/governance |
| 43 | Agent Capability Negotiation | medium | A2A protocols |
| 44 | Cross-Domain Agent Audit Trails | high | Agent identity/auth |
| 45 | Agent Failure Cascade Prevention | critical | AI safety/alignment |
| 46 | Human Override Standardization | high | Human-agent interaction |
| 47 | Agent Performance Benchmarking | medium | new |
Blog posts reference 12 gaps with different names (e.g., "Agent Resource Exhaustion Protection" vs DB's "Agent Resource Accounting and Billing"). The blog list appears to be an editorial rewrite, not raw pipeline output. The missing 12th gap may be "Cross-Protocol Translation" or "Agent Data Provenance" which appear in blog posts but not in the database.
## Ideas Count History
The database currently contains **419 ideas** across **377 drafts**. This is the third different count encountered:
| Source | Count | Date | Likely Explanation |
|--------|-------|------|-------------------|
| Blog post 5 filename | 1,262 | ~2026-03-03 | Pre-expansion dataset (260 drafts), before dedup |
| Blog post 5 text / master stats | 1,780 | ~2026-03-05 | Post-expansion (361 drafts), before dedup |
| Current database | 419 | 2026-03-08 | After `dedup_ideas` run (0.85 threshold) or re-extraction with different params |
### Ideas by Type (current DB)
| Type | Count |
|------|-------|
| protocol | 96 |
| architecture | 95 |
| extension | 79 |
| mechanism | 68 |
| requirement | 42 |
| pattern | 35 |
| framework | 3 |
| format | 1 |
### Ideas per Draft Distribution
| Ideas/Draft | Drafts |
|-------------|--------|
| 1 | 337 |
| 2 | 38 |
| 3 | 2 |
| 0 (no ideas) | 57 |
The near-uniform 1-idea-per-draft (89% of drafts with ideas) suggests either aggressive dedup or a re-extraction with constrained output. The original pipeline extracted 1-4 ideas per draft, so the 1,780 figure likely reflects pre-dedup counts.
Excluding false positives: 365 ideas across 326 drafts.
## Actions Taken (2026-03-08)
1. **Category normalization**: Updated 21 ratings rows from legacy long-form category names to canonical short forms. All 11 categories now consistent.
2. **False positive flagging**: Added `false_positive` column to ratings table. Flagged 73 drafts (38 with relevance <= 2, 35 manually reviewed at relevance 3+).
3. **Schema migration**: Updated `db.py` schema and migration code to include `false_positive` column.
4. **This document**: Created as single source of truth for counts.