ietf-draft-analyzer/data/reports/reviews/review-statistics.md

Statistical Review

Reviewed: 2026-03-08 Reviewer: Statistics & Data Analysis Agent (Claude Opus 4.6) Scope: All blog posts (00-08), data packages (00-06), master stats, and key reports -- cross-checked against data/drafts.db via sqlite3 queries.

---

Data Integrity Issues

CRITICAL: Database Has Grown Beyond Blog Series Claims

The blog series consistently claims 361 drafts, 557 authors, 1,780 ideas, and 12 gaps. The current database contains:

Metric	Claimed	Actual (DB)	Delta
Total drafts	361	434	+73 (20% more)
Total authors	557	557	Match
Total ideas	1,780	419	-1,361 (76% fewer)
Total gaps	12	11	-1
Total ratings	361	434	+73
Total embeddings	361	434	+73
Draft-author links	1,057	1,057	Match
LLM cache entries	703	1,397	+694

Root cause: The database was updated on 2026-03-07 with a new fetch of 431 drafts, bringing the total to 434. The blog series was written against a snapshot taken around 2026-03-03. The master stats file (00-master-stats.md) is dated 2026-03-03 and reflects the 361-draft corpus. However, the blog posts do not carry a "data frozen as of" disclaimer -- they state numbers as absolute facts.

Recommendation: Add a clear data freeze date to each blog post header (e.g., "Data current as of 2026-03-03, reflecting 361 of 434 drafts now in the database"). Alternatively, update all posts to reflect the 434-draft corpus.

CRITICAL: Ideas Count Mismatch (1,780 Claimed vs 419 in DB)

The most serious discrepancy. The ideas table contains only 419 rows, not 1,780. The idea type distribution also diverges sharply:

Type	Claimed	Actual (DB)
mechanism	663	68
architecture	280	95
pattern	251	35
protocol	228	96
requirement	171	42
extension	168	79
framework	9	3
format	--	1
other	10	--

Only 377 of 434 drafts have any ideas extracted. The 1,780 figure may come from a prior pipeline run whose results were overwritten, or from an in-memory analysis that was not persisted. Either way, the blog series' core claims about "1,780 ideas," "96% appear in only one draft," "628 cross-org convergent ideas (43% of 1,467 clusters)," and the entire idea taxonomy are not reproducible from the current database.

Recommendation: Re-run idea extraction to populate the database, or clearly note that the 1,780 figure comes from a specific pipeline run that is no longer reflected in the DB. This is the single most important data integrity issue -- Post 5's entire thesis rests on these numbers.

HIGH: Gap Count and Topics Differ

The DB has 11 gaps, not 12. The gap topics in the database are:

1. Multi-Agent Consensus Protocols
2. Agent Behavioral Verification
3. Cross-Protocol Agent Migration
4. Real-Time Agent Rollback Mechanisms
5. Agent Resource Accounting and Billing
6. Federated Agent Learning Privacy
7. Agent Capability Negotiation
8. Cross-Domain Agent Audit Trails
9. Agent Failure Cascade Prevention
10. Human Override Standardization
11. Agent Performance Benchmarking

The blog series lists different gap topics (e.g., "Agent Resource Exhaustion Protection" vs DB's "Agent Resource Accounting and Billing"; "Agent Error Recovery and Rollback" vs "Real-Time Agent Rollback Mechanisms"). Post 4's gap list appears to be a curated/rewritten version. The blog's 12-gap list includes "Cross-Protocol Translation" and "Agent Data Provenance" which do not appear as named gaps in the DB.

Recommendation: Reconcile the gap list. Either the DB was re-run and lost a gap, or the blog presents an edited version. If the latter, this should be acknowledged as editorial synthesis rather than raw pipeline output.

HIGH: Composite Rating Calculations Inconsistent

Multiple scoring methodologies are used without disclosure:

Draft	Blog Score	5-dim Composite (DB)	4-dim (excl overlap)
draft-aylward-daap-v2	4.8 (Post 1)	4.0	4.75
draft-cowles-volt	4.8 (Post 1)	4.0	4.75
draft-guy-bary-stamp-protocol	4.6 (Post 1)	3.8	4.5
draft-drake-email-tpm-attestation	4.6 (Post 1)	3.8	4.5

Post 1 claims DAAP and VOLT scored "4.8" -- this matches neither the 5-dimension composite (4.0) nor the 4-dimension composite excluding overlap (4.75). The master stats correctly uses 4.75 for the same drafts. Post 1 appears to round up (4.75 -> 4.8, 4.5 -> 4.6), which inflates perceived quality.

The "average score" also varies: Post 1 says "3.38/5.0", the master stats say "3.32" (novelty average), the DB 5-dim average is 3.13, and the 4-dim average is 3.27.

Recommendation: Pick one composite calculation, document it, and use it consistently. The 4-dim composite (excluding overlap, since overlap measures redundancy rather than quality) is defensible, but the rounding from 4.75 to 4.8 is not. Use exact values.

MEDIUM: Monthly Draft Counts Differ Between Sources

The master stats growth curve and the actual DB monthly counts diverge:

Month	Master Stats	Actual DB
2024-01	3	7
2024-02	1	3
2024-04	1	6
2024-09	2	11
2025-10	67	61
2025-11	61	53
2026-01	54	51
2026-02	86	85
2026-03	22	56

The master stats show a total of 361 across all months; the DB shows 434. Some of this is explained by the 73 new drafts fetched after the data freeze, but the per-month figures for 2024 are also significantly different (suggesting earlier months got new drafts from the keyword expansion that are counted differently).

The "43x acceleration" claim (from ~2/mo to 86/mo) uses the lowest trough and highest peak, which is cherry-picking. A more honest measure would compare rolling averages.

MEDIUM: Huawei Draft Counts Vary Across Posts

Source	Huawei Drafts	Huawei Authors
Post 1	66	53
Post 2	66	53
Data Package 02	"~60+ unique"	"~40+ unique"
Master Stats	57+	28+
Actual DB (all Huawei entities)	69 unique drafts	multiple entities
DB "Huawei" entity only	39	32

The consolidation of Huawei sub-entities (Huawei, Huawei Technologies, Huawei Canada, Huawei Singapore, etc.) is done informally. The blog confidently states "53 authors, 66 drafts" but the data package says "~60+ unique drafts, ~40+ unique authors (some overlap)." The actual DB shows 69 unique drafts across all Huawei-named affiliations. The author count depends entirely on deduplication, which is described as "hand-curated" with "40+ mappings."

Recommendation: Document the exact normalization rules used to arrive at "53 authors, 66 drafts" and make them reproducible.

---

Methodological Concerns

Sampling Bias

The dataset is keyword-filtered (12 keywords across draft names and abstracts). Multiple posts draw sweeping conclusions about "the IETF's AI agent landscape" without sufficient caveats about what this filter captures and misses.

Specific concerns:

• Post 1 claims "nearly 1 in 10 new Internet-Drafts is about AI agents" (9.3%). This figure depends on the denominator (total IETF drafts per year) which is stated but not sourced. Where do the numbers 1,651 (2024) and 2,696 (2025) come from? Are they verifiable?
• The keyword "intelligent" likely captures many non-agent-related drafts about intelligent networking, QoS, etc. The keyword "autonomous" captures autonomous systems (AS) networking drafts. No false-positive analysis is presented.
• Post 7 mentions "~90% accuracy" from spot-checking 50 drafts but provides no breakdown of error types, no inter-rater reliability, and no details on the spot-check methodology.

Rating Methodology (LLM-as-Judge)

The 1-5 rating scale scored by Claude is presented with minimal caveats in the blog posts. Key issues:

1. No inter-rater reliability: The same LLM rated all drafts. No human baseline or second-model comparison is provided.
2. Abstract-only analysis: Post 7 acknowledges switching from full-text to abstract-only analysis for ratings, claiming "equivalent ratings." No evidence is presented for this equivalence claim.
3. Overlap dimension ambiguity: The "overlap" dimension measures redundancy with other drafts, but since the LLM rates each draft independently, it cannot know the full corpus. The overlap score likely reflects the LLM's general knowledge of the field, not corpus-specific similarity.
4. Score compression: All ratings are on a 1-5 scale with integer values only. The max composite (5-dim) is 4.2 and the min is 1.8. The effective range is narrow, making distinctions between drafts less meaningful than the precise decimal composites suggest.

Clustering and Similarity

• The 0.85 and 0.90 cosine similarity thresholds for overlap clusters are stated but not justified. What threshold sensitivity analysis was performed?
• The "25+ near-duplicate pairs at 0.98" claim is used to argue for deduplication to "roughly 300 distinct proposals" -- but 25 duplicate pairs would reduce the count by at most 25, not 61.
• The SequenceMatcher threshold (0.75) for fuzzy idea matching is stated but not validated. How many false positives does this produce?

Cross-Org Convergence (628 Ideas)

The 628 cross-org convergent ideas figure is the blog series' lead metric for Post 5. However:

• The methodology (SequenceMatcher at 0.75 threshold across organizational boundaries) is described but the underlying data is not in the DB (only 419 ideas exist).
• No precision/recall analysis is presented. At a 0.75 sequence match threshold, generic titles like "Agent Communication Framework" will match across many drafts regardless of actual technical similarity.
• The claim "43% of unique idea clusters have cross-org validation" depends on the denominator (1,467 unique clusters), which itself depends on the 1,780 raw count that is not reproducible from the DB.

---

Misleading Claims

1. "4:1 Safety Deficit" Ratio

This ratio is presented as the series' signature metric, but its calculation shifts:

• Master stats says "~8:1 capability-to-safety" (after keyword expansion)
• Data package 01 says the safety ratio "improved from 4:1 due to keyword expansion"
• Posts 1-6 consistently use "4:1" as the headline
• Data package 06 says "45 safety drafts vs 316 capability drafts = 7:1"
• The deep analysis shows monthly ratios from 1.5:1 to 21:1

The blog presents "4:1" as a stable finding when the data shows it varies from 1.5:1 to 21:1 depending on the time period and from 4:1 to 8:1 depending on whether keyword-expansion drafts are included. The ratio also depends on multi-labeling: a draft tagged as both "A2A protocols" and "AI safety" counts as both capability and safety.

Recommendation: Present the ratio with ranges and context, not as a single stable number. The monthly trend data (Task #24) is more informative than any single ratio.

2. "36x Growth"

Post 1 claims "36x growth: 2 drafts/month (Jun 2025) to 72 drafts/month (Feb 2026)." The series overview says the same. But:

• Jun 2025 actually had 5 drafts (per DB), not 2
• Feb 2026 had 85 (per DB), not 72 or 86
• Picking the lowest month and highest month inflates the multiplier
• A rolling 3-month average would show more modest but still impressive growth

3. "96% of Ideas Appear in Exactly One Draft"

This is presented as evidence of extreme fragmentation. However:

• The idea extraction pipeline produces ~5 ideas per draft by design
• Many extracted "ideas" are draft-specific component descriptions, not standalone proposals
• Post 5 acknowledges this ("most are draft-specific component descriptions") but still leads with the 96% figure as a shock stat
• The true fragmentation question is whether the problems being solved are unique, not whether the component labels are unique

4. "120 A2A Protocol Drafts"

The category count depends on how "A2A protocols" is defined. The master stats says 136 A2A protocol drafts, but the blog posts use 120. Some posts say "136" (Post 4's gap data package), while others say "120" (Posts 1, 3, 4 text). The inconsistency appears to stem from the category count changing between pipeline runs.

5. Causal Language

Several claims use causal framing where only correlation exists:

• "The safety deficit is structural, not attitudinal" (Post 4) -- this is an interpretation, not a finding
• "Gap severity correlates with coordination difficulty" (Post 4) -- stated as found, but the correlation is between two human-assigned ordinal variables (severity levels assigned by Claude, coordination difficulty assessed by the Architect) with N=12 data points
• "The organizations doing the most drafting are focused on capability; the organizations doing the best safety work are doing the least drafting" (Post 2) -- the causal implication is that volume and safety focus are inversely related, but this could simply reflect different organizational missions

---

Improvement Suggestions

1. Add a Data Provenance Section

Each blog post should include a brief provenance note: data freeze date, pipeline version, exact query or command used to generate each key number. This would make claims verifiable.

2. Standardize the Composite Score

Choose one formula (recommend: 4-dimension excluding overlap, or 5-dimension with clear labeling) and use exact values (not rounded). Document the formula in Post 7 and use it consistently.

3. Validate Idea Extraction

Re-run idea extraction to ensure the DB reflects the claimed 1,780 ideas. If the pipeline was run differently (e.g., with a different prompt or batching strategy), document the exact parameters.

4. Add Confidence Intervals

For claims like "4:1 ratio," show the range across different time periods and calculation methods. For trend claims, show the underlying monthly data rather than cherry-picked endpoints.

5. Acknowledge LLM-as-Judge Limitations Prominently

Post 7 mentions LLM validation briefly. The rating methodology should include:

• A caveat in every post that uses ratings
• A note that overlap scores are based on LLM general knowledge, not corpus comparison
• Acknowledgment that abstract-only analysis may miss important content

6. De-duplicate Before Counting

The "361 drafts" count includes known near-duplicates. The blog acknowledges "probably closer to 300 distinct proposals" (Post 3) but continues using 361 everywhere. Either de-duplicate and use the lower number, or present both with context.

---

Post-by-Post Notes

Post 00 (Series Overview)

• Internal architecture document; numbers are consistent with master stats (361/557/628/12). No issues as an internal document.

Post 01 (Gold Rush)

• Score inflation: DAAP cited as 4.8, actual 4-dim composite is 4.75, 5-dim is 4.0. STAMP cited as 4.6, actual is 4.5/3.8. VOLT cited as 4.8, actual is 4.75/4.0.
• Category table inconsistency: The post lists "Data formats and interoperability: 145" as the top category, but the master stats shows "A2A protocols: 136" as the top. The post appears to use a different category set than the master stats.
• Growth figure: "36x growth" -- cherry-picked from lowest to highest month.
• "0.5% to 9.3%": The denominator (total IETF drafts) is stated but unsourced. The 9.3% figure assumes 1,748 total drafts in Q1 2026 -- where does this come from?
• Average score stated as "3.38" -- does not match any DB calculation (5-dim avg: 3.13; 4-dim avg: 3.27; novelty avg: 3.27).
• "~1,700 technical ideas": Post says "roughly 1,700" in one place; DB has 419.

Post 02 (Who Writes the Rules)

• Huawei "53 authors, 66 drafts" is stated with confidence but data package says "~60+" with caveats about entity dedup. DB shows 69 unique drafts across Huawei entities.
• "65% are at rev-00" for Huawei -- this figure is for "Huawei" entity only (57 drafts), not the combined 66/69. The denominator matters.
• "43 were submitted in the four weeks before IETF 121" -- data package says "43 of 69 across all entities." The blog says "43" out of Huawei's "66" implying 65%, vs data package's "62% of 69."
• "115 (23%) co-author with people from both Chinese and Western organizations" -- not verifiable from current DB without running the centrality analysis.
• Ericsson "4.8 average revision" claim (line 149) is inconsistent with data package showing Ericsson avg rev as 4.8 -- this appears correct.

Post 03 (OAuth Wars)

• The 14-draft OAuth list is well-documented with individual scores.
• Score for DAAP is listed as 4.8 but 4-dim composite is 4.75. Other scores in the table appear to be individual dimension values or different calculations (e.g., STAMP at 4.6 vs 4.5 4-dim).
• The data package actually lists 15 OAuth-related drafts (including draft-mw-spice-actor-chain and draft-gaikwad-south-authorization), but the blog says 14. The blog's list of 14 differs slightly from the data package's 15.
• "25+ near-duplicate pairs" leading to "roughly 300 distinct proposals" is a logical leap. 25 duplicate pairs reduce the count by 25 (one from each pair), yielding 336, not "roughly 300."

Post 04 (What Nobody Builds)

• Gap count: 12 in blog vs 11 in DB. Gap names differ from DB.
• "Ideas Addressing It" column (52, 117, 6, 0, 90, 5, 4, 10, 5, 26, 5, 79) -- these numbers cannot be verified because the ideas table has only 419 rows, not 1,780. With 419 ideas, these per-gap counts are implausible (they sum to 399, nearly the entire ideas table).
• "Only 6 extracted ideas address [error recovery], and all come from a single draft" -- this is a strong claim. With only 419 ideas in the DB, 6 ideas from one draft is plausible, but the DB has no gap-to-idea mapping table to verify.
• "12 (8.8%) of 136 A2A drafts also address safety" -- this requires the categories JSON field in the drafts table. Not independently verified but plausible.
• "Safety has zero co-occurrence with agent discovery/registration and zero co-occurrence with model serving/inference" -- sourced from deep analysis task #27, which is plausible but not verifiable from current DB without re-running the co-occurrence analysis.

Post 05 (1,262 Ideas / Where Drafts Converge)

• Title references "1262" in the filename but post content uses 1,780, 1,692, and 628. The filename appears to be from the pre-expansion dataset.
• "1,692 unique technical ideas" -- the DB has 419 ideas. This is the largest disconnect in the entire series.
• "Only 75 show up in two or more drafts" -- not verifiable from current DB.
• "628 ideas where different organizations are working on recognizably similar problems" -- the central claim of the post, not verifiable from current DB.
• The idea taxonomy table (mechanism: 663, architecture: 280, etc.) does not match DB (mechanism: 68, architecture: 95, etc.). Both the counts and the rank order differ.
• The convergence table (A2A Communication Paradigm: 8 orgs, etc.) is not verifiable.

Post 06 (Big Picture)

• Synthesis post; numbers are drawn from prior posts. Inherits all issues.
• "36 (10%) have been adopted by IETF working groups" -- based on naming convention (draft-ietf-*). This could be verified with a query but depends on the 361-draft corpus.
• "WG-adopted drafts score higher on average (3.54 vs. 3.31)" -- this uses 4-dim composite, which is consistent with the rest of the 4-dim usage but not labeled as such.
• "75 cross-draft convergent ideas (628 via fuzzy matching)" -- the parenthetical mixing of two very different numbers is confusing. 75 is exact-title matches; 628 is fuzzy cross-org. These are different metrics measuring different things.

Post 07 (How We Built This)

• Database table sizes: Claims 361 drafts, 1,780 ideas, 557 authors, 1,057 draft_authors, 4,231 draft_refs, 12 gaps, 703 llm_cache. DB now shows 434/419/557/1,057/4,231/11/1,397. Only draft_authors and draft_refs match.
• "43 CLI commands": Not verified but seems high. The source code would need to be checked.
• Cost figures: "$3.16 for 260 drafts" and total "~$9" are stated without supporting evidence (no token count logs in the DB). Not falsifiable but also not verifiable.
• "15 report types": Not verified.
• Describes rating as "1-5 scale" which matches the DB (max 5, not 10 as the reviewer checklist suggests).

Post 08 (Agents Building the Analysis)

• Meta post about the process. Numbers reference those from other posts, inheriting their issues.
• "20+ SQL queries" and "7 data packages" -- plausible but not independently verifiable.
• "30 dev-journal entries" -- could be verified by reading dev-journal.md.
• The cost table sums to "~$9" but the individual line items sum to ~$9.00 (2.50+5.50+0.80+0.20 = 9.00). Consistent.

State of Ecosystem (Vision Document)

• "36x increase" -- same cherry-picking issue as Post 1.
• Uses "72" drafts/month for Feb 2026 (differs from other sources: 86 in master stats, 85 in DB).
• Otherwise consistent with other posts.

Master Stats (00-master-stats.md)

• Gap count: Lists 12 gaps with different names than DB's 11.
• Idea count: 1,780 -- does not match DB's 419.
• Draft count: 361 -- does not match DB's 434 (but was correct at data freeze date).
• Composite scores: Uses 4-dim composite and gets 4.75 for top drafts -- correct for 4-dim, but unlabeled as such.
• Category distribution: Uses different category names/counts than the blog posts in some cases (e.g., master stats: "A2A protocols: 136" vs Post 1: "A2A protocols: 120").

---

Summary of Findings

Most Serious Issues (would undermine credibility if published):

1. Ideas count (1,780 claimed, 419 in DB) -- the foundation for Post 5's thesis is not reproducible
2. Composite score inflation (4.75 rounded to 4.8) and inconsistent calculation methods
3. Gap count (12 vs 11) and topic naming mismatches

Important Issues (should be fixed before publication): 4. Draft count stale (361 vs 434) 5. "4:1 ratio" is not stable -- varies 1.5:1 to 21:1 by month 6. "36x growth" cherry-picks endpoints 7. Category counts inconsistent between posts and master stats

Minor Issues (polish): 8. Huawei entity deduplication is informal 9. LLM-as-judge caveats are insufficient 10. No false-positive analysis for keyword filtering 11. The "25 duplicate pairs -> roughly 300" arithmetic does not work

What Holds Up Well:

• RFC cross-reference counts (4,231) match exactly
• Draft-author link count (1,057) matches exactly
• Author count (557) matches exactly
• The qualitative patterns (Huawei dominance, safety deficit, fragmentation) are directionally sound even if specific numbers vary
• The geopolitical analysis and team bloc detection methodology are well-described
• The cost analysis (~$9 total) is internally consistent