claude-archeflow-plugin/docs/dogfood-2026-04-04-batch.md

ArcheFlow Dogfood Report #2: Batch API Integration

Date: 2026-04-04 Task: Wire Anthropic Batch API into Colette's fanout pipeline with CLI commands and state persistence Project: writing.colette (Python, 27 modules, 457 tests) Complexity: High — 4 files, async API, state persistence, error recovery, CLI commands

Experimental Setup

Same task, same starting commit, two conditions:

1. Baseline: Plain Claude, no orchestration, single pass
2. ArcheFlow: PDCA standard workflow (Maker + Guardian review)

No Explorer or Creator used this time — task scope was clear enough to skip planning and go directly to Maker + Guardian (effectively a fast workflow).

Quantitative Comparison

Metric	Baseline	ArcheFlow	Delta
Lines added	189	279	+48%
Files touched	4	4	same
Time	~5 min	~12 min	+140%
Commits	1	4	cleaner history
Tests written	1	2	+1
Tests passing	13/13	14/14	+1
Bugs introduced	0	1	worse
Bugs caught by review	0	5	better
Real bugs in final code	1	0 (after fix)	ArcheFlow wins

Bug Analysis

Bugs found only by Guardian (not present in baseline)

#	Bug	Severity	Impact
3	hash() non-deterministic across processes for chapter index mapping	HIGH	Data loss on resume — chapters mapped to wrong files

This bug was introduced by ArcheFlow's Maker and caught by the Guardian. Baseline used enumerate(i) and avoided it entirely. Net: zero value.

Bugs present in BOTH versions, caught only by Guardian

#	Bug	Severity	Impact
4	Timeout marks variant as "done" — permanently loses batch state	HIGH	Silent data loss — timed-out batches can never be resumed

This is the key finding. Both implementations had this design-level bug. Only ArcheFlow's Guardian caught it. Plain Claude missed it because there was no review step.

Bugs in both, not caught by either initially

#	Bug	Severity	Impact
1	API key resolution inconsistency (env vs config)	CRITICAL	Wrong key used under mixed-key environments
5	No JSON error handling on corrupted state files	HIGH	Crash on truncated state file

Guardian flagged these. Baseline would have shipped them silently.

Qualitative Observations

Where Guardian added real value

1. Error path analysis: Guardian systematically checked "what happens when X fails?" for timeout, cancellation, corruption, and cross-process resume. Plain Claude focused on the happy path.
2. Cross-process state: The hash() non-determinism finding required reasoning about Python's hash randomization across interpreter invocations — a subtle runtime property that isn't visible from reading the code in isolation.
3. Data loss scenarios: Finding #4 (timeout → "done" → lost forever) requires understanding the interaction between wait_and_retrieve's timeout branch and the caller's unconditional status assignment. This is a 2-module interaction that single-pass implementation doesn't systematically check.

Where Guardian added noise

1. Finding #2 (batch_id validation): Technically valid but the Anthropic SDK already rejects malformed IDs. Low practical risk.
2. Finding #1 (API key source): Valid but matches existing patterns throughout the codebase — flagging it here without flagging it elsewhere is inconsistent.

The Maker problem

The ArcheFlow Maker introduced a bug (hash-based indexing) that the baseline avoided. This happened because:

• The Maker was working from a task description, not reading the existing sequential rewrite code as closely
• The Creator's plan (when used in dogfood #1) over-specified some things and under-specified others
• Working through an intermediary (plan → implementation) introduces information loss

This is a structural weakness of the PDCA model: the Plan-to-Do handoff can corrupt information.

Conclusions

Complexity threshold confirmed

Task type	Orchestration value
Simple (pattern-following, single file)	Negative — adds cost, Maker introduces bugs
Medium (multi-file feature, clear scope)	Neutral — extra code but similar outcome
Complex (error handling, state, async, resume)	Positive — Guardian catches design-level bugs

The differentiator is error path coverage. Guardian's systematic "what if this fails?" analysis catches bugs that single-pass implementation misses because implementers focus on making things work, not on making failures safe.

The honest ROI question

For this task: Guardian caught 1 bug the baseline missed (timeout data loss). That bug would have caused real data loss in production when a batch times out. The cost was ~7 extra minutes and a Maker-introduced bug that had to be fixed.

Is preventing a production data loss bug worth 7 extra minutes? Yes. But only because this was a task where data loss was possible. For a pure UI change or a refactor with no persistence, the answer would be no.

---

Improvement Hypotheses

Based on both dogfood runs, here are concrete hypotheses about how to improve ArcheFlow's value-to-cost ratio:

H1: Guardian-Only Mode (skip Plan/Do orchestration)

Observation: In both dogfoods, the Maker produced equivalent-or-worse code than plain Claude. The value came entirely from the Guardian review.

Hypothesis: A "review-only" mode where the user implements normally and then runs ArcheFlow as a post-implementation review would capture the Guardian's value without the Maker's overhead.

Test: Implement the same task plain, then run af-review (Guardian + Skeptic on the diff). Compare bug catch rate to full PDCA.

Expected outcome: Same bug catch rate, ~60% less cost.

H2: Pre-Implementation Threat Modeling (Guardian before Maker)

Observation: Guardian found error-handling bugs (timeout, corruption) that the Maker didn't anticipate. If Guardian's "what could go wrong?" analysis ran BEFORE implementation, the Maker could build in error handling from the start.

Hypothesis: Running a lightweight Guardian analysis on the Creator's plan (not the code) would produce a "threat list" that the Maker addresses during implementation, eliminating the need for a fix cycle.

Sequence: Creator → Guardian(plan) → Maker(plan + threats) → Guardian(code)

Expected outcome: Fewer Maker-introduced bugs, shorter fix cycle, Guardian's code review focuses on implementation correctness rather than missing error paths.

H3: Differential Review (only review what the Maker DIDN'T get from the plan)

Observation: The Maker copies most of the plan correctly. The bugs are in the gaps — things the plan didn't specify (error handling, cross-process state, timeout recovery).

Hypothesis: Instead of reviewing the entire diff, focus the Guardian on the delta between the plan and the implementation — what the Maker added, changed, or skipped that wasn't in the plan.

Test: Extract the plan's explicit instructions, diff against the implementation, and give Guardian only the unplanned additions.

Expected outcome: Higher signal-to-noise ratio (fewer false positives on code that correctly follows the plan), focused attention on the dangerous gaps.

H4: Project Convention Calibration (reduce false positives)

Observation: Guardian flagged API key handling (finding #1) and batch_id validation (finding #2) — both valid in absolute terms but inconsistent with the project's existing patterns. The project doesn't validate IDs or centralize key management anywhere else.

Hypothesis: Injecting a "project conventions" summary before Guardian review (e.g., "this project uses env vars for API keys, does not validate external IDs, handles errors via outer try/except") would let Guardian calibrate its expectations and only flag deviations from convention, not the convention itself.

Test: Run Guardian with and without convention context on the same diff. Count false positives.

Expected outcome: 30-50% reduction in noise findings without missing real bugs.

H5: Abandon PDCA for Implementation, Keep It for Review

Observation: Across both dogfoods, the cycle-back mechanism (Plan→Do→Check→Act→cycle back) never triggered. All reviews were APPROVED_WITH_FIXES, and fixes were applied in a single pass. The cyclic model added structural overhead (event tracking, artifact routing, convergence detection) that was never used.

Hypothesis: For most tasks, a linear pipeline (implement → multi-reviewer check → targeted fix) is sufficient. Reserve cyclic PDCA for tasks where reviewers fundamentally reject the approach (not just the implementation).

Test: Compare PDCA standard (cycle-back enabled) vs pipeline (no cycle-back) on 10 tasks. Measure: how often does cycle-back actually improve the outcome?

Expected outcome: Cycle-back triggers in <10% of tasks. Pipeline matches PDCA quality for 90%+ of cases at lower cost.

H6: Evidence-Gated Findings Actually Work

Observation: Of Guardian's 5 findings in this dogfood, 3 were substantive (timeout data loss, hash non-determinism, no JSON error handling) and 2 were low-value (API key pattern, batch_id format). The substantive ones cited specific code paths and failure scenarios. The low-value ones cited general principles without evidence of actual exploitation.

Hypothesis: The evidence-gating mechanism added in v0.7.0 (ban hedged phrases, require command output or code citation) would have automatically downgraded finding #2 ("could corrupt log output") while preserving findings #3 and #4 (which cite specific code paths and failure mechanisms).

Test: Re-run the Guardian review with evidence-gating active. Count how many findings survive vs. get downgraded.

Expected outcome: 1-2 findings correctly downgraded, 0 real bugs missed.

H7: Shadow Detection for the Maker

Observation: The Maker introduced a bug (hash-based indexing) because it deviated from the existing codebase pattern (enumerate-based indexing). This is the "Rogue" shadow — the Maker going off-script from what the codebase already does.

Hypothesis: A pre-commit check that compares the Maker's implementation against the existing codebase patterns (e.g., "how are chapter indices computed elsewhere in fanout.py?") would catch Rogue deviations before the Guardian review.

Test: Add a "pattern conformance" check to the Do phase that greps for how the modified variables/functions are used elsewhere in the file.

Expected outcome: Catches Rogue shadow bugs at implementation time rather than review time, saving a review cycle.

---

Recommended Next Steps (Priority Order)

1. H1: Build af-review mode (Guardian-only on existing diff) — lowest effort, highest expected ROI
2. H4: Project convention injection — reduce noise without missing signal
3. H2: Pre-implementation threat modeling — address the root cause of missing error handling
4. H5: Default to pipeline strategy, reserve PDCA for rejections
5. H7: Maker pattern conformance check — reduce Maker-introduced bugs