diff --git a/agents/guardian.md b/agents/guardian.md index 55b1ef9..657de8a 100644 --- a/agents/guardian.md +++ b/agents/guardian.md @@ -39,6 +39,7 @@ You see attack surfaces others walk past. You calibrate your response to actual - **Context isolation:** You receive only what the orchestrator provides. Do not assume knowledge from prior phases, other agents, or session history. If information is missing, use `STATUS: NEEDS_CONTEXT` rather than guessing. - APPROVED = zero CRITICAL findings - Every finding needs a suggested fix, not just a complaint +- **Evidence required:** Every CRITICAL or WARNING must cite a specific command output, exit code, or exact code with file path and line numbers. Findings without evidence are downgraded to INFO by the orchestrator. - Be rigorous but practical — flag real risks, not science fiction ## Status Token diff --git a/agents/sage.md b/agents/sage.md index 3a7f6bf..6fc9bba 100644 --- a/agents/sage.md +++ b/agents/sage.md @@ -49,6 +49,7 @@ You see the forest, not just the trees. "Will a new team member understand this - **Context isolation:** You receive only what the orchestrator provides. Do not assume knowledge from prior phases, other agents, or session history. If information is missing, use `STATUS: NEEDS_CONTEXT` rather than guessing. - APPROVED = code is readable, tested, consistent, and complete - REJECTED = significant quality issues that affect maintainability +- **Evidence required:** Quality findings must cite specific code (file:line, exact construct) or measurable criteria. Do not raise vague suggestions — if you cannot point to the code, do not raise the finding. - Focus on the next 6 months. Not the next 6 years. - Your review should be shorter than the code change. If it's not, you're over-reviewing. diff --git a/agents/skeptic.md b/agents/skeptic.md index 08149b1..d7a07c2 100644 --- a/agents/skeptic.md +++ b/agents/skeptic.md @@ -36,6 +36,7 @@ You make the implicit explicit. "The plan assumes X — but does X actually hold - **Context isolation:** You receive only what the orchestrator provides. Do not assume knowledge from prior phases, other agents, or session history. If information is missing, use `STATUS: NEEDS_CONTEXT` rather than guessing. - Every challenge MUST include an alternative. "This might not work" alone is not helpful. - Limit to 3-5 challenges. More than 7 is shadow behavior. +- **Evidence required:** Every challenge must reference specific code (file:line) or describe a concrete scenario with reproduction steps. Vague concerns without evidence are downgraded to INFO by the orchestrator. - Stay in scope. Challenge the task's assumptions, not the universe's. - APPROVED = no fundamental design flaws - REJECTED = the approach is wrong, and you have a better one diff --git a/skills/check-phase/SKILL.md b/skills/check-phase/SKILL.md index cb2169e..e4e7fd4 100644 --- a/skills/check-phase/SKILL.md +++ b/skills/check-phase/SKILL.md @@ -179,6 +179,51 @@ When the Act phase routes findings back to the Maker and the Maker applies fixes --- +## Evidence Requirements + +Every CRITICAL or WARNING finding must include concrete evidence. Findings without evidence are downgraded to INFO. + +### Evidence Types + +| Type | Example | When Required | +|------|---------|---------------| +| Command output | `npm test` output showing failure | Test-related findings | +| Exit code | `exit code 1 from eslint` | Tool-based validation | +| Code citation | `src/auth.ts:48 — \`if (token) { ... }\`` | Logic or security findings | +| Git diff | `+ db.query(userInput)` (unsanitized) | Implementation review | +| Reproduction steps | "1. Send POST with empty body, 2. Observe 500" | Runtime behavior findings | + +### Banned Phrases + +The following phrases are not permitted in CRITICAL or WARNING findings. They indicate speculation, not evidence: + +- "might be" +- "could potentially" +- "appears to" +- "seems like" +- "may not" + +A finding using these phrases must either be rewritten with evidence or downgraded to INFO. + +### Verification Protocol + +For each CRITICAL or WARNING finding, state: + +1. **What was tested** — the specific code path, input, or scenario examined +2. **What was observed** — the actual behavior or code construct found +3. **What correct behavior should be** — the expected alternative + +### Downgrade Rule + +If a reviewer produces a CRITICAL or WARNING finding without any of the evidence types above, the orchestrator downgrades it to INFO and emits a `decision` event: + +```bash +./lib/archeflow-event.sh "$RUN_ID" decision check "" \ + '{"what":"evidence_downgrade","from":"CRITICAL","to":"INFO","finding":"","reviewer":"","reason":"no evidence provided"}' +``` + +--- + ## Why Structured Findings Matter The standardized format enables: diff --git a/skills/run/SKILL.md b/skills/run/SKILL.md index 8baffea..bcb7124 100644 --- a/skills/run/SKILL.md +++ b/skills/run/SKILL.md @@ -477,6 +477,32 @@ Spawn all applicable reviewers in parallel (multiple Agent calls in one message) After each returns, emit `review.verdict` and save artifact. +#### 3c-ii. Evidence Validation + +After all reviewers complete, scan their outputs for banned phrases in CRITICAL/WARNING findings. Downgrade unsupported findings before proceeding to Act. + +```bash +BANNED_PHRASES=("might be" "could potentially" "appears to" "seems like" "may not") + +for artifact in .archeflow/artifacts/${RUN_ID}/check-*.md; do + REVIEWER=$(basename "$artifact" .md | sed 's/check-//') + while IFS= read -r line; do + SEVERITY=$(echo "$line" | grep -oE '(CRITICAL|WARNING)' | head -1) + [[ -z "$SEVERITY" ]] && continue + for phrase in "${BANNED_PHRASES[@]}"; do + if echo "$line" | grep -qi "$phrase"; then + echo "EVIDENCE DOWNGRADE: $REVIEWER finding uses '$phrase' — downgrading to INFO" + ./lib/archeflow-event.sh "$RUN_ID" decision check "" \ + '{"what":"evidence_downgrade","from":"'"$SEVERITY"'","to":"INFO","reviewer":"'"$REVIEWER"'","reason":"banned phrase: '"$phrase"'"}' + # Replace severity in artifact + sed -i "s/$line/$(echo "$line" | sed "s/$SEVERITY/INFO/")/" "$artifact" + break + fi + done + done < "$artifact" +done +``` + #### 3d. Phase Transition: Check to Act Collect all verdict seq numbers for the parent array.