feat: add process logging with DAG-based event sourcing

Event-sourced orchestration logging: JSONL events with parent relationships form a DAG for causal reconstruction of agent flows. Includes bash event emitter (jq-based) and markdown report generator.
2026-04-03 11:06:02 +02:00
parent 8dec44d199
commit 1753e69a9f
4 changed files with 605 additions and 1 deletions
--- a/skills/process-log/SKILL.md
+++ b/skills/process-log/SKILL.md
@@ -0,0 +1,278 @@
+---
+name: process-log
+description: |
+  Event-based process logging for ArcheFlow orchestrations. Captures every phase transition,
+  agent output, decision, and fix as structured JSONL events. Enables post-hoc reports,
+  dashboards, and process archaeology.
+  <example>Automatically loaded during orchestration</example>
+  <example>User: "Show me how this story was made"</example>
+---
+
+# Process Log — Event-Sourced Orchestration History
+
+Every ArcheFlow orchestration writes structured events to a JSONL file. Events are the **single source of truth** — all reports (Markdown, dashboards, timelines) are generated views.
+
+## Event Storage
+
+```
+.archeflow/events/<run-id>.jsonl     # One file per orchestration run
+.archeflow/events/index.jsonl        # Run index (one line per run, for listing)
+```
+
+**Run ID format:** `<date>-<slug>` (e.g., `2026-04-03-der-huster`)
+
+## When to Emit Events
+
+Emit an event at each of these points during orchestration:
+
+| Moment | Event Type | Trigger |
+|--------|-----------|---------|
+| Orchestration starts | `run.start` | After workflow selection, before first agent |
+| Agent spawned | `agent.start` | Before each Agent tool call |
+| Agent completes | `agent.complete` | After each Agent returns |
+| Phase transition | `phase.transition` | Plan→Do, Do→Check, Check→Act |
+| Decision made | `decision` | Plot direction chosen, fix applied, workflow adapted |
+| Review verdict | `review.verdict` | Guardian/Sage/Skeptic delivers verdict |
+| Fix applied | `fix.applied` | After each edit that addresses a review finding |
+| Cycle boundary | `cycle.boundary` | End of PDCA cycle, before next (or exit) |
+| Shadow detected | `shadow.detected` | Shadow threshold triggered |
+| Orchestration ends | `run.complete` | After final Act phase |
+
+## Event Schema
+
+Every event is one JSON line with these required fields:
+
+```jsonl
+{
+  "ts": "2026-04-03T14:32:07Z",
+  "run_id": "2026-04-03-der-huster",
+  "seq": 4,
+  "parent": [2],
+  "type": "agent.complete",
+  "phase": "plan",
+  "agent": "creator",
+  "data": { ... }
+}
+```
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `ts` | ISO 8601 | Timestamp |
+| `run_id` | string | Unique run identifier |
+| `seq` | integer | Monotonically increasing sequence number within run |
+| `parent` | int[] | Seq numbers of causal parent events. Forms a DAG. `[]` for root events. |
+| `type` | string | Event type (see table above) |
+| `phase` | string | Current PDCA phase: `plan`, `do`, `check`, `act` |
+| `agent` | string or null | Agent archetype that triggered the event |
+| `data` | object | Event-type-specific payload (see below) |
+
+### Parent Relationships (DAG)
+
+The `parent` field turns the flat event stream into a directed acyclic graph (agent call graph). This enables:
+
+- **Causal reconstruction:** which agent output caused which downstream action
+- **Parallel visualization:** agents sharing a parent ran concurrently
+- **Blame tracking:** trace a fix back through review → draft → outline → research
+
+Rules:
+- `run.start` has `parent: []` (root node)
+- An agent has `parent: [seq of event that triggered it]`
+- A phase transition has `parent: [seq of all completing events in prior phase]`
+- A fix has `parent: [seq of the review that found the issue]`
+- A decision has `parent: [seq of the agent that produced the alternatives]`
+- Parallel agents share the same parent (fan-out), phase transitions collect them (fan-in)
+
+Example DAG from a writing workflow:
+```
+#1 run.start []
+├── #2 agent.complete (explorer)       [1]
+│   └── #3 decision (plot direction)   [2]
+├── #4 agent.complete (creator)        [2]     ← explorer informs creator
+├── #5 phase.transition (plan→do)      [3,4]   ← fan-in
+│   └── #6 agent.complete (maker)      [5]
+├── #7 phase.transition (do→check)     [6]
+│   ├── #8 review (guardian)           [7]     ← parallel (fan-out)
+│   └── #9 review (sage)              [7]     ← parallel (fan-out)
+├── #10 phase.transition (check→act)   [8,9]   ← fan-in
+├── #11 fix (timeline)                 [8]     ← caused by guardian
+├── #12 fix (voice drift)             [9]     ← caused by sage
+└── #18 run.complete                   [17]
+```
+
+## Event Payloads by Type
+
+### `run.start`
+```json
+{
+  "task": "Write short story 'Der Huster'",
+  "workflow": "kurzgeschichte",
+  "team": "story-development",
+  "max_cycles": 2,
+  "config": {
+    "voice_profile": "vp-giesing-gschichten-v1",
+    "persona": "giesinger",
+    "target_words": 6000
+  }
+}
+```
+
+### `agent.start`
+```json
+{
+  "archetype": "story-explorer",
+  "model": "haiku",
+  "prompt_summary": "Research premise, find emotional core, suggest 3 plot directions"
+}
+```
+
+### `agent.complete`
+```json
+{
+  "archetype": "story-explorer",
+  "duration_ms": 87605,
+  "tokens": 21645,
+  "artifacts": ["docs/01-der-huster-research.md"],
+  "summary": "3 plot directions developed, recommended C (Mo krank + Koffer)"
+}
+```
+
+### `decision`
+```json
+{
+  "what": "plot_direction",
+  "chosen": "C — Mo krank + Koffer aus B",
+  "alternatives": [
+    {"id": "A", "label": "Mo ist weg", "reason_rejected": "Zu passiv für 6k-Story"},
+    {"id": "B", "label": "Huster gehört nicht Mo", "reason_rejected": "Zu Krimi-nah"}
+  ],
+  "rationale": "Stärkster emotionaler Kern, passt zum Voice Profile"
+}
+```
+
+### `review.verdict`
+```json
+{
+  "archetype": "guardian",
+  "verdict": "approved_with_fixes",
+  "findings": [
+    {"severity": "bug", "description": "Timeline: 'Montag' referenced but story starts Dienstag", "fix_required": true},
+    {"severity": "recommendation", "description": "Gentrification monologue too long for Alex register", "fix_required": false}
+  ]
+}
+```
+
+### `fix.applied`
+```json
+{
+  "source": "guardian",
+  "finding": "Timeline: Montag → Dienstag",
+  "file": "stories/01-der-huster.md",
+  "line": 302,
+  "before": "das Gegenteil von Montag",
+  "after": "das Gegenteil von Dienstag"
+}
+```
+
+### `phase.transition`
+```json
+{
+  "from": "plan",
+  "to": "do",
+  "artifacts_so_far": ["research.md", "outline.md"],
+  "notes": "Explorer recommended direction C, Creator produced 6-scene outline"
+}
+```
+
+### `cycle.boundary`
+```json
+{
+  "cycle": 1,
+  "max_cycles": 2,
+  "exit_condition": "all_approved",
+  "met": true,
+  "fixes_applied": 6,
+  "next_action": "complete"
+}
+```
+
+### `shadow.detected`
+```json
+{
+  "archetype": "story-explorer",
+  "shadow": "endless_research",
+  "trigger": "output >2000 words without recommendation",
+  "action": "correction_prompt_applied",
+  "occurrence": 1
+}
+```
+
+### `run.complete`
+```json
+{
+  "status": "completed",
+  "cycles": 1,
+  "agents_total": 5,
+  "fixes_total": 6,
+  "shadows": 0,
+  "duration_ms": 1295519,
+  "artifacts": [
+    "docs/01-der-huster-research.md",
+    "docs/01-der-huster-outline.md",
+    "stories/01-der-huster.md",
+    "docs/01-der-huster-guardian-review.md",
+    "docs/01-der-huster-sage-review.md",
+    "docs/01-der-huster-process.md"
+  ]
+}
+```
+
+## How to Emit Events
+
+During orchestration, write events using this pattern:
+
+```bash
+# Append one event to the run's JSONL file
+echo '{"ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","run_id":"RUN_ID","seq":SEQ,"type":"TYPE","phase":"PHASE","agent":"AGENT","data":{...}}' >> .archeflow/events/RUN_ID.jsonl
+```
+
+Or use the helper script:
+
+```bash
+./lib/archeflow-event.sh RUN_ID TYPE PHASE AGENT '{"key":"value"}'
+```
+
+The orchestration skill should call the event emitter at each trigger point listed in the table above.
+
+## Generating Reports
+
+After orchestration completes (or during, for live progress):
+
+```bash
+# Generate markdown process report
+./lib/archeflow-report.sh .archeflow/events/2026-04-03-der-huster.jsonl > docs/process-report.md
+
+# List all runs
+cat .archeflow/events/index.jsonl | jq -r '[.run_id, .status, .task] | @tsv'
+```
+
+## Run Index
+
+After each `run.complete`, append a summary line to `.archeflow/events/index.jsonl`:
+
+```jsonl
+{"run_id":"2026-04-03-der-huster","ts":"2026-04-03T16:00:00Z","task":"Write Der Huster","workflow":"kurzgeschichte","status":"completed","cycles":1,"agents":5,"fixes":6,"duration_ms":1295519}
+```
+
+## Integration with Existing Skills
+
+- **`orchestration`**: Emit events at phase transitions and after each agent
+- **`shadow-detection`**: Emit `shadow.detected` when thresholds trigger
+- **`autonomous-mode`**: Use `index.jsonl` for session summaries instead of separate session-log
+- **`workflow-design`**: Custom workflows inherit logging automatically
+
+## Design Principles
+
+1. **Append-only.** Never modify or delete events. They are immutable facts.
+2. **Self-contained.** Each event has enough context to be understood alone (no forward references).
+3. **Cheap.** One `echo >>` per event. No database, no service, no dependencies.
+4. **Optional.** If events dir doesn't exist, orchestration works fine without logging. Events are observation, not control flow.