feat: add memory injection audit trail for effectiveness tracking

This commit is contained in:
2026-04-04 07:36:55 +02:00
parent 8af9db2c12
commit dd82944529
3 changed files with 193 additions and 2 deletions

View File

@@ -215,6 +215,59 @@ Moves the lesson to `archive.jsonl` regardless of frequency.
| Before agent spawn (run start) | Inject relevant lessons | `archeflow-memory.sh inject <domain> <archetype>` |
| User command | Add/list/forget lessons | `archeflow-memory.sh add/list/forget` |
## Audit Trail
Track which lessons are injected into each run and whether they were effective.
### Storage
```
.archeflow/memory/audit.jsonl # Append-only audit log
```
### Injection Audit Record
When `--audit <run_id>` is passed to the `inject` command, an audit record is written:
```jsonl
{"ts":"2026-04-04T10:00:00Z","run_id":"2026-04-04-auth-fix","domain":"code","archetype":"","lessons_injected":["m-001","m-003"],"lesson_count":2}
```
Usage:
```bash
./lib/archeflow-memory.sh inject "$DOMAIN" "" --audit "$RUN_ID"
```
### Effectiveness Check
After a run completes, check whether injected lessons prevented issues:
```bash
./lib/archeflow-memory.sh audit-check <run_id>
```
This command:
1. Reads `audit.jsonl` for lessons injected in the given run
2. Reads the run's event file for `review.verdict` events
3. For each injected lesson, checks keyword overlap between the lesson's description and review findings
4. **No matching finding** = `helpful` (the lesson likely prevented the issue)
5. **Matching finding** = `ineffective` (the issue repeated despite the lesson being injected)
6. Appends effectiveness results to `audit.jsonl`
### Effectiveness Over Time
By querying `audit.jsonl` for effectiveness records, you can measure:
- Which lessons consistently prevent issues (high `helpful` count)
- Which lessons are not working (high `ineffective` count — consider rewording or removing)
- Overall memory system ROI (ratio of helpful to ineffective across all runs)
```bash
# Count effectiveness per lesson
jq -r 'select(.type == "effectiveness_check") | [.lesson_id, .effectiveness] | @tsv' .archeflow/memory/audit.jsonl | sort | uniq -c
```
---
## Design Principles
1. **Append-only storage.** `lessons.jsonl` is append-only during writes; decay rewrites the file in place but preserves all data (archived lessons move to `archive.jsonl`).