feat: add memory injection audit trail for effectiveness tracking
This commit is contained in:
@@ -215,6 +215,59 @@ Moves the lesson to `archive.jsonl` regardless of frequency.
|
||||
| Before agent spawn (run start) | Inject relevant lessons | `archeflow-memory.sh inject <domain> <archetype>` |
|
||||
| User command | Add/list/forget lessons | `archeflow-memory.sh add/list/forget` |
|
||||
|
||||
## Audit Trail
|
||||
|
||||
Track which lessons are injected into each run and whether they were effective.
|
||||
|
||||
### Storage
|
||||
|
||||
```
|
||||
.archeflow/memory/audit.jsonl # Append-only audit log
|
||||
```
|
||||
|
||||
### Injection Audit Record
|
||||
|
||||
When `--audit <run_id>` is passed to the `inject` command, an audit record is written:
|
||||
|
||||
```jsonl
|
||||
{"ts":"2026-04-04T10:00:00Z","run_id":"2026-04-04-auth-fix","domain":"code","archetype":"","lessons_injected":["m-001","m-003"],"lesson_count":2}
|
||||
```
|
||||
|
||||
Usage:
|
||||
```bash
|
||||
./lib/archeflow-memory.sh inject "$DOMAIN" "" --audit "$RUN_ID"
|
||||
```
|
||||
|
||||
### Effectiveness Check
|
||||
|
||||
After a run completes, check whether injected lessons prevented issues:
|
||||
|
||||
```bash
|
||||
./lib/archeflow-memory.sh audit-check <run_id>
|
||||
```
|
||||
|
||||
This command:
|
||||
1. Reads `audit.jsonl` for lessons injected in the given run
|
||||
2. Reads the run's event file for `review.verdict` events
|
||||
3. For each injected lesson, checks keyword overlap between the lesson's description and review findings
|
||||
4. **No matching finding** = `helpful` (the lesson likely prevented the issue)
|
||||
5. **Matching finding** = `ineffective` (the issue repeated despite the lesson being injected)
|
||||
6. Appends effectiveness results to `audit.jsonl`
|
||||
|
||||
### Effectiveness Over Time
|
||||
|
||||
By querying `audit.jsonl` for effectiveness records, you can measure:
|
||||
- Which lessons consistently prevent issues (high `helpful` count)
|
||||
- Which lessons are not working (high `ineffective` count — consider rewording or removing)
|
||||
- Overall memory system ROI (ratio of helpful to ineffective across all runs)
|
||||
|
||||
```bash
|
||||
# Count effectiveness per lesson
|
||||
jq -r 'select(.type == "effectiveness_check") | [.lesson_id, .effectiveness] | @tsv' .archeflow/memory/audit.jsonl | sort | uniq -c
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Append-only storage.** `lessons.jsonl` is append-only during writes; decay rewrites the file in place but preserves all data (archived lessons move to `archive.jsonl`).
|
||||
|
||||
Reference in New Issue
Block a user