Evaluation
How well the extractor finds commitments on conversations it was not tuned on. The test set is held out: drafted mostly by non-Claude models and by hand, and reviewed blind. The development set is the data the prompt was written against, so its scores confirm the pipeline rather than measure it.
Results on dev
| Configuration | Ledger F1 | Precision | Recall | Labels | Action | Due | Owner | Cost | p50 / p95 | Prompt | Run |
|---|---|---|---|---|---|---|---|---|---|---|---|
| haikulegacy, before sets | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | $0.0046 | 1.5 s / 2.6 s | v1 | 2026-10-22 04:01 |
The sets
| Set | Role | Gated | Items | Review | Labels (by span) | Written by | Hash |
|---|---|---|---|---|---|---|---|
| audioClips of test items: Polly (regenerable) and two people reading (fetched), scored against the same items as text. | test (audio) | no | 0 | – | – | – | e3b0c44298fc |
| devDevelopment set: the spec's 24 seed utterances plus later dev items. Prompts are tuned on it; its score is never a headline. | dev | no | 24 | accepted 24 | COMMITMENT 7, CONDITIONAL 2, HEDGED 4, NONE 4, REFUSAL 2, REPORTED 2, REQUEST 3, SUGGESTION 2 | spec 24 | 69836e7e0869 |
| testHeld-out headline set: reviewed drafts from non-Claude models plus hand-written items. | test | yes | 0 | – | – | – | e3b0c44298fc |
Every run
- dev · haiku · prompt v1 2026-10-22 04:01 · ledger F1 1.000 · legacy, before sets