Agent skill tests / November 4, 2026
What happened when
we tested the skills?
Two starter skills, an initial comparison and a follow-up revision. Inspect the actual answers and the limits of each test.
Latest revision: lead with the useful action
Daily Debrief now leads with follow-ups and meaningful changes. Choose Where Attention Belongs leads with one supported choice and a small experiment. Detailed records and maps stay available when needed.
- All six fresh revised answers met the original task checks.
- 5 of 6 answers were shorter. The full continuity-record case was longer and still needs work on readability.
This reused known test cases and historical skill answers. There was no new no-skill baseline, and no human scoring. Shorter output does not establish saved time or better real-world results.
Initial comparison: predefined checks met
These are AI-graded checks on fictional tasks, not success rates in real work. There was one run per case and condition, with no human scoring.
| Case | With skill | Without skill | Read outputs |
|---|---|---|---|
| leader conflicting commitmentDaily Debrief | 5 / 5 | 5 / 5 | Skill answerBaseline answer |
| ic stale ci and duplicatesDaily Debrief | 5 / 5 | 5 / 5 | Skill answerBaseline answer |
| catchup partial source continuityDaily Debrief | 5 / 5 | 5 / 5 | Skill answerBaseline answer |
| leader prep versus relationshipsChoose Where Attention Belongs | 5 / 5 | 5 / 5 | Skill answerBaseline answer |
| important oneoffs no repetitionChoose Where Attention Belongs | 5 / 5 | 3 / 5 | Skill answerBaseline answer |
| shared support growth and ownershipChoose Where Attention Belongs | 5 / 5 | 5 / 5 | Skill answerBaseline answer |
What this tells us
- Both conditions met every check in five cases. The baseline missed two overlapping checks about offering a useful next step when no automation candidate existed.
- Skill answers were longer in all six cases: approximately 3,052 words in total versus 1,934.
- A skill answer counted six supplied excerpts as seven. Passing the rubric did not make it error-free.
This small test does not establish a general quality improvement. The next iteration should shorten default answers, improve the checks and include human review.
Method, versions and limits
Fresh Codex subagents inheriting the same session configuration. Exact backend model identifier was not exposed; no model override was used.
Two separate fresh Codex graders scored three cases each with A/B condition labels hidden. The coordinator then inspected all 12 outputs and the grading. No human validation has been performed.
Tested source: commit c84292a. These results cover Daily Debrief and Choose Where Attention Belongs. The other four skills have not had behavioral evaluations.
- Six hand-authored fictional cases, not workplace outcome data.
- One execution per case and condition; no uncertainty interval or significance claim.
- Same inherited model configuration for execution and grading; exact backend identifier unavailable.
- Only two starter skills tested; the other four have not had behavioral evaluations.
- Explicit skill loading; installation, automatic triggering, MCP integrations, scheduling, multi-model portability, latency and cost were not tested.
- Rubric was frozen before output inspection, while execution was in progress; one wording typo was clarified before grading.
- Model-generated grading with coordinator AI review; no human scoring.
- Pass counts measure the stated checks, not real-world productivity, accuracy or hours saved.
Regex word-like sequences in raw Markdown, including source IDs and structured records; this is a reproducible length measure, not a readability score. Informal counts in raw grader notes are unverified estimates and can differ; the recomputation script defines the published counts.