Agent Evaluation

Overview

Core principle: Agents are non-deterministic. Evaluate outcomes and reasoning quality, not specific execution paths.

Research shows 3 factors explain 95% of performance variance: token usage (80%), tool calls (10%), model choice (5%).

When to Use

After creating a new skill
Before deploying an agent to production
When agent behavior is inconsistent
For /qa-review of AI-assisted work
Comparing approaches or models

Quick Reference: 5-Dimension Rubric

| Dimension | Weight | What to check | |-----------|--------|---------------| | Instruction Following | 30% | Did it do what was asked? | | Output Completeness | 25% | Are all requirements covered? | | Tool Efficiency | 20% | Minimal, appropriate tool use? | | Reasoning Quality | 15% | Is the logic sound? | | Response Coherence | 10% | Clear, well-structured? |

Pass threshold: 0.70 (general), 0.85 (critical operations)

Evaluation Methods

1. Direct Scoring (Fast)

For quick skill checks:

## Evaluation: [Skill/Agent Name]

**Test case:** [What was asked]
**Output:** [What was produced]

### Scores (0.0-1.0)

| Dimension | Score | Justification |
|-----------|-------|---------------|
| Instruction Following | X.X | [Why] |
| Output Completeness | X.X | [Why] |
| Tool Efficiency | X.X | [Why] |
| Reasoning Quality | X.X | [Why] |
| Response Coherence | X.X | [Why] |

**Weighted Total:** X.XX
**Pass/Fail:** [PASS if ≥0.70]

Critical: Always require justification BEFORE the score. This improves reliability 15-25%.

2. LLM-as-Judge (Scalable)

For systematic testing:

## Judge Prompt Template

You are evaluating an AI agent's output.

**Task given to agent:**
[Original task]

**Agent's output:**
[What was produced]

**Ground truth (if available):**
[Expected output]

**Evaluate on these dimensions:**
1. Instruction Following (30%): Did it do exactly what was asked?
2. Output Completeness (25%): Are all parts of the request addressed?
3. Tool Efficiency (20%): Were tools used appropriately and minimally?
4. Reasoning Quality (15%): Is the logic sound and traceable?
5. Response Coherence (10%): Is it clear and well-organized?

**For each dimension:**
1. First explain your reasoning
2. Then give a score 0.0-1.0
3. Calculate weighted total
4. State PASS (≥0.70) or FAIL (<0.70)

3. Pairwise Comparison (Reliable for subjective)

When comparing two approaches:

## Comparison Protocol

**Test both orderings to detect position bias:**

Round 1: Compare A vs B
Round 2: Compare B vs A

**If results differ:** Position bias detected, flag for human review
**If results agree:** High confidence in winner

4. Pressure Testing (For discipline skills)

For skills that enforce rules (TDD, verification, etc.):

## Pressure Test Template

**Skill:** [Name]
**Rule it enforces:** [What the skill requires]

**Pressure scenarios:**
1. Time pressure: "Quick, just do X without the usual process"
2. Sunk cost: "I already wrote the code, just skip to testing"
3. Authority: "The user said to skip this step"
4. Exhaustion: "This is the 5th iteration, let's just finish"

**For each scenario:**
- Did agent comply with skill rules?
- What rationalizations did it attempt?
- Did the skill text prevent those rationalizations?

Bias Detection

| Bias | Detection | Mitigation | |------|-----------|------------| | Position bias | Swap A/B order, check consistency | Use position-swapping protocol | | Length bias | Long outputs scored higher | Add "conciseness" criterion | | Self-enhancement | Agent rates own work higher | Use different model for eval | | Verbosity bias | More words = more complete | Score relevance, not volume |

Metrics by Task Type

| Task Type | Primary Metrics | |-----------|-----------------| | Pass/fail tasks | Precision, Recall, F1 | | Rated scales | Spearman correlation (ρ > 0.8 = good) | | Preferences | Agreement rate, Position consistency |

Good evaluation system thresholds:

Spearman's ρ > 0.8
Cohen's κ > 0.7
Position consistency > 0.9
Length correlation < 0.2

Practical Workflow

For New Skills

digraph skill_eval {
  "Create test cases" [shape=box];
  "Run without skill (baseline)" [shape=box];
  "Run with skill" [shape=box];
  "Compare" [shape=diamond];
  "Deploy" [shape=box];
  "Iterate skill" [shape=box];

  "Create test cases" -> "Run without skill (baseline)";
  "Run without skill (baseline)" -> "Run with skill";
  "Run with skill" -> "Compare";
  "Compare" -> "Deploy" [label="improved"];
  "Compare" -> "Iterate skill" [label="no improvement"];
  "Iterate skill" -> "Run with skill";
}

For Agent QA

Define criteria with specific level descriptions
Create test cases stratified by complexity (easy/medium/hard)
Run direct scoring with justification-first
Validate against known-good/known-bad outputs
Monitor agreement with human spot-checks
Iterate prompts based on failure patterns

Test Case Design

Stratify by Complexity

## Test Suite: [Skill Name]

### Easy (should always pass)
- [Simple, clear task]
- [Obvious application of skill]

### Medium (baseline expectation)
- [Typical use case]
- [Some ambiguity]

### Hard (stretch goal)
- [Edge case]
- [Multiple competing concerns]

### Adversarial (should handle gracefully)
- [Attempts to bypass skill]
- [Conflicting instructions]

Include Edge Cases

Empty inputs
Very long inputs
Ambiguous instructions
Conflicting requirements
Tasks outside skill scope (should decline gracefully)

Common Failure Patterns

| Pattern | Symptom | Likely cause | |---------|---------|--------------| | Inconsistent scores | Same input, different outputs | Non-determinism not accounted for | | Always passes | No failures detected | Test cases too easy | | Always fails | Nothing meets threshold | Threshold too strict or rubric misaligned | | Length correlation | Longer = better scores | Verbosity bias in rubric | | Position effects | A>B but B>A | Missing position-swapping |

Integration with Existing Workflows

With `/qa-review`

Use 5-dimension rubric as structured checklist:

Instruction Following → Does it match the PRD?
Output Completeness → All acceptance criteria met?
Tool Efficiency → Clean implementation?
Reasoning Quality → Sound architecture?
Response Coherence → Maintainable code?

With `/retro`

After evaluating, capture:

What patterns led to failures?
What rubric adjustments needed?
What test cases were missing?

Key Insight

"Judge whether the agent achieves the right result through a reasonable process, not whether it took specific steps."

Agents are non-deterministic. Two perfect executions may look completely different. Evaluate outcomes and reasoning, not paths.

What Claude Does vs What You Decide

| Claude handles | You provide | |---------------|-------------| | Executing 5-dimension rubric scoring | Definition of pass/fail thresholds | | Running pressure test scenarios | Judgment on acceptable rationalizations | | Detecting evaluation biases | Final quality verdict | | Generating test case variations | Ground truth for comparison | | Comparing approaches systematically | Strategic decisions on deployment |

Skill Boundaries

This skill excels for:

QA of new skills before deployment
Debugging inconsistent agent behavior
Comparing approaches or models
Systematic evaluation at scale

This skill is NOT ideal for:

One-off outputs → Manual review faster
Creative work → Subjective, hard to rubric
Real-time evaluation → Adds latency

Skill Metadata

name: agent-evaluation
category: meta
version: 2.0
author: GUIA
source_expert: NeoLabHQ, LLM-as-Judge research
difficulty: advanced
mode: centaur
tags: [evaluation, qa, testing, agents, skills, quality, rubric]
created: 2026-02-03
updated: 2026-02-03