Ships production agent evaluation.

Stop guessing whether your agents are getting worse. Capture trajectories, score them with three parallel metrics, detect regressions with statistical rigor, and visualize the full decision DAG—all offline, all local, all in one harness.

The Problem with Agent Evals

Tool-using agents fail in ways end-to-end answer checks miss. An agent can produce plausible final text while calling the wrong tool, skipping required lookups, or hallucinating facts. You ship the agent confident it works. A week later, a prompt change degrades performance without warning. You have no idea why, and no way to catch it fast.

AgentBench replaces guessing with data. It intercepts every tool call your agent makes, scores the entire trajectory—not just the final answer—and flags regressions before they reach production.

What You Get

• Trajectory capture: Every tool invocation recorded as a DAG node with args, output, latency, and reasoning traces. See exactly what your agent did, not just what it said.
• Three parallel metrics: Tool-call accuracy (edit distance), semantic similarity (embeddings), and faithfulness (claim grounding). All computed concurrently. One score per scenario.
• Regression detection: Z-score analysis against rolling history flags drops as small as 1.5σ. Catch quality degradation immediately after prompt or model changes.
• Streamlit dashboard: Pass rates, metric distributions, regression timeline, score trends. Filter by model, prompt version, scenario. Visualize the full history at a glance.
• YAML scenarios: Declarative test cases. Expected call graphs, gold outputs, grounding context, custom thresholds per scenario. No boilerplate.
• SQLite persistence: Everything stored locally. Offline analysis, historical comparison, no external services, no dependencies beyond Python.

Production Results

Real demo results: Good agent runs achieve 100% pass rate (all metrics 1.0). Degraded prompt drops to 75% with z-score regressions flagged automatically. Broken agent (wrong tools) fails as expected. You see all three scenarios instantly in the dashboard timeline.

Demo Results

This is what shipped. Nine runs, 36 scored results, four scenarios. Good agent, degraded variant, broken variant. Full history in SQLite. Dashboard renders all three at once.

Scenario	Runs	Passed	Tool Acc.	Semantic	Faithfulness
weather_austin	9	8 (89%)	0.89	0.88	0.89
company_lookup	9	8 (89%)	0.89	0.87	0.89
company_tax	9	6 (67%)	0.89	0.86	0.33
order_status	9	8 (89%)	0.89	0.87	0.89

How It Works

Clean, modular pipeline. Load scenarios from YAML. Intercept tool calls. Score in parallel. Persist to SQLite. Detect regressions with statistics. Visualize in Streamlit. No magic, no dependencies beyond what you already have.

The Pipeline

AgentBench follows a linear but flexible path from test case definition to visual regression detection:

The Three Metrics Explained

1. Tool-Call Accuracy

Compares the sequence of tools your agent called against the sequence it was supposed to call. Each tool invocation is normalized to (tool_name, arguments). Levenshtein distance measures how many edits (insertions, deletions, substitutions) are needed to transform actual into expected. Score = 1 - (distance / max_length), clipped to [0, 1]. Catches wrong tool calls, skipped steps, extra steps. A score of 1.0 means perfect tool usage.

2. Semantic Similarity

Embeds the agent's final answer and the gold output separately using all-MiniLM-L6-v2 (sentence-transformers). Computes cosine similarity. Catches answers that say the right thing in different words, and answers that say the wrong thing in confident language. Ranges 0 to 1. A score of 1.0 is bit-for-bit identical meaning.

3. Faithfulness (Claim Grounding)

Splits the answer into claim-like sentences. For each claim, checks if it's supported by either the context provided in the scenario or the outputs from the tools the agent called. Support measured by token overlap ≥ 0.4 or fuzzy trigram containment. Score = supported_claims / total_claims. Catches hallucinations, made-up facts, statements without grounding. A score of 1.0 means every claim is verifiable in the context.

Decision DAG Model

Each tool call becomes a node in a directed acyclic graph. Every node stores:

• node_id: Unique identifier within the trajectory
• tool_name: Which LangChain tool was invoked
• input_args: The arguments passed to the tool
• output: What the tool returned
• reasoning_trace: Optional chain-of-thought explanation
• latency_ms: How long the tool took to execute
• token_count: Tokens used by this step (if available)
• parent_ids: Which nodes came before this one
• error: Exception, if any
• timestamp: When it happened

The full trajectory aggregates all nodes plus final_output, total_tokens, model/prompt labels, and metadata. You can reconstruct the agent's exact thought process from the DAG: which tools it called, in what order, with what results.

Evaluation Methodology

Here's the exact sequence for each scenario:

1. Load — Parse YAML into a Pydantic Scenario object with prompt, expected_calls, expected_output, context, and pass_thresholds.
2. Instrument — Create TrajectoryInterceptor; agent wraps its LangChain tools through it.
3. Execute — Agent runs; every tool call becomes a ToolCallNode linked via parent_ids (chain DAG).
4. Finalize — Capture final natural-language answer; stamp finished_at.
5. Score (parallel) — ThreadPoolExecutor computes three metrics concurrently.
6. Gate — Scenario passes if ALL three metrics meet their thresholds (AND logic).
7. Persist — Save trajectory + result to SQLite; compare metrics to rolling history; write regression flags if z ≤ -1.5.
8. Summarize — CLI prints pass rate and per-scenario breakdown; optional JSON export to data/last_run.json.

Regression Detection in Depth

The regression detector is statistical and tolerant. For each metric in each scenario:

• Load history: Query up to 30 prior runs (excluding current).
• Minimum threshold: Need at least 3 historical runs to compute statistics. Otherwise, skip regression check.
• Compute z-score: Calculate mean and sample standard deviation of prior runs. Z = (current_value - mean) / std.
• Flag condition: If z ≤ -1.5 (default, configurable), emit RegressionFlag with metadata: scenario, metric, current score, mean, std, z-score, timestamp.
• Flat baseline edge case: If std ≈ 0 (all prior runs identical), flag only hard drops below the mean.
• Persist flags: Store in regressions table with full details for dashboard timeline replay.

Why z-score instead of just watching pass rate? A prompt change might drop semantic similarity from 0.98 to 0.94 while keeping pass rate at 100% (if threshold is 0.85). You'd miss it with binary pass/fail tracking. AgentBench catches that decline before it becomes a cascade.

Get It Running

Five minutes to your first eval. Bundled demo agent, four YAML scenarios, seeded SQLite history, full Streamlit dashboard.

Note: See data/last_run.json for CI-friendly JSON summary of latest run.

Plug In Your Agent

The protocol is clean. No modifications to AgentBench source needed. Wrap your LangChain tools with the interceptor, run your agent, record reasoning, return the final answer. Results write to SQLite immediately. Dashboard picks them up on refresh.

What Comes Next

v0.1.0 ships the core. Here's what's coming to make it even stronger: