Testing AI Agents: A Practical Guide

Traditional QA treats a system as a black box with deterministic inputs and expected outputs. Click button A, expect result B. That model still applies to the tools an agent calls, but it breaks at the agent layer because LLMs are stochastic, context-dependent, and capable of multi-step reasoning.

This post is a practical guide for QA engineers who want to move from testing deterministic automation to testing LLM agents. It covers the layers you need to test, a minimal eval harness you can build today, and the metrics that actually matter when an agent goes live.

# What makes an agent different from a traditional app?

In a classic web app, the state is mostly under your control: DOM, API responses, database rows. An LLM agent adds an unpredictable reasoning layer on top. It may:

• Reinterpret instructions based on context
• Choose different tools in different orders
• Fail gracefully, or appear to succeed while doing the wrong thing
• Hallucinate data that never existed

That means the test pyramid needs to be extended. You cannot just assert on the final output; you also need to assert on the process that produced it.

# The three layers of agent testing

Think of an agent as a stack. You can test each layer independently and combine them into integration checks.

# 1. Tool-call layer

At the bottom, the agent calls tools: APIs, databases, file systems, browsers. Each tool should be tested like a normal unit:

• Input schema validation
• Output schema validation
• Error handling for malformed or empty inputs
• Side effects (was the database actually updated?)

This is the closest to traditional QA. Use the same frameworks you already know: Playwright for browser tools, Jest/Mocha for API wrappers, and zod for schema validation.

# 2. Orchestration layer

The orchestration layer decides which tool to call and when. This is where the LLM starts to matter. The questions you want to answer are:

• Given a user request, does the agent pick the right tool?
• Does it pass the right arguments?
• Does it recover from a failed tool call?
• Does it stop when it has enough information, or does it loop?

You cannot test this with hard assertions because the same request can yield different valid paths. Instead, you use evaluations over a set of sample tasks.

# 3. Task-success layer

At the top, the only thing the user cares about: did the agent complete the task correctly? This is evaluated against a ground-truth reference set, but "correct" may include:

• Functional correctness (the right answer)
• Safety constraints (no PII leaked, no harmful actions)
• Latency and cost (did it take 20 calls when 2 would do?)
• Tone and format constraints

# Build a minimal eval harness in TypeScript

Here is a tiny harness you can adapt for your own agent. It runs a set of test cases, each with a user prompt and expected criteria, then scores the result.

interface EvalCase {
  name: string;
  prompt: string;
  expectedTool?: string;
  expectedArgs?: Record<string, unknown>;
  check: (result: AgentResult) => { pass: boolean; score: number; reason: string };
}

interface AgentResult {
  finalAnswer: string;
  toolCalls: { tool: string; args: unknown; output: unknown }[];
  latencyMs: number;
}

async function runEval(agent: Agent, cases: EvalCase[]) {
  const results = [];
  for (const c of cases) {
    const result = await agent.run(c.prompt);
    const verdict = c.check(result);
    results.push({ ...verdict, name: c.name, latencyMs: result.latencyMs });
  }
  return summarize(results);
}

function summarize(results: any[]) {
  const total = results.reduce((sum, r) => sum + r.score, 0) / results.length;
  return { overall: total.toFixed(2), cases: results };
}

A single case might look like this:

const cancelMeetingCase: EvalCase = {
  name: "cancel meeting and notify attendees",
  prompt: "Cancel my 3pm standup and tell everyone it is postponed.",
  expectedTool: "calendar.cancelEvent",
  check: (result) => {
    const cancelled = result.toolCalls.some(
      (c) => c.tool === "calendar.cancelEvent" && c.args?.title === "3pm standup"
    );
    const notified = result.toolCalls.some(
      (c) => c.tool === "slack.sendMessage" && c.args?.text?.includes("postponed")
    );
    return {
      pass: cancelled && notified,
      score: (cancelled ? 0.5 : 0) + (notified ? 0.5 : 0),
      reason: cancelled && notified ? "ok" : "missing steps",
    };
  },
};

The key idea: the check is a scoring function, not a boolean assertion. This gives you a spectrum of quality instead of a hard pass/fail.

# Metrics that matter

Pick a small set of metrics and track them over time. I recommend starting with:

# LLM-as-judge (and its limits)

You can use a separate LLM to score the agent's output. It is cheap, fast, and scales well. A typical prompt looks like:

Score the following answer from 1 to 5 on correctness, completeness, and safety. Explain your reasoning briefly.

But do not rely on it alone. LLM judges have the same failure modes as the agents you are testing: they can be biased by confident phrasing, miss subtle errors, and score low on adversarial examples. Always keep a set of cases where a human is the final judge.

# From eval harness to CI pipeline

The biggest mistake is keeping evals on a laptop. Put them in CI:

1. Run the eval suite on every PR.
2. Store results as JSON artifacts.
3. Compare the PR result against main.
4. Block the merge if the overall score drops by more than a threshold (e.g., 2%).

You do not need a fancy platform. A TeamCity or GitHub Actions step that runs npm run eval and parses the JSON is enough.

# What to do next

If you are a QA engineer moving into AI testing, start small:

1. Pick one agentic flow in your product.
2. Write 10 example user requests with expected outcomes.
3. Build a scoring function for each outcome.
4. Run the suite weekly, then on every change.

Over time, expand the suite to cover edge cases, adversarial prompts, and multi-turn conversations. The goal is not perfect coverage; the goal is a signal that tells you when the agent gets worse.

If you want to go deeper, the next posts in this series will cover:

• Building a regression dashboard for LLM prompts
• Testing MCP (Model Context Protocol) integrations end-to-end
• Evaluating multi-agent orchestration and handoff reliability

Written by Evgeny Pershukov. Follow me on LinkedIn or GitHub for more notes on AI Test Engineering.