Evgeny Pershukov — Blog

17 Bugs in My Backtest Engine (And What They Taught Me)

2026-08-06T00:00:00Z

Every trading strategy I run starts with a backtest: a simulator that replays years of market candles through the strategy and reports what would have happened. The bot behind it is a real system — ~26,000 lines of strict TypeScript, 272 tests, an LLM verifier that vets trade signals, a risk manager with hard limits.

Here is the uncomfortable part: the strategy code had tests, but the simulator was mostly trusted. And the simulator is the oracle. If the backtest lies, every strategy tuned against it is optimized for a market that does not exist. So I treated the backtest engine the way I would treat any production system under test — two full adversarial audit rounds, followed by three parallel external reviews. Result: 17 bugs found and fixed, two of them critical.

This post is a walkthrough of the most instructive ones, grouped by the failure pattern they represent — because the patterns generalize far beyond trading.

The setup

The engine simulates an exchange: balances (free/locked per asset), limit and market orders, leverage with borrowed margin, funding payments, and liquidation. Three files carry the simulation: BacktestExchange.ts (order execution and balances), BacktestRunner.ts (the candle loop), BacktestReport.ts (metrics).

Audit one covered the main flows and fixed 8 bugs. Audit two was a deliberate second pass — same files, fresh eyes, looking for what the first pass missed. It found 9 more. That alone is lesson zero: the first audit of a complex stateful system never gets everything.

Pattern 1: State that outlives itself

The liquidation handler had two bugs that compounded. First, it sold only free base balance — and only afterwards cancelled open orders, unlocking the rest. The unlocked base was never sold, leaving dead balance behind after a "full" liquidation. Second, and worse:

// after liquidation, none of these were reset:
positionQty      // funding kept being deducted
positionCost     // avgPrice = positionCost / positionQty → garbage
borrowedQuote    // net equity calculation → garbage

A liquidation mid-backtest poisoned every subsequent candle. The engine did not crash — it just quietly produced wrong numbers. That is the defining trait of oracle bugs: nothing throws, the report just lies.

A variant of the same pattern lived on the strategy side. executeDirectionalMode() recorded a new position in memory before the order was confirmed. If the exchange rejected the order, the position still existed — a ghost. The position manager then generated close signals for it, producing spurious orders that burned capital on nothing. The fix was structural: positions are created only in onOrderExecuted, on confirmed fill. Never record state for an action the world has not confirmed.

Pattern 2: The silent clamp

The most expensive habit in the codebase was Math.max(0, ...) used as error handling:

// market sell with insufficient free balance:
bBal.free = Math.max(0, bBal.free - qty)   // silently sells less than asked

// funding deduction with empty free balance:
qBal.free = Math.max(0, qBal.free - funding)  // funding silently vanishes

// fee deduction on a second fill in the same candle:
qBal.free = Math.max(0, qBal.free - fee)   // money from thin air

Each clamp turned an invariant violation into a slightly wrong number, with no signal to the caller. The strategy believed a position was closed; the balance said otherwise. A silent clamp is a swallowed exception with extra steps. The fix everywhere was the same: make the shortfall explicit — partial fill reported back to the caller, deduction taken from locked, or an outright reject.

This pattern is everywhere in application code, not just trading. Any place a value is forced into range "to be safe" is a place where a real problem becomes invisible.

Pattern 3: Accounting that double-counts or under-counts

Leverage broke equity math twice, in opposite directions.

Under-counting: equity was computed as quote + base × price, ignoring the borrowed margin entirely. A $10,000 position with $5,000 borrowed reported $15,000 in equity — max drawdown looked smaller than it was.

Double-counting: after the first fix, a second bug surfaced in the yearly runs. For market buys, margin was deducted from free but never returned after fill (limit orders unlocked it; market orders did not), and then getNetEquity() subtracted the same position cost again. Every directional entry "burned" its notional value from reported equity.

Together with the ghost-position bug, this produced my favorite number of the whole exercise: a backtest that started with $10,000 reported $795 in capital for 2025. Two bugs, each survivable alone, interacting into something absurd. The absurdity was the gift — $795 is obviously wrong, which is the only reason anyone dug in. The dangerous bugs are the ones that move your result by 8%.

Pattern 4: Reused evidence

The P&L tracker matched sells to buys FIFO — but never marked buys as consumed. Sell 1 BTC against a 1 BTC buy, then sell another 0.5 BTC, and the engine happily matched it against the same buy again. Every buy could be sold multiple times; realized P&L was fiction. The fix was one field: consumedQty on each trade record.

A related one: grid buys and DCA sells shared a single FIFO pool, so a DCA sell could be matched against a grid buy, producing meaningless per-strategy P&L. Matching needs to respect the boundaries of the thing being measured.

Pattern 5: Hardcoded context

Five bugs were the same bug: a constant baked in from whatever data the author had open that day.

Assumption	Reality	Effect
Funding interval = 15m candles	any interval	funding scaled wrong on other timeframes
Sharpe annualization = 1h candles	15m data	Sharpe inflated 2×
Average hold time = 1h/candle	15m data	hold time inflated 4×
Base asset = `'BTC'`	ETH, SOL, ETHBTC pairs exist	final positions never closed
Fee = flat 0.1%	exchange charges 0.06% for BTC	fees overstated 1.67×

The Sharpe one deserves emphasis: a headline quality metric, off by a factor of two, because of Math.sqrt(365 * 24). The fix in every case was to derive the constant from the data — candle gap from timestamps, base asset from symbol info — not to pick a better constant.

Pattern 6: What the external reviewers caught

After the two internal audits, I ran three parallel external reviews with different angles: order execution, financial realism, market microstructure. They found a different class of problem — not broken code, but a broken model of reality:

Fee structure. The reviewers assumed Binance-style maker/taker fees. Checking the actual exchange docs showed no maker/taker split at all in leverage mode — a flat 0.06%. Platform-specific research beats reasonable assumptions.
Free leverage. The default funding rate was 0, meaning borrowed margin cost nothing. At 3× leverage held for 30 days, real funding is roughly −8% of margin; the backtest showed 0.
Look-ahead bias. The strategy saw candle N's close and executed at candle N's close. Real strategies act on the previous close and fill at an unknown next price. This systematically inflates win rate, and no unit test will ever catch it — it is a modeling error, not a code error.

What this taught me

Test the oracle before the system. My 272 tests validated the bot against the simulator; almost nothing validated the simulator against reality. Any system whose output drives decisions — a backtest, an eval harness, a metrics pipeline — deserves its own adversarial pass.

Absurd numbers are leads, not noise. $795 from $10,000 was a bug report written by the data itself. Build the habit of chasing outputs that look wrong instead of rationalizing them.

Adversarial review finds what authorship cannot. I wrote the engine; my first audit found the bugs I was capable of seeing. The second pass found 9 more, and the external reviewers found the assumptions I did not know I was making. This is the same reason LLM-as-judge evals use a different model than the one being judged.

The same patterns apply to AI systems. Silent clamps = guardrails that swallow failures. Ghost state = tool calls assumed successful before confirmation. Hardcoded context = eval thresholds copied from someone else's benchmark. Reused evidence = test cases leaked into training prompts. The domain changes; the failure patterns do not.

Key takeaways

The component that produces your metrics is itself a system under test — audit it with the same rigor as the product.
Math.max(0, ...)-style clamps turn invariant violations into silent lies. Make shortfalls explicit.
Derive constants from data; never hardcode timeframe, symbol, or fee assumptions.
Two minor bugs can interact into an absurd result — and the absurdity is what makes them findable. Chase weird numbers.
One audit pass is never enough for stateful systems; fresh-eyes reviews catch a different class of bugs than code tests do.

Context Packs for AI Coding Agents: Building taskctx

2026-08-04T00:00:00Z

Every AI coding session starts with amnesia. The agent does not know what issue #123 is about, what was decided in its comments, which PRs are open against it, or what changed in the feature branch yesterday. For a real task in a multi-repo product, that context lives in at least four places: the issue tracker, pull requests, git branches, and the working copies themselves.

The usual workaround is copy-pasting fragments into the prompt until the agent stops asking questions. It works, but it is slow, lossy, and different every time. I got tired of doing it and built taskctx — a small CLI that assembles the whole context of a task into one deterministic Markdown file. This post covers what goes into a pack and the design decisions that make it work in daily agent-driven development.

The idea: a context pack, not a prompt

A context pack is a snapshot of everything an agent needs to start working on an issue, rendered as a single context-pack.md:

taskctx collect 123
# → .taskctx/123/context-pack.md

The agent does not get a giant prompt. It gets a file with stable paths to everything else — full per-commit diffs, raw GitHub JSON, the repository map — so it can read deeper only where needed. Context assembly stops being part of the prompt and becomes part of the workspace.

What goes into a pack

1. Delta first

The pack opens with what changed since the last collect, because that is what a returning session actually needs:

- api: 3 new commits (a1b2c3d..e4f5g6h)
- web: no new commits
- web#8636 state: open → merged
- issue: +2 comments

New commits are computed against the previous manifest, PR changes are rendered as ready-made transition lines, and the comment delta is a simple count. An agent that worked on the task yesterday reads five lines and is caught up.

2. Issue snapshot

Title, state, labels, body, and comments — the raw material. On top of that, one extraction proved surprisingly valuable: open checkboxes. The renderer scans the issue body and every comment for - [ ] items and lists them with their source:

- [ ] Verify timer state after re-login _(issue body)_
- [ ] Add retry backoff test _(@reviewer 2026-07-30)_

Checkbox items are usually the actual remaining work, scattered across a long thread. Collected in one place, they become the agent's implicit todo list.

3. Repository map and branch state

The config describes the workspace: each repo's local path (or null for remote-only), its GitHub slug, base branch, and branch naming pattern like issue-{N}. For every repo the pack shows the matched branch, the merge-base with the base branch, and the commit list with file statistics — additions and deletions per file, aggregated over the branch.

Full per-commit diffs are cached to disk during collect; the pack only carries their paths. The agent opens a diff when it needs one instead of paying for every diff in tokens up front.

4. PR states

For each linked PR: number, title, state, merged flag, review decision, and URL. Enough for the agent to know whether its previous work is under review, merged, or sent back.

Design decisions

Cache everything, render deterministically

Collect writes raw GitHub JSON, a manifest, and per-commit diffs into a .taskctx/<N>/ store. Rendering reads only the store:

taskctx pack 123 re-renders with zero network access — on a plane, in CI, or when the API rate limit says no.
The same store renders the same pack, byte for byte. Packs are diffable and safe to commit, and a changed pack is a signal, not noise.
--offline goes further and recomputes from local git refs only, without even gh.

Delta over snapshot

A pack that only describes current state forces the agent (and the human) to diff mentally against yesterday. Keeping the previous manifest makes the delta computable, and computing it makes the top section of the pack the most valuable one. This mirrors how I think about test reports: "what changed" beats "what is".

Instruction file over convention

taskctx init writes an instruction.md telling any agent how to consume packs: read the pack first, use the diff directory for details, search the working copies via the repo map, never re-run git diffs by hand. Any agent that picks up the repo inherits the workflow — no tribal knowledge in my head or in a chat history.

Graceful degradation per repo

Branches get deleted after merges; repos are not always checked out. Each repo entry in the pack is independent: a missing branch says "branch not found", a deleted one says "branch gone", a remote-only repo works through the API alone. One broken piece never fails the whole pack — same principle as classifying failures in the monitoring watchdog, applied to context collection.

Implementation notes

The tool is about 1,300 lines of TypeScript running on Bun, with Vitest covering the four risky seams: collection, git plumbing, GitHub API parsing, and pack rendering. The renderer is a pure function over the store — no I/O — which makes it trivial to test with fixtures and keeps the determinism promise enforceable.

The commands:

taskctx init                 # seed config + agent instruction.md
taskctx collect 123          # fetch + render the pack
taskctx pack 123             # re-render from cache
taskctx status 123           # delta only, to stdout
taskctx collect 123 --repo web --offline

What changed in practice

Session startup went from "ten minutes of copy-paste and one thing always forgotten" to one command and a file link. The unexpected win was continuity: because packs are deterministic and delta-first, an agent resuming a task after three days behaves like a colleague who read the thread, not like a new hire. And as a QA habit, I now treat stale context as a test failure — if the pack is old, you refresh it before trusting anything the agent says about the task.

Key takeaways

Every agent session starts with amnesia — context assembly should be a tool, not a prompt ritual.
Render the delta first: a returning session needs "what changed," not the whole history.
Cache everything and keep rendering deterministic, so packs are diffable and safe to commit.
Integrate via an instruction file any agent can read — no platform-specific plugin API required.
Degrade gracefully per repo: one missing branch should never fail the whole pack.

Project page: taskctx — Task Context Packs for LLM Sessions.

Monitoring a Chrome Extension in Production: Building a 24/7 Playwright Watchdog

2026-07-29T00:00:00Z

Browser extensions live inside someone else's DOM. Your extension can be perfectly engineered, fully covered by tests, and still break in production because the host app shipped a redesign overnight and renamed the CSS class your content script was anchoring to. No unit test catches that. No CI run against a saved fixture catches that. The only way to know is to open the real product, with the real extension, as a real user — and look.

This post walks through a monitoring system I built for exactly that problem: a 24/7 watchdog that loads the Everhour Chrome extension into Chromium, opens Asana pages every 10 minutes, verifies that 20+ injected UI controls are rendered, and sends a Slack alert with a pixel-diff screenshot when something disappears. The whole thing runs in Docker with no monitor attached.

The interesting parts are not the selectors. They are the reliability engineering around the checks: headed Chromium in a container, retries driven by error classification, a watchdog that watches the watcher, and an alerting pipeline designed to not cry wolf.

Why headed Chromium, and why Xvfb

Chrome extensions only load in a persistent context — browser.newContext() will not cut it. And while Chromium's newer headless mode has made progress with extensions, the battle-tested route for a 24/7 system is still a headed browser: --load-extension behaves predictably, extension service workers start the same way they do for real users, and the page's anti-automation heuristics treat the session as normal.

A headed browser needs a display. In Docker there is none, so you give it a virtual one:

# entrypoint: start virtual display, then the app
Xvfb :99 -screen 0 1920x1080x24 &
export DISPLAY=:99
node dist/main.js

The browser itself is launched once and kept alive between check cycles:

const context = await chromium.launchPersistentContext(userDataDir, {
  headless: false,
  args: [
    `--load-extension=${extensionPath}`,
    `--disable-extensions-except=${extensionPath}`,
  ],
});

Two practical consequences:

The profile persists. Login sessions survive restarts because userDataDir is a mounted volume. First run on an empty profile triggers an auto-login flow (including TOTP-based 2FA via otplib); after that, cookies do the work.
Profiles are platform-specific. A profile created on a Windows dev machine will not work inside a Linux container. Don't fight this — let auto-login mint a fresh profile per environment.

Checks as configuration, not code

Each monitored page is declared in YAML: a URL, a list of selectors with descriptions, and an optional chainAfter edge. Per-integration files (config/asana.yaml, config/everhour.yaml) are merged at startup and validated for id uniqueness:

checks:
  - id: asana-board-view-everhour
    name: "Board view"
    group: Asana
    url: https://app.asana.com/1/.../board/...
    chainAfter: asana-list-view-everhour
    waitAfterNavigationMs: 3000
    selectors:
      - selector: ".EverhourTimerButton"
        description: "Timer button on card"
        minCount: 1
      - selector: "text/Estimated time"
        description: "Estimate badge"

chainAfter exists because every check runs on one shared tab. Parallel navigation on a single page is a URL race; a sequential chain turns the whole cycle into a deterministic walk: extension popup → list view → board view → task details → home → my tasks. A failing check never blocks the chain — its dependents still run.

The check itself is dumb on purpose: navigate, wait, assert each selector exists (respecting minCount), record PASS/FAIL per control. All the intelligence lives in what happens around it.

Error taxonomy drives retry policy

The single most important design decision: not every failure is retried the same way. Every exception is classified into one of seven types, and the type determines the recovery strategy:

Error type	Meaning	Action
`ElementNotFound`	Selector missing	Retry (up to 3 attempts)
`NetworkError`	Navigation/request failure	Retry with exponential backoff
`Timeout`	Page or selector timeout	Retry with exponential backoff
`ExtensionFailure`	Extension controls not injected	Retry after extension health check
`SessionLost`	Auth session expired	Re-login, retry — attempt not consumed
`BrowserCrash`	Chromium process died	Restart browser, no retry
`Unknown`	Anything else	Retry conservatively

Two rules deserve emphasis.

SessionLost does not consume a retry attempt. An expired session is not a check failure — it is an environmental condition. The retry loop calls an onSessionLost callback (which re-authenticates), then tries again on the same attempt number. A separate counter caps this at 3 re-logins per check so a broken login flow cannot spin forever.

BrowserCrash gets no retry. Retrying against a dead browser process is meaningless; the only correct response is a full browser restart, after which the next cycle proceeds normally.

The retry budget itself is small — 3 attempts, 30-second base delay, doubling per attempt for network-class errors. A check that recovers on attempt 2 or 3 is logged as flaky, which feeds into alerting (more on that below):

for (let attempt = 0; attempt <= cfg.maxAttempts; attempt++) {
  try {
    const result = await runCheck(page, check);
    if (result.result === 'PASS') {
      if (attempt > 0) logger.warn(`Check ${check.id} recovered after ${attempt} retry(s) (flaky)`);
      return { result, retryCount: attempt, /* ... */ };
    }
    // FAIL: fall through to retry logic
  } catch (err) {
    if (err instanceof MonitorError && err.type === 'SessionLost') {
      if (++sessionLostCount > SESSION_LOST_MAX_RETRIES) throw err;
      await onSessionLost();          // re-login
      continue;                        // same attempt number
    }
    if (err instanceof MonitorError && err.type === 'BrowserCrash') throw err; // no retry
    // ...
  }
  await sleep(calculateDelay(baseDelaySec, exponential, attempt));
}

There is also a proactive layer: every 5 minutes, registered auth providers check isAuthenticated() and re-login before the next check cycle if the session expired. Most session losses are healed before any check notices.

Who watches the watcher

A monitoring system that hangs silently is worse than no monitoring — it gives you false confidence. Three mechanisms keep the monitor itself honest:

Heartbeat + watchdog process. The main process rewrites heartbeat.json after every cycle. A separate, deliberately tiny watchdog process reads it every 60 seconds; if the heartbeat is older than 90 seconds, it kills and respawns the main process. The watchdog has no browser, no network, no dependencies — it is too simple to hang the same way the main process can.

Memory guard. Long-lived Chromium leaks. The monitor polls the browser's RSS (tasklist on Windows, ps on Linux) and restarts Chrome when it crosses a configurable limit (2 GB by default).

Scheduled daily restart. Even below the memory limit, the browser gets a clean restart every night at 03:00. Extensions are re-injected via --load-extension on every launch, so a restart also picks up any extension files updated on the mounted volume.

The combination means the system recovers from hangs, leaks, and crashes without human involvement — and every recovery path is exercised regularly enough that you trust it when it matters.

Alerting without alert fatigue

A monitor that pages you for every transient blip gets muted within a week. The alerting pipeline has three filters:

Flaky ≠ broken. A check that failed but recovered on retry is recorded as FLAKY in the dashboard but does not alert. Only stable failures — still failing after the full retry budget — trigger Slack.

Deduplication with a signature. Repeat alerts for the same failure are suppressed for 30 minutes. The dedup key is not just the check id: it is a signature encoding which checks failed and how badly (:crit for browser crashes, :skip for auth-skipped). If the failure picture changes — a second page type starts failing, or a FAIL escalates to a crash — the signature changes and the alert goes through immediately, because this is genuinely new information.

Quiet hours with a digest, not a black hole. Between 21:00 and 09:00 (configured timezone), Slack alerts are suppressed — but written to an alert_queue table in SQLite. In the morning you get a digest of everything that happened overnight. Suppression never means loss.

Every alert carries the failure context: which controls are missing, the error details, and — the best part — screenshots.

Pixel-diff screenshots: show, don't tell

A text alert saying "Timer button not found on Board view" forces you to open the app and hunt. A screenshot shows you instantly. The system goes one step further and borrows from visual regression testing:

On PASS, the highlighted screenshot is written to {checkId}-baseline.png. Every green run refreshes the last-known-good reference.
On FAIL, the screenshot is written to a timestamped file, then pixel-diffed against the baseline using pixelmatch. Changed pixels are highlighted into {checkId}-diff.png.

// PASS → shot becomes the new last-known-good baseline
// FAIL → timestamped live shot + pixelmatch diff vs baseline
const sc = await takeHighlightedScreenshot(page, check.id, controls, mode);

The Slack alert then carries three links: 📸 live screenshot, 📋 baseline, 🔍 diff. When Asana ships a UI update and half the controls vanish, the diff shows you the redesigned toolbar in one glance — no dashboard visit required. When the diff is empty but selectors fail, you know the extension stopped injecting rather than the page changing. That distinction alone cuts triage time from minutes to seconds.

Everything is persisted in SQLite (check_history with per-control JSON detail, screenshot paths, retry counts, durations), with 30-day retention for detail rows and 90-day aggregates for trend reporting.

Lessons from running it 24/7

A few things only production time teaches you:

Monitoring code is production code. It needs its own reliability engineering — watchdogs, rotation, retention, restarts. Budget for it.
Classify before you retry. A single "retry 3 times on error" policy would have masked session expirations and hammered a dead browser. The taxonomy is the feature.
ERR_ABORTED is not an error. Single-page apps abort navigations during redirects all the time. A safeGoto() wrapper that swallows this specific case eliminates an entire class of false alarms.
Extension service workers are flaky to detect. waitForEvent('serviceworker') occasionally times out while content scripts work perfectly. Don't gate checks on the SW; gate them on what the user sees.
Flaky tracking pays off. The retry-recovery log turned out to be an early-warning system: a control that goes flaky before going stable-fail is usually a host app A/B test rolling out.
Give alerts a digest mode. Quiet hours that queue instead of drop keep nights silent and mornings informative.

What to do next

If you maintain a browser extension, an integration, or any UI that renders inside a third-party product:

Pick the 3–5 user-facing controls that define "the integration works."
Write one YAML-style check per page type with those selectors.
Classify your failures before writing a single retry.
Add a screenshot pipeline — first for failures, then a baseline/diff pair.
Put a heartbeat on the monitor itself from day one.

The full system — checker, retry engine, Slack alerts, dashboard, watchdog, Docker setup — is TypeScript with Playwright, Express, and SQLite, and the architecture is integration-agnostic: adding Jira or Trello monitoring means writing one auth provider and one YAML file. Source: github.com/evgeny-pershukov/ext-monitoring.

Key takeaways

Monitoring code is production code: it needs its own watchdog, memory guard, rotation, and retention budget.
Classify failures before writing retries — one "retry 3 times" policy masks session expirations and pages you at 3 AM for nothing.
Headed Chromium under Xvfb in Docker renders extensions exactly like a user machine; headless does not.
Pixel-diff screenshots in alerts cut triage from minutes to seconds: the diff shows what broke without a dashboard visit.
Flaky-before-fail is an early-warning signal — track recoveries, not just failures.

Written by Evgeny Pershukov. Follow me on LinkedIn or GitHub for more notes on AI Test Engineering.

Testing AI Agents: A Practical Guide

2026-07-16T00:00:00Z

Traditional QA treats a system as a black box with deterministic inputs and expected outputs. Click button A, expect result B. That model still applies to the tools an agent calls, but it breaks at the agent layer because LLMs are stochastic, context-dependent, and capable of multi-step reasoning.

This post is a practical guide for QA engineers who want to move from testing deterministic automation to testing LLM agents. It covers the layers you need to test, a minimal eval harness you can build today, and the metrics that actually matter when an agent goes live.

What makes an agent different from a traditional app?

In a classic web app, the state is mostly under your control: DOM, API responses, database rows. An LLM agent adds an unpredictable reasoning layer on top. It may:

Reinterpret instructions based on context
Choose different tools in different orders
Fail gracefully, or appear to succeed while doing the wrong thing
Hallucinate data that never existed

That means the test pyramid needs to be extended. You cannot just assert on the final output; you also need to assert on the process that produced it.

The three layers of agent testing

Think of an agent as a stack. You can test each layer independently and combine them into integration checks.

1. Tool-call layer

At the bottom, the agent calls tools: APIs, databases, file systems, browsers. Each tool should be tested like a normal unit:

Input schema validation
Output schema validation
Error handling for malformed or empty inputs
Side effects (was the database actually updated?)

This is the closest to traditional QA. Use the same frameworks you already know: Playwright for browser tools, Jest/Mocha for API wrappers, and zod for schema validation.

2. Orchestration layer

The orchestration layer decides which tool to call and when. This is where the LLM starts to matter. The questions you want to answer are:

Given a user request, does the agent pick the right tool?
Does it pass the right arguments?
Does it recover from a failed tool call?
Does it stop when it has enough information, or does it loop?

You cannot test this with hard assertions because the same request can yield different valid paths. Instead, you use evaluations over a set of sample tasks.

3. Task-success layer

At the top, the only thing the user cares about: did the agent complete the task correctly? This is evaluated against a ground-truth reference set, but "correct" may include:

Functional correctness (the right answer)
Safety constraints (no PII leaked, no harmful actions)
Latency and cost (did it take 20 calls when 2 would do?)
Tone and format constraints

Build a minimal eval harness in TypeScript

Here is a tiny harness you can adapt for your own agent. It runs a set of test cases, each with a user prompt and expected criteria, then scores the result.

interface EvalCase {
  name: string;
  prompt: string;
  expectedTool?: string;
  expectedArgs?: Record<string, unknown>;
  check: (result: AgentResult) => { pass: boolean; score: number; reason: string };
}

interface AgentResult {
  finalAnswer: string;
  toolCalls: { tool: string; args: unknown; output: unknown }[];
  latencyMs: number;
}

async function runEval(agent: Agent, cases: EvalCase[]) {
  const results = [];
  for (const c of cases) {
    const result = await agent.run(c.prompt);
    const verdict = c.check(result);
    results.push({ ...verdict, name: c.name, latencyMs: result.latencyMs });
  }
  return summarize(results);
}

function summarize(results: any[]) {
  const total = results.reduce((sum, r) => sum + r.score, 0) / results.length;
  return { overall: total.toFixed(2), cases: results };
}

A single case might look like this:

const cancelMeetingCase: EvalCase = {
  name: "cancel meeting and notify attendees",
  prompt: "Cancel my 3pm standup and tell everyone it is postponed.",
  expectedTool: "calendar.cancelEvent",
  check: (result) => {
    const cancelled = result.toolCalls.some(
      (c) => c.tool === "calendar.cancelEvent" && c.args?.title === "3pm standup"
    );
    const notified = result.toolCalls.some(
      (c) => c.tool === "slack.sendMessage" && c.args?.text?.includes("postponed")
    );
    return {
      pass: cancelled && notified,
      score: (cancelled ? 0.5 : 0) + (notified ? 0.5 : 0),
      reason: cancelled && notified ? "ok" : "missing steps",
    };
  },
};

The key idea: the check is a scoring function, not a boolean assertion. This gives you a spectrum of quality instead of a hard pass/fail.

Metrics that matter

Pick a small set of metrics and track them over time. I recommend starting with:

Metric	What it tells you	How to measure
Tool-call accuracy	Did the agent use the right tool with the right arguments?	Exact match on tool name + fuzzy match on args
Task success rate	Did it finish the user's task?	Human judgment or LLM-as-judge on final output
Hallucination rate	Did it invent facts or tools?	Check tool calls against allowed list and output against source data
Turn count / latency	Is it efficient?	Count tool calls and wall-clock time
Regression delta	Did a new model/prompt break old behavior?	Re-run eval set after every change and compare

LLM-as-judge (and its limits)

You can use a separate LLM to score the agent's output. It is cheap, fast, and scales well. A typical prompt looks like:

Score the following answer from 1 to 5 on correctness, completeness, and safety. Explain your reasoning briefly.

But do not rely on it alone. LLM judges have the same failure modes as the agents you are testing: they can be biased by confident phrasing, miss subtle errors, and score low on adversarial examples. Always keep a set of cases where a human is the final judge.

From eval harness to CI pipeline

The biggest mistake is keeping evals on a laptop. Put them in CI:

Run the eval suite on every PR.
Store results as JSON artifacts.
Compare the PR result against main.
Block the merge if the overall score drops by more than a threshold (e.g., 2%).

You do not need a fancy platform. A TeamCity or GitHub Actions step that runs npm run eval and parses the JSON is enough.

What to do next

If you are a QA engineer moving into AI testing, start small:

Pick one agentic flow in your product.
Write 10 example user requests with expected outcomes.
Build a scoring function for each outcome.
Run the suite weekly, then on every change.

Over time, expand the suite to cover edge cases, adversarial prompts, and multi-turn conversations. The goal is not perfect coverage; the goal is a signal that tells you when the agent gets worse.

If you want to go deeper, the next posts in this series will cover:

Building a regression dashboard for LLM prompts
Testing MCP (Model Context Protocol) integrations end-to-end
Evaluating multi-agent orchestration and handoff reliability

Key takeaways

Test agents on three layers: tool-call correctness, orchestration, and end-to-end task success — each catches failures the others miss.
Replace exact-match asserts with scored metrics and explicit thresholds; non-determinism is a test condition, not a failure.
Track tool-call accuracy, task success rate, hallucination rate, and regression delta — these four numbers tell you when the agent gets worse.
Use LLM-as-judge for open-ended outputs, but keep human-judged cases for calibration.
Evals belong in CI: run on every PR, compare against main, block merges on score drops.

Written by Evgeny Pershukov. Follow me on LinkedIn or GitHub for more notes on AI Test Engineering.

Evgeny Pershukov — Blog

17 Bugs in My Backtest Engine (And What They Taught Me)

# The setup

# Pattern 1: State that outlives itself

# Pattern 2: The silent clamp

# Pattern 3: Accounting that double-counts or under-counts

# Pattern 4: Reused evidence

# Pattern 5: Hardcoded context

# Pattern 6: What the external reviewers caught

# What this taught me

Context Packs for AI Coding Agents: Building taskctx

# The idea: a context pack, not a prompt

# What goes into a pack

# 1. Delta first

# 2. Issue snapshot

# 3. Repository map and branch state

# 4. PR states

# Design decisions

# Cache everything, render deterministically

# Delta over snapshot

# Instruction file over convention

# Graceful degradation per repo

# Implementation notes

# What changed in practice

Monitoring a Chrome Extension in Production: Building a 24/7 Playwright Watchdog

# Why headed Chromium, and why Xvfb

# Checks as configuration, not code

# Error taxonomy drives retry policy

# Who watches the watcher

# Alerting without alert fatigue

# Pixel-diff screenshots: show, don't tell

# Lessons from running it 24/7

# What to do next

Testing AI Agents: A Practical Guide

# What makes an agent different from a traditional app?

# The three layers of agent testing

# 1. Tool-call layer

# 2. Orchestration layer

# 3. Task-success layer

# Build a minimal eval harness in TypeScript

# Metrics that matter

# LLM-as-judge (and its limits)

# From eval harness to CI pipeline

# What to do next

The setup

Pattern 1: State that outlives itself

Pattern 2: The silent clamp

Pattern 3: Accounting that double-counts or under-counts

Pattern 4: Reused evidence

Pattern 5: Hardcoded context

Pattern 6: What the external reviewers caught

What this taught me

The idea: a context pack, not a prompt

What goes into a pack

1. Delta first

2. Issue snapshot

3. Repository map and branch state

4. PR states

Design decisions

Cache everything, render deterministically

Delta over snapshot

Instruction file over convention

Graceful degradation per repo

Implementation notes

What changed in practice

Why headed Chromium, and why Xvfb

Checks as configuration, not code

Error taxonomy drives retry policy

Who watches the watcher

Alerting without alert fatigue

Pixel-diff screenshots: show, don't tell

Lessons from running it 24/7

What to do next

What makes an agent different from a traditional app?

The three layers of agent testing

1. Tool-call layer

2. Orchestration layer

3. Task-success layer

Build a minimal eval harness in TypeScript

Metrics that matter

LLM-as-judge (and its limits)

From eval harness to CI pipeline

What to do next