LLM Eval Harness is a lightweight, framework-agnostic evaluation tool for agents and LLM pipelines. It is designed for QA engineers who want to move beyond ad-hoc prompts and into reproducible, CI-friendly test suites.

Goals

Reproducible evals: define tasks as data, not as one-off scripts.
Scoring functions: support exact, fuzzy, and LLM-as-judge scoring.
Regression tracking: compare scores across model versions, prompts, and code changes.
CI integration: exit codes and JSON reports for TeamCity, GitHub Actions, and GitLab CI.

Core concepts

Task

A task is a user request plus a ground-truth expectation:

interface Task {
  id: string;
  prompt: string;
  expectedTool?: string;
  expectedArgs?: Record<string, unknown>;
  expectedOutput?: string;
}

Evaluator

An evaluator scores the agent's result against the task. It returns a score between 0 and 1 and a human-readable reason.

type Evaluator = (result: AgentResult, task: Task) => {
  score: number;
  pass: boolean;
  reason: string;
};

Report

After running the suite, the harness produces a JSON report and a markdown summary:

Overall: 0.84
├─ tool-call accuracy: 0.92
├─ task success: 0.78
└─ hallucination rate: 0.03

Roadmap

Publish initial TypeScript package
Add CLI with config file support
Add built-in evaluators for common LLM tasks
Add GitHub Actions example

This project is in the design phase. If you want to collaborate, reach out on LinkedIn or Telegram.

# Goals

# Core concepts

# Task

# Evaluator

# Report