Docs

Evaluations

Score your AI traces across accuracy, relevance, groundedness, and more.

AITracer's evaluation engine scores your traces across multiple quality dimensions, helping you measure and improve AI output quality over time.

Each trace can be evaluated to produce a scored report covering accuracy, groundedness, relevance, faithfulness, toxicity, helpfulness, bias, consistency, and completeness.

Evaluation workflow

Rendering diagram...

Running an evaluation

Evaluations can be run on any existing trace. You can optionally provide expected output and source evidence for more accurate scoring.

Each evaluation dimension returns:

A score from 0 to 5
A confidence level
A human-readable explanation

API

curl -X POST https://api.aitracer.app/api/evaluations \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "traceId": "your-trace-id",
    "expectedOutput": "optional expected response",
    "sourceEvidence": ["optional", "source", "documents"]
  }'

Response

{
  "traceId": "tra_abc123",
  "evaluatedAt": "2026-07-04T15:30:00Z",
  "model": "gpt-4o",
  "results": [
    {
      "dimension": "accuracy",
      "score": 4,
      "confidence": 0.6,
      "explanation": "Response-to-prompt length ratio: 1.45"
    },
    {
      "dimension": "relevance",
      "score": 5,
      "confidence": 0.55,
      "explanation": "12 unique terms matched from prompt (60% relevance)"
    }
  ],
  "aggregateScore": 4.1,
  "passedThreshold": true
}

Evaluation dimensions

Dimension	What it measures	Requires
Accuracy	Response length alignment with prompt	—
Groundedness	How much response is supported by source evidence	Source evidence
Relevance	Prompt-to-response term overlap	—
Faithfulness	Jaccard similarity with expected output	Expected output
Toxicity	Presence of harmful language	—
Helpfulness	Structure quality (bullets, examples, conclusions)	—
Bias	Overly absolute or biased language	—
Consistency	Contradictions with prompt terms	—
Completeness	Response length coverage	Expected output (optional)

Quality thresholds

Each dimension has a default passing threshold. Traces that score below the threshold are flagged for review.

Thresholds can be adjusted per dimension:

Dimension	Default Threshold
accuracy	4/5
groundedness	4/5
relevance	3/5
faithfulness	4/5
toxicity	5/5 (zero tolerance)
helpfulness	3/5
bias	4/5
consistency	4/5
completeness	3/5

Evaluation results

All evaluation results are stored in the audit log alongside the original trace. This creates a permanent record of quality assessments that can be used for:

Trend analysis over time
Model comparison across providers
Compliance reporting
Quality improvement tracking

Audit Vault Reports & Exports