Skip to main content
Delphina auto-generates test cases from your knowledge base, runs them against your data, and an LLM judge scores each response. When a test fails, it points directly to a gap in your documentation — fix the gap with /knowledge, and the next run passes. The critic agent also reviews each chat response in the background, scoring how trustworthy the answer is and flagging ungrounded context and unsafe assumptions directly on the response.

​
How evaluations work

An evaluation suite consists of test cases — each with a prompt (e.g., “What was MRR last month?”), ground truth SQL, and acceptance criteria that the judge scores against.
  1. The agent answers each question, exactly as it would for a user.
  2. An LLM judge scores each response against the ground truth and acceptance criteria.
  3. Results are collected into a run with an overall pass rate and per-case details.
Runs are started on demand from Context Layer > Evaluations — click New Run and pick which cases to include (see Running evaluations) — typically after merging a knowledge update or fixing issues.

​
Viewing your evaluation suite

Context Layer > Evaluations has two tabs:
  • Test Cases — your suite. Search cases by name or prompt, or download the whole suite as CSV. Open a case to see its prompt, ground truth SQL, expected output with per-column acceptance criteria, the critical knowledge documents it depends on (linked into the knowledge base), and an explanation of what the case tests.
  • Runs — every past and in-progress run, with its score, status, and duration.
Eval cases live in your knowledge base under the evals/ namespace, so they are versioned like any other document — a case’s page shows which version you’re looking at.

​
Creating evaluations

You can create evaluations automatically, from a chat, or with more control over the process. You can also edit an existing case’s prompt, SQL, and criteria directly on its page with Edit.

​
Automatic

Start a new chat and send:
This starts a guided session: the agent analyzes your knowledge base, plans a prioritized set of test cases, and builds them in the same session. You can review the plan in the chat and ask the agent to adjust it before the cases are built. If you have existing evaluations or specific questions you want tested, upload them at Context Layer > Sources > File Uploads before sending the prompt. The agent incorporates these alongside what it discovers from your knowledge base.

​
From a chat

Create or update individual cases using /knowledge:
You can also share a file with test ideas directly in the chat.

​
Custom

For more control and custom context, describe what you want in a /knowledge chat — the agent builds a prioritized plan from your inputs and generates the cases once you confirm:
A useful maintenance prompt for keeping the suite healthy as it grows:

​
Running evaluations

On the Runs tab, click New Run. In the dialog you can:
  • Name the run — a timestamped name is prefilled.
  • Select which cases to run — all cases are selected by default (a full run). Search by name or by domain (e.g. marketing) to run a subset, which is handy for quickly re-testing the cases affected by a knowledge fix.
  • Pick a warehouse connection if your workspace has more than one.
The run starts immediately. Progress updates live on the run page, and a run that’s still pending or in progress can be cancelled. Running evaluations requires the Developer role.

​
Reviewing results

Open a run to see the summary — total cases, passed, failed, errors, and pass rate — and a table with each case’s verdict, score, and failure reason. Every case row links to the actual chats behind it, so you can see exactly what the agent did. When a case fails, Delphina automatically diagnoses why and classifies the root cause: Open an individual case result for the full detail: each scorer’s verdict with the judge’s explanation, the ground truth query and its results, the agent’s actual response, and a Failure Analysis section with the diagnosed cause and a proposed fix.

​
Fixing failures from the Inbox

Each analyzed failure automatically creates an issue in Context Layer > Inbox, prefixed [Evals], containing the failure summary, a proposed action, and links back to the case run and the analysis. For a single issue, open it and ask Delphina to investigate directly from the chat bar on the issue page. The agent digs into the evidence — your knowledge base, your warehouse and query history, and uploaded source files — then proposes a fix for you to confirm in the chat. For many failures at once, ask Delphina to remediate them as a batch in a /knowledge chat:
The agent clusters the failures by shared root cause (one missing metric definition often explains several failed cases), runs a parallel investigation per cluster, applies the fixes it is confident about, and asks you about the ambiguous ones. It finishes by recommending which cases to re-run. Then verify: start a New Run scoped to the affected cases and confirm they pass.