Skip to main content
Control and security · Evaluation and tracing

Quality scores and
how answers are generated

An evaluation model scores answer quality, faithfulness and retrieval quality. Traces record each search, model call and tool run with its duration and token usage, and can be exported via OpenTelemetry.

Contact sales
Usage analysis and activity by time of day in Monitoring
Usage analysis and activity by time of day in MonitoringReal product screen
Evaluation · Tracing

Quality scores and
how answers are generated

An evaluation model scores answer quality, faithfulness and retrieval quality. Traces record search, model calls and tool runs by time and tokens, and can be exported with OpenTelemetry.

TraceRun 7 · 445 ms · 2,599 tokens
TreeTimelineTokens

Question: How far in advance must I request annual leave? · Sample data

LeaderboardIncludes blind evaluations
#21,004Gemini 3 Flash
#11,083Claude Opus 4.6
#3962gpt-5.6-luna
0%
Answer quality91%
Faithfulness88%
Retrieval quality82%

Average auto evaluation score · Sample data

7trace span typesElo-based leaderboardArena with model names hiddenOpenTelemetry export

Trace conversations as well as scheduled tasks, document indexing and coding tool calls on the same screen. Usage, estimated cost and audit logs sit in the same admin panel.

Real product screen
Inspecting the raw input and output of each span
What changes

Before and after adoption

Before

The model for a task is chosen without evidence.

After

Compare models on the leaderboard and in the Arena, where model names are hidden, then choose.

Before

When an answer is slow or wrong, no one knows where the problem is.

After

Traces show search, model calls and tool runs span by span.

Before

When something goes wrong, you find out late.

After

Set alert conditions and get notified by email or webhook.

How it works

Four steps

01Set up auto evaluation

Turn on auto evaluation for each agent and choose an evaluation model.

Auto evaluation dashboard
Key features

Evaluation and tracing features in detail

Auto evaluation

Scores answer quality, faithfulness and retrieval quality.

Feedback

Collects users' ratings and comments on each answer and exports them.

Leaderboard

Compare the scores of models and agents.

Arena

Get two answers with the model names hidden and pick the better one.

Traces

View the processing steps as a tree and a timeline, and check token usage.

OpenTelemetry export

Send traces to your company's monitoring tools.

Alerts

Sends a notification by email or webhook when a condition you set is met.

Usage and cost

See usage and estimated cost by model and user.

Admin settings

Settings each company configures

  • Set the evaluation model and what it evaluates.
  • Set alert conditions and notification channels.
  • Choose whether to export traces via OpenTelemetry.
7Trace span types
Arena with model names hidden
OpenTelemetry export

Figures reflect the product as of October 2026.

Related features

Related pages

Model gatewaySet who can use each model, and its limitsAgentsBuild AI assistants for each departmentAdministration and auditManage users, sign-in, audit logs and retention periodsCode gatewayApplies company policy to developers' coding tools
FAQ

Frequently asked questions about evaluation and tracing

Does evaluation incur model costs?

Yes. An evaluation model does the scoring, so its calls cost money. The administrator decides what is evaluated and with which model.

Is work other than conversations traced too?

Yes. Scheduled tasks, document indexing and coding tool calls are traced on the same screen.

Does it integrate with our monitoring tools?

Yes. Traces are sent via OpenTelemetry and audit logs via syslog.

See it with
your own documents

In a consultation, we connect a few of your documents so you can check the answer quality for yourself. Send us your questions and our team will get back to you.

Contact salesRequest security documentation
Product videosReal product screens · Example company ACME Logistics · 16 videos