Understanding eval — the trust moat
Evaluation is what makes a connector trustworthy: every proposed tool is exercised against your real upstream in a sandbox before it can ship. Eval runs in the background — enqueue from the Evaluate screen and watch per-tool progress, or let the drift sweeper re-run it on a schedule later.
The suites
| Suite | What it proves |
|---|---|
| contract | The API does what the spec says: happy-path calls return the declared shapes. |
| behavior | Missing and invalid parameters fail the way the spec implies (no 500s from user input). |
| recovery | Transient upstream failures are retried with sane defaults. |
| safety | Read-only tools are proven read-only against their effect classes. Write tools get three sandboxed scenarios: invalid payloads must 4xx, replays stay consistent, and destructive calls require confirmation semantics — eval never fires a bare destructive call. |
| auth | Credentialed tools are probed with the real brokered key: accepted means not-401/403; a rejected key fails loudly (CREDENTIAL_REJECTED); no key at all fails as CREDENTIAL_REQUIRED. Tools with no credential requirement pass trivially. |
| custom | Your own scenarios, run as the authenticated caller. See below. |
A suite failure fails the run, and a failed run blocks release. That is the point: "passed" means something.
Certification
When a run completes, it's judged against the certification criteria:
- The run passed, with contract / behavior / recovery suites passing
(at least one scenario each, zero failures).
- Auth passed — or the tools honestly need no credential.
- Safety is verified (read-only for reads; the write-safety scenarios for writes).
- Every tool has suite coverage.
The verdict is stored on the run and shown in the run history as a Certified badge, with a checklist of all the checks. Prod promotion requires certification — see Publishing.
Your own scenarios
Built-in suites prove generic behavior, but only you know your edge cases: the city that must return current, the validation error that must be 422 with an error key, the deprecated argument combination that must keep working. Add custom scenarios per tool — given args → expect status and body keys:
- Enabled scenarios run inside every eval, as the authenticated caller,
on the same probe path as the built-ins.
- A failed scenario fails the run like any built-in failure, so it gates
release. Expectations are status + key-presence: pin the exact status for error cases, or leave it unpinned to expect 2xx.
Re-run and compare
Re-running eval shows a "Changes vs previous run" panel: regressions, new failures, fixed cases, added and removed scenarios — worst-first, with the pass-rate delta in percentage points. You always see whether your API got better or worse.
Sharing proof
Download report on the Evaluate screen exports the run as a self-contained HTML report (shareable, printable to PDF) via a signed URL. Suitcase for the team meeting, the security review, or the customer who asked "but does it work?"