> Documentation index: [/llms.txt](/llms.txt). Every page as one file: [/llms-full.txt](/llms-full.txt). # Certify an agent Certify an agent [#certify-an-agent] A certification plays each test case `n_runs` times, compares the share of runs that pass every criterion with `threshold_pct`, and waits for its reviewer to approve or reject the result in the web app. Endpoints [#endpoints] All paths start with `/api/v1/diagnostics/certifications` and need `X-Space-Slug`. | Method | URL | Does | Auth | | -------- | -------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------- | | `GET` | `/defaults/` | `{n_runs, threshold_pct, run_options, max_n_runs, min_threshold_pct, turn_time_limit_s, min_turn_time_limit_s, max_turn_time_limit_s, max_turns_limit, criteria}` | Key or bearer | | `POST` | `/` | Launch a certification; `201`; billed | Key or bearer | | `GET` | `/` | The workspace's certifications, newest first, without reports: `{items, counts, reviewers}`. Filters: `search` (agent name, or `#n`), `reviewer_id`, `outcome` (`in_progress`, `certified`, `not_certified`) | Key or bearer | | `GET` | `/{certification_id}/` | Status, members and report | Key or bearer | | `GET` | `/{certification_id}/chats/{chat_id}/` | One of its conversations | Key or bearer | | `GET` | `/{certification_id}/chats/{chat_id}/files/{file_id}/presigned-url/` | A short-lived link to a file in one of its conversations | Key or bearer | | `POST` | `/{certification_id}/retry/` | Measure a failed certification again, as a new one; `201` | Key or bearer | | `POST` | `/{certification_id}/members/` | Add a member: `{"email"}` | Key or bearer | | `DELETE` | `/{certification_id}/members/{user_id}/` | Remove a member | Key or bearer | | `PUT` | `/{certification_id}/reviewer/` | Change the reviewer: `{"user_id"}` | Key or bearer | | `POST` | `/{certification_id}/decision/` | Approve or reject | Bearer only | | `POST` | `/people/check/` | Check that an email belongs to a member of this workspace: `{"email"}`; `204`, or `422` | Key or bearer | * A certification belongs to the workspace that launched it: everyone there who can read agents sees it. * Changing members or the reviewer, and retrying, need edit access to agents, and stop once the certification is approved. Request fields [#request-fields] | Field | Rule | | ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | | `agent_id` | Required | | `dataset` | Required, 1–25 rows, each `{"simulated_user": …}` as in [Build an environment](./environments) | | `file_bucket` | The files the cases hold, `[{file_id, description}]` | | `n_runs` | 1–50, default 20 | | `threshold_pct` | 50–100, default 80 | | `turn_time_limit_s` | Seconds one agent turn may take, 60–3600, default 600. A turn over it ends the conversation there, and that run fails without being judged | | `max_turns` | Messages each simulated person sends at most, 1–20, the same for every case. Default: the highest any case asks for | | `members` | Optional, up to 50 `{"email"}`; each member is emailed | | `reviewer_email` | Optional: the launcher's own or a member's; default: the launcher | * The launcher is always a member. * `GET /{id}/` returns the `turn_time_limit_s` the certification ran with; a retry keeps it. * Every run is judged on the platform's default criteria, listed in `GET /defaults/` → `criteria`; an environment's checks do not apply. Launch and read a certification [#launch-and-read-a-certification] 1. Build the scenarios as in [Build an environment](./environments), steps 1–2, or reuse the `dataset` of a saved environment from `GET /api/v1/diagnostics/kits/{kit_id}/`. 2. Launch it: ```bash curl -X POST "$CENTRAL_HUB_URL/api/v1/diagnostics/certifications/" \ -H "X-API-Key: $ALO_API_KEY" \ -H "X-Space-Slug: $WORKSPACE_SLUG" \ -H "Content-Type: application/json" \ -d '{ "agent_id": 42, "dataset": [{"simulated_user": {"id": "deadline", "persona": "A bid manager at a construction firm", "context": "Preparing a bid for a public tender", "goal": "Learn the exact submission deadline", "opening": "When is the deadline for this tender?"}}], "file_bucket": [], "members": [{"email": "reviewer@example.com"}], "reviewer_email": "reviewer@example.com" }' ``` 3. Poll `GET /{id}/` until `status` is no longer `running`; see [Statuses](#statuses). While it runs, `report.partial` is `true` and the rate is so far, not a verdict. 4. Read the result in `report`; see [Report fields](#report-fields). Statuses [#statuses] | `status` | Means | | ----------------------- | --------------------------------------------- | | `running`, `cancelling` | In progress | | `awaiting` | Measured; waiting for the reviewer | | `approved`, `rejected` | The reviewer decided | | `completed` | Ended with nothing judged; nothing to approve | | `cancelled`, `failed` | Stopped; `failed` carries `error_message` | Report fields [#report-fields] | Field | Holds | | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `runs_passed`, `executions`, `meets_threshold` | The verdict: runs that passed every criterion, out of all runs | | `judged`, `conversations_total` | How far it got | | `criteria[]` | Each criterion with `{label, passed, evaluated}` | | `cases[].executions[]` | Each run: `met`, `failed_criteria[]` with `{id, label, reason}`, `timed_out_turn` (the turn that ran over the time limit, or `null`), and `chat_id`; read the conversation with `GET /{id}/chats/{chat_id}/` | | `avg_cost_usd`, `avg_latency_s` | Per conversation | To stop a certification, cancel its gym session: `POST /api/v1/diagnostics/gym/{gym_session_id}/cancel/`, with `gym_session_id` from `GET /{id}/`. Troubleshooting [#troubleshooting] | Response | Cause | Fix | | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- | | `422` `No one in this workspace signs in as …` | An address in `members` or `reviewer_email` is not a member of this workspace; nothing ran | Use the address the person signs in with, or invite them to the workspace first | | `422` `The reviewer must be one of the members.` | `reviewer_email` is not the launcher or in `members` | Add the reviewer to `members` | | `403` `Changing who a certification is shared with needs edit access.` | Members or reviewer changed without edit access to agents | Ask someone who can edit agents | | `409` `An approved certification can no longer change hands.` | Members or reviewer changed after approval | Nothing to do | | `409` `Choose another reviewer before removing this one.` · `Whoever launched it stays a member.` | Removing the reviewer or the launcher | Change the reviewer first; the launcher stays | | `422` `The reviewer must be a member first.` | `PUT /reviewer/` with someone not in the members | Add them with `POST /members/` first | | `409` `Only a failed certification can be retried.` · `Its agent was deleted.` | Retry on a certification that did not fail, or whose agent is gone | Launch a new one | | `401` `{"detail": "Not authenticated"}` on `/decision/` | This route takes a signed-in user only | The reviewer decides in the web app |