Configuration Schema

servers[ ]

• id (string, required) — unique identifier used in scenario server lists.
• transport (string, required) — connection type. Use "http" for HTTP/SSE.
• url (string, required) — full URL of the MCP endpoint.
• token (string, optional) — bearer token for authentication. Use a literal string or $ENV_VAR to read from the environment.

agents[ ]

• id (string, required) — unique identifier referenced by scenarios.
• provider (string, required) — LLM provider. One of: anthropic, openai, azure.
• model (string, required) — model identifier as used by the provider API.
• temperature (number, optional) — sampling temperature. Defaults to 0.

scenarios[ ]

• id (string, required) — unique identifier for the scenario.
• servers (string[], required) — list of server ids to give the agent access to.
• prompt (string, required) — the task description sent to the agent.
• agent (string, optional) — id of a specific agent to run. When omitted all agents run this scenario.
• eval (object, optional) — assertions to check after the agent completes.

eval.tool_constraints and eval.response_assertions

• tool_constraints.required_tools (string[], optional) — tool names the agent must call.
• tool_constraints.forbidden_tools (string[], optional) — tool names the agent must not call.
• response_assertions supports: contains, not_contains, starts_with, ends_with, equals, regex, jsonpath, jsonpath_exists, jsonpath_not_exists.
• For detailed examples of each assertion, see Reference / Tool and Response Assertions.

$ref Syntax

Use $ref inside a servers or agents list item to pull in a definition from an external YAML file.

servers:
  - $ref: ./shared/servers.yaml#my-server