Setting Up Evaluations

Recommended Project Layout

Use a consistent workspace layout so CLI and App commands resolve configs, libraries, and results predictably.

• Keep evaluation YAML files in mcplab/evals.
• Keep reusable servers and agents in library files.
• Store run output in mcplab/results/evaluation-runs.

mcplab/
  evals/
    eval.yaml
  results/
    evaluation-runs/
  servers.yaml
  agents.yaml

Author a Minimal, Valid Config

Start with one agent and one scenario. Validate this baseline first before adding more scenarios or models.

agents:
  - id: claude-haiku
    provider: anthropic
    model: claude-haiku-4-5-20251001
    temperature: 0

scenarios:
  - id: setup-check
    agent: claude-haiku
    servers: [demo-server]
    mcp_servers:
      - id: demo-server
        transport: http
        url: http://localhost:3000/mcp
    prompt: Use available tools to complete this setup verification task.

Configure Auth and Environment

Set provider keys and server auth variables before running evaluations. Keep secret values in environment variables, not in committed YAML.

• Use auth.type: bearer + env for bearer-token server auth.
• Use auth.type: oauth_client_credentials for client-credentials flows.
• Use auth.type: oauth_authorization_code when interactive/browser OAuth is required.

ANTHROPIC_API_KEY=...
OPENAI_API_KEY=...
MY_SERVER_TOKEN=...

Preflight Checklist

• MCP endpoint URL is reachable and returns MCP responses.
• Scenario IDs are unique and agent references resolve.
• Server labels in scenarios[].servers match your intended MCP server entries.
• All required env var names are set in your shell/session.
• You can run one scenario once successfully before scaling.

npx @inspectr/mcplab run -c mcplab/evals/eval.yaml -s setup-check -n 1

Next Setup Steps

• Add more scenarios after the baseline setup-check passes.
• Use --agents to compare models on the same scenarios.
• Open report.html or the App results view to inspect failures and tool traces.