Scenario Configuration

Scenario Fields

A scenario defines one test case the agent must execute against available MCP servers.

• id — unique scenario identifier. Use kebab-case.
• prompt — exact task instruction given to the agent.
• agent — optional pinned agent id. Omit to run all selected agents.
• servers — labels available in the scenario context.
• mcp_servers — concrete MCP server definitions (ref or inline server config).
• eval — assertions on tool usage, sequence, and response output.
• extract — capture values from final text using regex named group value.

Minimal Scenario Example

scenarios:
  - id: weather-baseline
    agent: claude-haiku
    servers: [weather-api]
    mcp_servers:
      - id: weather-api
        transport: http
        url: http://localhost:3000/mcp
    prompt: Get today's forecast for Brussels and summarize in one sentence.

Tool and Response Assertions

Add eval only after the baseline prompt run works, then tighten expectations incrementally.

For a full assertion catalog with examples for every type, see Reference / Tool and Response Assertions.

• Use required_tools to enforce critical tool calls.
• Use forbidden_tools to block unsafe or irrelevant tools.
• Use tool_sequence to require tools in order while allowing other tools in between.
• Use literal response assertions (contains, equals, etc.) for stable text checks.
• Use regex assertions for variable outputs (numbers, IDs, timestamps).
• Use JSONPath assertions when the response is structured JSON.

scenarios:
  - id: weather-asserted
    servers: [weather-api]
    mcp_servers:
      - ref: weather-api
    prompt: Return JSON with city and temperature_c for Brussels.
    eval:
      tool_constraints:
        required_tools: [get_weather]
        forbidden_tools: [delete_city]
      tool_sequence: [get_weather]
      response_assertions:
        - type: regex
          pattern: '([0-9]+)(\.[0-9]+)?\s?°?C'
        - type: jsonpath
          path: $.city
          equals: Brussels

Extract Structured Values

Use extract when you want reusable values from the final response for downstream checks or reporting.

scenarios:
  - id: extract-temperature
    servers: [weather-api]
    prompt: Report the current temperature in Brussels in Celsius.
    extract:
      - name: brussels_temp_c
        from: final_text
        regex: 'Temperature:\s*(?<value>[0-9]+(\.[0-9]+)?)\s*°?C'

Common Configuration Patterns

• Start with one scenario and one run (-n 1) until stable.
• Keep prompts deterministic; avoid broad, open-ended tasks first.
• Prefer mcp_servers refs for shared servers across many scenarios.
• Add one assertion at a time so failures are easy to diagnose.
• Split large workflows into multiple scenarios instead of one giant prompt.

Scenario Validation Checklist

• Scenario id is unique and readable.
• servers labels and mcp_servers mapping are coherent.
• Pinned agent exists in your agents list/library.
• Assertions test behavior, not cosmetic phrasing only.
• A focused command can run this scenario in isolation.

npx @inspectr/mcplab run -c mcplab/evals/eval.yaml -s weather-asserted -n 1