MCPLab

Reference

Tool and Response Assertions

Complete assertion guide with examples for tool checks and response checks.

When To Use Tool vs Response Assertions

  • Use tool assertions when behavior depends on action correctness (which tools were called, and in which order).
  • Use response assertions when behavior depends on final answer quality or format.
  • Use both together for high-confidence checks: action correctness plus answer correctness.

Tool Assertions

Tool assertions validate whether the agent used the right tools and sequence of calls.

required and forbidden tools
eval:
  tool_constraints:
    required_tools: [lookup_account, verify_identity]
    forbidden_tools: [delete_account]
allowed tool sequences
eval:
  tool_sequence:
    allow:
      - [lookup_account, verify_identity, process_refund]
      - [lookup_account, verify_identity, check_policy, process_refund]

Response Assertions

String response assertions are literal checks and are case-insensitive by default. Use regex only when you need pattern matching.

contains
eval:
  response_assertions:
    - type: contains
      value: refund processed
not_contains
eval:
  response_assertions:
    - type: not_contains
      value: internal error
starts_with
eval:
  response_assertions:
    - type: starts_with
      value: hello
ends_with
eval:
  response_assertions:
    - type: ends_with
      value: thank you
equals
eval:
  response_assertions:
    - type: equals
      value: success
regex
eval:
  response_assertions:
    - type: regex
      pattern: "refund\s+(processed|completed)"
jsonpath (exists or equals)
eval:
  response_assertions:
    - type: jsonpath
      path: $.status
    - type: jsonpath
      path: $.status
      equals: success
jsonpath_exists
eval:
  response_assertions:
    - type: jsonpath_exists
      path: $.data.id
jsonpath_not_exists
eval:
  response_assertions:
    - type: jsonpath_not_exists
      path: $.error

Behavior Notes and Edge Cases

  • contains/not_contains/starts_with/ends_with/equals are literal, case-insensitive string checks.
  • regex is case-insensitive by default and uses JavaScript regular expressions.
  • jsonpath/jsonpath_exists/jsonpath_not_exists require valid JSON in the final response.
  • If final response is not valid JSON, JSONPath assertions fail with an invalid JSON error.