MCP App Evals: How to Test Tool Calling Across GPT-4o, Claude, and Gemini (July 2026)

Unit tests tell you your code works. E2E tests tell you your MCP App renders inside a host-like runtime. Neither tells you whether GPT-4o, Claude, Gemini, or a newer model will pick the right MCP tool when a user asks for something in plain English.

That is what evals test. An eval sends realistic prompts to real models, exposes your tool list, then checks the tool call and arguments the model chooses. For MCP Apps, ChatGPT Apps, and Claude Connectors, that model-facing layer is where many bugs hide: vague tool names, overlapping descriptions, loose argument schemas, missing server instructions, and workflows that make sense to developers but not to a model.

TL;DR

Write evals for the prompts where model choice matters. Assert the selected tool, required arguments, argument shape, and expected no-tool cases. Run each case multiple times per model, because tool choice is probabilistic even when you lower temperature. Use evals to improve tool names, tool descriptions, inputSchema, outputSchema, and server instructions. Run them on main, releases, or manual CI jobs because they call paid model APIs.

What Evals Test

An MCP App exposes tools through the Model Context Protocol. The host gives those tool names, descriptions, schemas, and server instructions to the model. The model decides whether to call a tool, which tool to call, and what JSON arguments to pass.

That decision is separate from your app logic. Your handler can be correct, your resource can render perfectly, and your UI can pass visual tests. The app can still fail if the model calls search-photos when the user wanted show-albums, or if it passes "last month" into a field that expects an ISO date.

Evals cover that gap:

• Can the model find the right tool for the user’s intent?
• Can it extract required arguments from natural language?
• Can it avoid tool calls for unrelated prompts?
• Can it choose between tools with similar names?
• Can it follow server-wide rules, such as “preview before apply”?
• Can it keep behavior stable across ChatGPT, Claude, and other model families?

OpenAI’s Apps SDK docs describe MCP as the path through which the model consumes tool metadata, server guidance, structured content, and component state. That means your eval should treat metadata as part of the app contract, not as copy that only humans read.

The Layers Around Evals

Evals are useful because they test one narrow layer. Keep the other layers in place:

Layer	What it catches	Typical command
Unit tests	Pure functions, component branches, schema helpers	pnpm test:unit
Integration tests	MCP tool handlers, structuredContent, _meta, resource links	pnpm test:e2e
E2E tests	Host runtime rendering, iframe behavior, app interactions	pnpm test:e2e
Visual tests	Theme, viewport, display mode, layout drift	pnpm test:visual
Evals	Model tool selection and argument extraction	pnpm test:eval
Live tests	Real ChatGPT or Claude behavior after deployment	pnpm test:live

Run fast local tests on every push. Run evals when tool metadata changes, when a release is close, or when you need to compare model families. Run live tests when the deployed host behavior matters.

sunpeak keeps these layers in the same project. npx sunpeak new scaffolds an MCP App with tools, resources, simulations, inspector tests, and evals. npx sunpeak test init --server URL adds testing around an existing MCP server.

A Simple Eval

The smallest useful eval has a prompt and an expected tool.

// tests/evals/albums.eval.ts
import { expect } from 'vitest';
import { defineEval } from 'sunpeak/eval';

export default defineEval({
  cases: [
    {
      name: 'shows the album list',
      prompt: 'Show me my photo albums',
      expect: {
        tool: 'show-albums',
      },
    },
    {
      name: 'searches photos by subject',
      prompt: 'Find photos of sunsets',
      expect: {
        tool: 'search-photos',
        args: {
          query: expect.stringMatching(/sunset/i),
        },
      },
    },
  ],
});

The first case checks disambiguation. The second checks both tool selection and argument extraction. Use fuzzy matching for values that can be phrased more than one way, and exact matching when your schema requires a fixed enum or ID.

Run it explicitly:

pnpm test:eval

Evals are not part of the default cheap test path because every run can call real model APIs.

Configure Models Explicitly

Keep your model list in config so results can be compared over time.

// tests/evals/eval.config.ts
import { defineEvalConfig } from 'sunpeak/eval';

export default defineEvalConfig({
  models: [
    'gpt-4o',
    'gpt-5.6-terra',
    'claude-sonnet-4-20250514',
    'gemini-2.0-flash',
  ],
  defaults: {
    runs: 10,
    temperature: 0,
  },
});

Do not treat that list as permanent. OpenAI, Anthropic, and Google update their model families, and your users may sit behind hosts that change model routing. The point is to make the baseline visible. If you move from GPT-4o to a current GPT-5 model, add the new model next to the old one first, compare pass rates on the same evals, then remove the old baseline after you understand the difference.

OpenAI’s current model guidance recommends the GPT-5.6 family for complex production workflows and names Programmatic Tool Calling as a tool-heavy workflow feature. That does not make older eval baselines useless. It means you should keep tool-calling evals broad enough to catch behavior changes when a host or provider moves to a new model.

Choose Eval Cases by Risk

Start with the prompts that could route to more than one tool. A single app with one obvious tool does not need a large eval suite. A connector with ten tools absolutely does.

Good eval cases usually come from five places.

1. Tool disambiguation

When two tools overlap, write prompts that force the distinction.

{
  name: 'uses album view for album browsing',
  prompt: 'Show me my vacation albums',
  expect: { tool: 'show-albums' },
},
{
  name: 'uses search for individual photo lookup',
  prompt: 'Find the photo I took at the Eiffel Tower',
  expect: { tool: 'search-photos' },
}

If models split between the two tools, the names or descriptions are too close. Fix the schema before you blame the model.

2. Required argument extraction

Any required argument that comes from natural language deserves coverage.

{
  name: 'extracts date range',
  prompt: 'Show invoices from January through March 2026',
  expect: {
    tool: 'search-invoices',
    args: {
      startDate: expect.stringMatching(/^2026-01/),
      endDate: expect.stringMatching(/^2026-03/),
    },
  },
}

This catches weak parameter names like q, loose date formats, and missing required fields in inputSchema.

3. No-tool prompts

Your model should not call your app for unrelated questions.

{
  name: 'does not call a tool for an unrelated question',
  prompt: 'What is the capital of France?',
  expect: { tool: null },
}

No-tool cases matter for broad connectors because hosts may surface your tools in many conversations.

4. Multi-step workflow rules

Some apps require order. A code review app might need preview-changes before apply-changes. A payments app might need quote-transfer before submit-transfer.

Put the shared rule in MCP server instructions, then test it:

{
  name: 'previews before applying a change',
  prompt: 'Rename the project from Apollo to Atlas',
  expect: {
    sequence: ['preview-rename-project', 'apply-rename-project'],
  },
}

Use MCP server instructions for cross-tool rules. Use tool descriptions for single-tool behavior. Then add evals for the workflows that would be expensive or risky if the model skipped a step.

5. Regression prompts from real usage

When a user reports that the app “picked the wrong thing,” turn their wording into an eval. Keep the prompt close to what they typed. If you over-clean the prompt, you lose the bug.

Test Arguments, Not Just Tool Names

A green eval that only checks the tool name can still hide a broken call. Validate the arguments that matter:

• Required IDs
• Date and time formats
• Enum values
• Booleans for destructive or read-only paths
• Page size, sort order, and filter defaults
• User-provided text that must not be rewritten

For MCP Apps, this ties directly to inputSchema and outputSchema. The model fills the input. Your handler returns structuredContent. The resource reads that data. If any shape is too loose, the next layer gets fragile.

Use exact values when the value must match:

{
  name: 'uses read-only mode for a preview request',
  prompt: 'Preview what would change if I archived project A12',
  expect: {
    tool: 'preview-archive-project',
    args: {
      projectId: 'A12',
      destructive: false,
    },
  },
}

Use matchers when natural language can vary:

{
  name: 'normalizes customer name',
  prompt: 'Find invoices for Acme Incorporated',
  expect: {
    tool: 'search-invoices',
    args: {
      customer: expect.stringMatching(/acme/i),
    },
  },
}

If you keep accepting almost anything in the matcher, the eval stops telling you whether the schema works. Make matchers flexible around wording, not around the contract.

Read Results Like a Product Signal

Eval output should make the failure mode obvious.

tests/evals/albums.eval.ts
  shows the album list
    gpt-4o              10/10 passed (100%)  avg 1.1s
    gpt-5.6-terra        9/10 passed  (90%)  avg 1.4s
    claude-sonnet        8/10 passed  (80%)  avg 1.0s
    gemini-flash         6/10 passed  (60%)  avg 0.9s
      failures: called 'search-photos' instead of 'show-albums' (4x)

Do not chase 100 percent on every model if the prompt is genuinely ambiguous. A better goal is a stable pass threshold tied to user risk:

• 10/10 for destructive or irreversible actions
• 9/10 or better for primary user workflows
• 8/10 or better for low-risk discovery paths
• Lower thresholds only when you have a fallback, confirmation step, or live host difference you already understand

When a case fails, ask why that model made the choice. The answer is usually in your metadata.

Fixes That Usually Work

When evals fail, change the app contract before adding more prompt text.

Rename ambiguous tools

get-photos and show-albums both sound plausible for “show my albums.” Prefer names that encode the job:

• search-photo-assets
• list-photo-albums
• render-album-gallery

The model sees the tool name early, so names carry a lot of weight.

Put the difference in the first sentence

Tool descriptions should start with the distinction that matters:

List photo albums as grouped collections. Use this when the user asks for albums,
folders, collections, trips, or events. Do not use it to search individual photos.

Long descriptions with the useful bit in paragraph three tend to perform worse than short descriptions with the difference up front.

Tighten argument schemas

Use required fields, enums, minimums, maximums, and patterns where possible. Prefer cityName over q, startDate over from, and includeArchived over flag.

For MCP Apps, also test outputSchema and structuredContent in integration tests. Evals should verify that models can fill the input. Contract tests should verify that your handler returns the shape the resource expects.

Move shared rules into server instructions

If three tools must follow one policy, do not repeat that policy three times with slightly different wording. Use server instructions for cross-tool guidance, then keep individual tool descriptions focused.

Examples:

• “Preview tools must run before apply tools.”
• “Use read-only tools unless the user clearly asks to change data.”
• “For ambiguous account names, search accounts before reading account details.”

Then write evals for those rules.

Split or merge tools based on failures

If one tool has too many modes, split it. If two tools keep competing for the same user intent, merge them behind a mode enum. Fewer tools can improve routing when the user does not care about your internal distinction.

CI Without Burning Budget

Evals cost API credits and can vary. Keep them out of the default branch-push path unless your app is small and the budget is trivial.

name: Evals

on:
  push:
    branches: [main]
  workflow_dispatch:

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: pnpm/action-setup@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 22
          cache: pnpm
      - run: pnpm install --frozen-lockfile
      - name: Run MCP App evals
        run: pnpm test:eval
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          GOOGLE_GENERATIVE_AI_API_KEY: ${{ secrets.GOOGLE_GENERATIVE_AI_API_KEY }}

Use workflow_dispatch so a developer can run evals before merging a risky tool metadata change. Use scheduled runs if provider behavior drift matters for your app. Keep the raw failure output as an artifact when possible, because “called X instead of Y” is the useful part.

Where sunpeak Helps

You can write evals by hand around any MCP server, but the setup is repetitive. sunpeak packages it with the rest of the MCP App testing loop:

• npx sunpeak new scaffolds a full MCP App project with evals, simulations, resources, tools, and inspector tests.
• npx sunpeak test init --server URL adds tests around an existing local or remote MCP server.
• pnpm test:eval runs model tool-calling evals across your configured providers.
• The local inspector lets you debug the UI state that follows a tool call without spending host credits or refreshing a real ChatGPT or Claude session.

Use evals when the model has a real choice to make. Keep them small, tied to real prompts, and connected to the metadata you ship. That gives you a repeatable way to improve the part of an MCP App that normal tests cannot see: whether the model understands how to use it.