Fixing Flaky Tests in MCP Apps, ChatGPT Apps, and Claude Connectors (August 2026)

TL;DR: A flaky test has an uncontrolled input. In an MCP App, that input may come from the model, protocol revision, tool handler, external API, iframe lifecycle, host context, app-initiated tool call, browser, clock, or shared state. First reproduce the failure and identify its layer. Then replace live dependencies with fixed contracts, use simulation fixtures for UI states, wait on observable conditions, isolate every worker, and move model and real-host behavior into eval or live suites with their own thresholds. Retries can collect evidence, but they do not make a test reliable.

---

A flaky test passes and fails on the same commit. The failure may disappear on retry, under a debugger, or when you run the spec alone. That pattern tells you the test depends on something besides the code and declared fixture data.

MCP Apps, ChatGPT Apps, and Claude Connectors have more boundaries than a normal browser app. A single user action can involve model tool selection, an MCP request, an upstream API, a tool result, a sandboxed iframe, host-to-app notifications, and app-to-server calls. Treating that chain as one test creates a large search area when anything varies.

The practical fix is to split the chain into contracts. Make each contract deterministic, then reserve a small set of tests for the boundaries you cannot control.

Start With a Failure Taxonomy

Do not begin by increasing the timeout. Capture the failure and assign it to the first layer that behaved differently.

Layer	Typical flaky symptom	Evidence to capture	Deterministic test
Model	Different tool or arguments	Prompt, model ID, tool catalog, calls	Multi-run eval
Core MCP	Missing tool, lost state, wrong response	Protocol revision, method, headers, request ID	Protocol contract test
Tool handler	Timeout or changed payload	Input, upstream request, normalized result	Unit or integration test with stub
MCP App lifecycle	Blank iframe or missing initial data	UI messages, handshake completion, console	Host-replica E2E test
App-to-server tool call	Button stays pending or shows wrong state	Tool name, arguments, result or error	Simulation with server tool mock
Host context	Layout or theme assertion changes	Host, viewport, theme, display mode, locale	Explicit context matrix
Browser	Element detached or hidden	Trace, screenshot, DOM, browser version	Locator-based E2E test
Environment	CI-only failure	Worker count, timezone, port, process logs	Isolated worker test

This table keeps a model-routing issue out of a component test and keeps a CSS timing issue out of an OAuth investigation. Fix the first boundary that differs, not the last assertion that notices it.

Reproduce the Failure Before Fixing It

A weekly failure is hard to diagnose. Turn it into a local 1-in-20 failure with the smallest possible command.

For Playwright:

pnpm exec playwright test tests/e2e/search.spec.ts --repeat-each=20 --workers=1 --trace=retain-on-failure

One worker removes parallelism while you measure the baseline. If that stays green, rerun with the CI worker count. A failure that appears only under parallel load usually points to shared ports, files, server processes, database rows, caches, or browser storage.

For Vitest, repeat the focused test and then shuffle the suite:

for run in {1..50}; do pnpm exec vitest run tests/tools/search.test.ts || break; done
pnpm exec vitest run --sequence.shuffle

Keep these artifacts for every failed attempt:

• The exact seed, test order, worker index, retry index, and duration.
• Playwright trace, screenshot, DOM snapshot, browser console, and failed requests.
• MCP method, tool name, protocol revision, request ID, normalized input, and result type.
• Server and upstream logs joined by one correlation ID.

Do not log access tokens, cookies, authorization headers, or private tool data. Correlation IDs are useful because they connect the browser failure to the matching server request without exposing credentials.

Separate Deterministic Tests From Variable Tests

One suite should not mix two definitions of success.

A deterministic test has fixed inputs and one expected outcome. Unit, protocol, component, visual, and local end-to-end tests belong here. These tests should gate every pull request.

A variable test measures a distribution or a changing external system. Model evals, real-host tests, production smoke tests, and third-party API checks belong here. They may run on main, on a schedule, or before release, and they need thresholds plus enough evidence to distinguish a product regression from provider downtime.

This split makes the signal clear:

Suite	External model	External host	External API	Expected result
Unit and protocol	No	No	Stubbed	Exact
Local MCP App E2E	No	Host replica	Mocked or local	Exact
Visual regression	No	Host replica	Fixture	Exact within image tolerance
Model eval	Yes	No	Usually no	Pass-rate threshold
Live-host smoke	Yes	Yes	Production-like	Small critical-path set

If a pull-request test calls a real model and waits for a real host UI, it cannot promise the same output or latency on every run. Move that coverage to the right suite instead of adding retries to the wrong one.

Fix Model Variance With Evals

Models may choose different valid tools or arguments for the same prompt. Even a low temperature does not guarantee byte-for-byte output. A single exact assertion turns normal model variance into random CI failures.

Run each case several times and define the acceptable rate:

// tests/evals/search.eval.ts
import { expect } from 'vitest';
import { defineEval } from 'sunpeak/eval';

export default defineEval({
  runs: 10,
  threshold: 0.8,
  cases: [
    {
      name: 'searches invoices by customer',
      prompt: 'Find invoices for Acme',
      expect: {
        tool: 'search-invoices',
        args: {
          customer: expect.stringMatching(/acme/i),
        },
      },
    },
  ],
});

Partial argument matching matters because capitalization and harmless phrasing changes are often valid. Keep strict assertions for fields with one valid value, such as an account ID selected earlier in the conversation.

Pin the model identifier and tool catalog used by the run. A model update, tool-description edit, or added overlapping tool can change routing rates without changing the app UI. Store those inputs beside the aggregate result so a rate change is explainable.

An 8-of-10 threshold is only an example. Choose the threshold from product risk and enough historical runs. A destructive write tool may need stronger routing and confirmation evidence than a read-only search tool.

Pin Tool Results With Simulation Fixtures

Component and browser tests should not depend on a live tool handler when the goal is to verify the UI. Give the app a complete fixed tool input and result instead.

Current sunpeak projects discover JSON fixtures from tests/simulations/:

{
  "tool": "search-invoices",
  "userMessage": "Find invoices for Acme",
  "toolInput": { "customer": "Acme" },
  "toolResult": {
    "structuredContent": {
      "invoices": [{ "id": "inv_101", "status": "open", "amount": 12500 }]
    }
  }
}

Rendering without explicit input loads the fixture, so the tool handler and its upstream services do not run:

import { test, expect } from 'sunpeak/test';

test('renders the open invoice state', async ({ inspector }) => {
  const result = await inspector.renderTool('search-invoices');
  const app = result.app();

  await expect(app.getByRole('heading', { name: 'Acme invoices' })).toBeVisible();
  await expect(app.getByText('$125.00')).toBeVisible();
});

Create one fixture for each state the UI owns: success, empty, partial data, permission denied, validation error, timeout, oversized content, and cancellation. Use stable IDs and UTC timestamps. Sort collections in the fixture or application before asserting on order.

Keep a separate protocol test that calls the real local handler and verifies its schema and normalized result. That gives you handler coverage without making every UI assertion depend on the handler.

Mock App-Initiated Tool Calls

An MCP App can call a server tool after the initial render, such as when a user clicks Approve. That adds another asynchronous boundary. If the E2E test sends the call to a real backend, account data and latency can make the button flow flaky.

Define fixed responses under serverTools in the same simulation:

{
  "tool": "review-invoice",
  "toolResult": {
    "structuredContent": { "invoiceId": "inv_101", "status": "pending" }
  },
  "serverTools": {
    "approve-invoice": [
      {
        "when": { "invoiceId": "inv_101" },
        "result": {
          "content": [{ "type": "text", "text": "Invoice approved." }],
          "structuredContent": { "status": "approved" }
        }
      }
    ]
  }
}

Test the outgoing name and arguments at the tool boundary, then test the rendered approved state. Add separate simulations for rejection, timeout, and isError: true. A mock that only covers success leaves the most timing-sensitive UI paths untested.

Wait for the MCP App Lifecycle, Not a Delay

MCP App UI runs in a sandboxed iframe and has its own ui/initialize lifecycle. The host may deliver tool input or a tool result close to initialization, then send host-context changes later. Code that installs listeners too late can miss the first notification. A test that clicks before the app is ready can race the handshake.

Fixed delays hide that race:

// Avoid this
await page.waitForTimeout(1000);
await app.getByRole('button', { name: 'Approve' }).click();

Wait for the state a user or caller needs:

const approve = app.getByRole('button', { name: 'Approve' });

await expect(approve).toBeVisible();
await expect(approve).toBeEnabled();
await approve.click();
await expect(app.getByRole('status')).toHaveText('Approved');

Playwright locator assertions retry until the condition succeeds or the timeout expires. The test finishes quickly on a fast machine and waits on a slower one.

Also test events that can arrive more than once:

• Tool input partial updates followed by final input.
• Tool result, error, and cancellation.
• Theme, locale, viewport, safe-area, and display-mode changes.
• Repeated host-context values, which should not duplicate work.
• Resource teardown and remount.

Do not use networkidle as proof that an MCP App is ready. An iframe can finish network requests before its UI handshake or React commit, and long-lived analytics requests can prevent network idle after the app is usable.

Treat Protocol-Version Skew as a Matrix

The MCP 2026-07-28 release changed the core transport model. It removed the core initialize exchange and Mcp-Session-Id, made requests self-describing, added header-based method and tool routing, and made list results cacheable. Older clients still use the session-oriented protocol.

This can look like flakiness when different clients or workers use different revisions:

• A load balancer sends a session-bound request to another instance.
• A server expects initialization state that a 2026-07-28 request does not have.
• A test reuses a cached tool list after changing the server fixture.
• A gateway strips Mcp-Method or Mcp-Name headers on only one route.
• Hidden state survives one test and changes the next request.

Record the protocol revision with every request and run each supported revision in its own test project. Do not let one worker negotiate an older revision while another silently uses the latest. For 2026-07-28, verify that a tool call can reach any stateless server instance. For older revisions, verify whatever session affinity your server still promises.

The MCP Apps UI handshake is separate from the core MCP transport revision. Do not remove ui/initialize handling from the iframe because the new core protocol removed server initialization. They are different contracts.

Stub External APIs at the Network Boundary

A tool handler that calls a real billing, weather, search, or database service inherits that service’s data changes, rate limits, latency, and outages. Stub the client at a clean boundary and assert on your handler’s normalization logic.

Prefer dependency injection over spying on an imported object:

type InvoiceClient = {
  search(customer: string): Promise<Array<{ id: string; cents: number }>>;
};

export function createSearchInvoices(client: InvoiceClient) {
  return async ({ customer }: { customer: string }) => {
    const invoices = await client.search(customer);
    return {
      structuredContent: {
        invoices: invoices.map((invoice) => ({
          id: invoice.id,
          amount: invoice.cents / 100,
        })),
      },
    };
  };
}

The test passes a fixed client. A smaller live-contract job can call the real service on a schedule to detect upstream API drift. That job should report provider errors separately from application regressions.

Test timeout and cancellation explicitly. Use a promise you control rather than a short real timer, then release or reject it at the exact point the assertion needs.

Isolate State Between Workers

Parallel tests reveal hidden shared state. Common sources include:

• One fixed port used by several MCP server processes.
• A shared test user, workspace, database row, or OAuth grant.
• Module singletons that retain a connection or cached tool list.
• One simulation object mutated by multiple tests.
• Reused browser storage, IndexedDB, local storage, or service workers.
• Snapshot files written by more than one worker.

Give each worker unique resources. Derive ports, tenant IDs, database schemas, and temporary directories from the worker index. Create a fresh browser context for tests that change storage or permissions. Clone fixture objects before modifying them.

Reset mocks, timers, and environment variables in cleanup that runs even after a failed assertion:

import { afterEach, vi } from 'vitest';

afterEach(() => {
  vi.useRealTimers();
  vi.restoreAllMocks();
  vi.unstubAllEnvs();
});

If the spec passes with one worker and fails in parallel, keep it running in parallel while you diagnose it. Serializing the whole suite removes evidence and makes CI slower.

Freeze Time, Locale, and Randomness

Dates fail at midnight, daylight-saving boundaries, month ends, and timezone differences. Set an explicit UTC instant and locale:

import { beforeEach, afterEach, test, expect, vi } from 'vitest';

beforeEach(() => {
  vi.useFakeTimers();
  vi.setSystemTime(new Date('2026-08-14T15:00:00Z'));
});

afterEach(() => vi.useRealTimers());

test('formats a recent update', () => {
  expect(formatRelativeTime('2026-08-14T13:00:00Z', 'en-US')).toBe('2 hours ago');
});

Inject random and ID generators so tests can provide a fixed sequence. For visual tests, pin fonts, browser version, viewport, device scale factor, reduced-motion preference, theme, and locale. Mask truly dynamic regions such as a generated avatar only when that region is outside the behavior under test.

Use Stable Selectors and Assertions

Selectors built from DOM depth, generated classes, or inspector chrome break when markup changes. Prefer accessible roles and names because they match the user-facing contract:

await expect(app.getByRole('heading', { name: 'Invoice review' })).toBeVisible();
await app.getByRole('button', { name: 'Approve invoice' }).click();

Use data-testid when there is no accessible semantic selector, such as an invisible canvas layer. Keep the ID about the product concept, not the CSS layout.

Assert the final state instead of an intermediate implementation detail. A loading spinner may appear too briefly on a fast machine, while the final approved state is the behavior that matters. When the loading state itself is a requirement, hold the mocked promise open, assert the spinner, then resolve the promise and assert the final state.

Keep Real-Host Tests Small and Observable

The official MCP Apps testing guide recommends a reference host for local work and a compatible host for real conversational testing. Claude has no separate connector staging environment; its connector testing guide says custom connectors use the same runtime as directory connectors.

Real-host tests still depend on a model, host UI, account, network, tunnel, OAuth state, and production service. Use them for questions that a local replica cannot answer:

• Can the host connect and authorize against the deployed endpoint?
• Does the real model select the tool for one representative prompt?
• Does the host render the production resource and pass host context?
• Can one critical app action call the server and show the result?
• Does reconnect or token refresh work after a clean session?

Start each flow in a new conversation and with known account data. Record the host, account type, app version, server commit, prompt, tool call, and correlation ID. A host UI update should fail the live suite without making deterministic pull-request checks red.

Manage Retries and Quarantine With Deadlines

Retries are useful for measuring a failure, not declaring it fixed. Preserve the first-attempt trace and report retry counts by test name. If a retry passes, the original failure still needs an owner.

When a flaky test blocks unrelated work and cannot be fixed immediately:

1. Move it to a named quarantine job that runs on every relevant change.
2. Keep the job visible and alert on failures.
3. Link an issue with an owner, first-failure evidence, and removal date.
4. Keep equivalent deterministic coverage in the blocking suite where possible.
5. Delete the quarantine marker after the root cause is fixed and the test passes repeated stress runs.

Do not mute the test, raise global timeouts, or add global retries. Those choices reduce signal for every test.

A Flake-Fixing Runbook

Use this order when a test passes on retry:

1. Save the first failure’s trace, logs, request data, and environment metadata.
2. Run the smallest spec repeatedly and measure the failure rate.
3. Compare passing and failing runs at the first differing layer.
4. Remove parallelism to test for shared state, then restore it after the fix.
5. Replace live model, host, API, time, randomness, and mutable data with fixed inputs.
6. Replace sleeps and generic load waits with observable state assertions.
7. Pin protocol revision, host context, browser, locale, timezone, and viewport.
8. Add a regression test that fails for the identified race or leak.
9. Run the focused test repeatedly, shuffled, and in parallel.
10. Run the full validation suite before removing retries or quarantine.

Where sunpeak Helps

sunpeak gives MCP App tests a local host runtime, Playwright fixtures, fixed simulation data, protocol access, and host-context controls. You can use it with a sunpeak framework project or point it at an MCP server written in another language:

npx sunpeak test init --server http://localhost:8000/mcp

The generated tests can render fixed UI states with inspector.renderTool(), inspect protocol results with the mcp fixture, and mock app-initiated calls with simulation serverTools. That removes real accounts, host credits, repeated manual refreshes, and live model routing from the tests you run on every commit.

Keep a small live-host suite and model evals for the behavior that is actually variable. The rest of the test system should produce the same result every time, so a red build means the product changed.