Live Testing for Claude Connectors and ChatGPT Apps: Test Against Real Hosts with Playwright (August 2026)

A local MCP App test can prove that a tool returns the right data and that its UI works in a host replica. It cannot prove that a real ChatGPT or Claude account can reach your server, discover the current tool definition, select it from a prompt, pass host policy, and render the production resource. Live testing covers that last mile.

TL;DR: Keep broad regression coverage local. Add one or two Playwright smoke tests per real host for release-blocking flows. Record enough evidence to tell a product failure from a host, account, model, or network failure. sunpeak automates the real ChatGPT flow today and provides local ChatGPT and Claude replicas for the larger test matrix.

What Live Testing Means

A real-host test starts with a natural-language prompt and ends with a user-visible result. That path crosses several boundaries:

1. The host reaches the public MCP endpoint and completes the MCP handshake.
2. The host loads the expected tool names, descriptions, schemas, annotations, and linked UI resources.
3. The model maps the prompt to the intended tool and produces valid arguments.
4. The server handles auth and returns a valid tool result.
5. The host loads the app resource in its sandbox and connects the app bridge.
6. The user completes the one interaction that matters to the release.

A direct tools/call request covers step four well, but it skips model selection and most host behavior. A local inspector can cover the app bridge, host context, display modes, themes, and error fixtures without spending host credits. A live test earns its cost by testing the boundaries that neither lower layer owns.

The stable MCP Apps extension describes tool-linked interactive views rendered by a host. Host support still varies, so a passing test in one client does not certify another. Treat ChatGPT and Claude as separate deployment targets even when they consume the same MCP server and ui:// resource.

The August 2026 Host Reality

Both hosts have changed their setup flows since the first version of this guide.

OpenAI now documents custom MCP apps under Settings > Apps and workspace app controls, not the older Settings > Security and login plus Plugins path. Developer mode access depends on plan, role, and workspace policy. Creating an app includes entering the endpoint, choosing auth, scanning tools, and creating a draft. OpenAI also documents a frozen tool snapshot for approved workspace apps, so an admin must review refreshed actions when tool definitions change.

Claude custom connectors use remote MCP endpoints. Individual users can add one under Settings > Connectors, while Team and Enterprise owners can add connectors for the organization. Users then enable the connector for a conversation. Anthropic states that real connector testing happens in the production Claude client because there is no separate staging host.

Claude reaches remote connectors from Anthropic infrastructure, even when the user opens Claude Desktop. The endpoint needs public DNS and host reachability. Redirects to a different host can drop the Authorization header, and OAuth discovery failures can look like connection failures. Those are good examples of defects that a laptop-only test will miss.

There is also an important sunpeak boundary to state clearly. The current sunpeak@0.20.81 live fixture ships a ChatGPT page adapter. It does not yet ship a real-Claude browser adapter. sunpeak’s inspector still reproduces both ChatGPT and Claude runtimes locally, so you can run a cross-host regression matrix there. For real Claude, use a focused manual smoke check or maintain a small Claude Playwright page object until your framework version includes that adapter.

Build a Test Matrix by Ownership

The fastest test suite puts each assertion at the lowest layer that can answer it.

Question	Best layer	Typical trigger
Does the tool validate inputs and return the right schema?	Unit or MCP protocol test	Every change
Does the UI handle loading, empty, error, and large-data states?	Inspector e2e test with fixtures	Every pull request
Does layout work in ChatGPT and Claude host contexts?	Cross-host inspector and visual tests	Every UI change
Will several models select the tool for realistic prompts?	Repeated model eval	Tool metadata changes
Can the real host reach, select, call, and render the app?	Live Playwright smoke test	Pre-release or scheduled
Does OAuth survive expiry and reconnect?	Protected live auth test	Auth changes and scheduled checks

This split keeps the live suite small. A 200-row result, a denied permission, a slow backend, and every viewport size belong in deterministic fixtures. The real-host suite should answer a short release question, such as “Can a workspace user open the current incident dashboard and acknowledge one seeded incident?”

Design One High-Signal Smoke Test

Pick a flow with these properties:

• It invokes one named app or connector from a direct prompt.
• It reads deterministic data from a dedicated test tenant.
• It renders a UI element with a stable accessible name.
• It performs a reversible or harmless action.
• It leaves a server-side event you can find by run ID.

Seed the tenant before the test and include a unique run ID in the record or prompt. Cleanup should be idempotent, which means it can run twice without deleting unrelated data. Avoid personal accounts and live customer records. Give the test account only the scopes the flow needs.

A useful prompt is specific:

Open incident LIVE-204 in the incident dashboard.

“What needs my attention?” may be a good eval prompt, but it gives a live smoke test too many valid outcomes. Test broad discovery with repeated evals. Use the live prompt to test the host path with less model variance.

Automate a Real ChatGPT Test with sunpeak

In a sunpeak project, the live Playwright config can stay small:

// tests/live/playwright.config.ts
import { defineLiveConfig } from 'sunpeak/test/live/config';

export default defineLiveConfig({
  devOverlay: false,
});

The test must import the live fixtures from sunpeak/test/live:

// tests/live/incidents.spec.ts
import { test, expect } from 'sunpeak/test/live';

test('opens a seeded incident in ChatGPT', async ({ live }) => {
  const app = await live.invoke(
    'Open incident LIVE-204 in the incident dashboard.',
  );

  await expect(
    app.getByRole('heading', { name: 'Incident LIVE-204' }),
  ).toBeVisible();

  const acknowledge = app.getByRole('button', { name: 'Acknowledge' });
  await expect(acknowledge).toBeEnabled();
  await acknowledge.click();

  await expect(app.getByText('Acknowledged by live-test')).toBeVisible();
});

live.invoke() starts a fresh conversation, sends host-formatted input, waits for the nested app iframe, and returns a Playwright FrameLocator. Keep assertions inside that frame. Model response wording and ChatGPT chrome are outside your product contract.

Run the local suite first, then the live test:

pnpm test
pnpm test:e2e
pnpm test:visual
pnpm test:live

The live runner uses a visible browser because real ChatGPT blocks common headless automation. It starts sunpeak with production resources, imports or requests a browser session, and refreshes the connected app before tests run. The app name must match the project package name because the current adapter uses that name when addressing the app.

Do not add hosts: ['chatgpt', 'claude'] to this config with sunpeak@0.20.81. The config type accepts host names, but the installed live fixture registry currently implements ChatGPT only. Check the release you have installed before enabling another host.

Automate Claude Without Coupling Tests to Its DOM

For a real Claude test, keep host navigation behind a page object and reuse app-level assertions. A small interface is enough:

type LiveHost = {
  openNewConversation(): Promise<void>;
  enableApp(name: string): Promise<void>;
  sendPrompt(prompt: string): Promise<void>;
  waitForApp(): Promise<import('@playwright/test').FrameLocator>;
};

Your Claude adapter owns login detection, connector selection, conversation setup, streaming completion, and nested iframe lookup. The test owns the product result:

async function expectIncidentApp(host: LiveHost) {
  await host.openNewConversation();
  await host.enableApp('incident-dashboard');
  await host.sendPrompt('Open incident LIVE-204 in the incident dashboard.');

  const app = await host.waitForApp();
  await expect(
    app.getByRole('heading', { name: 'Incident LIVE-204' }),
  ).toBeVisible();
}

Do not spread host selectors through every spec. Chat products change their DOM often. One adapter gives you one place to update. Prefer roles, accessible labels, and frame URLs over generated classes. Keep a manual runbook beside the automation so someone can reproduce a failure when login, bot detection, workspace policy, or a host rollout blocks the runner.

Before the Claude test, add the endpoint under Settings > Connectors, connect the test account, and enable the connector in the new conversation. If you changed the server URL or auth configuration, Claude may require you to remove and re-add the custom connector.

Assert on Contracts You Own

Stable live assertions usually target:

• A heading, table, form, or status inside the app iframe.
• A deterministic record ID from the test tenant.
• A button enabled only after tool data arrives.
• A state change backed by a server request ID.
• A host context that your product truly needs, such as dark theme readability.

Avoid exact model prose, host spacing, generated CSS classes, timestamps, random production data, and pixel snapshots of host chrome. The host owns those details and can change them without breaking your app.

Use Playwright’s web-first assertions instead of sleeps:

await expect(app.getByRole('status')).toHaveText('Ready');

A fixed waitForTimeout(10_000) is slow when the result arrives early and unreliable when it arrives late. A locator assertion waits for the state you need and reports the missing contract when it times out.

Record Evidence, Not Just Pass or Fail

A live test crosses systems owned by several teams, so the failure report needs enough context to route the problem. Store:

• Host and surface, such as ChatGPT web or Claude.ai.
• Account or workspace type, without personal details.
• App or connector identifier and metadata refresh time.
• MCP server commit, deployment ID, and endpoint hostname.
• App resource URI or bundle hash.
• Exact prompt, selected tool, and arguments when available.
• Tool result category and server request ID.
• Playwright trace, screenshot, browser console errors, and failed requests.

Redact tokens, cookies, tool secrets, and customer data before uploading artifacts. Keep traces from dedicated test tenants only.

This evidence also helps detect a stale tool snapshot. If the deployment ID is current but the host calls an old tool name or omits a new argument, refresh or rescan metadata before changing application code.

Classify Failures Before Retrying

Blind retries make an unstable suite look green. Classify the failure first:

Connection failed before initialize. Check public DNS, TLS, WAF rules, endpoint redirects, transport response, and host egress. For Claude, copy the ofid_ reference ID shown by connector setup and match it with server and edge logs.

OAuth failed. Check protected resource metadata, authorization server discovery, PKCE support, client registration, scopes, redirect URIs, and refresh-token issuance. Test expiry and revocation with a separate protected job.

The host did not call the tool. Confirm the connector is enabled, refresh the host’s tool snapshot, inspect the prompt and tool description, and check whether workspace action controls disabled the tool. Put ambiguous prompts into repeated evals instead of retrying until one passes.

The tool ran but returned the wrong data. Reproduce the request with a protocol test using the captured arguments. This is usually a server contract issue, so add the regression below the live layer.

The tool result arrived but no app appeared. Check the linked resource URI, MIME type, production asset URLs, CSP metadata, iframe console, app bridge connection, and non-zero app height. Claude’s MCP App troubleshooting guide also warns that very large results may be moved out of the inline result path, so paginate and load detail on demand.

The app rendered but the assertion failed. Compare the screenshot and trace with the server event. Fix brittle selectors or seed drift before blaming the host.

Retry navigation, a transient 429, or session bootstrap after logging the first failure. Do not automatically retry a wrong tool choice or broken UI state because those may be the exact regressions the test exists to catch.

Refresh Metadata as Part of Release

Tool metadata is deployed state. Refresh it when you change:

• Tool names or descriptions.
• Input and output schemas.
• Read-only, destructive, or approval annotations.
• Linked app resource URIs.
• OAuth scopes or discovery metadata.

For ChatGPT drafts, rescan or refresh the app before testing. For published workspace apps, follow the admin review flow because OpenAI says tool updates are not applied automatically. For Claude, reconnect or re-add the custom connector when its URL or configuration changes, and verify the connector is enabled in the test conversation.

Make the smoke test run against the exact server revision you plan to release. A tunnel to an uncommitted local branch can prove that a developer machine works while saying nothing about the deployed bundle.

Run Live Tests on a Deliberate Schedule

Most teams should run live tests in two places:

1. Before a release that changes tool metadata, auth, resources, or host-facing UI.
2. On a protected schedule that catches host changes, expired sessions, and policy drift.

Do not make every pull request wait on a real host by default. Real sessions expire, models vary, hosts rate-limit automation, and UI rollouts can break selectors. A scheduled smoke test should alert an owner with the trace and classification data. A local failure should block the pull request.

Keep concurrency low and start each test in a fresh conversation. Use a dedicated tenant with seeded records, least-privileged OAuth scopes, reversible actions, and explicit cost limits. Never let a browser smoke test send email, move money, delete customer data, or approve a real workflow.

A Release Checklist

Before running the host test:

• Unit and MCP protocol tests pass.
• ChatGPT and Claude inspector tests pass with fixed simulations.
• Visual and accessibility checks pass for changed UI.
• Tool-selection evals meet their threshold when metadata changed.
• The production-like endpoint has valid HTTPS and no cross-host redirect.
• OAuth metadata is reachable from outside the development network.
• The host has the current tool snapshot.
• The test tenant contains the seeded record.

After the run, confirm the app state and the matching server event, then keep the trace long enough to compare with the next failure.

Where sunpeak Fits

You can build real-host tests with raw Playwright, but authentication, chat setup, host refresh, prompt formatting, streaming waits, and nested iframe lookup create ongoing maintenance. sunpeak’s live fixture owns those details for ChatGPT, while your test keeps normal Playwright assertions:

const app = await live.invoke('Open incident LIVE-204.');
await expect(app.getByRole('heading', {
  name: 'Incident LIVE-204',
})).toBeVisible();

Use the sunpeak testing framework and MCP App inspector for the broad, deterministic suite across ChatGPT and Claude replicas. Add the real ChatGPT smoke test after that. Keep a separate real-Claude smoke path until the installed sunpeak release ships a Claude live adapter.

That split gives each test one job: local tests explain whether your code works, evals measure whether models find it, and live tests confirm that the current host can reach and render the release.