In Development

PilotForge

Usability studies conducted by AI participants who have never seen the product.

PilotForge tests how usable a web app is by sending it AI participants. A researcher agent briefs each participant, a persona who has never seen the product, and sets it a real task. The participant works through the app in a browser, thinking aloud as it goes, and the researcher writes up what went wrong, ranked by severity.

Because a single run can mislead, every study is repeated, and a finding is treated as confirmed only when it recurs in most runs of the same persona. A review app replays each study step by step.

PilotForge is in development and used within the firm, where its one subject so far is Umami.

Particulars

  • Participants with distinct personas, kept away from the answer key
  • Each study run four times; only findings that repeat are filed
  • Findings graded by severity, with differences between personas noted
  • Cost metered on every run, with an optional spending cap
  • A replay viewer for reviewing each study

Built withTypeScript, Playwright and the Anthropic API

All projects