How to measure AI visibility without fooling yourself

Uncertainty matters: NIST's guide to confidence intervals for proportions explains why a measured rate is an estimate. Its statistical assumptions need checking; correlated AI answers, changing retrieval and model updates can make a simple independent-sample interval inappropriate.

The most common way teams check their AI visibility is also the worst: type a question into ChatGPT, screenshot the answer, and draw a conclusion. Run the same question five more times and you'll often get a different list. Measuring a non-deterministic system takes the same discipline as polling - and skipping that discipline produces confident, wrong conclusions.

Sample, don't spot-check

A single answer describes one observation. Repeating the same question helps estimate how often a brand appears, but even 8 of 10 runs versus 1 of 10 is a sample estimate, not proof of a stable difference. Trends and the effect of changes need repeated comparable measurements and an honest account of uncertainty.

There's no universal correct sample size: it trades off against cost and the uncertainty you can accept. Keep the questions, providers and sampling settings comparable, and report the denominator. Rates from 5 and 50 observations can be compared, but they have different uncertainty; a change in rate alone does not establish a trend.

What Arcanoris actually samples

Within one Arcanoris audit, N = 1: one answer per question per provider. Free uses 5 questions × 1 provider = 5 checks. Plus uses 20 questions × 5 providers = 100 checks, not 100 repetitions of one question. Pro asks 20 questions once per selected provider, up to 10. Each additional audit or monitoring run consumes checks from your plan. Scheduled monitoring repeats selected questions over time; a single audit is a snapshot and should not be presented as evidence of a stable trend.

Hold the method constant

• Label the provider and model. "AI visibility" isn't one number - GPT-5 with web search, Gemini, and Perplexity behave differently and should be reported separately.
• Keep prompts unbiased. Asking "is Acme good?" tells the model the answer you want. Ask what a buyer would ask, with no brand in the prompt.
• Version the methodology. When you change prompts, models, or sample size, mark the break - otherwise you'll read a method change as a market change.
• Timestamp everything. Answers drift when models update; a number without a date is unusable.

Measure more than mentions

Alongside mentions, record position and citations. Position describes where a name appeared; it does not establish whether a buyer chose it. Citations identify the sources an answer exposes, not every influence on the model. Use them to inspect supporting evidence and content gaps, and measure actual customer outcomes separately.

Track competitors in the same runs

Your mention rate alone has no scale. Appearing in 40% of answers is excellent if your nearest competitor appears in 10%, and alarming if they appear in 90%. Because competitors get measured in the very same sampled answers, the comparison is apples to apples by construction - you're not comparing your Tuesday sample against their sample from a different week with a different model version behind it.

A minimal setup you can build yourself

You don't need a platform to start - a script and a spreadsheet gets you a real baseline. The shape of it is a nested loop: for each prompt, for each provider, run N times and record the result.

for prompt in buyer_prompts:
  for provider in [openai, anthropic, perplexity, gemini]:
    for run in range(N):
      answer = provider.ask(prompt)          # no brand name in the prompt
      log(
        prompt, provider, run,
        mentioned = brand in answer.named_brands,
        position  = answer.position_of(brand),
        sources   = answer.cited_urls,
        timestamp = now(),
      )

That log is the whole dataset. Mention rate, position, and cited-source frequency are all just aggregations over those rows - the discipline is entirely in running it the same way every time, not in the analysis.

What good audit results look like

Provider-labelled mention rates with sample sizes, position when mentioned, the cited sources ranked by frequency, competitor rates from the same runs, and a methodology version plus timestamp on all of it. This is the shape we built Arcanoris's audit results around - but the principles hold whether you use a tool or a spreadsheet and an API key.