Methodology

How ShortlistTrace measures AI recommendation visibility.

ShortlistTrace is designed to measure repeated recommendation behavior, not to claim a permanent “rank” inside an AI model.

1. We build a buyer-question panel

We crawl a limited set of public pages from the company website and extract service, audience, positioning and trust signals. From those signals we generate non-branded questions that resemble real commercial research: vendor shortlists, comparisons, use cases, evaluation criteria and problem-solution queries.

2. We freeze the panel for comparisons

A baseline stores the exact buyer-question panel and a panel hash. A recheck reuses that same panel so changes are measured against the same questions rather than against a moving target. A new baseline intentionally creates a new panel when the business or methodology changes.

3. We repeat measurements across four AI engines

Paid measurement uses ChatGPT, Gemini, Claude and Perplexity. The standard audit collects three samples per question per engine. ChatGPT and Gemini are observed through search-oriented LLM scraper surfaces; Claude and Perplexity are observed through structured LLM Responses with web search. These are reported separately as different measurement surfaces rather than treated as identical products.

4. We separate visibility signals

Mention RateHow often the target company is mentioned in measured answers.
Recommendation RateHow often the target company is included in a recommendation context.
Citation RateHow often the target domain appears as a cited source.
Weighted Recommendation RateRecommendation visibility weighted toward higher-commercial-intent questions.

5. Measurement Confidence is not statistical significance

Our confidence label combines observation coverage, repeat agreement within each engine and cross-engine agreement. A HIGH label means the measured panel was well-covered and comparatively stable. Cross-engine agreement describes convergence between providers; it is not evidence that an answer is factually correct, unbiased, or statistically representative of all possible users and sessions.

6. We trace evidence, not just outcomes

We record cited sources, mentioned companies and recurring competitor patterns. Brand aliases are normalized where appropriate so product and parent-brand variants do not automatically appear as separate competitors.

7. We turn gaps into Fix Briefs

Fix Briefs connect measured buyer questions to a concrete page strategy, required sections, proof requirements, acceptance criteria and a same-panel validation plan. They are recommendations for what to investigate or improve, not guarantees that an AI system will change its answers.

Important limitations

  • Generative AI outputs can vary by time, provider, model version, account context and retrieval behavior.
  • ShortlistTrace observes selected AI surfaces; it does not represent every AI product, geography or user.
  • External citations can be vendor-owned, editorial, directories or other source types. Ownership should be verified before outreach.
  • Historical audits created under different methodologies may not be directly comparable.

Current repeated-measurement methodology: coverage + within-engine repeat agreement + cross-engine shortlist agreement. We version material methodology changes rather than silently rewriting historical baselines.