What I Learned Trying to Build My Own AI Visibility Tracking Tool

Why testing AI visibility tools made me more skeptical of AI visibility scores

I really wanted AI visibility tracking to work. Not because it’s the latest marketing trend, but because clients are asking a very reasonable question: Are we showing up when people use ChatGPT and other AI tools to research the things we sell? As a marketer, I want to know the answer too.

Over the past year, I’ve tested different AI visibility platforms and experimented with building our own tracking system at WellFound. At one point, I had a fairly elaborate process running through a spreadsheet that queried AI results, tracked brand and competitor appearances, captured citations and query fan-outs, and tried to turn all of that into something we could actually use.

What I learned in the process is that AI visibility absolutely matters. I'm just increasingly skeptical that we can measure it with anywhere near the precision some dashboards suggest. A recent LinkedIn post from Rand Fishkin caught my attention because he raised several of the same concerns I’d run into while testing these tools and trying to build our own.

I Ran Into This Trying to Build Our Own Tool

My frustration with the existing tools was what pushed me toward trying to build something ourselves. I wasn’t looking for another dashboard that simply told me a client’s brand was mentioned 32% of the time. I wanted to know which questions caused our clients to appear, which questions they weren’t appearing for, which competitors appeared instead, what websites and pages were being cited, and whether there was something those competitors were doing that we could learn from.

Most importantly, I wanted the data to tell us what we could actually do next.

That last part is where I think a lot of AI visibility reporting still falls short. Knowing that a competitor appears more frequently than you isn’t particularly useful by itself. I want to know why that might be happening and whether there is an actionable marketing opportunity behind it.

Maybe competitors have better comparison content. Maybe third-party publications mention them more frequently. Maybe their product or service information is structured more clearly. Maybe they have stronger coverage of a particular topic. Or maybe there simply isn’t enough evidence to know why they appeared and you didn’t. That last possibility needs to be an acceptable answer, too.

The Problem With an AI “Visibility Score”

The basic idea behind most AI tracking tools makes sense. Pick a set of prompts that matter to your business, run them through ChatGPT, Google AI, or other systems, record which brands appear, and repeat the process. From there, you can calculate something resembling share of voice or AI visibility.

The problem is that an AI answer isn’t a search engine results page. Ask an AI the same question repeatedly, and you may get different companies, different ordering, different wording, and different recommendations. That makes a metric like “Your AI visibility increased from 18% to 27%” look considerably more precise than the underlying data actually is.

That doesn’t necessarily mean the number is useless. It means we need to be much more careful about what we think it represents.

AI Results Aren’t Rankings

There’s another complication that makes this substantially harder: The AI answering my question isn’t necessarily the AI answering yours.

Previous conversations and other available context can influence the response someone receives. Two people can ask essentially the same question and get different answers. That creates a major measurement problem because we can measure what happened in a controlled test environment, but we can’t automatically assume those results represent what actual customers are seeing across thousands of individual conversations.

That distinction gets lost surprisingly quickly once AI visibility data makes its way into a marketing report. A dashboard turns an experiment into a percentage, puts it on a graph, and suddenly it feels like a KPI.

But what we may actually be measuring is something much narrower: how frequently a brand appeared when a specific set of prompts was run, under specific conditions, a specific number of times. That can still be useful information. It just isn’t the same thing as saying that a brand has 32% visibility in AI search.

We Don’t Really Know Prompt Volume Either

Traditional SEO has plenty of imperfect data, but at least we have decades of search behavior, keyword datasets, Google Search Console data, analytics, and other sources to work with. AI prompting doesn’t have an equivalent source of truth.

We don’t really know how many people ask a particular question across ChatGPT and other AI platforms, which becomes especially important when tools start estimating prompt volume or weighing some questions more heavily than others.

A business showing up for 80% of a collection of prompts sounds fantastic, but what if hardly anyone asks those questions? Meanwhile, that same company could be absent from a smaller set of questions that actually influence purchasing decisions. The visibility percentage alone can’t tell us that.

Attribution Is Even Messier

This may be the part of AI measurement that concerns me most. Marketers have spent years being asked to prove ROI, and while the measurement systems we use for SEO, paid media, email, and other channels are far from perfect, there is usually some kind of trail connecting activity to results.

With AI, that trail gets murky very quickly. Someone could have a 20-minute conversation with ChatGPT, discover a company they had never heard of, return three days later through a direct visit, and submit a form. Maybe AI played an important role in that sale. Maybe it didn’t. In most cases today, we have very little ability to know.

Until attribution improves, I don’t think we should manufacture certainty simply because we want another clean metric for a marketing report.

Citations Are Useful, but They Aren’t Proof of Causation

This was another area I became particularly interested in while building our own process. If ChatGPT recommends Company A and cites three websites in its response, it’s very tempting to conclude that those websites caused Company A to be recommended.

Maybe they did. But seeing a citation and proving that it influenced the recommendation are two very different things.

I still think citations are enormously useful from a marketing perspective. If the same publications, directories, industry resources, competitor pages, or types of content repeatedly appear around important topics, I want to know about them. They can uncover content gaps, digital PR opportunities, and places where a brand might benefit from having a presence.

What I don’t want to do is tell a client that getting mentioned on Website X will cause ChatGPT to recommend them. We simply don’t have enough evidence to make that kind of promise.

So Is AI Visibility Tracking Worth Doing?

I think it is. I just think we’re asking it to do things it can’t reliably do yet.

The measurement is getting better, particularly when prompts are tracked repeatedly and across broader topics rather than treated like individual keyword rankings. But that still doesn't make an AI visibility score equivalent to a traditional search ranking or a measure of business impact.

Instead of treating AI visibility percentages like an SEO ranking report, I’m becoming much more interested in using this data as competitive intelligence. Show me where competitors consistently appear and we don’t. Show me the types of questions where that happens. Show me which third-party sources repeatedly surface and which pages from our own website are being cited. Show me topics where an AI system appears to misunderstand what the company does.

Those findings can lead to actual work: improving content and SEO, clarifying positioning, strengthening important pages, earning third-party mentions, creating new resources, or simply doing a better job answering the questions prospective customers care about. Those are useful outcomes regardless of whether an AI visibility score is 23% or 31%.

How We’re Thinking About AI Visibility at WellFound

We’re continuing to pay attention to AI visibility because understanding how brands appear in AI-generated answers is becoming an important part of search strategy. But our approach is becoming less about chasing a single visibility score and more about looking for patterns we can actually use.

We want to know whether a client appears around strategically important topics, which competitors appear alongside or instead of them, which sources and pages keep surfacing, and whether that information points us toward something worth improving.

There’s a simple test I keep coming back to when looking at recommendations that come out of AI visibility research: Would this still be a good marketing decision if the AI visibility dashboard disappeared tomorrow?

If improving a page would make it more useful to prospective customers, do it. If earning coverage from a respected industry publication would strengthen the brand regardless of whether ChatGPT cites it, that’s probably worthwhile. If creating a new resource fills a genuine information gap for customers, there’s value in doing that even if we can’t prove it will increase an AI visibility score.

But if the entire justification for an initiative is moving an opaque percentage from one number to another, I’m not there yet.

AI search is changing quickly, and the ways we measure it will almost certainly get better. For now, I’d rather use imperfect AI data to find good marketing opportunities than turn imperfect data into a very precise-looking score and pretend we know exactly what it means.

To all my fellow marketers out there...let's keep at it.