AI search · 9 min read
How AI assistants pick sources, and which parts you can actually influence
Almost everything written about this mixes three very different kinds of statement: things the vendors have documented, things anyone can observe from outside, and things nobody outside the companies knows. Sorting them apart leaves a short list of real work and a long list of tactics with nothing behind them.
Written by Zubair Afzal, FounderUpdated
Start here
Three kinds of statement, routinely mixed together
Before evaluating any advice on this subject, work out which of these three it belongs to. Most disagreements dissolve once you do.
Documented
Published by the company that operates the system: which user agents fetch pages, how to allow or block each one, which robots and snippet directives are respected, and Google’s statement that its AI features require no special markup beyond ordinary indexing. This is the only tier where a claim can be checked against a primary source.
Observable
Things anyone can see from outside without knowing why they happen: which domains appear in a citation list, whether a page you changed starts being used, whether referral visits from an assistant appear in analytics. Real evidence, but correlational, small-sample and easily overread.
Unknown
How candidates are scored, how corroboration is weighted, what makes one of three near-identical pages the cited one, and whether any of that is stable between model versions. Nobody outside these companies knows, and the confident articles are extrapolating from a handful of prompts.
Tier one
What the vendors have actually put in writing
The documented layer is smaller than the discourse suggests, and it is almost entirely about access rather than preference. Vendors publish the user agents that fetch pages on their behalf and how to permit or refuse each one in robots.txt. Those are separate decisions: a crawler that gathers training data, a crawler that builds a search index, and a fetcher that retrieves a page live because a user asked something right now are three different agents, and a site can easily end up allowing one and refusing another without anyone deciding to.
Google documents that its AI features draw on the same index as ordinary search, that no special markup is required to be eligible, and that the standard snippet controls apply. Those controls are worth knowing because they are the one documented lever that reliably reduces your presence: nosnippet, a max-snippet limit and data-nosnippet on a specific element all restrict what can be shown, and a site that applied them years ago to protect content is applying them here too.
Microsoft is the transparent case, because Copilot grounds its web answers in Bing. That means the ordinary Bing Webmaster Tools console tells you what was crawled, what is indexed and what was rejected, and IndexNow lets you push a change the moment it is published rather than waiting to be recrawled. It is the only assistant surface where you can inspect the retrieval layer directly, which makes it the best place to find access faults that are probably affecting the others too.
Notice what is absent from all of it. No vendor publishes what makes a source likely to be selected. That silence is itself informative: the documented controls are negative, and everything positive is inference.
The work
Five things worth doing, in the order they pay off
Every step here is defensible on its own terms, which is the test. If a change only makes sense as an attempt to influence a model, it does not belong on this list.
Establish that you can be fetched
Check robots.txt, then the bot rules in your CDN or firewall, then the server logs for the published agents. Then check that the page renders its content in the HTML rather than assembling it in the browser afterwards. A source that cannot be retrieved is excluded before any of the interesting questions arise, and access faults are the single most common finding.
You get: A written record of which agents are allowed, and a decision behind each one
Give each claim one home
For every question you want to be the source on, pick one page, state the answer completely in the opening paragraph, and make the heading say the claim rather than tease it. If the same answer currently exists in fragments across four pages, consolidate it. This is a writing job, and it is the part that has an effect on human readers regardless of what any machine does.
You get: One canonical page per claim, with the answer stated at the top
Remove the contradictions you control
Your own site, your directory listings, your social profiles, your Companies House or registry entry, your Google Business Profile, the boilerplate at the bottom of your press releases. Make the description of what you do, who for and where match across all of them. Systems that cannot resolve you to one consistent organisation tend to describe a competitor instead.
You get: One canonical description, applied everywhere you control
Become the origin of something
A survey of your own customers, a dataset from your own operations, a documented method, a price index for your category. A source that restates what four other pages already say is interchangeable with them. A source that holds the only version of a fact is not, and that is the one advantage that does not depend on how retrieval happens to work this year.
You get: At least one publishable claim that only you can make
Keep a dated prompt log
Fix a set of prompts a real buyer would use. Run them monthly across the assistants that matter to you. Record the date, the exact wording, the assistant, and the full response rather than a score. It is sampling, it is noisy, and it is still the only honest way to see whether the description of your business is changing.
You get: A running log of raw responses with dates and wording
Tier three
Tactics with no documented mechanism behind them
These circulate widely. None is supported by vendor documentation, and none has an observable effect anyone has managed to isolate. Some are harmless, some are not.
- Publishing a file that instructs assistants how to read your site. No major vendor documents reading one, so the honest description is a hypothesis, not a deliverable.
- Writing instructions to a model inside your page, hidden or otherwise. Retrieval does not work that way, and where it is detected it is treated as manipulation.
- Markup described as AI-specific or answer-engine schema. There is no such vocabulary. Structured data is worth using for reasons that predate all of this.
- Adding a question-and-answer block to every page regardless of whether anyone asks those questions. It makes pages longer and less usable and does not make a claim more extractable.
- Buying mentions on sites that exist to be scraped. It creates corroboration that is trivially identifiable as coordinated, and the risk sits with you rather than the vendor.
- Any product offering a share-of-voice figure for an assistant without stating how many prompt runs it rests on. That number is sampling noise given a decimal point.
Everything durable on this subject is a consequence of being reachable, being correct and being consistent. Everything that only makes sense as an attempt to manipulate a model is undocumented, unmeasurable and short-lived.
That is not a satisfying conclusion for anyone hoping for a lever, and it is the one the evidence supports. The work that survives a model update is the work that would have been worth doing if none of these systems existed.
If you want the same argument applied to your own site rather than in general, the honest starting point is finding out what the assistants currently say about you, and where those descriptions came from.
Questions
What people ask once they stop looking for a trick
Does adding an llms.txt file to my site do anything?
No major assistant vendor has documented support for it, which means anyone telling you it works is reporting a belief rather than a behaviour. It costs almost nothing to publish, so it is not worth arguing about, but it should not appear on an invoice as a deliverable.
The general test is worth keeping: if a tactic has no documentation from the company whose system it is meant to influence, and no observable effect you could isolate, it is a guess. Guesses are allowed. Charging for them is not.
Does structured data make an assistant more likely to cite us?
There is no documented mechanism connecting markup to citation, and Google has stated plainly that no special markup is required for its AI features. What structured data reliably does is help a machine resolve what a page is about and which entity it belongs to, which is a real benefit for a different reason.
Use it because it makes your pages unambiguous and eligible for search features that are documented. Do not use it because somebody described it as a citation lever.
How do I know whether AI crawlers can actually reach my site?
Check three places, in this order: the robots.txt file, the bot-management rules in your CDN or firewall, and your server logs for the published user agents. The second is where the surprises are, because those rules are usually configured by an infrastructure team with a security brief and no marketing input.
Blocking is a legitimate choice for some publishers. What is not defensible is discovering two years later that a default rule made the decision for you.
Why does a smaller competitor get cited when we do not?
The most common reason is not authority but availability: they have stated the specific claim plainly in one place and you have distributed it across a case study, a services page and a PDF. Retrieval favours a clean, complete statement over a strong brand with a vague page.
The second most common reason is contradiction. If your own profiles, directory listings and old pages describe your business three different ways, a system has no stable version of you to use, and describing somebody else is the safer output.
Can we pay to appear in an assistant’s answer?
Not in the organic answer. Some products carry advertising, which is a separate, labelled placement and is bought like any other ad inventory. Nobody sells inclusion in the answer itself, and an offer to arrange one is describing something that does not exist.
This matters commercially because it means the only route is being genuinely usable as a source, which is slower and considerably harder to fake.
How would we ever know if any of this worked?
By sampling, honestly labelled. Write a fixed set of prompts a real buyer would use, run them on a schedule across the assistants that matter to you, and record the date, the exact wording and the full response. Over months that shows a direction.
It does not show a position, and two runs of the same prompt can differ. Any report that turns this into a single score without stating how many samples it rests on is presenting noise as a measurement.
Have someone check what is actually reaching your site
Access faults are the most common finding and the least interesting to talk about, which is why nobody checks. We will look at your robots file, your bot rules and your rendering, and tell you which agents can currently read you.
Related
Where to go next
- the work of becoming a sourceWhat an engagement to do this actually involves.
- how a generated answer gets assembled
- making every machine resolve you to one company
- the one surface with a webmaster console behind itCopilot grounds in Bing, which you can actually inspect.
- how we measure this, with the limits stated
- producing a fact of your own
- making the page fetchable in the first place
Last updated