You Can’t Bullshit an LLM
The headline is rhetorical. You absolutely can mislead a large language model. LLMs can hallucinate, be manipulated and give confident answers that are wrong. They do not automatically know the person asking, the product being assessed or the full market. They cannot guarantee an honest—or even correct—recommendation.
They can also be overly agreeable. Anthropic’s research on sycophancy found that models may favour answers aligned with a user’s stated beliefs over more truthful ones. That is a warning against treating an AI assistant as an infallible buyer’s agent.
But it does not make the strategic shift disappear.
When an AI assistant helps someone compare products, a brand cannot assume its advertising controls the whole evaluation. The assistant may work from the buyer’s question, the context they supply, information available online and—where enabled—memory or custom instructions. The result can be a more contextual comparison than a generic search query, even though it remains incomplete and fallible.
That changes what brand has to do.
The buyer’s context is becoming part of the brief
OpenAI says its shopping results consider the user’s query and context, including memory or custom instructions where those features are in use. It also cautions that not every available product will be shown and recommends that people verify whether a product meets their needs. Its newer shopping research experience asks clarifying questions, researches product information and develops a personalised guide. OpenAI explicitly acknowledges that details such as price and availability can still be wrong.
In other words, this is not a neutral machine that knows everything. It is a tool attempting to answer a specific person’s brief using the context and evidence it can access.
That distinction matters. A claim such as “New Zealand’s most trusted choice” may sound impressive in an advertisement. It is far less useful when the buyer asks: “Which option suits a small team, integrates with our current systems, has local support and can be implemented this quarter?”
The assistant needs specifics. What does the product do? Who is it for? What are the constraints? What evidence supports the claim? What do independent sources say? Where are the trade-offs?
This is not proof of a universal set of AI ranking factors. It is a strategic interpretation of how contextual, research-oriented interfaces are developing. The practical implication is straightforward: broad promises have less room to hide when the evaluation includes detailed product information, independent evidence and the buyer’s actual requirements.
Brand is not becoming less important
It is becoming harder to reduce brand to messaging.
Brand still creates differentiation. It still shapes recognition, expectation and preference. But an AI-mediated comparison can bring the gap between a brand’s promise and its delivered experience into the same conversation.
Reputation matters because buyers and their assistants may encounter reviews, expert commentary and public discussion. Product experience matters because it creates that reputation. Clear information matters because an assistant cannot reliably compare a capability that is vaguely described, inconsistently named or missing altogether.
SEO formatting may help information become discoverable and understandable. It is not a substitute for a distinctive proposition, a credible reputation or a product that delivers. Being machine-readable is useful; being worth recommending is the point.
A hypothetical comparison
Imagine a marketing director choosing software for a distributed team. She asks an AI assistant for three options that support New Zealand users, meet a defined budget, integrate with her existing stack and provide responsive onboarding.
Brand A has the biggest campaign. Its website promises a “seamless, world-class platform”, but the integration pages are vague, support coverage is unclear and pricing requires a sales call.
Brand B publishes detailed capabilities, limitations, pricing ranges and implementation requirements. Its terminology is consistent across its website and documentation. Independent reviews discuss both the strengths and the learning curve. The company responds publicly and specifically when customers raise issues.
The assistant might still get the comparison wrong or miss a better option. But Brand B has supplied more useful material for a contextual evaluation. Its advantage is not that it has discovered a magic LLM optimisation trick. It has made a differentiated promise clear, supported it with credible information and allowed the delivered experience to be examined.
That is brand work.
Five practical actions for senior marketers
1. Replace unsupported adjectives with decision-grade detail
Audit the claims that carry the most weight in a buying decision. Replace “leading”, “seamless” and “innovative” with specific capabilities, constraints, service levels and use cases. If a claim cannot be supported, narrow it or remove it.
2. Make product information consistent and accessible
Align names, specifications, pricing logic and availability across product pages, documentation, feeds and third-party listings. Structured data can help machines interpret information, but accuracy and consistency come first. Assign an owner and review high-value information regularly.
3. Build evidence outside your own advertising
Identify the sources a careful buyer would consult: independent reviews, recognised experts, credible publications, customer communities and transparent case studies approved for public use. Earn evidence through performance and service; do not manufacture consensus or disguise promotion as independence.
4. Treat customer experience as part of your information environment
Marketing cannot separate the promise from what happens after purchase. Repeated confusion, weak support or unaddressed complaints become part of the public record. Work with product, service and operations teams to close the gap between the campaign and the experience.
5. Test real buyer briefs—not just branded prompts
Ask AI assistants the detailed questions your customers might ask, varying the supplied context and constraints. Record omissions, inaccuracies and weak evidence. Use the exercise as research, not as a definitive ranking report: outputs vary, personalisation depends on context and memory settings, and human judgement remains essential.
The uncomfortable advantage
Brands used to be able to keep the promise and the proof in different places. AI-assisted evaluation increasingly brings them together.
That will not always produce the right answer. It will not eliminate persuasion, bias or manipulation. It will not make advertising irrelevant. But it can reduce a brand’s ability to rely on assertion alone—especially when the buyer asks a detailed question and the assistant looks beyond the campaign for evidence.
The opportunity is not to “optimise for LLMs” and call the job done. It is to build a brand that is clear, differentiated and credible wherever the evaluation happens.
You can bullshit an LLM. You just cannot build a durable brand by bullshitting the market.