Large WordPress platforms do not disappear from AI-generated answers simply because their content is poor. They often lose visibility because, after years of development, no one has taken ownership of the information architecture, while crawler access may be restricted at a level that is not visible from the WordPress admin panel.
This article explains what you can control, what is documented, and what remains a hypothesis.
What is changing and what remains the same
Users are increasingly getting a single synthesized answer instead of a list of links. Google AI Overviews and AI Mode, ChatGPT, Perplexity, and Copilot can synthesize information from multiple sources and point users to sources supporting the generated answer.
What changes is the way users reach content. In Google’s case, however, this does not mean that an entirely separate set of optimization rules has emerged. Google confirms that AI Overviews and AI Mode use its core ranking and quality systems. From Google’s perspective, optimization for generative AI is therefore still SEO.
The SEO fundamentals remain. Proper indexing, logical site structure, reliable information, and technical quality matter for both traditional search results and generative features. Google uses mechanisms including retrieval-augmented generation (RAG) and query fan-out, where the system may perform multiple related searches to collect the information required to answer a user’s question.
There is another point worth stating clearly because part of the market tends to overlook it: meeting technical requirements and following best practices does not guarantee indexing or inclusion in AI features. What you can responsibly do is establish strong technical foundations, make your content accessible, and ensure that it provides genuine value to the audience. Google specifically emphasizes unique, valuable content that offers information and perspectives that are not widely available elsewhere.
Four layers you can actually control
The order matters. Working on content while crawlers cannot access it means spending money before removing the underlying obstacle.
| Layer | Question it answers | Owner within the organization |
|---|---|---|
| AI crawler access | Can AI systems access our content at all? | Infrastructure, DevOps |
| Structured data and entities | Is information about the organization, authors, and content described consistently? | Development team |
| Information architecture | Can users and search engines understand how our knowledge is structured? | Marketing and platform architect |
| Technical quality | Can search engines discover, render, and index all relevant content? | Development team, hosting |
1. AI crawler sccess: a common single point of failure
AI providers have separated crawlers used for model training from those responsible for retrieving content in response to a search or user request. This distinction affects how content can be used, while many robots.txt files on large websites were created before these differences became relevant.
| User-agent | Role | Potential consequence of blocking |
|---|---|---|
| GPTBot | OpenAI training crawler | Content unavailable for relevant model-training use |
| OAI-SearchBot | Crawler associated with ChatGPT Search | Reduced ability for content to appear in ChatGPT Search |
| ChatGPT-User | Fetches pages following user requests | ChatGPT may be unable to retrieve requested content |
| ClaudeBot | Anthropic crawler | Restricts Anthropic’s access to the content |
| Claude-User | Fetches pages following user actions | Restricts content retrieval in response to user actions |
| PerplexityBot / Perplexity-User | Indexing and on-demand retrieval | Restricts the use of content by Perplexity |
| Google-Extended | Control token, not a crawler | Controls certain uses of content by Google’s generative AI systems but does not control Google Search indexing |
| Applebot-Extended | Control token | Controls certain uses of content in Apple’s AI features |
| CCBot | Common Crawl | Restricts inclusion in Common Crawl datasets |
The decision is a business decision, not merely a technical one. An organization can deliberately restrict particular uses of its content or allow access. What matters is that the decision is intentional rather than the accidental consequence of a rule copied into the configuration three years ago.
Google works differently from some independent AI assistants. For a page to appear in generative AI features in Google Search, it first needs to be indexed and eligible to appear in standard Google Search results. Google explicitly states that the existing crawling and indexing requirements remain the foundation.
A correct robots.txt file is not enough. On large platforms, restrictions often exist at a higher level:
- WAF and bot-management rules at the CDN level, including AI bot settings,
- rate limits that reject crawlers when request volumes increase,
- settings controlling eligibility for generative AI features in Google Search.
Search Console. Google provides performance reporting for generative AI that can be used to understand how content is discovered through generative AI features in Search and Discover. This is currently Google’s official data source for evaluating visibility in these features.
Access by other AI crawlers can be checked relatively cheaply: filter server and CDN logs by user-agent and use reverse DNS where the provider offers an appropriate verification method, since any bot can spoof a user-agent header.
There is also the question of llms.txt. Google’s position is now clear: Google Search does not use llms.txt to improve a site’s visibility or rankings, including in generative AI features. The file can still be maintained for other services that actually use it, but it is not part of Google Search optimization.
2. Structured data after the 2026 changes
Structured data remains part of SEO, but it is not a dedicated optimization mechanism for generative AI. Google explicitly states that structured data is not required to appear in generative AI features and that websites do not need any additional or special schema markup for AI Search.
Its practical role remains important where it helps Google interpret page content correctly and makes content eligible for supported rich results. Structured data should correspond to content that is actually visible to users.
One change is particularly important because it makes many older recommendations obsolete: on May 7, 2026, Google retired FAQ rich results from Search. Support in Rich Results Test and the Search Console report disappeared in June 2026, with Search Console API data following in August 2026. HowTo has not generated rich results since 2023.
FAQPage remains a valid Schema.org type. The practical conclusion is that FAQ sections are worth keeping when they are genuinely useful to users and reflect the page’s actual content. FAQ schema should not, however, be treated as a technique for increasing generative AI visibility. Google explicitly warns against over-focusing on structured data in this context.
| Type | What it is useful for today | Note |
|---|---|---|
| Organization | Describing the organization | Implement according to documentation and actual company information |
| Person | Describing the author | Data should correspond to a real person and visible site content |
| Article | Topic, author, publication and modification dates | dateModified should represent an actual update |
| BreadcrumbList | Providing context about the page’s position in the site structure | Can still qualify for relevant Google features |
| Service | Semantic description of a service | Not a dedicated AI Search markup |
| Product, Offer, AggregateRating | E-commerce | Data must match what the user sees |
| FAQPage | Semantic description of questions and answers | No need to implement it specifically for generative AI |
| HowTo | Semantic description of instructions | No rich-result effect since 2023 |
Use validator.schema.org to validate Schema.org markup and Rich Results Test only for types that Google still supports.
A common issue in large WordPress installations is duplicate or contradictory markup: two active SEO plugins, or a theme generating its own markup alongside a plugin.
3. Information architecture: why 300 articles can lose to 30
For a system that needs to find and understand information relevant to a user’s question, publication volume matters less than quality, usefulness, and organization.
Google describes query fan-out as a mechanism through which the system may perform multiple related searches to collect the information required to answer a complex question. This does not mean that you should create a separate page for every variation of a query. Google explicitly warns against producing content at scale to manipulate rankings or generative AI answers and points out that its systems can understand relevance without exact keyword matching.
If a website contains hundreds of posts on unrelated topics, turning them into a useful and intuitive knowledge base becomes difficult. When content is organized into clusters, users can move logically between related subjects.
For an enterprise platform, a cluster around the platform lifecycle might include architecture, technical audits, technical debt, performance and Core Web Vitals, security, migrations, maintenance, and infrastructure scaling. Each resource answers a different question and links to adjacent topics.
Three things damage this structure particularly quickly in large organizations:
Publishing instead of developing existing content. Publishing dozens of new articles every year while never updating existing resources creates multiple pages competing to answer the same question. Developing one comprehensive resource is cheaper and more useful than publishing a fifth article about essentially the same topic.
Inconsistent terminology. Calling the same service “maintenance” on one page, “technical support” on another, and “website care” somewhere else makes the offer harder for users to understand and reduces consistency across the website. Choose one primary term and use synonyms intentionally.
No owner for information architecture. Landing pages are created for campaigns, documentation belongs to the product team, and the blog belongs to marketing. Each element may be correct in isolation while the overall website fails to form a coherent structure. This is a governance problem, not a content problem, and no plugin will solve it.
Unique knowledge matters more than nnother article on the same topic
Expanding a topic cluster alone is not enough. Google now places particular emphasis on content that provides information that is not widely available elsewhere. This can include expert experience, proprietary analyses, original data, observations from projects, or detailed solutions to specific problems.
For a technology company, this changes the way content should be approached. Another article compiling publicly available information about WordPress optimization adds little. An analysis explaining how a specific architectural decision affected the performance of a large platform, what problems emerged during scaling, or which solution proved effective in a real project provides substantially more value.
Keyword research should therefore not be the only source of content ideas. Knowledge already available inside the organization matters as well: developer experience, audit findings, solved technical problems, architectural decisions, and their consequences.
4. Technical quality: what is documented and what is not
There is no public evidence that Core Web Vitals are a criterion for being cited in AI-generated answers. This is worth stating explicitly because the opposite claim is often presented as fact.
Google does clearly state that a page must meet Search’s technical requirements, be available for crawling and indexing, and be eligible to appear in standard Google Search results. Standard JavaScript SEO best practices continue to apply to sites using JavaScript. Google also recommends providing a good page experience, working properly across devices, reducing latency, and eliminating unnecessary content duplication.
Performance therefore stays on the list, but for a defensible reason.
| Metric | Good threshold | Typical cause on a large WordPress platform |
|---|---|---|
| LCP (Largest Contentful Paint) | < 2.5 s | Unoptimized hero images, no CDN, long server response time |
| INP (Interaction to Next Paint), replacing FID since 2024 | < 200 ms | Excessive scripts, plugins blocking the main thread |
| CLS (Cumulative Layout Shift) | < 0.1 | Images without dimensions, dynamically loaded elements |
Two operational notes are important.
First, field data from the Core Web Vitals report in Search Console and CrUX is more important than a single laboratory PageSpeed Insights score because it reflects the experience of actual users.
Second, if your audit procedure still includes Google’s Mobile-Friendly Test or the mobile usability report in Search Console, that procedure is outdated. Google retired both on December 1, 2023. PageSpeed Insights, Lighthouse, and the Core Web Vitals report are among the tools that should now be used instead.
How to measure it
Without an agreed metric, this work may be evaluated six months later using the wrong measure and incorrectly classified as ineffective.
Search Console. Google provides generative AI performance reporting that can be used to understand how users discover content through generative AI features in Search and Discover. Official Search Console data should form the basis for evaluating visibility in Google’s generative features.
Server and CDN logs. These answer the most fundamental question for external AI systems: are their crawlers visiting the website at all, and are they receiving 403 responses?
GA4. Separate referral traffic from AI assistant domains in reporting rather than evaluating it only within aggregated traffic channels. Volumes may still be relatively small, but the traffic can represent users with a specific research intent.
Citation monitoring tools. Treat them as sampling tools rather than a source of Google’s internal data. Google emphasizes that external tools do not have access to its internal ranking or AI systems.
The reporting consequence is important: visibility within a generative answer does not always lead directly to a website visit. Agree with management on what constitutes a meaningful result before the work begins, not after the first quarter.
AI-readiness checklist for a large WordPress platform
| # | Area | Control question | Where to check |
|---|---|---|---|
| 1 | Crawling and indexing | Is all important content accessible to crawlers and eligible for indexing? | robots.txt, Search Console, logs |
| 2 | CDN layer | Is the WAF or bot management blocking crawlers that should have access? | CDN panel, logs |
| 3 | Search Console | Are settings related to eligibility for generative AI features aligned with the organization’s decision? | Search Console |
| 4 | Structured data | Does the structured data reflect actual page content without contradictions? | validator.schema.org |
| 5 | Unique knowledge | Do key resources contain proprietary experience, data, analysis, or expert knowledge? | Content audit |
| 6 | Freshness | Are the most important articles regularly reviewed and genuinely updated? | CMS, content audit |
| 7 | Clusters | Do key business areas have logically connected sets of expert content? | Site map |
| 8 | Terminology | Are services and technologies named consistently across the website? | Terminology audit |
| 9 | Rendering | Can Google correctly render and process key content, including on JavaScript-heavy pages? | URL Inspection, rendering tests |
| 10 | Performance | Does the website provide a good experience across devices? | CWV report, CrUX, Lighthouse |
Two or three negative answers may be a one-sprint job. Six or more suggest that AI Search itself is not the main issue. It is exposing weaknesses in the underlying architecture. At that point, isolated fixes may cost more than systematically reorganizing the platform.
How we approach this at WLC
We start with diagnosis, not implementation.
The order is: crawler and indexing accessibility, technical quality, structural consistency and information architecture, followed by a content development plan.
There is a practical reason for this sequence. In many audits, the most expensive problem sits deeper in the stack than the client initially expects, while the cheapest improvement is often found in a layer that no one had been monitoring.
In practice, we rarely recommend rebuilding an entire website. More often, the existing platform needs to be reorganized and supported by governance that keeps it coherent as new campaigns, content, and sections are added.
If you are developing a WordPress platform and want to understand where you actually stand before investing further in content, talk to us about a technical diagnosis.
FAQ – AI Search and WordPress
Will AI Search replace SEO?
No. Google’s official position is clear: its generative AI features use the core ranking and quality systems of Google Search. Google treats activities described as GEO or AEO as part of SEO rather than a separate discipline requiring an entirely new set of optimization techniques.
Is structured data required to appear in Google’s generative AI features?
No. Google explicitly states that structured data is not required for generative AI search and that no special schema markup needs to be implemented. Structured data still has a role in the broader SEO strategy, particularly in relation to supported rich results.
Does FAQ schema still make sense after the May 2026 changes?
It can still make sense as a valid way of describing actual page content, but it should not be implemented specifically for AI Search. FAQ sections should primarily answer genuine user questions. Google emphasizes that structured data is not required for generative AI features.
Do we need to rebuild the website?
In most cases, no. Improving internal linking, terminology, indexing, and key content can produce meaningful improvements without changing the application architecture. A rebuild becomes relevant when the underlying architecture itself or accumulated technical debt is the problem.
Should AI crawlers be blocked?
This is a business decision that should be made separately for individual providers and types of content use. For Google, the foundation for visibility in AI Overviews and AI Mode remains the ability to crawl and index content in Google Search.
How quickly can we expect results?
There is no reliable universal timeframe. Removing a technical issue may enable a page to be crawled and indexed again, but meeting all requirements does not guarantee indexing or inclusion in generative AI features. Google explicitly states this in its documentation.
Does AI Search matter in B2B?
Yes. B2B buying processes begin with research, and generative AI features create another way for decision-makers to discover information about solutions, vendors, and technologies. From a content strategy perspective, this makes proprietary expert knowledge, original analysis, and first-hand experience increasingly valuable compared with reproducing information already available elsewhere.
