Keep Google discoverable.
Check your current file, back it up and test every rule after publishing.
Robots.txt tells compliant crawlers which paths they may request. Googlebot, OAI-SearchBot and GPTBot have different roles, so evaluate them separately. Robots.txt does not secure private content or guarantee AI citations.
What robots.txt can and cannot control
A robots.txt file provides crawler access instructions for bots that choose to respect the Robots Exclusion Protocol. It is usually published at the root of a host, such as https://example.co.ke/robots.txt. Rules are interpreted per host and protocol context; a rule on one subdomain does not automatically govern another.
robots.txt does not securely protect private content. A blocked URL can still be discovered through links, and malicious crawlers may ignore the file. For genuinely private material, use authentication and appropriate access controls.
It also does not function as a universal “AI opt-out” switch. Different providers publish different crawler identities and policies, and some AI-related access can involve ordinary user-initiated retrieval rather than bulk training crawlers.
Crawler access is not the same as indexing
Blocking crawling can prevent a bot from reading page content, but it does not necessarily erase a known URL from search results. For removal, use the correct indexing or HTTP response controls.
The difference between Googlebot, OAI-SearchBot and other AI crawlers
Googlebot is associated with Google Search crawling. Google also documents crawler controls relevant to some AI uses, including Google-Extended. Google-Extended is a control token for certain Google AI uses; it is not a separate ordinary crawler that must be allowed for standard Google Search indexing.
OpenAI publishes distinct identities such as OAI-SearchBot for search discovery and GPTBot for model-training-related crawling, with policies that can change. Other providers may have their own published user-agent tokens. Before adding a rule, confirm the current official documentation rather than copying an outdated list from social media.
For a business that wants search discovery but has reservations about training access, these distinctions matter. Do not block every bot containing “AI” without understanding the consequences for the channels you actually want to support.
| Crawler / token | Broadly associated purpose | What to verify |
|---|---|---|
| Googlebot | Google Search crawling | Normal indexing access |
| Google-Extended | Specified Google AI use controls | Official Google documentation |
| OAI-SearchBot | OpenAI search discovery | Current OpenAI crawler documentation |
| GPTBot | OpenAI model-training crawling | Current OpenAI crawler documentation |
A cautious example for a Kenyan business website
If your aim is to allow ordinary search crawling, the safest starting point is often to avoid unnecessary blanket blocks. You can include specific rules for crawler tokens where you have a deliberate policy, but make sure the file is valid and publicly accessible.
The example below illustrates explicit allowance for search discovery and leaves the broader site available to normal crawlers. It is not a universal recommendation to allow all AI training uses, nor a promise that an assistant will recommend the business.
WordPress installations may already generate robots.txt virtually. If Rank Math, another SEO plugin and a physical robots.txt file all attempt to control the endpoint, inspect the actual public response before editing anything.
User-agent: OAI-SearchBot
Allow: /
User-agent: Googlebot
Allow: /
User-agent: *
Allow: /
Sitemap: https://example.co.ke/sitemap_index.xml
How to inspect the robots.txt file WordPress is actually serving
Open https://yourdomain.co.ke/robots.txt in a private browser window and inspect the raw text. A WordPress plugin dashboard may show one version while a physical file, server cache or CDN serves another. Always trust the live response over a settings screen.
If your site uses Rank Math or Squirrly, identify which system owns robots.txt and XML sitemaps. Two SEO plugins with overlapping controls can create confusing results. The same applies to host-level security settings and Cloudflare rules.
If you need help with a stable setup, website maintenance and security should cover configuration ownership, backups and testing before edits are deployed.
- Visit the live robots.txt URL and read every user-agent group.
- Check whether the file is physical or generated by WordPress.
- Identify the SEO plugin or server setting that controls it.
- Verify sitemap URLs are correct and accessible.
- Check CDN cache behavior after any edit.
Robots.txt syntax rules that cause real problems
Each group begins with a User-agent directive and can include Allow and Disallow rules. A blank Disallow value does not mean “block everything”; Disallow: / is the broad rule that blocks crawling of the entire site for the matched group.
Be careful with copied wildcard patterns and assumptions about query strings. Some crawler implementations support pattern matching, but a complicated set of rules can unintentionally block legitimate pages. Always test representative URLs before publishing a large rewrite.
Also remember that robots.txt is public. Do not list sensitive directories in the hope of hiding them; use authentication and server permissions. For pages that should be removed from search, use a suitable noindex directive or HTTP status where appropriate, while allowing Google to crawl the page long enough to observe that signal.
How robots.txt interacts with noindex, canonical and XML sitemaps
Robots rules determine whether a compliant crawler can fetch a URL. A noindex directive tells a supporting search engine not to index a page after it can access and process the directive. Canonical tags indicate a preferred URL among duplicates but are signals rather than commands.
An XML sitemap lists important URLs for discovery. Including a URL in a sitemap does not override a robots block, and blocking a URL does not automatically remove it from search. These systems should be configured coherently rather than as independent checkboxes.
For example, a spam URL that must disappear from Google may need a 404 or 410 response, or an accessible noindex directive where appropriate. Blocking the URL in robots.txt first can prevent Google from seeing the removal signal. Your technical SEO process should diagnose the specific case.
| Control | Primary job | Common misunderstanding |
|---|---|---|
| robots.txt | Crawler access | Guaranteed deindexing |
| noindex | Indexing directive for supported crawlers | Works when crawler cannot access page |
| canonical | Preferred duplicate URL signal | Guaranteed URL replacement |
| XML sitemap | URL discovery aid | Guaranteed indexing |
How to make a considered AI crawler policy
Start with the business goal. Do you want your public service pages discoverable in search and assistant search products? Do you have copyrighted resources, sensitive datasets or contractual obligations that require a different policy? Different content types may justify different access decisions.
Read the official documentation for each relevant provider. Keep a written record of which user-agent groups you allow or block and why. Revisit that policy periodically because provider identities and use cases evolve.
If you serve customers in Nairobi, Mombasa, Nakuru or elsewhere in Kenya, keep your public service locations content accurate and accessible where intended. A crawler policy cannot compensate for weak local information.
How to test changes safely without damaging rankings
Create a copy of the existing robots.txt content before editing. Make one small change at a time, deploy it, and immediately inspect the live endpoint. Check important URLs including the homepage, primary services, blog articles, images and sitemaps.
Use Search Console URL Inspection and relevant crawl tests to verify that Google can access important pages. Remember that Search Console reports may lag and cannot prove that every third-party AI crawler behaves the same way.
If a rule causes unintended blocking, revert it promptly and clear relevant caches. A reliable WordPress website architecture includes change control and recovery procedures, not just a list of directives.
- Back up the current rules and record the change.
- Test homepage, service pages, articles and sitemap endpoints.
- Check public HTTP responses after cache purge.
- Monitor Search Console for crawl access changes.
- Revert immediately if important content becomes blocked.
What to do if AI assistants still cannot find your business
Do not assume robots.txt is the only reason. Check whether the relevant service pages are publicly accessible, whether the site has consistent business identity, whether the pages contain clear answers and whether search engines have discovered them.
Inspect the Nairobi service page and other commercial pages for accurate scope, genuine examples and contact details. If your site has thin pages or duplicate descriptions, improving the content may be more useful than changing crawler rules.
Finally, avoid promises that a particular robots.txt line guarantees ChatGPT recommendations. Search discovery, model behavior and AI responses involve many factors outside the control of one website.
How to decide whether a crawler should access a particular section
Begin with the content’s purpose. Public service pages, useful guides and contact information are generally intended for discovery. Administrative screens, internal dashboards and confidential client material are not. But robots.txt is not a secure boundary for private information, so authentication and server access rules must protect genuinely sensitive resources.
For public pages, decide whether you want normal search discovery and whether you have a separate policy about particular AI uses. Review each provider’s documented crawler identities and keep the decision written down. A broad block copied from another website may be inappropriate for your own business goals.
Pay attention to WordPress-generated paths. Blocking /wp-admin/ is common in some configurations, but other site functions may use endpoints or assets that must remain accessible. Test actual pages and resources rather than assuming every path beginning with “wp-” is expendable.
When publishing location-focused content, keep the service location pages discoverable where intended and avoid adding crawler rules that accidentally isolate entire categories of legitimate content.
How to recover after an accidental robots.txt block
If you discover that a broad Disallow rule has blocked the whole site, first restore the last known good file or configuration. Check the public endpoint immediately after the change. If a caching layer serves an old version, purge that specific cache and test again from outside the WordPress dashboard.
Next, verify representative pages using Search Console URL Inspection and any suitable live crawl tests. Confirm that the homepage, primary services, blog articles and XML sitemap can be fetched as intended. Search Console reporting can lag, so do not interpret one unchanged status as proof that the fix failed.
Review how the bad rule reached production. Was it a plugin setting, a physical file, a deployment script or a host-level rule? Assign a single owner for the endpoint and keep a backup of the working configuration. This prevents the same issue recurring during the next plugin update.
Finally, check the wider SEO setup. If the site was inaccessible for some time, review important pages and the technical SEO configuration for unrelated noindex or canonical errors. Restoring crawler access is the first step; maintaining a consistent setup is the longer-term fix.
Frequently asked questions
Should I allow OAI-SearchBot in robots.txt?
If you want your public pages eligible for OpenAI search discovery, review OpenAI’s current crawler documentation and consider allowing OAI-SearchBot. Allowing it does not guarantee a citation or recommendation.
What is the difference between GPTBot and OAI-SearchBot?
OpenAI documents different purposes for these crawler identities. Review its current official crawler documentation before deciding your access policy.
Does blocking Google-Extended remove my website from Google Search?
Google describes Google-Extended as a control for specified AI uses rather than ordinary Google Search indexing. Verify the latest documentation before making changes.
Can robots.txt remove spam URLs from Google?
Not reliably. Robots.txt blocks crawling, not guaranteed indexing. Removal often requires the correct HTTP status or an accessible noindex directive.
Why is WordPress robots.txt different from Rank Math settings?
A physical file, another plugin, hosting configuration or caching layer may control the public endpoint. Inspect the live robots.txt response.
Does Disallow: / block the entire website?
For a matching crawler group, Disallow: / requests that compliant crawlers avoid crawling the whole site. It can seriously affect discovery if applied broadly.
Do I need an AI-specific robots.txt file?
No. Standard robots.txt is the usual place for supported user-agent rules. There is no universal AI-specific file that guarantees discovery or citation.
