Robots.txt Audit for WordPress Websites in Kenya
A robots.txt audit checks which crawlers can access important pages and whether the rules accidentally block useful content. Robots.txt controls crawling, not guaranteed removal from Google. Inspect the live file, test rules and check indexing signals separately.
Read the live file before editing
Open the domain’s /robots.txt URL and record what the server actually returns. WordPress plugins, caching layers and physical files can produce different versions.
Do not assume the SEO plugin editor is authoritative. A physical robots.txt file or server rule may override a virtual file.
Check important URLs first
List the homepage, service pages, blog posts, product pages and sitemap URLs that should be crawlable. Test them against the rules for Googlebot and other relevant user agents.
A broad Disallow rule can block resources needed to understand a page. Check CSS, JavaScript and image paths before deciding that a rule is harmless.
Understand the difference between crawling and indexing
Disallow prevents compliant crawlers from fetching a path, but an already-known URL can still appear in search results without content. A noindex directive must be discoverable by the crawler to work.
For removed spam URLs, use genuine 404 or 410 responses rather than hiding them behind robots.txt. Otherwise the crawler may not see that the page is gone.
Check sitemap references and syntax
Use an absolute sitemap URL where appropriate and verify that the sitemap itself loads. Inspect rule groups for accidental scope changes and unsupported directives.
A sitemap is a discovery aid, not a command to index every URL. Include canonical, useful pages and avoid adding hacked or duplicate addresses.
Test before publishing
Use Search Console’s URL Inspection and the applicable robots testing guidance. Test representative URLs from each important site section.
Keep a copy of the previous file. A small typo in a broad rule can affect thousands of URLs, so changes deserve a careful review.
Monitor after the change
Check crawl activity, indexing reports and server responses over time. If a page remains unindexed, investigate canonical tags, content quality and technical responses rather than blaming robots.txt automatically.
Document why each restrictive rule exists. That makes future maintenance safer when another plugin or developer changes the configuration.
Why one robots.txt line can matter
A site owner adds a broad Disallow rule while trying to hide hacked search URLs. Unfortunately, the rule also prevents crawlers from fetching URLs that should return 404. The owner sees fewer crawls and assumes the spam has disappeared, but some results remain.
The better approach is to remove the hacked content, return appropriate status codes and keep useful pages crawlable. A robots audit should examine actual server responses, not just whether a text file contains the expected words.
A quick decision guide
| Area to review | Useful next step |
|---|---|
| Read the live file before editing | Open the domain’s /robots. |
| Check important URLs first | List the homepage, service pages, blog posts, product pages and sitemap URLs that should be crawlable. |
| Understand the difference between crawling and indexing | Disallow prevents compliant crawlers from fetching a path, but an already-known URL can still appear in search results without content. |
| Check sitemap references and syntax | Use an absolute sitemap URL where appropriate and verify that the sitemap itself loads. |
Details worth checking before you act
Robots.txt applies to paths, not an abstract list of WordPress page types. Verify how the live URL paths are constructed before adding wildcard patterns.
If an SEO plugin generates a virtual robots.txt file, check whether a physical file already exists. Edit the source that actually serves the live response, not a settings page that is being overridden.
A blocked URL may still be referenced by external links. If it should disappear from search, allow the crawler to observe the appropriate noindex or removal response rather than hiding the page blindly.
Frequently asked questions
Does robots.txt stop Google indexing a page?
Open the domain’s /robots.txt URL and record what the server actually returns. WordPress plugins, caching layers and physical files can produce different versions. Check the specific requirements and your actual business situation before taking action. Keep a record of what you changed and whether the result improved.
Can I block hacked URLs using Disallow?
A broad Disallow rule can block resources needed to understand a page. Check CSS, JavaScript and image paths before deciding that a rule is harmless. If you are unsure, verify the facts in the official dashboard or with the responsible provider rather than relying on an online shortcut. Avoid making changes that you cannot reverse.
Which WordPress folders should I block?
Disallow prevents compliant crawlers from fetching a path, but an already-known URL can still appear in search results without content. A noindex directive must be discoverable by the crawler to work. The safest decision is the one supported by real evidence from your website or account. Test carefully, and do not promise a result that depends on another platform.
Can two plugins control robots.txt?
A sitemap is a discovery aid, not a command to index every URL. Include canonical, useful pages and avoid adding hacked or duplicate addresses. Check the specific requirements and your actual business situation before taking action. Keep a record of what you changed and whether the result improved.
Should I put the sitemap in robots.txt?
Use Search Console’s URL Inspection and the applicable robots testing guidance. Test representative URLs from each important site section. If you are unsure, verify the facts in the official dashboard or with the responsible provider rather than relying on an online shortcut. Avoid making changes that you cannot reverse.
How do I test robots.txt rules?
Document why each restrictive rule exists. That makes future maintenance safer when another plugin or developer changes the configuration. The safest decision is the one supported by real evidence from your website or account. Test carefully, and do not promise a result that depends on another platform.
When should I change robots.txt?
Check crawl activity, indexing reports and server responses over time. If a page remains unindexed, investigate canonical tags, content quality and technical responses rather than blaming robots.txt automatically. Document why each restrictive rule exists. That makes future maintenance safer when another plugin or developer changes the configuration. Check the specific requirements and your actual business situation before taking action. Keep a record of what you changed and whether the result improved.
Where to go from here
If your business needs help with the issues covered in this guide, review our SEO and search visibility services. You can also contact the Afix Smart Web team to understand the next practical step.
