Robots.txt:
configure and test

What to decide before writing the first robots.txt line

Robots.txt is a publicly accessible text file containing crawling rules. It helps control which URLs a search crawler requests. Start by listing your pages: which should attract visitors from search, which serve internal functions and which must never be accessible to unauthorised people.

These objectives require different solutions. A product catalogue should usually remain crawlable. A public thank-you page can be excluded from the index with noindex. An account area needs server-side access checks. Trying to solve all three with Disallow creates a false sense that the site is configured correctly.

Google warns that blocking crawling does not guarantee that the URL itself will disappear from search results. See the introduction to robots.txt. Below, we focus on configuring crawling: one part of technical SEO, rather than a standalone way to gain rankings.

Where to find robots.txt and how to open it

For the website https://example.com/ open https://example.com/robots.txt. Replace example.com with your domain. You should see plain-text rules. If an error page, login form or your site’s visual layout appears, first fix how the file is served.

Use the filename robots.txt in lowercase. Do not upload it to an images folder or save it as robots.txt.txt. Subdomains, protocols and non-standard ports matter: rules at one address do not automatically apply to every version of the site. The general creation and placement process is covered in Google’s instructions.

1. Open the homepage and copy its final address after redirects.
2. Append /robots.txt to the site root and open it in a private window.
3. Save the current version as a separate text file with a date.
4. Record what generates the response: server, CMS, SEO plugin or caching system.
5. Check subdomains separately if they host independent sites.

List the URLs before changing the rules

Checking only the homepage will not reveal whether a block affects filters, images or part of the catalogue. Build a small test set that represents your site’s actual structure. Define the expected result for each URL in advance, then compare it with the observed result after the change.

The examples in the table are illustrative. Do not create these paths just for testing or copy blocks before defining what your existing pages do. Add URLs with parameters, similar folder names and different capitalisation: unexpected matches often appear at the boundaries of a pattern.

How to read User-agent, Disallow, Allow and Sitemap

User-agent identifies the crawler or group of crawlers the rules apply to. Disallow blocks crawling of a matching path; Allow allows it. Sitemap gives the full XML sitemap URL. Comments start with a hash. Write each directive on a separate line.

User-agent: *
Disallow: /draft-catalog/
Allow: /draft-catalog/public-guide/

Sitemap: https://example.com/sitemap.xml

The example blocks a drafts directory but allows its nested public reference section. Robots.txt does not protect the drafts from people: non-public material requires access controls.

Google selects the most specific matching User-agent group; the general asterisk group does not automatically supplement a dedicated Googlebot group. When paths conflict, the rule’s specificity matters, not its position in the file. The symbol * matches any sequence of characters; $ marks the end of a URL. Paths are case-sensitive. Exact conditions and exceptions are set out in Google’s specification.

Test similar paths, parameters and exceptions

Suppose you have an internal section /search/ and a useful article /search-marketing/. A block using the short prefix /search may affect both. Test not only the URL you want to block, but also the similar page that should remain accessible.

Do not apply a blanket rule to every URL containing a question mark before understanding the parameters. They may control language, catalogue pagination, products or useful filters. Compare URL pairs with and without each parameter for content, accessibility and canonical URL.

Getting started with robots.txt in WordPress

WordPress can serve a virtual robots.txt. The source code of do_robots() blocks crawling of the admin area and allows admin-ajax.php. Plugins and filters can modify the final response, so check the public URL after saving your settings.

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml

This is a starting point for review, not a universal WordPress configuration. Replace the sitemap URL with one that actually exists: some sites use sitemap_index.xml, others wp-sitemap.xml. Open the sitemap and check its child files. WordPress installations in a subdirectory also require a path check.

Do not block the entire wp-content folder: it may contain images and assets required for rendering. Assess archives, tags and author pages by their content and purpose. Coordinate individual page settings with your SEO plugin instead of trying to express the entire indexing strategy through path prefixes.

When to use Clean-param and what must not be treated as insignificant

Clean-param helps Yandex handle parameters that do not change content. Suppose an internal campaign_id parameter only tracks an ad click while the product page stays the same. After verifying that condition, you can prepare a rule for the relevant part of the site:

Clean-param: campaign_id /products/

Do not declare product, language or catalogue-page parameters insignificant simply because they generate many URLs. First compare the page with and without the parameter: heading, main text, product list, price and available language version.

According to Yandex’s documentation, some analytics tags may be removed automatically if the algorithms consider them insignificant. There is no need to add rules for those tags merely to have an extra line. Clean-param does not replace a site-structure review and is not a Google directive.

Record the decision in a table: parameter name, example URL, whether it changes content, chosen treatment and owner. When a new analytics system is added, the team can see what has already been checked and what needs a separate decision.

Why Disallow and noindex are not interchangeable

To exclude a publicly accessible HTML page from Google Search, you can place this meta tag in its head:

<meta name="robots" content="noindex">

The crawler must fetch the page to read this instruction. Blocking it in robots.txt at the same time may prevent that. For non-HTML files such as PDFs, use an X-Robots-Tag HTTP header. The methods and limitations are explained in Google’s noindex guide.

For example, you decide to keep a submission-confirmation page out of search. Specify its URL, the presence of noindex in the response and the absence of any crawl block that would hide it. Then check the result with the search engine’s tool. An admin-settings screenshot is not enough: a template or cache may serve something different.

If the page contains personal data from the enquiry, search settings alone are not enough. It must enforce access permissions. Robots.txt is public: listing a path in it does not make that path secret.

How to test robots.txt in Yandex Webmaster

Open Yandex Webmaster and go to Tools → Robots.txt analysis. Select your site or enter the address to analyse. The exact procedure for added and unverified sites is described in the tool’s help page.

1. Load the published file and make sure its text matches the latest change.
2. Review errors and crawling rules. A syntax error differs from a valid rule that produces an unwanted result.
3. Add your test URLs to the URL list under Check if URLs are allowed.
4. Run the check and compare every result with your expectation.
5. If you edited the text inside the tool, copy the tested version to your site: editing the analyser’s field does not update the site itself.

For the developer, record the URL, desired result, actual result and the rule responsible. Attach the test date and the published file’s text. This turns “the crawler cannot see the catalogue” into a reproducible issue.

Checking the file and an individual page in Search Console

In Google Search Console you handle two different tasks: checking whether robots.txt was fetched and diagnosing an individual URL. Start with the robots.txt report: it shows the fetched version, fetch date and processing problems. You can request a file recrawl for a critical fix.

1. Choose the property matching the domain and protocol you are checking.
2. Check the robots.txt status and open the latest fetched version.
3. Compare it with the file currently on the site. An older date is a reason to distinguish the search engine’s cached copy from the change already published.
4. Enter an important URL in the URL Inspection tool.
5. Run Test Live URL to check the currently published page.
6. Review whether crawling is allowed, whether the page was fetched and whether indexing is allowed.

A successful live test does not prove that the page is indexed or will rank. Stored indexing data and a live URL test refer to different points in time. This distinction is explained in the URL Inspection tool’s help.

Track two separate milestones: “the site serves the corrected response” and “the search engine has processed the change”. This prevents repeated edits to a correct file just because the report updates later.

What to do when the result differs from your expectation

A robots.txt 404 and a 5xx response are not the same problem: Google handles them differently. For server failures, use the official specification and the fetch report instead of assuming “no file means the site is definitely blocked”.

If a block appeared alongside a traffic drop, preserve the change timeline and investigate other causes too. Do not promise recovery by a fixed date: crawl access, revisits, indexing and rankings are separate stages. For a broader diagnosis, also use our guide to SEO mistakes.

How to hand over a change and accept the result

Agree on a small change with a clear objective. Do not change crawling rules, section URLs and indexing settings at once without a shared plan: unexpected results would be hard to trace. Save the old version and decide in advance how to restore it.

Check the file immediately after release, then review the reports after search crawlers next fetch it. Repeat the checks when the CMS, SEO plugin, domain, catalogue structure or cache system changes. The overall acceptance process is covered in “How to accept a website from a developer”.

Robots.txt configuration checklist

• Search pages, internal sections and data requiring authentication are identified.
• The file’s source is known and the original version is saved.
• The public URL returns the agreed rules.
• Dedicated crawler groups and similar paths have been tested.
• Important pages and rendering resources are not blocked by mistake.
• The XML sitemap URL exists and belongs to the current site.
• Parameters have been assessed for their effect on content.
• A crawl block does not hide noindex from the crawler.
• Expected and actual results are recorded for the test set.
• A change owner and a method for restoring the previous version are defined.

This checklist assesses crawling configuration. It does not replace an audit of content, structure, duplicates or user journeys. If the task is broader than one file, start with a website SEO plan and an agreed list of technical tasks.

Frequently asked questions about robots.txt