How to check
website indexing

Indexing, crawling and rankings are different outcomes

A page opens in a browser but cannot be found in search. That is not yet a diagnosis: the search engine may not have discovered the URL, fetched its content, or selected it as the canonical version; it may simply not show it for a particular query. Each situation needs its own evidence. Start with the page’s state, rather than rewriting its text or repeatedly requesting a recrawl.

For example, a service page may be indexed yet receive no impressions for a competitive query. That calls for analysing demand, content and promotion. Another service page may have been accidentally blocked from indexing: while the block remains, changing the heading cannot remove the cause.

The aim is a list of important URLs, verified statuses and clear next steps. There is no need to get every technical URL the site can generate into search.

List the pages that should appear in search

Export published pages from the CMS. Identify services, categories, products, articles and regional pages separately. Decide whether each type needs its own search result. A cart, account area, draft or technical thank-you page usually serves a different purpose.

Reconcile the register with internal links and the XML sitemap. These sources help reveal differences but do not replace one another: the sitemap may be incomplete, while a link-based crawl may miss a page with no incoming links. Record the export date and preferred domain version.

• For each entry, keep the complete URL, page type and expected status.
• Separate new publications, older important pages and intentionally excluded documents.
• For a moved URL, record the final redirect destination.
• Keep a history of recent changes: site migration, template updates, CMS settings and catalog filters.

Check access to the correct property in Google Search Console and Yandex Webmaster. Choosing the wrong domain or prefix can explain an empty report. Without verified access, you can perform external checks, but exclusion reasons and the full information available to the owner remain unverified.

What the site: operator shows—and what it cannot prove

Open a search engine and run the query site:example.com, replacing the domain with your own. Then narrow the check to a section or a specific URL: site:https://example.com/services/. Do not put a space between the colon and the URL. Review the URLs and titles that appear in the results.

This can quickly reveal a test subdomain, old titles, unwanted parameters or an unsuitable URL version. However, a missing URL in these results is not sufficient evidence that the page is absent from the index. Google explicitly warns that site: results are incomplete, and their order without an additional query does not represent the site’s normal rankings.

Do not calculate an indexing percentage from an approximate result count. Do not expect Google and Yandex to show the same page count: they update independently. Comparing specific important URLs and the reasons for their absence is more useful.

Inspect a URL in Search Console

Select the property and paste the complete URL into the inspection bar at the top. The initial result describes the version known to Google. Expand the indexing details and save the last crawl date, fetch availability and selected canonical URL. The tool is explained in the official URL Inspection help.

1. Check that you have the correct protocol, subdomain and path.
2. Record the overall status and the specific reason if the URL is not indexed.
3. Compare the site-declared canonical page with the one selected by Google.
4. Check the crawl date: the crawler may not yet have seen today’s fix.
5. To check the current version, select Test Live URL.

The live test and stored data answer different questions. The first checks current availability; the second describes the last processing result. A successful live test does not guarantee indexing or predict which canonical version will be selected. These distinctions are explained in the single-page troubleshooting guide.

If the page redirects visitors, inspect the original and destination URLs separately. Do not transfer one URL’s status to the other. Keep both entries and their relationship in the log; otherwise a corrected redirect can be mistaken for a missing page.

Analyse the Page indexing report

Open Indexing → Pages. Review the data date, trends and reasons why URLs are not indexed. If needed, filter by a selected sitemap. The overall count can reveal a change, but a repair plan must be based on reason groups and specific URLs.

In the Page indexing report help, Google explains why “not indexed” is not always an error. Intentional duplicates and redirected URLs may need no correction. What matters is whether distinct services, products or useful articles have ended up in those groups.

Do not treat an exported examples table as a guaranteed complete list of indexed pages. Reports have coverage and export limits. Record which URLs were inspected directly, which were grouped by a shared template and which still lack data.

Use Searchable pages and Check page

In Yandex Webmaster, select the site and open Indexing → Searchable pages. Review changes, URLs included in search and excluded pages. Filter the reasons, export the required data and compare it with your register. Status names and recommendations are in the help for this report.

For a specific URL, open Indexing → Check page. Enter the URL, choose the device type, run the check and open the details. Save the status, server response and information about the content the crawler sees. The features are described in the tool instructions.

Also review crawl statistics: they help establish when the crawler requested the page and what response it received. A visit alone does not mean the page is included in search. Yandex distinguishes these checks in its indexing management guide.

From status to the next action

“The page is missing” can describe very different situations. Do not apply one remedy to the entire list: distinguish expected exclusions from lost pages that are needed in search. The table below is a diagnostic guide; record the exact status wording from your own account.

A processed-but-not-indexed status does not prove a specific cause such as exhausted crawl budget. Conversely, duplicates do not mean you should delete an entire section. Decide after checking examples and the intended role of each group.

Check server responses, restrictions and canonical

Open the affected URL while signed out, then check it with httpstatus.io: enter the address, select Check status, then expand the redirect chain and headers. A live page must return a response appropriate to its content. A 200 status on a “nothing found” screen does not make it a substantive document.

Check robots.txt rules for the specific path. Then inspect robots meta tags and the X-Robots-Tag header. Blocking crawling and blocking indexing work differently: Google must be able to access a page to read its noindex directive. This is explained in the noindex documentation. A step-by-step explanation of the rules is available in our robots.txt guide.

Compare canonical declarations with the actual content. If every product page points to one category as its canonical version, investigate the template. Do not automatically replace every canonical: intentional duplicates may correctly point to the main document. Internal links, redirects and the sitemap should support a consistent scheme, not contradict it. The methods are described in Google’s canonical URL recommendations.

After a fix, inspect several URLs using the same template, including exceptions. Editing one page does not prove that the error has been resolved across the entire catalog.

Make sure the crawler receives the intended content

Compare the browser page with the content shown by inspection tools. A service page must contain its heading, description, terms and links. An empty container, loading message or sign-in screen needs technical investigation, even if everything looks correct to a signed-in editor.

If important text appears through JavaScript, check resource access and the rendered result. Google provides a separate JavaScript and search guide. The check must confirm that useful content is present, not merely that the console has no red error messages.

Then assess the page’s purpose. How does it help visitors, and how does it differ from neighbouring pages? An equipment page may need specifications, applications, compatibility, documentation and ordering terms. A service page may need a precise scope, limitations, process and outcome criteria. Adding paragraphs just to meet a character count is not enough.

Do not multiply identical text by changing a city or product name. Equally, do not merge different services just because they share a layout. Compare user needs, the offer and actual content first. If a whole section is affected, start with its shared page model, not a cosmetic edit to one heading.

Check important URLs, not just the total

Create a table with one agreed URL per row. Add separate fields for Google and Yandex, the data date, reason and a link to the evidence. Do not mechanically merge different protocols, paths and parameters: first determine which addresses really are versions of the same page.

For a large catalog, use exports and automation, but first examine the service’s method. An HTTP 200 response, sitemap inclusion and a site: query result are different signals. If a tool does not explain the source and date of its status, treat its table as a list for further investigation, not a definitive search-index register.

Address the shared cause first

Prioritise tasks by impact. An accidental site-wide noindex after a release, an unavailable server or an incorrect canonical in a template deserves attention before one recently published article. Still, support every hypothesis with observations: a drop coinciding with a release helps start an investigation but does not itself prove causation.

1. Save the original statuses and several reproducible examples.
2. Describe the expected behaviour for the affected page type.
3. Find the source: template, plugin, server configuration or publishing rules.
4. Verify the fix on control URLs and exceptions.
5. After deployment, inspect live responses and wait for search data to update.

A task needs a completion criterion. For example: “After the update, five service pages contain noindex. Remove the accidental template rule while preserving account-area restrictions. Check live responses for all service pages and reinspect selected URLs in the search consoles.”

If the problems are systemic, include this register in your SEO plan, and assign CMS and server changes to technical maintenance. A generic “improve SEO” list does not help estimate the work or accept the outcome.

Recrawling is the next step, not the fix itself

First confirm that the published page serves the corrected version. A cache may keep the old response after a CMS save. Run the available live check, compare the content and only then submit important updated URLs for reprocessing.

In Search Console, request indexing for the selected URL. For larger sets, keep the sitemap up to date. Google warns that repeated requests do not accelerate the process and that quotas apply; details are in its recrawl instructions.

In Yandex Webmaster, use Indexing → Reindex pages: add the required URLs and track the outcome. A successful submission means the request was accepted, not that the document is guaranteed to appear in search. Features and limits are explained in the reindexing help.

Separate two outcomes in the report: “the technical obstacle is removed” and “the search status has updated”. The first is evidenced by the site’s current response, the second by new search-engine data. Do not promise one deadline for every site or start another round of random changes just because the report still shows the previous crawl date.

Track changes to important pages

Keep a separate list of pages whose loss affects enquiries and sales: core services, priority categories and useful articles. Yandex Webmaster offers important-page monitoring and notifications. The procedure is described in the guide to monitoring search inclusion.

Set the checking frequency according to how often the site changes. Migrations, template updates and bulk publication require a specific follow-up. For a small, stable site, checking every URL manually each day is pointless when there is no sign of change.

• Compare page groups, reasons and dates, not just the overall count.
• Keep a release log alongside your observations.
• Track new pages separately from unexpected losses of older ones.
• Recheck fixes at the original URLs.
• Do not close a “no data” status without checking.

At project launch, include the indexing register in the website acceptance process with the developer. Related metadata, duplicate-content and navigation issues can be reviewed using the SEO error checklist. Monitoring then becomes part of website development, rather than a one-off reaction to falling traffic.

More questions about checking indexing