How to fix noindex, robots.txt, sitemap and canonical problems
Four small files and tags decide whether search engines can crawl and list your site. Here's how to check each one and fix it on WordPress, hosted builders and your own server.
Short answer: for Google to list your site, the page must not carry a noindex tag or header, and robots.txt must not say Disallow: / to every crawler. Those two are the serious ones, and both are often left over from a staging copy.
After that, publish a sitemap at /sitemap.xml, list it in robots.txt with a Sitemap: line, and give each page a canonical tag that points at its own preferred address.
Crawling (fetching pages) is controlled by robots.txt; indexing (listing them) is controlled by noindex. A sitemap helps crawling, and the canonical tag says which address to list when a page has several. The checker reads your homepage, /robots.txt and /sitemap.xml.
The page tells search engines not to list it
Serious Shown in the checker's Search (SEO) tab as “Page is marked noindex”.
The checker flags a <meta name="robots"> tag whose content includes noindex, or an X-Robots-Tag HTTP header that includes it.
Why it matters
When Google sees noindex, it drops the page from search results entirely, even if other sites link to it (Google: block indexing with noindex). On a homepage, that usually means the business can't be found by name. It's almost always an accident: a setting switched on while the site was being built and never switched off.
How to fix it
WordPress
Go to Settings → Reading and untick Discourage search engines from indexing this site. Since WordPress 5.3 that box adds a noindex, nofollow robots tag to every page (WordPress: Reading settings). If the tag remains, check your SEO plugin's per-page “allow search engines to show this page” option and its site-wide settings.
Wix, Squarespace and Shopify
Look in the SEO settings for an option that hides the site or a page from search engines, and turn it off. A password-protected or “coming soon” site can't be indexed either, so remove the site password when you launch.
Server header (nginx, Apache, staging hosts)
If the page source has no robots tag, the noindex is in a header. Search the server config for X-Robots-Tag (nginx add_header, Apache Header set) and remove it. Some hosts add it to staging domains automatically.
How to check it's fixed
Run curl -sI https://yourbusiness.com and confirm there's no X-Robots-Tag: noindex line, and search the page source for noindex. In Google Search Console, the URL Inspection tool shows “Indexing allowed?” for the live page, and you can request indexing from there.
robots.txt blocks every crawler
Serious Shown in the checker's Search (SEO) tab as “robots.txt blocks all crawlers”.
The checker reads the rules in your robots.txt that apply to all crawlers (the User-agent: * group) and flags it when they forbid the homepage. The classic cause is these two lines:
User-agent: *
Disallow: /Why it matters
Well-behaved crawlers, including Google's and Bing's, won't fetch any page the rules forbid (RFC 9309). Google can still list a blocked address it finds through links, but without being able to read it, the result usually has no description. And because Google can't fetch the page, it can't see any noindex on it either (Google: introduction to robots.txt).
How to fix it
Change Disallow: / to an empty Disallow:, which allows everything, or delete the line. One character changes the meaning, so check carefully. Keep Disallow rules only for areas that really shouldn't be crawled, such as an admin folder.
- WordPress: if there's a physical
robots.txtfile in the site's root folder (check with your host's file manager or FTP), edit or delete it. Otherwise, WordPress generates one itself, and an SEO plugin may offer a robots.txt editor. - Shopify generates
robots.txtand lets you customise it through arobots.txt.liquidtheme template. Wix has a robots.txt editor in its SEO settings. Squarespace generates the file for you. - cPanel or static hosting: edit
robots.txtin the site's document root (oftenpublic_html).
How to check it's fixed
Open https://yourbusiness.com/robots.txt in a browser. The robots.txt report in Search Console shows the version Google last fetched and any errors.
There's no robots.txt file
Minor Shown in the checker's Search (SEO) tab as “No robots.txt”.
The checker flags this when /robots.txt doesn't return a normal 200 response.
Why it matters and how to fix it
A missing file (a 404) isn't harmful. Google treats it as “no restrictions”, but you lose the easiest place to point crawlers at your sitemap. A file that returns a server error (5xx) is different: Google stops crawling the site for a time while it retries (Google: how it reads robots.txt). Create a plain-text file named robots.txt at the root of the domain:
User-agent: *
Disallow:
Sitemap: https://www.yourbusiness.com/sitemap.xmlThe Sitemap line must be a full address, including https://. If you want to set rules for AI crawlers too, see AI search visibility.
There's no sitemap at /sitemap.xml
Minor Shown in the checker's Search (SEO) tab as “No sitemap.xml”.
The checker requests /sitemap.xml, following up to two redirects on your own site. It flags the result unless the response is 200 and contains a <urlset> or <sitemapindex> element.
Why it matters
A sitemap is a list of the pages you want found. Google says most sites benefit from one, especially new sites with few inbound links. It also says a sitemap doesn't guarantee every listed page gets indexed (Google: sitemaps overview).
How to fix it
- WordPress (5.5 and later) publishes one at
/wp-sitemap.xml, and SEO plugins usually replace it with their own (WordPress core announcement). The checker only looks at/sitemap.xml, so list the real address inrobots.txt(next section) and, if you like, redirect/sitemap.xmlto it. - Squarespace (Squarespace help), Shopify and Wix generate
/sitemap.xmlautomatically once the site is public. - Hand-built sites: write an XML file that lists each page's full address. A single sitemap can hold up to 50,000 URLs or 50 MB (sitemaps.org protocol).
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url><loc>https://www.yourbusiness.com/</loc><lastmod>2026-10-21</lastmod></url>
<url><loc>https://www.yourbusiness.com/services/</loc></url>
</urlset>How to check it's fixed
Open /sitemap.xml in a browser, then submit it in Search Console's Sitemaps report, which shows whether Google could read it.
robots.txt doesn't list the sitemap
Minor Shown in the checker's Search (SEO) tab as “robots.txt doesn't list the sitemap”.
The checker found a working sitemap and a robots.txt, but no line starting with Sitemap:.
How to fix it
Add one line anywhere in robots.txt, using the full address. You can list more than one sitemap. The address doesn't even have to be on the same host (Google: robots.txt). Every search engine that reads robots.txt then finds the sitemap without you submitting it to each one.
Sitemap: https://www.yourbusiness.com/sitemap.xmlWordPress's generated robots.txt normally adds this line itself. If it's missing, a physical robots.txt file is probably overriding it, so add the line to that file.
The page has no canonical tag
Minor Shown in the checker's Search (SEO) tab as “No canonical URL”.
The checker flags a page whose <head> has no <link rel="canonical">.
Why it matters
The same page is often reachable at several addresses: with and without www, over http and https, or with tracking parameters. The canonical tag says which one is the real one, so Google combines the signals for all of them in one place. Google calls it a strong signal, and redirects a strong one too, while listing a page in your sitemap is a weak one (Google: consolidate duplicate URLs).
How to fix it
Add a self-referencing canonical with the full address to each page. Google recommends absolute URLs over relative ones:
<link rel="canonical" href="https://www.yourbusiness.com/">WordPress adds a canonical tag to single posts and pages on its own, but not to a homepage that shows your latest posts. SEO plugins add one to every page. Hosted builders generally add canonicals automatically; if one is missing, look for a canonical field in the page's advanced SEO settings. Also redirect the other versions (http to https, and one of www or non-www) to the preferred one, which is the strongest fix of all.
How to check it's fixed
Search the page source for rel="canonical". In Search Console, URL Inspection shows both the canonical you declared and the one Google chose.
The canonical tag points at a different site
Fix soon Shown in the checker's Search (SEO) tab as “Canonical points at otherdomain.com”.
The checker compares the domain in your canonical tag with the domain the page was served from, ignoring www, and flags any mismatch.
Why it matters
A canonical pointing at another domain asks Google to credit that domain with your page. Typical causes are a domain move with the old address still in the settings, a template copied from another site, or a staging address hard-coded in the theme.
How to fix it
Point the canonical at the page's own address. In WordPress, check Settings → General: both WordPress Address and Site Address should be your live domain. Then check the SEO plugin's canonical field and the theme's header.php. Elsewhere, check the custom canonical field and the primary domain setting. If you deliberately syndicate a page that started on another site, the canonical is correct and you can ignore this finding.
How to check it's fixed
View the source and read the href of the canonical tag, then run the checker again.
Sources
- Google Search Central: block indexing with noindex
- Google Search Central: introduction to robots.txt
- Google Search Central: how Google interprets robots.txt
- RFC 9309: Robots Exclusion Protocol
- Google Search Central: sitemaps overview
- sitemaps.org: sitemap protocol
- Google Search Central: consolidate duplicate URLs (canonical)
Common questions
Why is my website not showing up on Google?
The two most common technical causes are a leftover noindex tag and a robots.txt that disallows everything; both stop Google listing the site. If neither applies, the site may simply be new: submit your sitemap in Google Search Console and use URL Inspection to request indexing of the homepage.
Does robots.txt keep a page out of Google?
No. It stops crawling, not indexing, and Google can still list a blocked address it finds through links. To keep a page out of results, allow it to be crawled and add noindex, or put it behind a password.
Do I need a sitemap for a small website?
Google says small, well-linked sites may not strictly need one, but most sites benefit and every common platform makes one automatically. It costs nothing to have and to list in robots.txt.
What is a self-referencing canonical tag?
A canonical tag on a page that points at that page's own preferred address. It tells search engines which version to list when the same page is reachable with or without www, over http, or with tracking parameters.
How long does it take Google to reindex after removing noindex?
It depends on how often Google crawls the site, typically days rather than hours. Requesting indexing in Search Console's URL Inspection tool and resubmitting the sitemap usually speeds it up.
Find out if Google can see you. The website checker reads your robots.txt, sitemap, noindex and canonical tags in one pass, along with your titles, structured data and security. If your site has dropped out of search and you want help, get in touch.