Crawling
Search engines discover and access URLs — via links, sitemaps, and other hints.
Loading content
Loading content
WEBSITE INDEXING
Find out why important pages are not being indexed and build a practical SEO strategy to improve crawlability, discovery, indexing, and search visibility.
DEFINITION
A search engine generally needs to discover and process a page before it can potentially appear in results. Indexed does not mean ranking well. A page can be in the index and still have poor visibility.
The URL can be discovered and fetched. If the door is locked, later stages never start.
The page is processed as a document — including important HTML that JavaScript may add.
The system decides whether to store the page in the search index. Indexed does not mean ranking well.
Indexed pages can compete for queries. Visibility still depends on relevance, competition, and many other signals.
If the symptom is “our site does not show at all,” start with whether important URLs are in the index — not with ranking tactics. Related problem framing is on website not indexed.
CAUSES
Every indexing issue does not have the same cause. Match the URL to evidence before changing the whole site.
HTML or HTTP noindex asks systems not to keep the URL in the index. Check staging leftovers on production.
Disallow can stop fetching. That is not the same as noindex, and it can hide the very directive you meant to send.
A canonical to another URL, staging, or the homepage can consolidate away the page you wanted indexed.
Several URLs with the same job can lead to one being chosen and others skipped or clustered.
If nothing important links to a URL, discovery is slower and the page looks less connected.
URLs with no internal path often depend on sitemaps or external links alone.
Important pages buried many clicks deep are easier to miss on large sites.
Repeated 5xx responses can delay or drop crawl of those URLs.
The original URL is not the indexed one. Chains, loops, or redirects to weak destinations confuse the picture.
A 200 page that looks like “not found” or empty can be treated as low value.
A live product returning 404, or a gone page still 200, sends the wrong index signal.
Little unique purpose can reduce the chance of indexation. Thin does not automatically mean “never indexed.”
Location or tag pages that repeat the same boilerplate often compete with each other.
Important copy or links missing from what systems can process can leave a URL looking empty.
Discovery takes time. New URLs are not a failure on day one.
Missing, stale, or full of non-indexable URLs. A sitemap listing does not guarantee indexing.
Filters, parameters, and templates can create more URLs than should compete. Causes differ by site.
STAGES
Identifying the correct stage matters. Changing titles will not help a URL that cannot be fetched. Requesting indexing will not help a URL that is noindexed.
Search engines discover and access URLs — via links, sitemaps, and other hints.
Pages are processed and a decision is made about including them in the search index.
Indexed pages can compete for visibility based on many systems and signals. That is a later conversation.
AUDIT
Indexing problems should be diagnosed from evidence — not by submitting URLs repeatedly and hoping.
EVIDENCE
URL Inspection can confirm a single URL. It is not a substitute for fixing site-wide technical or content problems. Requesting indexing does not guarantee inclusion.
STATUS LABELS
Not every excluded URL is an SEO error. Some pages should stay out of the index on purpose.
Means: The URL was fetched, then not selected for the index at that time.
Check: Duplicates, thin templates, canonicals, quality, and whether the URL should compete at all.
Action: Action is not automatic. Some crawled-not-indexed URLs are correctly skipped.
Means: The URL is known but has not been crawled yet, or crawl has not completed.
Check: Internal links, sitemap, site size, and whether crawl is focused elsewhere.
Action: Improve discovery paths before repeatedly requesting indexing.
Means: A noindex directive was processed.
Check: Whether noindex is intentional (thank-you pages, filters) or accidental on production.
Action: Remove noindex only on URLs you want in the index.
Means: Fetching is discouraged. Indexing of a blocked URL is unreliable.
Check: Disallow rules on folders that hold important pages.
Action: Allow crawl if you need the URL indexed; use noindex on a crawlable URL if you need it out.
Means: Systems see overlap and chose a canonical you did not mark.
Check: Parameters, copies, and missing or conflicting rel=canonical.
Action: Align canonicals and internal links with the URL you want as primary.
Means: This URL points at another preferred URL. Often expected.
Check: Whether the canonical target is the page you actually want indexed.
Action: Usually no “fix” if the pair is intentional.
Means: This URL is not the destination. The target may be indexed instead.
Check: Redirect destination quality, chains, and loops.
Action: Needed if the redirect is wrong; not an error if the move is intended.
Means: The URL returned not found when crawled.
Check: Still linked internally, still in the sitemap, or still earning links.
Action: Restore, redirect to a relevant replacement, or remove dead links.
Means: The server failed during crawl.
Check: Hosting, timeouts, and error rates on that template.
Action: Fix reliability before asking for recrawl.
DISCOVERY
Important pages should have clear paths from other relevant pages. Do not create artificial internal links simply for SEO.
SITEMAPS
Submitting a URL in an XML sitemap does not guarantee indexing. It is a discovery hint.
Sitemaps help systems find URLs, especially new or deep ones. They do not guarantee indexing.
List the preferred live URLs you want considered — not every parameter variant.
Avoid noindex, 404, redirected, and blocked URLs in the sitemap as a rule.
Stale lastmod, broken sitemap indexes, and Search Console sitemap errors are worth fixing.
Split files and keep each sitemap valid. Organization helps maintenance, not ranking magic.
DIRECTIVES
These are not interchangeable. If you do not want a page in Google, blocking it in robots.txt is often the wrong tool — the noindex directive may never be fetched.
| Topic | robots.txt | noindex |
|---|---|---|
| What it mainly does | Suggests which paths not to fetch | Asks that a processed URL stay out of the index |
| Needs the URL to be crawled? | No — it tries to prevent the fetch | Yes — the directive must be seen |
| Typical use | Admin, infinite filters, faceted junk you do not want crawled | Thank-you pages, internal search, pages that must stay out of the index |
| Wrong use | Blocking a URL you want indexed, or blocking it hoping it “deindexes” | Putting noindex on a URL you still need in search |
PREFERRED URL
Canonical tags are signals. Search engines may choose a different canonical. Adding a tag does not guarantee indexing of that URL.
A live preferred URL usually points at itself so the signal is clear.
HTTP/HTTPS, www, trailing slashes, and parameters can look like copies.
Pointing at the wrong page, or mixing tags and headers, can send mixed signals.
That other URL is the one you are asking to represent the content.
Sort and session URLs often should not be treated as unique indexable pages.
VALUE
Thin content does not automatically mean a page will never be indexed. Quality is one part of a broader diagnosis — alongside access and duplicates.
Does this URL do a job no other URL does, for a search someone actually makes?
Tag pages, near-identical location pages, and thin categories often cluster.
Auto-created URLs with boilerplate can flood discovery without helping users.
Real inventory and useful category copy can still fail indexation for technical reasons — quality is one part of diagnosis.
When uniqueness and intent are the main gap, see content SEO.
NO PATH IN
An orphan page has little or no internal linking path from the rest of the website.
Example: a new service URL live at /services/migration-seo that is not in the services nav, not linked from related articles, and only listed in a sitemap. Discovery is slower. Example of a page that should often stay orphaned or be removed: an old campaign landing page with no remaining use — redirect or 410 instead of forcing links.
Identify orphans by comparing a crawl of linked URLs with sitemaps, analytics landing pages, and Search Console. Connect important URLs into architecture with links people would use.
RENDERING
JavaScript websites can be indexed. The question is whether search engines can access, render, understand, and index important content.
Content may exist only after JavaScript runs.
Important HTML can be in the first response.
Tabs, infinite scroll, and widgets can hide copy and links.
If links and titles never appear in what systems process, discovery and understanding suffer.
CATALOGUES
Shops can generate very large numbers of URLs and need a deliberate indexation strategy — not “index everything.”
Catalogue merchandising sits alongside this. See e-commerce SEO for that layer.
Decide which item and collection URLs should compete.
Combinations can explode. Many should stay out of the index on purpose.
Site-search URLs are often poor candidates for the index.
Need a consistent rule: keep, noindex, 404, or redirect — not a mix by accident.
Same SKU on multiple paths needs a preferred canonical and matching internal links.
Product sitemaps should match live canonicals. Categories should link to items you want discovered.
SCALE
Publishing, e-commerce, multi-location, and international sites with thousands of URLs share one idea: not every URL should compete. There is no universal crawl-budget number that fits every host.
AFTER A MOVE
Domain, CMS, redesign, URL, and platform migrations often show up first as coverage changes.
Planning redirects and URL mapping belongs in website migration SEO. After launch, indexing checks still include:
OVERLAP
Indexing problems often overlap with technical SEO. Diagnosis here still focuses on why URLs are or are not entering the search index — not on rewriting the whole technical stack by default.
For crawl, architecture, performance, and rendering as a system, see technical SEO.
MISTAKES
Allow crawl if you want them indexed. Do not use Disallow as a substitute for noindex.
Check templates and headers on production, not only staging.
Point at the live URL you want as primary, then match internal links.
Give important URLs a real path from related pages — not a hidden sitemap-only existence.
URL Inspection is a check, not a site-wide repair. Fix causes first. Requests do not guarantee inclusion.
Keep the list aligned with live, preferred, indexable URLs.
Fewer, clearer URLs usually beat more copies of the same job.
Treat production robots and meta as a separate config.
Fix hops, loops, and 5xx before expecting recrawl.
Move important URLs closer to navigation and related content.
Ensure crawlers can process the copy and links that define the page.
Indexation follows what the new URL actually contains.
Thank-you pages, filters, and alternates are often correctly out.
PROCESS
A practical sequence. I do not invent client counts, case studies, rankings, traffic, revenue, or awards.
Step 1
Which URLs actually matter — not the whole crawl dump.
Step 2
Samples from Search Console, sitemaps, and important landing pages.
Step 3
Groups, trends, and a few inspections — evidence first.
Step 4
Status codes, fetch, noindex, robots.
Step 5
How would a person — or a crawler — find this URL?
Step 6
Signals that choose a different URL or block this one.
Step 7
Is this URL doing a unique job?
Step 8
A blocked money page outranks a unused tag archive.
Step 9
Recrawl, headers, and live checks — not only a ticket closed.
Step 10
Right URLs in the index, then impressions and clicks — without promising either.
MODERN SEARCH
Pages generally need to be accessible and understood before they can contribute to search visibility. There is no indexing shortcut that guarantees AI Overviews, ChatGPT citations, or AI answers. An llms.txt file is not required for Google Search.
TIMING
There is no universal guaranteed timeframe. I do not promise a page will be indexed within a set number of hours or days.
If an important URL stays unindexed for an unusual period — compared with similar pages on the same site — inspect access, noindex, canonicals, internal links, and Search Console rather than waiting without a diagnosis.
FIT
MEASUREMENT
More indexed pages is not automatically better SEO. The goal is to get the right valuable pages indexed, not to maximize the URL count.
FAQ
Direct answers about crawl versus index, Search Console statuses, sitemaps, and requests — without guaranteed indexing.
Indexing is when a search system stores a processed version of a URL so it can potentially appear in results. It is not the same as ranking well.
Causes vary: noindex, robots.txt, canonicals, weak discovery, duplicates, errors, thin templates, rendering, or simply a new URL. Diagnosis starts with Search Console and the URLs that matter — not one generic fix.
Different templates, signals, and value. One product URL can be fine while a filter combination is correctly skipped.
Crawling is fetching the URL. Indexing is deciding to keep it in the search index. A URL can be crawled and still not indexed.
Use Google Search Console URL Inspection and indexing reports for your verified property. Site: queries are a rough hint, not a complete audit.
The URL was fetched, then not selected for the index at that time. Investigate duplicates, quality, and whether it should compete. It is not always an error.
The URL is known but has not been fully crawled or processed yet. Improve internal links and sitemap quality before treating it as a failed request.
No. It helps discovery. Inclusion still depends on access, signals, and how systems evaluate the URL.
No. Inspection and request indexing can prompt a recrawl of that URL. They do not replace site-wide fixes and do not guarantee inclusion.
It mainly prevents crawling. A blocked URL may still appear in limited ways if it is known from other sources. It is a poor way to “deindex” a URL you can still fetch.
robots.txt suggests not to fetch. noindex asks that a crawled, processed URL stay out of the index. To send noindex, the URL generally needs to be crawlable.
Overlap can lead to one URL being chosen and others skipped or clustered. Canonicals and internal links should point at the preferred version.
Yes. Useful links from related pages are a primary discovery path. Do not add artificial links that do not help people.
There is no guaranteed number of hours or days. Size, discovery, access, and processing all play a role. If an important URL stays out for an unusual period, inspect access and Search Console rather than waiting indefinitely.
No. Filters, thank-you pages, internal search, and true duplicates often should stay out. The goal is the right valuable pages, not the maximum count.
NEXT STEP
Find the technical, structural, or content-related reasons behind your indexing problem and turn them into a practical SEO action plan.