Technical SEO: structure, crawling and indexing

Last time

You learned what Google calls spam, and how to tell a penalty from an algorithm update. Read day 02 again.

In this lesson

• Plan a site structure with shallow depth, clear folders and strong internal links.
• Read HTTP status codes the way Googlebot reads them, and find redirect chains and loops.
• Choose the correct control: robots.txt, noindex, nofollow, a sitemap or a canonical URL.
• Explain crawl budget, index bloat and why filters, sort orders and parameters waste crawling.
• Check what a crawler really did, with Search Console reports and server logs.

Day 01 showed the pipeline: a search engine discovers a URL, crawls it, indexes it and then ranks it. Technical SEO is the work that makes this pipeline run well on your site. It does not make content better. It makes sure that search engines can find, fetch and understand the content that you have.

Technical SEO is a large part of the roadmap, so it has two days. This lesson covers structure, crawling and indexing. Day 03b covers JavaScript rendering, page speed and crawl tools.

Website architecture

Website architecture is how the pages of a site are grouped and linked. Users and crawlers both use it to move through the site. It has three parts: the hierarchy, the URLs and the internal links.

Hierarchy and depth

A site hierarchy goes from the home page to categories, then to single pages. Click depth is the number of clicks from the home page to a page. Crawlers find pages by following links, so a page that is ten clicks deep is found later and less often.

A flat structure keeps important pages a few clicks from the home page. The SEO industry often says “three clicks or fewer”. Google does not give a number, so treat this as a guide, not a rule. Google’s starter guide also says: do not rebuild a site only for structure, because search engines “will likely understand your pages as they are right now.”

Folders and URL structure

Google gives clear rules for URLs:

• Use readable words, not long ID numbers: /wiki/Aviation, not /index.php?topic=42&area=3a5e….
• Use hyphens between words, not underscores: /summer-clothing/.
• Use = and & for parameters. Brackets or commas between parameters confuse crawlers.
• Do not use fragments (the part after #) to change content. Google “generally doesn’t support URL fragments.”
• Remember that URLs are case-sensitive. Google treats /APPLE and /apple as two URLs.

Folders help on large sites. Google says that if you have “more than a few thousand URLs,” group similar pages in folders. Then Google can learn how often each folder changes. For example, /policies/ seldom changes, but /promotions/ changes often, so Google can crawl them at different rates.

Internal linking

Internal links are links between pages of the same site. They are the main way that crawlers find your pages. They also show which pages you think are important, and what each page is about.

Google can only crawl a link that is an <a> element with an href attribute. A <span> with a click handler, or <a onclick="goto(…)"> with no href, is not a reliable link. The visible text of the link is the anchor text. Google uses it to understand the target page, so write “ghost pepper recipes”, not “click here”. Day 04 teaches anchor text in more detail.

A page with no internal links to it is an orphan page. Crawlers can find it only from a sitemap or from another site. Fix orphans first, because the fix is cheap and the effect is direct.

HTTP status codes

Each time a crawler asks for a URL, the server sends a three-digit HTTP status code with the response. The first digit gives the class. Google’s crawlers act on each class in a different way:

Class	Example	What Google does
2xx success	200	Sends the content to indexing. Indexing is not guaranteed.
3xx redirect	301, 302	Follows the redirect, up to 10 hops, and processes the target.
4xx client error	404, 410	Does not index the URL. Removes it if it was indexed. Crawls it less often over time.
429 and 5xx	429, 500, 503	Slows down crawling. Keeps indexed URLs for a time, then drops them if the error stays.

Three details matter most in practice:

• 301 and 308 are permanent. Google uses them as a strong signal that the target is the canonical URL. 302 and 307 are temporary, so they are only a weak signal.
• Google treats 404 and 410 the same way. Use either one for a page that is gone.
• A soft 404 is a page that says “not found” but returns 200. Google detects it and reports it. It wastes crawling, so return a real 404.

Use 503 for planned maintenance. It tells Google to come back later. Do not use 401 or 403 to slow Google down; Google says that they “have no effect on crawl rate.”

Redirect chains and loops

A redirect chain is a set of redirects in a row: A → B → C → D. Each hop costs time for users and crawlers. Google’s crawl budget guide says: “Avoid long redirect chains, which have a negative effect on crawling.” Googlebot stops after 10 hops. Search Console then reports a “redirect error.”

A redirect loop is a chain that comes back to an earlier URL: A → B → A. The page never loads, for users or crawlers. Chains often grow after site changes, for example http:// → https:// → www → a new path. Fix a chain: make each old URL redirect in one hop to the final URL. Also update internal links so that they point to the final URL.

HTTPS

HTTPS is HTTP with encryption (TLS). Browsers show it as a secure connection. In 2014, Google made HTTPS a ranking signal, but a “very lightweight” one: it affected “fewer than 1% of global queries.” Today, Google lists HTTPS as one part of page experience. HTTPS is a baseline, not a way to rank higher. Redirect all http:// URLs to https:// with a 301 in one hop.

Crawl and index controls

This is the most important section of the lesson. Each control does one job. Most mistakes come from using a control for the wrong job.

Control	Where it goes	What it does
robots.txt Disallow	A file at the root of the host	Asks crawlers not to crawl URLs. It does not stop indexing.
noindex	A <meta name="robots"> tag or an X-Robots-Tag header	Tells search engines not to index the page.
rel="nofollow"	On one link	A hint not to follow or credit that link.
Sitemap	An XML file that lists URLs	Helps search engines discover URLs.
rel="canonical"	A <link> element or a header	A strong hint about which duplicate URL to index.

Robots.txt

robots.txt is a plain text file at the root of a host, for example https://example.com/robots.txt. It lists rules for each crawler, by User-agent. The rules are Disallow and Allow, with paths. The IETF published the standard as RFC 9309 in September 2022.

Google’s own words are clear: robots.txt is “not a mechanism for keeping a web page out of Google.” A disallowed URL can still appear in results if other pages link to it. It then shows without a description. robots.txt is also a request, not a lock. Googlebot obeys it, but other crawlers might not. Use a password to protect private pages.

Some details from Google’s specification:

• When rules conflict, the most specific rule (the longest path) wins. If two rules are equal, Google uses the less restrictive one.
• Google reads only the first 500 KiB of the file.
• If robots.txt returns a 4xx error, Google acts as if the file does not exist. Then it can crawl everything.
• If robots.txt returns a 5xx error, Google stops crawling the site for the first 12 hours.

Day 07 applies robots.txt to AI crawlers.

Noindex and nofollow

noindex keeps a page out of Google. You can put it in the HTML head as <meta name="robots" content="noindex">. For PDFs and other files, send the HTTP header X-Robots-Tag: noindex. Google does not support noindex in robots.txt.

The common trap: Google must crawl the page to see noindex. If robots.txt blocks the page, Google never reads the rule. Then the URL can stay in results. Try this in the diagram.

Settings for one page

robots.txt: Disallow the page Page has noindex Other sites link to the page

Turn the three settings on and off. Notice that Disallow with noindex is a trap: Google cannot crawl the page, so it never reads the noindex rule.

nofollow is a value of the rel attribute on a link. Since March 1, 2020, Google treats nofollow as a hint for crawling and indexing, not as a strict rule. For links inside your own site, Google says to use robots.txt instead. Day 06 covers nofollow, sponsored and ugc for links to other sites.

Sitemaps

A sitemap is a file that lists the URLs that you want search engines to find. One sitemap holds up to 50,000 URLs or 50 MB. A large site uses a sitemap index that lists many sitemaps. You can submit it in Search Console, or add a Sitemap: line to robots.txt.

A sitemap helps discovery. It “doesn’t guarantee” crawling or indexing. Google says that a small site (about 500 pages or fewer) with good internal links might not need one. Google ignores <priority> and <changefreq>. It uses <lastmod> only if the dates are accurate. List only canonical URLs that return 200.

Canonicalization

Many sites show the same content at more than one URL. Examples are http and https versions, tracking parameters, and sort orders. Canonicalization is how Google picks one URL from a set of duplicates. The URL it picks is the canonical URL. Google indexes the canonical and crawls the duplicates less often. Duplicate content on your own site is normal. It is not spam.

You can tell Google your preference. Google lists the methods “in order of how strongly they can influence canonicalization”:

1. Redirects: a strong signal.
2. rel="canonical": a strong signal. Put <link rel="canonical" href="https://example.com/page"> in the head of each duplicate, and on the canonical page itself.
3. Sitemap inclusion: a weak signal.

All of these are hints. Google “may choose a different page as canonical than you do.” Your signals must agree. Do not put one URL in the sitemap and a different one in rel="canonical". Also link to the canonical URL from your own pages. Do not use robots.txt or noindex to pick a canonical. Day 05 covers duplicate content and consolidation.

Crawl budget and index coverage

Crawl budget is “the set of URLs that Google can and wants to crawl” on one host. It has two parts. The crawl capacity limit is how much Google can crawl without overloading your server. Crawl demand is how much Google wants to crawl, based on popularity and how often pages change.

Most sites do not need to think about crawl budget. Google’s guide is for large sites (about 1 million pages that change weekly) or medium sites (about 10,000 pages that change daily). For other sites, an up-to-date sitemap and a check of the Page indexing report are enough.

Index coverage is how many of your URLs Google indexes, and why it does not index the others. Search Console shows this in the Page indexing report. Not every URL must be indexed. Duplicates, noindex pages and removed pages should stay out. Look for important pages in these states:

Status	What it means
Discovered – currently not indexed	Google knows the URL but did not crawl it yet, often to avoid overloading the site.
Crawled – currently not indexed	Google crawled the page but chose not to index it, for now.
Duplicate, Google chose different canonical than user	Google picked a different canonical than the one that you set.
Soft 404	The page looks empty or like an error, but returns 200.
Redirect error	A chain that is too long, a loop or a bad URL.

Index bloat is the opposite problem: many thin or duplicate pages in the index. This is an SEO industry term; Google does not use it. The fix is the same as for crawl waste. Remove, consolidate or block the low-value URLs.

URL parameters, faceted navigation and pagination

These three features cause most crawl waste on large sites.

URL parameters are the key=value pairs after the ? in a URL, for example ?sort=price or ?utm_source=news. Parameters that do not change the content still make new URLs. Use as few as you can.

Faceted navigation is the filter list on shop and search pages: colour, size, brand and others. Each mix of filters can make a new URL. The number of URLs grows very fast. Google calls the result “infinite URL spaces.” Google’s advice: if you do not need filter pages in Search, block them with robots.txt. Or use fragments (#color=green) for filters, because Google ignores fragments. rel="canonical" and nofollow also help, but Google says that they are “generally less effective in the long term.” Return a 404 for a filter mix that has no results.

Filters (for example colour, size, brand):

3

Values in each filter:

5

Three sort orders (?sort=price and others)

How the site handles filter URLs

Nothing robots.txt Disallow rel="canonical" to the main list

Example data. Add filters and values, and turn the sort orders on and off. Notice how fast the orange bar grows, and that only robots.txt makes it match the green bar.

Pagination splits a long list into pages: page 1, page 2, page 3. Follow these rules from Google. Give each page its own URL, for example ?page=2. Link each page to the next with a normal <a href>. Give each page a self-referencing canonical; do not point all pages to page 1. Google no longer uses rel="next" and rel="prev", but other search engines might.

“Load more” buttons and infinite scroll need care. Google’s crawlers “don’t ‘click’ buttons.” Make sure that each item also has a crawlable link or a sitemap entry.

Log file analysis

Search Console shows what Google reports. Your server logs show what really happened. A web server writes one line for each request, with the time, the URL, the status code and the user agent. Log file analysis is the work of reading these lines for crawler requests.

Logs show which folders Googlebot crawls most, which parameter URLs waste its time, and which important pages it never visits.

Anyone can fake the Googlebot user agent. To verify a request, Google says to run a reverse DNS lookup on the IP address with the host command. The name must end in googlebot.com, google.com or googleusercontent.com. Then do a forward lookup on that name, and check that it gives the same IP. For a quick view without logs, use the Crawl Stats report in Search Console. It shows requests, response codes and response times. Google notes that it does not count every request, so its numbers can differ from your logs.

What matters most

1. Use the right control for the job. robots.txt controls crawling. noindex controls indexing. Canonical tags and redirects pick one URL from duplicates.
2. Make important pages easy to reach. Crawlable links, shallow depth and no orphans do more than any tag.
3. Return honest status codes. 200 for real pages, one-hop 301s for moved pages, 404 for missing pages, 503 for maintenance.
4. Control URL growth on large sites. Filters, sort orders and parameters can create more URLs than you have pages.
5. Check, do not guess. Use the Page indexing report, URL Inspection and server logs to see what crawlers really did.

Key terms

• Technical SEO: the work that helps search engines crawl, render and index a site.
• Click depth: the number of clicks from the home page to a page.
• Orphan page: a page with no internal links to it.
• Anchor text: the visible text of a link.
• HTTP status code: a three-digit code that tells the result of a request.
• Soft 404: a page that says “not found” but returns status 200.
• Redirect chain: two or more redirects in a row before the final URL.
• Redirect loop: a redirect chain that comes back to an earlier URL.
• robots.txt: a file at the root of a host that asks crawlers not to crawl some URLs.
• noindex: a rule in a meta tag or HTTP header that keeps a page out of the index.
• Sitemap: a file that lists URLs that you want search engines to find.
• Canonical URL: the one URL that a search engine picks to index from a set of duplicates.
• Crawl budget: the set of URLs on a host that Google can and wants to crawl.
• Index bloat: an industry term for many low-value pages in the index.
• Faceted navigation: filters on a list page that make a new URL for each mix of filters.
• Log file analysis: reading server logs to see what crawlers really requested.

Exercise

Exercise · about 15 min

Audit the crawl controls of MDN

Use your browser on MDN Web Docs, Mozilla’s documentation site. You can also use your own site.

1. Open https://developer.mozilla.org/robots.txt. Write down each Disallow rule and the Sitemap line.
2. Open the sitemap URL from step 1. Is it a sitemap or a sitemap index? Look at the <lastmod> values.
3. Open https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status. Open the page source (Ctrl+U, or Cmd+Option+U on a Mac). Search for canonical and robots. What do you find?
4. Open the browser developer tools (F12). On the Network tab, turn on Preserve log. Then go to http://developer.mozilla.org/docs/Web/HTTP. Write down each status code and URL in the chain.
5. Open https://developer.mozilla.org/en-US/docs/this-page-does-not-exist-xyz. In the Network tab, check the status code. Is it a real 404 or a soft 404?
6. For each finding, write one line: is it good practice, and why?

Show a possible solution

These were the results on 7 October 2026. Your results can be different.

1. robots.txt has User-agent: * with Disallow: /api/, Disallow: /*/files/ and Disallow: /media. It also has Sitemap: https://developer.mozilla.org/sitemap.xml. Good: it blocks API and file URLs that are not useful in Search, and it tells crawlers where the sitemap is.
2. The file is a sitemap index. It lists one compressed sitemap for each language, such as /sitemaps/en-us/sitemap.xml.gz, each with a <lastmod> date. Good: a large site splits its sitemaps to stay under 50,000 URLs each.
3. The page has <link rel="canonical" href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status">, which points to itself. It has no robots noindex tag. Good: a self-referencing canonical is what Google recommends.
4. The chain has two hops: 301 from http:// to https://developer.mozilla.org/docs/Web/HTTP, then 302 to /en-US/docs/Web/HTTP, then 200. The 301 for HTTPS is correct. The second hop is a 302 because it picks a language for each user. This is a short chain, not a problem. But a link that goes straight to the /en-US/ URL saves both hops.
5. The server returns 404. Good: a real 404, not a soft 404.

Check yourself

A site owner wants to remove a private test page from Google results. The page has other sites linking to it. Which plan works?

Add a Disallow rule for the page to robots.txtAdd noindex to the page and make sure that robots.txt allows the crawlAdd a Disallow rule and a noindex tag at the same timeRemove the page from the sitemap

An old URL goes through four redirects before it reaches the final page. Each hop is a 301. What is the best fix?

Nothing, because Googlebot follows up to 10 hopsChange the redirects to 302Redirect the old URL to the final page in one hop, and update internal links to the final URLBlock the old URL in robots.txt

A shop has 300 products. Search Console shows 80,000 URLs as 'Discovered – currently not indexed', most of them with ?color= and ?sort= in the URL. What is the most likely cause?

A manual action for spamFaceted navigation and sort parameters that make many crawlable URLsThe sitemap is too largeThe site uses HTTPS

Two URLs show the same article. The site puts URL A in the sitemap and sets rel=canonical to URL B. What happens?

Google must index URL B, because rel=canonical is a ruleGoogle indexes both URLsThe signals conflict, so Google picks the canonical from other signals and might not pick BGoogle gives the site a manual action

Your logs show many requests with the user agent 'Googlebot' to /admin/. The reverse DNS lookup of the IP gives a name that ends in 'example-hosting.net'. What does this tell you?

Googlebot ignores your robots.txtThe requests probably do not come from Google, because the name is not a Google domainGoogle uses many hosting companies for its crawlersYou must add /admin/ to your sitemap

Sources

• URL structure best practices for Google Search (Google Search Central)
• SEO Starter Guide (Google Search Central)
• Link best practices for Google (Google Search Central)
• How HTTP status codes affect Google’s crawlers (Google Crawling Infrastructure)
• Redirects and Google Search (Google Search Central)
• HTTP response status codes (MDN)
• HTTPS as a ranking signal (Google Search Central Blog, 2014)
• Understanding page experience in Google Search results (Google Search Central)
• Introduction to robots.txt (Google Search Central)
• How Google interprets the robots.txt specification (Google Crawling Infrastructure)
• RFC 9309: Robots Exclusion Protocol (IETF, 2022)
• Block Search indexing with noindex (Google Search Central)
• Qualify your outbound links to Google (Google Search Central)
• Evolving “nofollow” (Google Search Central Blog, 2019)
• Learn about sitemaps and Build and submit a sitemap (Google Search Central)
• What is canonicalization and How to specify a canonical URL (Google Search Central)
• Optimize your crawl budget (Google Crawling Infrastructure)
• Page indexing report and Crawl Stats report (Search Console Help)
• Managing crawling of faceted navigation URLs (Google Crawling Infrastructure)
• Pagination, incremental page loading, and their impact on Google Search (Google Search Central)
• Verifying Googlebot and other Google crawlers (Google Crawling Infrastructure)
• roadmap.sh SEO roadmap: Index Coverage & Bloat, Site Hierarchy & Depth (roadmap.sh)

Next

Day 03b finishes technical SEO: JavaScript rendering, SSR and CSR, Core Web Vitals, mobile pages and crawl tools such as Screaming Frog.