Technical SEO: rendering, speed and crawl tools

In this lesson

Explain how Google renders JavaScript, and why other crawlers can see less than Google.
Compare client-side rendering, server-side rendering, static rendering and hydration for SEO.
Make content machine-readable, so that crawlers and AI systems can extract it.
Read Core Web Vitals in PageSpeed Insights, and tell field data from lab data.
Check a site for mobile-first indexing, and pick a crawl tool for an audit.

Day 03 was about which URLs a search engine crawls and indexes. This lesson is about what the search engine finds when it gets to a URL. Can it see the content? Is the page fast and stable? Does the page work on a phone? Then you meet two tools that check all of this for a full site.

Rendering and JavaScript SEO

Rendering is the work of turning code into the page that a person sees. A browser reads the HTML, loads the CSS and runs the JavaScript. JavaScript can add text, links and images to the page after the HTML arrives.

This matters for SEO because the HTML response and the finished page can be different. A crawler that reads only the HTML response sees only what the server sent.

How Google processes JavaScript

Google’s docs give three phases: crawling, rendering and indexing.

Crawl. Googlebot fetches the URL, if robots.txt allows it. It reads the links in the HTML and adds them to the crawl queue.
Render. Google puts every page with status 200 in a render queue, with or without JavaScript. A headless Chromium then runs the JavaScript. Google calls this its Web Rendering Service (WRS). The wait “may stay on this queue for a few seconds, but it can take longer than that.”
Index. Google reads the rendered HTML for more links, and uses it to index the page.

Two details in this pipeline cause many problems:

If the HTML response has noindex, Google “may skip rendering and JavaScript execution.” So JavaScript that removes noindex later may not work. Day 03 showed that robots.txt also stops Google from reading the page.
Google renders with “an evergreen version of Chromium”. This is the current version of Chrome, so modern JavaScript works. Other crawlers are different.

Not every crawler runs JavaScript

Google says that server-side rendering “is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript.” Google’s dynamic rendering page adds: “Other search engines may choose to ignore JavaScript.”

AI crawlers are the clearest case. In December 2024, Vercel studied crawler traffic on its network. It reported that “none of the major AI crawlers currently render JavaScript.” The list included GPTBot (OpenAI), ClaudeBot (Anthropic) and PerplexityBot. This is one company’s study at one time, not a promise from the crawler owners. But the safe rule is clear. If the content is only in JavaScript, assume that some crawlers cannot see it. Day 07 covers AI crawlers in detail.

JavaScript SEO rules

JavaScript SEO is the work that makes JavaScript sites crawlable and indexable. Google’s guides give these rules:

Rule	Why
Use `<a href="/products">` for links.	Google finds links only in `<a>` elements with an `href`.
Use the History API for routes, not `#/products` fragments.	Googlebot “can’t reliably resolve” fragment URLs.
Give missing pages a real 404, or add `noindex` with JavaScript.	A single-page app often returns 200 for every route, which makes soft 404s.
Put the canonical URL and robots meta tag in the original HTML.	Do not let JavaScript change them to a different value.
Do not load main content only after a click, swipe or typing.	Google “won’t load content that requires user interactions.”
Do not depend on cookies, permissions or WebSockets.	Googlebot clears cookies, declines permission requests and uses only HTTP.
Add a content hash to file names, such as `main.2bb85551.js`.	The WRS caches files aggressively and “may ignore caching headers.”

To check a page, use the URL Inspection tool in Search Console, or the Rich Results Test. Both show the rendered HTML that Google sees.

SSR vs CSR vs hydration

A rendering strategy is where and when the HTML of a page is made. The web.dev guide “Rendering on the Web” defines the main strategies:

Client-side rendering (CSR): the server sends a small HTML shell and JavaScript. The browser builds the page.
Server-side rendering (SSR): the server builds the full HTML for each request.
Static rendering: the full HTML for each URL is built once, at build time. The site that you read now uses static rendering.
Hydration: the server sends full HTML, then client-side JavaScript adds “application state and interactivity” to it.

How the site builds the page Client-side rendering (CSR) Server-side rendering (SSR) Static rendering (at build time) SSR with hydration

The crawler runs JavaScript (like Googlebot)

Example timings, not measurements. Pick each strategy. Then turn off 'The crawler runs JavaScript'. Notice that only client-side rendering hides the content, and that hydration makes users wait before the page responds.

For SEO, the question is simple: is the content in the HTML response? With SSR, static rendering and hydration, it is. Every crawler sees the text and links at once. With CSR, Google must render the page first, and some crawlers never do.

Speed is the other difference. web.dev says that SSR “generally produces a fast FCP”, which is the time until content shows. But building pages on the server “can increase your page’s TTFB”, the time to the first byte. Static rendering is fast on both. CSR can be slow on phones, because the JavaScript grows with the app.

Hydration has a cost too. A hydrated page can “appear to be loaded and interactive,” but it cannot respond until its scripts run. web.dev says that on mobile “this can take minutes.” This hurts responsiveness, which Core Web Vitals measure (see below).

Dynamic rendering is an older workaround. The server sends pre-rendered HTML to crawlers and the JavaScript app to users. Google now says that dynamic rendering “was a workaround and not a long-term solution.” Google recommends SSR, static rendering or hydration. Note that two Google sources do not agree here: the web.dev guide still calls dynamic rendering “an option worth considering.” The Search Central page is the one about Search, so follow it.

What matters most: put your important content and links in the HTML response. The framework does not matter, if it can do SSR or static rendering.

Machine-readable content

Machine-readable content is content that a program can parse from the page code, without looking at the page like a person. A crawler reads text, links and markup. It does not read a picture of a table.

Follow these rules:

Put text in HTML, not in images. Text inside an image is hard to extract. If you must use an image, add alt text: a short text in the alt attribute that describes the image. Google’s starter guide says that alt text “helps search engines understand what your image is about.”
Use HTML elements for their meaning. Use <h2> for headings, <table> for tables and <a href> for links. Do not build them from <div> elements and CSS.
Keep content in the HTML response where you can. The rendering section explains why.
Make structured data match the page. Structured data is code that labels facts on a page for machines. Day 05 teaches it.

The same rules apply to AI search. Google’s page on AI features has no extra technical requirements. It asks that “important content is available in textual form.” It also asks for structured data that “matches the visible text on the page.”

Core Web Vitals

Core Web Vitals are three metrics that measure real users’ experience of a page. Google publishes them through Chrome, web.dev and Search Console.

Metric	Measures	Good	Poor
LCP, Largest Contentful Paint	Loading: when the largest image or text block shows	2.5 s or less	over 4 s
INP, Interaction to Next Paint	Responsiveness: delay from a click, tap or key press to the next frame	200 ms or less	over 500 ms
CLS, Cumulative Layout Shift	Visual stability: how much the page moves without warning	0.1 or less	over 0.25

Values between “good” and “poor” are “needs improvement.” INP replaced First Input Delay (FID) as a Core Web Vital in 2024.

Each common problem has a common cause. A slow LCP often comes from a slow server or a large hero image. A poor INP comes from long JavaScript tasks, for example during hydration. web.dev lists the causes of layout shift: “images or videos with unknown dimensions,” web fonts that change size, and ads or widgets “that dynamically resize themselves.”

How the assessment works

The values come from real Chrome users. Google takes the 75th percentile: the value that 75% of visits are better than. A page passes the Core Web Vitals assessment only if all three metrics are good at the 75th percentile. If there is not enough INP data, LCP and CLS decide.

LCP (loading), 75th percentile: INP (responsiveness), 75th percentile: CLS (visual stability), 75th percentile: Not enough INP data (a page with few interactions)

Move the sliders. Notice that one metric in the orange or red zone fails the whole page. Then tick 'Not enough INP data' and see which metrics still count.

How much Core Web Vitals matter

Google says that “Core Web Vitals are used by our ranking systems.” Google also says that relevance comes first: it shows “the most relevant content, even if the page experience is sub-par.” Good scores do not guarantee top positions. Google warns that a perfect score “just for SEO reasons may not be the best use of your time.”

So treat Core Web Vitals as a tie-breaker and a user issue. Fix pages that are poor, because users leave slow and jumpy pages. Do not chase a score of 100.

The other page experience aspects are HTTPS, mobile display, no intrusive interstitials and not too many ads. Google says that they “don’t directly help your website rank higher.” Day 03 covered HTTPS.

PageSpeed Insights

PageSpeed Insights (PSI) is Google’s free tool at pagespeed.web.dev. Enter a URL and it gives a report for mobile and desktop. The report has two parts, and you must not mix them up.

Field data is at the top. It comes from the Chrome User Experience Report (CrUX), real Chrome users over the last 28 days. It shows the Core Web Vitals assessment. A new or small page might not have enough data. Then PSI shows data for the whole origin, or no field data at all.

Lab data is below. PSI runs Lighthouse, an automated test, once in a simulated browser. For mobile it simulates “a mid-tier device (Moto G4)” on a mobile network. It gives a performance score from 0 to 100, where 90 or above is good. It also lists fixes.

	Field data (CrUX)	Lab data (Lighthouse)
Source	Real Chrome users, 28 days	One simulated page load
Use it for	What users really get, and the CWV assessment	Finding causes and testing fixes
INP	Yes	No. It uses Total Blocking Time (TBT) as a stand-in, because a lab test has no user.

The field data is what Google uses for Core Web Vitals. The lab score is a debugging aid. For a full site, use the Core Web Vitals report in Search Console. It groups your URLs into good, needs improvement and poor.

Mobile friendliness

Mobile friendliness means that a page works well on a phone. The text is readable, the buttons are easy to tap and the page does not scroll sideways.

For Google, mobile is not optional. With mobile-first indexing, “Google uses the mobile version of a site’s content, crawled with the smartphone agent, for indexing and ranking.” In October 2023, Google said that this change was complete. Content that is only on the desktop version is not indexed.

Google lists three ways to build a mobile site:

Setup	How it works	Google’s view
Responsive design	Same HTML on the same URL; CSS changes the layout	Recommended: “the easiest design pattern to implement and maintain”
Dynamic serving	Same URL, different HTML for each device	Works, with the `Vary: User-Agent` header
Separate URLs	For example `m.example.com`	Works, but it needs the most care

For dynamic serving and separate URLs, check that the mobile version has the same content, headings, robots meta tags and structured data as the desktop version.

Google retired its Mobile-Friendly Test and the Mobile Usability report on 1 December 2023. Google said that mobile usability “remains critical for users”, and pointed to Lighthouse in Chrome instead. Today, you can check a page with Lighthouse, PSI and the device mode in Chrome DevTools.

Crawl tools: Screaming Frog and Sitebulb

Search Console shows Google’s view, but it is slow and it shows only your own sites. A site crawler (or audit crawler) is a program that crawls a site like a search engine and reports problems. It finds broken links, redirect chains, missing titles, noindex pages, canonical tags and orphan pages in minutes. These are the day 03 problems, found at scale.

	Screaming Frog SEO Spider	Sitebulb
Runs as	Desktop app for Windows, macOS and Linux	Desktop app, or Sitebulb Cloud in the browser
Free use	Free for up to 500 URLs	14-day free trial
Output	Tables and exports, “over 300 SEO issues”	Prioritised “Hints”, charts and PDF reports
JavaScript	Can render pages with built-in Chromium	JavaScript crawling at no extra cost

Screaming Frog is the common default for SEO professionals. Sitebulb gives more explanation and visuals, which helps when you report to people who are not SEOs. Both can crawl with JavaScript rendering. Then you can compare the raw HTML with the rendered HTML across a full site. This is the fastest way to find content that only JavaScript adds.

Both tools can also pretend to be Googlebot, follow your robots.txt and read your sitemaps. A crawl does not show what Google really did; server logs and Search Console do that (day 03). A crawl shows what Google can find.

What matters most

Put content and links in the HTML response. SSR or static rendering makes every crawler see the page. CSR depends on each crawler’s renderer.
Make the mobile version complete. Google indexes the mobile version only.
Read field data, not only the lab score. The CWV assessment uses real users at the 75th percentile.
Treat speed as a tie-breaker. Fix poor pages for users, but relevance ranks first.
Crawl your own site regularly. A site crawler finds technical problems before Google does.

Key terms

Rendering: turning HTML, CSS and JavaScript into the page that a person sees.
Web Rendering Service (WRS): Google’s headless Chromium that renders pages for indexing.
JavaScript SEO: the work that makes JavaScript sites crawlable and indexable.
Client-side rendering (CSR): the browser builds the page with JavaScript.
Server-side rendering (SSR): the server builds the full HTML for each request.
Static rendering: the full HTML of each URL is built once, at build time.
Hydration: client-side JavaScript adds interactivity to server-rendered HTML.
Dynamic rendering: a workaround that sends different HTML to crawlers and users; Google no longer recommends it.
Machine-readable content: content that a program can parse from the page code.
Alt text: a short text in an image’s alt attribute that describes the image.
Core Web Vitals: three metrics of real user experience: LCP, INP and CLS.
75th percentile: the value that 75% of visits are better than.
Field data: measurements from real users, in CrUX.
Lab data: measurements from one simulated page load, in Lighthouse.
Mobile-first indexing: Google indexes and ranks the mobile version of a page.
Responsive design: one HTML page whose layout changes with the screen size.
Site crawler: a program that crawls a site like a search engine and reports problems.

Exercise

Exercise · about 15 min

Check what crawlers see, and how fast it loads

Use Chrome. Use MDN Web Docs or your own site.

Open https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status. Open the page source (Ctrl+U, or Cmd+Option+U on a Mac). Search for the main heading and for the words Informational responses. Are they in the raw HTML?
Open the developer tools (F12). Press Ctrl+Shift+P (Cmd+Shift+P on a Mac), type Disable JavaScript and press Enter. Reload the page. What is still there? Turn JavaScript on again.
In the developer tools, turn on device mode (Ctrl+Shift+M). Pick a phone size. Does the page scroll sideways? In the source, search for name="viewport".
Open PageSpeed Insights and test the same URL. Look at the mobile tab. Write down: does field data exist for the URL or only for the origin? Does it pass the Core Web Vitals assessment? Write down LCP, INP and CLS.
Scroll to the lab data. Write down the performance score. Does the lab result agree with the field result?
Write one sentence: which rendering strategy does the page use, and is it good for crawlers?

Show a possible solution

These were the results on 8 October 2026. Your results can be different.

Yes. The raw HTML has <h1>HTTP response status codes and the section Informational responses. The server sends the full content.
The text, headings and links are still there without JavaScript. Some interactive parts, such as the theme switch or search, may not work.
The page fits a phone screen and has a <meta name="viewport"> tag, which tells the phone browser to use the screen width. This is responsive design.
A popular page such as this one usually has its own field data. The values change every day, so compare yours with the thresholds: LCP 2.5 s, INP 200 ms, CLS 0.1. If one metric is not good, the assessment fails.
The lab score can be lower or higher than the field result suggests. The lab simulates a slow phone once. The field data shows real users over 28 days. Trust the field data for the assessment, and use the lab data to find causes.
MDN sends full HTML from the server (server-side or static rendering, with JavaScript added on top). Every crawler, also one that does not run JavaScript, can read the content and links.

Check yourself

A shop built as a single-page app shows products only after its JavaScript runs. In Search Console, URL Inspection shows the products in the rendered HTML. What is still a risk?Google cannot index the productsCrawlers that do not run JavaScript, such as many AI crawlers, see an empty pageThe products will fail the Core Web Vitals assessmentGoogle will give the site a manual action

A page sends noindex in its HTML. A script removes noindex after the page loads. Why can the page still stay out of Google?Google never runs JavaScriptGoogle may skip rendering when the HTML has noindex, so it never sees the changenoindex in HTML is ignored, only robots.txt countsThe script makes the page fail Core Web Vitals

PageSpeed Insights shows a lab performance score of 45 for a page. The field data shows LCP 2.1 s, INP 150 ms and CLS 0.04. What is the correct reading?The page fails Core Web Vitals, because the score is below 50The page passes the Core Web Vitals assessment; the lab score shows a simulated slow phone and is a debugging aidThe field data is wrong, because it does not match the labThe page passes only if the lab score is over 90

A site has a desktop version with full articles and an m. version that shows only the first paragraph. What happens with mobile-first indexing?Google indexes the desktop version, because it has more contentGoogle indexes the mobile version, so most of each article is not indexedGoogle merges both versionsNothing, because separate URLs are not allowed

You must audit a 2,000-page site today, at no cost, and check which pages add content with JavaScript. Which plan fits best?Screaming Frog free version with JavaScript renderingScreaming Frog with a paid licence, or a Sitebulb free trial, with JavaScript renderingThe Mobile-Friendly TestPageSpeed Insights on each page

Sources

Understand the JavaScript SEO basics (Google Search Central)
Fix Search-related JavaScript problems (Google Search Central)
Dynamic rendering as a workaround (Google Search Central)
Rendering on the Web (web.dev)
The rise of the AI crawler (Vercel, December 2024)
SEO Starter Guide (Google Search Central)
AI features and your website (Google Search Central)
Understanding Core Web Vitals and Google search results (Google Search Central)
Understanding page experience in Google Search results (Google Search Central)
Web Vitals, Largest Contentful Paint, Interaction to Next Paint and Cumulative Layout Shift (web.dev)
About PageSpeed Insights (Google for Developers)
Mobile site and mobile-first indexing best practices (Google Search Central)
Mobile-first indexing has landed (Google Search Central Blog, October 2023)
More on how we’re evolving page experience (Google Search Central Blog, April 2023)
Screaming Frog SEO Spider and How to crawl JavaScript websites (Screaming Frog)
Sitebulb and Sitebulb pricing (Sitebulb)
roadmap.sh SEO roadmap: Rendering & JavaScript SEO, Machine-Readable Content (roadmap.sh)

Day 04 moves from technical SEO to content: keyword research, on-page SEO and the features of the results page.