Lesson kit

This page shows the parts that lessons use. It is not in the sidebar. Lessons follow the rules in CLAUDE.md.

Recap note

Last time

You learned what a crawler is. Read lesson 7 again. Internal links start with /learn/.

Interactive diagram

Pages on the site:

6

Toy example. Move the slider. The diagram adds one box for each page.

Exercise

Exercise · about 5 min

Find the robots.txt file of a site

1. Open https://example.com/robots.txt in your browser.
2. Write down each User-agent line that you see.

Show a possible solution

The file has no User-agent lines. Many small sites do not have a robots.txt file.

Check yourself

Which item tells a crawler which pages it must not crawl?

The title tagThe robots.txt fileThe meta description

What does the diagram above show?

One box for each pageOne box for each linkOne box for each visitor