How Search Engines Work: Crawling, Indexing and Ranking

Google describes Search as fully automated, using crawlers that regularly explore the web to find pages for its index. Most pages in results are found automatically rather than submitted.

Google says search works in three stages, and not all pages make it through each one: crawling, indexing and serving results.

Stage 1: Crawling

There’s no central registry of web pages, so Google constantly looks for new and updated ones. This is called URL discovery. Some pages are already known; others are found through links, such as a category page linking to a new post, or through sitemaps.

Googlebot decides algorithmically which sites to crawl, how often and how many pages to fetch, and slows down if a site struggles (for example, returning server errors). It renders pages and runs JavaScript using a recent version of Chrome.

• Common crawl problems: server issues, network issues, and robots.txt rules blocking access.
• Pages behind a login aren’t crawled.

Stage 2: Indexing

Google analyzes the text, key tags such as the <title> element and alt attributes, images and video. It groups pages with similar content and chooses the most representative as the canonical, the version that may be shown in results. It also collects signals like language, country and usability.

• Common indexing problems: low-quality content, robots meta rules (noindex) and site designs that make indexing difficult.
• Search Console’s Page indexing report shows why pages aren’t indexed; see how to fix indexing problems.

Stage 3: Serving and ranking

When someone searches, Google looks in its index for matching pages and returns those it judges most relevant and highest quality. Google says relevancy depends on hundreds of factors, which can include location, language and device: a search for bicycle repair shops shows different results in Paris than in Hong Kong. The search features shown also change with the query.

For the systems involved, see how Google ranks pages: what is documented.

Indexed but not showing?

Google lists three common reasons a page can be indexed but not appear: the content is irrelevant to the queries, the content quality is low, or robots meta rules prevent serving.

Frequently asked questions

Sources

• Google Search Central: In-depth guide to how Google Search works
• Google Search Central: SEO Starter Guide

Last reviewed October 14, 2026.