How to Use Robots.txt for Your Website: A Complete Guide

If you want search engines like Google, Bing, or Yahoo to index your website correctly, using a robots.txt file is essential. This small but powerful text file helps you control how search engine bots crawl and index your site. In this guide, we’ll explain what robots.txt is, why it matters, and how to use it effectively.

📌 What is a Robots.txt File?

The robots.txt file is a plain text file located in the root directory of your website (e.g., www.example.com/robots.txt). It gives instructions to search engine crawlers (also called bots or spiders) about which parts of your site should or shouldn’t be crawled.

✅ Example:


User-agent: *
Disallow: /admin/
Allow: /public/

This tells all bots (*) not to crawl the /admin/ folder but to allow access to /public/.

💡 Why is Robots.txt Important?

Here are a few reasons to use a robots.txt file:

Control crawler traffic and reduce server load
Prevent indexing of sensitive pages (e.g., login, admin, staging)
Avoid duplicate content issues
Guide bots to your sitemap
Protect user experience by hiding unfinished or private sections

🛠️ How to Create and Use Robots.txt

Step 1: Open a Text Editor

Use Notepad (Windows), TextEdit (Mac), or any code editor to create a new text file. Save it as robots.txt.

Step 2: Define Rules

Each rule includes:

User-agent: Defines the bot (use * for all)
Disallow: Blocks access to folders or pages
Allow: Grants access (used mainly when you're unblocking something inside a disallowed directory)
Sitemap: (Optional) Link to your XML sitemap

Example Configuration:


User-agent: *
Disallow: /admin/
Disallow: /temp/
Allow: /public/
Sitemap: https://www.example.com/sitemap.xml

Step 3: Upload to Your Website

Place the robots.txt file in your root directory (e.g., https://www.example.com/robots.txt).
Use FTP, cPanel, or your CMS (e.g., WordPress file manager) to upload.

Step 4: Test with Google Search Console

Use the Google Robots.txt Tester to check for:

Syntax errors
Blocked resources
Bot behavior validation

🔒 What NOT to Do with Robots.txt

❌ Don’t use it to hide private data (it's publicly visible!)
❌ Don’t block important content you want ranked in Google
❌ Don’t rely on it to remove URLs from search results (use "noindex" meta tag or Search Console)

🔎 Best Practices for Robots.txt

Tip	Description
✅ Use `*` to apply to all bots	`User-agent: *`
✅ Use trailing slashes for folders	e.g., `Disallow: /private/`
✅ Submit your sitemap	`Sitemap: https://example.com/sitemap.xml`
⚠️ Use caution with `Disallow: /`	Blocks entire site!
✅ Regularly review the file	Especially after site updates or redesigns

📘 Common Use Cases

Goal	Robots.txt Example
Block admin area	`Disallow: /admin/`
Allow only Googlebot	`User-agent: Googlebot`
Block image crawling	`User-agent: Googlebot-Image` + `Disallow: /`
Guide bots to sitemap	`Sitemap: https://example.com/sitemap.xml`

✍️ Final Thoughts

Using robots.txt effectively can improve your site's crawl efficiency, protect sensitive content, and enhance your SEO strategy. Whether you're a beginner or a developer, this simple file gives you big control over how your site is seen by search engines.

Robotstxt