Blogger robots.txt Generator

The tool is fully ready and tested, but errors can still happen. If you find any errors or mistakes in the guide, please let me know in the comments. Thanks in advance!

A robots.txt file is tiny, but one wrong line can keep Google away from your whole blog. The generator above helps you build a correct file for Blogger in a few clicks. Below, you will learn what every option does, how to choose the best settings, and the basics of robots.txt that every blogger should know.

Why I Built This Tool

When I was blogging myself, and later when I got to know many other bloggers, I kept seeing the same problem. Most bloggers are confused about robots.txt. They do not know why it exists or how to use it properly. So they do what feels easiest: they copy a random template from a website, or they follow what a YouTuber says, paste the code into Blogger and never look at it again.

This is not because they are careless. Many tutorials just give a block of code without explaining what each line does. But a robots.txt file is not something you should treat as a copy-and-paste job, because a single wrong line can block pages you want in Google.

That is why I made this page. It gives you a tool to generate the file and a complete guide to understand it, in one place. I personally believe this is a meaningful step in every blogger's journey. Once you understand robots.txt, you stop guessing, you make better SEO decisions, and blogging becomes less confusing.

How to Use the Robots.txt Generator

Enter your blog address. Use your full main address, for example https://www.example.com.
Set the AdSense crawler. Keep Include User-agent: Mediapartners-Google ticked and leave the dropdown on Disallow: (Default).
Choose your Disallow and Allow rules. Tick the ready-made rules, or add your own paths with the custom boxes.
Choose your sitemaps. Keep the ones you need. Tick Blog has more than 500 posts? if your blog is large.
Check the output. Read the Generated robots.txt box line by line, then click Copy Code or Download.

The output updates live as you change options. The tool runs in your browser, and the rules you choose are not sent to any server.

What Each Option Does

Blogger Site Address

The tool uses your address to build full sitemap URLs. If you forget https://, it adds it. If you paste a full page address, it keeps only the domain part. Use the same version of your domain that Google indexes (with or without www), because a robots.txt file only applies to the exact host where it is served.

Media Partner (Google)

Mediapartners-Google is the Google AdSense crawler. It is separate from Googlebot, which builds the Google Search index. According to Google AdSense Help, it is not controlled by the general User-agent: * group, so it needs its own group. If you untick the checkbox, the group is removed. The crawler is not blocked, but your intention is no longer written in the file.

Dropdown option	What it does
Disallow: (Default)	Writes an empty `Disallow:`. Nothing is blocked, so the AdSense crawler can read every page. Recommended.
Allow: /	Same practical result, written explicitly.
Same as User-agent: *	Copies all your rules into the AdSense group. Ads on blocked pages may be less relevant, or may not show.

Disallow and Allow Rules

A Disallow rule asks crawlers not to fetch URLs that start with a path. An Allow rule opens a path inside a blocked area. When two rules match, Google follows the more specific one, which is the one with the longer path.

Rule	What it does	Our pick
`Disallow: /search`	Blocks Blogger search results, label pages and their pagination. Everything that starts with `/search` is covered.	Keep on
`Disallow: /b/`	Blocks Blogger internal utility URLs, such as post previews.	Keep on
`Disallow: /?q=`	Blocks any URL with a `?q=` parameter. The `*` is a wildcard.	Optional
`Disallow: /feeds/posts/default`	Blocks your main Atom feed.	Leave off. Feeds can help discovery.
`Disallow: /?m=1` and `/?m=0`	Blocks the mobile and desktop view copies of your URLs.	Leave off
`Allow: /search/label/`	Keeps label pages open even though `/search` is blocked. It works because it is longer, so it is more specific.	On only if you have a few useful labels
`Allow: /p/` and `Allow: /feeds/posts/default`	Explicitly allow static pages and the feed.	Optional, already allowed by default

Why are ?m=1 and ?m=0 marked "Not Recommended"? These URLs are copies of your normal pages, and in most Blogger themes they point to the clean URL with a canonical tag. If you block them, Google cannot open the page, so it cannot read that tag. The blocked URL may still show up in Google without a description. It is better to let Google crawl them.

The Custom Disallow and Custom Allow boxes accept your own paths, like /p/private-notes.html. The tool adds a leading / if you forget it. The two feed rules cannot be active together, so ticking one unticks the other. The tool also adds Allow: / as the final rule, which simply says that everything else is open.

Sitemaps

A Sitemap: line tells crawlers where your sitemap files are. It must be a full URL, and it is not tied to any user-agent group.

Primary Atom Sitemap: /atom.xml?redirect=false&start-index=1&max-results=500 is a feed of up to 500 posts. Google accepts Atom feeds as sitemaps.
Standard Posts Sitemap: /sitemap.xml is the sitemap Blogger creates automatically.
Pages Sitemap: /sitemap-pages.xml covers your static pages.
More than 500 posts: each Atom request returns up to 500 posts, so the tool adds one line for each extra block (501, 1001 and 1501, up to 2,000 posts). For larger blogs, add the next ranges in the Custom Sitemap box.

Blogger's sitemap.xml may not list every post on a large blog, which is why the Atom lines are a useful backup. After you submit your sitemaps, check the Sitemaps report in Google Search Console to see how many URLs Google found.

The Best robots.txt for a Blogger Blog

There is no perfect file for every blog, but the tool's defaults are a safe starting point for most Blogger sites. Keep the AdSense crawler open, block internal search pages, leave the ?m= rules off, and show your sitemaps. The result looks like this:

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Disallow: /b/
Allow: /search/label/
Allow: /

Sitemap: https://www.example.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-pages.xml

Here is how Google reads it:

/2026/09/my-post.html is allowed, because only Allow: / matches.
/search?q=robots is blocked by Disallow: /search.
/search/label/Git is allowed, because Allow: /search/label/ is longer than Disallow: /search.

Choose one strategy for label pages. In Blogger, label pages usually count as search pages. If you allow them here, do not also set noindex for "Archive and search page tags" in Blogger's custom robots header tags, because the two settings work against each other. If your labels are many and thin, untick Allow: /search/label/ instead.

How to Add It to Blogger

Click Copy Code in the generator.
In your Blogger dashboard, open Settings and find Crawlers and indexing.
Turn on Enable custom robots.txt, click Custom robots.txt, paste the code and save.
Open yourdomain.com/robots.txt and confirm you see exactly what you pasted.
In Google Search Console, check the robots.txt report in Settings, and submit your sitemaps in the Sitemaps report.
Use URL Inspection on one important post and one blocked URL to confirm the result is what you expect.

Info! Google generally caches robots.txt for up to 24 hours, so a new version may not take effect immediately.

What Is a robots.txt File?

A robots.txt file is a plain text file at the root of your site, such as https://www.example.com/robots.txt. It tells crawlers which paths they may fetch, and well-behaved crawlers read it before they crawl your pages. The idea comes from the Robots Exclusion Protocol, used since the 1990s and published as an official standard, RFC 9309, in 2022.

The file applies only to the host, protocol and port where it is served. So https://example.com and https://www.example.com are treated separately, and a file placed in a subfolder is not used at all.

Important! robots.txt is a request, not security. Bad bots can ignore it, the file is public, and a blocked URL can still appear in Google without a description if other sites link to it. Never use it to hide private pages.

Why robots.txt Matters for SEO

robots.txt is not a ranking factor, and changing it will not push a page higher in Google. Its value is indirect:

Better crawl focus. Blogger creates many URLs you never wrote, such as search results and pagination. Google says crawl budget matters mostly for very large sites, but keeping bots on your real posts is a good habit for any blog.
Clear sitemap discovery. The Sitemap: line is one of the easiest ways to show crawlers your important URLs.
AdSense access. A correct Mediapartners-Google group lets the ad crawler read your content.
Risk control. A wrong rule can block important pages. One accidental Disallow: / blocks your entire site from crawling.

It also helps to know which tool does which job. Use robots.txt to control crawling. Use a noindex tag to keep a page out of search results, and keep that page crawlable so Google can see the tag. Use a canonical tag to merge duplicate URLs.

Warning! Do not combine a robots.txt block with a noindex tag on the same URL. If the URL is blocked, Google cannot open the page, so it never sees the noindex instruction.

Standard robots.txt Rules

Directive	Meaning	Example
`User-agent`	Starts a group of rules for one crawler. `*` means all crawlers without their own group.	`User-agent: Googlebot`
`Disallow`	A path crawlers should not fetch. An empty value blocks nothing.	`Disallow: /search`
`Allow`	A path crawlers may fetch, often an exception inside a blocked path.	`Allow: /search/label/`
`Sitemap`	The full URL of a sitemap file.	`Sitemap: https://www.example.com/sitemap.xml`

You can also use * to match any characters, $ to mark the end of a URL (for example, Disallow: /*.pdf$), and # to start a comment.

Name the file robots.txt in lowercase, save it as UTF-8 plain text, and place it at the root of the host.
Google reads only the first 500 KiB of the file. A normal Blogger file is far smaller.
Paths are case-sensitive. /Search does not match /search.
The most specific rule wins. If an Allow and a Disallow rule are equally specific, Google uses the less restrictive one, which is Allow.
Anything not blocked is allowed by default.
A crawler follows only the most specific group that matches its name. It does not merge that group with the * group.

Google supports only four fields: user-agent, allow, disallow and sitemap. It ignores others like Crawl-delay, and it stopped supporting noindex inside robots.txt in 2019. If Google cannot find the file (a 404 error), it assumes there are no restrictions.

Common robots.txt Mistakes

Leaving Disallow: / in the file, which blocks the whole site.
Using Disallow to remove pages from Google. Use noindex instead.
Blocking CSS, JavaScript or images that Google needs to render your pages.
Blocking ?m=1 or ?m=0, so Google cannot read the canonical tag.
Using a sitemap URL with the wrong domain version, or a relative URL instead of a full one.
Changing the AdSense group without a clear reason.
Adding lines like Crawl-delay that Google ignores.

Frequently Asked Questions

Is the Robots.txt Generator free and safe to use?

Yes. It is free, needs no signup, and runs in your browser. The rules you choose are not sent to a server.

Does robots.txt improve my Google ranking?

Not directly. It helps by guiding crawlers away from low-value URLs and toward your sitemaps. Good content, clear structure and a healthy site still decide your rankings.

Does Disallow remove a page from Google Search?

No. Disallow only stops crawling, and a blocked URL can still be indexed without a description if other pages link to it. To keep a page out of search results, use a noindex tag and let Google crawl the page.

Why should I block /search on a Blogger blog?

Blogger creates search and label URLs automatically, and pagination can multiply them. These pages are thin and repeat your content, so most blogs block /search to keep crawling focused on real posts and pages.

What is Mediapartners-Google, and should I keep it?

It is the Google AdSense crawler. Keeping its group with an empty Disallow: lets it read all your pages so it can understand your content and serve relevant ads. It does not decide your Google Search rankings.

How long do robots.txt changes take to work?

Blogger shows the new file right away, but Google generally caches robots.txt for up to 24 hours. Changes in search results can take longer, because pages Google already crawled are not re-evaluated at once.

Can I use this tool for WordPress or another platform?

The ready-made options are built for Blogger, such as /search, /b/ and Atom sitemap URLs. Other platforms use different paths. You can still use the custom boxes, but check that every rule matches your own site.

Final Thoughts

A good robots.txt file is short, clear and careful. Use the generator above to build yours, read every line of the output, and test it in Google Search Console before you rely on it. For official details, read Google's guides on robots.txt basics and how Google interprets the robots.txt specification, and the AdSense Help page about the AdSense crawler.

BytebaseX