Robots.txt Generator at a glance
- What it does
- Build a valid robots.txt file with the right allow and disallow rules, and understand what it actually controls - crawling, not indexing.
- Where it runs
- Entirely in your browser — no data is uploaded
- Works offline
- Yes, once the page has loaded
- Cost
- Free, with no account and no usage limit
- Category
- Webmaster & SEO
How to use the generator
- Choose which crawlers your rules apply to -
*for all, or a specific user agent. - Add the paths you want to disallow or explicitly allow.
- Add your sitemap URL, which should be absolute.
- Copy the file and upload it to your site root, at
/robots.txt.
The syntax
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /*?sort=
Allow: /admin/public-page
Sitemap: https://example.com/sitemap.xml| Directive | Meaning |
|---|---|
User-agent | Which crawler the following rules apply to. * matches any crawler without its own block. |
Disallow | A path prefix the crawler should not request. An empty value allows everything. |
Allow | An exception carved out of a broader Disallow. |
Sitemap | An absolute URL to a sitemap. Applies site-wide regardless of user-agent blocks. |
* and $ | Wildcard and end-of-URL anchor. Disallow: /*.pdf$ blocks PDFs. |
Rules match by prefix, so Disallow: /admin blocks /admin/, /administrator and /admin-panel. Add a trailing slash when you mean a directory. Paths are case-sensitive.
Blocking a page does not remove it from Google
This is the mistake worth understanding properly. Disallow stops a crawler fetching a URL. It does not stop the URL appearing in search results.
If other pages link to a blocked URL, Google can still index it based on those links, producing the familiar result with no description and the note that information is unavailable. Worse, because the crawler is not permitted to fetch the page, it cannot see a noindex tag you may have placed there - so blocking a page in robots.txt actively prevents it from being removed.
To remove a page from search results, allow crawling and add <meta name="robots" content="noindex">. Only add a robots.txt block after the page has dropped out of the index, if you want to save crawl budget.
Robots.txt is not a security control
The file is public, at a predictable URL, and lists exactly which paths you consider sensitive. Anyone can read it, and the first thing many automated scanners do is fetch it for a list of interesting directories. Compliance is voluntary - well-behaved crawlers respect it, and malicious ones treat it as a map.
Never list an admin panel, a backup directory, a staging area or an unreleased path here. Protect those with authentication, IP restrictions or server configuration. If a URL genuinely must not be accessed, robots.txt is the wrong mechanism entirely.
Rules worth having, and rules to avoid
Usually worth blocking: internal search result pages, faceted navigation parameters that generate near-infinite combinations, cart and checkout pages, and any print or duplicate view of existing content. These waste crawl budget without adding value.
Never block: your CSS and JavaScript. Google renders pages to evaluate them, and a page it cannot style or script may be judged as broken or as failing mobile usability. This was a widespread problem when sites blocked /assets/ wholesale.
The most expensive mistake:
User-agent: *
Disallow: /That blocks the entire site. It belongs on staging servers and is copied to production more often than anyone would like to admit. If organic traffic disappears overnight, check this file first.
Testing and verification
Google Search Console includes a robots.txt report that shows the fetched file and flags syntax errors. Use it after any change - a stray character can invalidate a directive silently.
Also confirm the file returns HTTP 200 and is served as text/plain from the root of each domain and subdomain you run. A subdomain needs its own file; example.com/robots.txt does not govern blog.example.com. And a robots.txt returning a 500 error causes Google to pause crawling entirely, which is a subtle way to lose a lot of traffic.
Frequently asked questions
No. It stops the page being crawled, not indexed. A blocked URL can still appear in results if other pages link to it - and because the crawler cannot fetch it, it will never see a noindex tag. Use noindex and allow crawling instead.
The root of each domain: https://example.com/robots.txt. It must be at the root, and each subdomain needs its own.
Not strictly - a missing file means everything may be crawled, which is fine for many sites. It is still worth having one to declare your sitemap and to avoid 404s in your logs.
Several publish user agent names you can disallow, such as GPTBot and CCBot. Compliance is voluntary, as with any crawler, so treat it as a request rather than a control.
The paths are. The directive names are not. /Admin/ and /admin/ are different rules.
Nothing you enter here leaves your browser
Robots.txt Generator does its work in JavaScript running on your own device. The page loads once, and after that there is no upload step and no server involved — which matters here because site structure and configuration are worth keeping to yourself.
You can verify this rather than taking our word for it: load the page, disconnect from the internet, and the tool keeps working. Our privacy policy sets out what is and is not collected, and this guide explains why the distinction matters.