Free Robots.txt and LLMs.txt Checker
See how Googlebot, Bingbot, GPTBot and PerplexityBot read your crawl rules, check llms.txt, and generate a clean file. No signup.
- ✓ 100% free
- ✓ No signup
- ✓ Works for any website
robots.txt & llms.txt generator
Runs in your browser. Edit the options, then copy or download the files and upload them to your site root.
User-agent: * Allow: /wp-admin/admin-ajax.php Disallow: /wp-admin/ Disallow: /cart/ Disallow: /checkout/ Sitemap: https://www.example.com/sitemap.xml
# Example Business > Example Business is a digital agency in Delhi NCR, India, offering SEO, Google Ads and website design for small businesses. ## Key pages - [Services](https://www.example.com/services): What we offer and pricing - [About](https://www.example.com/about): Our team and story - [Contact](https://www.example.com/contact): Phone, email and address
A robots.txt file tells crawlers which parts of your site they may request. A typo, a leftover Disallow rule, or a plugin that blocks the wrong path can keep important pages from being crawled. This free robots.txt checker fetches your file, shows how selected bots are treated, and can generate a clean starting file. It also looks for an llms.txt file. You do not need an account.
How to use the robots.txt checker
- Enter your site's root URL, such as https://www.example.com. The tool looks for /robots.txt and, where relevant, /llms.txt.
- Review the result for Googlebot, Bingbot, and AI crawlers such as GPTBot and PerplexityBot. Confirm that pages you want crawled are allowed, and that paths you meant to keep private are the ones being blocked.
- If you need a new file, use the generator, edit the rules so they match your site, then upload the file to the root of the domain and fetch the live URL again.
What the tool checks
For each selected bot, the checker shows whether robots.txt allows or disallows a path, and it flags obvious problems such as a missing file, a malformed line, or a rule that blocks the whole site. It also reports whether an llms.txt file is present at the root. The generator can create a simple robots.txt, including a Sitemap line when you provide a sitemap URL. Treat that output as a draft and read every line before you upload it.
Why robots.txt and llms.txt matter
Robots.txt controls crawling, not indexing
If you Disallow a URL, well-behaved crawlers should stop requesting it. That does not always keep the URL out of Google's index. If other sites link to the page, Google may still list the URL, often with little or no snippet. To keep a page out of the index, use a noindex tag or HTTP header on a URL that Google is still allowed to crawl, or protect the page with a login.
llms.txt is optional
llms.txt is a proposed convention: a plain text file at the root of the site that describes the business and points to useful pages for AI tools. It is not an official standard, and Google does not use it. If you add one, keep it short, accurate and optional. It should not replace robots.txt, and it should not be treated as a ranking signal.
Tips from our team
Do not block CSS, JavaScript or image paths that Google needs to render the page. Do not use robots.txt as the only way to hide a private tool or an admin area; those should also sit behind authentication. If you block some AI bots and allow others, do it on purpose, not by copying a file from another site.
If you would like our team to review crawl rules as part of a wider look at the site, Technical Speaks offers a free website and SEO audit.
Want an expert to fix it for you?
Our team handles technical SEO, content and local SEO for businesses in India and worldwide. Send us a message and we'll reply with clear next steps.
More free SEO tools
View all SEO tools →Frequently asked questions
Is this robots.txt checker free?
Yes. You can check robots.txt and llms.txt, and use the generator, with no signup.
If I block a page in robots.txt, will it stay out of Google?
Not necessarily. Robots.txt can stop Googlebot from crawling the page, but the URL can still appear in the index if Google found it through links. Use noindex on a crawlable page when you want it kept out of the index.
Does Google use llms.txt?
No. llms.txt is a proposed convention, not an official standard, and Google does not use it. Other AI tools may choose to read it, but adding the file is optional.
Where should robots.txt and llms.txt live?
At the root of the host, for example https://www.example.com/robots.txt and https://www.example.com/llms.txt. A file in a subfolder will not be used as the site-wide crawl policy.