Tell search engines what to crawl. Pick a preset, add your own rules and sitemap, and copy a clean, valid robots.txt in seconds.
Upload the file to the root of your domain so it lives at https://yourdomain.com/robots.txt. Test it in Google Search Console before relying on it.
A misconfigured robots.txt can quietly hide your best pages from Google. Get a free technical SEO audit and we'll check crawlability end-to-end.
Get my free technical audit →A robots.txt file lives at the root of your domain and tells search-engine crawlers which parts of your site they may or may not request. It's the first file most bots check. Used well, it keeps low-value pages (carts, admin areas, internal search) out of crawl budget; used carelessly, it can accidentally hide pages you want to rank.
Never Disallow a page you want indexed — robots.txt controls crawling, not indexing. Always include your sitemap. And remember robots.txt is public: don't list sensitive URLs there, since anyone can read the file.
noindex meta tag instead.