## What Is Robots.txt?
A **robots.txt file** is a text document used to give directives to search engine bots and spiders on how to crawl and index pages on a website.
## Why Is Robots.txt Important In SEO?
Robots.txt is an important tool for a website to use and it has a few different functions.
It is an effective way of **controlling** **[crawl budget](https://growhackscale.com/marketing-glossary/crawl-budget)**. By blocking certain sections of a website from being crawled, [Google crawlers](http://growhackscale.com/marketing-glossary/google-crawlers) can direct their efforts to more important sections of your website.
It is also important to use robots.txt in order to **prevent internal search results pages from being** **[indexed](http://growhackscale.com/marketing-glossary/indexing-seo)**, or similarly other pages that you don’t want to be seen in [SERPs](https://growhackscale.com/marketing-glossary/serp) like login pages.
Robots.txt is also useful if **you need to hide pages that are under construction** so that your audience won’t see them until they’re ready.
## What Should You Watch Out For?
While robots.txt is a useful tool, it does have its disadvantages.
- While it enables webmasters to stop certain pages from being crawled, this doesn’t necessarily prevent the URLs from appearing in SERPs. For this, you should use a [noindex tag](http://growhackscale.com/marketing-glossary/no-index-meta-tag).
- Using a robots.txt on a web page also prevents the spread of [link equity](http://growhackscale.com/marketing-glossary/link-equity) on that page.
- Additionally, **if a website’s security is not up to scratch attackers can use robots.txt to discover private data**.
## Robots.txt Best Practices
There are a few other things that you need to take into consideration when choosing to use robots.txt.
- Pay close attention when making changes to robots.txt, one small mistake can have a big impact and make certain sections of your site unindexable
- Don’t use robots.txt to block sensitive data from SERPs because it can still be indexed, use a noindex tag instead
- Ensure that you’re placing robots.txt on the right sections of your website, you don’t want to block essential pages from being indexed
- Make sure to add your [sitemap’s](https://growhackscale.com/marketing-glossary/xml-sitemap-seo) location to robots.txt
- To ensure that your robots.txt is discoverable, put it in your website’s root directory
## When Should You Avoid Using Robots.txt?
Robots.txt is not the go to tool for every situation. There are some circumstances when you should avoid using it.
- You may have heard that [duplicate pages](https://growhackscale.com/marketing-glossary/duplicate-content) can be fixed with a robots.txt, however this is wildly outdated. You should instead use a [canonical tag](http://growhackscale.com/marketing-glossary/meta-tags-seo). This will allow you to keep the duplicate page and preserve link equity.
- If a web page is no longer in use, then you shouldn’t be using robots.txt to remedy the situation. Instead, use a [301 redirect](https://growhackscale.com/marketing-glossary/301-redirect) in order to send your users to the right web page.
- In a situation where a website doesn’t want its web page to appear in SERPs but wants to preserve link equity, opt for a noindex tag instead of a robots.txt.
### How Can We Help?
Still confused? Get in touch and we’ll help you clarify with a free strategy plan.
In the meantime, you could also check out [our training courses](https://growhackscale.com/academy) to give you some essential pointers and practises to becoming a pro at SEO.
