What is this Robot.txt I hear about?

What is this Robot.txt I hear about?

It's a file that can guide search engine crawlers in your website

RE:

It is one type of file where you can set command to search engine crawler for allowing or disallowing about your site pages.

I totally support your definition

Robots.txt file is a text file that tells the search engine spiders what it can and cannot index. For instance, you wouldn;t want it indexing your cgi-bin folder, so that would be added in yoru Robots.txt file telling the spiders to stay out of that folder.

I use it with at my site to restrict access to some folders

It use used to drive crawlers to that pages you need

Thanks for sharing

Robot.txt files tell your instructions to a search engine robot. it depends on what you what it to do. if you dont have the robot.txt file then when a robot visits your site and if it doesnt find the file, it will start to visit all the pages and contents of your site. these files are like walls which helps in preventing some files like java scripts, some images or any other files from indexing. it is also important to make a note that it is not a method that prevents search engines from crawling your site. so it is important to create a robot.txt file. if you need this file to be created you can get it from TheWebPole.com