|
The Proper Way To Use The robot.txt File |
| When optimizing your web site most webmasters don?t consider using the robot.txt file. This is a very important file for your site. It let the spiders and crawlers know what they can and can not index. This is helpful in keeping them out of folders that you do not want index like the admin or stats folder.
Here is a list of variables that you can include in a robot.txt file and there meaning:
1. User-agent: In this field you can specify a specific robot to describe access policy for or a ?*? for all robots more explained in example.
2. Disallow: In the field you specify the files and folders not to include in the crawl.
3. The # is to represent comments
Here are some examples of a robot.txt file
User-agent: *
Disallow:
The above would let all spiders index all content.
Here another
User-agent: *
Disallow: /cgi-bin/
The above would block all spiders from indexing the cgi-bin directory.
User-agent: googlebot
Disallow:
User-agent: *
Disallow: /admin.php
Disallow: /cgi-bin/
Disallow: /admin/
Disallow: /stats/
In the above example googlebot can index everything while all other spiders can not index admin.php, cgi-bin, admin, and stats directory. Notice that you can block single files like admin.php. |
Author Info:
Jimmy Whisenhunt is the webmaster at VIP Enterprises http://www.vipenterprises.org
vipenter@vipenterprises.org
Articles distributed free by http://www.ArticleInfoz.net
|
|
| Most Recent
Free Articles added to our site : |
Vodafone Smart Tab 7 Tablet Device
Vodafone has fixed their steps into the market of top gadgets. Vodafone is planning to step out in the tough air of competition. Vodafone is an Australian brand and has the business of providing the network connectivity to mobile or broadband. In India, Vodafone has acquired the Hutch few years ago.[ Read Article] LG A290 Triple 3 SIM Mobile Phone
LG is very popular and reliable brand in the view of many people. LG Mobile Smart Phones introduced many advanced and heavily loaded featured smartphones in the market. Brand manufactures the ph[ Read Article]
Making Articles Work for You about The Proper Way To Use The robot.txt File from our free Computers article resources.
|
| Adding themed content is valuable to your site |
A webmaster or website operator do not risk any chance of getting sued for copyright infringement because they are public domain, once again meaning that anybody can use it. Writing articles by using public domain wont require as much work as writing one from scratch would. You save a lot of time also.
|
