How to configure your "robots.txt" file?

Hundreds of search engines send out their robots every day to crawl the web. Whether it is for indexing or spam purposes you may not want them to crawl some of your files or folder like the /images folder of your site as it is meaningless to you and is also a waste of your bandwidth.

The file robots.txt is a file that includes directives instructing these robots on what should be excluded from their visits. This file has to be named exactly 'robots.txt' and has to be placed in your root directory (no other place or folder). If, for example your web site address is www.mydomain.com, then the web address to this file should be:

http://www.mydomain.com/robots.txt

Configuring the robots.txt file

The syntax used in this file is simple and uses two different keywords:

User-agent:
Disallow:

The keyword 'User-agent' is used to specify the name of a robot (e.g. User-agent: Googlebot). The keyword 'Disallow' is used to provide the name of a file or folder that the search engine spider should not crawl.

The following robots.txt file was engineered to suit any of our standard store. Stores with additional custom modification may require additional changes. If your store does not have any major custom changes or additional folder then the following robots.txt file should be used. Please note that at delivery time, we already included a copy of this file with your store.

User-agent: Googlebot
Disallow: /*printable=Y*
Disallow: /*js=*
Disallow: /*print_cat=*

User-agent: *
Disallow: /cart.php
Disallow: /config.php
Disallow: /admin/
Disallow: /files/
Disallow: /register.php
Disallow: /COPYRIGHT
Disallow: /README
Disallow: /Smarty-2.6.12
Disallow: /VERSION
Disallow: /adaptive.php
Disallow: /admin_banner.php
Disallow: /adv_counter.php
Disallow: /antibot_image.php
Disallow: /auth.php
Disallow: /banner.php
Disallow: /banner_element.php
Disallow: /bonuses.php
Disallow: /change_password.php
Disallow: /check_requirements.php
Disallow: /choosing.php
Disallow: /cleanup.php
Disallow: /default_icon.gif
Disallow: /default_image.gif
Disallow: /default_logo.gif
Disallow: /download.php
Disallow: /error_message.php
Disallow: /fcategory.php
Disallow: /featured_products.php
Disallow: /giftcert.php
Disallow: /help.php
Disallow: /https.php
Disallow: /image.php
Disallow: /images/
Disallow: /include/
Disallow: /init.php
Disallow: /mail/
Disallow: /minicart.php
Disallow: /modules/
Disallow: /news.php
Disallow: /offers.php
Disallow: /order.php
Disallow: /orders.php
Disallow: /partner/
Disallow: /payment/
Disallow: /popup_edit_label.php
Disallow: /popup_image.php
Disallow: /popup_info.php
Disallow: /popup_poptions.php
Disallow: /prepare.php
Disallow: /process_order.php
Disallow: /prodconf.gif
Disallow: /products.php
Disallow: /provider/
Disallow: /recommends.php
Disallow: /referer.php
Disallow: /returns.php
Disallow: /schemes/
Disallow: /search.php
Disallow: /secure_login.php
Disallow: /send_to_friend.php
Disallow: /shipping/
Disallow: /shop_closed.html
Disallow: /skin1/
Disallow: /slabel.php
Disallow: /smarty.php
Disallow: /sql/
Disallow: /top.inc.php
Disallow: /upgrade/
Disallow: /var/
Disallow: /vote.php
Disallow: /wysiwyg.php
Disallow: /reports
Disallow: /skin1_original
Disallow: /skin1_preview

Blocking of the secure area

Because your store can also be accessed via HTTPS. Search engines robots accessing your site via both HTTP and HTTPS may think that you are providing duplicate content. To correct this issue you need to add the following lines to your basic authentication file '.htaccess' file:

RewriteEngine on
RewriteCond % on
RewriteRule ^robots\.txt$ robots-https.txt

Then you need to create a second robots.txt file but with a different name. It should be called robots-https.txt and have the following content:

User-agent: *
Disallow: /

This will actually prevent all search engines robots from doing any crawling of your site via HTTPS.

Please note that both changes, the addition to the .htaccess file and the new file robots-https.txt have already been made to your store before your store was first delivered to you.



	Support

		Support Overview
		FAQ
		HOW-TOs
		Agreements/Policies
		Sitemap
		Support Helpdesk