Ask any questions

Welcome to siteexplorer.info!



Thanks for using our products and services.



Any questions please Contact Us
About "stop crawling" requests:
as we can't verify the relationship between you and the site,
we will never responds to your "stop crawling" requests,
except our crawlers didn't follow your robots.txt rules.
We strongly recommend that you use robots.txt rules to limit the variety of crawlers.


Our bots absolutely works on the robots.txt protocol.
Generally, You needn't to change anything, as if it can't find our User-agent in the site robots.txt, it will auto use the other bot's rule(such as: googlebot)
Of course, You can prevent SiteExplorer from crawling your entire site or the specific contents by specifying them in robots.txt.
Please note that by doing this, the pages of your site will not be found in the search results and in any other the search results which is provided by siteexplorer.info,
Please keep in mind, any updates to robots.txt will take a period for our system updates.
For details of setting a robots.txt:

Example 1. To prevent search engine from accessing the entire site
User-agent: *
Disallow: /
Example 2. To allow all the robot access (or create a empty file: "/robots.txt") User-agent: *
Disallow:
or
User-agent: *
Allow: /
Example 3. To stop only SiteExplorer from accessing your site User-agent: SiteExplorer
Disallow: /
Example 4. To allow SiteExplorer only to access your site User-agent: SiteExplorer
Disallow:

User-agent: *
Allow: /
Example 5. To prevent Spider from accessing particular directories
In this example, there are 3 directories with limitation on accessing are set. This means that robot will not visit this 3 directories. Please note that you are required to separate "Disallow" line for every URL. The records of "Disallow: /cgi-bin/ /tmp/" will not work.
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/
Example 6. To allow access to ULRs on specified directories User-agent: *
Allow: /cgi-bin/see
Allow: /tmp/hi
Allow: /~joe/look
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/
Example 7. Use "*" to set access to url
To disable the access to the url ended with ".htm" which located under the directory /cgi-bin/ (including its subdirectories).
User-agent: *
Disallow:
/cgi-bin/*.htm