source: trunk/monsterz/web/htdocs/robots.txt @ 362

Last change on this file since 362 was 362, checked in by sam, 10 years ago

Fix logo and favicon location.

  • Property svn:keywords set to Id
File size: 1016 bytes
Line 
1# Do not crawl CVS and .svn directories (they are 403 Forbidden anyway)
2User-agent: *
3Disallow: CVS
4Disallow: .svn
5
6# "This robot collects content from the Internet for the sole purpose of
7# helping educational institutions prevent plagiarism. [...] we compare
8# student papers against the content we find on the Internet to see if we
9# can find similarities." (http://www.turnitin.com/robot/crawlerinfo.html)
10#  --> fuck off.
11User-Agent: TurnitinBot
12Disallow: /
13
14# "NameProtect engages in crawling activity in search of a wide range of
15# brand and other intellectual property violations that may be of interest
16# to our clients." (http://www.nameprotect.com/botinfo.html)
17#  --> fuck off.
18User-Agent: NPBot
19Disallow: /
20
21# "iThenticate® is a new service we have developed to combat the piracy
22# of intellectual property and ensure the originality of written work for
23# publishers, non-profit agencies, corporations, and newspapers."
24# (http://www.slysearch.com/)
25#  --> fuck off.
26User-Agent: SlySearch
27Disallow: /
28
Note: See TracBrowser for help on using the repository browser.