source: trunk/monsterz/web/htdocs/robots.txt @ 362

Last change on this file since 362 was 362, checked in by sam, 10 years ago

Fix logo and favicon location.

  • Property svn:keywords set to Id
File size: 1016 bytes
1# Do not crawl CVS and .svn directories (they are 403 Forbidden anyway)
2User-agent: *
3Disallow: CVS
4Disallow: .svn
6# "This robot collects content from the Internet for the sole purpose of
7# helping educational institutions prevent plagiarism. [...] we compare
8# student papers against the content we find on the Internet to see if we
9# can find similarities." (
10#  --> fuck off.
11User-Agent: TurnitinBot
12Disallow: /
14# "NameProtect engages in crawling activity in search of a wide range of
15# brand and other intellectual property violations that may be of interest
16# to our clients." (
17#  --> fuck off.
18User-Agent: NPBot
19Disallow: /
21# "iThenticate® is a new service we have developed to combat the piracy
22# of intellectual property and ensure the originality of written work for
23# publishers, non-profit agencies, corporations, and newspapers."
24# (
25#  --> fuck off.
26User-Agent: SlySearch
27Disallow: /
Note: See TracBrowser for help on using the repository browser.