User-agent: * Crawl-delay: 60 Disallow: /browser Disallow: /browser/ Disallow: /log Disallow: /log/ Disallow: /blog Disallow: /blog/ Disallow: /wiki/NewsArchive Disallow: /report Disallow: /report/ Disallow: /query Disallow: /query/ # make sure that googlebot obeys these rules User-agent: Googlebot Crawl-delay: 60 Disallow: /browser Disallow: /browser/ Disallow: /log Disallow: /log/ Disallow: /blog Disallow: /blog/ Disallow: /wiki/NewsArchive Disallow: /report Disallow: /report/ Disallow: /query Disallow: /query/ # no interesting images available User-agent: Googlebot-Image Disallow: / # wget in its recursive mode is a frequent problem. Please # read the man-page and use it properly. User-agent: wget Disallow: / # trouble-some bots / tools designed to suck the complete page # they might or might not obey robots.txt... User-agent: HTTrack Disallow: / User-agent: WebReaper Disallow: / User-agent: WebWalker Disallow: / User-agent: WebFetcher Disallow: / # stolen from madwifi.org and modified (pjf, 15.04.08)