Disallow: /cgi-bin/ # Teleport Pro is too dumb, rude and agressive User-agent: Teleport Pro Disallow: / User-agent: MSIECrawler Disallow: / User-agent: msnbot Crawl-delay: 120 User-agent: Slurp Crawl-delay: 120 User-agent: HyperRobot Disallow: / User-agent: Teoma Crawl-delay: 120 # Voila bot is too agressive, and there is no real benefit in being listed in voila.com User-agent: VoilaBot Disallow: / # iltrovatore bot is too agressive, and there is no real benefit in being listed in iltrovatore User-agent: iltrovatore-Setaccio Disallow: /