User-Agent: * Disallow: /cgi-bin/ Disallow: /trap/ Disallow: /todo/ User-Agent: LinkWalker Disallow: /cbot/mod_*html