User-agent: * Disallow: / Disallow: /dos/ Disallow: /images Disallow: /.asp Disallow: /.html Disallow: /.htm