# I had set up a separate Slurp section because Yahoo!'s crawler was ignoring the main one. # March 2009: I'm commenting these out for now to see if things have improved. # Next step is to use .htaccess files with # Header set X-Robots-Tag NOARCHIVE # #User-agent: Slurp #Disallow: /~mike/ #Disallow: /~muddy/mypictures/ #Disallow: /~muddy/toronto/ #Disallow: /pkpn/ #Disallow: /tmp/ User-agent: * Disallow: /*?C= Disallow: /~mike/ Disallow: /~muddy/mypictures/ Disallow: /~muddy/toronto/ Disallow: /pkpn/ Disallow: /tmp/