Lab Notes

I love Susan Bradley's Windows SBS weblog, which is why I've made frequent links to it from this blog. On this post, however, I have to disagree with her. She suggests that if you don't want your website (or portions of your website) to show up in Google, you need to create a robots.txt file.

In fact, what you need to do is implement some form of security for your website. Google and other well-behaved search engines may honor your robots.txt file, but others will not. The robots exclusion file provides some obscurity, but no real security. With IIS, you can set up directory security using the Internet Information Server configuration application, for Apache, using .htaccess files. Requiring a username and password will keep everyone (well, with the usual caveats about no security system being foolproof) from getting to your "under the radar" website.