What's the point of robots.txt?

From my limited understanding, this file is for detailing the files/pages that you don't want the search engine to index, correct? I'm just wondering why anyone wouldn't want the search engines to index their site?

Say you're a designer and you make a site for someone and keep a live preview for your portfolio. You would want to block that preview so that the search engine doesnt index the same content twice. Other reasons is for some reason you have personal information on a page and dont want the world to know about it. There's different reasons but so far I havent used it.

There is also bandwidth. You may have a big site and only a few pages that should really be of interest to the spider and you don't want all your bandwidth used up by a spider.

http://www.robotstxt.org - resource

If you create a spider your actually suppose to register it. I think few do though.

To add to the mystery, I looked at ebays robots.txt:

So apparently disney is a no touchy. I wonder why they wouldn't allow their policies to be spidered though. Maybe they don't want someone just copying the policy for their own use?

Humm, that is very weird. I can understand the policies I guess, but why Disney?

Some other things you may not want indexed by the SE's. An 'admin' area. Or, maybe a stats page that is vulnerable to 'referral spamming'.

Brett

disney was a promotion once but came up at the very top of most search enginey. I guess disney asked them to exclude it as they want to be the search result number one

Robots.txt simply specifes which folder/file should be index by search engine spiders.

I use robots.txt to prevent search engines from finding my secret content.

That is what the robots.txt is for, I use it to prevent google from addming my website images to Google Images

Does robots.txt work for all search engines?