Asking for a deposit.

After reading Julian's excellent How to write a web design contract thread, I decided that when I do go into business for myself, I should ask for a deposit before work begins as outlined by Julian.
However, after seeing the details of the Peter Avey situation (see the whole story), I started thinking that maybe it's not such a good idea. For instance, if I begin a design but for one reason or another are unable to complete it, is the client within their rights to ask for a refund? Or, if halfway through the job the client decides to go with someone else do I have to refund them then, or am I entitled to keep the money because whether or not the site was finished I did do work on it?
My main concern in all this is if I do take the deposit and something like this does happen but I don't refund them, I don't want to be labelled a scammer simply because I want to be paid for the work I've already done.

What are your thoughts on the issue?