I probably wouldn't....Sure it's money but is it really worth the annoyance of having to talk to the person you hate?
Maybe I'm just weird - for example, a couple of weeks ago I decided not to go for a job I could have gotten, purely because the guy in charge was a dickhead at the interview. I knew I would have had a completely miserable time working with him, so I decided not to go for it. The same applies for my design work I think.