Am i in the wrong?

Wow, so much wrong in so little time.

1. You priced the site at $80. That should have been the deal and YOU should have stuck to it. The extra labor should have been priced separately.

2. Because you didn't price your labor separately, it was worth the difference between the site price and the site price + labor price, which is to say: NOTHING. You devalued yourself. No one else did that.

3. Because your labor was worth nothing to begin with, you can't be upset that he wants his money back on the site if he didn't get the site. You might as well get annoyed at yourself and not the guy. You set the price of your labor by giving it away in the first place.

All that said: was the $40 given with an understanding that it was a non-refundable deposit? If no expectation was given either way, you should error on the side of angels and give him his money back. All you are out then is your time and labor, which you already valued at $0.