Yeah, it's fairly straight forward to go from HTML to XHTML, and start making use of CSS for your layouts, etc.

Actually I became familiar with CSS using Dreamweaver. If you have a recent version, or can get a demo, use it for learning - contrary to the opinion of many freelance web designers/developers, it is not the forbidden tool. Of course you may need to make manual changes to the code and CSS along the way (WYSIWYG don't always structure things nicely, or use efficient code - but making manual changes and tidying things up can be part of the learning process).

Don't forget to keep checking your work with the XHTML and CSS validators at http://www.w3.org, to get an idea of whether your work is valid, and conforms to web standards (which you should aim for).