It's up to you.
Upvotes and downvotes are anonymous to protect the voter. It's fine to explain your votes if you want to, but we shouldn't have a policy stating that it's necessary. I'd hesitate even to encourage it, because that could look like an unofficial requirement to comment, defeating the anonymity of the vote.
My point here is that if someone downvotes a post and leaves no comment, then the voter has done nothing wrong. Comments accompanying votes are fine and can be helpful, but we should not set up a culture here that makes people feel like they have to choose between commenting on their votes or refraining from voting. Anonymous votes, including anonymous downvotes, are always acceptable.
Upvotes
Leaving an explanation of your vote is a nice gesture, but it's not always necessary. We don't need 25 people all leaving comments that say, "this is a good question"; the upvotes make that clear enough already.
Downvotes
Many people on the site will graciously accept constructive criticism, but there will always be a few people who will take a negative comment the wrong way and reply rudely. I haven't seen that happen on this site yet (thankfully), but judging from other sites like Stack Overflow, and considering human nature, leaving any sort of negative comment exposes the commenter to risk of a backlash.
Related reading on Meta Stack Overflow: Require a comment explaining the reason for the first downvote on a question
Close votes
For close votes, if you're picking one of the pre-defined close reasons, it may be clear enough already why you voted that way. For a custom off-topic close reason, the system automatically posts a comment stating your close reason.