Do user moderated communities work?
Sometimes, but not often.
Members might flag an item of spam. They're far more likely to complain to you about it. Worse still, that might just flag stuff they don't like (rather than what's bad for the community).
This isn't to say that user moderated communities don't work, It's to say that it's not just a technology problem. It's a social problem. It takes time to build the structures through which a community can moderate itself.
I'd begin by restricting which members that can self-moderate the community. Only allow those with 100 posts or a great reputation score to be able to flag bad stuff. Make it a privilege that members have to earn. Remove it for members that don't use it.
You need to put the structures in place for a user-moderated community to succeed. Knowing who, what, and how to flag is just one side of the problem. Motivating them to do it is another.
Once again the main challenge isn't enabling the technology, but motivating the participants to use it.