As I said before, you can kick out people that you don't think are right for the community.
Don't be shy about doing it.
You don't have to tolerate the gradual degradation in quality or focus.
You can quickly remove the members that aren't a good fit.
It sounds harsh, but it's best for the community.