If your community has a job to do, then you should remove as many members as possible.
First, getting something done in knowledge-based tasks requires as few people as possible. Extra people don’t help, they burden. They make communication more difficult. They add to the frustration and reduce the consensus amongst the group.
Second, people are participating for fun not profit. If they don’t feel involved, which they wont as the community grows larger, then they wont be involved. You can’t have 60 people around a 10 man table discussing a project. So remove the people that can’t fit in and the whole group will benefit by being part of something slimmer, closer and more exclusive.
If you run a community that wants to achieve something, then remove as many members as possible.