The temptation for communities with low levels of activity is to grow.
This is usually a mistake. If you can't engage the members you have, why assume you can engage the members you don't have?
Right now, one of three things is happening: (1) Your community concept is wrong (2) You're attracting the wrong sort of members (3) Your tactics to convert newcomers into regulars is flawed.
Growth isn't the solution to any of these. If your community is wrong you need to start again and redefine what your community is about (tip: Look for that strong common interest).
If you're attracting the wrong sort of members, you need to rethink your approach to recruitment. Are you targeting specific people for a specific reason?
If your engagement approach isn't working, you need to find tactics that do engage members before attracting more.
Only once you can convert your existing members into active members, can you justify growth. You need to learn how to engage members before you grow the community, not after. You should have some proven tactics to get members active. Everything else is a waste of time and your precious first impression.
All the successful communities today; every single one was highly active in the early stages. The key to growth is high activity. Get the activity right early and growth will take care of itself.