How to foster gender diversity as an organizer
Aimed at a different point in the pipeline, you can try to make it more possible for speakers to accept your invitation. One step that can make a huge difference for certain people (in many cases, converting a 100% impossibility into an acceptance) is to provide resources for childcare at or near the conference.
For some ideas about specific steps you can take, Matilde Lalin recently wrote a nice overview of different ways conferences can and do support attendees traveling with children (published on Terry Tao's blog).
One that's not mentioned there is to allow attendees to purchase a hotel room in the conference-reserved block for a nanny. I know that the American Institute of Mathematics does this, and it can make the difference if the only other option would put the nanny somewhere far from the conference.
Depending on the size of your conference, this needn't necessarily mean that you subsidize or even organize the childcare; it can help even just to put attendees in touch with local daycares/nannies/babysitters, or provide other information. However unless your conference is very small, I hope you'll give thought to implementing some of the other steps that Lalin outlines.
Added February 2015:
Greg Martin has written an excellent paper "Addressing the underrepresentation of women in mathematics conferences" that addresses exactly this question, which is simultaneously scholarly and actionable. In particular, in Section 4 (pp 17-21) he gives thirty-eight concrete suggestions that organizers can follow. Not all of them will be applicable in a given situation (including that of the original question here), but many will be.
This idea is based purely on my own experience:
As a young researcher, cost is a reasonably significant factor in deciding whether to attend a conference. In some settings the cost can be reduced if you can find someone to share a hotel room with. Finding such a person though can be difficult, particularly if there is no public list of who is attending the conference and your gender is under-represented. I think it might be helpful if the registration form had an option along the lines of 'I would like to be put in touch with people who might be interested in sharing a room' (with appropriate follow-through, of course).
One helpful thing is to find lists of women in the relevant field to look through to find potential participants to invite to speak. For example, in Number Theory you can look at the list of participants in the Women in Number Theory conferences. Not every field has such easy sources of lists, but there are a lot of them out there, and it can help to find speakers whose research interests you but who you hadn't met before or otherwise didn't come immediately to mind. Similarly, social networks being what they are, having women co-organizers of the conference can also be quite helpful in brainstorming great women speakers who you might not have thought of on your own.