How to ask someone to collaborate on a joint paper?
I'm not a mathematician, and in my field sole-authored papers are unusual, so my answer might not be entirely relevant, but for what it's worth:
You don't usually ask someone to collaborate on a paper because you are stuck. Either the collaboration arises naturally through a joint project, or you ask someone to join your paper because they have some specific expertise or experience that you think can add to what is already going to constitute a good, publishable paper. In the latter case, I have had success by sending an email briefly outlining the goals of the paper, its importance, and the timeline on which I intend to have the paper completed. I finish the email by stating what I think they could add to the paper and asking them if they are interested in a co-authorship on this basis. I might already have a draft of the paper ready, and if so, I will attach it so that they can see that I am serious and the paper has a good prospect of timely completion.
Also: a bit of flattery doesn't go astray. Refer to "your important work on..." or "your extensive experience in..." among the reasons for approaching them.
If in fact, you are not approaching someone to collaborate because of something specific that they can add to an already solid paper, but merely looking for someone to help with a work in progress, I suggest approaching someone at a junior level who already knows you, and asking for help.
How mathematicians collaborate depends (among other things) on their career stage. At the student level, the most common kind of collaboration could be more accurately described as a form of mentorship: you are working with a more senior guide, and that they mean to help you with your work and professional development is a key feature. This is different from the kind of collaboration you do (in most cases) one you get past the student / postdoc stage. At that point it is fair and only natural for someone who is considering collaborating with you to ask "What's in it for me?"
The path of most graduate students is one of decreasing collaboration of the mentorship type (in math, mentorship collaborations do not always result in the mentor being the coauthor, so e.g. it is not necessarily viewed as inconsistent for an advisor to talk about "our work" when the final product is a paper written only by the student) and increasing independence from your mentors. Your trajectory sounds a bit different in that you already have solo papers (although, as in the previous parenthetical, that may not describe the whole story). Another feature of student research -- the key feature, really -- is that whether your advisor is joining you in doing the mathematics or not, she is supposed to be involved in helping you choose problems to work on that she finds, by virtue of her greater knowledge / experience / cultural familiarity / political savvy, to be especially worthwhile and significant. A student who works on a problem for a while with no significant progress raises a little red flag in my head. One must ask: why are you not working solely on problems that either you know you are making progress on or your advisor is there to help you make progress on?
In our department there is absolutely no one interested in that subject.
Another little red flag. What you are doing is not inherently bad but seems more risky than normal. I fear you are getting less than the standard amount of "mentorship collaboration" and I think you should consider moving somewhere else (as a student, or possibly graduating ASAP and taking a postdoc) where you can benefit from that.
Anyway, there is a different kind of mathematical collaboration. (By the way, @Significance has done a good job of describing it in her answer. But I want to write as a mathematician and in some more detail.) If you want to start a collaboration with someone of this other sort, then typically this is founded on either significantly overlapping research interests or complementary skill sets, approaches and/or partial results.
Maybe while reading a draft of a paper you see how to strengthen one of the results. In this case you can (perhaps) get added as a coauthor on a later draft, and especially if you are joining more than one other author this can be a good way to piggyback onto a longer term collaboration already in progress. Here is an example of mine.
Maybe you have a draft of a paper and you see that your Theorem 2 is a variant of their Theorem 5, so joining forces is a natural choice, if only for reasons of smoothing out the publication process. Here is an example of such a collaboration of my own, in which we each had overlapping manuscripts that we combined. I was happy with the resulting paper, which is a little more than the sum of its parts, and remain on good terms with the coauthor, but that was the extent of the collaboration.
Maybe you discover that someone whom you previously did not know (e.g. a recent PhD) is working in the same area as you, or even on the same problem. Learning that someone else has been working on the same problem as you and that you each have partial results which have been obtained using (even mildly) different ideas and techniques can be immensely helpful. If each person's perspective is rich enough so that the other cannot readily incorporate it just by reading the other's work, then this is often the beginning of a very fruitful collaboration. Here is an example of mine. Or maybe you run into "someone new" and discover that you both have aspirations to work on the same thing, and though you don't have finished product, you each like the sound of the other's ideas and think it will be better to work together. This is one of the purer forms of collaboration, but it seems to require a rather fortuitous set of circumstances. Here is an example of mine.
Maybe you have a really attractive half of a result, and you're looking for someone with distinctly complementary skills and/or results to form an attractive whole. Again, this can be hard to swing, but it's satisfying when it works. Here is an example of mine. In this case, I had 2/3 of a new proof of a theorem, so I wrote a mass email to my own mathematics department asking if everyone could help me out. One of the other students in the department wrote back within a few weeks with an answer. I added a lot of other material, wrote the entire paper myself, and still feel like I really lucked out.
Maybe you have a question that's truly attractive: a question that no one else thought to ask and that you happen to know that the answer would be very interesting...but you really can't answer (or anyway, aren't trying to answer: it comes to the same!) yourself. In this case you can try asking the question as widely as you can, and if the answer comes, you can try write a paper together. This is the least likely scenario yet. It seems that a lot of amateur mathematicians think mathematical research works this way...but it usually doesn't. "Here's what I want to do, just tell me how to do it" is usually not a route to publication in mathematics. (Maybe this is what people on this site mean when they say "An idea is worthless" and I start to flip out in response. In mathematics "an idea" doesn't mean an idea for a problem; it means an idea for a solution!) I do have an example of this. In fact this originated with a question I asked on math.SE. This was quite an unusual situation. Among other things, my coauthor is not a professional mathematician or academic, so he would never have written this up for publication on his own. The journal we published in accompanies the articles by little author bios. His was: "[NJD] is a private citizen of the Netherlands." But he is a private citizen of the Netherlands who has the skills and abilities of a clever research mathematician.
I don't claim that the above list is exhaustive, but it is certainly long enough. When I look back over it, I am struck by how many highly positive collaborative experiences I have had, but also how large a role chance and serendipity played in allowing these collaborations to be realized (and it makes me wonder how many collaborations didn't come to pass because I was not luckily connected with the right person at the right time.) I hope it gives you some ideas about how non-mentorship collaborations can get started. Good luck.
The other answers are good, but I'd like to make a few other points. (Like Pete, I'm also a mathematician.)
- Some problems are just hard, and you may be ill-equipped to attack them. Sometimes is just best to put them on the back-burner. Lots of projects get abandoned, or at least put to the side, and if people didn't do this, many would be stuck forever.
- If you're a grad student/postdoc, maybe you should try to work on problems people around you are interested in.
- If you do want to still pursue this problem, try to either visit people who might have ideas and talk to them about it or invite them to come visit you. Special semesters/summer schools/conferences are also great. Then talk to them about your project, and see if they're interested or if they have any ideas. Many times, a good suggestion from an expert, rather than a full-fledged collaboration, is all you need to make a break-through.
I know the first two are non-answers to your question, and I can't tell how much energy you are putting into this particular problem, but it's good to take a step back and think about what you are thinking about doing.