How can I determine whether a student has written an excellent paper themselves, or hired someone else to do it?

Are there any techniques I could use to determine whether an essay that I suspect was written by someone else, but not published elsewhere, is a student's own work?

Ask the student to come to your office hours or a private meeting.

Then, say: "I was really impressed by your recent paper, it was an excellent piece of work! Let's discuss it some more."

Student who did the work will be able to discuss said work intelligently.

Student who didn't do the work is unlikely to be able to carry on an intelligent conversation about said work.

(This still isn't proof of any kind, and you can't accuse student of cheating without some evidence - but if you're lucky, student will at least be spooked enough to never do it again.)

Are there any steps I can take to ensure I don't receive questionable essays in the first place?

Assign essays that are a few steps beyond, but still closely tied to, what's been discussed in class, such that somebody who wasn't in the class will not be able to produce the kind of essay you're expecting (the effectiveness of this depends very much on the subject matter).

First, begin any essay assignment with clear instructions on the amount of outside assistance that is allowed.

Many universities have "drop in writing tutors" who are paid (albeit often rather modestly) to help students with their writing assignments. These tutors are well-trained on how not to help too much: e.g. they probably do not write down as much as a complete sentence that the student has not already suggested.

Having students expressly acknowledge the outside help that they get, even if absolutely legitimate, helps towards fostering good academic practices.

Thus if they go to the university's writing center or some similar entity, that's fine (assuming that above it has been clarified by the instructor to be fine; but if your university has a writing center then it seems like a bad idea not to allow your students to use it in at least some ways), but they should still acknowledge it. This goes a long way to combating the problem of "mysterious diction", i.e., the isolated turn of phrase that you just don't quite believe came from the student.

You should really enforce this practice: if it turns out that a student went to the writing center and didn't report it, lower their grade at least slightly: this helps to reiterate that this is very important to you. In fact:

Communicate absolutely clearly that in academia there is no sin worse than passing off someone's ideas and/or work as your own.

I think that most undergraduates know that this is something they are not supposed to do, but many don't seem to understand that this is a very serious crime. To me at least, the crime is so serious that -- in light of the difficulties of definitively "catching" the student -- the penalties should be clarified in advance to be quite severe. At my university, the absolute minimum penalty for this is a grade of 0 on the assignment. I would say though that a student who paid someone else to write a paper for them and is caught deserves to fail the entire course. Make this clear.

Structure a written assignment more than just asking for a full-blown paper at the end.

This has the merit of being a good practice for other reasons. Even in mathematics courses I sometimes ask students to write papers, and I have learned from hard experience that if you don't check in with them while they are doing the work then the final quality of the product is going to be ridiculously spotty: e.g. it will turn out that some students simply didn't understand what you were asking them to do to the extent that you do not recognize what they turn in as being a specimen (however poor) of what you asked for.

For a significant paper in an undergraduate course it could be appropriate to meet with the student several times before they turn in the final version: once to determine and have you sign off on an appropriate topic, and at least once to show you their partial work, giving you a chance to adjust them back on track if need be.

If you break the assignment up into pieces like this, then it becomes at the same time harder for a student to pay someone else to do it and also easier for them to do it themselves. I have to imagine that a lot of this professional paper-writing is done when a student realizes that they have waited until the last 24-72 hours before the paper is due. Someone can write a good paper under those severe time constraints, but they can't.

Consider having a classroom component accompany any significant written work.

This works better in some courses than others, depending on both the size of the course and the subject matter. Hearing 25 different presentations on whether and in what way Madame Bovary was a proto-feminist sounds like a rather dreary use of class time. But let me say that in every graduate (math) course I teach, whether I assign written homework or not I always have a "problem session", usually about once a week and lasting at least an hour, in which the students present their solutions to me and to each other (with whatever written notes they want, of course). I don't do this because I'm worried that their written work will not be their own, not really, but it is definitely a different thing to cobble together a solution from two different texts and a website (this is a totally legitimate thing to do, in my courses: in fact this sounds like a reasonable description of some of my own research!) than to actually present it in front of other people. (Again, I've learned this from hard experience: in many of the courses I teach, the course text is my own notes, sometimes written on the fly and sometimes already written. It is remarkable to me how I can give a better or worse rendition of something that I looked up from my own lecture notes on any given day. Sometimes I don't succeed in getting the material as firmly in my head as I would like in the time allotted to prepare that day's lecture.)

If there is both an oral and a written component, then layering is again a good thing: maybe the student turns in some preliminary written thing, then they give a presentation on it, then they improve their written work based on the feedback they got from the presentation. Again, this seems more appropriate in some courses than others, but when it is appropriate the students learn a lot: perhaps they will even be too busy learning and working to think to get someone else to write their paper.

I wish these random thoughts can supplement the other good answers and comments. Mainly, I wish to address the second question: "Are there any steps I can take to ensure I don't receive questionable essays in the first place?"

Don't start off with negative thoughts/suspicion so soon: Some introverted people can perfectly blend into the background in class, but produce very thoughtful written work because they actually put good amount of thought into the piece. It's like you have never noticed this person's existence until you have read the student's written work. For this reason, I'd suggest taking a more trusting approach.

Set up a code of honor: In some professional degrees students often have to sign a code of honor. You may check with the department head if something like this can be set up for all the students. You can also create your own and have the students read, discuss, question, and sign on the first day of the class. A code of honor will help clarifying what is and is not expected in the course, and will help to i) remind students to be honest, and ii) serve as a foundation contract in case someone really does cheat.

I do believe there are students with malicious intention, trying the cheat their way through. However, I also believe there are some of them who did it because "Well, the lecturer never said I can't." It is saddening that nowadays teachers have to lay out all these criteria, but I also would see this chance as bringing the students, who may be from diverse backgrounds, onto the same page regarding ethics and professionalism.

Downplay the grade proportion of the paper: The incentive to hire a ghost writer is likely proportional to the percentage of the final grade attributed to that piece. Try lower it, and introduce some other forms of assessment that are less likely to be done by someone else: In-class exercise, individual/group presentation, etc. are some possibilities.

Similarly, give smaller and/or more frequent writing assignments: In questionnaire design, if one asks more questions on the same construct, the reliability tends to go up. Basing on this idea, you may consider giving more than one written assignment. This can let you i) detect within-student fluctuation (How did this person suddenly write so well?) and ii) perhaps allow students to express their thoughts or work on somehow different topics/from a different angle. Higher frequency may also introduce less incentive to hire a ghost writer.

Implement a progressive assignment to foster the sense of ownership: You may also consider modifying your assignment so that students will need to work on it and report to you step by step, from concept to review, from review to analysis, from analysis to draft, and from draft to final version. Spread them out across the course. This method will allow you to first get a sense of their thought process (which is a lot more objective than checking their in-class behaviors), and it will also enhance the student's sense of owning the piece as well as understanding of the contents. This can likely decrease the need or urgency to seek ghost writers.

Be very clear what is and is not allowed: I'd just put this in as a food for thought. We may often encourage people, especially those with English as second language, to either let native speakers look at their work or seek professional editors' help to improve their written piece. Is this allowed? Sometimes a run of very simple edit can make the piece quite eloquent, and some editors may either go to town in their editing or help the student to flush out their ideas more clearly during the face-to-face meeting. Is this allowed? I am not pro or against this practice of getting an editor's help; I just wish to point out that there can be much grey area when judging if no clear rules are laid out.