How should an advisor handle an underperforming PhD student?
I will share my perspective. To give a little background, I went from being the doctoral student you described above to being a postdoc with a strong publication rate at a prestigious university. I suppose my major professor was feeling very much the way you are. I know this because I was put on probation after my first year in my phd program. Suffice to say, though, I was completely unprepared to conduct research at a PhD level, even for a first year.
I was put under a junior faculty member to RA by my major professor during my first year and its interesting that what you refer to are the same problems I had. I even produced a terrible literature review as well! I imagine I was more of a hindrance than a help. My second semester was spent doing clerical/inventory work out of the way of the rest of the lab. My major professor and the junior faculty member told me that I was unfit for PhD.
So what changed for me?
I had a senior faculty member take interest in me and take me under his wing. He was closer to retirement and I imagine had more time for mentoring. He was patient with me, took time to talk with me, and took an active interest in me. Slowly and surely I started to turn things around and ended up graduating with a strong publication record that included multiple first authored journal articles and numerous conference proceedings.
In my reflection on my phd career I realize that my core issue was facing the reality of my unpreparedness and then emotionally retreating. This created a spiral of negativity that only kept making things worse and worse. I performed a task poorly, then was chastised, then my motivation declined, then I performed a task even more poorly, then was further chastised, then my motivation declined even further, etc, etc, etc.
For me, the best thing that happened was for my supervisor and the junior faculty member to let another faculty member mentor me. I realize that the relationship I had with the junior faculty member was likely poisoned as she perceived the issues relating to my work as character deficiencies and general inadequacy of abilities.
It seems that you are in this place with your graduate student. You view their performance due to character deficiencies (laziness) and inadequate abilities. It is likely that this will color your perception of any positive progress that the student might make.
If you feel that you can not make a clean start with the student, then perhaps letting someone else mentor them might be best. It was for me.
I will strongly caution you though in making judgments about their character, potential, and abilities. The core issue might be the relationship between you and the student rather than some inherent problem with the student.
Let's go with the assumption that this student is genuinely unsuited for a PhD. Such people do exist, they do get into graduate programs, and it is not doing them, their advisor, or the field any favors to string them along and waste their time. What should the advisor do with them?
There are two critical things: Communicate, and document.
Communicate with the department chair, with the dean, with the head of the graduate program, and anyone else who is involved with administration. Explain that you are concerned about this student's progress and potential, assure them that you are going to treat the student fairly, and ask them to help you set up a program to monitor the student's progress. Keep communicating with these people during this program. Prepare and send regular updates (at least quarterly, probably more).
Document the student's progress. To do this, you'll need to communicate with the student. Set clear, objective, unambiguous targets. These targets should be achievable by a PhD student without stress or difficulty; they are not stretch goals, they are the minimum that a PhD student should be able to perform. Set multiple goals, a realistic time apart, so that over a period of say a year a minimally competent student would be able to hit each goal on schedule.
Communicate these goals with your chair, grad student advisor, etc, and make sure everyone is on board. Communicate with the student. Explain to the student (ideally in the presence of the grad student coordinator, the departmental chair, the dean, etc) that he or she must hit each of these goals or they will be removed from the PhD program. This goes in writing, signed by you and the student and probably some or all administrators, and you store the document where you can refer to it repeatedly.
Each goal has a target date. Document whether the student did or did not achieve that goal. It will be a temptation to say that the student almost made it, or to decide that the goal was too hard and move the goalposts down. Don't do this. You set these goals so they're the minimum a PhD student should do; if the student doesn't completely achieve each goal, they are not a minimally acceptable PhD student.
If they don't reach every one of the goals, then it's up to the administration. The traditional solution at this point is to offer the student the choice of "Mastering out" (taking a MSc instead of a PhD and moving on), dropping out, or finding another advisor; the latter usually comes with warnings that they are on thin ice if they do.
Is this a lot of work? It sure is, and it should be; this is a major life decision for everyone concerned, and it shouldn't be the easy way out. But there are times when the best way forward for a student is to get out of a program, and there are ways to help them to that conclusion.
First, I think you were abused if you were forced to take on a student other than by mutual agreement. I feel bad for your situation. But you gave some level of buy in to it, even if under duress. This gives you responsibilities to the student that are no different than if your relationship had been more traditionally arranged. He is a person, not an artifact.
I think your responsibilities are clear. You must teach him what he needs to know. No professor can ever expect that a student is so well trained/educated in the past that nothing is expected of themselves and that all will be well. If that were true there would be no need for you in the first place.
Teach him how to do data collection. Teach him how to do a lit review. Don't complain about him or compare him to others. He isn't them. His background is different. If it is deficient that isn't your fault, of course, but someone has failed him and you have taken on the responsibility to mentor him.
If you are incapable of that then you need to find a way to sever the relationship in an equitable way. I don't know if that is possible here.
Note that this isn't a rant. I've been in similar situations with unprepared and (seemingly) unmotivated students. It was a royal pain but through a lot of work and coaching, the students I have in mind ended up at the very top of their class. But my office hours were always occupied by these students. At first they don't understand anything and require constant repetition. By the end, they are explaining things to me - correctly, it turns out. Easy? NO. Required. Well, that is the job description of a professor, really.
I'm sorry for your situation, but you need to either fulfill the responsibilities or find a way out. Letting the student naturally fail shouldn't be one of the options.
I include this addendum due to new information from the OP. While I still believe that the solution I stated above is the correct one (teach the student what he needs to know), there are ethical issues for the University itself as well as for its faculty. In fact, my view is that "the University" IS "the faculty".
It is not unethical for a university to accept unprepared students. It may be commendable, in fact. However if it also puts its head in the sand and refuses to see what it is doing and account for it then it is acting unethically. However, if it also wants to organize itself so that such unprepared students are given the specific help they need to be successful, then I congratulate them. Otherwise, however, they are acting unethically.
In the particular case, it seems that the student was admitted with no compensating help. Everything was simply thrown on the student. Sink or Swim, not my problem. Of course, such a student can't do a literature search effectively the first time. Neither can they swim two laps the first time they are pushed into the pool.
However, it isn't completely the responsibility (or shouldn't be) of the OP, here. I think it is equally unethical for a university (and especially a full professor) to push problematic/marginal students off on junior (read inexperienced) faculty. Such faculty typically don't know how to effectively handle such cases since they are, themselves in the pool for the first time.
So, either the university need to uphold stated standards or must find a way to equitably compensate when they don't. And it needs to be equitable for all concerned, both students and (junior) faculty.
But, to be doubly clear, the university and its faculty doesn't owe the candidate a degree. But it does, IMO, owe him all the support he needs. And that doesn't include lowering standards of performance or writing their papers for them. Support and encouragement.
As to advice to the OP, if at all possible, I hope you are looking for a better situation, in which you can more fairly do your own job and work more gradually into the advising game.