What to do if most of my students have switched to a better lecturer's class?

You should discuss this with your department chair (the person who runs your department). While the teaching chair (or graduate/undergraduate chair) might seem more relevant, it is the chair of the department who you have to keep happy. Their opinion matters more than ours.

Describe the dwindling number of students as a concern, then see where the conversation goes. Instead of offering your suggestion, invite your chair to offer their thoughts and suggestions first, before you launch into yours. Who knows? Perhaps they will suggest closing down your section on their own and you get what you want. Their response probably also depends on the attractiveness of your research alternative versus their need to find instructors to cover sections they've already committed to offering.

Your chair may also be glad you invited the conversation, because they may have some feedback based on their own observations and conversations with your colleagues and students they would like to share with you. They may be able to tell you how you could get better so you could one day be the instructor who draws away everyone else's students.

But I agree with and upvoted DBB's answer, that trying to dump the class could make you look bad and that if you care about getting good at teaching, you should simply "push through" and deliver the lectures.

I am sympathetic as to why you feel this way based on the loss of students. But I want to answer your question briefly and honestly. The worst possible implications of taking this step is you could look to your department chair like a (a) quitter (b) person who is not interested in teaching (c) bad teacher (d) bad colleague (e) all of the above... Seems like you also were not put in a position to succeed, matched up against a beloved senior instructor.

In my experience as a student, I left classes when I felt the material was over my head and the teacher did not care about it. The smaller class sizes seem like a good opportunity for you to improve as a teacher which will be invaluable in an academic job search or tenure push. If you want to stay in academia I advise you to push through this type of adversity and improve.

The answer to what you should do is "Keeping plugging away with the students you have." There are some side issues here, and I think they are important. First, your colleague is misbehaving. On the surface, it seems noble for him to double his class size for no compensation, and perhaps no immediate damage is done. But what happens when the dean notices that the class size can be doubled at no cost? 1. Probably half the lecturers will no longer be needed. Bye-bye. 2. The remaining lecturers have their class size doubled just because ONE lecturer was OK with it. If I were the chairman, I would have a clear policy against "course hopping" and encourage the faculty to take steps to reduce it. Assuming your chairman is sensible, I would think that the most likely implication from your meeting with him is that your colleague gets his leash yanked. Most chairmen know that it's vital to keep sections even.

Sometimes I teach two sections of the same course which has a TA-led discussion section attached. Sometimes one TA will be experienced and personable while the other is a first-year grad student who barely squeaked by the TOEFL. I absolutely forbid the students from one section attending the other. It's not fair to the popular TA, as his reward for being good, to have more students and have his office hours overrun by extra students. I help enforce this by requiring one quiz per week in discussion, so that the students attend the correct section at least once a week. (And this is a bit of free-but-worth-every-penny advice: Make attending your class worth some small amount of points. It will bring some students back. And it will help you measure how big the gap is between you and your colleague. If a 1% attendance grade brings back most students, then the gap is not that great.)

Second: A lecturer just has to be Dory (the fish in Finding Nemo.) No matter what happens (you fart real loud in class, you give the whole lecture with your pants unzipped, you accidentally offend all the women in class and don't know why, you give a test and the average is 12%, a student with emotional problems has a major meltdown in class, etc.) you "just keep teaching...just keep teaching...just keep teaching...