How can I increase the max selected units in Starcraft single player?

No.

There is no legitimate (ethically acceptable) way of changing the game to select more than 12 units.

Mainly because mods in Brood War aren't like Minecraft mods, it doesn't just make the game more fun, it potentially (in this case, definitely) gives you an unfair advantage over your opponent.

As a long time Brood War player, I entreat you to avoid using these methods and enjoy the game's imperfect beauty.

---

But this may quench your thirst if you're not picky about experiencing the pure, true Brood War.

Starcraft 2 - Brood War Remake

It seems that there's a tool called Oblivion that raises the limit to 255 units; should be enough given the supply limits.

It also introduces other sensible changes (like automatic mining), but I'm not sure if those can be enabled/disabled individually.