How do I unlock courses in Mario Kart Wii?

Basically, you just need to get a medal (bronze, silver or gold) in single player Grand Prix mode in each cup.

Star Cup => Finish 3rd or better in Mushroom Cup and Flower Cup
Special Cup => Finish 3rd or better in Star Cup (which means that you also finished 3rd or better in Mushroom Cup and Flower Cup)

Leaf Cup => Finish 3rd or better in Shell Cup and Banana Cup
Lightning Cup => Finish 3rd or better in Leaf Cup (which means that you also finished 3rd or better in Shell Cup and Banana Cup)