Will I miss out on anything if I don't invite the three new residents to the island?
Without moving in additional residents, you won't be able to progress the game!
That said, there is a way you can influence who shows up for the first 3 houses.
If you have previously visited islands with Nook Miles tickets, you can interact with the villagers there and encourage them to move to your island (even before Nook builds any houses!). If you go to an island and don't like the villager camping out there, you can ignore them without affecting your town.
When Nook does build the 3 house plots, any villagers you've encounter on Nook Miles islands will move in. If you haven't asked 3 villagers to move in, the houses will be populated with random villagers you haven't encountered on islands. (Though once the names are attached to the plot, there's no way to kick them out before they move in)
After the first 5, further islanders first "sample" island life by spending time on your island's campsite. With the exception of the 1st camper (as Riley mentions), no one will move-in without your suggesting it.
Chart of how all 10 villagers are determined
Villager 1 - NO CHOICE - Determined on island start
Villager 2 - NO CHOICE - Determined on island start
Villager 3 - LIMITED CHOICE - Meet on Mystery Island Tour and invite, otherwise random
Villager 4 - LIMITED CHOICE - Meet on Mystery Island Tour and invite, otherwise random
Villager 5 - LIMITED CHOICE - Meet on Mystery Island Tour and invite, otherwise random
Villager 6 - NO CHOICE - Must accept first camper
Villager 7 - FULL CHOICE - Talk at campsite / mystery island and invite to move-in (or build a house plot for random)
Villager 8 - FULL CHOICE - Talk at campsite / mystery island and invite to move-in (or build a house plot for random)
Villager 9 - FULL CHOICE - Talk at campsite / mystery island and invite to move-in (or build a house plot for random)
Villager 10 - FULL CHOICE - Talk at campsite / mystery island and invite to move-in (or build a house plot for random)
You won't be able to progress to the next stage of the game. Meaning your resident services tent will never upgrade into a building, you wont get isabelle, you wont be able to customize your town, etc.
Accept all the villagers (and future ones you will be "forced" to get) asap. You'll be able to kick out/invite who you want later.