Why do we need a bootloader separate from our application program in microcontrollers?
A bootloader on a microcontroller is responsible for updating the main firmware over a communication channel other than the programming header. This is useful for updating firmware in the field over BLE, UART, I2C, SD cards, USB, etc. It would be extremely inconvenient to require customers to purchase programmers just to update the firmware on their devices.
The reason why the bootloader is kept separate is for reliability. The bootloader and application code are placed in separate sections of flash, so that the application code can be erased and re-written by the bootloader without changing anything related to the bootloader code.
If the bootloader and application were kept together, then the bootloader code would need to be copied to RAM before it could run, since any firmware update would erase the bootloader code in flash. If power were cut with the bootloader code in RAM and the flash erased, the device would be bricked.
So that the loading process can recover from errors. Suppose there is a communication error or power disconnects during an upgrade. If the boot loader were part of the application you were upgrading then the user wouldn't be able to try again without using special hardware to reflash to boot loader.
Some microcontrollers can't execute code from RAM. If the boot loader was mixed in with the rest of the software then you wouldn't actually be able to upgrade your software because you can't erase pages of flash that you are currently executing out of. The work around is to first burn the new code to the second half of flash, then jump to it. The new code then copies itself to the first half of flash. Of course the downside is that burning flash is usually slow and now that you have to do it twice the loading process might take up to twice as long. Also this work-around limits your application size to be no larger than half of your total flash.
Well written boot loaders try to verify that valid code exists on the device before trying to execute it. If the boot loader and other code were mixed together then how could you be sure that your validation routine would work if all the code didn't load?
Authentication. Secure boot loaders try to verify that the loaded application matches a digital signature before executing. But if the boot loader and other code were mixed together then you can't control what runs on the device because once the user loads new code you can't control what happens at startup.
They're generally there to allow you to update your main application program.
You need some code which knows how to erase and reprogram some of the internal flash, that can't be the main program as when it's erased itself it wouldn't be able to reprogram.