Does it make any difference whether I declare my <observers> inside <global>, <frontend> , <default>, <admin>, <adminthtml>?

If you call observer between <frontend></frontend> then that observer event will only be executed on the frontend.if you write between <adminhtml></adminhtml> it will only be executed in the backend and if you write between <global></global> then the observer event will work on both frontend and backend.

for example take one event called "core_block_abstract_prepare_layout_after"

Case 1:-

<adminhtml>
    <events>
      <core_block_abstract_prepare_layout_after>
      </core_block_abstract_prepare_layout_after>
    </events>
</adminhtml>

Observer event "<core_block_abstract_prepare_layout_after>" will only work in the backend

Case 2:-

<frontend>
    <events>
      <core_block_abstract_prepare_layout_after>
      </core_block_abstract_prepare_layout_after>
    </events>
</frontend>

Observer event "<core_block_abstract_prepare_layout_after>" will only work in the frontend

Case 3:-

<global>
    <events>
      <core_block_abstract_prepare_layout_after>
      </core_block_abstract_prepare_layout_after>
    </events>
</global>

Observer event "<core_block_abstract_prepare_layout_after>" will only work in the frontend and backend

Magento has three "areas" by default: frontend, adminhtml and install, which are used to separate several things like translations and designs.

The area is loaded and set once when Magento is initialized, see Mage_Core_Model_App_Area::load() for details. You will notice an additional area admin, but it seems like this is not used (anywhere anymore?). Some modules might add new areas, for example test by EcomDev_PHPUnit.

Observers are also defined per area, but additionally you can define then in <global> so that they apply in all areas.

Only observers defined in the current area and in global will be executed. Global observers always come first.

Defining an observer in <default> or <admin> actually should not work at all.