Remove toolbar from product list via local.xml

And you won't be able to remove it without overriding something. Here is how the getToolbarBlock() method looks like:

public function getToolbarBlock()
{
    if ($blockName = $this->getToolbarBlockName()) {
        if ($block = $this->getLayout()->getBlock($blockName)) {
            return $block;
        }
    }
    $block = $this->getLayout()->createBlock($this->_defaultToolbarBlock, microtime());
    return $block;
} 

This means that if a block with a certain name (value returned bygetToolbarBlockName) exists in the layout then that block will be returned. otherwise a new block is created with type catalog/product_list_toolbar and that is returned by the method.

[EDIT] I just had a crazy idea. Why not change the toolbar block type? That way it won't be rendered as a toolbar. I haven't tried it but I think it's worth it.
Something like:

<reference name="product_list">
    <block type="core/template" name="product_list_toolbar" />
</reference>

I mean adding a block with the same name but a different type. If it doesn't work please don't donwvote; it is just in idea :)

In your local.xml file add the following:

<catalog_category_default>
    <!-- Remove Toolbar by setting a blank template -->
    <reference name="product_list_toolbar">
        <action method="setTemplate"><template /></action>
    </reference>
</catalog_category_default>

It seems you can't remove this or unset this via XML due to some weird way the devs built the thing. However you can create a blank template and point the toolbars to this blank template instead.

<catalog_category_default>
    <reference name="product_list_toolbar">
        <action method="setTemplate">
            <template>theme/package/blank.phtml</template>
        </action>
    </reference>
</catalog_category_default>
<catalog_category_layered>
    <reference name="product_list_toolbar">
        <action method="setTemplate">
            <template>theme/package/blank.phtml</template>
        </action>
    </reference>
</catalog_category_layered>
<catalogsearch_result_index>
    <reference name="product_list_toolbar">
        <action method="setTemplate">
            <template>theme/package/blank.phtml</template>
        </action>
    </reference>
</catalogsearch_result_index>

That will remove it from Normal categories, Anchored categories and the search page.