Difference between a Message Broker and an ESB
The difference between a Message Broker and an ESB (Enterprise Service Bus) is mainly the word 'bus'.
To me, a Message Broker is one (usally big) process that transforms data from one structure to another structure or modifies content.
An ESB is a message oriented middleware (MOM) plus additional services, one of which could be a Message Broker. So an ESB can include a Message Broker as one of it's components. A Bus consists of more than one processes, otherwise I wouldn't call it a 'bus'. The nature of a bus is that there are multiple components serving different tasks, each one communicating over a MOM and adhering to some form of 'common data format'. A bus would consist of: applications sending data to the MOM, database adapters, Message Brokers, MOM bridges, etc.
The separation is a bit gradual, but the biggest difference between a Message Broker architecture and a Bus is that of granularity. If your task is to integrate applications A, B, .., Z and a couple of databases, you can do this with one big Message Broker connecting each and everyone. Or with an ESB where multiple small components take over just little tasks. For example one adapter connects to A, another one to B (but they don't do transformation), then each one sends their stuff to one (or more) Message Broker, each of which should be kept as simple as possible - e.g. not having to know about the data model of 'A' or 'B'. A good ESB should have a common data definition on the bus, abstracting from the 'differentness' of individual applications.
TRANSFORMATION: an ESB doesn't help with transformation, unless it comes with a Message Broker. But each good ESB should include a Message Broker anyway. The Message Broker should be your bus's expert for transformations, but nothing else.
HORIZONTAL scaling: if you only have 3 things to connect (now and forever), it's probably not worth the effort to get a full-blown ESB. A Message Broker has the advantage of being just one big process. You can configure everything in there and have a central location for all your data mappings, filtering and routing.
But if you have 30 applications to connect, one Message Broker would probably come to grinding halt. Of course you can buy more instances, run things redundant, etc. but you should change your strategy to 'localize' jobs. Each application's adapter (could be one little Message Broker instance each) should be able to generate and/or receive an abstracted common data model (e.g XML with a shared XSD). There could also be a central Message Broker for transformation tasks, but that instance should be unaware of data model A or B. So an ESB should move the processing to the expert component instead of keeping everything in a central place.
Disclaimer: I am an IBM consultant and specialise in WebSphere ESB. This comment isn't left in any official capacity.
An ESB is more of an architectural pattern or concept than a product - broadly, a service-based way of engineering loose coupling. Its definition is fought over and not exactly set in stone. In general, an ESB is set of unrelated (in a technical sense) services - they expose interfaces, and they consume them from other services. Generally there isn't a hub and spoke architecture involved, although there can be.
IBM certainly markets both WebSphere Message Broker and WebSphere ESB as products that make it easy to build an ESB (along with the DataPower hardware appliance). They have different technological roots, but have some overlap in purpose. Also, that's not to say you can't build an ESB with lots of other things that aren't branded as an 'ESB product'.
That doesn't answer all your questions, but hopefully addresses the IBM part.
I just read this article by Udi Dahan a few days ago, which might give you a more clear view of what I feel is one fundamental difference.
http://www.udidahan.com/2011/03/24/bus-and-broker-pubsub-differences
Quoting:
The rule that there can only be a single publisher for a given event type is one of the things that differentiates buses from brokers, though both obviously allow you to have multiple subscribers
...
Unfortunately, there are many broker-style technologies out there that are being marketed under the banner of the Enterprise Service Bus. While some products have the ability to be deployed in both a centralized and distributed fashion (sometimes called “federated” or “embedded” mode), many do not enforce the “single publishing endpoint per event-type” rule.
Without this constraint, it is just too easy to make mistakes.
Hope it helps.
You can use a transformation broker without a service bus, and vice versa. In terms of specific products I don't think any one is purely one or the other because of the way each complements the other. Some products are stronger in the one area, other stronger in another. Perhaps a choice needs to be made based on which function best covers an individual problem.
A broker may have better built-in "lego blocks" for constructing a transformation chain than an ESB product does. A broker pressed into service as an ESB may be crushed under load and not scale well, or may lack robust journaling and tools for dealing with journals.
Some ESBs allow database updates to be rolled back and queues to be replayed into a corrected application once an egregious error in logic has been uncovered and fixed. I don't think most brokers integrate that level of transactional support. For this to work at all your "transactions" almost have to be business events (a sale, a renewal, a change of ownership, etc.) rather than something like RPCish "database updates."