You can use a transformation broker without a service bus, and vice versa. In terms of specific products I don’t think any one is purely one or the other because of the way each complements the other. Some products are stronger in the one area, other stronger in another. Perhaps a choice needs to be made based on which function best covers an individual problem.
A broker may have better built-in “lego blocks” for constructing a transformation chain than an ESB product does. A broker pressed into service as an ESB may be crushed under load and not scale well, or may lack robust journaling and tools for dealing with journals.
Some ESBs allow database updates to be rolled back and queues to be replayed into a corrected application once an egregious error in logic has been uncovered and fixed. I don’t think most brokers integrate that level of transactional support. For this to work at all your “transactions” almost have to be business events (a sale, a renewal, a change of ownership, etc.) rather than something like RPCish “database updates.”