I’d recommend staying away from hand-rolled solutions as there is a bunch of somewhat difficult stuff that needs to be gotten just right – like how transactions are handled, how exceptions cause rollbacks, how to stop rolling back endlessly (poison messages), how to integrate with long-running workflows so that the state management boundaries line up, and more.
You will probably want some kind of durable/transactional messaging infrastructure, so not using MSMQ you’d be left with Service Broker on the Microsoft platform, or some other alternative like ActiveMQ. MSMQ has the benefit of already being installed on all Windows machines, as opposed to Service Broker which isn’t.
In terms of choosing between NServiceBus, Mass Transit, and Rhino Service Bus – this Stackoverflow answer comparing NServiceBus to MassTransit would be a good place to start..
In our 3.1 release, we’re introducing NSB Studio – a set of Visual Studio integrated modeling tools that enable you to model your system at a higher level of abstraction and have much of the configuration and initialization of NServiceBus be done for you automatically. I’d say that this really tips the scales in favor of NServiceBus.
Hope that helps.
Disclaimer: I am the author of NServiceBus.