Why is there no IO transformer in Haskell?

Consider the specific example of IOT Maybe. How would you write a Monad instance for that? You could start with something like this:

instance Monad (IOT Maybe) where
    return x = IOT (Just (return x))
    IOT Nothing >>= _ = IOT Nothing
    IOT (Just m) >>= k = IOT $ error "what now?"
      where m' = liftM (runIOT . k) m

Now you have m' :: IO (Maybe (IO b)), but you need something of type Maybe (IO b), where–most importantly–the choice between Just and Nothing should be determined by m'. How would that be implemented?

The answer, of course, is that it wouldn’t, because it can’t. Nor can you justify an unsafePerformIO in there, hidden behind a pure interface, because fundamentally you’re asking for a pure value–the choice of Maybe constructor–to depend on the result of something in IO. Nnnnnope, not gonna happen.

The situation is even worse in the general case, because an arbitrary (universally quantified) Monad is even more impossible to unwrap than IO is.

Incidentally, the ST transformer you mention is implemented differently from your suggested IOT. It uses the internal implementation of ST as a State-like monad using magic pixie dust special primitives provided by the compiler, and defines a StateT-like transformer based on that. IO is implemented internally as an even more magical ST, and so a hypothetical IOT could be defined in a similar way.

Not that this really changes anything, other than possibly giving you better control over the relative ordering of impure side effects caused by IOT.