Cast from int to long is interpreted as conversion between the two types.
Cast from object to int is interpreted as unboxing a boxed int.
It is the same syntax, but it says two different things.
In the working cases (int→long, object (boxed int)→int), the compiler knows exactly what code to produce. If boxed int→long was to work, the compiler would have to somehow figure out which conversion to use, but it doesn’t have enough information to do it.
See also this blog post from Eric Lippert.