It is exactly as you say: The whole concept of public key authentication is that the private key should only be known to the owner, while the corresponding public key can be widely disseminated. The security of your authentication depends on the security of the private key, not of the security of the public key.
The fact that somebody else provides you with a private key automatically makes it compromised. (You don’t know if that other admin still has a copy which can be used to impersonate you.)