There is pam_ssh_agent_auth, which does exactly what you need. This package is available for both Fedora and RHEL so the process of setting up and installing is very straight-forward:
yum install pam_ssh_agent_auth
Add to your /etc/sudoers:
Defaults env_keep += \"SSH_AUTH_SOCK\"
Put your ssh-public key to /etc/security/authorized_keys (get it from yubikey for example using ssh-keygen -D /usr/lib64/pkcs11/opensc-pkcs11.so)
Add a line to the start of /etc/pam.d/sudo:
auth sufficient pam_ssh_agent_auth.so
Then you just add the pkcs11 library to your ssh-agent and you can run sudo without password (authenticating using key on the token):
ssh-add -s /usr/lib64/pkcs11/opensc-pkcs11.so
sudo -i
This process is also described in the manual page for pam_ssh_agent_auth.
Note that the keys on the yubikey needs to be generated before, but this is described in the Yubico documentation already.