Lack of IPv6 support on your site will hurt some of your users.
According to stats published by Google 20-25% of users currently have IPv6. A large fraction of those users will need to go through some kind of NAT to reach IPv4-only services, which will make connectivity less reliable.
The fraction of users without any IPv4 connectivity whatsoever is negligible. But for those users who have both IPv4 and IPv6 connectivity your site will be more reliable if you support both as well. Many clients now support RFC 6555 which will allow them to automatically fail over between IPv4 and IPv6.
Also keep in mind that if a network with both IPv4 and IPv6 support has a DHCP server outage the users on that network will still be able to reach mainstream sites, but if your site is IPv4-only they will not be able to reach your site. Thus they might conclude your site is down and not realize there is a problem on their network.