For starters, everyone behind a common NAT gateway will be authorized to send mail as the client that initiated the POP before SMTP, as it just means “IP x.x.x.x has sucessfully authorized, they may send mail now” and all clients behind the NAT will appear with the same IP.
Potentially, this might mean 1000s of completely random, unreleated users can use your SMTP server in case of modern carrier grade NAT networks, e.g. in cell phone networks.
This alone and the fact that every relevant mail client nowadays is able to do SMTP auth should be enough to convince you to turn it off.