I notice from the given configuration that you are running infront of an AWS ELB load balancer v2 and I am guessing that v2e points directly to an app server (which would otherwise be behind ELB)?
If so this will suggest to me that, along with the 503 error, the connection between your HAProxy instance and ELB is hitting a timeout, either the 4 second contimeout timeout or the 300 second server timeout. The more likely one is the 4 second contimeout and the sporadicalness of the error further confirms that it is likely to be a network issue between HAProxy and ELB.
I would attempt increasing the contimeout value as well as monitoring latency between HAProxy and ELB.