Based on the log you provided from Nginx, it seems that the connections between your server and users are unstable or slow. Please try traceroute to your client IP address or his/her gateway from your server. Also, ping your client IP address for a long time to see the packet loss rate and response time. MTU may be another source of this problem. Test if you can reach your client with MTU=1500 (Mac: ping -D -s 1472 xx.xx.xx.xx).
BTW: If your server or client resides in China, this problem usually not your fault. GFW is known to randomly discard packets between border to intentionally make international connection quality worse.