I found one of the servers was throwing up a 500 error quite regularly. I checked the logs and in /var/log/nginx/error.log I found the following errors:
socket() failed (24: Too many open files) while connecting to upstream, client
socket() failed (23: Too many open files in system) while connecting to upstream, client
Solution was to increase the number of worker connections and edit a few files to allow more open files on the file system.
// edit /etc/nginx/nginx.conf
worker_connections 8024;
//add line to /etc/sysctl.conf
fs.file-max = 120000
//add two lines to /etc/security/limits.conf
* soft nofile 30000
* hard nofile 60000
//then run:
sysctl -p
//restart nginx for good measure
/etc/init.d/nginx restart