We're running a web app on Tomcat 6 and Apache mod_proxy 2.2.3. Seeing a lot of 502 errors like this:
Bad Gateway!
The proxy server received an invalid response from an upstream server.The proxy server could not handle the request GET /the/page.do.
Reason: Error reading from remote server
If you think this is a server error, please contact the webmaster.
Error 502
Tomcat has plenty of threads, so it's not thread-constrained. We're pushing 2400 users via JMeter against the app. All the boxes are sitting inside our firewall on a fast unloaded network, so there shouldn't be any network problems.
Anyone have any suggestions for things to look at or try? We're heading to tcpdump next.
UPDATE 10/21/08: Still haven't figured this out. Seeing only a very small number of these under load. The answers below haven't provided any magical answers…yet. 🙂
Best Answer
Just to add some specific settings, I had a similar setup (with Apache 2.0.63 reverse proxying onto Tomcat 5.0.27).
For certain URLs the Tomcat server could take perhaps 20 minutes to return a page.
I ended up modifying the following settings in the Apache configuration file to prevent it from timing out with its proxy operation (with a large over-spill factor in case Tomcat took longer to return a page):
Some backgound
ProxyTimeout alone wasn't enough. Looking at the documentation for Timeout I'm guessing (I'm not sure) that this is because while Apache is waiting for a response from Tomcat, there is no traffic flowing between Apache and the Browser (or whatever http client) - and so Apache closes down the connection to the browser.
I found that if I left the Timeout setting at its default (300 seconds), then if the proxied request to Tomcat took longer than 300 seconds to get a response the browser would display a "502 Proxy Error" page. I believe this message is generated by Apache, in the knowledge that it's acting as a reverse proxy, before it closes down the connection to the browser (this is my current understanding - it may be flawed).
The proxy error page says:
...which suggests that it's the ProxyTimeout setting that's too short, while investigation shows that Apache's Timeout setting (timeout between Apache and the client) that also influences this.