Linux – Too many TIME_WAIT connections, getting “Cannot assign requested address”

linuxscalascalabilitysocketstcp

I have a small web application which opens a TCP socket connection, issues a command, reads the response and then closes the connection for every request to a particular REST endpoint.

I've started load testing the endpoint using Apache JMeter and am noticing that after running for some time, I start seeing errors like "Cannot assign requested address", the code opening this connection is:

def lookup(word: String): Option[String] = {
 try {
  val socket = new Socket(InetAddress.getByName("localhost"), 2222)
  val out = new PrintStream(socket.getOutputStream)
  val reader = new BufferedReader(new InputStreamReader(socket.getInputStream, "utf8"))
  out.println("lookup " + word)
  out.flush()

  var curr = reader.readLine()
  var response = ""
  while (!curr.contains("SUCC") && !curr.contains("FAIL")) {
    response += curr + "\n"
    curr = reader.readLine()
  }
  socket.close()
  curr match {
    case code if code.contains(SUCCESS_CODE) => {
      Some(response)
    }
    case _ => None
  }
 }
 catch {
   case e: Exception => println("Got an exception "+ e.getMessage); None
 }
}

When I run netstat I also see a lot of the following TIME_WAIT connection statues, which implies to me that I am running out of ports in the ephemeral space.

tcp6       0      0 localhost:54646         localhost:2222          TIME_WAIT  
tcp6       0      0 localhost:54638         localhost:2222          TIME_WAIT  
tcp6       0      0 localhost:54790         localhost:2222          TIME_WAIT  
tcp6       0      0 localhost:54882         localhost:2222          TIME_WAIT 

I am wondering what the best solution is for this issue. My current thought is that creating a connection pool where connections to this service running on port 2222 can be reused by different HTTP requests, rather than creating new requests each time. Is this a sensible way of fixing the issue and making the application scale better? It seems like a lot of overhead to introduce and definitely makes my application more complicated.

Are there any other solutions for helping this application scale and overcome this port issue that I'm not seeing? My web application is running in an Ubuntu linux VM.

Best Answer

Yes, creating a connection pool is a good solution. However, an easier solution is to have the server close connections, rather than the client. In this case, the server's sockets, rather than the client's, would end up in TIME_WAIT state, so the client would not run out of ports. On the server side, connections in TIME_WAIT state would not make the server run out of ports because they all use the same local port.

To make sure that the connection is closed by the server, you need to read from the socket (on the client) until you reach end-of-file condition. At this moment, it is safe to close the socket on the client side because the server has closed it already. Of course, you need to make sure that the server would close the socket, rather than, say, waiting for a new request.

Alternatively, if you have root access, there are some sysctl options that you can tweak:

  • net.ipv4.ip_local_port_range – range of ephemeral ports. Increase it to make more ports available for outgoing connections.
  • net.ipv4.tcp_tw_recycle – enable faster recycling of connections in TIME_WAIT state.
  • net.ipv4.tcp_tw_reuse – enable reuse of connections in TIME_WAIT state. Not recommended.

See man pages ip(7) and tcp(7) for more information.