Delphi Indy IdTcpClient read operation returning truncated data for one specific request

delphidelphi-2007indy

This is an interesting problem that I’ve not been able to solve yet.

I am writing a client that communicates across the Internet to a server. I am using the TIdTcpClient Internet Direct (Indy) component in Indy 10 using RAD Studio 2007 native personality.

To get data from the server, I issue an HTTP request using SSL over port 443 where my request details are contained in the HTTP message body. So far, so good. The code works like charm, with one exception.

There is one request that I am submitting that should produce a response of about 336 KB from the server (the HTTP response header contains Content-Length: 344795). The problem is that I am getting only 320KB back. The response, which is in XML, is clearly truncated in the middle of an XML element.

For what it’s worth, the XML is simple text. There are no special characters that can account for the truncation. My TIdTcpClient component is simply reporting that, after receiving the partial response, that the server closed the connection gracefully (which is how every response is expected to be completed, even those that are not truncated, so this is not a problem).

I can make nearly identical calls to the same server where the response is also more than a few K bytes, and all of these work just fine. One request I make returns about 850 KB, another returns about 300 KB, and so on.

In short, I encounter this problem only with the one specific request. All other requests, of which there are many, receive a complete response.

I have talked to the creator of the service, and have supplied examples of my request. He reported that the request is correct. He also told me that when he issues my same request to his server that he gets a complete response.

I’m at a loss. Either the creator of the service is mistaken, and there is actually a problem with the response on that end, or there is something peculiar about my request.

Is there a solution here that I'm missing? Note that I’ve also used a number of other read mechanisms (ReadString, ReadStrings, ReadBytes, etc) and all produce the same result, a truncation of this one specific response at the 320KB mark.

The code is probably not relevant, but I’ll include it anyway. Sorry, but I cannot include the XML request, as it includes proprietary information. (ReadTimeout is set to 20 seconds, but the request returns in about 1 second, so it's not a timeout issue.)

function TClient.GetResponse(PayloadCDS: TClientDataSet): String;
var
  s: String;
begin
  try
    try
      s := GetBody(PayloadCDS);
      IdTcpClient1.Host := Host;
      IdTcpClient1.Port := StrToInt(Port);
      IdTcpClient1.ReadTimeout := ReadTimeout;
      IdTcpClient1.Connect;
      IdTcpClient1.IOHandler.LargeStream := True;
      //IdTcpClient1.IOHandler.RecvBufferSize := 2000000;
      IdTcpClient1.IOHandler.Write(s);
      Result := IdTcpClient1.IOHandler.AllData;
    except
      on E: EIdConnClosedGracefully do
      begin
         //eat the exception
      end;
      on e: Exception do
      begin
        raise;
      end;
    end;
  finally
    if IdTcpClient1.Connected then
      IdTcpClient1.Disconnect;
  end;
end;

Best Answer

Since you are sending an HTTP request, you should be using the TIdHTTP component instead of the TIdTCPClient component directly. There are a lot of details about the HTTP protocol that TIdHTTP manages for you that you would have to handle manually if you continue using TIdTCPClient directly.

If you are going to continue using TIdTCPClient directly, then you should at least stop using the TIdIOHandler.AllData() method. Extract the 'Content-Length' reply header (you can Capture() the headers into a TStringList or TIdHeaderList and then use its Values[] property) and then pass the actual reported byte count to either TIdIOHandler.ReadString() or TIdIOHandler.ReadStream(). That will help ensure that the reading I/O does not stop reading prematurely because of the server's disconnect at the end of the reply.

Related Topic