C# – How to simulate a web browser so the website feeds me the correct HTML source

c++winforms

I'm trying to webscrape a website and it appears to be feeding me bogus HTML with the WebClient.DownloadData() method.

Is there a way for me to "fool" the website that I'm a browser of sorts?

Edit:

Adding this header still doesn't fix the issue:

Client.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");

Is there something else I can try? 🙂

Edit 2:

If it helps at all. I'm trying to download the source of a ThePirateBay Search.

This URL:
http://thepiratebay.org/search/documentary/0/7/200

As you can see, the source shows what is should, seed information for the movies etc. But when I use the DownloadData() method, I get random torrent results, nothing at all related to what I'm search for.

Best Solution

Try adding a user agent header so it thinks you are one of the major browsers (IE, FF, etc)

client.Headers.Add ("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.2; .NET CLR 1.0.3705;)");
Related Question