Please see this questions and the related questions. We have to study the HTML source of a webpage before we can scrap it properly. So login manually and get the source of the inbox page and then study it to scrape it.
Why dont you use yahoo's webmail API? Which is a better solution.
There are several ways to perform HTTP GET
and POST
requests:
Method A: HttpClient (Preferred)
Available in: .NET Framework 4.5+
, .NET Standard 1.1+
, .NET Core 1.0+
.
It is currently the preferred approach, and is asynchronous and high performance. Use the built-in version in most cases, but for very old platforms there is a NuGet package.
using System.Net.Http;
Setup
It is recommended to instantiate one HttpClient
for your application's lifetime and share it unless you have a specific reason not to.
private static readonly HttpClient client = new HttpClient();
See HttpClientFactory
for a dependency injection solution.
POST
var values = new Dictionary<string, string>
{
{ "thing1", "hello" },
{ "thing2", "world" }
};
var content = new FormUrlEncodedContent(values);
var response = await client.PostAsync("http://www.example.com/recepticle.aspx", content);
var responseString = await response.Content.ReadAsStringAsync();
GET
var responseString = await client.GetStringAsync("http://www.example.com/recepticle.aspx");
Method B: Third-Party Libraries
RestSharp
POST
var client = new RestClient("http://example.com");
// client.Authenticator = new HttpBasicAuthenticator(username, password);
var request = new RestRequest("resource/{id}");
request.AddParameter("thing1", "Hello");
request.AddParameter("thing2", "world");
request.AddHeader("header", "value");
request.AddFile("file", path);
var response = client.Post(request);
var content = response.Content; // Raw content as string
var response2 = client.Post<Person>(request);
var name = response2.Data.Name;
Flurl.Http
It is a newer library sporting a fluent API, testing helpers, uses HttpClient under the hood, and is portable. It is available via NuGet.
using Flurl.Http;
POST
var responseString = await "http://www.example.com/recepticle.aspx"
.PostUrlEncodedAsync(new { thing1 = "hello", thing2 = "world" })
.ReceiveString();
GET
var responseString = await "http://www.example.com/recepticle.aspx"
.GetStringAsync();
Method C: HttpWebRequest (not recommended for new work)
Available in: .NET Framework 1.1+
, .NET Standard 2.0+
, .NET Core 1.0+
. In .NET Core, it is mostly for compatibility -- it wraps HttpClient
, is less performant, and won't get new features.
using System.Net;
using System.Text; // For class Encoding
using System.IO; // For StreamReader
POST
var request = (HttpWebRequest)WebRequest.Create("http://www.example.com/recepticle.aspx");
var postData = "thing1=" + Uri.EscapeDataString("hello");
postData += "&thing2=" + Uri.EscapeDataString("world");
var data = Encoding.ASCII.GetBytes(postData);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.ContentLength = data.Length;
using (var stream = request.GetRequestStream())
{
stream.Write(data, 0, data.Length);
}
var response = (HttpWebResponse)request.GetResponse();
var responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
GET
var request = (HttpWebRequest)WebRequest.Create("http://www.example.com/recepticle.aspx");
var response = (HttpWebResponse)request.GetResponse();
var responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
Method D: WebClient (Not recommended for new work)
This is a wrapper around HttpWebRequest
. Compare with HttpClient
.
Available in: .NET Framework 1.1+
, NET Standard 2.0+
, .NET Core 2.0+
using System.Net;
using System.Collections.Specialized;
POST
using (var client = new WebClient())
{
var values = new NameValueCollection();
values["thing1"] = "hello";
values["thing2"] = "world";
var response = client.UploadValues("http://www.example.com/recepticle.aspx", values);
var responseString = Encoding.Default.GetString(response);
}
GET
using (var client = new WebClient())
{
var responseString = client.DownloadString("http://www.example.com/recepticle.aspx");
}
Best Answer
Definitely go with PHP Simple HTML DOM Parser. It's fast, easy and super flexible. It basically sticks an entire HTML page in an object then you can access any element from that object.
Like the example of the official site, to get all links on the main Google page: