|
|
.NET Screen Scraping in depth
|
by Damian Manifold
Feedback
|
Average Rating: This article has not yet been rated.
Views (Total / Last 10 Days):
94001/
84
|
|
|
|
| Introduction |
There have been articles on ASPAlliance about data scraping, today we will be looking at the different techniques. The WebRequest class is provided for accessing data via the web, it has two derived classes that will be looking at: Webclient and httpWebresponse.
Both classes are able to do anything you wish to do, it is more of a case of which to use for what job.
Here we will cover everything you would want to do with the two classes and see which comes out best.
|
| Simple scraping |
Here we are looking at just scraping a simple page. Where you want to do nothing but get back the page and do not have to pass up any data.
You will notice that at this level there are only small differences in the use of the classes, and at the moment Webclient seems to have the slight edge with it being slightly simpler code.
|
| Forms |
You have seen how simple it is to scrape any page using either webClient or httpWebResponse, today we will be looking into how you pass form data to the page you wish to scrape.
Here we are looking at passing the form data as a query.
You will notice that the difference between the two classes is now becoming apparent. So which is better Client or Request? Well since Request passes the data on the URL it is much simpler but as you can see from the example if you have many fields things may be a little unclear. Client however is much more structured in how it passes the data, and if you wanted you could always pass the data on the URL here too. So client at the moment is looking like the better solution.
|
| Posted Forms |
So you should have by now scraped a page and scraped the result of a passed form.
Here we are looking at posting the form data.
You now see that Client now passes the form as a byte encoded string and oddly now returns the page as the same. To post data using Request you still use the same byte encoded array but have to manually open a stream and write the data yourself. This does however give you a good picture of how things actually work.
So which is the best now? Well client is still the most concise, but Request is at least consistent and clearer as to what is happening. So for posting a form it seem to be a tie.
|
| Passing Headers |
Here we are looking in how to pass values in the header. Header values go unnoticed by the users but can carry important information such as the browser type.
Once you have looked at the above code you will see that both Client and Request both deal with headers in the same way. However with request not all values in the header can be set with this method, some more standard header values such as user-agent have there own property. This helps to make things clearer and when you start to deal with cookies this will be a big help
When you run the standard header example you will notice that not all header information can be set in code. Values such as Host are preset and cannot be modified.
|
| Scraping & passing cookies |
Finally are looking in how to pass values in the cookies. One of the most important things that cookies can be used for than can cause trouble when scraping is session variables.
Once again you will see that Client is much simpler than the Request method. It sets the cookie value directly in the header, where as Request uses a cookie container which may make things a little clearer but also as more powerful implications.
The best implication of using the cookie container is that if you are going to be scraping multiple sites you can keep all your cookies in the same container, the Request then only passes up the cookies with the corresponding domain.
So again it is a close call between which you wish to use. Since both are derived from the same base class performance is not much of an issue. But in conclusion if you are doing simple scraping webClient would appear to be the most convenient but can become unclear if you are passing lots of values in forms or cookies, httpWebRequest is a little more long winded but though its uses of classes is a little more clear. So the choice is yours.
|
| Scraping in one line using temporary objects |
After finishing the series on screen scraping the thought arose, how small can you make a working scrape. With this in mind the most extreme example was chosen, is a one line scrape Doable?
Surprisingly the answer is yes, and more surprisingly it is still quite clear.
Here is the example code for a conventional scrape and the one line scrape
You will notice in the conventional scrape that variables are declared for the Webclient, stream and stream reader objects, and that they are only used once before they are disposed of.
If you look at the one line example you will see that when the object is created it is no longer assigned to a variable it is just used. Placing () around the object creation allows the object to be accessed as a temporary object, since there is no other reference the object it will be disposed of after execution of the command.
|
| Other Articles |
|
|
|
|
|
|
Product Spotlight
|
|
|
