CodeSnip: Screen Scraping Lists
page 1 of 1
Published: 28 Oct 2003
Unedited - Community Contributed
Ever wanted to add a news, feed to you site? Here is an example of how to do just that, buy using simple screen scraping methods and a regular expression to extract news simply and easily.
by Damian Manifold
Average Rating: This article has not yet been rated.
Views (Total / Last 10 Days): 9033/ 13

There have been articles on ASPAlliance about data scraping, concerning returning an entire page or a particular element. Building on the base of the other articles we will be using the grouping constructs to easily retrieve a list of headlines from Guardian Unlimited.

Source Code
News.ascx News Control
News.ascx.vb Code behind to scrape the data
Example.aspx An Example

You will find the function to scrape the HTML functionally the same as with the other articles, but with the addition of a try-catch statement, as you can never be too cautious when using resources outside of your control.

The getNews function is where things differ. This is where all the work takes place and is what we will be concentrating on here. It is surprisingly small considering its size if you did not use grouping constructs.

  27  private function getNews() as System.Data.DataTable
  28      Dim rowNewsItem as System.Data.DataRow
  30      'create the table to be returned
  31      getNews = new System.Data.DataTable()
  32      getNews.Columns.Add("strURL")
  33      getNews.Columns.Add("strHeadline")
  34      getNews.Columns.Add("strSummary")
  36      'set up the regular expression for the news page
  37      Dim strRegex as string
  38      strRegex = _
  "<A HREF='(?<strURL>[^']+)'[\s]*?>(?<strHeadline>[^<]+)</A>[\s\w\W]*?<BR>(?<strSummary>[^<]+)<"
  39      Dim Regex as System.Text.RegularExpressions.Regex
  40      Regex = new System.Text.RegularExpressions.Regex(strRegex, _
  42      'scrape the data
  43      Dim Matches as System.Text.RegularExpressions.MatchCollection = _
  44      Dim Match as System.Text.RegularExpressions.Match
  46      'loop through all matches filling out the table as you go
  47      for Each Match in Matches
  48          rowNewsItem = getNews.NewRow()
  49          rowNewsItem("strURL") = Match.Groups("strURL").Value
  50          rowNewsItem("strHeadline") = Match.Groups("strHeadline").Value
  51          rowNewsItem("strSummary") = Match.Groups("strSummary").Value
  52          getNews.Rows.Add(rowNewsItem)
  53      Next
  54  End function
(Line Continuation Characters Above are for Display Only) Generated using CodeView

Lines 31-34 deal with the creation of a DataTable. DataTables are a useful feature of .NET they allow you to easily pass data between function, without loosing any clarity.

Take a look at line 38; you will see the strRegex String.  Now that is a monster of an expression but one hell of a powerful one.

Let us take a closer look at the construction of strRegex. The String can be broken down into four basic parts.


 <A HREF='(?[^']+)'[\s]*?>(?<strheadline>[^<]+)</A>[\s\w\W]*?<BR>(?<strsummary>[^<]+)<

These will be exactly matched against the string, helping to locate the text that you are interested in.

Character Sets

 <A HREF='(?[^']+)'[\s]*?>(?<strheadline>[^<]+)</A>[\s\w\W]*?<BR>(?<strsummary>[^<]+)<

The contents of the square brackets ([]) is the set of characters that you wish to dispose. In the above example's \s represents any white space character; \w represents any word character, and \W represents any non word character.


 <A HREF='(?[^']+)'[\s]*?>(?<strheadline>[^<]+)</A>[\s\w\W]*?<BR>(?<strsummary>[^<]+)<

The *? indicates that the preceding character is repeated multiple times; this differs from the * on its own in that the * will try to match the longest possible string where the *? matches the shortest. What is the difference, you may ask. Well let us imagine you want to get the first cell in a table. "<TD>[\s\w\W]*</TD>": this looks like it should work, but it would match is the <TD> of the first cell with the </TD> of the last.

Grouping Constructs

 <A HREF='(?[^']+)'[\s]*?>(?<strheadline>[^<]+)</A>[\s\w\W]*?<BR>(?<strsummary>[^<]+)<

Now this is where the magic happens. This enables us to extract the information we are after without all that messing about with indexOf or subString and all the validation that goes along with it. "(?[^']+)": this takes the value from the current position up to but not including the ‘ and assigns it into the strURL construct. Meaning you can now refer to the data by name.

Once you have matched the data, all that is left to do is loop though the collection and fill out the table.

So there you have it, a function to retrieve the news as a DataSet.
As an example, the DataList has been bound to a custom control, making a nice little control that can add content to any page.

Example.aspx The News Control in Action

.NET Screen Scraping in depth by Damian Manifold
Easy .NET Screen Scraping by Steven Smith
ASP.NET Data Scraping by G. Andrew Duthie
Regular Expressions Quickstart by Chris Garrett

Guardian Unlimited More News Feeds That Can Be Scraped
Regular Expression Library More on Regular Expressions
ASPFriends Discuss Regular Expressions

User Comments

No comments posted yet.

Product Spotlight
Product Spotlight 

Community Advice: ASP | SQL | XML | Regular Expressions | Windows

©Copyright 1998-2024  |  Page Processed at 2024-04-22 11:59:49 PM  AspAlliance Recent Articles RSS Feed
About ASPAlliance | Newsgroups | Advertise | Authors | Email Lists | Feedback | Link To Us | Privacy | Search