Build an ASP Search Engine using Website Utility
 
Published: 04 Nov 2004
Unedited - Community Contributed
Abstract
This article explains how to create a server-side ASP search engine for a website using Website Utility.
by Brett Burridge
Feedback
Average Rating: 
Views (Total / Last 10 Days): 23490/ 50

Why Websites need Search Facilities

Once a website grows beyond a couple of dozen pages then it can sometimes be difficult to create a site navigation scheme that allows users to quickly find exactly what they're looking for. One way to improve site navigation is to add a search facility to the website. Adding a search facility brings major benefits to a website, making it easier for users to find information as well as adding an additional method of navigating a website. Search facilities are generally well used, and will often appear within the top ten most requested pages on a website.

Search Engines Allow Visitors to Search your Content

One of the easiest ways to add a facility for searching the pages in your website is to link to search results for your website from one of the major search engines. Google and other major search engines allow you to do this. However, using this method it can be difficult to integrate the search results with the design of your website. It also carries the obvious risk of a website visitor leaving your website and not returning! Even worse, your website visitors may see an advert for a competitor on the search results page, and so go and do their business elsewhere!

Building Your Own Search Engine

There are a number of software solutions that allow you to put your own search engine on your website. These include server-side search solutions available such as Microsoft's Index Server or ht://Dig. Although they allow sophisticated search facilities to be created, they generally require a high level of technical knowledge to install and configure. Another complication is that your chosen Internet Service Provider or hosting company may not support the search software required, or you may not be able to install additional software onto the server hosting the website.

A useful alternative is to use a server-side solution that only makes use of ASP, and does not require any other components to be installed onto a web server.

The Website Utility Builds ASP Search Engines

The Website Utility is a Microsoft Windows application that is able to create a server-side ASP search facility for a website. The resulting search facility requires no 3rd party components, databases or search software. It is also able to create client-side JavaScript search facilities for smaller websites.

The walkthrough below shows the steps involved in creating an ASP based search facility using The Website Utility:

Configuring The Website Utility to Produce ASP Search Engines

The Website Utility is configured using a small Windows application. There is a Create ASP/JavaScript Search Facility checkbox in the Report Settings part of the window that needs to be ticked in order for the ASP search facility to be created:

A screenshot of The Website Utility's graphical user interface, showing the options used to create a JavaScript Search Engine for a website

Note that if your website uses query strings then it is a good idea to tick the checkbox under Web Robot Settings called Use URL Query Strings. This will ensure that in the search results pages with different query strings will be treated as different search results pages. So for example www.mywebsite.net/news.php?ID=12 will link to a different news article from www.mywebsite.net/news.php?ID=21 and so The Website Utility will ensure they are indexed separately.

Running The Website Utility

Clicking on the Run button will start The Website Utility's web robot. This web robot start at a user specified page in the website and will automatically crawl all of the pages in that website. The Website Utility extracts all of the words from these pages, and finds the most relevant pages in the website for each word. Common English words (e.g. got, like, then) are removed, as are words of one or two characters. Word rankings depend on many factors, including their distribution through the entire website and their distribution in the content of a specific page.

Incorporating the ASP Search Facility into any Website

The Website Utility creates two ASP include files that can be used on the website's search results page:

  • A Search Data ASP include file (TWUSearchData_JScript.asp) contains the rankings for each word and the most relevant pages for that word.
  • A Search Code ASP include file (TWUSearchCode_JScript.asp) contains the code required to parse the user's search query and finds the most relevant pages for that query.

Note that the data and code files are created in JScript, but they can easily be used from within ASP pages that use VBScript.

Pages are sorted in search results according to their ranking for the particular word or words being searched for. The ranking scale goes from 0 to 99. Rank is higher for pages that most closely match the search term. In general, searching for words that are common on the site will produce search results with a lower rank than very specific words that occur on only one or two pages.

The search facility also requires a search form and a search results page. The search form can either be put on a separate search page on the site, or the search form could be added to all of the pages in a website (e.g. in the top right hand corner of the website's navigation). The HTML code for a typical search form is shown below. The search form needs a text box called TWUQuery. The form should use either the GET or POST form submission method to submit to the search results page.



ASP JScript Search for <a href="http://www.brettb.com/</title">http://www.brettb.com/</title</a>>
</head>
<body>
<h1>Search <a href="http://www.brettb.com/</h1">http://www.brettb.com/</h1</a>>
<form name="frmSearch" method="GET" action="searchresults.asp">


Search for: <input type="text" name="TWUQuery" maxlength="50">
<input type="submit" name="submitbutton" value="Submit">
</form>


</body>
</html></pre></p>
<p><font face="Verdana" size="2">The search results page (i.e. searchresults.asp) needs to include references to the two ASP include files created by The Website Utility (TWUSearchData_JScript.asp and TWUSearchCode_JScript.asp):</font></p>
<p><pre><<a href='mailto:%@Language="VBScript"%'>%@Language="VBScript"%</a>>
<script language="JScript" runat="Server">
var TWU_MaximumSearchResults = 20;
var TWU_DisplayPageTitles = true;
var TWU_DisplayPageDescriptions = true;
var TWU_DebugMode = false;
</script>
<!--#include file="TWUSearchData_JScript.asp"-->
<!--#include file="TWUSearchCode_JScript.asp"-->
<html>
<head>
<title>Search Results




Search Results for "<%Response.Write(TWU_GetOriginalSearchQuery())%>"

<%Response.Write(TWU_DisplaySearchResults(TWU_GetModifiedSearchQuery()))%>

The search results page can of course be customised to fit in with the existing design of your website. If you want to display the search terms the user was searching for, then use this ASP code (assuming VBScript is the default scripting language):

 <%Response.Write(TWU_GetOriginalSearchQuery())%>

To display the search results, place this ASP code where you want the search results to appear:

 <%Response.Write(TWU_DisplaySearchResults(TWU_GetModifiedSearchQuery()))%>

The search results page also defines four ASP JScript variables that can be used to change the output:

  • TWU_MaximumSearchResults Controls the maximum number of pages that will be listed in the search results. This stops users getting confused by seeing large numbers of pages in the search results.
  • TWU_DisplayPageTitles If set to true then the pages displayed in the search results will show their HTML titles as clickable links. If set to false then the URL is displayed instead. URLs are also shown if a page does not have a title. If the website does not contain accurate page titles you might have to turn this feature off.
  • TWU_DisplayPageDescriptions If set to true then the pages displayed in the search results will show their HTML Description Meta Tags as page descriptions. If set to false then page descriptions are not included in the search results. If the website does not contain accurate page descriptions you might want to turn this feature off. Turning off descriptions will also increase search performance.
  • TWU_DebugMode If set to true then debugging information is displayed (you should not need to use this).

Performance Issues

A server-side ASP search engine is obviously going to have a larger performance overhead compared to using a dedicated search solution such as Microsoft's Index Server or SQL Server's full text indexing capabilities. The size of the TWUSearchData_JScript.asp include file will depend on the number of pages in the website indexed, and also the amount of content on each page in the website. It is also dependent upon the nature of the website itself - websites with pages about similar subjects will tend to require a smaller file than a website with pages about different subjects.

To create a search facility for medium to large websites of 1000+ pages, or to allow the simultaneous searching of several websites at once, then Microsoft's Indexing Service is recommended. The Index Server Companion may also be used to extend the functionality of Indexing Services.

Download the Evaluation Version

The evaluation version of The Website Utility will allow you to determine whether the ASP Search Engines it creates are suitable for use on your own websites:

Sample Search Facilities

Purchase The Website Utility



User Comments

Title: Wonder Article, very helpful   
Name: Terry Pearson
Date: 2005-05-05 12:10:31 AM
Comment:
Thank you. It was very helpful to me as a beginner to understand the search engine app. Keep up the awesome work!
Title: Mr M X   
Name: Lowcrazy
Date: 2005-04-08 7:20:18 AM
Comment:
Man this one is so helpful

Product Spotlight
Product Spotlight 





Community Advice: ASP | SQL | XML | Regular Expressions | Windows


©Copyright 1998-2024 ASPAlliance.com  |  Page Processed at 2024-03-29 11:28:25 AM  AspAlliance Recent Articles RSS Feed
About ASPAlliance | Newsgroups | Advertise | Authors | Email Lists | Feedback | Link To Us | Privacy | Search