Understanding Full Text Search in SQL Server 2005
 
Published: 03 Dec 2007
Abstract
This article exposes the concept of Full text search. It also explains the architecture of full-text search indexing along with the enhancements made to it in SQL Server 2005.
by Uday Denduluri
Feedback
Average Rating: This article has not yet been rated.
Views (Total / Last 10 Days): 67670/ 167

Introduction

Full text search is the search functionality that enables a fast and easy way of searching text based data in SQL Server. Although the concept of full text search is not new in SQL Server 2005, there are considerable enhancements in performance and manageability of SQL Server 2005. In this article we will discuss the usage of full text search and the architecture of full text search in detail. We will also compare full text search with other search options "Like" and discuss the reason for the superior functionality set of full-text search.

Why Full Text Search?

Let us assume that we have huge amounts of data in the database and we are having some search functionality involved in User Interface (UI). The UI for the database is a website open to the internet so response time is crucial. Now if the administrator of the website wants to search all the customer names which have a specific pattern, then he or she queries the database system as shown in Listing 1.

Listing 1

SELECT [NAME] FROM [CUSTOMER] WHERE [NAME] LIKE '%PATTERN%'

The query in Listing 1 is fine for getting the result. But in real time this may not suffice the actual requirement. Let us see why the query in Listing 1 may not yield us actual results.

·         In case of a large amount of data, the like query works very slowly. It may take minutes to return the result. This is true especially for unstructured text data.

·         If I want to query my data on formatted binary data [format], I cannot use a Like query.

So to summarize, the need to of the hour is for a search functionality that is fast, efficient and can also be run on unstructured text data with the support of search on different data types like formatted binary. Full text search qualifies all these and in fact has many advantages.

Architecture

Let us identify different components of Full text search before going ahead with understanding the architecture.

·         The Microsoft Full-Text Engine for SQL Server (MSFTESQL)

·         The Microsoft Full-Text Engine Filter Daemon (MSFTEFD) that comprises the following:

1.    Filter

2.    Protocol handler

3.    Word breaker

MSFTESQL is a windows service that is a tightly integrated component of SQL Server 2005 which is used as a search engine. This is built on the (MSSearch) technology. MSFTESQL service is mainly involved in providing three functionalities.

·         Implementing full-text catalogs and indexes for database

·         Querying the database with words, phrases and words in close proximity [meaning of a word]

·         Managing the full-text catalogs that are stored in SQL servers

MSSearch is a process in windows system that is responsible for doing the cataloguing and indexing. This file name is mssearch.exe. This process will be used by MSFTESQL service to implement the indexing and full-text catalogs.

Implementing Full Text search

Let us discuss the steps for implementing full-text search in SQL Server 2005.

·         Adding a full text search catalog to the database – To add a full text search catalog we need to open the database node in Management studio and click on the storage node. We can see the Full text catalog in this node. We have an option for adding the full-text catalog by which we can add a new catalog.

·         Adding the Full text indexing table – We need to specify to the database that we want to implement the full-text search functionality for a table. To do this we can select the respective table and select the right click option of Full-Text Index and Define Full-Text index. This opens up the Full text indexing wizard.

1.    Using the Full text indexing wizard we select a unique index.

2.    Once we select a unique index and move to the next step, we select the columns that are eligible for full text queries. This enables the full-text search for the selected columns.

3.    We need to select the option of how can we track the changes. If automatically selected then a full population of indexing occurs. To avoid a population at the end of this wizard, select the Do Not Track Changes option, and clear the Start Full Population When Index Is Created checkbox.

4.    Once we are done with step 3 we need to map the catalog out of all the catalogs at the database level. Or we could even create a new catalog for the full text search.

5.    This is the last step of the wizard. Here we can create a table schedule and catalog schedules and schedule them with time. A table schedule can have a name, time of schedule and occurrences. 

Once we create them and click on the finish button the full-text index is created.

Running the Full Text search

Until now, we have set-up the full text search catalogues and indexes. Let us run the some queries for the full-text search. There are four types of predicates used in SQL Server 2005 for running the Full text search queries. A predicate is an expression that evaluates to TRUE, FALSE or UNKNOWN. A predicate is used in a where condition in T-SQL statement.

1.    FREETEXT

2.    FREETEXTTABLE

3.    CONTAINS

4.    CONTAINSTABLE

Let us see each one of them individually with its usage.

FREETEXT

Freetext is the simplest form of predicates in the full text search. It searches for the words that give the similar meaning and not the exact wording of the word. This kind of predicate is used where we go to query a document as a whole for a word in it. The freetext not only returns the exact wording, but also the nearest meanings attached to it. Listing 1 shows the usage of the free text. To understand listing 1, the user should have a basic understanding of T-SQL queries. Let us also see the usage of Freetext with a help of an example. Listing 2 gives an example for the usage of FREETEXT.

Listing 2

FREETEXT ( { column_name | (column_list) | * } 
          , 'freetext_string' [ , LANGUAGE language_term ] )

Listing 3

SELECT CellularPhone FROM CART WHERE FREETEXT (CellularPhone, '1111')

Listing 4

SELECT CELLULARPHONE FROM CART WHERE CELLULARPHONE LIKE '%1111%'

FREETEXTTABLE

Freetexttable returns a collection of one or more rows as a table. The columns in the rows are selected based on the string given as the predicate. Freetexttable is similar to freetext except the former can return tables and the latter one can return columns. Listing 4 shows the usage of Freetexttable.

Listing 5

FREETEXTTABLE (table , { column_name | (column_list) | * } 
          , 'freetext_string' 
     [ ,LANGUAGE language_term ] 
     [ ,top_n_by_rank ] )

CONTAINS

As the name suggests, Contains actually works by verifying if a word or phrase is contained in the columns. It can search for a word, prefix of a word, synonym of a word, word formed from another word. We need to specify the language in case the languages of the words are not of the same language. Listing 5 shows the simple usage of contains column.

Listing 6

SELECT [Name], [Address]
FROM Customer
WHERE AGE > 30
AND CONTAINS([Name], 'Uday');

CONTAINSTABLE

Contains and Containstable are similar to freetext and freetexttable. The former one returns columns and the latter one returns the table.

References

Conclusion

Full-text search allows fast and flexible indexing for keyword-based query of text data stored in a Microsoft SQL Server 2005 database. There are many enhancements made to the full-text search. In sql server 2005 it uses the MSSearch functionality. The concept of ranking is also included in this version. Ranking can give a score about the relevance of the search made. In this article we tried to expose all the basic functionalities of full-text search.



User Comments

Title: nfl jerseys cheap   
Name: NIKE NFL jerseys
Date: 2012-07-02 10:11:34 AM
Comment:
http://www.jersey2shop.com
http://www.cheapjersey2store.com
http://www.jerseycaptain.com
http://www.yourjerseyhome.com
We are professional jerseys manufacturer from china,wholesal.cheap nike nfl jerseys, mlb jerseys, nhl jerseys,nba jerseys and shoes
Cheap NFL,NBA,MLB,NHL
,heap jerseys,2012 nike nfl Jerseys,nba jersey and shorts,oklahoma city thunder jersey,official jeremy lin new york knicks jersey,NFL Jerseys Wholesale,blake griffin jersey blue,NFL jerseys For Sale online.All Our Jerseys Are Sewn On and Directly From Chinese Jerseys Factory
,Wholesale cheap jerseys,Cheap mlb jerseys,]Nike NFL Jerseys,Cheap China Wholesae,Wholesale jerseys From China,2012 nike nfl Jerseys,Jerseys From China,,2012 nike nfl Jerseys,Revolution 30 nba jerseys,jersey of nba chicago bulls direk rose ,nfl jerseys,green bay packers jerseys wholesale,Buffalo Bills nike nfl jerseys sale,good supplier soccer jerseys,cool base mlb jerseys,Revolution 30 nba jerseys,2012 stanley cup nhl jersey,
We are professional jerseys manufacturer from china,wholesal.cheap nike nfl jerseys, mlb jerseys, nhl jerseys,nba jerseys and shoes. www.yourjerseyhome.com
Title: feedback   
Name: Deepali
Date: 2012-06-18 10:56:55 AM
Comment:
really helpful
Title: Full text Catalog   
Name: Neetu
Date: 2012-04-05 3:50:28 AM
Comment:
The whole article is very helpful but I didn't understood the concept of full text catalog.
What is the use of it?
Title: Nice Feed Back   
Name: Bhinu Baba
Date: 2012-03-15 2:19:52 AM
Comment:
Good Article... Easy to understand
Title: NICE ARTICLE   
Name: Prateek Gupta
Date: 2011-09-12 5:08:35 AM
Comment:
This article is effective and easy to understand. Many thanks to author..
Title: GOOD FEED BACK   
Name: RUPESH JOSHI
Date: 2011-01-18 8:29:44 AM
Comment:
NICEEEEE, Article is Too Too GOOD

CAN I GET ARTICLE REGARDING THE MAIL TRIGGER USING DATABASE.



THANKS
Title: feed back   
Name: gaurav dutt
Date: 2010-12-24 1:45:38 AM
Comment:
article is very good. It's very simple to understand
Title: feed back   
Name: murli dhar
Date: 2010-11-10 1:26:29 AM
Comment:
too good define
Title: Feedback   
Name: dhanesh
Date: 2010-10-27 12:02:45 PM
Comment:
Very good article
Title: Table Schedule vs. Catalog Schedule   
Name: Joe
Date: 2010-08-27 2:26:45 PM
Comment:
What is the difference between a table schedule and a catalog schedule?
Title: Good Article   
Name: Boon
Date: 2010-08-24 1:30:41 AM
Comment:
Good Article... Easy to understand
Title: Most Helpfull   
Name: Manoj Kumar Nayak
Date: 2010-07-30 1:18:19 AM
Comment:
Its a very good article but if u mention directly what are the advantages and disadvantages then its better to all. Immediately one person can understand how much help full this feature in my project
Title: Work Properly   
Name: Karan Singh
Date: 2010-07-28 5:08:26 AM
Comment:
It's work fine but i need multiple field condition in where clauses if u have please send me on karan.yogis@gmail.com.
Thanks for above code.
Title: Most Helpfull   
Name: surendra saini
Date: 2010-07-28 1:25:42 AM
Comment:
this example is very help full
thanks
Title: Good job   
Name: Vijay Sharma-Akruti Software
Date: 2010-04-01 6:08:47 AM
Comment:
its so simple & eassy to map.
Thanks.
Title: Now working   
Name: Hussain
Date: 2010-03-17 9:11:16 AM
Comment:
select * from tbltest where CONTAINS ([name],'LN') not showing any result but
select * from tbltest where like 'LN' showing Can you suggest what may be the problem
Title: Example for Full Text: Try it   
Name: Vipul Prasad
Date: 2010-02-26 2:32:38 AM
Comment:
EXEC sp_fulltext_database 'enable'
EXEC sp_fulltext_catalog 'KCatalog', 'create';

EXEC sp_fulltext_table 'Table1', 'create', 'KCatalog', 'pk_Table1'

EXEC sp_fulltext_column 'Table1', 'Subject', 'add'
EXEC sp_fulltext_column 'Table1', 'Message', 'add'

EXEC sp_fulltext_table 'Table1','activate'

EXEC sp_fulltext_catalog 'KForumCatalog', 'start_full'

my Table contains

ID(P.K) Subject(nvarchar) Message(text)

1 Hello All This is test

You have to set CHANGE_TRACKING AUTO while creating fulltext catalog.

Otherwise it will not populate automatically,
and you need to RUN last EXEC for every new row updation.
Title: Mr   
Name: Krishnaraj
Date: 2010-01-26 9:37:12 AM
Comment:
Its simple and easy get the point for what we looking for...

thanks..
Title: Mr   
Name: Pawan
Date: 2010-01-08 2:15:12 AM
Comment:
good one. simple and easy to understand
Title: Understanding Full Text Search in SQL Server 2005   
Name: Jerome D
Date: 2009-12-09 8:50:20 AM
Comment:
Good article for beginners! Everything in this article works. More examples are need, these are examples of the scenarios where you use the different predicates
Title: Good article   
Name: Hiren
Date: 2009-11-18 2:08:03 AM
Comment:
thanks for this article..
this article is very easy to understand..
Title: thanking you   
Name: harsh chhajer
Date: 2009-09-05 9:10:51 AM
Comment:
thanks for this article
this article is very simple and easy to understand
Title: Some query in Fulltext   
Name: Shingala Atul
Date: 2009-05-12 4:28:45 AM
Comment:
Ineed some help if anyone done this my problem is if i want to search only on character into word through free text.
i know it search only whole word but if anyone do this thing then please replay
Title: Problem   
Name: Howard Roarke
Date: 2009-03-24 7:27:49 AM
Comment:
dint work for me.need a detailed article please....
Title: Useful   
Name: Deepthi
Date: 2009-03-24 7:25:27 AM
Comment:
useful article.....
Title: Problem   
Name: Jig
Date: 2009-03-05 7:21:38 AM
Comment:
Good article, but i have a problem, when i try to search pendant in my table it gives me records bu if i search by "dndant" then it doesnt return any record.
Title: Understanding Full Text Search in SQL Server 2005   
Name: Harpal
Date: 2009-01-14 12:16:44 AM
Comment:
Gr8 article. thanx.
Title: Understanding Full Text Search in SQL Server 2005   
Name: Sanjeev
Date: 2008-11-13 12:24:53 PM
Comment:
Good article for Beginers , Thanks for sharing .
Please explain other advanced topics soon in full text .

Thanks
Title: make it simple   
Name: Neela
Date: 2008-11-01 12:26:48 AM
Comment:
want to make catalogue for differant cat.please explain in simple language
Title: Try to explain more next time....   
Name: WebEye
Date: 2008-09-05 11:48:52 PM
Comment:
Try to explain more next time....
Title: Mr.   
Name: Hitesh
Date: 2008-07-30 12:37:47 PM
Comment:
Good Article.
Title: Usless   
Name: Programmer
Date: 2008-07-08 5:02:18 AM
Comment:
More details please
Title: Mr   
Name: Prasad
Date: 2008-05-07 2:34:49 PM
Comment:
Good Article for beginers
Title: Mr   
Name: khodor
Date: 2008-04-21 4:52:41 PM
Comment:
it is a good in general but i want more example about freetext table and containstable
Title: SQL Full Text Search - storage   
Name: Jack Kangas
Date: 2007-12-24 3:46:39 PM
Comment:
You mention that section on MSFTESQL (a windows service) - manages "the full text catalogs taht are stored within SQL Server"

I just created an index and there is now 75 meg of files in a catalog folder on the machine. What is in those .ci, .dir and .wid files?

I am asking because I am concerned about security on the words that are in the full text index. If anyone could read the words in those files (that are external to SQL Server's normal MDB, NDB, and LDB files.
Title: SQL Full Text Search   
Name: Saurabh Sharma
Date: 2007-12-09 11:52:07 PM
Comment:
Another gr8 one article.
Title: SQL Full Text Search   
Name: SAuarbh Sharma Software Engineer.
Date: 2007-12-09 11:50:47 PM
Comment:
It's Really Helpful and gr8 article. Thanx.
Title: SQL Full Text Search   
Name: Nitin Sharma Software Engineer
Date: 2007-12-03 10:53:50 PM
Comment:
Great one..!!






Community Advice: ASP | SQL | XML | Regular Expressions | Windows


©Copyright 1998-2021 ASPAlliance.com  |  Page Processed at 2021-02-27 4:33:53 AM  AspAlliance Recent Articles RSS Feed
About ASPAlliance | Newsgroups | Advertise | Authors | Email Lists | Feedback | Link To Us | Privacy | Search