The scale of the robots problem largely depends on the type
of website as well as the type of content it offers. The following pointers
are consistent with robot activity.
Large numbers of requests from a single IP address or a range of
IP addresses within the same subnet (i.e. the first three numbers of the IP
address are identical).
Large numbers of requests for database driven content compared to
the rest of the website.
Many requests made from browsers that do not support ASP
Lots of and increasing numbers of website visitors, but no
corresponding increase in transactions (e.g. sales!).
Large numbers of spam or automated requests being generated from
Although some of these indicators will be identified by
website server statistical analysis packages, it is often necessary to manually
look at the log files in a text editor or use a specialized log reporting tool
such as Microsoft's Log