Hosting ASP.NET 1.1 and 2.0 applications on an IIS Server is not really a very difficult job in and of itself. But keeping the servers serving all those pages all the time, over multiple time zones, to millions of users worldwide can have quite a lot of its own kind of challenges.

In this article and some of the others which will follow soon, I will try to keep my focus on some basic knowledge of debugging tools and techniques. The reason why you should make yourself aware of these things beforehand (assuming everything is working fine on your server right now) is because this will save you some precious time in case your production website is down.

Without much ado, let me tell you a few problem descriptions (mostly dealt by the web administrators).

Problem 1: I have a few ASP.NET websites that work just fine for most of the time, but I do not know what happens at some sporadic intervals when things break and my clients start seeing Server Application Unavailable error. Sometimes, it gets fixed on its own and at other times I have to reset my IIS Server or reboot my server to get everything back to normal.

Problem 2: Everything was working fine until yesterday night when after rebooting, our IIS server has started behaving erratically, throwing all these weird error messages every few hours. I am not even sure what changed and I have no clue of how to fix this.

Problem 3: My w3wp.exe stars with low CPU but somehow over the course of time it starts to hover at 100% and eventually my server becomes pathetically slow. I restart the box or reset my IIS to fix the issue, but somehow it re-appears.

Problem 4: I have a lot of ASP.NET websites and a few classic ASP websites on the same web server. Whenever the load is high on the server during X AM - Y PM it responds very slowly and sometimes does not respond at all. We reset IIS and it works well for some time, but it then hangs again. The CPU stays low and I am not sure if it is IIS causing the issue or there is a bottleneck elsewhere.

Problem 5: Recently I have deployed a couple of ASP.NET applications and I have noticed that it sucks the memory pretty soon and gradually moves up to about 800 MB to 1GB and then my users start seeing System.OutOfMemory exceptions until I reset my IIS or reboot my box.

Do these problem descriptions sound familiar? These and many other similar problems (sometimes multiple issues at the same time) make the job of a web administrator really challenging. As you can easily figure out from the above problems, there is a common resolution to most of these - reset IIS or reboot the server or something on a similar ground. As you would agree, none of these resolutions are REAL solutions. They are just workarounds and act as just a simple band-aid to a chronic issue which is good enough to give sleepless nights to quite a few concerned. In this article (and some more to come), I will dissect the problems above and more in a more practical way using some debugging tools and techniques which is called Post-Production debugging. This simply means to debug issues that are very difficult to reproduce (and quite often not even anticipated) in the development scenario where we do not have as much load on the server as we do in production.

Please keep in mind that I am not talking about setting a breakpoint in the Visual Studio IDE and finding those small and pesky bugs. Here we are looking at the general picture of the server with problems (or sometimes, just to baseline the things so that you can keep yourself away from the problems!!) and using the tools which you might like to keep on any IIS server. For me, keeping some of the following tools installed on any IIS server is like having an ambulance or at least a good first aid kit at my disposal. No, I am not asking you to keep 20 different diagnostic services installed and running on your server. All I am telling you is to be prepared, paranoid and beware!

Believe me, when your server is in a bad shape, the first thing you would hate to do is to install some of the tools from the internet, especially if it has become a critical issue to you.

Okay, I am done with the preaching part. Let me start with the actual stuff now… the Tools (some you already have on your server, while some you need to download and install). In this article I will discuss the 4 most important tools which, in my humble opinion, are a must have for any server. I will build on this article in more detail in the forthcoming articles. Stay tuned!

In short, we call this PerfMon. To start it, you can go to Control Panel -> Administrative Tools and double click Performance. Or, you can simply click on Start -> Run, type Perfmon and hit the Enter key. Before I proceed further, let me tell you that Perfmon does take a few processor cycles and disk space, since it logs a lot of details based on the counters you have and the frequency with which you are logging. The idea here is not to let your Perfmon run for ever and ever and waste your CPU cycles along with disk space, but just to ensure that you have the counters added. Then, if there is any need of data, you can just start/stop logging and do not miss the counters in a spur of the moment.

NB: I am using Windows XP SP 2 to create this article and all the snapshots are from the same machine.

How to create a Perfmon log?

* Start your Performance Monitor.

* Expand Performance Logs and Alerts.

* Right click on Counter Logs and select New Log Settings.

* Give a meaningful name and click on OK (I gave a name PerformanceCheckup).

Figure 1

         

You can change the Current Log File’s location, but the interface will not allow you to do it unless you add a counter.

Click on Add Objects and select the following from the dialog box which comes up (you can use Shift and CTRL keys here to select multiple items).

Process
Processor
Memory
.NET CLR Data
.NET CLR Exceptions
.NET CLR Interop
.NET CLR Jit
.NET CLR Loading
.NET CLR LocksAndThreads
.NET CLR Memory
.NET CLR Networking
.NET CLR Remoting
.NET CLR Security
ASP.NET
ASP.NET Applications
ASP.NET State Service

If you have classic ASP application, you may like to add Active Server Pages and Web Service counters as well.

After you have added the counters, click on Close button.

At this time you will notice that the other textboxes are no longer grayed out. You can specify a time interval to sample the data. By default, the interval is 15 seconds and is good for most purposes. But let us say the issue is happening every 3 days. In that case, it will not make much sense to log data every 15 seconds for all the counters. Probably 60 or 90 seconds will be a better choice.  Hopefully, you get the idea. The thumb rule is that too little data can make the Perfmon logs meaningless and too much data can make it a little more difficult to analyze, but more data cannot hurt you as much as less data, since less data is almost like no data!

Now, right click on your Perfmon Log file and choose Start. In the screenshot below, you will find the PerformanceCheckup log file as green, which means that now the tool is gathering data every N seconds you have just configured in the last step.

Also, notice in the screenshot above that the name of the Perfmon log file is C:\PerfLogs\PerformanceCheckup_000001.blg. As soon as you stop and restart it, the Log File Name will change automatically to PerformanceCheckup_000002. Therefore, your existing log files are not touched and you may archive it to any storage medium.

Figure 2

Now that the counters are already added, you NEED NOT keep the logging started all the time since it will just keep taking space on your hard-disk. You may turn it on whenever you plan to fix any performance related issues on your server. The Perfmon logs will give you a decent picture of what is happening on your server. It is just a matter of getting yourself acquainted with different counters and what they mean.

Here are a couple of fantastic article from Microsoft:
Performance Counters for ASP.NET
Memory Performance Counters

This tool is one of the smartest tools around and comes with an extensive help file. It has a very intuitive GUI and can assist you in troubleshooting hang, crash, deadlock, slow performance or memory leak and fragmentation. It contains a lot of nifty, ready-made scripts which you can use to diagnose your memory dump with.

So… what is a memory dump?

A memory dump is a kind of snapshot of a process and contains useful data at the time you created that snapshot. It can tell you a lot of things, like which pages are being executed by the process, how many of them are waiting, what are the kinds of objects allocated, etc.  Say you are facing any of Problems 3, 4 or 5 stated in the beginning of the article. You can create memory dumps at times when you see the bad performance and get it automatically analyzed by Debug Diagnostic tool. In fact, even if you are seeing Crashes (Problem 1 and 2), you can use this tool to find out what is crashing your process and fix the issue, which could be very difficult to fix if you do not have a memory dump. In fact, the problem is how you will actually figure out if you are seeing crash related issues on your server. I have blogged about it in the past and hopefully that post should help you to some extent.

How can you create a memory dump for any process using Debug Diagnostic tool?

Start your debug diagnostic tool.

Click on the Cancel button on the wizard that you get by default.

Click on the Tools menu, select Create IIS/COM+ Hang Dump and choose Yes on the message box that pops up.

Figure 3

         

It will create the dumps and will show an appropriate message box to you.

Figure 4

          

If you look at the folder mentioned above, you will see two .DMP files (in your case it could be more if you have a lot of application pools) and those are your Memory dumps.

If you want to take dump of a specific process, Debug Diagnostic can help you there as well. Just switch to the Processes tab and you will see the list of all the processes.

Figure 5

    

Right click on any process, select Create Full User dump and you are done!

It is not a good practice to take one dump, analyze it and tell anyone… that it is okay. Here is the problem. Normally, it does not happen that way and has a good chance of misleading you. If you want to be sure of the issue, you should take multiple dumps about 3-4 minutes apart (if you are seeing slow performance) and analyze them to see if you find the same issue in all of them.

Learning to analyze memory dumps has a steep learning curve and although Debug Diagnostic will do the basic analysis for you, it might not be the PERFECT analysis simply because it is based on some scripts. It is a very good tool, but is by no means the only tool you need to have to fix all your performance issues!!! That said, you cannot write it off, since it is good enough to guide you and sometimes it is bang on target. Go through the help file. It is pretty neat and littered with appropriate pictures throughout.

How can you analyze a memory dump using Debug Diagnostic tool?

Once again, the help file is pretty detailed. Navigate to the following location and read through Advanced Analysis.

Figure 6

 

NOTE: While collecting the dumps, ensure that Perfmon has been logging the data for a while. Basically, a good practice is to ensure that Perfmon logging is enabled and you take at least 3 dumps about 3-4 minutes apart and THEN stop the Perfmon logging.

In my opinion, this tool is The One which gives you answers when Debug Diagnostic tool fails. The downside is that this tool has a very steep learning curve and the GUI is not one for the faint hearted. I do not want to scare you by saying all this, but unfortunately this is true. Debugging a memory dump using WinDBG is difficult, but there are a lot of experts doing just that at Microsoft and have a great level of expertise in analyzing the dump files and troubleshooting IIS/ASP.NET and all sorts of other issues. If you see that you have already collected a set of dumps along with the Perfmon data you can call Microsoft and ask them to help you troubleshoot your issue.

Microsoft help and support
Support Options from Microsoft Services

In case you are of a curious type, I would suggest just 3 links which can help you. http://mtaulty.com/communityserver/blogs/mike_taultys_blog/archive/2004/08/03/4656.aspx
http://www.codeproject.com/debug/windbg_part1.asp
http://blogs.msdn.com/dougste/articles/clrdebug.aspx

In the forthcoming articles, I will discuss more about WinDBG and some very specific issues which come up.

I have written a detailed article on Log Parser for Microsoft Support voice column and you can read it from this link. This is a long article but is worth the time you spend to know Log parser. Ensure that you download this tool and keep it handy on your web server!

Making and keeping yourself prepared for any eventuality cannot hurt. In this part I have just touched upon a few must-have tools which I would advice you to install on your IIS server and make yourself aware of how to use these tools. Please remember that this is my personal recommendation based on my personal experience. Going forward, if you see any issues on your web server and when you seek help from someone, you can tell them how you have approached the issue so far and how you have already captured the data (they will appreciate that you have already captured it). The chances are that you will not have to wait for the issue to happen again since you already have the correct memory dumps with you. Hence, the resolution could be much faster. Also, quite often you will see that dumps are like onions. You reveal one problem and after fixing it, another problem might surface up. I wish there was a silver bullet to fix all the issues with one dump collection. Unfortunately, there is not. There will be times when you will have to repeat the dump collection followed with analysis multiple times, but then it is not like you are shooting in the dark.

Happy troubleshooting!