Virus scanner killing Team Foundation Server performance

The last couple of weeks, we’ve been experiencing occasional lock ups of our TFS Server, sometimes once or twice a day, sometimes it doesn’t occur for a couple of days at all. Whenever it occurs, response times for certain requests skyrocket and eventually everything comes to a halt. We’re not quite sure yet what triggers this but while looking for a probable cause we noticed some strange behavior of the McAfee process. When logged on to the server, we could see McAfee taking up to 55% CPU power. After some digging around we found out that our operations department deploys standard settings for McAfee to all servers. They don’t really bother looking at the behavior of applications running on these servers nor do they monitor for extensive CPU usage by the scan software.

Because we wanted to find out what was causing this to happen, we decided to deploy the SysInternals FileMon for Windows tool to our server and configure it to filter out McAfee related file access. To make sure there would be enough traffic on the server while monitoring it, we chose our biggest project (+/- 400Mb in source control) and forced a get operation of all files. We couldn’t believe our own eyes when we saw that almost everything the TFS server was doing, went through the anti virus software. McAfee was scanning:

• Temporary ASP.Net files
• SQL Server Databases
• SQL Server Analysis Server files
• TFS Cache of Source Control files
• Windows Sharepoint Services templates
• Machine keys

No wonder this thing was taking that much CPU, it was scanning 400Mb worth of source code on the fly. For the extent of the download, our server was running at 100% CPU usage! It was very clear that when we wanted to gain back our server’s full horse power we had to get rid of McAfee’s eager scanning habits.

Unfortunately we cannot change the McAfee settings ourselves so we had to file a Change Request in order to get if reconfigured. It was at that point we slowly began to understand why they just scan everything on each and every server. Getting approval for excluding paths or file extensions is not an easy undertaking. They were asking for all kinds of stuff like ‘Who has access to these folders?’, ‘Which rights do they have?’, ‘What’s the performance gain you expect to get?’, ‘Did you measure this’ … Result of it is that we are still awaiting approval to exclude all TFS related locations from scanning. L

I will post an update as soon as operations changed the settings and let you know how it worked out. In the mean time we’ll be checking our build and proxy servers to see if we can optimize these ones too. Luckily for us we configured our continuous integration builds to do Incremental Builds, otherwise our TFS Server would have been hitting the 100% mark continuously.