WorkspaceMapping.xml and Continuous Integration

Here’s one that crossed my mind when writing my previous post. I was telling about the fact we configured our continuous integration build types to do Incremental Builds in order to optimize build time. While setting this up we noticed that nobody ever bothers to change the WorkspaceMapping.xml file to point to the actual source code needed to perform a build. Basically, a standard generated WorkspaceMapping.xml file points to the root of your Team Project like this:

<?xml version=”1.0″ encoding=”utf-8″?>
<SerializedWorkspace xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance&#8221; xmlns:xsd=”http://www.w3.org/2001/XMLSchema”&gt;
  <Mappings>
    <InternalMapping ServerItem=”$/MyTeamProject” LocalItem=”C:\Workspace\” Type=”Map” />
  </Mappings>
</SerializedWorkspace>

I don’t know how you guys organize your Team Project source control folder but we do it like this:

$/MyTeamProject

    +-MyTeamProject

        +-Branches

            +-Oct06

                +-Documentation

                +-Source

            +-Apr07

                +-Documentation

                +-Source

        +-Releases

        +-Trunk

            +-Documentation

            +-Source

So for us, a default setup would get all artifacts (documentation, source code, released binaries, …) from source control to the workspace located on the build server. While in fact you probably only need the ‘Source’ folder beneath the ‘Trunk’ or maybe one of the branches. This way you are eating up a lot of disk space and bandwidth.

Sample based upon our largest project in TFS:

	Download Size
Complete Project	2 Gb
Trunk Only	400 Mb
Savings	1.6 Gb

Updating the WorkspaceMapping.xml file takes care of this:

<?xml version=”1.0″ encoding=”utf-8″?>
<SerializedWorkspace xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance&#8221; xmlns:xsd=”http://www.w3.org/2001/XMLSchema”&gt;
  <Mappings>
    <InternalMapping ServerItem=”$/MyTeamProject/MyTeamProject/Trunk/Source” LocalItem=”C:\Workspace\” Type=”Map” />
  </Mappings>
</SerializedWorkspace>

A nice side effect of this is that the path everything gets downloaded to gets shorter and pointing to the solution file from the build project gets easier too.

Virus scanner killing Team Foundation Server performance

The last couple of weeks, we’ve been experiencing occasional lock ups of our TFS Server, sometimes once or twice a day, sometimes it doesn’t occur for a couple of days at all. Whenever it occurs, response times for certain requests skyrocket and eventually everything comes to a halt. We’re not quite sure yet what triggers this but while looking for a probable cause we noticed some strange behavior of the McAfee process. When logged on to the server, we could see McAfee taking up to 55% CPU power. After some digging around we found out that our operations department deploys standard settings for McAfee to all servers. They don’t really bother looking at the behavior of applications running on these servers nor do they monitor for extensive CPU usage by the scan software.

Because we wanted to find out what was causing this to happen, we decided to deploy the SysInternals FileMon for Windows tool to our server and configure it to filter out McAfee related file access. To make sure there would be enough traffic on the server while monitoring it, we chose our biggest project (+/- 400Mb in source control) and forced a get operation of all files. We couldn’t believe our own eyes when we saw that almost everything the TFS server was doing, went through the anti virus software. McAfee was scanning:

• Temporary ASP.Net files
• SQL Server Databases
• SQL Server Analysis Server files
• TFS Cache of Source Control files
• Windows Sharepoint Services templates
• Machine keys

No wonder this thing was taking that much CPU, it was scanning 400Mb worth of source code on the fly. For the extent of the download, our server was running at 100% CPU usage! It was very clear that when we wanted to gain back our server’s full horse power we had to get rid of McAfee’s eager scanning habits.

Unfortunately we cannot change the McAfee settings ourselves so we had to file a Change Request in order to get if reconfigured. It was at that point we slowly began to understand why they just scan everything on each and every server. Getting approval for excluding paths or file extensions is not an easy undertaking. They were asking for all kinds of stuff like ‘Who has access to these folders?’, ‘Which rights do they have?’, ‘What’s the performance gain you expect to get?’, ‘Did you measure this’ … Result of it is that we are still awaiting approval to exclude all TFS related locations from scanning. L

I will post an update as soon as operations changed the settings and let you know how it worked out. In the mean time we’ll be checking our build and proxy servers to see if we can optimize these ones too. Luckily for us we configured our continuous integration builds to do Incremental Builds, otherwise our TFS Server would have been hitting the 100% mark continuously.

The Big Picture

About us

We, meaning the .Net community within our company, are a very small part of the IT department of an international financial group with Belgian roots which is heavily investing in expansion towards Central Europe. The past years the group acquired banks, insurance companies, leasing companies … in several Central European countries. Since then most IT departments, formerly belonging to each acquired company itself, have been brought together under one management. We are now in the process of unifying the way we work and ultimately deliver software. As in most companies we also started outsourcing development but instead of working together with the usual suspects for off shore development, management decided to take over a company located in Chennai – India, virtually extending the local IT departments.

In an effort to standardize .Net usage for all IT departments within the group, we reorganized the Belgium based team into a service oriented structure. The picture below shows an overview of the different teams which make up our total .Net community. Bottom left you can see the .Net Service Center which basically manages methodology, toolset, framework, guidelines … This is the team I’m currently part of. The other teams, called Delivery Centers, actually take on real-life projects, each delivering software to their local customers while using the same guidelines, methodology, framework and centralized infrastructure.

Location			Team Size
Belgium	Head Office Leuven	Service Center	6
	Head Office Brussels/Leuven	Delivery Center	30
	Computer Centre Mechelen	Operations	n.a.
Czech Republic	Head Office Prague	Delivery Center	4
India	2 Office locations Chennai	Delivery Center	18

Table 1: Current Staffing .Net Teams

The infrastructure

All teams are connected to and collaborate using the infrastructure hosted by our operations department in Mechelen, Belgium. We’re currently using a single tier deployment of Team Foundation Server which is running on top of a virtualized Windows 2003 machine hosted on a Blade Server. Next to TFS we host a build server for the Service Center and one extra for each Delivery Center we add. The build servers are used for doing Continuous Integration builds and for building the MSI packages we need to deploy the software. The Load Test Environment enables load/performance/stress testing of both web and batch applications.

As you can see in the picture below every developer uses the Team Suite Edition of Visual Studio 2005. Project Leads and management can use the Visual Studio Team Explorer, Excel and Ms Project to connect to TFS …

Team members located in India access the Team Foundation Server thru a Team Foundation Proxy Server primaraly for reasons of limited bandwidth between our Computer Centre and the office in Chennai. I’ve been told bandwidth is still expensive in India .

Server		CPU	Memory	Disk Capacity
Team Foundation Server	Virtualized	Dual CPUUsing 4 cores of the Blade server	2 GB	Fixed – C: 10 GB Dynamic -D: 30 GB
Build Server	Virtualized	Single CPUUsing 2 cores of the Blade server	2 GB	Fixed – C: 10 GB Dynamic -D: 30 GB
Team Test Load Agent Server	Virtualized	Single CPUUsing 2 cores of the Blade Server	2 GB	Fixed – C: 10 GB Dynamic -D: 30 GB
TFS Proxy Server	Physical unit	Single CPU	2 GB	120 GB
Web Application Server	Virtualized	Single CPU	2 GB	Fixed – C: 10 GB Dynamic -D: 30 GB
Developer Machines	Physical unit	Dual Core CPU’s	2GB	120 GB

Table 2: Hardware specifications

That’s all for now, I’ll talk about TFS customization and the extra tools we developed ourselves in the next post.