15 years on and some teams don’t realise that CI is mandatory.

There’s several stages of evolution with TeamCity:

• The single project
• Multiple copies of the same project referencing a single template.
• Then we collapse back down to one project but a branch-agnostic project.

With ‘feature-branch’ support now in TeamCity we only need to copy our projects when we’re creating a new version of the build system, - not when we’re creating a new branch. TC can keep an eye on a range of branches and trigger a build on the right one. This removes the need for templating and keeps things simple again.

---

properties for trunk/branch, version checkout on agent

a project should be the level at which you branch at. I.e. you can have one with ‘trunk’ in the name.

Templates are good but you can’t template build dependencies so try to keep the dependencies within one TC project.

TeamCity

I’m assuming you’re fully bought into how awesome TeamCity is as a CI server. Have a look and see how your build scores against this list.

Maintenance

1. Thou shalt not select agents based on their name.

• Instead rely on Requirements 'version min'. See Supercharge your build agent properties

2. DRY: Templates are your friend.

• The only thing you can't template is configuration dependencies.
• You can set up properties in the root project to provide sensible defaults. For example: a system property named 'system.Configuration' set to a value of 'Release' will ensure all msbuild projects and solutions will build in release mode instead of debug mode by default.

3. Favour svn / package managers over agent installs

• If you have to, check it into source control, but if possible use a package manager.
• Using something like Nuget rather than checking into source control also reduces checkout footprint (so build agents need to do fresh checkouts less).

Speed

4. No more than 50 agents per server.

• Just because you can, doesn't make it right.

5. Keep it brief

• Keep the build logs small to stay quick (they all have to be shipped back to the head node).
• Keep the checkouts small to prevent disk thrashing. (Use package managers.)
• Keep the artifacts small to increase the length of history you can keep.

6. Always checkout on the agent

• Lower load on central server - it's outnumbered, be nice to it. (This is now the default from version 10)

Reliability

7. Always use the Free Disk Space build feature

• What's not to like - make sure there's enough disk space to finish the compile.

8. Use custom errors.

• Break the build on warnings and invent your own.
• E.g. fail on the word 'considering ' on a .net project if it's not finding an assembly.

9. Be hygienic.

• If you choose the certainty of a clean build and you're in .net
  • Build to single bin/obj dirs (for speed).
  • Delete the bin/obj/dist dirs just to be sure.
• Add a scheduled once a week build even if you've not made changes to ensure your tests are not permagreen when in reality your red.
• Clean your build agents weekly of everything.
• Save your settings to sourcecontrol - you'll get invaluable visibility on teamcity settings changes.
• Always click the clear down the folder dependencies checkbox and when doing nuget pack (JetBrains why is this not by default and why hide it in the advanced section?)

Above all,

10. Don't live with red.

• If things have got red and stayed red, you need to do *whatever it takes* to get it green again. If that means deleting a bunch of tests or long term ignoring them then do it - the last thing you want is people getting used to living in the red. (Obviously zero tolerance is best approach)

If you’ve got some great CI / TeamCity tips, please share!

Obligatory infographic:

Agent Management

TeamCity is awesome, but there’s always room for improvement. Here’s a great trick to help you always select the right agents…

Agent-Selection

Once you’ve got a big build farm, agent selection has challenges. How many times have we realised the tests failed because that agent didn’t have a particular package installed on it?

One technique that works very well is to have a weekly task running on all compatible agents that scans the registry for all MSI packages installed and which version numbers they are.

This is easy enough to do in PowerShell:

Get-WmiObject -Class Win32_Product

Returns the installed set of products, with the Name and Version properties being on each product.

The secret is to then write this information to the build agent’s conf\build.properties file. The agent will then disconnect from the TeamCity cluster, and a minute later will re-connect with the appropriate settings.

One slight gotcha at the moment is to try not to have any version number as part of the property name as TeamCity can’t do wildcards on property names - only on the version field.

Run as Admin

Obviously it’s strongly recommended to not have your build chain need admin privileges to run, but for some builds it may be the only way. You could subset to build agents which have admin privileges by doing something like this:

([Security.Principal.WindowsPrincipal] [Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] "Administrator")

(Again, if at all possible you can de-admin your build process that’s preferable!)

Hopefully you’ll find other good uses for this technique!