2 June 2016

Teamcity

                                                                                                                                         | | 15 years on and some teams don't realise that CI is mandatory.

There’s several stages of evolution with TeamCity:

The single project
Multiple copies of the same project referencing a single template.
Then we collapse back down to one project but a branch-agnostic project.

With ‘feature-branch’ support now in TeamCity we only need to copy our projects when we’re creating a new version of the build system, - not when we’re creating a new branch. TC can keep an eye on a range of branches and trigger a build on the right one. This removes the need for templating and keeps things simple again.

properties for trunk/branch, version checkout on agent

a project should be the level at which you branch at. I.e. you can have one with ‘trunk’ in the name.

Templates are good but you can’t template build dependencies so try to keep the dependencies within one TC project.

Cake

Freightliner Just Revealed The First Real Road-Legal Autonomous Big Rig: http://truckyeah.jalopnik.com/freightliner-just-revealed-the-first-real-road-legal-au-1702382986/+AnnaleeNewitz

Given most of the cyclists killed in London are by undertaking a lorry while not being seen, these automated trucks should have a much better job of spotting them.

I wonder how few years it will be before they don’t bother with the cab. My guess is that we will see cabless road trains first in Australia.

(Just so we are clear, you should still have a truck number greater than 1 in your team even if the chances of a team member getting run over are now smaller !)
| | I thought I was doing the right thing. I had every dll branded with the branch that it was built with and the subversion revision number that the repository was up to when the build was done.

I still think there’s nothing wrong with including the branch name, but there’s distinct disadvantages to just wiring the version number to the latest revision number. If you do so you loose an important property: your artefacts change all the time. Two otherwise identical builds will appear binary different for all built dlls.

From a human / operational perspective this is a big downside. It’s great that we can aways reproduce the exact build, but ideally if I’ve not changed a dll it would be great if that dll didn’t change.

Can we have our cake and eat it?

TeamCity

I’m assuming you’re fully bought into how awesome TeamCity is as a CI server. Have a look and see how your build scores against this list.

Maintenance

1. Thou shalt not select agents based on their name.

Instead rely on Requirements 'version min'. See Supercharge your build agent properties

2. DRY: Templates are your friend.

The only thing you can't template is configuration dependencies.
You can set up properties in the root project to provide sensible defaults. For example: a system property named 'system.Configuration' set to a value of 'Release' will ensure all msbuild projects and solutions will build in release mode instead of debug mode by default.

3. Favour svn / package managers over agent installs

If you have to, check it into source control, but if possible use a package manager.
Using something like Nuget rather than checking into source control also reduces checkout footprint (so build agents need to do fresh checkouts less).

Speed

4. No more than 50 agents per server.

Just because you can, doesn't make it right.

5. Keep it brief

Keep the build logs small to stay quick (they all have to be shipped back to the head node).
Keep the checkouts small to prevent disk thrashing. (Use package managers.)
Keep the artifacts small to increase the length of history you can keep.

6. Always checkout on the agent

Lower load on central server - it's outnumbered, be nice to it. (This is now the default from version 10)

Reliability

7. Always use the Free Disk Space build feature

What's not to like - make sure there's enough disk space to finish the compile.

8. Use custom errors.

Break the build on warnings and invent your own.
E.g. fail on the word 'considering ' on a .net project if it's not finding an assembly.

9. Be hygienic.

If you choose the certainty of a clean build and you're in .net
- Build to single bin/obj dirs (for speed).
- Delete the bin/obj/dist dirs just to be sure.
Add a scheduled once a week build even if you've not made changes to ensure your tests are not permagreen when in reality your red.
Clean your build agents weekly of everything.
Save your settings to sourcecontrol - you'll get invaluable visibility on teamcity settings changes.
Always click the clear down the folder dependencies checkbox and when doing nuget pack (JetBrains why is this not by default and why hide it in the advanced section?)

Above all,

10. Don't live with red.

If things have got red and stayed red, you need to do *whatever it takes* to get it green again. If that means deleting a bunch of tests or long term ignoring them then do it - the last thing you want is people getting used to living in the red. (Obviously zero tolerance is best approach)

If you’ve got some great CI / TeamCity tips, please share!

Obligatory infographic:

Agent Management

                                                        | | TeamCity is awesome, but there's always room for improvement. Here's a great trick to help you always select the right agents...

Agent-Selection

Once you’ve got a big build farm, agent selection has challenges. How many times have we realised the tests failed because that agent didn’t have a particular package installed on it?

One technique that works very well is to have a weekly task running on all compatible agents that scans the registry for all MSI packages installed and which version numbers they are.

This is easy enough to do in PowerShell:

Get-WmiObject -Class Win32_Product

Returns the installed set of products, with the Name and Version properties being on each product.

The secret is to then write this information to the build agent’s conf\build.properties file. The agent will then disconnect from the TeamCity cluster, and a minute later will re-connect with the appropriate settings.

One slight gotcha at the moment is to try not to have any version number as part of the property name as TeamCity can’t do wildcards on property names - only on the version field.

Run as Admin

Obviously it’s strongly recommended to not have your build chain need admin privileges to run, but for some builds it may be the only way. You could subset to build agents which have admin privileges by doing something like this:

([Security.Principal.WindowsPrincipal] [Security.Principal.WindowsIdentity]::GetCurrent()).IsInRole([Security.Principal.WindowsBuiltInRole] "Administrator")

(Again, if at all possible you can de-admin your build process that’s preferable!)

Hopefully you’ll find other good uses for this technique!
	<h2>Living with NuGet</h2>

As NuGet packages are immutable I’m really interested in using a shared package directory for NuGet as suggested here:

http://stackoverflow.com/questions/24617112/storing-nuget-packages-in-alternate-location-on-build-server

As well as time, this would save a ton of diskspace on build agents.

Death to Dll Hell

.Net assembly versioning has always been ‘tricky’. The GAC solves this problem but as many find out the GAC is wac. An alternative to mandatory strong naming of the world is to embed the version in the assembly name (which you bump the version number on each breaking change).

At least with Nuget this gets a bit easier when you’re then upgrading references to the next version.

Large Testbases

                                                                                                                                                                                                                                                                                     | | There's two possibilities, either you have a set of machines exactly the same or they're not. If you're using imaging then exactly the same may be possible.

If not there’s likely to be machines which don’t have every last install required to pass every test. You could be fighting a loosing battle trying to get everything installed everywhere. Instead nunit has ‘assume ‘ which allows you to check whether that component is available before running the test. If it’s not then the test is reported as inconclusive.

Using ‘Assume’s allows you to use a higher percentage of build agents and reduces the number of boring test failures.

In my book that’s a win-win that reduces the costs of keeping a large testbase on the road.

Testing in the Small

Before you get to having a large test base you need to know how to test. There’s a learning curve and the more you do the better you get, but I think this guy has hit the nail on the head (it’s not really about ruby):

https://www.youtube.com/watch?v=LfmAzLAKKoc
	There’s two possibilities, either you have a set of machines exactly the same or they’re not. If you’re using imaging then exactly the same may be possible.