Don’t Mix and Match

Do not use your CI/CD pipelines for routine maintenance.  Why not you ask, well its simple really routine operations are different to continuous delivery.  Whats the difference?

Let me start with a story, this is something I inherited and while it sort of worked I was never very happy with it. I was brought in to manage a bunch of servers that were being used to build and test an application, now do to an issue in the base OS they had to be rebooted once every 1000 runs or so, the decision had been made before me to reboot the machines every day just to be safe.  So the solution I inherited as for a VSTS build pipeline to run once a day that would look at each pool, wait until the current job was finished and then reboot the machine and re-enable.  it sort of worked, the number mornings I came in and found a some of the machines in an indeterminate state was to high for me and added some manual checks every morning.

So what was the issue? Well basically the VSTS build system while very good is not meant for maintenance jobs and is not actually very good at running them. If all this was in azure it would be pretty simple, set an automation runbook that performs the routine maintenance but this was a on premises environment with a mixture of physical and virtual machines. 

What would I of done, given the time, would of been to register a scheduled task on each machine that would of performed the reboot and re-enabled the machine in the pool, this could of been deployed by VSTS if necessary.  this would of kept the CI/CD separate from the actual environment and we could of if necessary replace the CI/CD component with another and the environment would of stayed working.

The moral, use the right tools for the right job.  Don’t think ah I have this tool it looks like a hammer and if I hit this problem hard enough with a hammer it might solve the problem.