Continuous Delivery (CD) facilitates the path of an evolving product from development through to production. Continuous Integration (CI) plays an important part of this process and defines the software development process, residing at the beginning of the CD process. There is a wealth of information in books and on the Internet describing CI processes, however with a large amount different CI tools available there is little information on the build jobs which compile and test the code that are central to the CI process. A typical CI process is as follows: The developer manually builds and tests source code on their own machine. They will then commit their modifications to a source control management system. A build tool will subsequently run jobs which compile and test the code. The built artefact is then uploaded to a central repository ready for further deployment and testing.
Automatic Creation of Build Jobs
So why should we set up a facility to automate the creation of other jobs? Build tool user guides describe how to create build jobs but they do not often explain how to manage them in any great detail, despite the functionality being available in the tool to do so. Normally the developer will know how their application is built and so take sole responsibility for the creation and configuration of the build job. However, this process has some disadvantages:- Software architectural constraints: Applications are likely to be built differently from one another, depending upon architecture. The developer may interpret how an application is built differently from another developer and so the build configuration will slightly differ from one job to another. Hence, a large number of jobs become unmaintainable if they are all configured differently.
- Human factor: Manually creating a job introduces the risk of making mistakes especially if a new job is created by copying an existing job and then modifying that job.
- Job timeline control: There is generally no record kept of job configuration for every build, i.e. If a job configuration is changed it could break previous builds.
Advantages for automating the creation of build jobs
There are several advantages of using a master build job to automatically create multiple jobs in order to facilitate the CI build process.- All build job configurations are created from a set of job templates or scripts. These can be placed under configuration control. Creation of a new job is then a simple matter of using the template or running the scripts. This can cut job creation times significantly (many minutes to a few seconds). Hence, Job configuration changes are made easier; configuration changes to the build job template ensures that all subsequent new jobs created will inherit the new changes.
- Consistency with all existing build job configurations implies that globally updating all configurations via tools or scripts is more straightforward than with a cluttered inconsistent system.
- The developer does not necessary need detailed knowledge of the build tool to create a build job.
- The ability to automatically create and teardown build jobs as part of an Agile CI build lifecycle.
Point 1: Configuration Control of Job Configuration
It is good practice that every component of a CI system should be placed under configuration control. That includes the underlying build systems and the build jobs, not just the source code. Most build tools store build job configuration as a file on the master system. These files are set to a common XML format and are normally accessible via the front end interface directly through a REST API of some kind. Some build tools have a built-in feature that can save job configuration as a file to any location. Since the job configuration can be saved as a file it can be stored in a configuration management (CM) system. Any job configuration change can be recorded through either modifying the file directly, then upload the file to the build tool or modifying the job configuration from the build tool, save the file and upload it manually to the CM system. The method carried out depends upon how easy it is to access the job configuration file directly from the build tool. Another important reason for storing job configuration in a CM system is that in the event of a catastrophic failure and all job configurations are lost, the build system can be recovered relatively quickly and all jobs, build logs and job history restored back to their last known good state.Point 2: Job Maintenance
Of course it goes without saying that maintaining a thousand jobs of different configurations would be a major headache, which is why it is so important to standardise job configurations. If a change is required that impacts on a large number of similar jobs, then writing scripts or creating special tasks to modify these jobs would be straightforward. However, having a thousand jobs in a build system is not the best strategy for CI. We will discuss this further under Point 4.Point 3: Developer Control
Although developers are encouraged to work autonomously, building their applications using a centralised build system, they need to work within guidelines outlined by the build process. A developer will expect fast feedback of their code changes and so not want to waste a lot of time messing around with build tools and job configuration. Providing developers with a one button 'self service' style solution to set up build jobs automatically, will help them achieve their development task objectives more quickly.Point 4: A Lean CI System
Although software developments teams adopt Agile methods in developing their products, the build tools they work with are not necessarily used in the same way, i.e. a properly configured build tool should not contain thousands of long lived build jobs. Ideally, jobs should be created on demand as part of a build CI lifecycle. As long there is the means to recreate any build job from any historical job configuration and that build relics are retained, i.e. logs, artifacts, test reports etc., then only a handful of jobs should exist in the build tool. Of course an on-demand job creation-teardown solution may not be an achievable goal, but the point is that the number of jobs in a build tool should be kept to a manageable level.Day-to-day Development
Build management governs the tools and processes necessary to assist developers in day-to-day activities with regards to build and continuous integration. Utilities can be set up which empowers development teams to create build jobs and provide flexibility in job configuration but still be governed by best software development practices.The CI Job Suite
Consider typical tasks that developers carry out during the course of day to day development:- Build a baseline
- Release the baseline
- Build a Release branch (often combined with releasing the baseline)
- Create a development branch, build the branch
- Run Integration Tests
When to implement a self-service job creation solution
Implementing a self-service solution for creating build jobs is applicable for the following cases:- Initialisation of new 'green-field' projects (those that have never been built before).
- Projects that already exist but never had any build jobs set up for them. i.e. building manually from a developer's own machine.
- If a product only has a few jobs, i.e. large monolithic applications, then it may be considered not of value to the business to spend a considerable amount of time and effort designing, testing and rolling out an automated create job process for just those few projects. However, as the business expands, the product architecture will become more complex as more development streams are set up to manage the extra workload, hence the number of jobs in the build tool will inevitably increase.
- Projects with existing jobs: Two scenarios can be considered when dealing with existing jobs in the build tool.
- Delete the existing jobs and recreate them with the job suite creation utility, thereby losing all historical build information and logs.
- Keep the existing jobs and attempt to modify their configuration to match the job configuration created by the job suite creation utility.
Setting Up a Self-Service Job Suite Creation Facility in a Build Tool
We have talked about the advantages of automatically creating build jobs, so now let's discuss how we can implement such a feature in a build tool. The principle should be the same for building applications for any language I.e. Java, .NET etc...Introducing the Job Creation Build Form
The build form is a means to pass information from the build tool directly to another feature or backend scripts that perform the job creation tasks. Ideally, as much project information should be provided upfront. This will reduce the amount of any additional effort required later to add extra configuration to the newly created build jobs. Information such as project title, location of source code and specific build switches are normally required. Hence, at a push of the 'Build' button the jobs are created just a few moments later.Build Tool Implementation:
We shall now discuss how such a job creation facility can be implemented in a build tool. Two tools for which I have had personal experience of setting up job suite creation facilities are Hudson and Anthill Pro 3.Hudson / Jenkins
For clarity we will just talk about Hudson, but the information is equally relevant for Jenkins. The Hudson CI tool has become very popular for Build Managers looking to minimise their development resource budget and do not want to be tied down with expensive licences. Hudson defines its build job configuration in XML format. A typical job config XML file is accessible from the top level page of the job url, i.e. http://hostname/job-name/config.xml Most of the sections in the config file are self-explanatory. Sections are added by plugins in the config file only when they are used in the Hudson GUI. These config files can be 'templated' by replacing specific project information in them with tokens. Several templates can be used to carry out different tasks, i.e. Checkout of source code from different SCMs e.g, Subversion or GIT. So now we need to link the create job build form to the scripts. The diagram below shows the workflow:
- If the Hudson security section is configured, ensure that the 'anonymous' user has 'create job' privileges otherwise the upload will fail authentication.
- If Hudson tabs are used to view specific groups of jobs, it will not be possible to automatically place any of the newly created jobs under those tabs, except for the generic 'all' tab (or any tab that uses a regex expression). The Hudson master configuration file will require updating as well as a manual restart of the Hudson interface. Once the jobs are created they can then be manually added to the relevant tab if so desired.
<project name="createjob" default="createHudsonjobsConfigs">
<!-- Ant script to update hudson job config.xml templates to create new jobs in hudson -->
<!-- Get external properties from Hudson build form -->
<property name="hudsonjobname" value="${HUDSON.JOB.NAME}" />
<property name="scmpath" value="${SCM.PATH}" />
<property name="mvngoals" value="${MVN.GOALS}" />
<!-- ...do same for rest of properties from the hudson form -->
...
...
<property name="hudson.createItemUrl"
value="http://hudson.server.location/createItem?name=" />
<!-- Include ant task extensions-->
<taskdef resource="net/sf/antcontrib/antlib.xml"/>
<target name = "createHudsonjobsConfigs" description="creates new config.xml file from input parameters">
<mkdir dir="${hudsonjobname}"/>
<!-- loop through each job template file replacing tokens in the job-templates with properties from Hudson -->
<for list="CI-build-trunk,RC-build-branch" param="jobName">
<sequential>
<delete file="${hudsonjobname}/${configFile}" failonerror="false"></delete>
<copy file="../job-templates/@{jobName}-config.xml" tofile="${hudsonjobname}/${configFile}"/>
<replace file="${hudsonjobname}/${configFile}" token="$HUDSON.JOB.NAME" value="${hudsonjobname}"/>
<replace file="${hudsonjobname}/${configFile}" token="$SCM.PATH" value="${scmpath}"/>
<!-- ...do same for rest of tokens in the job template -->
...
...
<antcall target="configXMLUpload">
<param name="job" value="@{jobName}"></param>
</antcall>
</sequential>
</for>
</target>
<!-- contruct the job config upload command -->
<target name="configXMLUpload">
<echo>curl -H Content-Type:application/xml -s --data @config.xml ${hudson.createItemUrl}${hudsonjobname}-${job}</echo>
<exec executable="curl" dir="${hudsonjobname}">
<arg value="-H" />
<arg value="Content-Type:application/xml"/>
<arg value="-s" />
<arg value="--data" />
<arg value="@config.xml" />
<arg value="${hudson.createItemUrl}${hudsonjobname}-${job}"/>
</exec>
</target>
</project>
Below is a typical tokenised Hudson job template:
<?xml version='1.0' encoding='UTF-8'?>
<project>
<actions/>
<description>Builds $POM.ARTIFACTID</description>
...
...
<scm class="hudson.scm.SubversionSCM">
<locations>
<hudson.scm.SubversionSCM_-ModuleLocation>
<remote>$SCM.PATH/trunk</remote>
<local>.</local>
<depthOption>infinity</depthOption>
<ignoreExternalsOption>false</ignoreExternalsOption>
</hudson.scm.SubversionSCM_-ModuleLocation>
</locations>
</scm>
<assignedNode>$BUILD.FARM</assignedNode>
...
...
<builders>
<hudson.tasks.Maven>
<targets>$MVN.GOALS</targets>
<mavenName>$MVN.VERSION</mavenName>
<usePrivateRepository>false</usePrivateRepository>
</hudson.tasks.Maven>
</builders>
...
...
</project>
CI-build-trunk-config.xml Hudson Job Template
Anthill Pro
Anthill Pro is a licensed build and continuous integration tool from Urban Code Inc. which provides the user almost complete custom control of build jobs via a GUI and an extensive API library. The latest incarnation from UrbanCode is uBuild, which is essentially a cut down version of Anthill and is geared primarily towards the continuous integration building of products rather than its parent product which was originally designed to manage to entire continuous delivery pipeline. uBuild does not have the API capability that Anthill has but it is possible to create a plugin that can do a similar thing. Build jobs in Anthill are called 'Workflows' which essentially are a pipeline of individual job tasks executed in a specific order (serial or parallel). Anthill's API scripting language is Beanshell which is based on standard Java methods. Beanshell scripts are stored within the Anthill graphic user interface and called from within a job task. The latest version of Anthill allows custom-built plugins which can be developed with any programming language. An Anthill job task is created using API calls which essentially 'constructs' all the necessary functions of that workflow. The script can be particularly long as every aspect of the workflow configuration has to be populated with data. Just like Hudson, build job information is collected via a build form which executes a master workflow, calling on the Beanshell scripts to create the relevant workflow.
private static Project createProject(User user) throws Exception {
// get the values from the buildlife properties
String groupId = getAnthillProperty("GROUPID");
String artifactId = getAnthillProperty("ARTIFACTID");
String build_farm_env_name = "linux-build-farm";
String branchName = "branchName";
Workflow[] workflows = new Workflow[3];
// Set up Project
Project project = new Project(artifactId + "_" + branchName);
// determine whether the project already exists and is active
boolean isProjectActive;
try {
Project projectTest =
getProjectFactoryInstance(artifactId + "_" + branchName);
isProjectActive = projectTest.isActive();
} catch (Exception err) {
isProjectActive = false;
}
if (!isProjectActive) {
project.setFolder(getFolder(groupId, artifactId));
setLifeCycleModel(project);
setEnvironmentGroup(project);
setQuietConfiguration(project);
String applications = getAnthillProperty("APPS");
// create project properties
createProjectProperty(project, "artifactId", artifactId, false, false);
// set the server group for the workflow and add environment properties
addPropertyToServerGroup(project, build_farm_env_name, applications);
project.store();
...
...
// Create the CI Build workflow
String workflowName = "CI-build";
workflows[0] = createWorkflow(
project,
workflowName,
perforceClientFor(branchName, groupId, artifactId, workflowName) + buildLifeId,
perforceTemplateFor(branchName, groupId, artifactId, workflowName),
"${stampContext:maven_version}",
build_farm_env_name,
"CI-build Workflow Definition"
);
// add trigger url to commit-build branch
addRepositoryTrigger(workflows[0]);
// Create the Branch workflow
workflowName = "Branch";
workflows[1] = createWorkflow(
project,
workflowName,
perforceClientFor(branchName, groupId, artifactId, workflowName) + buildLifeId,
perforceTemplateFor(branchName, groupId, artifactId, workflowName),
"From ${stampContext:maven_version}",
build_farm_env_name,
"Branch Workflow Definition"
);
// Create the Release workflow
workflowName = "Release";
workflows[2] = createWorkflow(
project,
workflowName,
perforceClientFor(branchName, groupId, artifactId, workflowName) + buildLifeId,
perforceTemplateFor(branchName, groupId, artifactId, workflowName),
"${stampContext:maven_release_version}",
build_farm_env_name,
"Release Workflow Definition"
);
...
...
} else {
// project already exixts
String msg = "Project " + artifactId + "_" + branchName + " already exists - check project name and pom";
throw new RuntimeException (msg);
}
return project;
}
Example CreateProject Java Beanshell script
Note: It is also possible to create a self contained package of Java Beanshell scripts such as a jar file which is placed in a folder on the Anthill Pro application server. Direct calls to classes within the jar file can then be made from a shell task within the application. This works well especially because the scripts can be unit tested before being applied in a live development environment.
Other Tools
So, as long there exists the ability to create jobs via an API or a template feature, the implementation described here can be similarly applied to other build tools such as Thoughtworks Go, Jetbrains Teamcity and Atlassian Bamboo to name but a few.Ultimate Build Tool Agility: Implementing the Lean Build Tool philosophy
An advantage mentioned earlier in this article discussed the concept of automatically creating on –demand CI projects and then tearing them down after completion. This would significantly reduce the amount of build jobs in the build tool. I have yet to implement an agile lean method for automatically creating and tearing down jobs on a build tool, so the technical challenges to implement such a task have yet to be realised. However, everything that has been discussed here can lead to implementing such a solution. In fact the Atlassian build tool Bamboo has the feature to detect branch creation from a main baseline and then automatically create an appropriate job to build it.Summary
We have discussed how to set up facilities to create other jobs that form part of the continuous delivery strategy for a business.- An automated approach to creating jobs in a build tool should be adopted as part of everyday software development operations.
- Automatic creation of jobs saves time and allows the developer to get on with more important tasks.
- Consistent build job configuration leads to easier job maintenance and drives consistent development practices across a company.
- Automatic job creation can ultimately lead to a true Agile adoption of software development right down to the build tool level.
