Our CI Journey
As a company, our first foray into CI involved a single instance of Jenkins and a single instance of Reviewboard (we currently link the two so that reviewers can see the associated test results when they are reviewing code - see this video for more information). These instances dealt with every project we were working on. To organise them, each project had its own version-control repository, a Reviewboard group, and a Jenkins folder.
The beauty of Jenkins is its modular architecture of open-source plugins. This makes it expandable, modifiable and, because you only install the plugins you need, you can avoid the clutter of unwanted features. However, these plugins can quickly become deprecated, or introduce new features that break backwards compatibility, which turned into an issue when we started needing different versions of specific plugins for different projects on our single Jenkins instance.
Another challenge was that we wanted to share more of the benefits of CI with our clients by giving them direct access to these tools. Although Jenkins and Reviewboard provide some levels of permissions and access control, those features are not at the level of security that we wanted. This was particularly needed for projects where we are collaborating with the client, as they need to have access to the source code, Jenkins and Reviewboard.
To address the issues of access control and differing CI requirements on different projects, our short-term solution was to create a separate Jenkins / Reviewboard pair for any projects that required it. The downside of this was that it introduced inconsistencies between projects in the CI tool's location, configuration, and available build, analysis and testing tools. This, combined with new projects being set-up by different people, led to more variations in the configurations. It became difficult for developers to easily change between projects without having to relearn each specific CI environment. Maintenance overheads also grew rapidly as each separate environment needed to be manually updated for any plugin or version changes, with the work repeated over every instance.
Although this system had brought the advantages of CI to our projects, we felt that there was still a lot of room for improvement. We decided we would invest some effort to address these limitations, which brings us to our list of requirements.
The Requirements
Project Isolation
All projects should be isolated from one another. We should have the ability to invite clients as collaborators to view every step of the CI solution, or even be able to package up the project, including the CI, analysis, and testing tools, and provide that as a complete CI solution to the client.
Automated
Automate as much as (reasonably) possible. This is not only to save labour and time at set up, but also for consistency, quality and ease of future updates.
Consistent
All projects should have consistency between one another. Consistency makes it easier for developers switching between projects, eliminating the need to relearn the CI environment. It also makes documentation of the CI solution apply to all instances.
Flexible
Clients can have a wide range of requirements and we offer a wide range of expertise. We needed a CI solution that could be set up on any server, and work with both hardware and software-based projects using a wide variety of build, analysis and testing tools.
Version Controlled
Configurations should be kept as code alongside the source code in the project repository. Previously our configurations had all been managed through the Jenkins GUI. This was useful for manual setup but difficult to maintain and made it very repetitive to update jobs across multiple projects. Keeping the configurations in the version control repository makes it easy to push the same update to many different projects. This can be invaluable when upgrading Jenkins or its plugins.
Open Source
For flexibility and expandability, we wanted to maximise our use of open source solutions. This approach allows us to make our own additions and modifications for features that are not yet implemented and give back to the community that develops these tools.
Modular
With different projects using different tools, it is important for us to be able to easily mix and match tools and their associated CI configurations as we deploy new CI instances for new projects, or manage the changing requirements of existing projects. Adopting a modular approach allows us to do this efficiently and to apply updates and enhancements relating to specific tools to all instances that use them, regardless of what else might also be in those environments.
Maintainable
In addition to the maintainability requirements already mentioned, we wanted to be able to make configuration changes and test that they performed correctly without impacting the development team. To achieve this, we wanted to be able to branch the CI configurations in the same way that we would branch our production code and then have Jenkins run tests using the branch changes, only merging back to master/trunk once everything is passing.
Our Solution
Our final solution kept the same tools that we had already been using: Jenkins, Reviewboard and the analysis and testing tools we have used in the past. However, the setup and configuration of these tools had to be changed. In order to completely isolate projects from one another, we are using individual Virtual Machines (VMs) for each project, each hosting Jenkins, Reviewboard and documentation. This lets us allocate resources to different VMs depending on project requirements. We can spin down or archive VMs if the project is completed, or transfer them to the client.
Using separate instances of Jenkins and Reviewboard means that every new project now has an entire new VM, configured to work with the tools on the relevant OS. To minimise human labour and time taken, the vast majority of these steps are automated. We use configuration management tools (such as Puppet) to configure the VM and install Jenkins and Reviewboard. The final steps of configuration for the tools are done manually, as the chosen tools are different for each project. Through this automation, we have cut down the time to deploy new templated CI VMs to less than a day. Of course, some customisations can then take additional time, and for client projects we also need to discuss and advise on requirements, potentially perform on-site installations and provide training in how to operate and maintain it. Overall though, the vast majority of setups will be completed within the first two week sprint.
The automation means the configuration steps of the Jenkins and Reviewboard instances are identical, outside of the tools used and visual customisation. This removes the overhead of the developer having to learn a new CI configuration for every new project. To keep the consistency across any application, the Jenkins jobs were made modular. The job itself, and all supporting code doesn't change, but calls a list of tools which are defined in small scripts that are easily added or removed from the job.
Moving the configurations to the version control repository meant translating the Jenkins jobs from freestyle projects into Jenkins declarative pipeline syntax. Unfortunately, some of the plugins we were using did not support this syntax, such as the very important jenkins-reviewbot plugin (which allows Jenkins to Reviewboard interactions) and the doxygen-plugin (which lets us do document generation within the Jenkins job). Because of our choice to use open source solutions, we were able to add support for the new syntax ourselves and create a pull request to share it back to the open source community.
Conclusion
Having put in place this automation, templating and plugin enhancements we have been so pleased with this CI approach that we have launched this as a new service, which we have named Accelerate CI. As part of this service, we will advise on tools and flows, create a customised CI environment optimised for the needs of the project, get everything up and running on site and provide training to both users and administrators. We now have templates for a wide range of tools covering analysis, test, build and documentation of both software and FPGA designs. We've found that CI systems tend to evolve slowly over several years before reaching their full potential. Our goal is to leap-frog that process and give our clients the benefits of a mature CI system in weeks rather than years. If you're interested in learning more about Accelerate CI, we'd love to hear from you.
We will be presenting more on how we achieved this and the benefits we've experienced at our CI Workshop on 7th March 2019. If you’re interested in learning more, register to attend, or if you can’t attend the Workshop join our CI Interest Group to get future updates...
