How to accelerate your FPGA development using CI

Call: +44 (0)23 8098 8890

Posted 29th May 2020, By Hugh J
Jenkins for FPGA

Here at ITDev, we believe FPGA development can be efficient and enjoyable but we’ve all had projects before where that hasn’t been the case. Maybe a bug crept in and you can’t tell when, or you went over 100% resource utilisation without noticing it. This is where using Continuous Integration (CI) tools like Jenkins can be useful. They can do all the tedious work so that us engineers can focus on the fun bits: implementation, verification and problem solving!

Over the course of this blog, I’m going to look at a few areas where we have seen great benefit to projects:

  • Regular Builds and Linting
  • Unit testing
  • Resource and Timing Tracking

Where we use third party tools, I’ll give an example but there are lots of free and paid for alternatives to choose from. Links to all the tools can be found at the end of the blog.

If you like the sound of this blog, but are completely lost don’t worry, take a look at our blogs on CI concepts.

Regular Builds and Linting

This is the easiest thing to set up and can save many headaches. We tend to split this into two:  a simulator compile (e.g. Modelsim or Verilator) and an FPGA build (usually using Xilinx’s Vivado or Intel’s Quartus). The reason for the split is that you may want to run a quick simulator compile on every commit but only run synthesis once a day (depending on how long it takes).

Having a job that does a build is useful but ideally, we want to pull out the pertinent information. That’s where tools such as the Warnings Next Generation plugin for Jenkins come in. This tool comes with a load of tools to parse standard log formats (e.g. GCC or similar). Most software engineers will be satisfied with this list but the list of supported FPGA tools is less impressive. They helpfully provide a way to create tools on the fly in the Jenkins GUI, but to save you time our own Accelerate CI product comes with built in Warning Parsers for Vivado, Quartus and Modelsim logs for instant set up.

The Warnings plugin gives you a sortable/filterable table of warnings for each build and even shows you trends over time. This can be useful for tools that are a bit over-enthusiastic about warnings; if you don’t want to fix them at least you can check the number of warnings doesn’t increase. This can be compared to the previous build, or a different branch or release.

Vivado Warning Parser for Accelerate CI 

Unit Testing

Unit testing can be very useful, especially on large projects where system testing won’t begin for a long time. Nobody wants to get weeks into a project with no feedback that functionality has been implemented correctly. As a design consultancy, it is especially important to us that clients can see a measure of progress throughout the project. Depending on the project, we have two approaches to unit testing:

  1. During feature implementation we develop a unit test. This unit test will be re-run in future CI builds to check nobody breaks the feature.
  2. Tests (or empty placeholders) are written up front and are expected to fail at the start of implementation. As features are implemented tests begin to pass.

Either way, progress can be tracked as tests are added or move from failing to passing. But we need a plugin to show this information in Jenkins. We tend to use xUnit, a plugin that supports multiple test result formats, to parse results. Separate to this, we use Test Results Analyzer plugin to display the results. This fits nicely with our choice of HDL unit testing: VUnit. For more information on this subject, why not read our blog or check out VUnit's home page. VUnit has a parameter -x XUNIT_XML that allows you to output your tests results as an XML file compatible with the xUnit plugin using the JUnit format. This works with both Jenkins and Bamboo.

Like the warnings plugin, we not only get current results but also get trends over time. This can help pinpoint which submission first broke a test.

Test Result Analyzer Dashboard

Resource and Timing Tracking

So far, the Warnings plugin has allowed us to make sure we maintain a high quality of code, and the xUnit plugin has allowed us to track progress and prevent regression. But at the end of the day, we need to know about resource usage and clock speed. Ideally, we want to know about any potential issues in either as soon as possible. That’s why we use the Plot plugin to show a plot of the useful figures. We first run a helper script which we wrote (shown below) to extract the figures from the tool reports and generate a csv file before the tool can generate the graphs. Using this information on how much of the implementation has been completed can be a useful way to estimate final resource utilisation and clock speed.

The following BASH snippet is a very crude way to convert a Quartus summary file to a csv table of the format required by the plot plugin:

# Grab the percentage data from summary
TABLE="$(sed -rn 's/([^:]*) :.* ([0-9]*) %.*/\1\t\2/p' $1)\n"
# Count the number of rows
ROWS=$(printf "$TABLE" | wc -l)
FMT=$(python -c "print '- ' * $ROWS")
# Transpose columns to rows
ROW1=$(printf "$TABLE" | cut -f1 | paste -d, $FMT)
ROW2=$(printf "$TABLE" | cut -f2 | paste -d, $FMT)
# Output
printf "$ROW1\n$ROW2\n"

Giving us a CSV output like this:

Logic utilization (in ALMs) Total pins Total block memory bits ... Total HSSI PMA TX
25 28 5 ... 0


Which results in a graph like this:

FPGA Utilisation Plots

So What?

All this information can be helpful. We believe that issues should be caught as early as possible, so we run a large amount of CI testing during code review (using integration with Reviewboard or Gerrit). Failing tests and Compile error reports can be used by the code author to fix their code before asking colleagues to perform a code review. In parallel with this, we tend to have Jenkins run checks periodically if new submissions to the master (or trunk) have been made. This ensures that if any issues slip through the net, we find them as quickly as possible.

Whether you use Jenkins or Bamboo, to set up your own environment or employ a setup such as Accelerate CI, I would advise you to pair CI with FPGA development. It helps you catch issues early, prevents human error from going unnoticed and helps track the progress of projects. What more could you want?

There are also plenty of other things you can add in, such as including Modelsim’s HTML code coverage report, or using system test tools to run nightly FPGA tests with real hardware, but these topics are for another day.

Tool Links

Main image: Jenkins on a computer chip

Interested in other ITDev CI blogs?

Here at ITDev we have been utilising Continuous Integration (CI) on client projects for many years and have built extensive experience on the subject. Helping the industry to adopt best practice we have written introductory and more advanced blogs on the subject. See this search index on other CI related blogs.

How ITDev Can Help

At ITDev we believe good engineering practice is essential to delivering high quality, engineered designs. Aligned with this, we run a series of events and workshops to help develop industry best practices through collaborative sharing and learning. 

ITDev provides technology consultancy as well as hardware & software design services; our team of engineers has extensive experience in FPGA design for both Xilinx and Intel (formerly Altera) devices.

In line with our other areas of expertise, we offer advice and assistance to companies considering investing in CI and other workflow improvements. We use our experience and library of in-house scripts to quickly deliver ready to use CI environments customised to fit your own project requirements. If you are interested in attending a workshop on CI, or receiving more information on this topic, please sign up to the ITDev Continuous Integration Interest Group.

If you have any questions or would like to find out how we can help you, initial discussions are always free of charge and we would be delighted to speak to you. Email us or call us on +44 (0)23 8098 8890.

IET Enterprise Partners logo

Latest Blog Posts

Our product development process
Posted 24th September 2021, By Steve W
In his first in a series of blogs addressing the product development process, Steve looks at how to avoid the pitfalls ...
An empty office and desserted Science Park
Posted 29th January 2021, By Jon O
Having started 2021 much as we ended 2020, we thought it would be good to look back to before the pandemic and reflect on how the year turned out against ...more
Higher level testbench reusing lower level components
Posted 5th January 2021, By Tom J
ITDev recently engaged with an ASIC design company. This blog outlines our verification strategy, and how our knowledge of UVM allowed us to achieve high test ...more
Ladybug in sight
Posted 3rd August 2020, By James H
If debugging a Linux driver is bugging you, this 'how to' guide explains how to approach this using the function tracer within ftrace to find out what’s going ...more

Latest News

Variety is the spice of life
Posted 15th July 2021
The newly re-modeled cafe on the Science Park has put us in a culinary mood but keep on reading and you'll see there's another reason why we chose a spice rack ...more
Supply chain management
Posted 7th June 2021
Recent activities in May include focusing on a number of product development enquiries, helping an aerospace client with support issues and starting ...more
Deep tech
Posted 10th May 2021
News this month includes details of further engagement with our client on Android audio device driver development and a new partnership with the world’s first ...more
SKA at night
Posted 31st March 2021
As our work tends to focus on ‘technologies’ rather than ‘markets’, it’s normal for our portfolio of projects to cover a range of industries.