Introduction to Part 2
Part 1 of my blog was about configuring tools and getting your environment set up; I must confess this was not the most exciting part. But everyone knows that to work properly, you need all your tools to be ready and perfectly suited for your task. Would a surgeon work with a kitchen knife on an IKEA sofa? I hope not. Our work, as developers, poses hardly any risk to human life compared to the work of a surgeon - but that doesn't mean we don't deserve the best tools, does it? ;)
This next part of my blog will focus on coding and committing so without any further ado, let's start!
Finding something to fix
So far so good, all our tools are on the toolbench, shiny and ready to be used. Your cup of tea (we're a UK-based company and I don't like coffee...) is here too. As the sun rises on the horizon, you start to feel this deep, unstoppable urge to do your best work today. Nothing could stop you from achieving this... except maybe for the fact that we do not have any specific pieces of code to work on.
Cloning Linux's sources
For anything you need which is related to the Linux kernel, you should check kernel.org. If you scroll to the bottom of the page, you will find a lot of links under 'Other readings'; this is where you will find git.kernel.org (named “Cgit”).
You may be thinking: this sounds exactly what I was looking for so let's 'git clone git.kernel.org'? Not really, git.kernel.org is a website hosting lots of repositories relating to the Linux kernel. There are hundreds of kernel repositories for different subsystems, different maintainers, different tools; indeed some of them have been idle for years...Be sure to clone a repository that is not outdated as some of them haven't been updated for years!
When I was first looking to clone a repository, I could not find any information about which tree to use for kernel developers. All the tutorials I found used different trees and I was a bit lost in this process. But after some research, I came up with a list of the most interesting Linux repositories, which I will share with you:
- torvalds/linux.git: the main tree, you'll find the latest Linux releases on this tree. Used to define Release Candidate version (-rc). This tree changes only when a new release (or Release Candidate) is created each 7 to 10 days approximately.
- stable/linux.git: Linux's stable tree repository. Used to store and update stable releases over time. The main difference between this repository and the main Linux tree is that this repository's releases will be updated (mainly security updates and bugfixes) over time if the related release is supported. For example, at the time of writing, the Linux kernel 3.16.y branch has been updated to 3.16.83 less than 2 weeks ago.
- next/linux-next.git: probably one of the most interesting trees. All the different subsystem trees and their modifications are merged into it each day. Thus, if you want the latest, most complete Linux kernel tree, this is the one!
- gregkh/stable.git: managed by Greg Kroah-Hartman, this tree stores all the modifications applied to the staging drivers (released drivers but isolated from the others as they are not finished yet). This tree is a nice source of contributions as all these drivers need to be cleaned up and adapted to the latest kernel's API.
All the subsystem trees at git.kernel.org host a lot of other kernel trees. If you want to fix a warning on a given subsystem, it's always better to clone its tree in order to fix the problem. But as linux-next is updated each night, it should be fairly up-to-date with the latest changes on your target subsystem.
And now? We've identifed some Linux repositories so we need to clone one of them! As I said, cloning linux-next should be fine, so that's what we'll do!
git clone https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git
And now, to quote Austin Powers: wait a tick. As of May 2020, this repository is 3.0GB; depending on your internet bandwidth, it could take up to a couple of hours to clone so be patient :) Also bear in mind that you will need up to 3GB for the kernel’s build artefacts.
Tackle those nasty bugs!
There are lots of ways to find something to fix in the Linux sources. This could range from a spelling mistake to a compiler warning or any cleanup that you could imagine. We will now look at the most straightforward ways you can contribute to the kernel.
Build the kernel... and display warnings!
By default, the Linux kernel build process won't output any warning message, but some flags have been defined in order to increase the verbosity.
Let's not wait any more! Build that kernel:
make
How does it feel? Exciting, isn't it? Sure, it is! Until... Until it fails! Damn. You should also see a clear error message 'Configuration file ".config" not found!'. The Linux kernel build system called Kbuild is a complex process, especially because it builds a lot of tools as well as the kernel, but it is also extremely modular. There are so many build options that, on my machine, the tiniest kernel possible is 423kb, built in 30 seconds, while the most complete configuration (with all the options activated) is 132MB built in 52 minutes. And how, with all these options (12000+ unique options on the current “next” tree) should the kernel build system know what we need to build?
To help Kbuild a bit in this task, we will ask for the creation of a file named .config to Kbuild itself! This file will store all the configuration options we asked for. To create a .config file, call one of these commands:
make defconfig: create a default .config file with all the options that suits your current architecture.make config/make nconfig/make menuconfig: used to interactively change kernel's options.make oldconfig: create a .config file using the one currently available. Useful if you want to build a new kernel with all the options used in an older one!make randconfig: randomly enable / disable options. Used to test purposes.make tinyconfig: create the tiniest kernel possible.make kvmconfig: update the current configuration with KVM support.
These are enough for us, but you can find more configurations with 'make help'. And now:
make
It should not crash this time ;) Feel free to append -j <THREADS> to speed up the build process, with <THREADS> being a placeholder for the number of threads you want to use for the build.
Now you may be wondering: wait, I see no warning message! As we created a .config file, the kernel can now be built, and it builds, I promise you! But, by default, Kbuild won't display any warning message from the compiler or any other tool invoked. It's now time to present your Kbuild's "blabbermouth" option! Using W=<LEVEL> with LEVEL equals 1, 2, 3 or a combination of them, it will make it a bit more verbose. W=1 will display rare and relevant warnings, W=2 warnings that happens quite often but still are relevant (maybe) and W=3 will open a gate to Hell and crush your body under hundreds of thousands of warning messages (don't use W=3).
make W=1
This should be just enough for now and it should give you enough warning message for a day or two.
Checkpatch and its magic
checkpatch.pl is a Perl script developed by the kernel contributors, for the kernel contributors. It is mainly used to check patches and commits’ coding style before submission but it can be used with files too. This is how to use it (from the repository folder):
scripts/checkpatch.pl --patch PATCH_FILE.patch: check patch file coding style.scripts/checkpatch.pl --git COMMIT[...COMMIT]: check commit's coding style.scripts/checkpatch.pl --file FILE1.c [FILE2.c ...]: check file's coding style.
This script will output a set of CHECK, WARNING and ERROR messages when the code does not comply to the kernel coding style, it should look a bit like:
WARNING: please, no spaces at the start of a line #49: FILE: mm/cma.c:49: + return PFN_PHYS(cma->base_pfn);$ WARNING: line over 80 characters #377: FILE: mm/cma.c:377: + next_zero_bit = find_next_zero_bit(cma->bitmap, cma->count, start
Fixing a coding style mistake is not always as easy as it looks and sometimes different subsystems have different policies regarding coding guidelines (except if explicitly defined by the kernel's coding style itself). However, style patches are hardly ever applied quickly by a subsystem's maintainer(s), but feel free to try, you may have a bit more luck than me!
"TODO" and other readings
Developers generally like to keep their to-do lists inside the code itself or close to it. And so do kernel developers. Find all these TODO with grep "TODO" -i -r --include=\*.{c,h}.
You can also read the source code itself in order to understand, for example, how a driver for a specific device works. If you try to understand how it works, you will definitely find something to fix, be it the error path of a function or an unused variable.
Other places to look
There are more ways to find out how you can contribute to the kernel than the 3 ways described here, so feel free to have a look:
- dmesg output and kernel's oops
- Coccinelle's output
- Sparse's output
- Other static analysis tools ('make help' can give you hints)
- Kernel's test suite
- Linux's Bugzilla
- …
Or maybe you are reading this blog because you have already found a bug and want to fix it, if so, go on!
Coding time
I have not defined C as a prerequisite to follow this guide, I hope it was obvious... So for those who know how to code, show time! For those who don't know, find someone who does and stick like glue to them!
Writing code or fixing a bug in the Linux kernel is a bit different than in a userspace program. Thus, the C standard library is not available; only a subset of these functions is defined in the kernel itself. A lot of simple userspace actions like locking or memory allocation can have a huge impact on the kernel’s behaviour.
That said, a lot of documentation is available for the Linux kernel, from API to BPF via filesystems management, for example. Most of it is in the kernel’s “Documentation” folder, or at kernel.org/doc. It is very important to read this documentation thoroughly, especially the one related to the code you're trying to fix.
Testing is also a very important requirement before committing your changes; the easiest test is to build the kernel. Sending a patch that can't even be compiled is a waste of time for everyone who will try to test it.
Also, you should pay attention to the Kernel's coding rules; not complying with these rules will mean your work is not mainlined. Checkpatch has specifically been designed for such tasks. And remember: special cases aren't special enough to break the rules (yes, that's The zen of Python).
Committing
The last step, at least for this part of the guide, is to commit your changes. First, review your changes and ensure each modification has a reason to be:
git diff
Then, stage changes to prepare your commit:
git add -p
Finally, commit! Using a Signed-off-by commit:
git commit -s -v
There are lots of ways to write a commit message depending on the module you worked on and what you modified. Some rules will be explained here but for a more detailed explanation, see PatchPhilosophy:
- Commit subject (commit message's first line): should be identical to previous commits on the same file. To look at a file history in git, use
git log --pretty=oneline --abbrev-commit FILE_PATH. For example, commits on staging drivers (under development drivers) must be prefixed with staging. - Commits fixing a static analysis error must include a copy of this error, same for checkpatch.pl errors.
- Huge changes (to add a new feature for example) must be split into different commits in order to create a patch series. Unfortunately, this guide would be too long if I had to explain how to send a patch series so...my overriding advice is to try not to change too many things ;)
Also, your commit's content should describe what has been changed and why. This is the only text that will explain your patch to the maintainer who has to read your code and understand all its potential side-effects on an existing code base he or she is accountable for. You have to convince the maintainer that he or she needs your changes, that their tree will be better with your patch than without! Be concise, clear and explain why your patch is good.
What's next?
That was enough new knowledge for one day. Take time to go through all this stuff again if you need to and try to explore the source code as well as the Kbuild system (e.g. How to build in a separate folder? Set configuration from command line? Build only the files you need?)
The third and final part of this guide will detail how to extract a patch from your commit in order to send it to the maintainers!
How ITDev Can Help
As a provider of software and electronics design services, we are often working with the Linux and Android operating systems. We have extensive experience in developing, modifying, building, debugging and bring up of these operating systems, be it for new developments or working with existing kernels to demonstrate new device technologies and features.
We offer advice and assistance to companies considering to use or already using Linux or Android on their embedded system. Initial discussions are always free of charge, so if you have any questions, or would like to find out more, email us or call us on +44 (0)23 8098 8890.
Embedded Linux workshop
We are investigating the possibility of running an 'online' workshop focused on the challenges around embedded Linux later this year. If you would be interested in attending a 'virtual' workshop, sign up to our Embedded Linux Interest Group to be kept informed.
