Monday, August 13, 2012

Automatically backporting the Linux kernel

Users should be able to get support for their hardware if they are on any supported kernel. Linux kernel developers should always prioritize working directly on the Linux kernel and there should be no excuse as to why not do to so. Linux kernel developers should ideally never have to backport anything they do onto older kernels. Linux distributions should share as much as possible if they are already putting some work into backporting. What if we could simply automatically backport the Linux kernel ? What if to automatically backport the Linux kernel we may have to take into consideration some design considerations of the technologies and semantics that would enable this? This  blog post is about all this, where we stand and where we are going.

At the 2012 Linux Collaboration summit in San Francisco I gave a talk about the idea and project objective I have on automatically backporting the Linux kernel. A pictures speaks more than 1000 words so lets try to get directly to shiny pictures and graphs so you have to read less and I can write less. If you haven't yet please go first read my last blog post on optimizing backporting collateral evolutions on the Linux kernel, and then go through the few images below. You can also just review the slides from the automatically backporting the Linux kernel presentation.

This is your Linux backport effort of a collateral evolution on drugs:


If you've been doing backporting or if you've read backported Linux kernel code before though you'll realize that this is what you typically see though. This change in fact would be present for all drivers, and the one above is a change for just one driver. A better idea is to wrap all this code up into a helper routine that could be used over different drivers. Here's what this would look like then:


And this is exactly where we are today with backporting the net_device_ops collateral evolution down to one line for each driver. We do this by implementing the actual helper in a shared module under the project compat. Once we have this implemented you can even express this in SmPL form (see previous blog post on optimizing collateral evolutions) and then we would not have to even code this collateral evolution for other new drivers. In fact, if we added a static inline for netdev_attach_ops() upstream to simply do that assignment to the data structure it would mean we would require 0 lines of code changed to backport this collateral evolution. I really want you to go review the first picture before you make any decisions on how you feel about this.

A few people like the strategy and thanks to all the help of a good large group of contributors today I can say we are striving towards these optimizations slowly. This effort started through the compat and compat-wireless (now renamed to compat-drivers) projects but the number of systems we backport keeps growing and the amount of code we can share keeps growing as well. Below exactly how much code we share is illustrated against the amount of code we are pulling directly from the Linux kernel.


That small green thing -- that's compat, and that red thing between the dark stuff and the green stuff is the delta of patches that we have not yet merged into a generic compat module or are things which we have not yet figured out how to generalize -- this goes into the compat-drivers project. In short: the more we share into a compat module the more subystems we can share at a lower cost and most effort is done automatically.

But not all work is automatic of course. Hauke Mehrtens, Johannes Berg, Bala Shanmugam, Ozan Çağlayan among others have contributed mounds of work to the projects and without their help we would not have such great advancements. Ozan in particular decided to take on backporting the video graphics kernel drivers as part of his Google Summer of Code project and with this making it the 4th subsystem we are working towards integrating into the project, previous to this we only backporte Ethernet, Wireless and Bluetooth subystems. The folks at the Linux Foundation driver backports working group expressed interest in the project at the last Linux Collaboration summit in San Francisco and at their request we will be renaming the project to compat-drivers to put emphasis we are interested in generalizing backporting, and we will eventually be making releases through kernel.org instead of the wireless front. At this point all relevant documentation has been migrated onto the kernel.org backports wiki page, please use that page to advance or read the required documentation. We expect to make the first release through kernel.org as of the v3.7 release cycle.

For now if interested in the project you can refer to the code:
At this point we aim to backport all 4 subystem kernel drivers and making both daily snapshots based on Stephen Rothwell's linux-next.git, and RC releases based on Linus's linux.git sand Greg's stable releases from linux-stable.git and all of these able to compile across 21 older kernels.

If interested in poking around first, you may want to simply clone compat.git and check out the following few scripts:

git clone git://github.com/mcgrof/compat.git
./bin/get-compat-trees
./bin/get-compat-kernels
./bin/ckmake

ckmake is intended to help you cross compile test any external module across 21 older kernels. 

Apart from backporting we realize at times you may need to integrate critical fixes not yet integrated upstream. The project has solution for this that respects and prioritizes Linux upstream development by providing code metrics on each type of delta possible of patches from the birth of a patch all the way up to its integration upstream and the effort required to backport it.

If you want to send patches please subscribe and send patches to the shiny new mailing list: backports@vger.kernel.org. To subscribe send an e-mail with anything on the subject to majordomo@vger.kernel.org with this on the subject:

subscribe backports

Read the following e-mails on a follow up of how to confirm subscription. Please read the new backports kernel wiki page on how you are expected to send patches, but please use the new mailing list.
Post a Comment