HOWTO: Compiling Feisty Kernel with Completely Fair Scheduler (CFS)

[volsungs]

Hi all,

There has been quite a bit of hype about potential 3D and desktop performance with the new Completely Fair Scheduler (CFS), which will be incorporated in kernel version 2.6.23. Word has it that CFS will not be added to Ubuntu Gutsy and some users have expressed disappointment about this decision (http://ubuntuforums.org/showthread.p...&highlight=cfs). I wanted try out this scheduler on my Ubuntu Feisty box, but I didn't find much information on how to go about it. One recent post (http://distrogue.blogspot.com/2007/0...rnel-2623.html) describes how to upgrade to the 2.6.23-rc2 kernel for Ubuntu, but I had serious problems compiling the Nvidia video driver with that kernel and I just want the new scheduler, not all the other changes with 2.6.23 (it's still an unstable kernel, after all).

Fortunately, Ingo Molnar (CFS's developer) has released a patch file for the 2.6.20 tree. This is a brief HOWTO for compiling the CFS into the current stable Feisty kernel: 2.6.20-16.29. This guide assumes that you have installed and are running the aforementioned kernel (since I copy the config file from it below). If not, you'll need to apt-get install two packages: linux-image-2.6.20-16-generic linux-headers-2.6.20-16-generic. NOTE: I have compiled many kernels over the years (particularly on Gentoo), but it's possible that the instructions below may have inaccuracies and I welcome corrective feedback.

If you don't want to compile the kernel yourself (which is what is described below), you can download the .deb files that I built. These were compiled against the 2.6.20-16.29 source for Ubuntu Feisty, and the kernel configuration was the same as the current generic kernel (with kernel debugging turned off), so they should be as usable as the existing stock Feisty kernel.

DISCLAIMER: I accept no liability for the files provided below. They work great on my machine, but I haven't tested them on other computers.

http://www.michaelhallquist.com/ubun...ustom_i386.deb
http://www.michaelhallquist.com/ubun...ustom_i386.deb

To install these .debs, see step 10 below (basically you need to run: sudo dpkg -i <package>).

Thanks to tseliot for putting together a newbie's kernel compilation guide on Ubuntu (http://ubuntuforums.org/showthread.php?t=56835). This guide is based largely on his documentation. Please read his guide before you jump into adding CFS to your kernel.

1) Prepare your box for kernel compilation. Follow the aforementioned guide up to the part where you have installed the necessary tools and have unpacked the kernel source. I skipped the xorg.conf modification from tseliot's guide above because I compiled the Nvidia module myself (see step 11 below). Tseliot's guide is a bit out of date, so you won't need gcc-3.4, and the latest source package is called linux-source-2.6.20. If you've done things correctly, you should now have the kernel source available at /usr/src/linux-source-2.6.20.

2) Obtain the patch from Ingo's site: http://people.redhat.com/mingo/cfs-s...16-v20.4.patch

(NOTE: there's a slightly newer v20.5 patch, but I haven't tested it and v20.4 is only a few days old)

3) Move the patch into the kernel source directory and run the patch against the source like so:

# sudo mv sched-cfs-v2.6.20.16-v20.4.patch /usr/src/linux-source-2.6.20
# cd /usr/src/linux-source-2.6.20
# sudo patch -p1 < sched-cfs-v2.6.20.16-v20.4.patch

4) You'll encounter two errors in the patch. The first is in the /usr/src/linux-source-2.6.20/Makefile and the second is /usr/src/linux-source-2.6.20/kernel/sched.c. These occur because the patch was written for 2.6.20.16 vanilla sources, whereas I think the Ubuntu is a heavily patched 2.6.20.3. You can resolve these simple conflicts yourself (look at Makefile.rej and sched.c.rej) or just download the versions I fixed:

http://www.michaelhallquist.com/ubuntu-cfs/sched.c
http://www.michaelhallquist.com/ubuntu-cfs/Makefile

Replace the old versions with those.

5) Additionally, the 2.6.20 vanilla sources didn't include a definition for arch_enter_lazy_cpu, but the Ubuntu source does (in include/asm-i386/paravirt.h). Ingo's patch adds a null definition for arch_enter_lazy_cpu (assuming vanilla sources), which results in a duplicate definition. If you compile without fixing the duplication, I think it will work (you'll get lots of warnings), but I would instead just comment out line 118 of the post-patch file /usr/src/linux-source-2.6.20/include/linux/sched.h. This line is:

#define arch_enter_lazy_cpu_mode() do { } while (0)

Again, the corrected file can be pulled from:

http://www.michaelhallquist.com/ubuntu-cfs/sched.h

6) Okay, now the source is in good shape to compile. Make sure the source is cleaned of object files and any old config files:

# cd /usr/src/linux-source-2.6.20
# sudo make-kpkg clean
# sudo rm .config .config.old

7) I didn't want to compile from scratch and wanted to stay close to the stock kernel, so I imported the 2.6.20-16-generic config like so:

# sudo cp /boot/config-2.6.20-16-generic /usr/src/linux-source-2.6.20/.config

8) By default, Ubuntu has kernel debugging turned on in the config, which makes the kernel and modules much bigger than they need to be (see also https://wiki.ubuntu.com/KernelCustomBuild). So turn that off. Run sudo make menuconfig, go under "Kernel Hacking" and turn off "Kernel debugging." You can also alter your config while you're in there to turn off SMP, compile just for your architecture, and so on... but that's beyond the scope of this discussion. Or, you can get the config file I used from:

http://www.michaelhallquist.com/ubun...untu-cfs-v20.4

9) Okay, now we're ready to compile. Create the .debs for the image and headers following tseliot's guide:

# sudo make-kpkg --initrd --append-to-version=-custom kernel_image kernel_headers

(you can omit the --append-to-version portion if you like or change it to whatever suits you)

10) Once it compiles (it may take a while), there should be two debs in the /usr/src directory. Mine were called:

linux-image-2.6.20.16-ubuntu-cfs-v20.4_2.6.20.16-ubuntu-cfs-v20.4-10.00.Custom_i386.deb
linux-headers-2.6.20.16-ubuntu-cfs-v20.4_2.6.20.16-ubuntu-cfs-v20.4-10.00.Custom_i386.deb

Whatever yours are called, install the debs with:

# sudo dpkg -i linux-image-<name here>.deb
# sudo dpkg -i linux-headers-<name here>.deb

11) (FOR NVIDIA CARD BINARY DRIVER USERS ONLY) Now, with luck, the new kernel and headers are installed. But wait! I run an Nvidia graphics card and I need a module compiled against the new kernel (If you reboot before you make the module, X windows will not start, which may be uncomfortable for those not used to a command line interface). I have no clue about ATI, so you're on your own if you have an ATI card. You'll need the nvidia-kernel-source package installed. This will give you the /usr/src/nvidia-kernel-source.tar.gz archive. Extract that and compile a module against the new kernel like so:

# cd /usr/src
# sudo tar xf nvidia-kernel-source.tar.gz
# cd modules/nvidia-kernel/nv
# SYSSRC=/usr/src/linux-headers-2.6.20.16-ubuntu-cfs-v20.4/ make module install

Note that the SYSSRC argument above must be set to wherever your kernel source headers were installed from step 10. Now, the new module should be installed in the proper /lib/modules directory. If you do get stranded at a CLI without the driver, login, then type:

# sudo apt-get remove nvidia-glx
# sudo apt-get install nvidia-glx
# sudo /etc/init.d/gdm restart

This should get the driver up and running for you.

12) Reboot the computer and choose the new CFS-based kernel (should have cfs-v20.4 at the end)! The new kernel worked great for me and I've had no problems. I haven't observed drastic responsiveness improvements over the old scheduler, but I haven't really put the kernel through its paces, either. You can verify that CFS is running by looking in the /proc/sys/kernel directory. If you have CFS running, you'll see something like this:

# ls /proc/sys/kernel

sched_batch_wakeup_granularity_ns sched_features sched_min_granularity_ns sched_stat_granularity_ns
sched_child_runs_first sched_latency_ns sched_runtime_limit_ns sched_wakeup_granularity_ns

All those sched parameters are for CFS. I don't know what tweaking them would do, so I'd leave it at the default for now.

I hope this guide proves useful to others interested in trying out the new Completely Fair Scheduler!

- Michael