Low-Latency HOWTO

by Paul Winkler 9/11/2000
Maintained by Patrick Shirkey
additions by Patrick Shirkey, Eric Rzewnicki, Mark Knecht

Last update Monday 27 October 2008 08:57

For the latest kernel patches and to stay informed about the bleeding edge of realtime kernel lowlatency development visit the Realtime Low Latency Wiki.

For most users the latest 2.6 series kernels are sufficient for low latency. However there are a few things you have to configure. For example make sure you have “Generic PCI bus-master DMA support” on in the kernel, run your root filesystem on ext2 and partition your discs so that your software is not stored on the root filesystem partition. It's also a good idea to have the disc you record to running on a seperate ide bus to the root filesystem disc and you definitely want to mount /tmp to tmpfs.

none /tmp tmpfs defaults 0 0

For audio you will want to use JACK and associated applications. Another important part of latency is getting your interrupt (IRQ) settings correct.

Contributed by Mark Knecht.

This is a simple, or difficult, subject. Due to the history of the PC architecture, PC's first had 1 interrupt controller, and then with the AT architecture, added a second. This accounts for the sort of strange numbers I'm going to give you next.

Here's the order of interrupt priority on a non-APIC machine:

0, 1, 8, 9, 10, 11, 12, 13, 14, 15, 3, 4, 5, 6, 7

0 is the highest priority
7 is the lowest priority

Potentially 9, 10, 11, 3, 4, 5, 6, 7 are made available to PCI slots on many motherboards. I won't go into why the order is this way, but it is. You MUST look at your SPECIFIC motherboard to know what's available to you. Many newer mobos allow you to change the irq settings for the pci interface in the BIOS. This is also useful for USB devices as the usb IRQ generally tags along with one of the pci slots. So you may be able to change the IRQ for your USB card by adjusting the the BIOS settings for the PCI slots. If you cannot adjust the settings in the BIOS your other option is to juggle the cards around the available slots. (NB - requires opening your case)

In my experience all audio cards want to be on IRQ's 9, 10 or 11. It isn't actually that important WHAT IRQ a card is on, but it IS IMPORTANT what devices are on higher IRQs. For instance, if Steve's audio card is on IRQ 11, and nothing is on IRQ 9 or 10, then no problem. But if Steve was to insert a PCI graphics adapter on IRQ9, his audio card could start having trouble because the graphics adapter would have higher priority.

Again, I won't go into all the reasons why, but just assume that any device with a higher priority number (from the list above) gets in the way of devices with lower priorities.

One last thing about non-APIC, interrupts may be shared. Just because your card is on IRQ9 doesn't mean that another device isn't sharing IRQ9. What matters in this case is how many interrupts the other device is going to generate, and how well it's driver is written.

There are a few ways to do this. The easiest is to

cat /proc/interrupts

But you can get more information from running

lspci -v

If you want to find the IRQs for your usb device you need to look for the usb bus controller and then compare its number with the results from.

lsusb

The kernel must be configured to assign different vector numbers to specific interrupts. The standard kernel cannot do this but Clemens Ladisch has written a patch which adds this capability.

• In an APIC system lower vector numbers have higher priorities.
• The priority level is determined by the upper four bits of the vector number.
• The system timer is redirected to interrupt 2.
• Vector number 00 is reserved, so these entries are unused.

This document is specifically aimed at those interested in using 2.4.x with Andrew Morton's low-latency patch for audio purposes.

It's not really hard when you know what to do, but there's a lot of stuff to upgrade before you can compile and install the kernel if you are moving from linux 2.2.x. So if you are, don't forget to read the section on updating your system files.

We suggest scanning through this whole document to get an overview before you start mucking with the kernel. Everything described here requires kernel configuration, and it's easier to do it all in one go rather than rebuilding the kernel again for each feature you want. You should also read through the necessary upgrades to make sure you have all your software in order before you start configuring.

1. Download Andrew Morton's latest low-latency patch.
2. Download the 2.4.x sources of the right version for Andrew's patch. When I say x, I mean for example get linux-2.4.17 if you have a Morton patch for 2.4.17. Kernel sources can be found at ftp.kernel.org and various mirrors.
3. It's good to keep your sources organized. /usr/src/linux is probably a link to your current kernel sources, something like this:

   $ ls -l
   total 4
   lrwxrwxrwx    1 root     src            12 Jul 22 13:46 linux -> linux-2.2.16
   drwxr-xr-x   14 root     root         4096 Jul 21 22:59 linux-2.2.16

   If that's the case, remove the link:

   rm /usr/src/linux

   Otherwise, if /usr/src/linux is a directory and not a symlink, rename it so the name includes the kernel version, e.g.:

   mv /usr/src/linux /usr/src/linux-2.2.16

4. Now unzip the 2.4.x source tarball and untar it in /usr/src/ . This will result in a /usr/src/linux directory. Rename this /usr/src/linux-2.4.x where x is the proper number.

   mv linux linux-2.4.x

5. Re-create /usr/src/linux as a link to your new sources:

   ln -s /usr/src/linux-2.4.x /usr/src/linux

6. Now apply Andrew's patches:

   cd /usr/src/
   patch -p0 < /dir/where/you/put/2.4.x-low-latency.patch

   This should patch without any problems. However some people have found that this works better:

   cd /usr/src/linux
   patch -p1 < /dir/where/you/put/2.4.x-low-latency.patch

   or

   gzip -cd patchXX.gz | patch -p0

   YMMV

1. Configuration may be easier if you copy your old kernel configuration to your new source directory:

     cp /usr/src/linux-2.2.x/.config /usr/src/linux

2. To save a copy of your old kernel config as .config.old and create a .config updated for the current kernel:

    make oldconfig

3. In /usr/src/linux, do make config, make xconfig or make menuconfig (your choice). You'll want to read /usr/src/linux/Documentation/Changes first! Options relevant to low-latency audio:
   • First thing you want to do in kernel configuration is get access to the experimental stuff. To do so, go to the first item - “Code maturity level options” - and say Y to “Prompt for development and/or incomplete code/drivers.”
   • Low-latency is enabled via the “Processor Type and Features” menu or x86. To enable “Control low latency with sysctl” first you have to enable “Low latency scheduling” above it
   • There is a new configuration option “Control low latency with sysctl”. If you enable this, you can turn the low-latency capability on and off with /proc/sys/kernel/lowlatency. The default value is zero - low-latency disabled or 'off'.

                cat /proc/sys/kernel/lowlatency

     should yield “1” or 'on'. if not

                echo 1 > /proc/sys/kernel/lowlatency

     to turn it on. Building your kernel with “Low latency scheduling” and “Control low latency with sysctl” will give you a /proc/sys/kernel/lowlatency file which you can echo 0 or 1 to to disable or enable low latency patches dynamically in the kernel. This is usually done through /etc/sysctl.conf:

        #
        # /etc/sysctl.conf - Configuration file for setting system variables
        # See sysctl.conf (5) for information.
        #
        kernel/lowlatency = 1

     This file is executed by /etc/init.d/procps on startup or can be run manually as root with 'sysctl -p'. If you build your kernel only with 'Low latency scheduling' then low latency will be enabled by default and there will be no way to disable the low latency patches short of recompiling your kernel. If you want to double check if your kernel is patched with the low latency patches, I'm assuming Andrew Morton's patches, either repatch or check for existence of /include/linux/lowlatency.h.

   • If you have IDE drives, go to the “ATA/IDE/MFM/RLL support” section. In a subwindow you will find “IDE, ATA and ATAPI Block devices” section. Say Yes to “Generic PCI bus-master DMA support” and “Use PCI DMA by default when available”. This will turn on DMA for your disks which is necessary for low-latency disk-intensive work. You may also want to browse this section for any options related to your specific motherboard/chipset.
   • Next, go to the “Character devices” section and enable “Enhanced Real-time Clock Support” for access to your computer's hardware clock.
   • Turn on sound support. In the “Sound” section, you want to select “M” for “Sound support”. Do NOT select “Y” for “Sound support.” It's supposed to work either way, but some users have occasionally had trouble with non-working drivers or missing functionality when selecting “Y” for “Sound Support”. Save yourself the hassle and select “M”.
   • Still in the “Sound” section, select “N” or “M” for EVERYTHING ELSE. Say “M” only to those modules you want to build OSS support for. If you're using ALSA, you don't need any of these at all. They won't prevent ALSA from working as long as they're not loaded; but they might confuse you when you're trying to get ALSA working: you can't load ALSA and OSS modules for the same card at the same time. Similarly, if you are using USB audio or midi devices, you will want to say Y to “Preliminary USB device filesystem”. If you want to use the ALSA driver for it, be sure to disable “USB Audio support” (or set it as a module) in the USB section (not in the Sound Support section). You'll then need to get ALSA sources from www.alsa-project.org and follow the installation directions. Be sure to get the 0.9.x series; ALSA 0.5.x is ancient and should be ignored, in spite of its “stable” status.
   • Journalling filesystems are great because they log all disk transactions and are very tolerant of sudden power loss (either accidental, or because you pulled the plug on purpose to get out of a frozen system). This is useful for anybody who runs buggy programs with high priority and root privileges… namely, just about anyone running high- performance audio on linux. It's especially important if you're running in a live performance or broadcast situation where you need to be able to recover from just about any catastrophe quickly. There are a couple of choices of journaling filesystem available. I've tried reiserfs and ext3 with good success. Currently I'm using ext3 because it's a lot easier to set up, being backwards-compatible with ext2, which means you can convert an existing filesystem and later, if necessary, boot with a kernel that only recognizes ext2. So you don't need to be sure to always have an ext3-enabled boot floppy for emergencies. During kernel configuration, go into the “File systems” section of the kernel configurator. Say Y to “Ext3 journalling file system support” and/or “Reiserfs support”. While you're at it, turn on support for anything else you're likely to need: ISO 9660 for CD-ROMs, Microsoft Joliet CDROM extensions, DOS FAT fs, VFAT fs, anything else you like. I usually say Y instead of M to all filesystems I want to support, because I want to be sure I can mount anything even if there's a problem with modules. It's happened… I'll assume you're using ext3 like me. Next, you want to read this quick little howto, which tells you everything else you'll need to know about running with ext3: http://www.symonds.net/~rajesh/howto/ext3/ext3-5.html Note that you DON'T need to patch your kernel if you have a recent 2.4.x Linux kernel source tree. Ext3 now comes standard, though it's still labelled “experimental.”
   • It's possible, when running the latest multimedia application or game, to suddenly find your system totally unresponsive. Journaling filesytems will help you avoid fsck and possible data loss, but it's even better if we can find a way to reboot gracefully, or even get the s Often, the “Magic SysReq” key combinations can save you. To enable them, go into the “Kernel Hacking” section of the kernel configuration. (Don't let the name scare you.) Say Y to “Magic SysRq key”. Read /usr/src/linux/Documentation/ sysrq.txt for instructions; more importantly, PRINT IT since you won't be able to get at the file when you need it most!!! If you forget to print it Alt-SysRq-q might help you (but probably not when you're running X). q is currently not a valid sysrq key so using it causes sysrq to print a usage message with a list of the available commands. The appropriate letter for each is given in uppercase. This is a simple “cheat sheet” I've used in the past when I can't find my printed copy of sysrq.txt. This may not work in the future if q becomes used by sysrq.

     Here's a particularly useful sequence:
     Alt-Sysrq-k    - kills all programs on current console
     Alt-Sysrq-e    - Drops to a login terminal
     Alt-Sysrq-s    - attempts to sync all mounted filesystems
                      (wait a moment after this... if you have a working
                       console, you should see an "OK" message when it's done)
     Alt-Sysrq-u    - attempts to remount all filesystems read-only
     Alt-Sysrq-b    - force immediate reboot without sync or unmount

     By doing this, you can greatly reduce the chance of filesystem damage, and even avoid needing to fsck on reboot. I find that if the freeze happens while I'm in X, I'm usually unable to get the screen back, but I can do a safe reboot. Note that some of the key combinations seem to work with one Alt key and not the other… but it's not consistent, I don't know why. Try both Alt keys.

4. Most likely, you have an IDE CDROM or CD-R or CD-RW drive. If you want to exercise your rapidly-diminishing fair use rights under U.S. law and make a copy of something in your CD collection for your own personal use, you'll need to do a few things before you can “rip” music from an audio CD. As it happens, exactly the same setup is necessary to support “burning” your own data or audio CDs. You first need to enable SCSI emulation so that the system treats the IDE drive as if it were a SCSI drive. During kernel configuration: First, go into “ATA/IDE/MFM/RLL support”. In the subsection for “IDE, ATA and ATAPI Block devices”, make sure to choose N for “Include IDE/ATAPI CDROM support”. Then choose Y or M for “SCSI emulation support”. Go back to the main menu, enter SCSI support, and be sure to say Y to “SCSI support”, Y to “SCSI CD-ROM support”, and Y to “SCSI generic support”.
5. (Optional step) Edit line 4 or your kernel Makefile to set an Extraversion. I find this useful for tracking various compiles of the same kernel version in case I make a mistake or want to test different configuration options. I usually use the hostname and a sequence number in addition to indicating any patches applied:

     vi Makefile
     ...
     EXTRAVERSION = -ll-box1-0

6. The 2.4 kernel uses a new device file for shared memory. It's a good idea to use this, so add this to /etc/fstab. Put this line in /etc/fstab:

   none                /dev/shm              tmpfs     defaults      0 0

   And make the directory: mkdir /dev/shm
   It won't affect booting an old linux 2.2 kernel - you'll just get a complaint that the kernel does not support the shm filesystem. While you are editing /etc/fstab you can also mount your /tmp/jack directory in RAM. This solution reduces the audio buffer overruns and therefore greatly improves JACKs performance if you are using a journalling filesystem (as recommended above) but you will loose all data in this dir on shutdown. Other *nix's use this approach.

   none                /tmp/jack              tmpfs     defaults      0 0

   There is more information on this in the JACK FAQ.

7. Now build your kernel:

     make dep
     make clean
     make bzImage

   Next check to see that the compile succeeded. There is nothing more frustrating than removing the files below before you find out that there was a problem you didn't notice while compiling. Usually compile problems are because of missing module dependencies. Often the output will look normal but you will know for sure there was something missing from the config if you don't have a System.map. If you do then proceed below, if you don't then you need to hunt down the missing dep. Sometimes the location of the dependency changes between kernel versions. If you are using a 'pre' kernel you may even have to to patch a file by hand.

     ls System.map (check to see that the compile succeeded)
     rm -f /boot/System.map
     cp System.map /boot/System.map
     rm -f /boot/bzImage
     cp arch/i386/boot/bzImage /boot/bzImage-2.4.x

8. Update LILO. Edit your /etc/lilo.conf file and put in a new image entry:

   image=/boot/bzImage-2.4.x
           label=linux-2.4.x
           read-only
           root=/dev/whatever_you_use

   If you also want to use your cdwriter, or rip audio from a CD, there's one more thing to add in lilo.conf before you'll be able to use the drive:

   append="hdX=scsi hdX=ide-scsi"

   … replacing X with the drive letter of your IDE cd drive, e.g. my CD drive is /dev/hdc. (I'm not clear on which of those options is actually needed, I think it may have changed at some point. I have both options, which seems to be safe - the invalid one is ignored.) If you already have an “append” line in lilo.conf, just add the text in quotes.

9. Now run LILO so it finds the new image:

   /sbin/lilo

   As always, running LILO is VERY important. You won't be able to boot the new kernel until you do so. This is especially important if you ever remove your old kernel!

10. Finally!! Shut down and boot your new kernel.
11. Extra step for CDWriting support - After rebooting the new kernel, the only remaining step is (as root):

    rm /dev/cdrom
    ln -s /dev/srN /dev/cdrom  
    OR
    ln -s /dev/scdN /dev/cdrom

    (either one should work) Once again, replace N in these commands with the appropriate number. In my case, I have no real SCSI devices, so I choose /dev/sr0. Be sure that your /etc/fstab has an entry for your CDROM device that points to the correct device. The device can either be /dev/cdrom if you've updated that symlink, or it can be /dev/srN as described above. Now that you've got everything ready, try mounting a data CD to make sure that still works. Also try playing an audio cd. Next, try ripping and/or burning. Useful tools include cdrecord and cdparanoia. More information can be found in the CD-Writing HOWTO, which may be in your installed documentation, or check here: http://www.tldp.org/HOWTO/CD-Writing-HOWTO.html

There's a few things you need to do before you can load modules with 2.4.

1. First, if you are running an old modutils version (probably not these days), upgrade modutils.

   insmod -V

   See below for the correct version number. Hopefully you won't have to, but if you do then you need to download the modutils package. Compile and install it now, or use your distribution's latest available package. Make sure your old modutils is gone!

2. If your module configuration is in /etc/conf.modules, be advised that this name is deprecated - you should rename it to /etc/modules.conf . Don't worry, this won't harm anything with your old 2.2 kernel. Note that some linux distributions, e.g. Debian, do not advise editing /etc/modules.conf directly; instead, you edit or add files in /etc/modutils and run a script called update_modules which generates /etc/modules.conf for you. This allows you to organize things a little better.
3. Now you should build your kernel modules.

   cd /usr/src/linux
   make modules; make modules_install

4. Try to load a module using modprobe or insmod … hopefully it should work! You might need to run depmod -a as root. If there's a problem, double-check your configuration for that module in /etc/modules.conf.

The above directions will work just as well as the following, but this method using make-kpkg can save some work for Debian users. Starting after the kernel configuration:

1. Install kernel-package

   apt-get install kernel-package

2. Build a kernel and pack it into a .deb file

   make-kpkg clean
   make-kpkg kernel-image

   or, if you want to time the kernel build so you'll know how long it takes if you need to do it again:

   time sh -c "make-kpkg clean; make-kpkg kernel_image"

3. Install the kernel .deb package

   cd ..
   dpkg -i linux-2.4.x-ll-box1-0.deb

   The kernel package will take care of configuring and running LILO. It also handles the modules. For more information on the organization of kernel package management in Debian see the dpkg and make-kpkg man pages.

4. Shut down and boot your new kernel.

Now you're finally ready to get your soundcard working again. I won't go into detail on this because ALSA installation and configuration is well covered elsewhere. Go to The ALSA howto. I didn't have to do anything differently with the 2.4 kernel; ALSA just worked fine!

There are a number of packages you may need to upgrade. These are listed in /usr/src/linux/Documentation/Changes. The listing below is taken from Documentation/Changes for a 2.4.17 kernel.

Current Minimal Requirements
============================

Upgrade to at *least* these software revisions before thinking you've
encountered a bug!  If you're unsure what version you're currently
running, the suggested command should tell you.

Again, keep in mind that this list assumes you are already
functionally running a Linux 2.2 kernel.  Also, not all tools are
necessary on all systems; obviously, if you don't have any PCMCIA (PC
Card) hardware, for example, you probably needn't concern yourself
with pcmcia-cs.

o  Gnu C                2.95.3                # gcc --version
o  Gnu make             3.77                  # make --version
o  binutils             2.9.1.0.25            # ld -v
o  util-linux           2.10o                 # fdformat --version
o  modutils             2.4.2                 # insmod -V
o  e2fsprogs            1.25                  # tune2fs
o  jfsutils             1.0.12                # fsck.jfs -V
o  reiserfsprogs        3.x.1b                # reiserfsck 2>&1|grep reiserfsprogs
o  pcmcia-cs            3.1.21                # cardmgr -V
o  PPP                  2.4.0                 # pppd --version
o  isdn4k-utils         3.1pre1               # isdnctrl 2>&1|grep version

Additional test commands that take advantage of your distribution packaging system:

• For gcc (egcs >= 1.2 should also work!):

  gcc --version
  rpm -q egcs gcc
  dpkg -l gcc

• For binutils:

  ld -v
  rpm -q binutils
  dpkg -l binutils

  Notes from Documentation/Changes: “If you can, upgrade to the latest 2.9.5 binutils release. Older releases such as 2.8, 2.8.xx, and the FSF's 2.9.1 should be avoided if at all possible. The later releases of 2.9.1.0.x (anything where x >= 22) can and do compile the kernel properly, but there are many benefits to upgrading to 2.9.5 if you're up to it.” Note: there is a problem with later 2.4.x kernels and recent binutils versions. This bit me with kernel 2.4.16 and binutils 2.11.92.0.12.3. Depending on the kernel configuration building the kernel fails with this error:

  undefined reference to 'local symbols in discarded section .text.exit'

  There are several workarounds to be found in various mailing list archives. Using binutils version 2.11.92.0.10 instead of 2.11.92.0.12.3 worked for me.

• util-linux Test commands:

        chsh -v
        rpm -q util-linux
        dpkg -l util-linux

  Note that you *can* compile util-linux yourself, but this is potentially dangerous - you MUST carefully read the MCONFIG file and make sure everything's right. I tried to compile it and didn't know what I was doing, and I made it impossible to log in to my system!! After some hairy battling with broken boot floppies, I managed to get my system working again. I have since used only prepackaged versions with no trouble.

• modutils Test commands:

        insmod -V
        rpm -q modutils
        dpkg -l modutils

  Installation Notes: If you want to compile modutils, Wait until after you've compiled and booted your 2.4 kernel before you try to compile modutils! I was unable to compile modutils until after I'd booted 2.4; then it went fine. This means also that you should NOT compile anything ESSENTIAL to your 2.4 kernel as a module, because you won't be able to load the modules the first time you boot 2.4.
  I had modutils-2.1.121-4 previously, and this version WILL NOT load modules for a 2.4 kernel.

• e2fsprogs Test commands:

        /sbin/tune2fs --version
        rpm -q e2fsprogs
        dpkg -l e2fsprogs

• pcmcia-cs (only if you have PCMCIA devices …) Test commands:

        cardmgr -V

• PPP Test commands:

        pppd --version
        rpm -q ppp
        dpkg -l ppp

• isdn4k-utils (only if you need ISDN. I don't, so I have no experience with this.) Test commands:

        isdnctrl 2>&1|grep version

Questions, comments, corrections: Linux audio users.