Debugging Kernel Modules with GDB 7

When you're debugging your Linux kernel, every try to browse into a module? That only works well if you can read assembly easily. GDB has debug info for the linked-in part of the kernel, but being separately-compiled entities, modules are a big unknown. There is a command in GDB to fix this: add-symbol-file, which informs GDB of the symbols in the given file. You just have to tell GDB where that file lives in the process's address space.

Figuring that out isn't necessarily straightforward. VMware's solution (from their tech note Replay Debugging on Linux) isn't very satisfactory. To paraphrase, you need to go into the guest, examine files in /sys/modules/<foo>/sections to extract the module's segment offsets, and then build an appropriate GDB command line from that. That's fine, unless your load addresses change every now and then (preventing you from hard-coding this into your GDB init script). And of course the guest system isn't exactly accessible when a debugger is attached.

I thought I'd get around this issue by having GDB look at the running kernel to figure out where it should locate the module. Linux keeps a linked list of all loaded modules in a list named modules. So GDB just needs to traverse this list until it finds the named module, then extract the info and build its own command line. Sounds easy, right?

I fought with GDB's scripting language to implement this, and it won. The killer: I couldn't figure out how to do a string compare in it. Anyway, I shelved this idea and waited. Since then, GDB 7 has been released with support for Python scripting, and I couldn't help but take another look at this problem.

The script below adds the new GDB command add-kernel-module. It takes one argument, the file name of the kernel module to load. The code does about what I mentioned above: it walks kernel data structures, extracts relevant info, and builds a GDB command line from it. It's not very pretty code, but it works.

Be sure to name this "vmlinux-gdb.py" so that it can be automatically loaded when you debug your kernel.

import os.path

long_type = gdb.lookup_type('unsigned long')

# These functions are counterparts to the Linux kernel macros
def offset(gdb_type, member):
    '''Return the offset of a member within a structure.'''
    ptr = gdb.Value(0).cast(gdb_type.pointer())
    return ptr[member].address.cast(long_type)

def list_entry(list_head_ptr, gdb_type, off):
    '''Return the base element that holds a list_head struct.'''
    base = list_head_ptr.cast(long_type) - off
    return base.cast(gdb_type.pointer())

def listhead_iter(list_head, typename, member):
    '''Iterate over elements of a list_head.  The first argument should
be a *reference* to the main list_head.'''
    type = gdb.lookup_type(typename)
    off = offset(type, member)
    pos = list_head['next']
    while pos != list_head.address:
        yield list_entry(pos, type, off)
        pos = pos['next']

class AddKernModCmd(gdb.Command):
    '''Load symbols for a currently-running kernel module.'''

    def __init__(self):
        super(AddKernModCmd, self).__init__('add-kernel-module',
                                            gdb.COMMAND_FILES,
                                            gdb.COMPLETE_FILENAME)

    def invoke(self, filename, from_tty):
        if filename == '':
            print 'USAGE: add-kernel-module FILE.ko'
            return
        if not os.path.isfile(filename):
            if os.path.isfile(filename + '.ko'):
                filename += '.ko'
            else:
                print 'Could not find module file %s' % filename
                return
        base = os.path.splitext(os.path.basename(filename))[0]
        try:
            frame = gdb.selected_frame()
            modules = frame.read_var("modules")
        except:
            print 'A running kernel must be attached in order to get section information'
            return
        for mod in listhead_iter(modules, 'struct module', 'list'):
            if mod['name'].string() == base:
                # Found it!  Now, get the section attributes
                sect = mod['sect_attrs']
                num_sect = int(str(sect['nsections']))
                attrs = [ sect['attrs'][i] for i in range(num_sect) ]
                adict = dict([ (a['name'].string(), str(a['address']))
                               for a in attrs ])
                cmd = 'add-symbol-file %s %s' %(filename,adict['.text'])
                for sec in ('.bss', '.data', '.init.text'):
                    if sec in adict:
                        cmd += ' -s %s %s' % (sec, adict[sec])
                gdb.execute(cmd)
                return
        print 'Module %s not found' % base

AddKernModCmd()

"Fixing" my PowerBook's Lid

I love my little PowerBook G4, but it's starting to show its age. In particular, the case itself has taken some wear over the years. Dings in the aluminum don't matter that much, but one piece of damage has seriously affected its usability.

Macs have always worked well with power management. You close the lid and it goes to sleep. When you open the lid again, everything just pops up. It's power management without thinking. Macs were doing this before Windows thought to copy them and before Linux knew how to resume. (Suspend is easy; it's brining things back that's tricky.)

Anyway, this handy feature worked fine, up until my Mac wouldn't stay shut. There is a little spring-loaded magnetic hook that would pop down from the top of the case when the lid was close to the base, and that would hold the laptop shut. That hook broke off earlier this year. So the case will naturally rest right on the edge of the open/shut sensor. If I travel with it in my backpack, it goes in and out of sleep every time I shift my weight. It's strange picking up my bag and then hearing the beep that tells me I have new mail. You can imagine what this does to my battery life.

I finally figured out how to "fix" my PowerBook without disassembling the case. There's a simple command line that will prevent the laptop from waking up when the case is opened:

sudo pmset -a lidwake 0

After running this, my laptop won't wake unless I hit the power button on the keyboard. The pmset command is a utility for changing power settings. It performs the same changes that the Energy Saver preference pane does, plus a few. The "lidwake" setting doesn't show up in Energy Saver, but you can still set it directly.