Kernel Module Debugging

From OpenFlow Wiki

Debugging the kernel module

To debug the kernel module you need to maintain all the debugging information in the module and do NO optimizations else when you step through it it jumps all over the place. The flags used for compiling OpenFlow can be specified in the command line of a configure, and are also held in configure.ac. Example configure for debugging:

./configure --with-l26=/lib/modules/`uname -r`/build CFLAGS="-g3 -ggdb3 -O0 -fno-inline"

One option for debugging is running VMware Workstation with a VM that matches the host's environment, and putting the OpenFlow code in an NFS mount that is available to both the VM and the host. This allows you to build on the host, and use something like tcpreplay to replay packet flows into the VM. If it crashes you can roll back to a snapshot, saving massive amounts of time. A few random notes for setting something like this up:

• For the tcpreplay nic use Host only networking, and ensure that the vmnetX interface has +rw permissions from the user who launched workstation else the VM cannot put its NIC into promiscuous mode to capture all packets being replayed.

• To setup NFS over a VM that is using NAT you must add the following to the nat.conf in /etc/vmware/vmnetX/nat/nat.conf

[privilegedUDP]
autodetect = 1

[privilegedTCP]
autodetect = 1

• To enable GDB on the image, edit the .vmx file and add (for 32 bit installs)

debugStub.listen.guest32=1

• Download the debuginfo for your kernel, CentOS debug info can be had from http://debuginfo.centos.org/5/i386/,

which will put the vmlinux file into /usr/lib/debug/lib/modules/`uname -r`/

% gdb
(gdb) file vmlinux
(gdb) target remote localhost:8832

• To locate a source file line number with a crash frame+offset perform the following:

EIP: [<e0bbcccf>] dp_frame_hook+0xf0/0x108 [openflow_mod] SS:ESP 0068:c0774f84
 <0>Kernel panic - not syncing: Fatal exception in interrupt

(gdb) info line *dp_frame_hook+0xf0
Line 551 of "/mnt/openflow/workspace/openflow/datapath/linux-2.6/datapath.c"
   starts at address 0xccf <dp_frame_hook+240>
   and ends at 0xcde <dp_frame_hook+255>.

• Here is a command log of what it took to get crash working on a CentOS machine so that kernel crashes were dumped and then could be opened in gdb to determine the crash point. [todo edit this for clarity]

  215  vim kdump.conf 
  216  vim /etc/grub.conf 
  217  shutdown -r now
  218  cd /var/log
  219  ls -la
  220  grep -HinR panic *
  221  grep -HinR openflow *
  222  vim messages
  223  ls -la
  224  vim faillog 
  225  vim dmesg 
  226  cd /
  227  ls -la
  228  cd boot
  229  ls
  230  cd ..
  231  cd tmp
  232  ls
  233  cd ..
  234  lkcd
  235  which lkcd
  236  kdump
  237  man kdump
  238  which kdump
  239  kdump -h
  240  vim /var/log/messages
  241  vim /var/log/dmesg 
  242  cd /etc
  243  ls
  244  vim kdump.conf 
  245  ls /var
  246  ls /var/crash
  247  ls
  248  ls -la /var/crash
  249  man crash
  250  cd /var/crash/127.0.0.1-2008-05-07-13\:21\:15/
  251  cat /etc/grub.conf | more
  252  find /usr/ -name vmlinux
  253  find /usr/ -name vmlinz
  254  find /usr/ -name vmlinuz
  255  uname -a
  256  crash /boot/vmlinuz-2.6.18-53.1.14.el5PAE vmcore
  257  yum search kernel | grep -i debug
  258  yum install kernel-debug
  259  ls /boot
  260  vim /etc/grub.conf 
  261  shutdown -r now
  262  yum search kernel-debug
  263  find /usr/lib -name vmlinux
  264  find /usr/lib -name vmlinuz
  265  find /usr/lib -name vmlinu*
  266  find /usr/lib -name vmlinu*
  267  history
  268  yum search kernel-PAE-debug
  269  yum search kernel | rep -i PAE
  270  yum search kernel | grep -i PAE
  271  yum install kernel-PAE-debug
  272  yum install kernel-debug-PAE
  273  vim /etc/grub.conf 
  274  shutdown -r now
  275  man yum
  276  vim /etc/grub.conf 
  277  crash /boot/vmlinuz-2.6.18-53.1.14.el5PAE vmcore 
  278  cd ~
  279  cd ~
  280  wget http://debuginfo.centos.org/5/i386/kernel-PAE-debuginfo-2.6.18-8.1.10.el5.i686.rpm
  281  rpm -i kernel-PAE-debuginfo-2.6.18-8.1.10.el5.i686.rpm 
  282  rm kernel-PAE-debuginfo-2.6.18-8.1.10.el5.i686.rpm 
  283  uname -a
  284  wget http://debuginfo.centos.org/5/i386/kernel-debuginfo-common-2.6.18-53.1.14.el5.i686.rpm
  285  wget http://debuginfo.centos.org/5/i386/kernel-PAE-debuginfo-2.6.18-53.1.14.el5.i686.rpm
  286  rpm -i kernel-debuginfo-common-2.6.18-53.1.14.el5.i686.rpm 
  287  rpm -i kernel-PAE-debuginfo-2.6.18-53.1.14.el5.i686.rpm 
  288  crash /boot/vmlinuz-2.6.18-53.1.14.el5PAE vmcore 
  289  cd /var/crash/127.0.0.1-2008-05-07-14\:47\:16/
  290  crash /boot/vmlinuz-2.6.18-53.1.14.el5PAE vmcore 
  291  crash
  292  ls /dev/crash
  293  man cras
  294  man crash
  295  man crash
  296  cd ~
  297  ls
  298  cd temp
  299  ls
  300  mkdir temp
  301  cd temp
  302  tar xvzf /boot/vmlinuz-2.6.18-53.1.14.el5PAE 
  303  bzimage
  304  pwd
  305  cd /var/crash
  306  ls
  307  cd 127.0.0.1-2008-05-07-14\:47\:16/
  308  ls
  309  crash /usr/lib/debug/lib/modules/`uname -r`/vmlinux vmcore 
  310  cat /etc/grub.conf 
  311  cat /etc/kdump.conf 
  312  crash /usr/lib/debug/lib/modules/`uname -r`/vmlinux vmcore