Gdb

References

• 7.2 Example Debugging Session: Segmentation Fault Example (unkownroad.com)
• Debugging with GDB (delorie.com)
• GDB commands (tutorialspoint.com)
• An Interactive Guide to Faster, Less Frustrating Debugging
• Norm Matloff's Debugging Tutorial (also guide to fast editing and gui debuggers DDD, GVD)
• 8 gdb tricks you should know (oracle.com)

GDB front-ends

Reference list:

• GDB Front Ends
• How to highlight and color gdb output during interactive debugging?

Ideally front-ends must use the GDB/MI2 interface. There is also the built-in Text User Interface to GDB (C-x C-a: http://davis.lbl.gov/Manuals/GDB/gdb_21.html

Good candidates

Mainly from SO post above:

• .gdbinit
• GDB Dashboard (my fork)
• pwndbg
• voltron

Vim:

• https://github.com/vim-scripts/Conque-GDB

Use old shell implementation, not the new neovim terminal.

• http://pyclewn.sourceforge.net/

Read some comment it was not convenient, and would require extra plugin like https://github.com/notEvil/vim-debug.

Neovim:

• https://github.com/critiqjo/lldb.nvim

Only for LLDB.

GDBInit hacks

• Using colout one may color almost any gdb output.

Old front-ends

These are using the old "annotation" mechanisms:

• DDD
• KDBG
• cgdb (text mode with vi keymap)

GDB configuration

GDB reads file ~/.gdbinit at start.

Some references:

• http://stackoverflow.com/questions/209534/how-to-highlight-and-color-gdb-output-during-interactive-debugging

Bare minimum configuration

From StackOverflow [1]:

set history save on
set print pretty
set output-radix 16
set height 0

Install Python modules for GDB

From pwndbg [2]:

# Find the Python version used by GDB.
PYVER=$(gdb -batch -q --nx -ex 'pi import platform; print(".".join(platform.python_version_tuple()[:2]))')
PYTHON=$(gdb -batch -q --nx -ex 'pi import sys; print(sys.executable)')
PYTHON="${PYTHON}${PYVER}"

# Find the Python site-packages that we need to use so that
# GDB can find the files once we've installed them.
SITE_PACKAGES=$(gdb -batch -q --nx -ex 'pi import site; print(site.getsitepackages()[0])')
# or to install in user mode:
SITE_PACKAGES=$(gdb -batch -q --nx -ex 'pi import site; print(site.getusersitepackages())')

# Install Python dependencies
sudo ${PYTHON} -m pip install --target ${SITE_PACKAGES} -Ur requirements.txt

Example of file requirements.txt

pip
pycparser
psutil>=3.1.0

Front-ends

GDB dashboard

GDB dashboard is a modular visual interface for GDB in Python.

To install simply copy .gdbinit as ~/.gdbinit

cp gdb-dashboard/.gdbinit ~/.gdbinit

Alternatively, source it from ~/.gdbinit:

source ~/.gdbinit-dashboard

Install pygments

Install pygments to get source highlighting

sudo pip install Pygments        # Globally
pip install Pygments             # Locally

If GDB uses python3 (ldd $(which gdb))), you'll need to install with pip3:

sudo pip3 install Pygments       # Globally
pip3 install Pygments            # Locally

To get the list of available styles:

python from pygments.styles import get_all_styles as styles
python for s in styles(): print(s)

Alternative styles:

• gruvbox

Voltron

See Voltron.

Prepare debug session

• Compile with debug symbols, use option -g:

gcc -g program.c               # -g : debug symbols
gcc -g -O0 program.c           #  ... -O0: disable optimization

• Force core dumps (see bash help ulimit):

ulimit -c unlimited
./a.out
# Segmentation fault (core dumped)

GDB invocation

gdb a.out
gdb a.out core.1234           # If coredump available

GDB commands

Reference:

• GDB manual
• https://beej.us/guide/bggdb/

Break points and watch points
b [+-][NUMBER] break [+-][NUMBER]	Set a breakpoint at current line, at given line NUMBER or NUMBER lines after/before current line.
b LOCATION break LOCATION	Set breakpoint at LOCATION. b main sets a breakpoint at beginning of function main(). b foo.c:42 sets a breakpoint at file foo.c, line 42.
watch EXPR	Stop execution when EXPR changes
awatch EXPR	Stop execution when EXPR is accessed
i b info b info break	list breakpoints
cl LOCATION clear LOCATION	Clear breakpoint by LOCATION
d delete	Delete all breakpoints
d NUMBER delete NUMBER	Clear breakpoint by NUMBER (as listed by i b)
dis NUMBER disable NUMBER	Disable breakpoint by NUMBER (as listed by i b)
save b FILE save breakpoints FILE	Save current breakpoints as script FILE. Use source to reload.

Execute program
r [ARGS] run [ARGS]	Start (or restart) program. Arguments may include wildcards (*) and redirections (<, <<...)
kill	Kill current program.
c cont	Continue and interrupted program.
s [NUMBER] step [NUMBER]	Step (into) current line, or NUMBER lines.
n next	Run to next line (over current line)
fin finish	Execute till returning from current selected frame.
advance LOCATION	Run until temporary breakpoint set at LOCATION.
u [NUMBER] until [NUMBER]	Execute until the program reaches a source line greater than current (very handy).

View stack
bt bt [COUNT] backtrace [COUNT] where [COUNT]	Print backtrace of all stack frames, or innermost (outermost) COUNT frames if COUNT>0 (COUNT<0)
f [FRAME] frame [FRAME]	Select frame FRAME and print stack frame
up	Go up a level in the stack (frame calling current frame).
do down	Go down a level in the stack (frame called by current frame).

View memory
disp EXPR display EXPR	Display EXPR at each prompt (if within scope).
x/FMT ADDRESS	Examine memory at ADDRESS as FMT(x)
i locals i args info locals info args	Print information on local variables / function arguments in the current frame
print EXPR	print EXPR.
undisplay NUMBER	Undisplay expression by NUMBER.

View code
l list	List (10 by default) lines of current frame
disassemble /m disassemble /s disassemble 'foo.c'::bar	Disassemble a specified section of memory

Miscellaneous
q quit	Quit gdb.
help COMMAND apropos WORD	Get help on COMMAND, or search commands related to WORD.
source FILE	Source script FILE.
RETURN	Repeat last command.

Tips

Define a custom label for breakpoint in C/C++

Say we want to set a breakpoint at a specified location in source file, but this position may move over time. The easiest is to use an asm statement to define the label [3]:

#include <stdio.h>

int main () {
  void *ret_p = &&ret;
  printf("ret: %p\n", ret_p);
  goto *ret_p;

  return 1;

ret:
  asm("RET:")

    return 0;
}

This will add a symbol table entry as follows.

gcc  -Wl,--export-dynamic t.c  -ldl
readelf -s a.out | grep RET
# 41: 0804858a     0 NOTYPE  LOCAL  DEFAULT   13 RET

Use help to test a command abbreviation

Don't know if f stands for finish or frame? Just use help f:

help f
# Select and print a stack frame.
# ...

8 gdb tricks you should know

From https://blogs.oracle.com/ksplice/entry/8_gdb_tricks_you_should:

• Use break WHERE if COND

For instance break context_switch if next == init_task.

• Use command

This sets commands to be executed when a breakpoint is hit. For instance

b do_mmap_pgoff 
# Breakpoint 1 at 0xffffffff8111a441: file mm/mmap.c, line 940.
command 1
# Type commands for when breakpoint 1 is hit, one per line.
# End with a line saying just "end".
>print addr
>print len
>print prot
>end

• Use gdb --args to specify runtime arguments

gdb --args pizzamaker --deep-dish --toppings=pepperoni
# ...
show args
# Argument list to give program being debugged when it is started is
#  " --deep-dish --toppings=pepperoni".
b main
# ...

• Finding source files

Use directory to add directory to search for source files.

list main
# 1192    ls.c: No such file or directory.
#     in ls.c
directory ~/src/coreutils-7.4/src/
# Source directories searched: /home/nelhage/src/coreutils-7.4:$cdir:$cwd
list main

Use set substitute-path to fix absolute paths

list schedule
# 5519    /build/buildd/linux-2.6.32/kernel/sched.c: No such file or directory.
#     in /build/buildd/linux-2.6.32/kernel/sched.c
set substitute-path /build/buildd/linux-2.6.32 /home/nelhage/src/linux-2.6.32/
list schedule

• Use gcc -ggdb3 to add MACRO's symbol as gdb macro

• Use gdb's variable like $1 but also define your own

set $foo = 4
p $foo
# $3 = 4

• Use register variables

All CPU registers are available as variables like $R0, $R1, but gdb also defines cross-architectures ones like $sp and $pc. For instance, if $rsi register is used to pass the 1st parameter,

break write if $rsi == 2

• The x command

x/s 0xffffffff81946000
# ffffffff81946000 <>:     "root=/dev/sda1 quiet"

# Use x/i as a quick way to disassemble memory
x/5i schedule
#   0xffffffff8154804a <schedule>:   push   %rbp
#   0xffffffff8154804b <schedule+1>: mov    $0x11ac0,%rdx
#   0xffffffff81548052 <schedule+8>: mov    %gs:0xb588,%rax
#   0xffffffff8154805b <schedule+17>:    mov    %rsp,%rbp
#   0xffffffff8154805e <schedule+20>:    push   %r15

# Print code surround current pc
x/20i $ip-40

• Use the @ symbol

The following works if source code is something like int a[ 10 ] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };:

p a
# $1 = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}

But if the array is much bigger, or array is a C++ vector, we can use @:

p *&a[0]@10
# $1 = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
p *&a[550]@4
# $1 = {551, 552, 553, 554}

Use value history variables $

See summary list below. For more details, check GDB manual on Value History.

$1          # History value 1
$2          # History value 2
$           # MOST RECENT value in the history
$$          # ... and value before that
$$2         # ... and value before that
$$0         # Same as $
$$1         # Same as $$

For instance:

p *$        # Dereference last value
p *$.next   # ... follow the chain
<RETURN>    # ... again
<RETURN>    # ... again

show values    # Print the last 10 values.

Examples

Simple Segmentation Fault Example

(From [4])

Example program segfault.c: #include <stdio.h> #include <stdlib.h> int main(int argc, char **argv) { char *buf; buf = malloc(1<<31); fgets(buf, 1024, stdin); printf("%s\n", buf); return 1; }

Compile and launch gdb: gcc -g segfault.c gdb a.out The debug session run backtrace frame 3 print buf kill break segfault.c:8 run print buf next print buf Fix the bug, then start again, watching now buf: watch buf # Start again, answer 'y' when asked to start from beginning run # Break at watch point, let's _c_ontinue c

More GDB stuff

display i            # Display i at each stop
display/4i $pc       # Display the 4 next instruction at each stop
u                    # Continue until we reach an higher pc address

Using gdb-dashboard

da                   # Print dashboard again
da reg               # Remove register
da reg               # Add it back
da m w 0x1000 0x10   # Watch memory zone at 0x1000 for 16 bytes
da m w 0x2000 0x10   # Add a second zone at 0x2000