Gdb

I don't like debuggers. Never have, probably never will. I use gdb all the time, but I tend to use it not as a debugger, but as a disassembler on steroids that you can program.

The most effective debugging tools are: your brain, a unit test, and the print statement.

Any programmer not writing unit tests for their code in 2007 should be considered a pariah (*).

GDB configuration

GDB reads file ~/.gdbinit at start.

Some references:

http://stackoverflow.com/questions/209534/how-to-highlight-and-color-gdb-output-during-interactive-debugging

Bare minimum configuration

From StackOverflow [3]:

set history save on
set print pretty
set output-radix 16
set height 0

Install Python modules for GDB

From pwndbg [4]:

# Find the Python version used by GDB.
PYVER=$(gdb -batch -q --nx -ex 'pi import platform; print(".".join(platform.python_version_tuple()[:2]))')
PYTHON=$(gdb -batch -q --nx -ex 'pi import sys; print(sys.executable)')
PYTHON="${PYTHON}${PYVER}"

# Find the Python site-packages that we need to use so that
# GDB can find the files once we've installed them.
SITE_PACKAGES=$(gdb -batch -q --nx -ex 'pi import site; print(site.getsitepackages()[0])')
# or to install in user mode:
SITE_PACKAGES=$(gdb -batch -q --nx -ex 'pi import site; print(site.getusersitepackages())')

# Install Python dependencies
sudo ${PYTHON} -m pip install --target ${SITE_PACKAGES} -Ur requirements.txt

Example of file requirements.txt

pip
pycparser
psutil>=3.1.0

Front-ends

Gef

Simple install script [5]:

# via the install script
$ wget -q -O- https://github.com/hugsy/gef/raw/master/scripts/gef.sh | sh

# manually
$ wget -O ~/.gdbinit-gef.py -q https://github.com/hugsy/gef/raw/master/gef.py
$ echo source ~/.gdbinit-gef.py >> ~/.gdbinit

Dependencies:

Install keystone-engine, NOT keystone (fix error AttributeError: 'module' object has no attribute 'KS_ARCH_X86'):

pip3 install --no-binary keystone-engine keystone-engine

Tips:

Gef may work with a second pane in tmux. This uses the tmux-setup / tmux-start commands, but the most reliable is to identify the PTS device with tmux list-panes -a -F '#D #{pane_tty}', and then assign the appropriate device to variable redirect in ~/.gef.rc.

tmux list-panes -F '#D #{pane_tty}'
# %0 /dev/pts/9
# %1 /dev/pts/10

Use context (or alias ctx) to re-print gef context.
Use db &myvar to dump (byte) at some variable location.

GDB dashboard

GDB dashboard is a modular visual interface for GDB in Python.

To install simply copy .gdbinit as ~/.gdbinit

cp gdb-dashboard/.gdbinit ~/.gdbinit

Alternatively, source it from ~/.gdbinit:

source ~/.gdbinit-dashboard

Install pygments

Install pygments to get source highlighting

sudo pip install Pygments        # Globally
pip install Pygments             # Locally

If GDB uses python3 (ldd $(which gdb))), you'll need to install with pip3:

sudo pip3 install Pygments       # Globally
pip3 install Pygments            # Locally

To get the list of available styles:

python from pygments.styles import get_all_styles as styles
python for s in styles(): print(s)

Alternative styles:

gruvbox

Issues

Cannot write the dashboard: [Errno 10] No child process issue #56

This is a bug in the gdb vendor. Fix file platform.py:

@@ -1009,7 +1009,10 @@
     except (AttributeError,os.error):
         return default
     output = string.strip(f.read())
-    rc = f.close()
+    try:
+        rc = f.close()
+    except:
+        rc = 0
     if not output or rc:
         return default
     else:

Cannot write the dashboard: Invalid character '?' in expression.

Likely due to incorrect character in info reg. The register is evaluated via parse_and_eval

Voltron

See Voltron.

GDB TUI

Some example inspired from Greg Law video on Youtube:

Start TUI mode with Ctrl-X A, or gdb --tui, or tui enable.
Ctrl-L repaint the screen.
Ctrl-X 2 to split the layout (repeat to cycle through layout), or layout split.
Ctrl-X 1 for back to source-only or disas-only layout (depending on the split-layout we left).
tui reg float for instance to change the register view.

Note that while in TUI mode, up and down keys will scroll the source code:

Up / Down scroll the source code.
Ctrl-P / Ctrl-N Previous or Next command in command history.
Ctrl-O switch focus (does not work for me).

CGDB

Probably one of the best lightweight front-end out there, with vim-like bindings.

Esc or i to switch between modes.
Ctrl-W to switch between vertical / horizontal layout (requires latest cgdb from source).
j, k, Ctrl-D, Ctrl-U to navigate (and more vim-like bindings as well)
Space to set a breakpoint, t for a temporary breakpoint
/ and n for search and next
:help for help (when in cgdb mode)
:set dis or :set nodis to view disassembly.

cgdb works nicely with rr:

rr replay -d cgdb

Prepare debug session

Compile with debug symbols, use option -g:

gcc -g program.c               # -g : debug symbols
gcc -g -O0 program.c           #  ... -O0: disable optimization

Force core dumps (see bash help ulimit):

ulimit -c unlimited
./a.out
# Segmentation fault (core dumped)

GDB invocation

gdb a.out
gdb a.out core.1234           # If coredump available

GDB commands

Reference

GDB manual
https://beej.us/guide/bggdb/

Break points and watch points
`b [+-][NUMBER]` `break [+-][NUMBER]` `b LOCATION` `break LOCATION`	Set a breakpoint at current line, at given line NUMBER or NUMBER lines after/before current line. Set breakpoint at LOCATION. `b main` sets a breakpoint at beginning of function `main()`. `b foo.c:42` sets a breakpoint at file foo.c, line 42. `b *0x400120` sets a breakpoint at address 0x400120.
`tb [+-][NUMBER]` `tbreak [+-][NUMBER]` `tb LOCATION` `tbreak LOCATION`	Idem, but temporary breakpoint.
`ignore N COUNT`	Set ignore-count of breakpoint N to COUNT
`command [BKPIP]` `...` `end`	Set commands to execute when a breakpoint is hit
`watch EXPR`	Stop execution when EXPR changes
`awatch EXPR`	Stop execution when EXPR is accessed
`i b` `info b` `info break`	list breakpoints
`cl [LOCATION]` `clear [LOCATION]`	Clear breakpoint by LOCATION, or current line
`d` `delete`	Delete all breakpoints
`d NUMBER` `delete NUMBER`	Clear breakpoint by NUMBER (as listed by `i b`)
`dis NUMBER` `disable NUMBER`	Disable breakpoint by NUMBER (as listed by `i b`). `dis $bpnum` to disable last bkp (useful for .gdbinit).
`save b FILE` `save breakpoints FILE`	Save current breakpoints as script FILE. Use `source` to reload.
Registers
`i r` `info reg`	List integer registers

Execute program
`r [ARGS]` `run [ARGS]`	Start (or restart) program. Arguments may include wildcards (*) and redirections (<, <<...)
`start [ARGS]`	Start and break on `main`
`kill`	Kill current program.
`c` `cont`	Continue and interrupted program.
`s [NUMBER]` `step [NUMBER]`	Step (into) current line, or NUMBER lines.
`si [NUMBER]` `stepi [NUMBER]`	Step one (assembly) instruction exactly (N times)
`n` `next`	Run to next line (over current line)
`ni [NUMBER]` `nexti [NUMBER]`	Step one (assembly) instruction exactly (N times), but over subroutine calls.
`fin` `finish`	Execute till returning from current selected frame.
`adv LOCATION` `advance LOCATION`	Run until temporary breakpoint set at LOCATION.
`u [NUMBER]` `until [NUMBER]`	Execute until the program reaches a source line greater than current (very handy).

View stack
`bt [COUNT]` `backtrace [COUNT]` `where [COUNT]`	Print backtrace of all stack frames, or innermost (outermost) COUNT frames if COUNT>0 (COUNT<0)
`f [FRAME]` `frame [FRAME]`	Select frame FRAME and print stack frame
`up`	Go up a level in the stack (frame calling current frame).
`do` `down`	Go down a level in the stack (frame called by current frame).

View memory
`disp/FMT EXPR` `disp EXPR` `display EXPR`	Display EXPR at each prompt (if within scope), using format FMT. If omitted, reuse last format FMT (of last `x`, `p` or `disp`). Use `ptr` to display pointed data, and `ptr@LEN` to display array of LEN elements.
`x[/nfu] ADDR` `x/8zb ADDR` `x/16i ADDR` `x/16zw ADDR`	Examine n memory locations at ADDR, format as n (`xduotacfszr`) and unit size u (`bhwg`). If omitted, reuse last format FMT. Pro tip: Use always format 'z' over 'x'. `/i` for a single assembly instruction. `/16i` for 16 instructions `/8zb` for 8 bytes with leading zeroes. `/16zw` for 16 32-bit words with leading zeroes. `x/16i $PC` use current PC addr
`i lo` `i args` `info locals` `info args`	Print information on local variables / function arguments in the current frame
`p/FMT EXPR` `p EXPR` `print EXPR`	print EXPR, using format FMT. If omitted, reuse last format FMT (of last `x`, `p` or `disp`). Use `ptr` to display pointed data, and `ptr@LEN` to display array of LEN elements.
`undisplay NUMBER`	Undisplay expression by NUMBER.

View code
`l` `list`	List (10 by default) lines of current frame
`disas /m` `disas /s` `disas 'foo.c'::bar` `disas main,+40` `disas $pc,+40` `disassemble`	Disassemble a specified section of memory

Miscellaneous
`q` `quit`	Quit gdb.
`help COMMAND` `apropos WORD`	Get help on COMMAND, or search commands related to WORD.
`source FILE`	Source script FILE.
`RETURN`	Repeat last command.
`add-symbol-file FILE ADDR`	Load symbols from FILE at given ADDR.

Change memory

set VARIABLE = VALUE
For instance, given a program

int main(void){
    char[] person = "Bob";
    char[] p2 = "Alice";

    printf("Hello %s\n");
}

set main::person = { 'S', 'a', 'm', 0x00 }
set main::person = "Sam"
# To change memory directly:
set {char [4]}0x43f800 = {0x01, 0x02, 0x03, 0x04}

Alternatively we can use strcpy [6]:

(gdb) p malloc(20)
$3 = (void *) 0x6ce81808
(gdb) p strcpy($3, "my string")
$4 = 1827149832
(gdb) x/s $3
0x6ce81808: "my string"

Dump / restore

For instance, to load an Intel hex file:

restore firmware.hex

To generate the same file:

dump ihex memory firmware.hex 0x00400000 0x0040FFFF

Note that dump can also save the result of an expression: dump [format] memory filename start_addr end_addr dump [format] value filename expr

Reverse debugging

References

GDB news - reversible debug.

Example session

Say we have the following program:

void error(void) 
{
    // ...
}

void foo(void)
{
    // ...
    bar();          // Calls error() if something goes wrong
    baz();
    // ...
}

We want to debug function bar. We can use gdb reverse debugging as follows:

gdb myprog

# Start recording when entering 'bar'
b bar
commands
    record
    continue
end
# Break if error
b error
# Stop recording when we exit bar
tbp baz
    record stop
    continue
end

If an error occurs, we can use the following commands to walk backward (see reference for more):

The usual step or next, but in reverse: reverse-next or rn, reverse-nexti or rni, reverse-step or rs, reverse-stepi or rsi.
Reverse continue (until next breakpoint, or record start): reverse-continue or rc.
Reverse finish: reverse-finish.

Recording may be very slow. A faster alternative is to use rr.

Troubleshooting

When getting the following messages:

Process record does not support instruction 0xfae64 (xsave)

Process record does not support instruction 0xc5 at address

...

This mostly happens when gdb encounters an instruction it cannot handle yet, such as AVX instructions. Usually it occurs when a library is loaded to detect which CPU binary to use [7], [8], [9]

The fix is either to run gdb with LD_BIND_NOW=1 (LD_BIND_NOW=1 gdb a.out) to avoid exotic instruction due to ld.
Or compile with -static (clang -g -static hello.c).

gdb prints the following message and exits:

# [1]+   Stopped                LD_BIND_NOW=1 gdb a.out
fg
# LD_BIND_NOW=1 gdb a.out
# Process record: failed to record execution log.
#
# Program stopped

This is simply indicative that program probably segfaulted. Do fg to resume gdb and start debug backward.

This does not seem to happen when using cgdb.

Watch point ignored when reverse executing

When setting a watchpoint, gdb goes up to program start and prints No more reverse-execution history.

watch *(long **)0x7fffffffffd4a8
# Hardware watch point 4: *(long **)0x7fffffffffd4a8
rc
# Continuing
#
# No more reverse-execution history.
# main () at bubble_sort.c:34
# 34          unsigned  i = 0;

There is a bug in gdb that makes it ignore hardware watchpoint in reverse execution... The fix [10], [11]

set can-use-hw-watchpoints 0
watch *(long **)0x7fffffffffd4a8

Python

See gdb python.

Tips

Use RETURN to repeat last command

Pressing RETURN repeats all last command:

Commands like s (step) or n (next). Very handy to step in the code.
Command like x/16w 0x10100000 or x/16i 0x10100000, each time printing 16 new words.

Define a custom label for breakpoint in C/C++

Say we want to set a breakpoint at a specified location in source file, but this position may move over time. The easiest is to use an asm statement to define the label [12]:

#include <stdio.h>

int main () {
  void *ret_p = &&ret;
  printf("ret: %p\n", ret_p);
  goto *ret_p;

  return 1;

ret:
  asm("RET:")

    return 0;
}

This will add a symbol table entry as follows.

gcc  -Wl,--export-dynamic t.c  -ldl
readelf -s a.out | grep RET
# 41: 0804858a     0 NOTYPE  LOCAL  DEFAULT   13 RET

Use `help` to test a command abbreviation

Don't know if f stands for finish or frame? Just use help f:

help f
# Select and print a stack frame.
# ...

8 gdb tricks you should know

From https://blogs.oracle.com/ksplice/entry/8_gdb_tricks_you_should:

Use break WHERE if COND

For instance break context_switch if next == init_task.

Use command

This sets commands to be executed when a breakpoint is hit. For instance

b do_mmap_pgoff 
# Breakpoint 1 at 0xffffffff8111a441: file mm/mmap.c, line 940.
command 1
# Type commands for when breakpoint 1 is hit, one per line.
# End with a line saying just "end".
>print addr
>print len
>print prot
>end

Use gdb --args to specify runtime arguments

gdb --args pizzamaker --deep-dish --toppings=pepperoni
# ...
show args
# Argument list to give program being debugged when it is started is
#  " --deep-dish --toppings=pepperoni".
b main
# ...

Finding source files

Use directory to add directory to search for source files.

list main
# 1192    ls.c: No such file or directory.
#     in ls.c
directory ~/src/coreutils-7.4/src/
# Source directories searched: /home/nelhage/src/coreutils-7.4:$cdir:$cwd
list main

Use set substitute-path to fix absolute paths

list schedule
# 5519    /build/buildd/linux-2.6.32/kernel/sched.c: No such file or directory.
#     in /build/buildd/linux-2.6.32/kernel/sched.c
set substitute-path /build/buildd/linux-2.6.32 /home/nelhage/src/linux-2.6.32/
list schedule

Use gcc -ggdb3 to add MACRO's symbol as gdb macro

Use gdb's variable like $1 but also define your own

set $foo = 4
p $foo
# $3 = 4

Use register variables

All CPU registers are available as variables like $R0, $R1, but gdb also defines cross-architectures ones like $sp and $pc. For instance, if $rsi register is used to pass the 1st parameter,

break write if $rsi == 2

The x command

x/s 0xffffffff81946000
# ffffffff81946000 <>:     "root=/dev/sda1 quiet"

# Use x/i as a quick way to disassemble memory
x/5i schedule
#   0xffffffff8154804a <schedule>:   push   %rbp
#   0xffffffff8154804b <schedule+1>: mov    $0x11ac0,%rdx
#   0xffffffff81548052 <schedule+8>: mov    %gs:0xb588,%rax
#   0xffffffff8154805b <schedule+17>:    mov    %rsp,%rbp
#   0xffffffff8154805e <schedule+20>:    push   %r15

# Print code surround current pc
x/20i $ip-40

Use the @ symbol

The following works if source code is something like int a[ 10 ] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };:

p a
# $1 = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}

But if the array is much bigger, or array is a C++ vector, we can use @:

p *&a[0]@10
# $1 = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
p *&a[550]@4
# $1 = {551, 552, 553, 554}

Use value history variables `$`

See summary list below. For more details, check GDB manual on Value History.

$1          # History value 1
$2          # History value 2
$           # MOST RECENT value in the history
$$          # ... and value before that
$$2         # ... and value before that
$$0         # Same as $
$$1         # Same as $$

For instance:

p *$        # Dereference last value
p *$.next   # ... follow the chain
<RETURN>    # ... again
<RETURN>    # ... again

show values    # Print the last 10 values.

Disassemble at an arbitrary address

disas FUNCTION command only works for defined FUNCTION. To disassemble at an arbitrary address:

x/i 0x10100000 one instruction at address 0x10100000,
x/16i 0x10100000 (16 instruction) at address Ox10100000,
x/16i *0x10100000 (16 instruction) at address contained at Ox10100000,
x/16i $PC disassemble 16 instructions at PC.

Redirect STDIN

Redirect STDIN with the run command:

gdb ./myprog
run <input.txt            # Run with stdin redirected

One can also pass parameters with the run command (or use gdb --args command-line parameter).

Exit gdb if run fails

Use python script

def tryRun():
    try:
        gdb.execute("run")
    except:
        print("Exception occured during gdb command: run")
        gdb.execute("quit 1")

Define custom gdb command

Define custom gdb command using define:

# define in .gdbinit
define sf
  where        # find out where the program is
  info args    # show arguments
  info locals  # show local variables
end

See also Gdb python to define more powerful custom commands.

Load symbols from another ELF file

This is useful in embedded systems where some part of the code are already loaded or stored in a ROM. When the debugger stops in a function for which it doesn't have the symbols, it cannot decode the stack and provide a correct backtrace. The fix is to load the symbols for that function using the command add-symbol-file.

add-symbol-file a.out 0               # Add symbol from file a.out, at offset 0

Typically the offset to give is the lowest address in the ELF file.

Modify memory content

Use set [13]. Simplest is to modify a variable:

set variable i = 10

But GDB can write in any memory location:

set {int}0x83040 = 4
set *((int *)0x83040) = 4

Remote debug with `gdbserver`

Start the gdb server on the remote machine:

gdbserver --multi 192.168.20.1:6666     # CLIENT_ADDR:REMOTE_PORT

This will listen on port 6666, waiting for connection from client 192.168.20.1

On the client, start the GDB client:

gdb-multiarch ./my_exec                 # multiarch necessary if client/server are different arch

Then, in GDB:

target extended-remote {REMOTE_ADDR}:{REMOTE_PORT}
set remote exec-file /path/on/remote/to/my_exec

If the remote executable uses dynamic libraries, these must be sent to the client. To accelerate the loading, copy first the libraries locally on the client, then tell GDB where to find them [14]:

set sysroot /d/st/courses/training_obfu/formation_reverse/dynamic/tp/qmessage/pi_root
set solib-absolute-prefix /d/st/courses/training_obfu/formation_reverse/dynamic/tp/qmessage/pi_root
set solib-search-path /d/st/courses/training_obfu/formation_reverse/dynamic/tp/qmessage/pi_root

Reverse debugging

From Greg Law's incredible video.

Assume we have an application that crashes at random. We want to debug it when it crashes. First we setup some breakpoint and start recording automatically:

gdb a.out
start

break main
command
record
continue
end

break _exit.c:32              # Hack. to break on exit
command
run
end

We also make sure pagination is disabled:

set pagination off

Now we cont until the code crashes. We start to reverse debug the program:

p $pc
# 1 = (void (*)()) 0x5e0c5d00
x 0x5e0c5d00
# 0x5e0c5d00     Cannot access memory at address 0x5e0c5d00
bt
# ... the stack is corrupted. Let's step back...
reverse-stepi
# 0x00000000004806ac in main () at bubble_sort.:43
# 43      }
# ... we're back in normal land
bt
# 0x00000000004806ac in main () at bubble_sort.:43
disas
# 0x00000000004806ac <+135>:     retq
# ... so indeed we tried to return, but likely the stack is corrupted

Let's look at our stack:

print $sp
# 2 = (void *) 0x7fffffffdc98
print *(long **) 0x7fffffffdc98
# 3 = (long *) 0x5e4c5d00
# ... this the address we would jump to if RETQ
watch *(long **) 0x7fffffffdc98
# Hardware watchpoint 4: *(long **) 0x7fffffffdc98

We have set our hardware watchpoint, and will reverse continue! This will bring us to the point where the corruption occured:

reverse-continue
# old value = (long *) 0x5e4c5d00
# new value = (long *) 0xffff7a36ec5 <__libc_start_main+245>
# 0x00000000004806a8 in main () at bubble_sort.:37
print i
4 = 35

We were indeed overwriting the stack because of array overflow!

Use Catch to break on syscall

This was used by HN to debug a complex bugs in some Python profiler [15]:

# This looks for "lseek" followed by "close" without intervening "read".
set height 0

catch syscall close
catch syscall read
catch syscall lseek
disable 1 2

commands 2
 disable 1 2
 continue
end

commands 3
 if $rdi == 31
  enable 1 2
  continue
 else
  continue
 end
end

Printing and Examing memory

Getting rid of octal

By default, gdb uses octal' to print char string, which is a major PITA. Use p/x or p/a to work-around that, or use examine (x)

Dealing with examine default format / unit size

The complete examine syntax is x/nfu addr, where n is the repeat count, f is the format (same as for print, ie. one of x, d, u, o, t, a, c, f, s), and u is the unit size (one of b, h, w, g).
If not specified, examine uses a default value for the format (x initially) and unit size (w for x format, g for a format). But the default changes every time you use print or examine!

Printing arrays and pointers (to arrays)

Given the C code:

uint8_t arr[8] = {0x12, 0x34, 0x56, 0x78, 0x11, 0x22, 0x33, 0x44};
uint8_t *p = arr;

Using print or p:

# -- print: arr and *p
p arr
p *p@8
# $1 = "\022\064Vx\021\"3D"
p/a arr
p/x arr
p/a *p@8
p/x *p@8
# $2 = {0x12, 0x34, 0x56, 0x78, 0x11, 0x22, 0x33, 0x44}
p *p
# $3 = 0x12

# -- print: &arr, p and &p
p &arr
# $4 = (uint8_t (*)[8]) 0x404030 <arr>
p p
# $4 = (uint8_t *) 0x404030 <arr> "\022\064Vx\021\"3D"
p/a &arr
p/a p
# $5 = 0x404030 <arr>
p/x &arr
p/x p
# $3 = 0x404030
p &p
# $1 = (uint8_t **) 0x404040 <p>
p/a &p
# $2 = 0x404040 <p>
p/x &p
# $3 = 0x404040

# -- print: More cases with @
p arr@2
# $8 = {"\022\064Vx\021\"3D", "\000\000\000\000\001\000\000"}
p/a arr@2
p/x arr@2
# $9 = {{0x12, 0x34, 0x56, 0x78, 0x11, 0x22, 0x33, 0x44}, {0xa2, 0xa1, 0xb2, 0xb1, 0x0, 0x0, 0x0, 0x0}}

p p@8
# $7 = {0x404030 <arr> "\022\064Vx\021\"3D", 0x100000001 <error: Cannot access memory at address 0x100000001>, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}
p/a p@8
# $8 = {0x404030 <arr>, 0x100000001, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}
p/x p@8
# $9 = {0x404030, 0x100000001, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}

Using examine or x:

# -- examine: arr, &arr, p (all points to same address)
# ... We force FORMAT and UNIT SIZE to avoid relying on previous default setting.
x/xw arr
x/xw &arr
x/xw p
# 0x404030 <arr>: 0x78563412
# ... From now on, all examine will use hexadecimal, and unit size for hex is 'word'.
x/xg arr
x/xg &arr
x/xg p
# 0x404030 <arr>: 0x4433221178563412
# ... From now on, all examine will use hexadecimal, and unit size for hex is 'giant'.
x/xb arr
x/xb &arr
x/xb p
# 0x404030 <arr>: 0x12
# ... From now on, all examine will use hexadecimal, and unit size for hex is 'byte'.
x/8b arr
x/8b &arr
x/8b p
# 0x404030 <arr>: 0x12    0x34    0x56    0x78    0x11    0x22    0x33    0x44

# -- examine: address 'a' format is always SIGNED, which may gives strange results:
# ... same as 'x' if positive:
x/xw arr
x/aw arr
# 0x404030 <arr>: 0x78563412
# ... but different is negative. This is because 'a' takes the value as an address (negative=stack).
set arr[3]=0x80
x/aw arr
# 0x404030 <arr>: 0xffffffff80563412
x/xw arr
# 0x404030 <arr>: 0x80563412

# -- examine: &p (address of pointer p itself)
x/aw &p
# 0x404050 <p>:   0x404030 <arr>
# ... note again difference with 'hexadecimal' format:
x/xw &p
# 0x404050 <p>:   0x00404030

# -- examine: bad cases (usually)
x *p
# 0x1:  Cannot access memory at address 0x12
x &p@8
# Only values in memory can be extended with '@'.
x/xb arr
x/xb arr@8
# 0x404030 <arr>: 0x12
# ... @8 has no effect. One must use x/8xb

Notes

We can use both p/x and p/a to remove octal format. The small advantage for p/a is that it will also prints to nearest preceding symbol.

p/a some_struct                      # Address mode... but strangely it removes octal. WTF?

p and x always default to last used format, unless one is specified. The default for x is x, and changes each time you use x or print [16]:

Printing struct and pointers (to struct)

Given the C code:

struct {
    uint16_t a;
    uint16_t b;
} S = {0xa1a2,0xb1b2}, *s = &S;

Using print or p:

# -- print: S and *s
p S
p/a S
p/x S
p *s
p/a *s
p/x *s
# $9 = {
#   a = 0xa1a2, 
#   b = 0xb1b2
# }

# -- print: &S, s and &s
p &S
p s
# $1 = (struct {...} *) 0x404038 <S>
p/a &S
p/a s
# $2 = 0x404038 <S>
p/x &S
p/x s
# $3 = 0x404038
p &s
# $4 = (struct {...} **) 0x404040 <s>
p/a &s
# $5 = 0x404040 <s>
p/x &s
# $6 = 0x404040

# -- print: More cases with @
p S@2
p/a S@2
p/x S@2
# $2 = {{
#     a = 0xa1a2, 
#     b = 0xb1b2
#   }, {
#     a = 0x0, 
#     b = 0x0
#   }}
p s@2
p/a s@2
# $2 = {0x404038 <S>, 0x0}
p/x s@2
# $2 = {0x404038, 0x0}

Using examine or x:

# -- examine: &S, s, in hexadecimal format
x/xw &S
x/xw s
# 0x404038 <S>:   0xb1b2a1a2
x/xg &S
x/xg s
# 0x404038 <S>:   0x00000000b1b2a1a2
x/xb &S
x/xb s
# 0x404038 <S>:   0xa2

# -- examine: &S, s, in address format
x/aw &S
x/aw s
# 0x404038 <S>:   0xffffffffb1b2a1a2
x/ab &S
x/ab s
# 0x404038 <S>:   0xffffffffffffffa2
x/ag &S
# 0x404038 <S>:   0xb1b2a1a2
# ... This is because address is 0x00000000b1b2a1a2, and leading zeroes are removed.

# -- examine: bad cases (usually)
x/xb S
x/xb *s
# Value can't be converted to integer.
x/xb &S@8
# Only values in memory can be extended with '@'.
x/xw s
x/xw s@4
# 0x404040 <s>:   0x00404038
# ... @8 has no effect. One must use x/4xw

Hook on quit

From Stackoverflow:

define hook-quit
    set confirm off
end

Hook on stop

Say you have a custom cmd that you'll like to execute whenever GDB breaks:

define hook-stop
    cmd
end

Enable core dump and load in gdb

Enable core dump file with ulimit:

ulimit -c unlimited

Check they are well enabled:

ulimit -a
# core file size          (blocks, -c) unlimited
# data seg size           (kbytes, -d) unlimited
# ...

Use gdb -c core to load the core dump:

./a.out
# Segmentation fault (core dumped)
l core*
# -rw-------+ 1 user user 240K Jun 16 16:35 core
gdb -c core

Or use any other front-end:

cgdb -c core

Check /proc/sys/kernel/core_pattern for the naming pattern for core files.

Send SIGINT automatically after some time

Use this timeout:

#! /bin/bash

# Source: https://github.com/dscho/msys/blob/master/msys/packages/bash/2.04/examples/scripts/timeout

#Newsgroups: comp.unix.admin,comp.unix.solaris,comp.unix.shell
#From: gwc@root.co.uk (Geoff Clare)
#Subject: Re: timeout -t <sec> <unix command> (Re: How to give rsh a shorter timeout?)
#Message-ID: <EoBxrs.223@root.co.uk>
#Date: Fri, 13 Feb 1998 18:23:52 GMT

#
# Conversion to bash v2 syntax done by Chet Ramey <chet@po.cwru.edu
# UNTESTED
#

prog=${0##*/}
usage="usage: $prog [-signal] [timeout] [:interval] [+delay] [--] <command>"

SIG=-TERM	# default signal sent to the process when the timer expires
timeout=60	# default timeout
interval=15	# default interval between checks if the process is still alive
delay=2		# default delay between posting the given signal and
		# destroying the process (kill -KILL)

while :
do
	case $1 in
	--)	shift; break ;;
	-*)	SIG=$1 ;;
	[0-9]*)	timeout=$1 ;;
	:*)	EXPR='..\(.*\)' ; interval=`expr x"$1" : "$EXPR"` ;;
	+*)	EXPR='..\(.*\)' ; delay=`expr x"$1" : "$EXPR"` ;;
	*)	break ;;
	esac
	shift
done

case $# in
0)	echo "$prog: $usage" >&2 ; exit 2 ;;
esac

(
	for t in $timeout $delay
	do
		while (( $t > $interval ))
		do
			sleep $interval
			kill -0 $$ || exit
			t=$(( $t - $interval ))
		done
		sleep $t
		kill $SIG $$ && kill -0 $$ || exit
		SIG=-KILL
	done
) 2> /dev/null &

exec "$@"

Example of use:

timeout -INT 300 :5 -- gdb $(APP)

Examples

Simple Segmentation Fault Example

(From [17])

Example program segfault.c:

#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
  char *buf;

  buf = malloc(1<<31);

  fgets(buf, 1024, stdin);
  printf("%s\n", buf);

  return 1;
}

Compile and launch gdb:

gcc -g segfault.c
gdb a.out

The debug session

run
backtrace
frame 3
print buf
kill
break segfault.c:8
run
print buf
next
print buf

Fix the bug, then start again, watching now buf:

watch buf
# Start again, answer 'y' when asked to start from beginning
run
# Break at watch point, let's _c_ontinue
c

More GDB stuff

display i            # Display i at each stop
display/4i $pc       # Display the 4 next instruction at each stop
u                    # Continue until we reach an higher pc address

Using gdb-dashboard

da                   # Print dashboard again
da reg               # Remove register
da reg               # Add it back
da m w 0x1000 0x10   # Watch memory zone at 0x1000 for 16 bytes
da m w 0x2000 0x10   # Add a second zone at 0x2000

Frequently used commands

info reg
x/i $pc

GDB script

Breakpoint command lists

See help commands (or here)

Example script:

break foo if x>0
commands
silent
printf "x is %d\n",x
cont
end

Use silent to make the breakpoint silent.
Use condition (here if x>0) on breakpoint to break conditionally.

Troubleshooting

Python GDB

Deadlocks in multi-threaded Python extensions

Gdb

References

GDB front-ends

GDBINIT Hacks

Tools and libraries

Tutorials

Alternatives to GDB

rr

LLDB

No debugger

GDB configuration

Bare minimum configuration

Install Python modules for GDB

Front-ends

Gef

GDB dashboard

Voltron

GDB TUI

CGDB

Prepare debug session

GDB invocation

GDB commands

Reference

Change memory

Dump / restore

Reverse debugging

Python

Tips

Use RETURN to repeat last command

Define a custom label for breakpoint in C/C++

Use help to test a command abbreviation

8 gdb tricks you should know

Use value history variables $

Disassemble at an arbitrary address

Redirect STDIN

Exit gdb if run fails

Define custom gdb command

Load symbols from another ELF file

Modify memory content

Remote debug with gdbserver

Reverse debugging

Use Catch to break on syscall

Printing and Examing memory

Printing arrays and pointers (to arrays)

Printing struct and pointers (to struct)

Hook on quit

Hook on stop

Enable core dump and load in gdb

Send SIGINT automatically after some time

Examples

Simple Segmentation Fault Example

More GDB stuff

Using gdb-dashboard

Frequently used commands

GDB script

Breakpoint command lists

Troubleshooting

Python GDB

Navigation menu

Search

Use `help` to test a command abbreviation

Use value history variables `$`

Remote debug with `gdbserver`