Gdb: Difference between revisions

From miki
Jump to navigation Jump to search
 
(15 intermediate revisions by the same user not shown)
Line 235: Line 235:
* <code>:set dis</code> or <code>:set nodis</code> to view disassembly.
* <code>:set dis</code> or <code>:set nodis</code> to view disassembly.


cgdb works nicely with '''rr''':
cgdb works nicely with '''[[rr]]''':
<source lang="bash">
<source lang="bash">
rr replay -d cgdb
rr replay -d cgdb
</source>
</source>

See [[rr]] page for more options.


== Prepare debug session ==
== Prepare debug session ==
Line 281: Line 283:
:<code>b main</code> sets a breakpoint at beginning of function <code>main()</code>.
:<code>b main</code> sets a breakpoint at beginning of function <code>main()</code>.
:<code>b foo.c:42</code> sets a breakpoint at file {{file|foo.c}}, line 42.
:<code>b foo.c:42</code> sets a breakpoint at file {{file|foo.c}}, line 42.
:<code>b foo.c:loop</code> sets a breakpoint at file {{file|foo.c}}, label <code>loop:</code>.
:<code>b *0x400120</code> sets a breakpoint at address 0x400120.
:<code>b *0x400120</code> sets a breakpoint at address 0x400120.
|-
|width=150pt|
<code>hb [+-][NUMBER]</code><br/>
<code>hb LOCATION</code><br/>
|
Set a HARDWARE breakpoint at current line, at given line ''NUMBER'' or ''NUMBER'' lines after/before current line.

Set a HARDWARE breakpoint at ''LOCATION''.
|-
|width=150pt|
<code>b LOC if COND</code>
|
Set a conditional breakpoint.
:<code>b main.c:24 if a == 2</code> Set a breakpoint at <code>main.c:24</code> when var <code>a</code> is 2.
|-
|-
|width=150pt|
|width=150pt|
Line 437: Line 454:
{| width=100%
{| width=100%
|-
|-
!colspan=2|View memory
!colspan=2|View / Change memory
|-
|-
|width=150pt|
|width=150pt|
Line 454: Line 471:
|
|
Examine ''n'' memory locations at ADDR, format as ''n'' (<code>xduotacfszr</code>) and unit size ''u'' (<code>bhwg</code>). If omitted, '''reuse last format FMT'''.
Examine ''n'' memory locations at ADDR, format as ''n'' (<code>xduotacfszr</code>) and unit size ''u'' (<code>bhwg</code>). If omitted, '''reuse last format FMT'''.
'''Pro tip: Use always format 'z' over 'x''''.
'''Pro tip: Use always format 'z', not 'x''''.
* <code>/i</code> for a single assembly instruction. <code>/16i</code> for 16 instructions
* <code>/i</code> for a single assembly instruction. <code>/16i</code> for 16 instructions
* <code>/8zb</code> for 8 bytes with '''leading zeroes'''.
* <code>/8zb</code> for 8 bytes with '''leading zeroes'''.
Line 472: Line 489:
<code>print EXPR</code>
<code>print EXPR</code>
|print ''EXPR'', using format FMT. If omitted, '''reuse last format FMT''' (of last <code>x</code>, <code>p</code> or <code>disp</code>).
|print ''EXPR'', using format FMT. If omitted, '''reuse last format FMT''' (of last <code>x</code>, <code>p</code> or <code>disp</code>).
Use <code>*ptr</code> to display pointed data, and <code>*ptr@LEN</code> to display array of LEN elements.
Use <code>*ptr</code> to display pointed data, and <code>*ptr@LEN</code> to display array of LEN elements. '''Pro tip: Use always format 'z', not 'x''''
|-
|-
|
|
<code>undisplay NUMBER</code>
<code>undisplay NUMBER</code>
|Undisplay expression by ''NUMBER''.
|Undisplay expression by ''NUMBER''.
|-
|
<code>set VARIABLE=VALUE</code>
|Write VALUE in VARIABLE (this modifies memory).
|}
|}


Line 527: Line 548:
</div>
</div>


;Keyboard shortcuts
=== Change memory ===
* {{kb|C-L}} &mdash; Clear the screen (useful when pgm writes to the screen).
<code>set VARIABLE = VALUE</code><br/>
For instance, given a program
<source lang="c">
int main(void){
char[] person = "Bob";
char[] p2 = "Alice";

printf("Hello %s\n");
}
</source>

<source lang=c>
set main::person = { 'S', 'a', 'm', 0x00 }
set main::person = "Sam"
# To change memory directly:
set {char [4]}0x43f800 = {0x01, 0x02, 0x03, 0x04}
</source>

Alternatively we can use <code>strcpy</code> [https://stackoverflow.com/questions/19503057/in-gdb-how-can-i-write-a-string-to-memory]:
<pre>
(gdb) p malloc(20)
$3 = (void *) 0x6ce81808
(gdb) p strcpy($3, "my string")
$4 = 1827149832
(gdb) x/s $3
0x6ce81808: "my string"
</pre>


=== Dump / restore ===
=== Dump / restore ===
Line 1,282: Line 1,277:


=== Send SIGINT automatically after some time ===
=== Send SIGINT automatically after some time ===
Use <code>timeout</code>:
Use this <code>timeout</code>:
<source lang="bash">
#! /bin/bash

# Source: https://github.com/dscho/msys/blob/master/msys/packages/bash/2.04/examples/scripts/timeout

#Newsgroups: comp.unix.admin,comp.unix.solaris,comp.unix.shell
#From: gwc@root.co.uk (Geoff Clare)
#Subject: Re: timeout -t <sec> <unix command> (Re: How to give rsh a shorter timeout?)
#Message-ID: <EoBxrs.223@root.co.uk>
#Date: Fri, 13 Feb 1998 18:23:52 GMT

#
# Conversion to bash v2 syntax done by Chet Ramey <chet@po.cwru.edu
# UNTESTED
#

prog=${0##*/}
usage="usage: $prog [-signal] [timeout] [:interval] [+delay] [--] <command>"

SIG=-TERM # default signal sent to the process when the timer expires
timeout=60 # default timeout
interval=15 # default interval between checks if the process is still alive
delay=2 # default delay between posting the given signal and
# destroying the process (kill -KILL)

while :
do
case $1 in
--) shift; break ;;
-*) SIG=$1 ;;
[0-9]*) timeout=$1 ;;
:*) EXPR='..\(.*\)' ; interval=`expr x"$1" : "$EXPR"` ;;
+*) EXPR='..\(.*\)' ; delay=`expr x"$1" : "$EXPR"` ;;
*) break ;;
esac
shift
done

case $# in
0) echo "$prog: $usage" >&2 ; exit 2 ;;
esac

(
for t in $timeout $delay
do
while (( $t > $interval ))
do
sleep $interval
kill -0 $$ || exit
t=$(( $t - $interval ))
done
sleep $t
kill $SIG $$ && kill -0 $$ || exit
SIG=-KILL
done
) 2> /dev/null &

exec "$@"
</source>

Example of use:

<source lang="bash">
<source lang="bash">
timeout -INT 300 :5 -- gdb $(APP)
timeout -INT 300 :5 -- gdb $(APP)
</source>

=== Write into memory ===
Use <code>set VARIABLE = VALUE</code>.

;Modify program variables
For instance, given a program
<source lang="c">
int main(void){
char[] person = "Bob";
char[] p2 = "Alice";

printf("Hello %s\n");
}
</source>
We can do
<source lang="python">
set main::person = { 'S', 'a', 'm', 0x00 }
set main::person = "Sam"
</source>

;Write at memory address
<source lang="python">
set {char}0x43f800 = 0x01
set {int}0x43f800 = 0x04030201
set {char [4]}0x43f800 = {0x01, 0x02, 0x03, 0x04}
</source>

;Call functions with side effect
Alternatively we can use <code>strcpy</code> [https://stackoverflow.com/questions/19503057/in-gdb-how-can-i-write-a-string-to-memory]:
<source lang="python">
p malloc(20)
# $3 = (void *) 0x6ce81808
p strcpy($3, "my string")
# $4 = 1827149832
x/s $3
# 0x6ce81808: "my string"
</source>

=== Detect current architecture ===
<source lang="python">
show architecture
# The target architecture is set automatically (currently i386)
</source>
See also gdb python.

=== Source file on specific architecture ===
For instance, to source gdb dashboard only on i386:
<source lang="python">
arch=gdb.execute("show architecture", to_string=True).strip()
if arch.find('i386')>=0:
gdb.execute("source ~/.gdbinit-gef.py")
end
</source>

=== Breakpoint tips ===
Source: https://nullprogram.com/blog/2024/01/28/

;Continuable assertions
* On X86, we can trigger an INT 3 followed by a NOP so that GDB breaks at the point of the assert

<source lang="c">
#define breakpoint() asm('int3; nop')
</source>

;Named position
* Use a label in a function <code>example</code> (eg. <code>loop</code>), then use <code>break example:loop</code> to set a symbol break that will adapt at each rebuild.
<source lang="c">
// gdb: b example:loop
void example(double *nums, int n, ...)
{
for (int i = 0; i < n; i++) {
loop: // named position at the start of the loop
// ...
}
}
</source>
* Use an ASM label (one by module allowed). If defined in several module, all will be covered by a single bkp.
<source lang="c">
#//gdb: b b
for (int i = 0; i < n; i++) {
asm ("b:");
// ...
}
</source>
</source>



Latest revision as of 12:45, 29 January 2024

References

GDB front-ends

There are several types of front-end for GDB:

  • GDB built-in front-end (using TUI, Text User Interface)
  • Front-end that customizes heavily .gdbinit file.
  • Front-end using the newer MI2 interface.
  • Front-end using the older MI interface.

More front-ends:

GDB built-in
gdbinit front-end
IDE with gdb integration
Emacs is frequently touted as having the best GDB integration (better than DDD for instance) [1]. Must check for some tutorials [2].
MI2 front-ends
Seen on Mozilla rr.
Only for LLDB.
Use old shell implementation, not the new neovim terminal.
Read some comment it was not convenient, and would require extra plugin like https://github.com/notEvil/vim-debug.
MI front-end (old interface)

GDBINIT Hacks

  • Using colout one may color almost any gdb output.

Tools and libraries

  • Capstone, the ultimate disassembly engine.

Tutorials

GDB TUI mode, python, reverse debugging (with record).

Alternatives to GDB

rr

rr is the Record and Replay Framework, developped by Mozilla, and is a replacement for and powerful enhancement of gdb.

See rr for more information.

LLDB

 ☒  TODO: Have a look at LLDB

The debugger from Apple, mostly compatible with GDB with more powerful feature, but for LLVM/CLang only.

No debugger

Some people don't use / like debuggers:

I don't like debuggers. Never have, probably never will. I use gdb all the time, but I tend to use it not as a debugger, but as a disassembler on steroids that you can program.
The most effective debugging tools are: your brain, a unit test, and the print statement.
Any programmer not writing unit tests for their code in 2007 should be considered a pariah (*).

GDB configuration

GDB reads file ~/.gdbinit at start.

Some references:

Bare minimum configuration

From StackOverflow [3]:

set history save on
set print pretty
set output-radix 16
set height 0

Install Python modules for GDB

From pwndbg [4]:

# Find the Python version used by GDB.
PYVER=$(gdb -batch -q --nx -ex 'pi import platform; print(".".join(platform.python_version_tuple()[:2]))')
PYTHON=$(gdb -batch -q --nx -ex 'pi import sys; print(sys.executable)')
PYTHON="${PYTHON}${PYVER}"

# Find the Python site-packages that we need to use so that
# GDB can find the files once we've installed them.
SITE_PACKAGES=$(gdb -batch -q --nx -ex 'pi import site; print(site.getsitepackages()[0])')
# or to install in user mode:
SITE_PACKAGES=$(gdb -batch -q --nx -ex 'pi import site; print(site.getusersitepackages())')

# Install Python dependencies
sudo ${PYTHON} -m pip install --target ${SITE_PACKAGES} -Ur requirements.txt

Example of file requirements.txt

pip
pycparser
psutil>=3.1.0

Front-ends

Gef

Simple install script [5]:

# via the install script
$ wget -q -O- https://github.com/hugsy/gef/raw/master/scripts/gef.sh | sh

# manually
$ wget -O ~/.gdbinit-gef.py -q https://github.com/hugsy/gef/raw/master/gef.py
$ echo source ~/.gdbinit-gef.py >> ~/.gdbinit

Dependencies:

  • Install keystone-engine, NOT keystone (fix error AttributeError: 'module' object has no attribute 'KS_ARCH_X86'):
pip3 install --no-binary keystone-engine keystone-engine

Tips:

  • Gef may work with a second pane in tmux. This uses the tmux-setup / tmux-start commands, but the most reliable is to identify the PTS device with tmux list-panes -a -F '#D #{pane_tty}', and then assign the appropriate device to variable redirect in ~/.gef.rc.
tmux list-panes -F '#D #{pane_tty}'
# %0 /dev/pts/9
# %1 /dev/pts/10
  • Use context (or alias ctx) to re-print gef context.
  • Use db &myvar to dump (byte) at some variable location.

GDB dashboard

GDB dashboard is a modular visual interface for GDB in Python.

To install simply copy .gdbinit as ~/.gdbinit

cp gdb-dashboard/.gdbinit ~/.gdbinit


Alternatively, source it from ~/.gdbinit:

source ~/.gdbinit-dashboard
Install pygments

Install pygments to get source highlighting

sudo pip install Pygments        # Globally
pip install Pygments             # Locally

If GDB uses python3 (ldd $(which gdb))), you'll need to install with pip3:

sudo pip3 install Pygments       # Globally
pip3 install Pygments            # Locally

To get the list of available styles:

python from pygments.styles import get_all_styles as styles
python for s in styles(): print(s)

Alternative styles:

Issues
  • Cannot write the dashboard: [Errno 10] No child process issue #56
This is a bug in the gdb vendor. Fix file platform.py:
@@ -1009,7 +1009,10 @@
     except (AttributeError,os.error):
         return default
     output = string.strip(f.read())
-    rc = f.close()
+    try:
+        rc = f.close()
+    except:
+        rc = 0
     if not output or rc:
         return default
     else:
  • Cannot write the dashboard: Invalid character '?' in expression.
Likely due to incorrect character in info reg. The register is evaluated via parse_and_eval

Voltron

See Voltron.

GDB TUI

Some example inspired from Greg Law video on Youtube:

  • Start TUI mode with Ctrl-X A, or gdb --tui, or tui enable.
  • Ctrl-L repaint the screen.
  • Ctrl-X 2 to split the layout (repeat to cycle through layout), or layout split.
  • Ctrl-X 1 for back to source-only or disas-only layout (depending on the split-layout we left).
  • tui reg float for instance to change the register view.

Note that while in TUI mode, up and down keys will scroll the source code:

  • Up / Down scroll the source code.
  • Ctrl-P / Ctrl-N Previous or Next command in command history.
  • Ctrl-O switch focus (does not work for me).

CGDB

Probably one of the best lightweight front-end out there, with vim-like bindings.

  • Esc or i to switch between modes.
  • Ctrl-W to switch between vertical / horizontal layout (requires latest cgdb from source).
  • j, k, Ctrl-D, Ctrl-U to navigate (and more vim-like bindings as well)
  • Space to set a breakpoint, t for a temporary breakpoint
  • / and n for search and next
  • :help for help (when in cgdb mode)
  • :set dis or :set nodis to view disassembly.

cgdb works nicely with rr:

rr replay -d cgdb

See rr page for more options.

Prepare debug session

  • Compile with debug symbols, use option -g:
gcc -g program.c               # -g : debug symbols
gcc -g -O0 program.c           #  ... -O0: disable optimization
  • Force core dumps (see bash help ulimit):
ulimit -c unlimited
./a.out
# Segmentation fault (core dumped)

GDB invocation

gdb a.out
gdb a.out core.1234           # If coredump available

GDB commands

Reference

Break points and watch points

b [+-][NUMBER]
break [+-][NUMBER]
b LOCATION
break LOCATION

Set a breakpoint at current line, at given line NUMBER or NUMBER lines after/before current line.

Set breakpoint at LOCATION.

b main sets a breakpoint at beginning of function main().
b foo.c:42 sets a breakpoint at file foo.c, line 42.
b foo.c:loop sets a breakpoint at file foo.c, label loop:.
b *0x400120 sets a breakpoint at address 0x400120.

hb [+-][NUMBER]
hb LOCATION

Set a HARDWARE breakpoint at current line, at given line NUMBER or NUMBER lines after/before current line.

Set a HARDWARE breakpoint at LOCATION.

b LOC if COND

Set a conditional breakpoint.

b main.c:24 if a == 2 Set a breakpoint at main.c:24 when var a is 2.

tb [+-][NUMBER]
tbreak [+-][NUMBER]
tb LOCATION
tbreak LOCATION

Idem, but temporary breakpoint.

ignore N COUNT

Set ignore-count of breakpoint N to COUNT

command [BKPIP]
...
end

Set commands to execute when a breakpoint is hit

watch EXPR

Stop execution when EXPR changes

awatch EXPR

Stop execution when EXPR is accessed

i b
info b
info break

list breakpoints

cl [LOCATION]
clear [LOCATION]

Clear breakpoint by LOCATION, or current line

d
delete

Delete all breakpoints

d NUMBER
delete NUMBER

Clear breakpoint by NUMBER (as listed by i b)

dis NUMBER
disable NUMBER

Disable breakpoint by NUMBER (as listed by i b).

dis $bpnum to disable last bkp (useful for .gdbinit).

save b FILE
save breakpoints FILE

Save current breakpoints as script FILE. Use source to reload.
Registers

i r
info reg

List integer registers

Execute program

r [ARGS]
run [ARGS]

Start (or restart) program. Arguments may include wildcards (*) and redirections (<, <<...)

start [ARGS]

Start and break on main

kill

Kill current program.

c
cont

Continue and interrupted program.

s [NUMBER]
step [NUMBER]

Step (into) current line, or NUMBER lines.

si [NUMBER]
stepi [NUMBER]

Step one (assembly) instruction exactly (N times)

n
next

Run to next line (over current line)

ni [NUMBER]
nexti [NUMBER]

Step one (assembly) instruction exactly (N times), but over subroutine calls.

fin
finish

Execute till returning from current selected frame.

adv LOCATION advance LOCATION

Run until temporary breakpoint set at LOCATION.

u [NUMBER]
until [NUMBER]

Execute until the program reaches a source line greater than current (very handy).
View stack

bt [COUNT]
backtrace [COUNT]
where [COUNT]

Print backtrace of all stack frames, or innermost (outermost) COUNT frames if COUNT>0 (COUNT<0)

f [FRAME]
frame [FRAME]

Select frame FRAME and print stack frame

up

Go up a level in the stack (frame calling current frame).

do
down

Go down a level in the stack (frame called by current frame).
View / Change memory

disp/FMT EXPR
disp EXPR
display EXPR

Display EXPR at each prompt (if within scope), using format FMT. If omitted, reuse last format FMT (of last x, p or disp). Use *ptr to display pointed data, and *ptr@LEN to display array of LEN elements.

x[/nfu] ADDR
x/8zb ADDR
x/16i ADDR
x/16zw ADDR

Examine n memory locations at ADDR, format as n (xduotacfszr) and unit size u (bhwg). If omitted, reuse last format FMT. Pro tip: Use always format 'z', not 'x'.

  • /i for a single assembly instruction. /16i for 16 instructions
  • /8zb for 8 bytes with leading zeroes.
  • /16zw for 16 32-bit words with leading zeroes.
  • x/16i $PC use current PC addr

i lo
i args
info locals
info args

Print information on local variables / function arguments in the current frame

p/FMT EXPR
p EXPR
print EXPR

print EXPR, using format FMT. If omitted, reuse last format FMT (of last x, p or disp).

Use *ptr to display pointed data, and *ptr@LEN to display array of LEN elements. Pro tip: Use always format 'z', not 'x'

undisplay NUMBER

Undisplay expression by NUMBER.

set VARIABLE=VALUE

Write VALUE in VARIABLE (this modifies memory).
View code

l
list

List (10 by default) lines of current frame

disas /m
disas /s
disas 'foo.c'::bar
disas main,+40
disas $pc,+40
disassemble

Disassemble a specified section of memory
Miscellaneous

q
quit

Quit gdb.

help COMMAND
apropos WORD

Get help on COMMAND, or search commands related to WORD.

source FILE

Source script FILE.
RETURN Repeat last command.

add-symbol-file FILE ADDR

Load symbols from FILE at given ADDR.
Keyboard shortcuts
  • C-L — Clear the screen (useful when pgm writes to the screen).

Dump / restore

For instance, to load an Intel hex file:

restore firmware.hex

To generate the same file:

dump ihex memory firmware.hex 0x00400000 0x0040FFFF

Note that dump can also save the result of an expression: dump [format] memory filename start_addr end_addr dump [format] value filename expr

Reverse debugging

References
Example session

Say we have the following program:

void error(void) 
{
    // ...
}

void foo(void)
{
    // ...
    bar();          // Calls error() if something goes wrong
    baz();
    // ...
}

We want to debug function bar. We can use gdb reverse debugging as follows:

gdb myprog
# Start recording when entering 'bar'
b bar
commands
    record
    continue
end
# Break if error
b error
# Stop recording when we exit bar
tbp baz
    record stop
    continue
end

If an error occurs, we can use the following commands to walk backward (see reference for more):

  • The usual step or next, but in reverse: reverse-next or rn, reverse-nexti or rni, reverse-step or rs, reverse-stepi or rsi.
  • Reverse continue (until next breakpoint, or record start): reverse-continue or rc.
  • Reverse finish: reverse-finish.

Recording may be very slow. A faster alternative is to use rr.

Troubleshooting
  • When getting the following messages:
Process record does not support instruction 0xfae64 (xsave)

Process record does not support instruction 0xc5 at address

...
This mostly happens when gdb encounters an instruction it cannot handle yet, such as AVX instructions. Usually it occurs when a library is loaded to detect which CPU binary to use [6], [7], [8]
  • The fix is either to run gdb with LD_BIND_NOW=1 (LD_BIND_NOW=1 gdb a.out) to avoid exotic instruction due to ld.
  • Or compile with -static (clang -g -static hello.c).
  • gdb prints the following message and exits:
# [1]+   Stopped                LD_BIND_NOW=1 gdb a.out
fg
# LD_BIND_NOW=1 gdb a.out
# Process record: failed to record execution log.
#
# Program stopped
This is simply indicative that program probably segfaulted. Do fg to resume gdb and start debug backward.
This does not seem to happen when using cgdb.
  • Watch point ignored when reverse executing
When setting a watchpoint, gdb goes up to program start and prints No more reverse-execution history.
watch *(long **)0x7fffffffffd4a8
# Hardware watch point 4: *(long **)0x7fffffffffd4a8
rc
# Continuing
#
# No more reverse-execution history.
# main () at bubble_sort.c:34
# 34          unsigned  i = 0;
There is a bug in gdb that makes it ignore hardware watchpoint in reverse execution... The fix [9], [10]
set can-use-hw-watchpoints 0
watch *(long **)0x7fffffffffd4a8

Python

See gdb python.

Tips

Use RETURN to repeat last command

Pressing RETURN repeats all last command:

  • Commands like s (step) or n (next). Very handy to step in the code.
  • Command like x/16w 0x10100000 or x/16i 0x10100000, each time printing 16 new words.

Define a custom label for breakpoint in C/C++

Say we want to set a breakpoint at a specified location in source file, but this position may move over time. The easiest is to use an asm statement to define the label [11]:

#include <stdio.h>

int main () {
  void *ret_p = &&ret;
  printf("ret: %p\n", ret_p);
  goto *ret_p;

  return 1;

ret:
  asm("RET:")

    return 0;
}

This will add a symbol table entry as follows.

gcc  -Wl,--export-dynamic t.c  -ldl
readelf -s a.out | grep RET
# 41: 0804858a     0 NOTYPE  LOCAL  DEFAULT   13 RET

Use help to test a command abbreviation

Don't know if f stands for finish or frame? Just use help f:

help f
# Select and print a stack frame.
# ...

8 gdb tricks you should know

From https://blogs.oracle.com/ksplice/entry/8_gdb_tricks_you_should:

  • Use break WHERE if COND
For instance break context_switch if next == init_task.
  • Use command
This sets commands to be executed when a breakpoint is hit. For instance
b do_mmap_pgoff 
# Breakpoint 1 at 0xffffffff8111a441: file mm/mmap.c, line 940.
command 1
# Type commands for when breakpoint 1 is hit, one per line.
# End with a line saying just "end".
>print addr
>print len
>print prot
>end
  • Use gdb --args to specify runtime arguments
gdb --args pizzamaker --deep-dish --toppings=pepperoni
# ...
show args
# Argument list to give program being debugged when it is started is
#  " --deep-dish --toppings=pepperoni".
b main
# ...
  • Finding source files
Use directory to add directory to search for source files.
list main
# 1192    ls.c: No such file or directory.
#     in ls.c
directory ~/src/coreutils-7.4/src/
# Source directories searched: /home/nelhage/src/coreutils-7.4:$cdir:$cwd
list main
Use set substitute-path to fix absolute paths
list schedule
# 5519    /build/buildd/linux-2.6.32/kernel/sched.c: No such file or directory.
#     in /build/buildd/linux-2.6.32/kernel/sched.c
set substitute-path /build/buildd/linux-2.6.32 /home/nelhage/src/linux-2.6.32/
list schedule
  • Use gcc -ggdb3 to add MACRO's symbol as gdb macro
  • Use gdb's variable like $1 but also define your own
set $foo = 4
p $foo
# $3 = 4
  • Use register variables
All CPU registers are available as variables like $R0, $R1, but gdb also defines cross-architectures ones like $sp and $pc. For instance, if $rsi register is used to pass the 1st parameter,
break write if $rsi == 2
  • The x command
x/s 0xffffffff81946000
# ffffffff81946000 <>:     "root=/dev/sda1 quiet"

# Use x/i as a quick way to disassemble memory
x/5i schedule
#   0xffffffff8154804a <schedule>:   push   %rbp
#   0xffffffff8154804b <schedule+1>: mov    $0x11ac0,%rdx
#   0xffffffff81548052 <schedule+8>: mov    %gs:0xb588,%rax
#   0xffffffff8154805b <schedule+17>:    mov    %rsp,%rbp
#   0xffffffff8154805e <schedule+20>:    push   %r15

# Print code surround current pc
x/20i $ip-40
  • Use the @ symbol
The following works if source code is something like int a[ 10 ] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };:
p a
# $1 = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
But if the array is much bigger, or array is a C++ vector, we can use @:
p *&a[0]@10
# $1 = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
p *&a[550]@4
# $1 = {551, 552, 553, 554}

Use value history variables $

See summary list below. For more details, check GDB manual on Value History.

$1          # History value 1
$2          # History value 2
$           # MOST RECENT value in the history
$$          # ... and value before that
$$2         # ... and value before that
$$0         # Same as $
$$1         # Same as $$

For instance:

p *$        # Dereference last value
p *$.next   # ... follow the chain
<RETURN>    # ... again
<RETURN>    # ... again
show values    # Print the last 10 values.

Disassemble at an arbitrary address

disas FUNCTION command only works for defined FUNCTION. To disassemble at an arbitrary address:

  • x/i 0x10100000 one instruction at address 0x10100000,
  • x/16i 0x10100000 (16 instruction) at address Ox10100000,
  • x/16i *0x10100000 (16 instruction) at address contained at Ox10100000,
  • x/16i $PC disassemble 16 instructions at PC.

Redirect STDIN

Redirect STDIN with the run command:

gdb ./myprog
run <input.txt            # Run with stdin redirected

One can also pass parameters with the run command (or use gdb --args command-line parameter).

Exit gdb if run fails

Use python script

def tryRun():
    try:
        gdb.execute("run")
    except:
        print("Exception occured during gdb command: run")
        gdb.execute("quit 1")

Define custom gdb command

Define custom gdb command using define:

# define in .gdbinit
define sf
  where        # find out where the program is
  info args    # show arguments
  info locals  # show local variables
end

See also Gdb python to define more powerful custom commands.

Load symbols from another ELF file

This is useful in embedded systems where some part of the code are already loaded or stored in a ROM. When the debugger stops in a function for which it doesn't have the symbols, it cannot decode the stack and provide a correct backtrace. The fix is to load the symbols for that function using the command add-symbol-file.

add-symbol-file a.out 0               # Add symbol from file a.out, at offset 0

Typically the offset to give is the lowest address in the ELF file.

Modify memory content

Use set [12]. Simplest is to modify a variable:

set variable i = 10

But GDB can write in any memory location:

set {int}0x83040 = 4
set *((int *)0x83040) = 4

Remote debug with gdbserver

Start the gdb server on the remote machine:

gdbserver --multi 192.168.20.1:6666     # CLIENT_ADDR:REMOTE_PORT

This will listen on port 6666, waiting for connection from client 192.168.20.1

On the client, start the GDB client:

gdb-multiarch ./my_exec                 # multiarch necessary if client/server are different arch

Then, in GDB:

target extended-remote {REMOTE_ADDR}:{REMOTE_PORT}
set remote exec-file /path/on/remote/to/my_exec

If the remote executable uses dynamic libraries, these must be sent to the client. To accelerate the loading, copy first the libraries locally on the client, then tell GDB where to find them [13]:

set sysroot /d/st/courses/training_obfu/formation_reverse/dynamic/tp/qmessage/pi_root
set solib-absolute-prefix /d/st/courses/training_obfu/formation_reverse/dynamic/tp/qmessage/pi_root
set solib-search-path /d/st/courses/training_obfu/formation_reverse/dynamic/tp/qmessage/pi_root

Reverse debugging

From Greg Law's incredible video.


Assume we have an application that crashes at random. We want to debug it when it crashes. First we setup some breakpoint and start recording automatically:

gdb a.out
start

break main
command
record
continue
end

break _exit.c:32              # Hack. to break on exit
command
run
end

We also make sure pagination is disabled:

set pagination off

Now we cont until the code crashes. We start to reverse debug the program:

p $pc
# 1 = (void (*)()) 0x5e0c5d00
x 0x5e0c5d00
# 0x5e0c5d00     Cannot access memory at address 0x5e0c5d00
bt
# ... the stack is corrupted. Let's step back...
reverse-stepi
# 0x00000000004806ac in main () at bubble_sort.:43
# 43      }
# ... we're back in normal land
bt
# 0x00000000004806ac in main () at bubble_sort.:43
disas
# 0x00000000004806ac <+135>:     retq
# ... so indeed we tried to return, but likely the stack is corrupted

Let's look at our stack:

print $sp
# 2 = (void *) 0x7fffffffdc98
print *(long **) 0x7fffffffdc98
# 3 = (long *) 0x5e4c5d00
# ... this the address we would jump to if RETQ
watch *(long **) 0x7fffffffdc98
# Hardware watchpoint 4: *(long **) 0x7fffffffdc98

We have set our hardware watchpoint, and will reverse continue! This will bring us to the point where the corruption occured:

reverse-continue
# old value = (long *) 0x5e4c5d00
# new value = (long *) 0xffff7a36ec5 <__libc_start_main+245>
# 0x00000000004806a8 in main () at bubble_sort.:37
print i
4 = 35

We were indeed overwriting the stack because of array overflow!

Use Catch to break on syscall

This was used by HN to debug a complex bugs in some Python profiler [14]:

# This looks for "lseek" followed by "close" without intervening "read".
set height 0

catch syscall close
catch syscall read
catch syscall lseek
disable 1 2

commands 2
 disable 1 2
 continue
end

commands 3
 if $rdi == 31
  enable 1 2
  continue
 else
  continue
 end
end

Printing and Examing memory

Getting rid of octal
  • By default, gdb uses octal' to print char string, which is a major PITA. Use p/x or p/a to work-around that, or use examine (x)
Dealing with examine default format / unit size
  • The complete examine syntax is x/nfu addr, where n is the repeat count, f is the format (same as for print, ie. one of x, d, u, o, t, a, c, f, s), and u is the unit size (one of b, h, w, g).
  • If not specified, examine uses a default value for the format (x initially) and unit size (w for x format, g for a format). But the default changes every time you use print or examine!

Printing arrays and pointers (to arrays)

Given the C code:

uint8_t arr[8] = {0x12, 0x34, 0x56, 0x78, 0x11, 0x22, 0x33, 0x44};
uint8_t *p = arr;

Using print or p:

# -- print: arr and *p
p arr
p *p@8
# $1 = "\022\064Vx\021\"3D"
p/a arr
p/x arr
p/a *p@8
p/x *p@8
# $2 = {0x12, 0x34, 0x56, 0x78, 0x11, 0x22, 0x33, 0x44}
p *p
# $3 = 0x12

# -- print: &arr, p and &p
p &arr
# $4 = (uint8_t (*)[8]) 0x404030 <arr>
p p
# $4 = (uint8_t *) 0x404030 <arr> "\022\064Vx\021\"3D"
p/a &arr
p/a p
# $5 = 0x404030 <arr>
p/x &arr
p/x p
# $3 = 0x404030
p &p
# $1 = (uint8_t **) 0x404040 <p>
p/a &p
# $2 = 0x404040 <p>
p/x &p
# $3 = 0x404040

# -- print: More cases with @
p arr@2
# $8 = {"\022\064Vx\021\"3D", "\000\000\000\000\001\000\000"}
p/a arr@2
p/x arr@2
# $9 = {{0x12, 0x34, 0x56, 0x78, 0x11, 0x22, 0x33, 0x44}, {0xa2, 0xa1, 0xb2, 0xb1, 0x0, 0x0, 0x0, 0x0}}

p p@8
# $7 = {0x404030 <arr> "\022\064Vx\021\"3D", 0x100000001 <error: Cannot access memory at address 0x100000001>, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}
p/a p@8
# $8 = {0x404030 <arr>, 0x100000001, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}
p/x p@8
# $9 = {0x404030, 0x100000001, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}

Using examine or x:

# -- examine: arr, &arr, p (all points to same address)
# ... We force FORMAT and UNIT SIZE to avoid relying on previous default setting.
x/xw arr
x/xw &arr
x/xw p
# 0x404030 <arr>: 0x78563412
# ... From now on, all examine will use hexadecimal, and unit size for hex is 'word'.
x/xg arr
x/xg &arr
x/xg p
# 0x404030 <arr>: 0x4433221178563412
# ... From now on, all examine will use hexadecimal, and unit size for hex is 'giant'.
x/xb arr
x/xb &arr
x/xb p
# 0x404030 <arr>: 0x12
# ... From now on, all examine will use hexadecimal, and unit size for hex is 'byte'.
x/8b arr
x/8b &arr
x/8b p
# 0x404030 <arr>: 0x12    0x34    0x56    0x78    0x11    0x22    0x33    0x44

# -- examine: address 'a' format is always SIGNED, which may gives strange results:
# ... same as 'x' if positive:
x/xw arr
x/aw arr
# 0x404030 <arr>: 0x78563412
# ... but different is negative. This is because 'a' takes the value as an address (negative=stack).
set arr[3]=0x80
x/aw arr
# 0x404030 <arr>: 0xffffffff80563412
x/xw arr
# 0x404030 <arr>: 0x80563412

# -- examine: &p (address of pointer p itself)
x/aw &p
# 0x404050 <p>:   0x404030 <arr>
# ... note again difference with 'hexadecimal' format:
x/xw &p
# 0x404050 <p>:   0x00404030

# -- examine: bad cases (usually)
x *p
# 0x1:  Cannot access memory at address 0x12
x &p@8
# Only values in memory can be extended with '@'.
x/xb arr
x/xb arr@8
# 0x404030 <arr>: 0x12
# ... @8 has no effect. One must use x/8xb
Notes
  • We can use both p/x and p/a to remove octal format. The small advantage for p/a is that it will also prints to nearest preceding symbol.
p/a some_struct                      # Address mode... but strangely it removes octal. WTF?
  • p and x always default to last used format, unless one is specified. The default for x is x, and changes each time you use x or print [15]:

Printing struct and pointers (to struct)

Given the C code:

struct {
    uint16_t a;
    uint16_t b;
} S = {0xa1a2,0xb1b2}, *s = &S;

Using print or p:

# -- print: S and *s
p S
p/a S
p/x S
p *s
p/a *s
p/x *s
# $9 = {
#   a = 0xa1a2, 
#   b = 0xb1b2
# }

# -- print: &S, s and &s
p &S
p s
# $1 = (struct {...} *) 0x404038 <S>
p/a &S
p/a s
# $2 = 0x404038 <S>
p/x &S
p/x s
# $3 = 0x404038
p &s
# $4 = (struct {...} **) 0x404040 <s>
p/a &s
# $5 = 0x404040 <s>
p/x &s
# $6 = 0x404040

# -- print: More cases with @
p S@2
p/a S@2
p/x S@2
# $2 = {{
#     a = 0xa1a2, 
#     b = 0xb1b2
#   }, {
#     a = 0x0, 
#     b = 0x0
#   }}
p s@2
p/a s@2
# $2 = {0x404038 <S>, 0x0}
p/x s@2
# $2 = {0x404038, 0x0}

Using examine or x:

# -- examine: &S, s, in hexadecimal format
x/xw &S
x/xw s
# 0x404038 <S>:   0xb1b2a1a2
x/xg &S
x/xg s
# 0x404038 <S>:   0x00000000b1b2a1a2
x/xb &S
x/xb s
# 0x404038 <S>:   0xa2

# -- examine: &S, s, in address format
x/aw &S
x/aw s
# 0x404038 <S>:   0xffffffffb1b2a1a2
x/ab &S
x/ab s
# 0x404038 <S>:   0xffffffffffffffa2
x/ag &S
# 0x404038 <S>:   0xb1b2a1a2
# ... This is because address is 0x00000000b1b2a1a2, and leading zeroes are removed.

# -- examine: bad cases (usually)
x/xb S
x/xb *s
# Value can't be converted to integer.
x/xb &S@8
# Only values in memory can be extended with '@'.
x/xw s
x/xw s@4
# 0x404040 <s>:   0x00404038
# ... @8 has no effect. One must use x/4xw

Hook on quit

From Stackoverflow:

define hook-quit
    set confirm off
end

Hook on stop

Say you have a custom cmd that you'll like to execute whenever GDB breaks:

define hook-stop
    cmd
end

Enable core dump and load in gdb

Enable core dump file with ulimit:

ulimit -c unlimited

Check they are well enabled:

ulimit -a
# core file size          (blocks, -c) unlimited
# data seg size           (kbytes, -d) unlimited
# ...

Use gdb -c core to load the core dump:

./a.out
# Segmentation fault (core dumped)
l core*
# -rw-------+ 1 user user 240K Jun 16 16:35 core
gdb -c core

Or use any other front-end:

cgdb -c core

Check /proc/sys/kernel/core_pattern for the naming pattern for core files.

Send SIGINT automatically after some time

Use this timeout:

#! /bin/bash

# Source: https://github.com/dscho/msys/blob/master/msys/packages/bash/2.04/examples/scripts/timeout

#Newsgroups: comp.unix.admin,comp.unix.solaris,comp.unix.shell
#From: gwc@root.co.uk (Geoff Clare)
#Subject: Re: timeout -t <sec> <unix command> (Re: How to give rsh a shorter timeout?)
#Message-ID: <EoBxrs.223@root.co.uk>
#Date: Fri, 13 Feb 1998 18:23:52 GMT

#
# Conversion to bash v2 syntax done by Chet Ramey <chet@po.cwru.edu
# UNTESTED
#

prog=${0##*/}
usage="usage: $prog [-signal] [timeout] [:interval] [+delay] [--] <command>"

SIG=-TERM	# default signal sent to the process when the timer expires
timeout=60	# default timeout
interval=15	# default interval between checks if the process is still alive
delay=2		# default delay between posting the given signal and
		# destroying the process (kill -KILL)

while :
do
	case $1 in
	--)	shift; break ;;
	-*)	SIG=$1 ;;
	[0-9]*)	timeout=$1 ;;
	:*)	EXPR='..\(.*\)' ; interval=`expr x"$1" : "$EXPR"` ;;
	+*)	EXPR='..\(.*\)' ; delay=`expr x"$1" : "$EXPR"` ;;
	*)	break ;;
	esac
	shift
done

case $# in
0)	echo "$prog: $usage" >&2 ; exit 2 ;;
esac

(
	for t in $timeout $delay
	do
		while (( $t > $interval ))
		do
			sleep $interval
			kill -0 $$ || exit
			t=$(( $t - $interval ))
		done
		sleep $t
		kill $SIG $$ && kill -0 $$ || exit
		SIG=-KILL
	done
) 2> /dev/null &

exec "$@"

Example of use:

timeout -INT 300 :5 -- gdb $(APP)

Write into memory

Use set VARIABLE = VALUE.

Modify program variables

For instance, given a program

int main(void){
    char[] person = "Bob";
    char[] p2 = "Alice";

    printf("Hello %s\n");
}

We can do

set main::person = { 'S', 'a', 'm', 0x00 }
set main::person = "Sam"
Write at memory address
set {char}0x43f800 = 0x01
set {int}0x43f800 = 0x04030201
set {char [4]}0x43f800 = {0x01, 0x02, 0x03, 0x04}
Call functions with side effect

Alternatively we can use strcpy [16]:

p malloc(20)
# $3 = (void *) 0x6ce81808
p strcpy($3, "my string")
# $4 = 1827149832
x/s $3
# 0x6ce81808: "my string"

Detect current architecture

show architecture
# The target architecture is set automatically (currently i386)

See also gdb python.

Source file on specific architecture

For instance, to source gdb dashboard only on i386:

arch=gdb.execute("show architecture", to_string=True).strip()
if arch.find('i386')>=0:
    gdb.execute("source ~/.gdbinit-gef.py")
end

Breakpoint tips

Source: https://nullprogram.com/blog/2024/01/28/

Continuable assertions
  • On X86, we can trigger an INT 3 followed by a NOP so that GDB breaks at the point of the assert
#define breakpoint() asm('int3; nop')
Named position
  • Use a label in a function example (eg. loop), then use break example:loop to set a symbol break that will adapt at each rebuild.
// gdb: b example:loop
void example(double *nums, int n, ...)
{
    for (int i = 0; i < n; i++) {
        loop:  // named position at the start of the loop
        // ...
    }
}
  • Use an ASM label (one by module allowed). If defined in several module, all will be covered by a single bkp.
#//gdb: b b
for (int i = 0; i < n; i++) {
    asm ("b:");
    // ...
}

Examples

Simple Segmentation Fault Example

(From [17])

Example program segfault.c:
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char **argv)
{
  char *buf;

  buf = malloc(1<<31);

  fgets(buf, 1024, stdin);
  printf("%s\n", buf);

  return 1;
}
Compile and launch gdb:
gcc -g segfault.c
gdb a.out

The debug session

run
backtrace
frame 3
print buf
kill
break segfault.c:8
run
print buf
next
print buf

Fix the bug, then start again, watching now buf:

watch buf
# Start again, answer 'y' when asked to start from beginning
run
# Break at watch point, let's _c_ontinue
c

More GDB stuff

display i            # Display i at each stop
display/4i $pc       # Display the 4 next instruction at each stop
u                    # Continue until we reach an higher pc address

Using gdb-dashboard

da                   # Print dashboard again
da reg               # Remove register
da reg               # Add it back
da m w 0x1000 0x10   # Watch memory zone at 0x1000 for 16 bytes
da m w 0x2000 0x10   # Add a second zone at 0x2000

Frequently used commands

info reg
x/i $pc

GDB script

Breakpoint command lists

See help commands (or here)

Example script:

break foo if x>0
commands
silent
printf "x is %d\n",x
cont
end
  • Use silent to make the breakpoint silent.
  • Use condition (here if x>0) on breakpoint to break conditionally.

Troubleshooting

Python GDB