Rr

rr is the Record and Replay Framework, developped by Mozilla, and is a replacement for and powerful enhancement of gdb.

References

• rr homepage.
• Using rr in an IDE
• Building and Installing and rr Usage, from Mozilla wiki.

• Pernosco A nice front-end for rr (free for users on Github, see HN thread).

Install and configuration

On Debian, rr is available as a package:

sudo apt install rr

rr also requires the linux performance analysis package for the current kernel.

# Install linux perf
sudo apt install linux-perf-4.19

# Enable perf for user land (temporary)
sudo sysctl kernel.perf_event_paranoid=0

# Enable perf permanently - note: =1 might be enough
echo 'kernel.perf_event_paranoid=0' | sudo tee '/etc/sysctl.d/51-rr-enable-perf-events.conf'

# To squeeze max perf on laptop (even plugged on AC!), disable CPU scaling:
sudo apt-get install cpufrequtils
sudo cpufreq-set -g performance          # Must be redone at each reboot !!!
# CPU scaling can be disabled permanently in BIOS

See rr usage page for more performance tuning and setting perf flag permanently.

Integration with cgdb

rr works nicely with cgdb:

rr replay -d cgdb

Example of use

Start recording the application:

rr record my_app

Then replay it with:

rr replay

In case the application fires several processes, we can tell rr to fast-forward until child is forked. First we use rr ps to get the list of recorded processes, then we start rr with -f:

rr ps
rr replay -f 12345

During replay, rr behaves likes gdb, with the usual reverse debugging command (rc, rn, rs...). See referenced links for more information.

NOTE: rr seems to ignore breakpoints if we do run instead of cont...

Troubleshoot

Potential issues in VM

• See this bug in VMWare [1]. Fix is to add monitor_control.disable_hvsim_clusters = true.
• See this bug in Xen [2].
• Discuss on rr in Hacker News.