Enable and Analyse Core Dumps in Linux
Purpose
To debug an executable one can run it under control of gdbserver/gdb. This also allows to debug issues with abnormal termination caused by receiving signals.
See Hello World application on Embedded Linux for a possible setup.
For cases which do not allow to run gdb core dumps can help in debugging issues with abnormal termination.
The Linux kernel can write a file containing the state of a process when the process receives certain signals, e.g. segmentation fault or illegal instruction. Such files are called core dumps and contain a snapshot of the allocated memory and registers and can later be used with gdb to analyse the causes of the signal. See also 'man core'.
Core Dumps on CPU Signals
Preconditions
Make sure the Linux kernel has the needed configuration options set as we don't enable these by default. Check and if needed recompile the kernel with enabled core dump options.
Check for 3.0, 3.1 kernels:
zcat /proc/config.gz | grep CONFIG_ELF_CORE
Check for 3.7 and later kernels:
zcat /proc/config.gz | grep CONFIG_COREDUMP
Enabling Core Dumps
The following enables core dump generation when a user process triggers a signal and sets the core file location to /tmp/ with a sensible name. Note that these settings aren't stored persistently.
ulimit -c unlimited
echo 1 > /proc/sys/kernel/core_uses_pid
echo "/tmp/core-%e-%s-%u-%g-%p-%t" > /proc/sys/kernel/core_pattern
Should you run an executable with 'suid' or 'guid' rights you also need the following:
echo 2 > /proc/sys/fs/suid_dumpable
Depending on the memory usage of a process core files can be huge. Enabling core dumps on a system can fill up its mass storage over time. To enable core dumps on production systems it is highly recommended to use a daemon to manage the core dumps (e.g. make sure that the core dumps do not use more than a certain amount of space). Systemd might be a viable solution for this, however the coredump functionality is not enabled by default in OpenEmbedded (see recipes-core/systemd/systemd_xxx.bb in OpenEmbedded core).
Trigger a Core Dump
To test the creation of core dumps, one can use the kill command to send a segmentation fault signal to the process:
kill -11 [PID]
Or alternatively using killall which also allows to use the signal name:
killall -SIGSEGV firefox
Create a Core Dump of a Running Application
To create a single core dump without provoking an abnormal process termination the utility gcore can be used.
This allows to create a core dump at any time of execution. The utility gcore is part of gdb, one can install it using the package feeds:
opkg install gdb
Use gcore with the process ID as argument to create a core dump of a user process.
# gcore -o /tmp/core-myapp 280
0x76f46588 in read () from /lib/libc.so.6
Saved corefile /tmp/core-myapp.280
Analysing Core Dumps
A cross gdb can be used to analyse a core file. Additionally to the core dump gdb needs the executable and all linked static object files. One can use the OpenEmbedded built cross gdb and the target sysroot for that.
Copy the core file to your development PC into the OpenEmbedded sysroot and use gdb to do the analysis:
In oe-core/build/out-glibc/sysroot/colibri-t30:
../x86_64-linux/usr/bin/armv7ahf-vfp-neon-angstrom-linux-gnueabi/arm-angstrom-linux-gnueabi-gdb -ex 'set sysroot .' -ex 'core-file ./core-firefox-4-0-0-596-1402701200' usr/lib/firefox/firefox