Need to get a dump from fro ma PHP process on Debian 9.
In this post will take a Linux kernel mechanism to create and manage processes dumps.
Kernel’s dumps are created in another way, check Kdump на Arch Wiki.
Linux Core Dump
The kernel will create a process dump if it performed an invalid operation and has to be stopped.
To do so, the kernel will send a special signal to such a process so the process can handle it itself or using standard mechanisms and will cause the kernel to apply the mechanism to create a dump of the memory of this process.
The complete list can be found in the kernel source in the SIG_KERNEL_COREDUMP_MASK macros:
...
#define SIG_KERNEL_COREDUMP_MASK (\
rt_sigmask(SIGQUIT) | rt_sigmask(SIGILL) | \
rt_sigmask(SIGTRAP) | rt_sigmask(SIGABRT) | \
rt_sigmask(SIGFPE) | rt_sigmask(SIGSEGV) | \
rt_sigmask(SIGBUS) | rt_sigmask(SIGSYS) | \
rt_sigmask(SIGXCPU) | rt_sigmask(SIGXFSZ) | \
SIGEMT_MASK
...
Which is used by the sig_kernel_coredump macros:
...
#define sig_kernel_coredump(sig) siginmask(sig, SIG_KERNEL_COREDUMP_MASK)
...
Which is called in case of Fatal-errors and in its turn will call the do_coredump() function:
...
fatal:
...
if (sig_kernel_coredump(signr)) {
if (print_fatal_signals)
print_fatal_signal(ksig->info.si_signo);
proc_coredump_connector(current);
...
do_coredump(&ksig->info);
}
...
And do_coredump() will create a memory dump and will save it on a hard disk.
Signals and dump creation
Let’s check how it’s working.
Take a simple code in C:
#include <stdio.h>
#include <unistd.h>
int main() {
while(1) {
pid_t pid = getpid();
printf ("Working with PID %lu\n", pid);
sleep(5);
}
}
Build and run:
$ gcc make_dump.c -o make_dump
./make_dump
Working with PID 2714790
In another terminal – send a signal, for example SIGSEGV (Segmentation violation), code 11:
$ kill -s SIGSEGV 2714790
In a terminal with the app running check its output:
$ ./make_dump
Working with PID 2714790
...
Working with PID 2714790
Segmentation fault (core dumped)
Check a dump file:
$ ls -l /tmp/ | grep 2714790
-rw------- 1 setevoy setevoy 380928 Mar 10 11:24 coredump-make_dump.2714790
In the same way, you can create a dump for any already running process, for instances – run sleep:
$ sleep 100 &
[1] 2761144
$ kill -s SIGSEGV 2761144
[1]+ Segmentation fault (core dumped) sleep 100
$ file /tmp/coredump-sleep.2761144
/tmp/coredump-sleep.2761144: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from 'sleep 100', real uid: 1000, effective uid: 1000, real gid: 1000, effective gid: 1000, execfn: '/usr/bin/sleep', platform: 'x86_64'
GDB – create a core dump
Besides of a signal sending you can use gcore from the gdb:
$ sleep 100 &
[1] 2762961
$ sudo gcore 2762961
...
Saved corefile core.2762961
$ file core.2762961
core.2762961: ELF 64-bit LSB core file
GDB – read a core dump
To check a dump’s content you can use gdb and pass an executable as a first argument and a path to a dump file as a second argument:
$ gdb make_dump coredump-make_dump.2714790
...
[New LWP 2714790]
Core was generated by `./make_dump'.
Program terminated with signal SIGSEGV, Segmentation fault.
0 0x00007f50d566e27e in clock_nanosleep@GLIBC_2.2.5 () from /usr/lib/libc.so.6
(gdb)
kernel.core_pattern
During dump creation kernel will check the kernel.core_pattern parameter which determines how the dump will be handled.
Here you can specify a path and a filename with specificators or pass it external dumps handlers like systemd-coredump (рассмотрим ниже).
Check the Naming of core dump files documentation here>>>.
Most used options here are:
-
%eexecutable filename (without path prefix) -
%pPID of dumped process, as seen in the PID namespace in which the process resides -
%ttime of dump, expressed as seconds since the Epoch, 1970-01-01 00:00:00 +0000 (UTC)
Our file /tmp/coredump-make_dump.2714790, which we observed in the GDB above consists of the kernel.core_pattern = /tmp/coredump-%e.%p:
-
/tmpcatalog -
%e– file name started with the coredump + an executable name файла make_dump -
%p– and a killed process PID – 2714790
Also, instead of using the direct path and file name you can pass dump data via a pipe | to the /dev/null or to a handler like systemd-coredump.
limits.conf
Before creating a dump the kernel also will check soft and hard limits for the core in the /etc/security/limits.conf:
$ cat /etc/security/limits.conf | grep core
- core - limits the core file size (KB)
* soft core 0
Or check directly in your current environment:
$ ulimit -c
unlimited
A running process limits can be found in its limits from the /proc:
$ ./make_dump
Working with PID 2753034
And limits:
$ cat /proc/2753034/limits
Limit Soft Limit Hard Limit Units
...
Max core file size unlimited unlimited bytes
...
fs.suid_dumpable
Sometimes a dump can be not created if a process made a suid-operation for example by using the setuid() syscall.
In this case, the behavior is controlled by the fs.suid_dumpable:
$ sysctl fs.suid_dumpable
fs.suid_dumpable = 2
It can accept 0, 1 or 2, see the man proc:
- 0: (default) This provides the traditional (pre-Linux 2.6.13) behaviour. A core dump will NOT be produced for a process which has changed credentials (by calling seteuid or similar) or whose binary does not have read permission enabled.
- 1: (“debug“) All processes dump core when possible.
- 2: (“suidsafe“) Any binary which normally would not be dumped (see “0” above) is dumped readable by root only.
systemd-coredump
The systemd of course, has its own dumps handler – already mentioned systemd-coredump.
On Arch Linux is used by default, for Debian 9 can be installed with apt:
$ sudo apt -y install systemd-coredump
Config file – /etc/systemd/coredump.conf.
After installation configure the kernel – send dumps via a pipe to the systemd-coredump:
root@bttrm-production-app-1:/home/admin# echo '|/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %e' > /proc/sys/kernel/core_pattern
Check it:
root@bttrm-production-app-1:/home/admin# cat /proc/sys/kernel/core_pattern
|/lib/systemd/systemd-coredump %P %u %g %s %t 9223372036854775808 %e
Create some dump:
root@bttrm-production-app-1:/home/admin# sleep 10 &
[1] 27117
root@bttrm-production-app-1:/home/admin# kill -s 11 27117
[1]+ Segmentation fault (core dumped) sleep 10
coredumpctl
To work with dumps managed with systemd-coredump the coredumpctl utility can be used:
root@bttrm-production-app-1:/home/admin# coredumpctl
TIME PID UID GID SIG COREFILE EXE
Mon 2020-03-09 20:10:16 EET 20882 0 0 11 present /home/admin/dump_test
...
Tue 2020-03-10 09:04:14 EET 16786 1003 1003 11 present /usr/sbin/php-fpm7.4
Tue 2020-03-10 12:23:43 EET 27117 0 0 11 present /bin/sleep
In the SIG column, we can see a signal sent to the process killed, in those cases it were 11 == SIGSEGV.
Dump files are located at the /var/lib/systemd/coredump directory:
root@bttrm-production-app-1:/home/admin# ll /var/lib/systemd/coredump/
total 125376
-rw-r----- 1 root root 25208 Mar 9 20:10 core.dump_test.0.6bb23c691d354e9dbf4382d109a5c1d4.20882.1583777416000000000000.lz4
...
-rw-r----- 1 root root 33962 Mar 10 12:23 core.sleep.0.6bb23c691d354e9dbf4382d109a5c1d4.27117.1583835823000000000000.lz4
To observe a dump’s content use info with some MATCH condition, for example, a PID:
root@bttrm-production-app-1:/home/admin# coredumpctl info 27117
PID: 27117 (sleep)
UID: 0 (root)
GID: 0 (root)
Signal: 11 (SEGV)
Timestamp: Tue 2020-03-10 12:23:43 EET (3min 3s ago)
Command Line: sleep 10
Executable: /bin/sleep
...
Hostname: bttrm-production-app-1
Storage: /var/lib/systemd/coredump/core.sleep.0.6bb23c691d354e9dbf4382d109a5c1d4.27117.1583835823000000000000.lz4
Message: Process 27117 (sleep) of user 0 dumped core.
Stack trace of thread 27117:
0 0x00007fc1a8053270 __nanosleep (libc.so.6)
1 0x000056159fa6f91f n/a (sleep)
2 0x000056159fa6f700 n/a (sleep)
3 0x000056159fa6c9a4 n/a (sleep)
4 0x00007fc1a7fbb2e1 __libc_start_main (libc.so.6)
5 0x000056159fa6ca7a n/a (sleep)
Debian – core dump is not created
Everything described above is working perfectly on two laptops with Arch Linux, but on my project’s servers with Debian 9 dumps are created in a usual way, without systemd-coredump.
The core_pattern config:
root@bttrm-production-app-1:/home/admin# cat /proc/sys/kernel/core_pattern
/tmp/core-%e.%p
Run a process, kill it with 11 signal, check /tmp – no dumps here.
Run an app again:
root@bttrm-production-app-1:/home/admin# ./dump_test
Working with PID 27954
And check its limits:
root@bttrm-production-app-1:/home/admin# cat /proc/27954/limits
Limit Soft Limit Hard Limit Units
...
Max core file size 0 unlimited bytes
...
The Soft limit is set to the zero value, e.g. dumps are disabled at all as kernel can’t use more than 0 bytes for a dump file.
Set it to the unlimited:
root@bttrm-production-app-1:/home/admin# ulimit -S -c unlimited
Run an app again, check limits now:
root@bttrm-production-app-1:/home/admin# ./dump_test 1>/dev/null &
[3] 28399
root@bttrm-production-app-1:/home/admin# cat /proc/$!/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size unlimited unlimited bytes
...
Create a dump:
root@bttrm-production-app-1:/home/admin# sleep 10 &
[4] 28613
root@bttrm-production-app-1:/home/admin# kill -s 11 $!
Check it:
root@bttrm-production-app-1:/home/admin# ls -l /tmp/coredump-sleep.28613
-rw------- 1 root root 380928 Mar 10 12:47 /tmp/coredump-sleep.28613
Done.
Useful links
- Understand and configure core dumps on Linux
- Hunting the Core
- Core dump (Arch Wiki)
- Generate PHP core dumps on segfaults in PHP-FPM
- Generating a gdb backtrace
- Проблема с eaccelerator и генерация coredump для php-fpm
- How to get a core dump for a segfault on Linux
- kill: Creating a core dump
Similar posts
- 01/23/2020 Debian: php7.3-curl : Depends: libcurl3 (>= 7.44.0) but it is not installable
- 01/27/2017 [Debian: apt – GPG error: The following
- 02/24/2020 Linux: PHP-FPM, Docker, STDOUT and STDERR – no an application’s error logs
Top comments (0)