Troubleshooting

Segmentation fault (core dumped): what SIGSEGV and exit code 139 mean, and how to fix it

Segmentation fault (core dumped) means SIGSEGV, exit code 139. Find the crashing library in dmesg, enable core dumps, get a backtrace with gdb or Python.

Published ~8 min read
On this page

Short answer: “Segmentation fault” means the program touched memory it is not allowed to touch, so the kernel killed it with signal 11 (SIGSEGV); the shell reports exit code 139 (128 + 11), and adds “(core dumped)” when a core file was written. Find out where it crashed before you change anything:

first look
sudo dmesg -T | grep -i segfault | tail -5     # which binary or library crashed
sudo journalctl -k -g segfault --since today    # same lines from the journal
ulimit -c                                       # 0 means no core file is written
cat /proc/sys/kernel/core_pattern               # where a core file would go

If the crash is in a program you run from a package, the kernel line usually tells you which library to look at. If it is your own code or a Python script, a backtrace (gdb, faulthandler or AddressSanitizer, all below) points at the exact line.

What the error looks like in bash, sh, zsh and containers

The message is not printed by the program: it dies before it can print anything. The shell that started it sees the signal and prints its own text, so the wording depends on the shell:

WhereWhat you seeExit status
bash, interactiveSegmentation fault, or Segmentation fault (core dumped) when a core file was written139
bash -c or a scriptbash: line 1: 3017049 Segmentation fault python3 ...139
sh (dash on Debian and Ubuntu)Segmentation fault139
zshzsh: segmentation fault ./program139
Docker, Kubernetescontainer exits with exit code 139139
systemd servicestatus=11/SEGV in systemctl statusunit fails

To show the real thing without a buggy program, Python can crash itself on purpose by reading address 0 through ctypes. This is what bash 5.2 and dash printed on our Debian 12 host, run in a scratch folder:

output · Debian 12
$ bash -c 'python3 -c "import ctypes; ctypes.string_at(0)"; echo "exit=$?"'
bash: line 1: 3017049 Segmentation fault      python3 -c "import ctypes; ctypes.string_at(0)"
exit=139
$ sh -c 'python3 -c "import ctypes; ctypes.string_at(0)"; echo "exit=$?"'
Segmentation fault
exit=139
$ kill -l 139
SEGV

No “(core dumped)” here: ulimit -c is 0 on that host, so no core file was written. The zsh wording above comes from the zsh source code (its signal message table and job report), which also sets $? to 128 plus the signal number; we did not run zsh. A similar looking status, 126, means the file could not be started at all, which is an exec format error and a different problem.

What SIGSEGV and exit code 139 mean

The signal(7) man page lists SIGSEGV as signal 11 on x86 and ARM, described as “Invalid memory reference”, with the default action “Core”: terminate the process and write a core dump. The GNU bash manual says a command killed by fatal signal N exits with 128+N, so SIGSEGV gives 139. A program can only cause it, never ignore it safely: typical triggers are dereferencing a NULL or freed pointer, writing past the end of a buffer, writing to read-only memory, and running out of stack.

Do not confuse it with running out of memory. The OOM killer sends SIGKILL, the shell says Killed and the exit status is 137 (128 + 9). If that is what you see, read why Linux killed your process instead.

Read the kernel's segfault line

A segfault in a process without its own signal handler normally leaves one line in the kernel log (the kernel rate-limits them when many arrive at once). These are the lines from our test runs, read with dmesg (the same lines are in journalctl -k):

output · Debian 12
python3[3017049]: segfault at 0 ip 00007ff53dac6ad8 sp 00007ffe3fa717a8 error 4 in libc.so.6[7ff53d985000+156000] likely on CPU 1 (core 1, socket 0)
python3[3017681]: segfault at 0 ip 00007f0e5444cad8 sp 00007ffc1fd8ed38 error 4 in libc.so.6[7f0e5430b000+156000] likely on CPU 5 (core 5, socket 0)
crash[3017998]: segfault at 0 ip 00007fb4e6d24a32 sp 00007fffae23a258 error 6 in libc.so.6[7fb4e6be3000+156000] likely on CPU 7 (core 7, socket 0)
deep[3020410]: segfault at 7ffea69b2c6c ip 000055ed6f4a6144 sp 00007ffea69b2c60 error 6 in deep[55ed6f4a6000+1000] likely on CPU 5 (core 5, socket 0)
  • python3[3017049]: the program and its PID.
  • segfault at 0: the bad address. 0 or a small number is a NULL pointer. An address within a few bytes of sp (the stack pointer), as in the fourth line, is a stack overflow.
  • error 4 or error 6: the x86 page fault code. 4 is a read from user space, 6 a write.
  • in libc.so.6[...]: the file the crashing instruction lives in. A crash inside libc almost always means the caller passed it a bad pointer; libc is the place, not the cause. A crash in a library you recently upgraded or copied in by hand is a strong lead.

Common causes and how to tell them apart

CauseHow it shows upFix
Bug in your own C, C++ or Rust unsafe codeSame address and line every runBacktrace with gdb or AddressSanitizer, fix the pointer
Mismatched shared library after an upgradeCrash in a .so from /usr/local/lib or LD_LIBRARY_PATHCheck ldd ./program, remove the stray copy
Broken Python C extension or wheelfaulthandler stops in one import or package callReinstall that package in a fresh venv
Stack overflow (deep recursion)Fault address next to spFix the recursion; raise ulimit -s only as a test
Faulty RAM (rare)Different programs crash at random addressesMemory test on bare metal; on a VPS ask the provider

After an upgrade: check which libraries are loaded

A program built against one version of a library can crash when it gets an incompatible one with the same file name. That happens with a hand-copied .so in /usr/local/lib, an LD_LIBRARY_PATH left in a profile, or an application bundle that ships its own copies. Compare what the program resolves with what the package manager installed:

bash
ldd ./program                      # every library and the file it resolves to
echo "$LD_LIBRARY_PATH"            # should usually be empty
env -u LD_LIBRARY_PATH ./program   # does it still crash without it?

If a library is missing instead of wrong, the program does not even start and you get a different message, covered in error while loading shared libraries.

Python scripts that segfault

Python code by itself should not segfault, its errors become exceptions. When a Python process dies with SIGSEGV, the crash is almost always in compiled code: a C extension (numpy, a database driver, an image library), a wheel built for a different version, or ctypes. The other route is very deep recursion after sys.setrecursionlimit() was raised, when every level passes through C code (a __repr__ that calls repr() on itself, for example): on our Python 3.11 that overflowed the C stack and segfaulted, while plain Python recursion ended in RecursionError. Python 3.12 added a separate guard for such C level recursion. Turn on the faulthandler module, which the Python documentation enables with -X faulthandler or PYTHONFAULTHANDLER=1, and Python prints its own traceback just before it dies. Python 3.11 on our host:

output · Python 3.11
$ python3 -X faulthandler -c "import ctypes; ctypes.string_at(0)"
Fatal Python error: Segmentation fault

Current thread 0x00007f3410e26040 (most recent call first):
  File "/usr/lib/python3.11/ctypes/__init__.py", line 519 in string_at
  File "<string>", line 1 in <module>
/bin/bash: line 1: 3017676 Segmentation fault      python3 -X faulthandler -c "import ctypes; ctypes.string_at(0)"

The top frames name the package whose compiled part crashed. Recreate the virtual environment and reinstall that package so pip fetches a wheel that matches your Python; on Debian 12 and Ubuntu 24.04 that has to happen in a venv anyway, see externally managed environment.

Enable core dumps

A core file is a snapshot of the process memory at the moment of the crash. The core(5) man page names the two settings that matter most for whether one is written: the RLIMIT_CORE limit (what ulimit -c shows) and /proc/sys/kernel/core_pattern. Raise the limit for the current shell only, then reproduce the crash:

bash
ulimit -c unlimited          # this shell and its children only
cat /proc/sys/kernel/core_pattern
./program                    # crash again, now "(core dumped)"

What core_pattern contains decides where the file goes:

  • core (the kernel default): a file named core in the current directory of the crashed process. With the limit raised in a scratch folder, our test printed Segmentation fault (core dumped) and left a 5 MB core that file core identified as ELF 64-bit LSB core file ... from 'python3 -c import ctypes...'.
  • A line starting with | and systemd-coredump: the core is stored by systemd. List and inspect it with coredumpctl.
  • A line with apport: Ubuntu's crash reporter takes it and writes a report to /var/crash.
systemd-coredump
coredumpctl list                     # recent crashes: time, PID, signal, executable
coredumpctl info 3017049             # details and a short stack trace
coredumpctl dump 3017049 -o core     # write the core to a file
coredumpctl debug 3017049            # open it in gdb directly

Get a backtrace with gdb

A backtrace lists the chain of function calls that led to the crash. With gdb installed (sudo apt install gdb), either load a core file or run the program under gdb. The flags are from the GDB manual: -batch exits after the commands, -ex runs one command, --args passes the rest of the line to the program:

bash
gdb -batch -ex bt ./program core                       # from a core file
gdb -batch -ex run -ex bt --args ./program --its-flags # run until it crashes
gdb -batch -ex "thread apply all bt" ./program core    # every thread

Frames that show ?? instead of function names lack debug symbols. For packaged programs, Debian and Ubuntu serve them over debuginfod: with DEBUGINFOD_URLS set, an interactive gdb asks whether to use it, while a -batch run needs -iex "set debuginfod enabled on". For your own code, rebuild with -g.

C and C++: build with AddressSanitizer

For code you compile yourself, GCC and Clang can add checks that catch the bad access and print where it happened, before or instead of a plain segfault. Our test program passes a NULL pointer to strcpy. Built with GCC 12 on the same host, the plain build only says Segmentation fault, the -fsanitize=address build names the line:

output · gcc 12, paths shortened
$ gcc -g -fsanitize=address -o crash-asan crash.c && ./crash-asan
AddressSanitizer:DEADLYSIGNAL
==3018005==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0x7f5d98985a32 bp 0x7ffe1f04b110 sp 0x7ffe1f04b0f8 T0)
==3018005==The signal is caused by a WRITE memory access.
==3018005==Hint: address points to the zero page.
    #0 0x7f5d98985a32  (/lib/x86_64-linux-gnu/libc.so.6+0x167a32)
    #1 0x55c46aa9e1ab in copy_name .../crash.c:5
    #2 0x55c46aa9e1d4 in main .../crash.c:10

The same build caught our deep recursion test as AddressSanitizer: stack-overflow ... deep.c:2 in depth. Use it on a test or staging build: it makes the program slower and uses more memory. A sanitizer build exits with status 1, not 139.

On a VPS or dedicated server

Before you suspect hardware, read the segfault line. If the same program crashes in the same library at the same offset every time, it is software, whatever machine it runs on. The offset is ip minus the start address in the brackets: for both python3 lines from our runs it is 0x141ad8 in libc.so.6, although the raw addresses differ because of address space randomisation. On a VPS the guest has no access to the host's memory, so faulty RAM is the provider's to test; report it only when unrelated programs crash at random addresses. On a dedicated server, that random pattern (and the CPU number at the end of the line, if it keeps repeating) is the point to run a memory test during a maintenance window.

Frequently asked

What does Segmentation fault (core dumped) mean?

The program accessed memory it was not allowed to, so the kernel killed it with signal 11 (SIGSEGV). The shell prints the message because the program dies before it can print anything. "(core dumped)" means a core file with the process memory was also written, which happens only when ulimit -c allows it or a crash handler such as systemd-coredump collects it.

Why is the exit code 139?

Shells report a process killed by signal N as exit status 128 plus N. SIGSEGV is signal 11, so a segfault gives 139, and kill -l 139 prints SEGV. Docker and Kubernetes report the same 139 for a container whose main process segfaulted.

How do I find out where a program segfaulted?

Start with sudo dmesg -T | grep segfault, which names the program and the library it crashed in. For the exact function, enable core dumps with ulimit -c unlimited and run gdb -batch -ex bt ./program core, or use coredumpctl debug where systemd-coredump is active. Python scripts show their own traceback with python3 -X faulthandler.

Is a segmentation fault caused by bad RAM?

Rarely. If the same program crashes in the same library at the same offset every time, it is a software bug or a mismatched library. Faulty memory shows up as different programs crashing at random addresses; on a dedicated server that is the time for a memory test, on a VPS the host hardware is the provider's to check.

Related articles

All troubleshooting