Troubleshooting

Kernel panic: not syncing: read the message, boot an older kernel, rebuild initramfs

Kernel panic not syncing: decode VFS: Unable to mount root fs, Attempted to kill init and out of memory panics, boot an older kernel and rebuild initramfs.

Published ~8 min read
On this page

Short answer: Kernel panic - not syncing: is the prefix the Linux kernel prints before it stops for good; the words after the colon are the actual reason. Read that reason, boot the previous kernel from the GRUB menu, and if that one works, rebuild the initramfs or reinstall the broken kernel from the running system.

the line to read · formats from the kernel source
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
Kernel panic - not syncing: System is deadlocked on memory
Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. ...
---[ end Kernel panic - not syncing: <same reason> ]---

Every line above is built from a format string in the mainline kernel source (kernel/panic.c, init/do_mounts.c, kernel/exit.c, mm/oom_kill.c and init/main.c), read for this article; the device numbers and exit code are examples. "Not syncing" is a leftover from old kernels: Linux 2.4 called sys_sync() to write cached data to disk before halting, unless the panic hit an interrupt handler or the idle task, and then printed "not syncing". Current kernels never sync on panic, so the prefix always says it. Nothing on this page was triggered on purpose here, a panic is not something to test on a server that matters.

Which panic is it?

Text after not syncing:What happenedFirst thing to try
VFS: Unable to mount root fs on unknown-block(0,0)The kernel could not turn root= into any block deviceOlder kernel, then rebuild the initramfs
VFS: Unable to mount root fs on "sda1" or unknown-block(8,1)The device opened, but no filesystem the kernel tried could mount itCheck root= and the filesystem type
Attempted to kill init! exitcode=0x...PID 1 (systemd, or the initramfs /init) exited or was killedDecode the exit code, older kernel, repair from a rescue ISO
System is deadlocked on memoryOut of memory and the OOM killer found no process it was allowed to killFind what pinned memory or set oom_score_adj
Out of memory: ... panic_on_oom is enabledvm.panic_on_oom is set, so OOM panics instead of killing a processDecide whether you want that setting
No working init foundNone of /sbin/init, /etc/init, /bin/init or /bin/sh could be startedWrong root mounted, or a broken userland

The lines just above the panic line often say more than the panic line itself. When the kernel cannot open the root device it first prints VFS: Cannot open root device "..." or unknown-block(...): error ..., then Please append a correct "root=" boot option; here are the available partitions: and a list of every partition it does see. If that list is empty, the kernel has no driver for your disk at this point of the boot. When it opened the device but could not mount it, it prints No filesystem could mount root, tried: and the filesystem types it tried. A Call Trace: block follows for panics raised deeper in the kernel; for those, the function names at the top of the trace are what to search for.

VFS: Unable to mount root fs

The numbers in brackets are the major and minor device number the kernel ended up with. unknown-block(0,0) means the kernel tried to mount root itself and got no device at all from root=. The kernel alone understands device names, PARTUUID=, PARTLABEL= and device numbers, but not the root=UUID=... that distributions use; that form is resolved by the initramfs. So on a distribution kernel this panic usually means the initramfs for this kernel was not loaded or could not be unpacked. Typical causes:

  • A kernel update was interrupted after the kernel was installed but before its initramfs was built.
  • The /boot partition ran full during the update, leaving a truncated initramfs.
  • A GRUB entry has no initrd line, so the kernel starts without an initramfs. (A wrong UUID with a working initramfs ends in an initramfs shell instead of this panic.)

The form with a quoted name and a non-zero pair, such as "sda1" or unknown-block(8,1), means the device was opened but no filesystem the kernel tried could mount it: a wrong partition, a filesystem module that is not in the initramfs, or a damaged filesystem. Press e on the GRUB entry and check that the linux line carries the right root= and that an initrd line follows it. The kernel’s parameter list also documents rootwait and rootdelay= for disks that show up late, such as USB.

Attempted to kill init! exitcode=0x...

PID 1 is not allowed to end. When it exits or dies, kernel/exit.c panics with the exit status in hex. The kernel stores a normal exit as the code shifted left by 8 bits, and a fatal signal as the signal number in the low bits, so you can decode it in any shell (this ran here):

bash
$ printf '%d\n' $((0x00000100 >> 8))     # exited with status 1
1
$ printf '%d\n' $((0x00007f00 >> 8))     # status 127: a command was not found
127
$ printf '%d\n' $((0x0000000b & 0x7f))   # killed by signal 11 (SIGSEGV)
11

An exit status means PID 1 (the initramfs /init or the system’s own init) ended with an error; the lines printed just before the panic usually say which step failed. A signal such as 11 (segmentation fault) or 4 (illegal instruction) means init itself crashed, so check its binary and libraries from a rescue system, for example after an upgrade that was cut off halfway. For exit status 127 in general, see exit code 127.

Out of memory and no killable processes

Normally the OOM killer ends one process and the system lives on, which is covered in OOM-killed processes. Two cases end in a panic instead:

  • Nothing can be killed. The kernel logs Out of memory and no killable processes... and, for a system-wide shortage, panics with System is deadlocked on memory. This happens when every candidate is protected, for example with oom_score_adj set to -1000, or when the memory is held by the kernel itself rather than by processes.
  • You asked for a panic. With vm.panic_on_oom = 1 the kernel panics with Out of memory: system-wide panic_on_oom is enabled; with 2 it says compulsory and panics even for a cgroup limit. The kernel documentation lists these settings for cluster failover; the default is 0.
bash · read only
sysctl vm.panic_on_oom kernel.panic
free -h
grep -lxF -e -1000 /proc/[0-9]*/oom_score_adj 2>/dev/null | wc -l   # protected processes

If the box simply has too little memory for its workload, adding swap buys headroom while you size it properly.

Boot an older kernel from the GRUB menu

If the panic started right after an update, the previous kernel is usually still installed. Open the console and reboot. When the menu is hidden, the GRUB manual says holding Shift or pressing Esc or F4 during the countdown shows it. On Debian and Ubuntu, grub-mkconfig puts older kernels in the Advanced options submenu, so pick the second newest there; Fedora and the RHEL family build the menu from one Boot Loader Specification entry per kernel in /boot/loader/entries, so the kernels are listed directly. Once it is up:

bash
uname -r                       # the kernel that works
ls /lib/modules                # every installed kernel version
sudo dmesg --level=err,warn    # anything odd on this boot

Then rebuild the initramfs for the broken kernel as shown next, or remove and reinstall that kernel package. dmesg shows driver errors on the working boot that may explain why the newer one fails.

Regenerate the initramfs, also from a rescue system

Debian · Ubuntu
sudo update-initramfs -u -k all        # every installed kernel that has one
sudo update-initramfs -c -k 6.1.0-XX-amd64   # create it if it is missing
sudo update-grub
RHEL · Rocky · Alma · Fedora
sudo dracut -f --regenerate-all
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Per its man page, update-initramfs -k all with -u works on every installed kernel that already has an initramfs; -c creates one. Replace the version with one from ls /lib/modules. dracut’s --regenerate-all rebuilds every image at the default location for the kernel versions found on the system, and -f lets it overwrite existing files.

If no kernel boots, start a live or rescue ISO if your provider offers one, mount the installed system and chroot into it with the same bind mounts as in grub rescue, then run the commands above inside the chroot. Do not rely on uname -r there: it reports the rescue system’s kernel, not yours. Check free space on /boot with df -h /boot first, since a full /boot is a common reason the image was broken.

kernel.panic: reboot automatically after a panic

By default the kernel waits forever after a panic: the PANIC_TIMEOUT build option defaults to 0, and the sysctl documentation defines kernel.panic as 0 to loop forever, a positive number to reboot after that many seconds, and a negative number to reboot at once. Your distribution or image may set another value, so check it:

bash
sysctl kernel.panic kernel.panic_on_oops
# reboot 10 seconds after a panic, now and on every boot:
echo 'kernel.panic = 10' | sudo tee /etc/sysctl.d/90-panic.conf
sudo sysctl --system

The same setting exists as the boot parameter panic=10. With kernel.panic_on_oops = 1 an oops also panics, and the documentation notes that a non-zero kernel.panic then reboots the machine. On a server that runs unattended a short timeout is usually what you want, because it comes back on its own. While you are still hunting a panic, set it to 0 for a while so the message stays on the console instead of vanishing into a reboot.

If the system gets past the kernel but stops in a maintenance shell, read emergency mode instead.

Frequently asked

What does kernel panic not syncing mean?

It is the prefix the Linux kernel prints when it hits an error it cannot recover from and stops. Not syncing is a leftover from old kernels, which wrote cached data to disk before halting unless they could not; current kernels never sync on panic. The text after the colon is the real reason, for example VFS: Unable to mount root fs or Attempted to kill init.

How do I fix VFS: Unable to mount root fs on unknown-block(0,0)?

unknown-block(0,0) means the kernel tried to mount root itself and could not resolve root= to any device; it cannot read root=UUID= without an initramfs, so the initramfs for that kernel was usually missing, truncated or not loaded. Boot the previous kernel from the GRUB menu, then run sudo update-initramfs -u -k all and sudo update-grub on Debian and Ubuntu, or sudo dracut -f --regenerate-all on RHEL, Rocky and Alma. If no kernel boots, do the same from a rescue ISO inside a chroot.

What does Attempted to kill init exitcode=0x00000100 mean?

PID 1 exited, which the kernel never allows. A normal exit is stored as the status shifted left by 8 bits, so 0x00000100 is exit status 1 and 0x00007f00 is 127; a value in the low bits such as 0x0000000b is a signal (11, SIGSEGV). An exit status means init ended with an error, and the lines just above the panic say which step failed; a signal means init itself crashed.

How do I make a Linux server reboot automatically after a kernel panic?

Set kernel.panic to a number of seconds, for example kernel.panic = 10 in /etc/sysctl.d/90-panic.conf followed by sudo sysctl --system, or add panic=10 to the kernel command line. The kernel's built-in default is 0, which waits forever. While you are debugging, leave it at 0 so the message stays on the console.

Related articles

All troubleshooting