Read-only file system: why Linux stops writing, and how to repair or remount safely

Short answer: “Read-only file system” (error code EROFS, errno 30) means the kernel refuses to write to the filesystem that holds the path. There are two very different reasons. Either it is mounted read-only on purpose (a container mount, a systemd sandbox, a snap, the macOS system volume), or the kernel switched it to read-only after errors to protect your data, which is what happens when a server’s disk suddenly stops taking writes. Two commands tell you which:

findmnt -T /PATH/YOU/WRITE/TO -o TARGET,SOURCE,FSTYPE,OPTIONS
sudo dmesg -T | grep -iE 'remounting|I/O error|EXT4-fs error|XFS|BTRFS'

• The kernel log is quiet and the options say ro: the filesystem was mounted read-only on purpose. Find out by whom: containers, systemd, snap, macOS, WSL, or an intentional read-only mount.
• The log shows Remounting filesystem read-only or I/O errors: the filesystem hit an error. Do not just remount it read-write; follow the repair steps. On current kernels the options can still say rw in this state (why).

Which message do you see?

Every program words the same EROFS differently. The Linux lines are real output from Debian 12 against read-only mounts; WSL and macOS use their own wording:

“No space left on device” is a different error (ENOSPC): a full disk refuses new data but does not turn read-only. That one is covered in No space left on device.

Why the kernel turns a filesystem read-only

Debian’s installer and the Debian and Ubuntu cloud images put errors=remount-ro on the root filesystem in /etc/fstab. On a Debian 12 server:

/dev/sda1 ext4   rw,relatime,discard,errors=remount-ro

With it, ext4 refuses every further write as soon as it detects corruption, so damage already done is not written over. Without it, the same happens whenever the journal aborts, which is what a failed write to the journal causes. This is the kernel log from a test filesystem on Debian 12 (kernel 6.1), where we injected an error through ext4’s own test switch:

EXT4-fs error (device loop0): trigger_test_error:126: comm bash: test error from scratch
Aborting journal on device loop0-8.
EXT4-fs (loop0): Remounting filesystem read-only
EXT4-fs error (device loop0): ext4_journal_check_start:83: comm touch: Detected aborted journal

On a real server the first EXT4-fs error names the function that found the problem, and when the disk is at fault it follows lines beginning I/O error, dev vda, sector or Buffer I/O error on dev vda1, logical block. Those two decide the repair: I/O errors point at the disk or the storage under it, EXT4-fs error lines alone at the filesystem.

Other filesystems react differently. XFS, the default on Rocky and AlmaLinux, does not turn read-only: it logs Shutting down filesystem and Please unmount the filesystem and rectify the problem(s), and every access then fails with Input/output error. Btrfs logs forced readonly.

findmnt still says rw

Since a 2024 kernel fix (CVE-2024-50191), ext4 no longer flips the mount itself to read-only after an error; it blocks writes internally. What the mount options show therefore depends on the kernel:

On our Debian 12 test, mount -o remount,rw even exited with success, and the next touch failed with Read-only file system again; a remount to ro was refused with cannot mount /dev/loop0 read-only. The reliable check for ext4 is the superblock:

sudo dumpe2fs -h /dev/DEVICE | grep -E '^Filesystem state|^FS Error count'

Filesystem state:         clean with errors
FS Error count:           1

DEVICE is the SOURCE from findmnt, such as vda1 or sda1. clean with errors means: repair offline before you trust this filesystem again.

Repair a filesystem the kernel made read-only

1. 1Save the kernel log first

   If /var/log is on the filesystem that went read-only, these lines never reach the disk, and after a reboot journalctl -k -b -1 will not have them. Copy them to memory-backed /dev/shm now and download the file:

   sudo dmesg -T > /dev/shm/dmesg-readonly.txt

2. 2Copy off what you cannot lose

   A read-only filesystem is still readable. Pull uploads, configuration and database files to another machine before anything else touches the disk, for example with rsync -a root@SERVER:/var/www/ ./rescue/ run from that machine.

3. 3Decide whether the disk is failing

   I/O errors in the log on a dedicated server: check the drive with sudo smartctl -a /dev/sda (or /dev/nvme0) and plan its replacement. On a VPS the virtual disk sits on the provider’s storage: send the log lines to support before you repair, since a filesystem check cannot fix a disk that keeps failing.

4. 4Check the filesystem while it is not mounted

   Reboot. A filesystem marked with errors is checked automatically at boot, and if the check can fix it unattended the server comes up normally. If not, Debian and Ubuntu stop at an (initramfs) prompt with The root filesystem on /dev/vda1 requires a manual fsck, and Rocky and Alma at an emergency shell. Neither is reachable over SSH; use the provider’s console, then run the check and continue:

   fsck -y /dev/vda1
   exit

   To force a full check on the next boot instead, add fsck.mode=force fsck.repair=yes to the kernel line in the GRUB menu (press e there). If the server does not boot at all, start your provider’s rescue system if it has one, find the partition with lsblk -f, and check it from there; otherwise restore a snapshot.

5. 5Confirm it is clean

   After the boot, findmnt -no OPTIONS / should show rw, dumpe2fs -h should say clean, and a test write should work. Watch the kernel log for a few days: errors that come back mean the disk, not the filesystem.

The repair command depends on the filesystem; run each only on an unmounted one:

e2fsck -f -y /dev/DEVICE

xfs_repair /dev/DEVICE

btrfs check --readonly /dev/DEVICE

A clean e2fsck run ends with a summary line and exit code 0; exit code 1 means it corrected errors:

e2fsck 1.47.0 (5-Feb-2023)
ro.img: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
ro.img: 11/16384 files (0.0% non-contiguous), 9513/65536 blocks

When mount -o remount,rw is the right fix

Only when the filesystem was mounted read-only on purpose and nothing is wrong with it: a root mounted read-only in single-user mode or after booting with init=/bin/bash (the usual way to reset a lost root password from the console), an ro in /etc/fstab, or a disk you mounted read-only yourself. Then:

sudo mount -o remount,rw /MOUNTPOINT

After an error it is no fix, as the test above shows: depending on the kernel it is refused, or it reports success while writes keep failing.

Docker and Kubernetes

Inside a container, read-only is almost always configuration. A bind mount with :ro is read-only, docker run --read-only makes the container’s own root read-only, and Kubernetes does the same with readOnlyRootFilesystem: true. Keep the read-only root, which is good hardening, and give the program writable places for what it needs:

services:
  app:
    image: IMAGE
    read_only: true
    tmpfs:
      - /tmp
      - /run
    volumes:
      - ./config:/etc/app:ro
      - app-data:/var/lib/app
volumes:
  app-data:

With docker run, the same is --read-only --tmpfs /tmp --tmpfs /run; in Kubernetes, an emptyDir volume mounted at the path. Kubernetes mounts ConfigMap and Secret volumes read-only in any case: change the object, not the file in the pod. Docker also mounts /proc/sys read-only, so writing a sysctl from inside fails: a redirect reports Read-only file system, while procps sysctl -w says permission denied on key "net.ipv4.ip_forward". Set it from outside with docker run --sysctl net.ipv4.ip_forward=1 or sysctls: in Compose; this works for sysctls the kernel keeps per container, such as the net.* ones.

A service fails, the same command works in your shell

That is systemd sandboxing. With ProtectSystem=strict a service sees the whole filesystem read-only except /dev, /proc and /sys; full makes /usr, the boot directories and /etc read-only, and DynamicUser=yes implies strict. Check the unit:

systemctl show SERVICE -p ProtectSystem -p ReadWritePaths -p StateDirectory

Then allow exactly the directories it writes, with a drop-in from sudo systemctl edit SERVICE, and restart the service:

[Service]
ReadWritePaths=/srv/myapp/uploads

For state and logs, StateDirectory=myapp and LogsDirectory=myapp create /var/lib/myapp and /var/log/myapp with the right owner and keep them writable under strict. Writing a unit from scratch is covered in Create a systemd service.

Snap, macOS and WSL

Snap packages

Everything under /snap/NAME is a read-only squashfs image, so editing a file in there fails by design. Change settings with snap set NAME KEY=VALUE where the snap supports it, or in its writable directory /var/snap/NAME/current.

macOS

Since Catalina the system volume is read-only, and since Big Sur it is sealed as well, so sudo mkdir /data fails and mount -uw / is no way around it. Write under your home directory, or, if a tool insists on a top-level path, create the directory in your home, add a link to it in /etc/synthetic.conf and reboot. The two columns are separated by a tab:

printf 'data\tUsers/USER/data\n' | sudo tee -a /etc/synthetic.conf

WSL 2

After a hard shutdown, WSL may mount a distribution read-only and print An error occurred mounting the distribution disk, it was mounted read-only as a fallback. Microsoft’s repair runs e2fsck from Windows, in an administrator PowerShell, with Docker Desktop closed:

wsl.exe --shutdown
wsl.exe --mount PATH\TO\ext4.vhdx --vhd --bare
wsl.exe lsblk
wsl.exe sudo e2fsck -f /dev/sdX
wsl.exe --unmount

lsblk shows which sdX is the disk you just attached. With only one distribution installed, Microsoft notes you may need to install a second one to run the commands in.

On a VPS: the console matters

A server whose root went read-only usually still answers SSH, but the repair in step 4 happens before SSH starts. A VMHeaven KVM server has a VNC console in the panel that shows the (initramfs) prompt or the emergency shell, power controls for the reboot, and backups to restore if the repair cannot save the filesystem. If a server goes read-only, then reboots in a loop, see VPS keeps rebooting.

To see what fills a disk once it is writable again, see Check disk usage on Linux.

Frequently asked