dm-crypt Causing System Freezes
For the past year, I had been having frequent system freezes on my NixOS desktop and laptop. The system would become completely unresponsive, to the point where only SysRq commands would work.
The freezes occured particularly often, and repeatably, during the following events:
nixos-rebuild- Downloading large files e.g. a 33GB Immich album
Initially I thought the issue was due to the IO scheduler being used (mq-deadline on my desktop with a SATA SSD, none on my laptop with an NVME SSD). However, even after changing the scheduler, there was no improvement.
The IO scheduler
You can check the scheduler being used for a particular device with:
To change the scheduler:
Generally, bfq is recommended for interactive systems as it guarantees disk latencies similar to idle, at the expense of throughput.
You can see a comparison of schedulers here, as well as on the Arch Linux Wiki.
The Culprit
As part of exploration I ran sar -d 1, and noticed that the average disk queue size (aqu-sz) became excessively large during the above operations.
I found that by simulating large writes (e.g. with openssl rand -out bigfile.bin $((1024 * 1024 * 30000))), I was able to increase the queue size, to the point where disk utilization hovered at 100% and thereafter the system would lockup.
Finally, I found an article on Reddit that suggested dm-crypt was the culprit. Turns out, Cloudflare also did a writeup on the issue.
Essentially, dm-crypt maintains a separate workqueue for IO, which is actually unnecessary on fast drives like SSDs. For reasons I do not fully understand yet, this can severely degrade performance under load. I suspect this is due to running the IO scheduler over a workqueue (in this case, a sorted binary tree), instead of allowing it to access block IO directly.
Fixing
You can check whether dm-crypt uses the workqueues:
❯ sudo dmsetup table
luks-primary: 0 487315456 crypt aes-xts-plain64 :64:logon:cryptsetup:7b61f149-1dc8-499b-b5b6-078e05031c1b-d0 0 8:2 32768 2 no_read_workqueue no_write_workqueue
vg-nixos: 0 487309312 linear 254:0 2048
In the example above, the workqueues have already been disabled (no_read_workqueue and no_write_workqueue respectively).
To change them:
# Temporary
# To make persistent, add --persistent
cryptsetup --perf-no_read_workqueue --perf-no_write_workqueue refresh DEVICE