After migrating to ZFS storage for VM and containers I quickly found that ZFS really loves to use RAM. Which normally is fine as long as it gives up it’s cache space when a VM or container need it, unfortunately I’ve had a handful of times where ZFS is just too slow to free RAM and OOM killer will kick in.

Thankfully ZFS allows us to set a limit to cache size with zfs_arc_max and zfs_arc_min

echo "$[N * 1024*1024*1024]" > /sys/module/zfs/parameters/zfs_arc_min
echo "$[N * 1024*1024*1024]" > /sys/module/zfs/parameters/zfs_arc_max

With N being the amount of RAM in gigabytes to use for ARC cache And to make it permanent

echo "options zfs zfs_arc_min=$[N * 1024*1024*1024]" > /etc/modprobe.d/xfs.conf
echo "options zfs zfs_arc_max=$[N * 1024*1024*1024]" >> /etc/modprobe.d/xfs.conf

Documentation for ZFS suggesting 4gb + 1gb per 1tb of raw storage, on my 12tb array I ended up with 24gb as an easy number and so far the array has been happy.