PVE Tinkering Notes (3) Creating ZFS on the Hard Disk Originally Used in QNAP

Introduction

After some research, I decided to use ZFS as the file system for the two mechanical hard drives. I could have used ext4 just like QNAP does, but ZFS is safer — it has self-healing capabilities and can handle bit-flip issues. In short, it's just better to use.

If you're looking for flexibility, you can use Btrfs. There's a great write-up on this topic that you can refer to: Data Disk Partitioning: Btrfs File System - Building a NAS/Small Server from Scratch

In one sentence: ZFS is not just a file system — it is an intelligent storage solution.

ZFS (Zettabyte File System) is an open-source, advanced file system and logical volume manager originally developed by Sun Microsystems.

Unlike traditional file systems (such as ext4, NTFS), ZFS combines file system + RAID + volume management all in one.

Key features: self-repair, snapshots, high reliability, large-capacity support.

Check Hard Disk

Mainly uses the two commands fdisk -l and lsblk.

• fdisk -l: View disk partition information (lower-level, raw)
• lsblk: View disk block device tree structure (more intuitive, human-oriented)

Command Explanation:

🧱 fdisk -l

• Full name is "list partition table"

• Displays all disks and their partition information, capacity, file system type (MBR/GPT), etc.

• Requires root privileges (so it is commonly used as sudo fdisk -l)

• Example output:

  pgsql复制编辑Disk /dev/sda: 512 GB
  Device     Boot Start       End   Sectors   Size Type
  /dev/sda1  *     2048    999423    997376   487M EFI System
  /dev/sda2        999424 976773119 975773696 465G Linux filesystem
  

• 📌 More "low-level," helpful for understanding the disk partition table structure.

🌳 lsblk

• Full name is "list block devices"

• Displays all block devices (disks, partitions, LVM, RAID, etc.) and their mount status

• Presents in a tree-like structure, which is more intuitive

• Does not display loop devices with empty mount points (unless additional parameters are added)

• Does not require root privileges by default

• Example output:

  pgsql复制编辑NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
  sda      8:0    0   512G  0 disk
  ├─sda1   8:1    0   487M  0 part /boot/efi
  └─sda2   8:2    0 511.5G  0 part /
  

• 📌 More suitable for quickly identifying which device is mounted to which directory. For example, when you connect a new SSD, it will appear with a name like /dev/sdX or /dev/nvmeXn1.

Backup Data

Refer to my previous content

There are several methods:

• rsync: rsync -a --info=progress2 /mnt/disk1/your-folder /mnt/disk2/
• Midnight Commander (mc) (Visual)

I recommend using mc. Note that if you are operating on a server via SSH, be sure to use a tool like screen to start a background session to prevent the connection from dropping halfway through the copy.

After the backup is complete, please verify the data yourself. You can also refer to my previous article for the method.

Format Hard Disk

The simplest way is to use the Wipe Disk feature in PVE.

Error

However, I encountered an error during usage: disk/partition '/dev/sda1' has a holder

Analyzing the cause, "holder" indicates that there are still kernel objects using this partition.

Common Holders

As long as holders exist, Proxmox's wipe_disk script (essentially sgdisk --zap-all + wipefs -a) will be blocked from writing by the kernel, resulting in a 500 error.

Troubleshooting Method

lsblk -f
ls /sys/block/sda/sda1/holders

Output like dm-0 dm-1 → indicates it is occupied by LVM/MD, etc.

Solution

⚠️ Please verify the disk name for all commands (using /dev/sda as an example here). Incorrect operations will immediately destroy data!

After completing the steps, run ls /sys/block/sda/sda1/holders —— the directory should be empty.

Then, returning to the Web UI and clicking Wipe Disk should no longer result in an error.

Creating ZFS

In the Web UI, select the node (mine is called PVE) - Disks - ZFS - Create ZFS

TL;DR: Choose the disk, fill in the name, select single disk for RAID level (you can choose Mirror if you want redundancy), select lz4 for compression, and set ashift to 12

For those who want to delve deeper, please refer to the following explanation.

Recommended Configuration

RAID Level: Choose based on trade-offs between capacity, performance, fault tolerance, and rebuild speed (mirror/raid10 favors performance and small-scale setups, raidzN favors capacity and simplicity, draidN favors capacity and fast self-healing).

Compression Algorithm: For general use, select lz4 (or on); for maximum compression, consider zstd or gzip.

ashift: For modern disks, choose 12.

What Do These Parameters Mean?

RAID Level (Pool Layout)

Note:

1. Formula: Total Disks ≥ Parity Disks (P) + Data Disks (D) + 1 (hot spare partition); the Proxmox UI only lists draid1/draid2/draid3—no need to manually specify D, just meet the minimum disk requirement.
2. raidzN vs. draidN comparison:
   • raidzN: Dedicated parity disk, single-task rebuild after failure, impacts array performance;
   • draidN: Integrated hot spare, rebuild process distributed in parallel across all disks—faster speed and less impact on online I/O.

Compression

ZFS supports compressing data blocks during writes, saving space and often improving I/O performance (reading with slight CPU cost in exchange for less disk I/O).

ashift (Alignment Size)

• Meaning
  ZFS performs I/O on physical disks in byte sizes that are powers of two. ashift is the exponent (ashift = log2(block size)).
• Common Values
  • ashift=9: 2^9 = 512 B, corresponds to traditional 512-byte sectors.
  • ashift=12: 2^12 = 4096 B, corresponds to modern 4K sectors (SSDs also often write physically in 4K).
• Recommendations
  1. Modern disks (especially SSDs or 4K-sector drives): Always use ashift=12 to avoid performance and lifespan penalties from cross-sector writes.
  2. If you confirm all disks are classic 512B-sector disks, ashift=9 may slightly save space, but it is riskier and now uncommon.

Note: ashift can only be specified when creating the pool and cannot be changed once established, so ensure you choose correctly from the start.

Common Commands

After creating ZFS, you can use the following commands to view all ZVOLs and datasets.

zfs list

Use the following command to view all ZFS pools.

zpool list

Modifying Mount Points

The default mount point is the root directory, using the ZFS pool name as the mount point.

For example, /data1

This can be manually modified.

sudo zfs set mountpoint=/storage/data1 data1

Check the current mount point.

zfs get mountpoint data1

However, after modification, you also need to synchronously update the PVE configuration /etc/pve/storage.cfg.

It will contain a configuration similar to this:

zfspool: data1
    pool data1
    content rootdir,images
    mountpoint /data1

zfspool: data2
    pool data2
    content rootdir,images
    mountpoint /data2

You need to change the corresponding mountpoint to /storage/data1.

Save and exit. /etc/pve is automatically managed and hot-reloaded by Proxmox's cluster configuration filesystem (pmxcfs), and related PVE services (such as container startup) will instantly read changes to this file.

You can use the pvesm status command to view storage, though this command does not output paths.

root@pve:/storage# pvesm status
Name             Type     Status           Total            Used       Available        %
backups           dir     active        98497780        13794364        79653868   14.00%
data1          zfspool     active      3771203584       650276484      3120927100   17.24%
data2          zfspool     active      3771203584      1778071764      1993131820   47.15%
local             dir     active        98497780        13794364        79653868   14.00%
local-lvm     lvmthin     active       847638528         2966734       844671793    0.35%