ZFS Filesystem Review

Published Date: 2020/10/17 by: DaVieS

ZFS Filesystem Review

ZFS is a transactional filesystem by Sun Microsystems (ORACLE)
This means: data corruption never should be.

This can be achieved by: checksuming, duplications, and various raid-levels that built-in to ZFS. Yes. ZFS is a volume manager too.

ZFS is popular since FreeBSD introduced it in their releases few years ago.
ZFS has its functionality above other common filesystems and a well-tuned ZFS filesystem mostly can achieve more speed.

I'm using ZFS since its available and here is my review and some tips.

How ZFS works?
Pools and RAID Levels

ZFS its a volume manager too, you are directly can use the disks as "software-raid" arrays.

  • RAIDZ1 - Act as Raid5 with one parity disk

  • RAIDZ2 - Act as Raid6 with two parity disk

  • RAIDZ3 - Triple Parity (Uncommon)

Example: zpool create storage raidz1 /dev/vtbd0 /dev/vtbd1


On 1-Parity (RAID5, RAIDZ1) systems you can continue to use your array in "DEGRADED" mode when you have 1 failing disk from array.

On Double-Parity (RAID6, RAIDZ2) systems you can continue to use your array in "DEGRADED" mode when you have 2 failing disk from array.

On Triple-Parity (RAIDZ3) systems you can continue to use your array in "DEGRADED" mode when you have 3 failing disk from array.

DEGRADED means fully-functional and danger.

Please note that if you are using large disks on large array then be careful with their URE ratings, array from single Desktop HDD may could not be rebuild when its failing due to Disk specific "Unrecoverable Read Errors" which is normal.

Commonly Accepted formula is at least one parity per 3-4 disks. So if you have 12 disks you should user triple-parity or more raid arrays.

If you have 15 disks don't build a single array, make 2-3 arrays then stripe them into one, or use as SAN.

ZFS Array with 12 * 12 TB Enterprise SATA Disks with Triple-Raid (RAIDZ-3)
root@storage:~ # zpool status
  pool: zroot
 state: ONLINE
  scan: none requested
        NAME              STATE     READ WRITE CKSUM
        zroot             ONLINE       0     0     0
          raidz3-0        ONLINE       0     0     0
            mfisyspd0p3   ONLINE       0     0     0
            mfisyspd1p3   ONLINE       0     0     0
            mfisyspd10p3  ONLINE       0     0     0
            mfisyspd11p3  ONLINE       0     0     0
            mfisyspd2p3   ONLINE       0     0     0
            mfisyspd3p3   ONLINE       0     0     0
            mfisyspd4p3   ONLINE       0     0     0
            mfisyspd5p3   ONLINE       0     0     0
            mfisyspd6p3   ONLINE       0     0     0
            mfisyspd7p3   ONLINE       0     0     0
            mfisyspd8p3   ONLINE       0     0     0
            mfisyspd9p3   ONLINE       0     0     0
errors: No known data errors

root@storage:~ # zpool iostat
               capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  ----- 
zroot       57.8T  72.2T      0    484  3.66K  43.7M

ZFS has many misunderstanding let me clear them.
    • ZFS only for servers - No, I'm using it my notebook too, working well, even in linux.


    • ZFS will not work on any LVM, HW Raid, pseudo / virtual disk - Well ZFS meant for JBOD (Just-bunch-of-disks), however working on all scenario, but not safe and fast as much on JBOD.


    • ZFS eats all of the RAM but when the system requires more RAM ZFS will be freeing RAM for system. - Oh well simply no, if even trying to that will be takes more time that you have for it.


    • ZFS eats all of the RAM. - No if you set one parameter you can limit, surprise :)


  • ZFS is very SLOW - No, just on misconfigured environments, but it has a more Hardware Requirement like at least 4GB RAM and 64bit instruction set.

What is the Recommended Requirements for ZFS?
  • JBOD, Just Bunch Of Disks & Direct Access, if this not possible then RAID0 Arrays

  • Any Hardware Cache could slowdown the ZFS

  • 4K Disks with shift=12 alignment

  • RAM depending on filesystem size and files, large files requires less RAM, lot of small files requires more. 128k default recordsize requires less RAM than 4k recordsize. Healthy RAM amount starting from 16GB - 256GB.

Wait.. RecordSize What the hell is it?
In ZFS all files are stored either as a single block of varying sizes (up to the recordsize) or using multiple recordsize blocks.

ZFS doing checksum and metadata by blocks (with size set by recordsize), if you have a lot of random Read/Write like torrent you may set recordsize to physical blocksize, 4K or 16K (torrent write/read 16K chunks at usual).

If you have large files with sequential read / write than you can setup recordsize more than 128K.

For databases 16/32k, for images 128k, virtual images (VDI, VMDK) 4K.

ZFS Tuning Tips

There are two userspace utility: zpool, zfs

'zpool' can handle the "LVM" things.
while 'zfs' can handle the filesystem level.

ZFS has more advantage that you ever imagine, there are partitions but they are dynamically growing types. That means you can set maximum size to partition (quota) but you can have almost unlimited partitions, because they are not reserving space.

ZFS has a power utility, you can realtime set filesystem variables, even recordsizes too. but notice that setting compression, recordsize, ... will be takes effect on newly created files, existing content only can be inherit new settings when you copy, rename (mv) not works.

ZFS can compress your files with zfs set compression=[on/off/[other_algo]] I recommend lz4 because it's CPU friendly.

Turning off sync is not that dangerous as it usually we think, but you can loose data when you set it to disabled but wont affect filesystem integrity.

Dedup is a nice feature on papers, but useless until you not have as much GB of RAM as you TB*7 you have.

ZFS has its own CACHE feature like L1, L2ARC.

L1 ARC is based on RAM while L2ARC only if you have cache (SSD) drive to array.
L1 and L2 is not meant for write-cache, therefore no significant changes on write-performance occurs.

Soo Tuning ..

Some sysctl variables are also available to you.

    • zfs_arc_max = 23750508544 (Allow 24GB RAM to be eaten by L1ARC)
    • zfs_prefetch_disable = 1 (When its disabled could gain some performance speed)
    • zfs_nocacheflush = 1 (if you have NON-JBOD config you can turn off, btw nothing harmful)
    • zfs_txg_timeout = 5 / 60 (Lower degrades performance but smaller data loss on power outage and less hungs, Higher value can increase performance but avoid setting to hard it can freeze the system for short times. )

On Linux: These parameters are can be found in /sys/module/zfs/parameters


root@storage:~ # zfs get all zroot
NAME   PROPERTY              VALUE                  SOURCE
zroot  type                  filesystem             -
zroot  creation              Tue Mar 13 18:06 2018  -
zroot  used                  42.0T                  -
zroot  available             49.5T                  -
zroot  referenced            256K                   -
zroot  compressratio         1.00x                  -
zroot  mounted               yes                    -
zroot  quota                 none                   default
zroot  reservation           none                   default
zroot  recordsize            128K                   default
zroot  mountpoint            /zroot                 local
zroot  sharenfs              off                    default
zroot  checksum              on                     default
zroot  compression           lz4                    local
zroot  atime                 off                    local
zroot  devices               on                     default
zroot  exec                  on                     default
zroot  setuid                on                     default
zroot  readonly              off                    default
zroot  jailed                off                    default
zroot  snapdir               hidden                 default
zroot  aclmode               discard                default
zroot  aclinherit            restricted             default
zroot  canmount              on                     default
zroot  xattr                 off                    temporary
zroot  copies                1                      default
zroot  version               5                      -
zroot  utf8only              off                    -
zroot  normalization         none                   -
zroot  casesensitivity       sensitive              -
zroot  vscan                 off                    default
zroot  nbmand                off                    default
zroot  sharesmb              off                    default
zroot  refquota              none                   default
zroot  refreservation        none                   default
zroot  primarycache          all                    default
zroot  secondarycache        all                    default
zroot  usedbysnapshots       0                      -
zroot  usedbydataset         256K                   -
zroot  usedbychildren        42.0T                  -
zroot  usedbyrefreservation  0                      -
zroot  logbias               latency                default
zroot  dedup                 off                    default
zroot  mlslabel                                     -
zroot  sync                  standard               default
zroot  refcompressratio      1.00x                  -
zroot  written               256K                   -
zroot  logicalused           42.0T                  -
zroot  logicalreferenced     31K                    -
zroot  volmode               default                default
zroot  filesystem_limit      none                   default
zroot  snapshot_limit        none                   default
zroot  filesystem_count      none                   default
zroot  snapshot_count        none                   default
zroot  redundant_metadata    all                    default
root@storage:~ # 

Did you know?
FreeBSD can use ZFS natively, also ZFS is more powerful under BSD.
See this pretty 'top' showing ZFS parameters by native.

last pid: 35802;  load averages:  0.71,  0.59,  0.52                   up 12+20:47:44  15:17:05
73 processes:  1 running, 72 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 7436K Active, 54M Inact, 31M Laundry, 61G Wired, 1296M Free
ARC: 56G Total, 325M MFU, 53G MRU, 36M Anon, 149M Header, 2121M Other
     53G Compressed, 68G Uncompressed, 1.29:1 Ratio
Swap: 24G Total, 11M Used, 24G Free

Am I just dreaming or unlimited snapshots, replication is really a thing? On filesystem-block level?
Yes, you can issue a snapshot from your "partition" by one command, then you can clone, send to another ZFS (replication), rollback, destroy.. as you want.

What About Speed?
No question, if you can tune and configure your filesystem to real-usage ZFS is safest choice even could be fastest.

Finally ...
We have this GEM, use IT! This article created from 5+ years of heavy ZFS usage, no data loss and acceptable speed, and not worrying since replication started :)

If you like the article then don't forget to share!

bhyve webadmin, web control panel

FreeBSD uses bhyve as hypervisor! So.. no questions we are using FreeBSD for web, mailing, devel, for everything.Now we would like to drop KVM /..

Side Quest, BVCP WebUI for FreeBSD Bhyve

Okay, so we are in middle to upgrade and realign our infrastructure and happened days ago with a random facebook talk, someone hinted FreeBSD..

Corrupted innoDB on linux ext4, data recovery

I could say I saw a everthing but not, here is the case: There is a VM Host with ZFS Storage, direct attached, and there is a Linux VPS with ext4..

FreeBSD 13 Just Released

FreeBSD The best operating system for serve WEB and FTP has just announced yesterday, that the newest version is ready for production. One of the..

MVCP 2.0 is our biggest thing ever

We promised to be released back in 2020 but COVID make us busy too.MVCP is our "Webhosting" packed into a single appliance top on the WORLD..

What is R3 as Certificate Issuer?

Don't panic, it is still Let's Encrypt. Let's Encrypt is a free SSL Certificate provider, issuing certificates automatically but only for 3..

Happy New Year, 2021

Thank you, we are happy to see you here again! nPulse.net site and it's services are performed better by more than 50% overall in 2020 compared..

Seems Google have serious problems with gmail.com

It started about a week ago, gmail.com started an agressive rate limiting (DEFER), seems gmail.com have serious problems regarding to all of our..

System Upgrade, PHP 8.0 Failures

So today we had a little downtime partially due to security upgrades, and major version upgrades.nPulse.net is linked to many services, most of..

PayPal going to support crypto currencies

According to the latest news we can confirm that PayPal has entered the cryptocurrency market allowing for their customers to buy and sell..

All rights reserved nPulse.net 2009 - 2021
Powered by: MVCP 2.0-RC / ASPF / PHP 7.4 / NGINX / FreeBSD