{"id":113,"date":"2013-10-02T12:34:16","date_gmt":"2013-10-02T12:34:16","guid":{"rendered":"http:\/\/blogs.kent.ac.uk\/unseenit\/?p=113"},"modified":"2013-10-02T12:34:16","modified_gmt":"2013-10-02T12:34:16","slug":"effects-of-zfs-fragmentation-on-underlying-storage","status":"publish","type":"post","link":"https:\/\/blogs.kent.ac.uk\/unseenit\/effects-of-zfs-fragmentation-on-underlying-storage\/","title":{"rendered":"Effects of ZFS Fragmentation on Underlying Storage"},"content":{"rendered":"

For some time we’ve been aware of the effects of ZFS Fragmentation, which typically becomes an issue as a zpool passes 80% full, although we’ve seen it start anywhere between 70% and over 90% depending on the workload. Typically, you see degradation of performance and increased IO load as\u00a0disk writes require finding and sticking together small chunks of free space. There’s a good summary of the gory technical details here<\/a>.<\/p>\n

The proper fix for this is to have a separate ZFS Intent Log (ZIL) device but given the number of physical and virtual servers we have that utilise ZFS root disks, this isn’t practical. Unfortunately, this also doesn’t fix the issue for existing pools and there’s no ZFS defragmentation tool. To resolve the issue you need to delete data or present additional storage.<\/p>\n

The following charts neatly identify this issue – the volume below had a pair of competing jobs – one doing a lot of reads (green), and one doing a small amount of write I\/O (blue, negative axis). A new volume was presented and attached to the zpool at 12:00.<\/p>\n

\"ZFS<\/a>
ZFS IO before presenting new storage<\/figcaption><\/figure>\n

At this point the writes stopped going to the existing fragmented disk, freeing up a couple of hundred IOPS for the read job (to the point that is now CPU-bound). In the chart below, you can see the massively reduced write operations on the new unfragmented volume:<\/p>\n

\"ZFS<\/a>
ZFS IO after adding new storage.<\/figcaption><\/figure>\n

That’s ~200 IOPS, down to ~4 IOPS, a 50-fold amplification of write operations. Couple that with the extra I\/O caused by RAID6 writes, and the underlying Tier-3 SATA disks were being worked extremely hard.<\/p>\n

\"Drive<\/a>
Drive busy percentage<\/figcaption><\/figure>\n

Unsurprisingly, the performance of this filesystem (and the others that share this Raid Group) has increased significantly.<\/p>\n","protected":false},"excerpt":{"rendered":"

For some time we’ve been aware of the effects of ZFS Fragmentation, which typically becomes an issue as a zpool passes 80% full, although we’ve … Read more<\/a><\/p>\n","protected":false},"author":12,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[28946,28948],"tags":[],"_links":{"self":[{"href":"https:\/\/blogs.kent.ac.uk\/unseenit\/wp-json\/wp\/v2\/posts\/113"}],"collection":[{"href":"https:\/\/blogs.kent.ac.uk\/unseenit\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blogs.kent.ac.uk\/unseenit\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blogs.kent.ac.uk\/unseenit\/wp-json\/wp\/v2\/users\/12"}],"replies":[{"embeddable":true,"href":"https:\/\/blogs.kent.ac.uk\/unseenit\/wp-json\/wp\/v2\/comments?post=113"}],"version-history":[{"count":2,"href":"https:\/\/blogs.kent.ac.uk\/unseenit\/wp-json\/wp\/v2\/posts\/113\/revisions"}],"predecessor-version":[{"id":116,"href":"https:\/\/blogs.kent.ac.uk\/unseenit\/wp-json\/wp\/v2\/posts\/113\/revisions\/116"}],"wp:attachment":[{"href":"https:\/\/blogs.kent.ac.uk\/unseenit\/wp-json\/wp\/v2\/media?parent=113"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blogs.kent.ac.uk\/unseenit\/wp-json\/wp\/v2\/categories?post=113"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blogs.kent.ac.uk\/unseenit\/wp-json\/wp\/v2\/tags?post=113"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}