This is an old revision of the document!
This page documents my experiences in converting a 3-way RAIDZ1 pool into a 6-way RAIDZ2 pool (adding 3 new disks to the 3 original disk), with minimal downtime. The system boots off a UFS gmirror and this remains unchanged (though I have allocated equivalent space on the new disks to allow potential future conversion to ZFS).
server# gpart show => 34 1953522988 ada0 GPT (932G) 34 94 1 freebsd-boot (47K) 128 6291456 2 freebsd-ufs (3.0G) 6291584 6291456 3 freebsd-swap (3.0G) 12583040 4194304 4 freebsd-ufs (2.0G) 16777344 1936745678 5 freebsd-zfs (924G) => 34 1953522988 ada1 GPT (932G) 34 94 1 freebsd-boot (47K) 128 6291456 2 freebsd-ufs (3.0G) 6291584 6291456 3 freebsd-swap (3.0G) 12583040 4194304 4 freebsd-ufs (2.0G) 16777344 1936745678 5 freebsd-zfs (924G) => 34 1953525101 ada2 GPT (932G) 34 94 1 freebsd-boot (47K) 128 6291456 2 freebsd-ufs (3.0G) 6291584 6291456 3 freebsd-swap (3.0G) 12583040 4194304 4 freebsd-ufs (2.0G) 16777344 1936747791 5 freebsd-zfs (924G) server# zpool status -v pool: tank state: ONLINE status: The pool is formatted using an older on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on older software versions. scrub: scrub completed after 14h22m with 0 errors on Thu Oct 28 18:22:28 2010 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ada0p5 ONLINE 0 0 0 ada1p5 ONLINE 0 0 0 ada2p5 ONLINE 0 0 0 errors: No known data errors
server% gpart show => 34 1953525101 da0 GPT (932G) 34 94 1 freebsd-boot (47K) 128 10485760 2 freebsd-zfs (5.0G) 10485888 6291456 3 freebsd-swap (3.0G) 16777344 1936747791 5 freebsd-zfs (924G) => 34 1953525101 da1 GPT (932G) 34 94 1 freebsd-boot (47K) 128 10485760 2 freebsd-zfs (5.0G) 10485888 6291456 3 freebsd-swap (3.0G) 16777344 1936747791 5 freebsd-zfs (924G) => 34 1953522988 ada0 GPT (932G) 34 94 1 freebsd-boot (47K) 128 6291456 2 freebsd-ufs (3.0G) 6291584 6291456 3 freebsd-swap (3.0G) 12583040 4194304 4 freebsd-ufs (2.0G) 16777344 1936745678 5 freebsd-zfs (924G) => 34 1953522988 ada1 GPT (932G) 34 94 1 freebsd-boot (47K) 128 6291456 2 freebsd-ufs (3.0G) 6291584 6291456 3 freebsd-swap (3.0G) 12583040 4194304 4 freebsd-ufs (2.0G) 16777344 1936745678 5 freebsd-zfs (924G) => 34 1953525101 ada2 GPT (932G) 34 94 1 freebsd-boot (47K) 128 6291456 2 freebsd-ufs (3.0G) 6291584 6291456 3 freebsd-swap (3.0G) 12583040 4194304 4 freebsd-ufs (2.0G) 16777344 1936747791 5 freebsd-zfs (924G) => 34 1953525101 ada3 GPT (932G) 34 94 1 freebsd-boot (47K) 128 10485760 2 freebsd-zfs (5.0G) 10485888 6291456 3 freebsd-swap (3.0G) 16777344 1936747791 5 freebsd-zfs (924G) server% zpool status pool: tank state: ONLINE scrub: scrub completed after 3h54m with 0 errors on Sat Nov 6 16:35:22 2010 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz2 ONLINE 0 0 0 ada0p5 ONLINE 0 0 0 ada1p5 ONLINE 0 0 0 ada2p5 ONLINE 0 0 0 ada3p5 ONLINE 0 0 0 da0p5 ONLINE 0 0 0 da1p5 ONLINE 0 0 0 errors: No known data errors
In my case, I have root and swap on the same disks, so I needed to carve out space for that. Even if you use the disks solely for ZFS, it's probably a good idea to partition a couple of MB off the disks in case a replacement disk is slight smaller.