This is an old revision of the document!
This page documents my experiences in converting a 3-way RAIDZ1 pool into a 6-way RAIDZ2 pool (adding 3 new disks to the 3 original disk), with minimal downtime. The system boots off a UFS gmirror and this remains unchanged (though I have allocated equivalent space on the new disks to allow potential future conversion to ZFS).
server# gpart show
=> 34 1953522988 ada0 GPT (932G)
34 94 1 freebsd-boot (47K)
128 6291456 2 freebsd-ufs (3.0G)
6291584 6291456 3 freebsd-swap (3.0G)
12583040 4194304 4 freebsd-ufs (2.0G)
16777344 1936745678 5 freebsd-zfs (924G)
=> 34 1953522988 ada1 GPT (932G)
34 94 1 freebsd-boot (47K)
128 6291456 2 freebsd-ufs (3.0G)
6291584 6291456 3 freebsd-swap (3.0G)
12583040 4194304 4 freebsd-ufs (2.0G)
16777344 1936745678 5 freebsd-zfs (924G)
=> 34 1953525101 ada2 GPT (932G)
34 94 1 freebsd-boot (47K)
128 6291456 2 freebsd-ufs (3.0G)
6291584 6291456 3 freebsd-swap (3.0G)
12583040 4194304 4 freebsd-ufs (2.0G)
16777344 1936747791 5 freebsd-zfs (924G)
server# zpool status -v
pool: tank
state: ONLINE
status: The pool is formatted using an older on-disk format. The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the
pool will no longer be accessible on older software versions.
scrub: scrub completed after 14h22m with 0 errors on Thu Oct 28 18:22:28 2010
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1 ONLINE 0 0 0
ada0p5 ONLINE 0 0 0
ada1p5 ONLINE 0 0 0
ada2p5 ONLINE 0 0 0
errors: No known data errors
server% gpart show
=> 34 1953525101 da0 GPT (932G)
34 94 1 freebsd-boot (47K)
128 10485760 2 freebsd-zfs (5.0G)
10485888 6291456 3 freebsd-swap (3.0G)
16777344 1936747791 5 freebsd-zfs (924G)
=> 34 1953525101 da1 GPT (932G)
34 94 1 freebsd-boot (47K)
128 10485760 2 freebsd-zfs (5.0G)
10485888 6291456 3 freebsd-swap (3.0G)
16777344 1936747791 5 freebsd-zfs (924G)
=> 34 1953522988 ada0 GPT (932G)
34 94 1 freebsd-boot (47K)
128 6291456 2 freebsd-ufs (3.0G)
6291584 6291456 3 freebsd-swap (3.0G)
12583040 4194304 4 freebsd-ufs (2.0G)
16777344 1936745678 5 freebsd-zfs (924G)
=> 34 1953522988 ada1 GPT (932G)
34 94 1 freebsd-boot (47K)
128 6291456 2 freebsd-ufs (3.0G)
6291584 6291456 3 freebsd-swap (3.0G)
12583040 4194304 4 freebsd-ufs (2.0G)
16777344 1936745678 5 freebsd-zfs (924G)
=> 34 1953525101 ada2 GPT (932G)
34 94 1 freebsd-boot (47K)
128 6291456 2 freebsd-ufs (3.0G)
6291584 6291456 3 freebsd-swap (3.0G)
12583040 4194304 4 freebsd-ufs (2.0G)
16777344 1936747791 5 freebsd-zfs (924G)
=> 34 1953525101 ada3 GPT (932G)
34 94 1 freebsd-boot (47K)
128 10485760 2 freebsd-zfs (5.0G)
10485888 6291456 3 freebsd-swap (3.0G)
16777344 1936747791 5 freebsd-zfs (924G)
server% zpool status
pool: tank
state: ONLINE
scrub: scrub completed after 3h54m with 0 errors on Sat Nov 6 16:35:22 2010
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz2 ONLINE 0 0 0
ada0p5 ONLINE 0 0 0
ada1p5 ONLINE 0 0 0
ada2p5 ONLINE 0 0 0
ada3p5 ONLINE 0 0 0
da0p5 ONLINE 0 0 0
da1p5 ONLINE 0 0 0
errors: No known data errors
In my case, I have root and swap on the same disks, so I needed to carve out space for that. Even if you use the disks solely for ZFS, it's probably a good idea to partition a couple of MB off the disks in case a replacement disk is slight smaller.