Overview
Eventually your ZFS pool will no longer have enough storage for you. When this happens you will need to add some disks or replace your old disks with new larger ones. Since i no longer have any spare SATA ports, I am going to do the latter, replacing all my 2TB disks with 6TB ones.
This article will show you how to replace your existing disks with larger capacity ones on a ZFS pool that is using raidz.
The following assumptions are made regarding your setup:-
- Your computer or server has ZFS on Linux installed.
- Your ZFS pool uses raidz1, raidz2 or raidz3
Backup your data!!
Please make sure you have a full backup of your data before replacing any disks. This article is a useful guide, but we cannot take any responsibility if anything goes wrong.
Get current pool information
Before you start swapping disks, you should gather as much information about your existing ZFS pools
zpool status -v pool: DUMPSTER
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: resilvered 0 in 0h0m with 0 errors on Tue Apr 12 09:24:45 2016
config:
NAME STATE READ WRITE CKSUM
DUMPSTER ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sdc2 ONLINE 0 0 0
sdb2 ONLINE 0 0 0
sdd2 ONLINE 0 0 0
sdf2 ONLINE 0 0 0
sdg2 ONLINE 0 0 0
sde2 ONLINE 0 0 0
errors: No known data errorsWe need some more detailed information, so will also do the following:-
sudo zdbDUMPSTER:
version: 5000
name: 'DUMPSTER'
state: 0
txg: 10893419
pool_guid: 66180509451502810599
errata: 0
hostid: 8399329
hostname: 'bob'
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 66180509451502810599
children[0]:
type: 'raidz'
id: 0
guid: 4466980275526910809
nparity: 1
metaslab_array: 23
metaslab_shift: 36
ashift: 12
asize: 11989479849984
is_log: 0
children[0]:
type: 'disk'
id: 0
guid: 2277585469358930581
path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9GB903918-part2'
phys_path: '/dev/gptid/c1cbdbc7-00c8-11e1-a37e-f46d046fc73d'
whole_disk: 0
DTL: 84
children[1]:
type: 'disk'
id: 1
guid: 12154086360534719056
path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2HGJ9KB705656-part2'
phys_path: '/dev/gptid/c215d0a5-00c8-11e1-a37e-f46d046fc73d'
whole_disk: 0
DTL: 83
children[2]:
type: 'disk'
id: 2
guid: 99122293356262304622
path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2HGJ9KB705655-part2'
phys_path: '/dev/gptid/c258b10f-00c8-11e1-a37e-f46d046fc73d'
whole_disk: 0
DTL: 82
children[3]:
type: 'disk'
id: 3
guid: 9833596632931584564
path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2HGJ9KB705653-part2'
phys_path: '/dev/gptid/c2989f3a-00c8-11e1-a37e-f46d046fc73d'
whole_disk: 0
DTL: 81
children[4]:
type: 'disk'
id: 4
guid: 887129227780467899
path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9GB903914-part2'
phys_path: '/dev/gptid/c2da76e5-00c8-11e1-a37e-f46d046fc73d'
whole_disk: 0
DTL: 80
children[5]:
type: 'disk'
id: 5
guid: 1256734120672923987
path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2HGJ9KB705654-part2'
phys_path: '/dev/gptid/c32365af-00c8-11e1-a37e-f46d046fc73d'
whole_disk: 0
DTL: 79
features_for_read:You might want to save the above information about your disks for later reference.
The key information here is that my pool DUMPSTER is made up of 6 disks, with a raidz configuration. raidz, also known as raidz1, means that up to one disk can fail without losing any data.
Also, it is important to note the guid and path for each disk.
Configure expansion
We need to tell our pool to expand automatically once all the new disks have been added. We do this with the following:-
sudo zpool set autoexpand=on DUMPSTERSwap out the disk
The method i am going to go through assumes that you don't have any spare SATA ports. So the current disk will be removed, and a new bigger disk will be put in its place.
Do the following for starters:-
- Shutdown the server
- Unplug and remove one of your current disks
- Make a note of the serial number on the new disk
- Put the new disk in its place, and connect it
- Turn the server back on
Now check that status of the pool.
sudo zpool status pool: DUMPSTER
state: DEGRADED
status: One or more devices could not be used because the label is missing or
invalid. Sufficient replicas exist for the pool to continue
functioning in a degraded state.
action: Replace the device using 'zpool replace'.
see: http://zfsonlinux.org/msg/ZFS-8000-4J
scan: resilvered 0 in 0h0m with 0 errors on Tue Apr 12 09:24:45 2016
config:
NAME STATE READ WRITE CKSUM
DUMPSTER DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
2277585469358930581 UNAVAIL 0 0 0 was /dev/sdc2
sdb2 ONLINE 0 0 0
sdd2 ONLINE 0 0 0
sdf2 ONLINE 0 0 0
sdg2 ONLINE 0 0 0
sde2 ONLINE 0 0 0
errors: No known data errorsAs you can see, the pool has a state of DEGRADED and the disk we removed is shown as UNAVAIL. Helpfully, this also shows the guid of the removed disk, being 2277585469358930581, and that the location of the removed disk was /dev/sdc2. We will need this information shortly.
Now we want to list all the disks to find the newly added one. Mine should be listed under /dev/sdc.
sudo fdisk -l...
Disk /dev/sdc: 6001.2 GB, 6001175126016 bytes
255 heads, 63 sectors/track, 729601 cylinders, total 11721045168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000
Disk /dev/sdc doesn't contain a valid partition table
...So the new 6TB disk is there but it isn't part of the pool yet.
Replace the disk
We have physically swapped our disk, but we need to tell our ZFS pool that we have replaced the old disk with a new one.
First, we need to find the path of the new disk:-
ls -la /dev/disk/by-id...
lrwxrwxrwx 1 root root 9 Apr 14 10:31 ata-WDC_WD60EZRZ-00GZ5B1_WD-WX31D95N3KPA -> ../../sdc
...This shows me that my new disk, which has a serial number of WX31D95N3KPA, has a path of _/dev/disk/by-id/ata-WDC_WD60EZRZ-00GZ5B1WD-WX31D95N3KPA
Create GPT table
Before our new disk can become part of the pool, it needs to have a GPT table. This can be created as follows:-
sudo parted /dev/disk/by-id/ata-WDC_WD60EZRZ-00GZ5B1_WD-WX31D95N3KPAGNU Parted 2.3
Using /dev/sdc
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel GPT
(parted) q
Information: You may need to update /etc/fstab.Now we are ready to make the new disk part of our pool. Make sure you replace the following with the guid of your old disk, and the path of your new disk.
sudo zpool replace DUMPSTER 2277585469358930581 /dev/disk/by-id/ata-WDC_WD60EZRZ-00GZ5B1_WD-WX31D95N3KPAThe new disk will now be resilvered. This process can take several hours. But you can check its progress:-
sudo zpool status pool: DUMPSTER
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Thu Apr 14 10:31:36 2016
74.0M scanned out of 10.6T at 5.28M/s, 11h8m to go
11.9M resilvered, 0.00% done
config:
NAME STATE READ WRITE CKSUM
DUMPSTER DEGRADED 0 0 0
raidz1-0 DEGRADED 0 0 0
replacing-0 UNAVAIL 0 0 0
2277585469358930581 UNAVAIL 0 0 0 was /dev/sdc2
ata-WDC_WD60EZRZ-00GZ5B1_WD-WX31D95N3KPA ONLINE 0 0 0 (resilvering)
sdb2 ONLINE 0 0 0
sdd2 ONLINE 0 0 0
sdf2 ONLINE 0 0 0
sdg2 ONLINE 0 0 0
sde2 ONLINE 0 0 0
errors: No known data errorsThis shows me that resilvering progress is going to take another 11 hours. This is only an estimate. I would not recommend using the pool whilst it is resilvering.
Check that is worked
Once complete, the pool should return to an online state.
zpool status -v pool: DUMPSTER
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: resilvered 0 in 0h0m with 0 errors on Tue Apr 12 09:24:45 2016
config:
NAME STATE READ WRITE CKSUM
DUMPSTER ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sdc2 ONLINE 0 0 0
sdb2 ONLINE 0 0 0
sdd2 ONLINE 0 0 0
sdf2 ONLINE 0 0 0
sdg2 ONLINE 0 0 0
sde2 ONLINE 0 0 0
errors: No known data errorsThis process can now be repeated to replace as many of the disk as you need to. However, I will not get the bennefit of the larger capacity disks until all the old smaller ones have been replaced.