Swap out your old disks for some new ones with greater capacity

Published on: - 1 year, 4 months ago

An article tagged as: linux, nas, ubuntu, zfs

Twitter Google+ Facebook Reddit

Overview

Eventually your ZFS pool will no longer have enough storage for you. When this happens you will need to add some disks or replace your old disks with new larger ones. Since i no longer have any spare SATA ports, I am going to do the latter, replacing all my 2TB disks with 6TB ones.

This article will show you how to replace your existing disks with larger capacity ones on a ZFS pool that is using raidz.

The following assumptions are made regarding your setup:-

  • Your ZFS pool uses raidz1, raidz2 or raidz3
  • Your computer/server is running ZFS on Linux

Backup your data!!

Please make sure you have a full backup of your data before replacing any disks.

This article is a useful guide, but we cannot take any responsibility if anything goes wrong.

Get current pool information

Before you start swapping disks, you should gather as much information about your existing ZFS pools

$ zpool status -v

  pool: DUMPSTER
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: resilvered 0 in 0h0m with 0 errors on Tue Apr 12 09:24:45 2016
config:

        NAME        STATE     READ WRITE CKSUM
        DUMPSTER   ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            sdc2    ONLINE       0     0     0
            sdb2    ONLINE       0     0     0
            sdd2    ONLINE       0     0     0
            sdf2    ONLINE       0     0     0
            sdg2    ONLINE       0     0     0
            sde2    ONLINE       0     0     0

errors: No known data errors

We need some more detailed information, so will also do the following:-

$ sudo zdb

DUMPSTER:
    version: 5000
    name: 'DUMPSTER'
    state: 0
    txg: 10893419
    pool_guid: 66180509451502810599
    errata: 0
    hostid: 8399329
    hostname: 'bob'
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 66180509451502810599
        children[0]:
            type: 'raidz'
            id: 0
            guid: 4466980275526910809
            nparity: 1
            metaslab_array: 23
            metaslab_shift: 36
            ashift: 12
            asize: 11989479849984
            is_log: 0
            children[0]:
                type: 'disk'
                id: 0
                guid: 2277585469358930581
                path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9GB903918-part2'
                phys_path: '/dev/gptid/c1cbdbc7-00c8-11e1-a37e-f46d046fc73d'
                whole_disk: 0
                DTL: 84
            children[1]:
                type: 'disk'
                id: 1
                guid: 12154086360534719056
                path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2HGJ9KB705656-part2'
                phys_path: '/dev/gptid/c215d0a5-00c8-11e1-a37e-f46d046fc73d'
                whole_disk: 0
                DTL: 83
            children[2]:
                type: 'disk'
                id: 2
                guid: 99122293356262304622
                path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2HGJ9KB705655-part2'
                phys_path: '/dev/gptid/c258b10f-00c8-11e1-a37e-f46d046fc73d'
                whole_disk: 0
                DTL: 82
            children[3]:
                type: 'disk'
                id: 3
                guid: 9833596632931584564
                path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2HGJ9KB705653-part2'
                phys_path: '/dev/gptid/c2989f3a-00c8-11e1-a37e-f46d046fc73d'
                whole_disk: 0
                DTL: 81
            children[4]:
                type: 'disk'
                id: 4
                guid: 887129227780467899
                path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2H7J9GB903914-part2'
                phys_path: '/dev/gptid/c2da76e5-00c8-11e1-a37e-f46d046fc73d'
                whole_disk: 0
                DTL: 80
            children[5]:
                type: 'disk'
                id: 5
                guid: 1256734120672923987
                path: '/dev/disk/by-id/ata-SAMSUNG_HD204UI_S2HGJ9KB705654-part2'
                phys_path: '/dev/gptid/c32365af-00c8-11e1-a37e-f46d046fc73d'
                whole_disk: 0
                DTL: 79
    features_for_read:

You might want to save the above information about your disks for later reference.

The key information here is that my pool DUMPSTER is made up of 6 disks, with a raidz configuration. raidz, also known as raidz1, means that up to one disk can fail without losing any data.

Also, it is important to note the guid and path for each disk.

Configure expansion

We need to tell our pool to expand automatically once all the new disks have been added. We do this with the following:-

$ sudo zpool set autoexpand=on DUMPSTER

Swap out the disk

The method i am going to go through assumes that you don't have any spare SATA ports. So the current disk will be removed, and a new bigger disk will be put in its place.

Do the following for starters:-

  • Shutdown the server
  • Unplug and remove one of your current disks
  • Make a note of the serial number on the new disk
  • Put the new disk in its place, and connect it
  • Turn the server back on

Now check that status of the pool

$ sudo zpool status

  pool: DUMPSTER
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-4J
  scan: resilvered 0 in 0h0m with 0 errors on Tue Apr 12 09:24:45 2016
config:

        NAME                     STATE     READ WRITE CKSUM
        DUMPSTER                DEGRADED     0     0     0
          raidz1-0               DEGRADED     0     0     0
            2277585469358930581  UNAVAIL      0     0     0  was /dev/sdc2
            sdb2                 ONLINE       0     0     0
            sdd2                 ONLINE       0     0     0
            sdf2                 ONLINE       0     0     0
            sdg2                 ONLINE       0     0     0
            sde2                 ONLINE       0     0     0

errors: No known data errors

As you can see, the pool has a state of DEGRADED and the disk we removed is shown as UNAVAIL. Helpfully, this also shows the guid of the removed disk, being 2277585469358930581, and that the location of the removed disk was /dev/sdc2. We will need this information shortly.

Now we want to list all the disks to find the newly added one. Mine should be listed under /dev/sdc.

$ sudo fdisk -l
...
Disk /dev/sdc: 6001.2 GB, 6001175126016 bytes
255 heads, 63 sectors/track, 729601 cylinders, total 11721045168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk identifier: 0x00000000

Disk /dev/sdc doesn't contain a valid partition table
...

So the new 6TB disk is there but it isn't part of the pool yet.

Replace the disk

We have physically swapped our disk, but we need to tell our ZFS pool that we have replaced the old disk with a new one.

First, we need to find the path of the new disk:-

$ ls -la /dev/disk/by-id

...
lrwxrwxrwx 1 root root   9 Apr 14 10:31 ata-WDC_WD60EZRZ-00GZ5B1_WD-WX31D95N3KPA -> ../../sdc
...

This shows me that my new disk, which has a serial number of WX31D95N3KPA, has a path of /dev/disk/by-id/ata-WDC_WD60EZRZ-00GZ5B1_WD-WX31D95N3KPA

Create GPT table

Before our new disk can become part of the pool, it needs to have a GPT table. This can be created as follows:-

$ sudo parted /dev/disk/by-id/ata-WDC_WD60EZRZ-00GZ5B1_WD-WX31D95N3KPA

GNU Parted 2.3
Using /dev/sdc
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel GPT
(parted) q
Information: You may need to update /etc/fstab.

Now we are ready to make the new disk part of our pool. Make sure you replace the following with the guid of your old disk, and the path of your new disk.

$ sudo zpool replace DUMPSTER 2277585469358930581 /dev/disk/by-id/ata-WDC_WD60EZRZ-00GZ5B1_WD-WX31D95N3KPA

The new disk will now be resilvered. This process can take several hours. But you can check its progress:-

$ sudo zpool status

  pool: DUMPSTER
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Apr 14 10:31:36 2016
    74.0M scanned out of 10.6T at 5.28M/s, 11h8m to go
    11.9M resilvered, 0.00% done
config:

        NAME                                            STATE     READ WRITE CKSUM
        DUMPSTER                                       DEGRADED     0     0     0
          raidz1-0                                      DEGRADED     0     0     0
            replacing-0                                 UNAVAIL      0     0     0
              2277585469358930581                       UNAVAIL      0     0     0  was /dev/sdc2
              ata-WDC_WD60EZRZ-00GZ5B1_WD-WX31D95N3KPA  ONLINE       0     0     0  (resilvering)
            sdb2                                        ONLINE       0     0     0
            sdd2                                        ONLINE       0     0     0
            sdf2                                        ONLINE       0     0     0
            sdg2                                        ONLINE       0     0     0
            sde2                                        ONLINE       0     0     0

errors: No known data errors

This shows me that resilvering progress is going to take another 11 hours. This is only an estimate. I would not recommend using the pool whilst it is resilvering.

Check that it has worked

Once complete, the pool should return to an online state.

$ zpool status -v
  pool: DUMPSTER
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: resilvered 0 in 0h0m with 0 errors on Tue Apr 12 09:24:45 2016
config:

        NAME        STATE     READ WRITE CKSUM
        DUMPSTER   ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            sdc2    ONLINE       0     0     0
            sdb2    ONLINE       0     0     0
            sdd2    ONLINE       0     0     0
            sdf2    ONLINE       0     0     0
            sdg2    ONLINE       0     0     0
            sde2    ONLINE       0     0     0

errors: No known data errors

This process can now be repeated to replace as many of the disk as you need to. However, I will not get the bennefit of the larger capacity disks until all the old smaller ones have been replaced.

References