So the case was that we had MetroCluster with 3040 heads. New 3240 heads came out and since 3040 was old enough, we decided to replace it with new ones.
First we thought that "ok, we can just takeover the other side -> change the head -> give it back to a new head and we're good to go". Well, after reading some documentation it was pretty clear that you can't do this. I'm not sure what's the exact reason for it, but at least 3240 has newer NVRAM and 3040 <> 3240 can't sync each others NVRAMs (and usually MC heads should be pretty much same HW, same modules and things like that). Problem was that we have a whole lot of servers using this storage and (big surprise) our boss didn't like the total blackout idea, so we had to figure something else out. I wonder how big companies usually do head upgrades at first place(?)
PUFF and we got an idea.. ;) Just create a new MC and use same backend switches which we are using with old MC as well. Then we asked this from Netapp and they said that twin fabric MC is supported (so that both MC have their own backend switches). And twin fabric MC sharing same backend switches should be supported later this year -- or something like that at least..?
So, non-supported feature. And this twin fabric support is only "available" when both MetroClusters are using same head-types, so our 3040 + 3240 twin won't be supported. ever(?) :) One idea was to buy new backend switches for new MC, but since we didn't need any extra gear, we just wanted to save our money.
Remind, NON-SUPPORTED. Use with caution and if it fails, it fails and nobody won't help you :)
To start with, we we had 2 brand new shelves, so we used those to build both sides up locally (one local shelf per site to install ontap and built up root aggregate). This included all basic stuff.
Next step was to build zoning for backend (it's pretty much recommended anyway, so this was a good thing to do). Basically you have:
- 4 switches
- 1 port for ISL (E-port)
- 1 port for FCVI per switch
- 1-2 ports used by NA itself for connecting shelves per switch
- X ports for shelves per switch
No zoning for ISL ports at all.
Brocade configuration could look something like this:
zonecreate "FCVI_3040", "1,0; 2,0"
(first number (1) is domain id and second (0) is port number. Zone is for fabric, so it will automatically spread to partner switch as well. You have to define partner's ports on same zone. You can find domain ID's with 'switchsshow' -command.
zonecreate "FCVI_3240", "1,1; 2,1"
zonecreate "STORAGE_3040", "1,3; 1,4; 2,3; 2,4"
zonecreate "STORAGE_3240", "1,5; 1,6; 2,5; 2,6"
cfgcreate "SAN_FABRIC1", "FCVI_3040; FCVI_3240; STORAGE_3040; STORAGE_3240"
cfgsave
cfgenable "SAN_FABRIC1"
That should do it.
When zoning is done, power up both heads and see if they are ok and can see local and partner disks (eg. disk show, sysconfig -r, storage show disk -p and so on...). You need to assign both local disks and disks which you are going to use for mirroring the root aggregate. You can see unassigned disk with 'disk show -n' -command.
To assign disk, just type: disk assign switch2:5.16 -p 1 -s
This will assign port 5, disk 16 (on switch2) on pool 1 with systemid you defined. You can find systemid eg. by typing 'sysconfig'. After that it will automatically assign all disk on pool 1 (might take a minute or two).
When disks are assigned, check that everything seems ok, eg. sysconfig -r
toaster1> sysconfig -r
Aggregate aggr0 (online, raid_dp, mirrored) (block checksums)
Plex /aggr0/plex0 (online, normal, active, pool0)
RAID group /aggr0/plex0/rg0 (normal)
RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
dparity switch2:13.16 0d 1 0 FC:A 0 FCAL 15000 560000/1146880000 560208/1147307688
parity switch1:13.17 0c 1 1 FC:B 0 FCAL 15000 560000/1146880000 560208/1147307688
data switch1:13.18 0c 1 2 FC:B 0 FCAL 15000 560000/1146880000 560208/1147307688
data switch1:13.28 0c 1 12 FC:B 0 FCAL 15000 560000/1146880000 560208/1147307688
Plex /aggr0/plex6 (online, normal, active, pool1)
RAID group /aggr0/plex6/rg0 (normal)
RAID Disk Device HA SHELF BAY CHAN Pool Type RPM Used (MB/blks) Phys (MB/blks)
--------- ------ ------------- ---- ---- ---- ----- -------------- --------------
dparity switch1:10.16 0c 1 0 FC:B 1 FCAL 15000 560000/1146880000 560208/1147307688
parity switch2:10.17 0d 1 1 FC:A 1 FCAL 15000 560000/1146880000 560208/1147307688
data switch2:10.28 0d 1 12 FC:A 1 FCAL 15000 560000/1146880000 560208/1147307688
data switch2:10.27 0d 1 11 FC:A 1 FCAL 15000 560000/1146880000 560208/1147307688
And:
toaster1> storage show disk -p
PRIMARY PORT SECONDARY PORT SHELF BAY
--------------------- ---- --------------------- ---- ---------
switch1:10.16 B switch2:10.16 A 1 0
switch2:10.17 A switch1:10.17 B 1 1
switch1:10.18 B switch2:10.18 A 1 2
switch2:10.19 A switch1:10.19 B 1 3
switch2:10.20 A switch1:10.20 B 1 4
When everyhing seems to be ok, you can create mirror.
Just type: aggr mirror aggr0
It should automatically do it. You can use 'aggr mirror aggr0 -n' if you want to simulate what NA is trying to do (good thing to check it).
And to complete MC, just enable clustering (you need cf and cf_remote licenses to do this).
Type: cf enable
So basically it's all same things when you're building normal MC. Just do zoning and it should work.
And when MC is up, plan is just to migrate servers from old MC to new MC. When one shelf (aggregate) is empty, just remove shelf and plug it to a new one and keep going until migration is done. Maybe slow way to upgrade, but at least you can minimize downtime a bit.
And when MC is up, plan is just to migrate servers from old MC to new MC. When one shelf (aggregate) is empty, just remove shelf and plug it to a new one and keep going until migration is done. Maybe slow way to upgrade, but at least you can minimize downtime a bit.
