Weekend Overtime Sucks.

Ok, so it goes like this. Crappy old third party SCSI drive fails in the staff mail server. No problem, we'll just replace it. Ok, no spares available in a decent time frame from any local suppliers. (It's an old 18G with an SCA connector). Right, next plan. Lets just remove the partner disk of the failed one from the mirror set, and reduce the size of the mirror. Afterall, The single 75G volume served both Staff and Students until a month ago, so staff wont miss ~18G or so.

Tonight rolls around, when I'm supposed to do this work. Ok, so I try putting a spare 36G into the slot where the 18G failed. BAD idea:

d20: Submirror of d21
    State: Needs maintenance 
    Invoke: metareplace d21 c2t3d0s1 <new device>
    Size: 143302506 blocks
    Stripe 0: (interlace: 32 blocks)
        Device              Start Block  Dbase State        Hot Spare
        c2t0d0s1                   0     No    Maintenance  
        c2t1d0s1                   0     No    Maintenance  
        c2t2d0s1                   0     No    Maintenance  
        c2t3d0s1                   0     No    Maintenance

For some reason, it cut power to the entire disk enclosure. Although, there are two enclosures, and half of the mirror set is in each. (4 18's striped in each enclosure, then mirrored.) No problem, after futzing with power in the enclosure, things come back. Good good. Right, New plan. I use the silly Solaris DiskSuite tools to remove the failed submirror from the mirror, and recreate a new stripeset with the 3 good disks. We'll copy to that, then do some fun data shuffling. No problem.

Start rsync. Wait. Wait some more. And some more. Go get food... Right ok, I'm back, and what do I find? but it copying the old student spool, sitting in cyrus/PRE_STUDENTMOVE/ - Mike, you're a dumbass. Lets see if I can get by for the rest of the evening without screwing up too badly again... *sigh*

UPDATE 2004/02/01: Finally...
... I can go home.

d21: Mirror
    Submirror 0: d20
      State: Okay         
    Submirror 1: d19
      State: Resyncing    
    Resync in progress: 11 % done

Still had to wait ages on rsync though....

[10:07] [/Work] [permanent link]